Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29829

Reduce execution time of ci_hsc_gen3

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc_gen3
    • Labels:
      None
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      ci_hsc_gen3 takes a very long time to execute and there is a bottleneck during deblending where a single job blocks everything and takes around 1500 seconds to execute.

      ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'deblend' on quantum {skymap: 'discrete/ci_hsc', tract: 0, patch: 69} took 1552.789 seconds
      

      Given what ci_hsc_gen3 is really testing we don't really need to do all that work.

      On Slack Fred Moolekamp suggested that the quickest fix might be to add a new config option for deblending that specifies the maximum number of sources to process.

      Note also that we have a similar problem in obs_decam tests where DM-29762 reports:

      4.25 minutes running nopytest_test_processCcd.py, much of it spent deblending 7 sources out of 303 into 2525(!!) children

      Similarly in obs_decam the actual deblended sources aren't really part of what is being tested so capping deblending at 10 sources would help a lot. It's also possible that for this test dataset we have a bad configuration because 2500 children is probably telling us that something is broken.

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            Adding Sophie Reed to the conversation because we plan to test a DRP with fakes in ci_hsc soon too, and are thinking of ways to do so without doubling the runtime. See DM-29110

            Show
            yusra Yusra AlSayyad added a comment - Adding Sophie Reed to the conversation because we plan to test a DRP with fakes in ci_hsc soon too, and are thinking of ways to do so without doubling the runtime. See DM-29110
            Hide
            fred3m Fred Moolekamp added a comment - - edited

            This would be a different fix for obs_decam, since the test mentioned uses meas_deblender instead of meas_extensions_scarlet. But it makes sense to implement this there as well so that both deblenders will have this option.

            Show
            fred3m Fred Moolekamp added a comment - - edited This would be a different fix for obs_decam, since the test mentioned uses meas_deblender instead of meas_extensions_scarlet. But it makes sense to implement this there as well so that both deblenders will have this option.
            Hide
            fred3m Fred Moolekamp added a comment -

            This should be resolved by DM-29888.

            Show
            fred3m Fred Moolekamp added a comment - This should be resolved by DM-29888 .
            Hide
            yusra Yusra AlSayyad added a comment -

            Marking as a redundant of DM-29888 (no story points or assignee on this particular ticket)

            Show
            yusra Yusra AlSayyad added a comment - Marking as a redundant of DM-29888 (no story points or assignee on this particular ticket)

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              tjenness Tim Jenness
              Watchers:
              Fred Moolekamp, Jim Bosch, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.