Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ci_hsc_gen3
-
Labels:None
-
Team:Data Release Production
-
Urgent?:No
Description
ci_hsc_gen3 takes a very long time to execute and there is a bottleneck during deblending where a single job blocks everything and takes around 1500 seconds to execute.
ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'deblend' on quantum {skymap: 'discrete/ci_hsc', tract: 0, patch: 69} took 1552.789 seconds
|
Given what ci_hsc_gen3 is really testing we don't really need to do all that work.
On Slack Fred Moolekamp suggested that the quickest fix might be to add a new config option for deblending that specifies the maximum number of sources to process.
Note also that we have a similar problem in obs_decam tests where DM-29762 reports:
4.25 minutes running nopytest_test_processCcd.py, much of it spent deblending 7
sources out of 303 into 2525(!!) children
Similarly in obs_decam the actual deblended sources aren't really part of what is being tested so capping deblending at 10 sources would help a lot. It's also possible that for this test dataset we have a bad configuration because 2500 children is probably telling us that something is broken.
Adding Sophie Reed to the conversation because we plan to test a DRP with fakes in ci_hsc soon too, and are thinking of ways to do so without doubling the runtime. See
DM-29110