# Track down Gen3 processing failures in MergeMeasurementsTask

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
1
• Team:
Data Release Production

#### Description

Hsin-Fang Chiang writes:

The other SciPi error I saw was

 ValueError: Error in inputs to MergeCoaddMeasurements: source IDs do not match

But if I try to reproduce it directly, I'm seeing

 File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.0.0-7-g4840a288+1/python/lsst/pipe/tasks/mergeMeasurements.py", line 298, in run     mergedCatalog.reserve(len(orderedCatalogs[0]))  IndexError: list index out of range

This affects 19/79 quanta.  An example of one that failed is at

 /project/hchiang2/bps/submit/G3M19c/000001/drp/025/input/mmt/quantum083381.pickle

That's (tract=9615 patch=77).

#### Activity

Hide
Jim Bosch added a comment -

Problem appears to be happening earlier: the input deepCoadd_meas catalog for g in this patch has 18851 records, while those for all other bands have 18852.

Show
Jim Bosch added a comment - Problem appears to be happening earlier: the input deepCoadd_meas catalog for g in this patch has 18851 records, while those for all other bands have 18852.
Hide
Hsin-Fang Chiang added a comment -

(Corrected the number of affected quanta, from a new run fixing memory issue and ignoring two troublesome patches of DM-20695)

Show
Hsin-Fang Chiang added a comment - (Corrected the number of affected quanta, from a new run fixing memory issue and ignoring two troublesome patches of DM-20695 )
Hide
Jim Bosch added a comment -

Found it, and it's just a missing config override (which escaped our notice because the PipelineTask and CmdLineTask for coadd deblending are different, and just call the same subtask).  I've already pushed that to the master branch of the lsst-dm/gen3-hsc-rc2 repo, if you've been using that, but the relevant line is this one:

It looks like the configs in the lsst/ci_hsc_gen3 should already be getting that via a slightly different mechanism that isn't obviously better or worse (they use the overrides for the Gen2 Task and assume that those are valid for the Gen3 one, which happens to be true right now).

Show
Jim Bosch added a comment - Found it, and it's just a missing config override (which escaped our notice because the PipelineTask and CmdLineTask for coadd deblending are different, and just call the same subtask).  I've already pushed that to the master branch of the lsst-dm/gen3-hsc-rc2 repo, if you've been using that, but the relevant line is this one: https://github.com/lsst-dm/gen3-hsc-rc2/blob/master/config/deblendCoaddSourcesSingle.py#L4 It looks like the configs in the lsst/ci_hsc_gen3 should already be getting that via a slightly different mechanism that isn't obviously better or worse (they use the overrides for the Gen2 Task and assume that those are valid for the Gen3 one, which happens to be true right now).
Hide
Jim Bosch added a comment -

Update: the configs in ci_hsc_gen3 were not ok, because on the Gen2 side this is set in the CmdLineTask's setDefaults, not in the obs_subaru config override file it was loading.  And that highlights the fact that the right fix for Gen3 is clearly to put it in the PipelineTask's setDefaults, so there is now a (trivial) branch and PR for this ticket:

Show
Jim Bosch added a comment - Update: the configs in ci_hsc_gen3 were not ok, because on the Gen2 side this is set in the CmdLineTask's setDefaults, not in the obs_subaru config override file it was loading.  And that highlights the fact that the right fix for Gen3 is clearly to put it in the PipelineTask's setDefaults, so there is now a (trivial) branch and PR for this ticket: https://github.com/lsst/pipe_tasks/pull/304
Hide
Hsin-Fang Chiang added a comment -

Thanks for solving the mystery!

Show
Hsin-Fang Chiang added a comment - Thanks for solving the mystery!

#### People

Assignee:
Jim Bosch
Reporter:
Jim Bosch
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, Jim Bosch