Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20705

Track down Gen3 processing failures in MergeMeasurementsTask

    XMLWordPrintable

    Details

      Description

      On Slack (https://lsstc.slack.com/archives/C2JPT1KB7/p1563485540182700), 

      Hsin-Fang Chiang writes:

      The other SciPi error I saw was

      ValueError: Error in inputs to MergeCoaddMeasurements: source IDs do not match 

      But if I try to reproduce it directly, I'm seeing

      File "/software/lsstsw/stack_20190330/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/18.0.0-7-g4840a288+1/python/lsst/pipe/tasks/mergeMeasurements.py", line 298, in run
          mergedCatalog.reserve(len(orderedCatalogs[0]))
       IndexError: list index out of range 

      This affects 19/79 quanta.  An example of one that failed is at

      /project/hchiang2/bps/submit/G3M19c/000001/drp/025/input/mmt/quantum083381.pickle 

      That's (tract=9615 patch=77).

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          Problem appears to be happening earlier: the input deepCoadd_meas catalog for g in this patch has 18851 records, while those for all other bands have 18852.

          Show
          jbosch Jim Bosch added a comment - Problem appears to be happening earlier: the input deepCoadd_meas catalog for g in this patch has 18851 records, while those for all other bands have 18852.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          (Corrected the number of affected quanta, from a new run fixing memory issue and ignoring two troublesome patches of DM-20695)

          Show
          hchiang2 Hsin-Fang Chiang added a comment - (Corrected the number of affected quanta, from a new run fixing memory issue and ignoring two troublesome patches of DM-20695 )
          Hide
          jbosch Jim Bosch added a comment -

          Found it, and it's just a missing config override (which escaped our notice because the PipelineTask and CmdLineTask for coadd deblending are different, and just call the same subtask).  I've already pushed that to the master branch of the lsst-dm/gen3-hsc-rc2 repo, if you've been using that, but the relevant line is this one:

          https://github.com/lsst-dm/gen3-hsc-rc2/blob/master/config/deblendCoaddSourcesSingle.py#L4

          It looks like the configs in the lsst/ci_hsc_gen3 should already be getting that via a slightly different mechanism that isn't obviously better or worse (they use the overrides for the Gen2 Task and assume that those are valid for the Gen3 one, which happens to be true right now).

          Show
          jbosch Jim Bosch added a comment - Found it, and it's just a missing config override (which escaped our notice because the PipelineTask and CmdLineTask for coadd deblending are different, and just call the same subtask).  I've already pushed that to the master branch of the lsst-dm/gen3-hsc-rc2 repo, if you've been using that, but the relevant line is this one: https://github.com/lsst-dm/gen3-hsc-rc2/blob/master/config/deblendCoaddSourcesSingle.py#L4 It looks like the configs in the lsst/ci_hsc_gen3 should already be getting that via a slightly different mechanism that isn't obviously better or worse (they use the overrides for the Gen2 Task and assume that those are valid for the Gen3 one, which happens to be true right now).
          Hide
          jbosch Jim Bosch added a comment -

          Update: the configs in ci_hsc_gen3 were not ok, because on the Gen2 side this is set in the CmdLineTask's setDefaults, not in the obs_subaru config override file it was loading.  And that highlights the fact that the right fix for Gen3 is clearly to put it in the PipelineTask's setDefaults, so there is now a (trivial) branch and PR for this ticket:

          https://github.com/lsst/pipe_tasks/pull/304

           

          Show
          jbosch Jim Bosch added a comment - Update: the configs in ci_hsc_gen3 were not ok, because on the Gen2 side this is set in the CmdLineTask's setDefaults, not in the obs_subaru config override file it was loading.  And that highlights the fact that the right fix for Gen3 is clearly to put it in the PipelineTask's setDefaults, so there is now a (trivial) branch and PR for this ticket: https://github.com/lsst/pipe_tasks/pull/304  
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thanks for solving the mystery! 

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thanks for solving the mystery! 

            People

            Assignee:
            jbosch Jim Bosch
            Reporter:
            jbosch Jim Bosch
            Reviewers:
            Hsin-Fang Chiang
            Watchers:
            Hsin-Fang Chiang, Jim Bosch
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.