Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19645

MatchPessimisticB fails on DC2.1 data

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      While processing DC2.1 data we found many instances of the astrometry matcher failing. Here an example from a log file:

      processCcd.calibrate.astrometry INFO: Purged 2008 sources, leaving 164 good sources
      processCcd.calibrate.astromRefObjLoader INFO: Loading reference objects using center (70.054077, -44.428199) and radius 0.18204469504951715 deg
      processCcd.calibrate.astromRefObjLoader INFO: Loaded 1506 reference objects
      processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references
      processCcd FATAL: Failed on dataId={'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}: RuntimeError: Unable to match sources
      Traceback (most recent call last):
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
          result = self.runTask(task, dataRef, kwargs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
          return task.runDataRef(dataRef, **kwargs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/processCcd.py", line 198, in runDataRef
          icSourceCat=charRes.sourceCat,
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 540, in runDataRef
          icSourceCat=icSourceCat,
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 670, in run
          sourceCat=sourceCat,
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 152, in run
          res = self.solve(exposure=exposure, sourceCat=sourceCat)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 234, in solve
          match_tolerance=match_tolerance,
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 330, in _matchAndFitWcs
          match_tolerance=match_tolerance,
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/matchPessimisticB.py", line 272, in matchObjectsToSources
          raise RuntimeError("Unable to match sources")
      RuntimeError: Unable to match sources
      

      I tried to run with the optimsticB matcher and that seemed to succeed without any problems. In previous data challenges, the pessimisticB matcher was more robust to failures, so this error is confusing.

      I am in the process of copying a problematic visit over to lsst-dev at /scratch/rearmstr/Run2.1i/w_2019_17. This visit had failures for detectors: 78, 81, 85, 89, 135, 139, 150, 181, 185.

        Attachments

          Activity

          Hide
          erykoff Eli Rykoff added a comment -

          So I see that only 33 reference sources made it through the selection: processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references, which is selecting on resolved (now the default). This seems low, and surprisingly lower than the number of good detected sources (I believe also selected to be unresolved sources?). Is this typical in this run, and have you checked the reference catalog if something is strange?

          Show
          erykoff Eli Rykoff added a comment - So I see that only 33 reference sources made it through the selection: processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references , which is selecting on resolved (now the default). This seems low, and surprisingly lower than the number of good detected sources (I believe also selected to be unresolved sources?). Is this typical in this run, and have you checked the reference catalog if something is strange?
          Hide
          rearmstr Bob Armstrong added a comment -

          I only looked at this single visit, but a number of them had a low number like this one, while others had 100-200. I haven't checked the reference catalog. I wasn't sure the best way to debug it.

          Show
          rearmstr Bob Armstrong added a comment - I only looked at this single visit, but a number of them had a low number like this one, while others had 100-200. I haven't checked the reference catalog. I wasn't sure the best way to debug it.
          Hide
          swinbank John Swinbank added a comment -

          Based on Slack, I think we're attacking this in two ways. First, Eli Rykoff is checking to see if his work on DM-18643 & DM-19304 could be relevant. If it isn't, the AP team (likely Chris Morrison, when he is back from his travels) will take it over and look at matcher issues.

          Based on that, I'm assigning to Eli Rykoff; Eli, please pass it on when you're done!

          Show
          swinbank John Swinbank added a comment - Based on Slack, I think we're attacking this in two ways. First, Eli Rykoff is checking to see if his work on DM-18643 & DM-19304 could be relevant. If it isn't, the AP team (likely Chris Morrison , when he is back from his travels) will take it over and look at matcher issues. Based on that, I'm assigning to Eli Rykoff ; Eli, please pass it on when you're done!
          Hide
          erykoff Eli Rykoff added a comment -

          So this is clearly a bug in the reference catalog. I loaded the reference objects in the region of the failed ccd logged above ({{dataId=

          {'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}

          }}), and made a simple plot of red points: all objects; blue points: those marked as ~refCat['resolved']. For some reason, most of this region the reference catalog either (a) has only galaxies (resolved) objects or (b) most of this region the stars are not properly flagged as such.

          One could probably get this to run if you removed the reference catalog selection, but the real solution is to fix the reference catalog, which is outside the scope of DM.

          Though we should also think about more helpful errors in this case, but the extreme down-selection of the reference catalog was the key clue.

          Show
          erykoff Eli Rykoff added a comment - So this is clearly a bug in the reference catalog. I loaded the reference objects in the region of the failed ccd logged above ({{dataId= {'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0} }}), and made a simple plot of red points: all objects; blue points: those marked as ~refCat ['resolved'] . For some reason, most of this region the reference catalog either (a) has only galaxies (resolved) objects or (b) most of this region the stars are not properly flagged as such. One could probably get this to run if you removed the reference catalog selection, but the real solution is to fix the reference catalog, which is outside the scope of DM. Though we should also think about more helpful errors in this case, but the extreme down-selection of the reference catalog was the key clue.
          Hide
          erykoff Eli Rykoff added a comment -

          I don’t know to mark this done or won’t fix.

          Show
          erykoff Eli Rykoff added a comment - I don’t know to mark this done or won’t fix.
          Hide
          wmwood-vasey Michael Wood-Vasey added a comment -

          Workflow -> Won't Fix

          Show
          wmwood-vasey Michael Wood-Vasey added a comment - Workflow -> Won't Fix
          Hide
          swinbank John Swinbank added a comment -

          Thank you for the quick check, Eli Rykoff!

          Since there was genuine effort expended on this (albeit only a little), we should claim it as “done”. I'll mark it as such.

          Show
          swinbank John Swinbank added a comment - Thank you for the quick check, Eli Rykoff ! Since there was genuine effort expended on this (albeit only a little), we should claim it as “done”. I'll mark it as such.

            People

            • Assignee:
              erykoff Eli Rykoff
              Reporter:
              rearmstr Bob Armstrong
              Watchers:
              Bob Armstrong, Colin Slater, Eli Rykoff, Heather Kelly, John Swinbank, Michael Wood-Vasey
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel