MatchPessimisticB fails on DC2.1 data

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
1
• Team:
Data Release Production

Description

While processing DC2.1 data we found many instances of the astrometry matcher failing. Here an example from a log file:

 processCcd.calibrate.astrometry INFO: Purged 2008 sources, leaving 164 good sources processCcd.calibrate.astromRefObjLoader INFO: Loading reference objects using center (70.054077, -44.428199) and radius 0.18204469504951715 deg processCcd.calibrate.astromRefObjLoader INFO: Loaded 1506 reference objects processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references processCcd FATAL: Failed on dataId={'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}: RuntimeError: Unable to match sources Traceback (most recent call last):  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__  result = self.runTask(task, dataRef, kwargs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask  return task.runDataRef(dataRef, **kwargs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/processCcd.py", line 198, in runDataRef  icSourceCat=charRes.sourceCat,  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 540, in runDataRef  icSourceCat=icSourceCat,  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 670, in run  sourceCat=sourceCat,  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 152, in run  res = self.solve(exposure=exposure, sourceCat=sourceCat)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 234, in solve  match_tolerance=match_tolerance,  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 330, in _matchAndFitWcs  match_tolerance=match_tolerance,  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper  res = func(self, *args, **keyArgs)  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/matchPessimisticB.py", line 272, in matchObjectsToSources  raise RuntimeError("Unable to match sources") RuntimeError: Unable to match sources 

I tried to run with the optimsticB matcher and that seemed to succeed without any problems. In previous data challenges, the pessimisticB matcher was more robust to failures, so this error is confusing.

I am in the process of copying a problematic visit over to lsst-dev at /scratch/rearmstr/Run2.1i/w_2019_17. This visit had failures for detectors: 78, 81, 85, 89, 135, 139, 150, 181, 185.

Attachments

1. Figure_1.png
33 kB

Activity

Hide
Eli Rykoff added a comment -

So I see that only 33 reference sources made it through the selection: processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references, which is selecting on resolved (now the default). This seems low, and surprisingly lower than the number of good detected sources (I believe also selected to be unresolved sources?). Is this typical in this run, and have you checked the reference catalog if something is strange?

Show
Eli Rykoff added a comment - So I see that only 33 reference sources made it through the selection: processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references , which is selecting on resolved (now the default). This seems low, and surprisingly lower than the number of good detected sources (I believe also selected to be unresolved sources?). Is this typical in this run, and have you checked the reference catalog if something is strange?
Hide
Bob Armstrong added a comment -

I only looked at this single visit, but a number of them had a low number like this one, while others had 100-200. I haven't checked the reference catalog. I wasn't sure the best way to debug it.

Show
Bob Armstrong added a comment - I only looked at this single visit, but a number of them had a low number like this one, while others had 100-200. I haven't checked the reference catalog. I wasn't sure the best way to debug it.
Hide
John Swinbank added a comment -

Based on Slack, I think we're attacking this in two ways. First, Eli Rykoff is checking to see if his work on DM-18643 & DM-19304 could be relevant. If it isn't, the AP team (likely Chris Morrison, when he is back from his travels) will take it over and look at matcher issues.

Based on that, I'm assigning to Eli Rykoff; Eli, please pass it on when you're done!

Show
John Swinbank added a comment - Based on Slack, I think we're attacking this in two ways. First, Eli Rykoff is checking to see if his work on DM-18643 & DM-19304 could be relevant. If it isn't, the AP team (likely Chris Morrison , when he is back from his travels) will take it over and look at matcher issues. Based on that, I'm assigning to Eli Rykoff ; Eli, please pass it on when you're done!
Hide
Eli Rykoff added a comment -

So this is clearly a bug in the reference catalog. I loaded the reference objects in the region of the failed ccd logged above ({{dataId=

{'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}

}}), and made a simple plot of red points: all objects; blue points: those marked as ~refCat['resolved']. For some reason, most of this region the reference catalog either (a) has only galaxies (resolved) objects or (b) most of this region the stars are not properly flagged as such.

One could probably get this to run if you removed the reference catalog selection, but the real solution is to fix the reference catalog, which is outside the scope of DM.

Though we should also think about more helpful errors in this case, but the extreme down-selection of the reference catalog was the key clue.

Show
Eli Rykoff added a comment - So this is clearly a bug in the reference catalog. I loaded the reference objects in the region of the failed ccd logged above ({{dataId= {'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0} }}), and made a simple plot of red points: all objects; blue points: those marked as ~refCat ['resolved'] . For some reason, most of this region the reference catalog either (a) has only galaxies (resolved) objects or (b) most of this region the stars are not properly flagged as such. One could probably get this to run if you removed the reference catalog selection, but the real solution is to fix the reference catalog, which is outside the scope of DM. Though we should also think about more helpful errors in this case, but the extreme down-selection of the reference catalog was the key clue.
Hide
Eli Rykoff added a comment -

I don’t know to mark this done or won’t fix.

Show
Eli Rykoff added a comment - I don’t know to mark this done or won’t fix.
Hide
Michael Wood-Vasey added a comment -

Workflow -> Won't Fix

Show
Michael Wood-Vasey added a comment - Workflow -> Won't Fix
Hide
John Swinbank added a comment -

Thank you for the quick check, Eli Rykoff!

Since there was genuine effort expended on this (albeit only a little), we should claim it as “done”. I'll mark it as such.

Show
John Swinbank added a comment - Thank you for the quick check, Eli Rykoff ! Since there was genuine effort expended on this (albeit only a little), we should claim it as “done”. I'll mark it as such.

People

• Assignee:
Eli Rykoff
Reporter:
Bob Armstrong
Watchers:
Bob Armstrong, Colin Slater, Eli Rykoff, Heather Kelly, John Swinbank, Michael Wood-Vasey