Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
Description
While processing DC2.1 data we found many instances of the astrometry matcher failing. Here an example from a log file:
processCcd.calibrate.astrometry INFO: Purged 2008 sources, leaving 164 good sources
|
processCcd.calibrate.astromRefObjLoader INFO: Loading reference objects using center (70.054077, -44.428199) and radius 0.18204469504951715 deg
|
processCcd.calibrate.astromRefObjLoader INFO: Loaded 1506 reference objects
|
processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references
|
processCcd FATAL: Failed on dataId={'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}: RuntimeError: Unable to match sources
|
Traceback (most recent call last):
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 388, in __call__
|
result = self.runTask(task, dataRef, kwargs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/cmdLineTask.py", line 447, in runTask
|
return task.runDataRef(dataRef, **kwargs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/processCcd.py", line 198, in runDataRef
|
icSourceCat=charRes.sourceCat,
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 540, in runDataRef
|
icSourceCat=icSourceCat,
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_tasks/17.0.1-11-gf64fdeb0/python/lsst/pipe/tasks/calibrate.py", line 670, in run
|
sourceCat=sourceCat,
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 152, in run
|
res = self.solve(exposure=exposure, sourceCat=sourceCat)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 234, in solve
|
match_tolerance=match_tolerance,
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/astrometry.py", line 330, in _matchAndFitWcs
|
match_tolerance=match_tolerance,
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/pipe_base/17.0.1-2-g3e5d191+19/python/lsst/pipe/base/timer.py", line 150, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2019_17/stack/miniconda3-4.5.12-1172c30/Linux64/meas_astrom/17.0.1-8-g7a3d54a+6/python/lsst/meas/astrom/matchPessimisticB.py", line 272, in matchObjectsToSources
|
raise RuntimeError("Unable to match sources")
|
RuntimeError: Unable to match sources
|
I tried to run with the optimsticB matcher and that seemed to succeed without any problems. In previous data challenges, the pessimisticB matcher was more robust to failures, so this error is confusing.
I am in the process of copying a problematic visit over to lsst-dev at /scratch/rearmstr/Run2.1i/w_2019_17. This visit had failures for detectors: 78, 81, 85, 89, 135, 139, 150, 181, 185.
Attachments
Activity
I only looked at this single visit, but a number of them had a low number like this one, while others had 100-200. I haven't checked the reference catalog. I wasn't sure the best way to debug it.
Based on Slack, I think we're attacking this in two ways. First, Eli Rykoff is checking to see if his work on DM-18643 & DM-19304 could be relevant. If it isn't, the AP team (likely Chris Morrison [X], when he is back from his travels) will take it over and look at matcher issues.
Based on that, I'm assigning to Eli Rykoff; Eli, please pass it on when you're done!
So this is clearly a bug in the reference catalog. I loaded the reference objects in the region of the failed ccd logged above ({{dataId=
{'visit': 2188, 'detector': 185, 'run': '2188', 'raftName': 'R43', 'detectorName': 'S12', 'snap': 0}}}), and made a simple plot of red points: all objects; blue points: those marked as ~refCat['resolved']. For some reason, most of this region the reference catalog either (a) has only galaxies (resolved) objects or (b) most of this region the stars are not properly flagged as such.
One could probably get this to run if you removed the reference catalog selection, but the real solution is to fix the reference catalog, which is outside the scope of DM.
Though we should also think about more helpful errors in this case, but the extreme down-selection of the reference catalog was the key clue.
I don’t know to mark this done or won’t fix.
Workflow -> Won't Fix
Michael Wood-Vasey
added a comment - Workflow -> Won't Fix Thank you for the quick check, Eli Rykoff!
Since there was genuine effort expended on this (albeit only a little), we should claim it as “done”. I'll mark it as such.
So I see that only 33 reference sources made it through the selection: processCcd.calibrate.astrometry.referenceSelector INFO: Selected 33/1506 references, which is selecting on resolved (now the default). This seems low, and surprisingly lower than the number of good detected sources (I believe also selected to be unresolved sources?). Is this typical in this run, and have you checked the reference catalog if something is strange?