Details
-
Type:
Bug
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: astrometry_net, meas_astrom
-
Labels:None
-
Epic Link:
-
Team:SQuaRE
Description
Running processCcd on monocam data segfaults during astrometry.
Steps to reproduce:
- Download attached datafile.
- conda install v12.1 of the stack
- setup ticket branch
DM-8401of obs_monocam -
ingestImages.py path/to/downloaded/file/2016-05-12CAblank5_y4_70.fits
-
processCcd.py filename --rerun test_1 --id visit=1333 --doraise -c isr.doLinearize=False isr.doBias=False isr.doDark=False isr.doFlat=False
Should see segfault after
processCcd.calibrate.astrometry.solver INFO: Number of selected sources for astrometry : 552
|
Interestingly, running processCcd with -c calibrate.doAstrometry=False results in the astrometric solver still being run, and a segfault still occurring, but in a different place. Just mentioning as this may or may not be useful debug info.
Attachments
Issue Links
Activity
Ran this using the shared stack (ie, built from source with eups distrib, not installed through conda) on lsst-dev01. It burns CPU for several minutes trying to match, then ultimately fails but doesn't segfault:
processCcd.calibrate.astrometry.solver INFO: Number of selected sources for astrometry : 552
|
processCcd.calibrate.astrometry.solver WARN: Did not get an astrometric solution from Astrometry.net
|
Traceback (most recent call last):
|
File "/ssd/lsstsw/stack/Linux64/pipe_tasks/12.1+1/bin/processCcd.py", line 25, in <module>
|
ProcessCcdTask.parseAndRun()
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/cmdLineTask.py", line 472, in parseAndRun
|
resultList = taskRunner.run(parsedCmd)
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/cmdLineTask.py", line 208, in run
|
resultList = list(mapFunc(self, targetList))
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/cmdLineTask.py", line 343, in __call__
|
result = task.run(dataRef, **kwargs)
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/timer.py", line 121, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/ssd/lsstsw/stack/Linux64/pipe_tasks/12.1+1/python/lsst/pipe/tasks/processCcd.py", line 181, in run
|
icSourceCat = charRes.sourceCat,
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/timer.py", line 121, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/ssd/lsstsw/stack/Linux64/pipe_tasks/12.1+1/python/lsst/pipe/tasks/calibrate.py", line 383, in run
|
icSourceCat=icSourceCat,
|
File "/ssd/lsstsw/stack/Linux64/pipe_tasks/12.1+1/python/lsst/pipe/tasks/calibrate.py", line 462, in calibrate
|
sourceCat=sourceCat,
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/timer.py", line 121, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetAstrometry.py", line 195, in run
|
return self.solve(exposure=exposure, sourceCat=sourceCat)
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/timer.py", line 121, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetAstrometry.py", line 226, in solve
|
results = self._astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
|
File "/ssd/lsstsw/stack/Linux64/pipe_base/12.1+1/python/lsst/pipe/base/timer.py", line 121, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetAstrometry.py", line 391, in _astrometry
|
astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetBasicAstrometry.py", line 468, in determineWcs
|
return self.determineWcs2(sourceCat=sourceCat, **margs)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetBasicAstrometry.py", line 490, in determineWcs2
|
wcs, qa = self.getBlindWcsSolution(sourceCat, **kwargs)
|
File "/ssd/lsstsw/stack/Linux64/meas_astrom/12.1+1/python/lsst/meas/astrom/anetBasicAstrometry.py", line 578, in getBlindWcsSolution
|
raise RuntimeError("Unable to match sources with catalog.")
|
RuntimeError: Unable to match sources with catalog.
|
I'll see if I can replicate the segfault using the Conda stack.
Sorry, yes, you are correct, using astrometry_net_data sdss-dr9-fink-v5b.
And, whilst it's not the desired behaviour, the "correct" behaviour for this with the current state of the the astrometry_net matching, is this to sit and burn CPU and then crash.
Confirmed this fails with the stack installed through conda on lsst-dev01.
The segfault happens in astrometry_net's starxy_sort_by_flux, ie not in LSST written code. Easy to reproduce, without using Monocam: from a fresh stack installation, we can just run
$ python ${MEAS_ASTROM_DIR}/tests/testCreateWcsWithSip.py
|
which segfaults within a fraction of a second.
I built and installed my own astrometry_net into the conda-provided stack. That is:
$ git clone https://github.com/lsst/astrometry_net.git
|
$ cd astrometry_net
|
$ eupspkg -er fetch
|
$ eupspkg -er prep
|
$ eupspkg -er config
|
$ eupspkg -er build
|
$ eupspkg -er install
|
$ eupspkg -er decl -F
|
With that, everything seems to run fine (insofar as failing to match and throwing a RuntimeError on the Monocam data is "fine"; seems like it's expected, at least).
Given that the code in question wasn't written by LSST and that it works fine when not using the binary distribution, I'm not inclined to spend much more time trying to debug it. J Matt Peterson [X], do you have any idea what the problem might be? Perhaps this might be resolved in a future binary distribution?
Setting team to SQuaRE since this seems to be an issue with the conda packaging.
Hey J Matt Peterson [X], just watned to be sure you're aware of this. It's not an immediate crisis, but until we can get it resolved it renders binaries a lot less useful than they might otherwise be.
Using astrometry_net_data sdss-dr9-fink-v5b I surmise. Am I correct?