Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_astrom, pipe_tasks
-
Labels:None
-
Story Points:2
-
Sprint:Science Pipelines DM-S15-6
-
Team:Data Release Production
Description
processCcd.py /lsst3/HSC/data/ --output /raid/price/test --id visit=904400 ccd=50
|
[...]
|
processCcd.calibrate.astrometry.solver.loadAN: Loading reference objects using center (1023.5, 2091) pix = Fk5Coord(319.8934727, -0.0006943, 2000.00) sky and radius 0.111920792477 deg
|
processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: Could not find flux field(s) y_camFlux, y_flux
|
Traceback (most recent call last):
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/cmdLineTask.py", line 320, in __call__
|
result = task.run(dataRef, **kwargs)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processCcd.py", line 85, in run
|
result = self.process(sensorRef, postIsrExposure)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processImage.py", line 160, in process
|
calib = self.calibrate.run(inputExposure, idFactory=idFactory)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/calibrate.py", line 457, in run
|
astromRet = self.astrometry.run(exposure, sources1)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 177, in run
|
results = self.astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 292, in astrometry
|
astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 409, in determineWcs
|
return self.determineWcs2(sourceCat=sourceCat, **margs)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 437, in determineWcs2
|
astrom = self.useKnownWcs(sourceCat, wcs=wcs, **kw)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 308, in useKnownWcs
|
calib = None,
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 173, in loadPixelBox
|
loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)
|
File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
|
res = func(self, *args, **keyArgs)
|
File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 141, in loadSkyCircle
|
fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName)
|
File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 40, in getRefFluxField
|
raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList)))
|
RuntimeError: Could not find flux field(s) y_camFlux, y_flux
|
We should be able to fix this by setting config parameters (e.g., calibrate.astrometry.solver.defaultFilter or calibrate.astrometry.solver.filterMap), but how do we keep that synched with the choice of reference catalog? And once we get past astrometry, we also have the same problem in photocal.
Attachments
Issue Links
- relates to
-
- Invalid
Activity
Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms.
Paul Price
added a comment - Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms . We want the following configuration settings:
root.calibrate.astrometry.solver.filterMap = {'y': 'z'}
|
root.calibrate.photocal.applyColorTerms = True
|
root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b"
|
As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.
Paul Price
added a comment - We want the following configuration settings:
root.calibrate.astrometry.solver.filterMap = {'y': 'z'}
root.calibrate.photocal.applyColorTerms = True
root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b"
As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem. Russell Owen, would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome.
price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub
|
commit 7bc113c086c0fac3161a9eded863804df091f8c4
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Wed Aug 12 10:54:52 2015 -0500
|
|
|
photocal: ensure essential data is present when applyColorTerms
|
|
Putting the check in the Config.validate means we fail early
|
with a clear error message.
|
|
|
python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++-----
|
1 files changed, 11 insertions(+), 5 deletions(-)
|
|
|
commit 4c56986bf922ceab73b42d0547ab44100c3f9047
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Wed Aug 12 10:57:04 2015 -0500
|
|
|
add functions for setting config from eups setups
|
|
Both astrometry and photocal have settings that depend on which
|
astrometry_net_data reference catalog is setup, so this provides
|
a convenient means of setting them.
|
|
This is a temporary solution until:
|
1. The butler includes support for reference catalogs; and
|
2. The configuration system supports coordinated changes.
|
|
|
python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++
|
1 files changed, 62 insertions(+), 0 deletions(-)
|
|
|
|
|
price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub
|
commit cc1666bca5c65225944b9eb264734db85c7906d8
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Wed Aug 12 12:38:16 2015 -0500
|
|
|
config: activate color terms in photocal
|
|
|
config/hsc/processCcd.py | 2 --
|
config/processCcd.py | 2 ++
|
2 files changed, 2 insertions(+), 2 deletions(-)
|
|
|
commit 3a2fbbe21fb35026309165a624f4b1e09ead8291
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Wed Aug 12 12:39:21 2015 -0500
|
|
|
config: set astrometry, photocal from eups
|
|
Using new functionality in pipe_tasks to ensure these are set to
|
the correct value according to the astrometry_net_data we're using.
|
|
|
config/processCcd.py | 7 +++++++
|
1 files changed, 7 insertions(+), 0 deletions(-)
|
Paul Price
added a comment - Russell Owen , would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome.
price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub
commit 7bc113c086c0fac3161a9eded863804df091f8c4
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 12 10:54:52 2015 -0500
photocal: ensure essential data is present when applyColorTerms
Putting the check in the Config.validate means we fail early
with a clear error message.
python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++-----
1 files changed, 11 insertions(+), 5 deletions(-)
commit 4c56986bf922ceab73b42d0547ab44100c3f9047
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 12 10:57:04 2015 -0500
add functions for setting config from eups setups
Both astrometry and photocal have settings that depend on which
astrometry_net_data reference catalog is setup, so this provides
a convenient means of setting them.
This is a temporary solution until:
1. The butler includes support for reference catalogs; and
2. The configuration system supports coordinated changes.
python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++
1 files changed, 62 insertions(+), 0 deletions(-)
price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub
commit cc1666bca5c65225944b9eb264734db85c7906d8
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 12 12:38:16 2015 -0500
config: activate color terms in photocal
config/hsc/processCcd.py | 2 --
config/processCcd.py | 2 ++
2 files changed, 2 insertions(+), 2 deletions(-)
commit 3a2fbbe21fb35026309165a624f4b1e09ead8291
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Aug 12 12:39:21 2015 -0500
config: set astrometry, photocal from eups
Using new functionality in pipe_tasks to ensure these are set to
the correct value according to the astrometry_net_data we're using.
config/processCcd.py | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)
This looks like a good short-term fix.
I only have two minor requests for setConfigFromEups.py:
- Please document the astrometryMenu argument of setConfigFromEups.
- Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.
Thanks, Russell Owen.
As discussed on HipChat, I documented the astrometryMenu, but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary.
Merged to master.
Paul Price
added a comment - Thanks, Russell Owen .
As discussed on HipChat, I documented the astrometryMenu , but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary.
Merged to master. 
Photocal needs a different solution (at least if you want to apply color terms).
I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?