XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2
• Sprint:
Science Pipelines DM-S15-6
• Team:
Data Release Production

#### Description

 processCcd.py /lsst3/HSC/data/ --output /raid/price/test --id visit=904400 ccd=50 [...] processCcd.calibrate.astrometry.solver.loadAN: Loading reference objects using center (1023.5, 2091) pix = Fk5Coord(319.8934727, -0.0006943, 2000.00) sky and radius 0.111920792477 deg processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: Could not find flux field(s) y_camFlux, y_flux Traceback (most recent call last):  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/cmdLineTask.py", line 320, in __call__  result = task.run(dataRef, **kwargs)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processCcd.py", line 85, in run  result = self.process(sensorRef, postIsrExposure)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processImage.py", line 160, in process  calib = self.calibrate.run(inputExposure, idFactory=idFactory)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/calibrate.py", line 457, in run  astromRet = self.astrometry.run(exposure, sources1)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 177, in run  results = self.astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 292, in astrometry  astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 409, in determineWcs  return self.determineWcs2(sourceCat=sourceCat, **margs)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 437, in determineWcs2  astrom = self.useKnownWcs(sourceCat, wcs=wcs, **kw)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 308, in useKnownWcs  calib = None,  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 173, in loadPixelBox  loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper  res = func(self, *args, **keyArgs)  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 141, in loadSkyCircle  fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName)  File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 40, in getRefFluxField  raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList))) RuntimeError: Could not find flux field(s) y_camFlux, y_flux 

We should be able to fix this by setting config parameters (e.g., calibrate.astrometry.solver.defaultFilter or calibrate.astrometry.solver.filterMap), but how do we keep that synched with the choice of reference catalog? And once we get past astrometry, we also have the same problem in photocal.

#### Activity

Hide
Russell Owen added a comment -

Photocal needs a different solution (at least if you want to apply color terms).

I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?

Show
Russell Owen added a comment - Photocal needs a different solution (at least if you want to apply color terms). I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?
Hide
Paul Price added a comment -

Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms.

Show
Paul Price added a comment - Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms .
Hide
Paul Price added a comment -

We want the following configuration settings:

 root.calibrate.astrometry.solver.filterMap = {'y': 'z'} root.calibrate.photocal.applyColorTerms = True root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b" 

As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.

Show
Paul Price added a comment - We want the following configuration settings: root.calibrate.astrometry.solver.filterMap = {'y': 'z'} root.calibrate.photocal.applyColorTerms = True root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b" As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.
Hide
Paul Price added a comment -

Russell Owen, would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome.

 price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $git sub commit 7bc113c086c0fac3161a9eded863804df091f8c4 Author: Paul Price  Date: Wed Aug 12 10:54:52 2015 -0500    photocal: ensure essential data is present when applyColorTerms    Putting the check in the Config.validate means we fail early  with a clear error message.    python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++-----  1 files changed, 11 insertions(+), 5 deletions(-)   commit 4c56986bf922ceab73b42d0547ab44100c3f9047 Author: Paul Price  Date: Wed Aug 12 10:57:04 2015 -0500    add functions for setting config from eups setups    Both astrometry and photocal have settings that depend on which  astrometry_net_data reference catalog is setup, so this provides  a convenient means of setting them.    This is a temporary solution until:  1. The butler includes support for reference catalogs; and  2. The configuration system supports coordinated changes.    python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++  1 files changed, 62 insertions(+), 0 deletions(-)     price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442]$ git sub commit cc1666bca5c65225944b9eb264734db85c7906d8 Author: Paul Price  Date: Wed Aug 12 12:38:16 2015 -0500    config: activate color terms in photocal    config/hsc/processCcd.py | 2 --  config/processCcd.py | 2 ++  2 files changed, 2 insertions(+), 2 deletions(-)   commit 3a2fbbe21fb35026309165a624f4b1e09ead8291 Author: Paul Price  Date: Wed Aug 12 12:39:21 2015 -0500    config: set astrometry, photocal from eups    Using new functionality in pipe_tasks to ensure these are set to  the correct value according to the astrometry_net_data we're using.    config/processCcd.py | 7 +++++++  1 files changed, 7 insertions(+), 0 deletions(-) 

Show
Paul Price added a comment - Russell Owen , would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome. price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $git sub commit 7bc113c086c0fac3161a9eded863804df091f8c4 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:54:52 2015 -0500 photocal: ensure essential data is present when applyColorTerms Putting the check in the Config.validate means we fail early with a clear error message. python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-) commit 4c56986bf922ceab73b42d0547ab44100c3f9047 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:57:04 2015 -0500 add functions for setting config from eups setups Both astrometry and photocal have settings that depend on which astrometry_net_data reference catalog is setup, so this provides a convenient means of setting them. This is a temporary solution until: 1. The butler includes support for reference catalogs; and 2. The configuration system supports coordinated changes. python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++ 1 files changed, 62 insertions(+), 0 deletions(-) price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442]$ git sub commit cc1666bca5c65225944b9eb264734db85c7906d8 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:38:16 2015 -0500   config: activate color terms in photocal   config/hsc/processCcd.py | 2 -- config/processCcd.py | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-)   commit 3a2fbbe21fb35026309165a624f4b1e09ead8291 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:39:21 2015 -0500   config: set astrometry, photocal from eups Using new functionality in pipe_tasks to ensure these are set to the correct value according to the astrometry_net_data we're using.   config/processCcd.py | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
Hide
Russell Owen added a comment -

This looks like a good short-term fix.

I only have two minor requests for setConfigFromEups.py:

• Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.
Show
Russell Owen added a comment - This looks like a good short-term fix. I only have two minor requests for setConfigFromEups.py: Please document the astrometryMenu argument of setConfigFromEups. Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.
Hide
Paul Price added a comment -

Thanks, Russell Owen.

As discussed on HipChat, I documented the astrometryMenu, but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary.

Merged to master.

Show
Paul Price added a comment - Thanks, Russell Owen . As discussed on HipChat, I documented the astrometryMenu , but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary. Merged to master.

#### People

Assignee:
Paul Price
Reporter:
Paul Price
Reviewers:
Russell Owen
Watchers:
Jim Bosch, Lauren MacArthur, Paul Price, Russell Owen