[DM-3442] Processing y-band HSC data fails in loading reference sources - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: meas_astrom, pipe_tasks
Labels:
None

Story Points:
2
Epic Link:
Test data development and HSC stack integration
Sprint:
Science Pipelines DM-S15-6
Team:
Data Release Production

Description

processCcd.py /lsst3/HSC/data/ --output /raid/price/test --id visit=904400 ccd=50

[...]

processCcd.calibrate.astrometry.solver.loadAN: Loading reference objects using center (1023.5, 2091) pix = Fk5Coord(319.8934727, -0.0006943, 2000.00) sky and radius 0.111920792477 deg

processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: Could not find flux field(s) y_camFlux, y_flux

Traceback (most recent call last):

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/cmdLineTask.py", line 320, in __call__

    result = task.run(dataRef, **kwargs)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processCcd.py", line 85, in run

    result = self.process(sensorRef, postIsrExposure)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processImage.py", line 160, in process

    calib = self.calibrate.run(inputExposure, idFactory=idFactory)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/calibrate.py", line 457, in run

    astromRet = self.astrometry.run(exposure, sources1)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 177, in run

    results = self.astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 292, in astrometry

    astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 409, in determineWcs

    return self.determineWcs2(sourceCat=sourceCat, **margs)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 437, in determineWcs2

    astrom = self.useKnownWcs(sourceCat, wcs=wcs, **kw)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 308, in useKnownWcs

    calib = None,

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 173, in loadPixelBox

    loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)

  File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 141, in loadSkyCircle

    fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName)

  File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 40, in getRefFluxField

    raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList)))

RuntimeError: Could not find flux field(s) y_camFlux, y_flux

We should be able to fix this by setting config parameters (e.g., calibrate.astrometry.solver.defaultFilter or calibrate.astrometry.solver.filterMap), but how do we keep that synched with the choice of reference catalog? And once we get past astrometry, we also have the same problem in photocal.

Attachments

Issue Links

relates to

DM-3445 Overhaul the way the right colorterm dictionary is selected

Invalid

Activity

Ascending order - Click to sort in descending order

No builds found.

Paul Price created issue - 11/Aug/15 3:37 PM

Hide

Permalink

Russell Owen added a comment - 11/Aug/15 4:34 PM

Photocal needs a different solution (at least if you want to apply color terms).

I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?

Show

Russell Owen added a comment - 11/Aug/15 4:34 PM Photocal needs a different solution (at least if you want to apply color terms). I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?

Hide

Permalink

Paul Price added a comment - 11/Aug/15 5:06 PM

Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms.

Show

Paul Price added a comment - 11/Aug/15 5:06 PM Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms .

Hide

Permalink

Paul Price added a comment - 11/Aug/15 5:13 PM

We want the following configuration settings:

root.calibrate.astrometry.solver.filterMap = {'y': 'z'}

root.calibrate.photocal.applyColorTerms = True

root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b"

As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.

Show

Paul Price added a comment - 11/Aug/15 5:13 PM We want the following configuration settings: root.calibrate.astrometry.solver.filterMap = {'y': 'z'} root.calibrate.photocal.applyColorTerms = True root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b" As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.

Hide

Permalink

Paul Price added a comment - 12/Aug/15 10:43 AM

Russell Owen, would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome.

price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub

commit 7bc113c086c0fac3161a9eded863804df091f8c4

Author: Paul Price <price@astro.princeton.edu>

Date:   Wed Aug 12 10:54:52 2015 -0500

    photocal: ensure essential data is present when applyColorTerms

    Putting the check in the Config.validate means we fail early

    with a clear error message.

 python/lsst/pipe/tasks/photoCal.py |   16 +++++++++++-----

 1 files changed, 11 insertions(+), 5 deletions(-)

commit 4c56986bf922ceab73b42d0547ab44100c3f9047

Author: Paul Price <price@astro.princeton.edu>

Date:   Wed Aug 12 10:57:04 2015 -0500

    add functions for setting config from eups setups

    Both astrometry and photocal have settings that depend on which

    astrometry_net_data reference catalog is setup, so this provides

    a convenient means of setting them.

    This is a temporary solution until:

    1. The butler includes support for reference catalogs; and

    2. The configuration system supports coordinated changes.

 python/lsst/pipe/tasks/setConfigFromEups.py |   62 +++++++++++++++++++++++++++

 1 files changed, 62 insertions(+), 0 deletions(-)

price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub

commit cc1666bca5c65225944b9eb264734db85c7906d8

Author: Paul Price <price@astro.princeton.edu>

Date:   Wed Aug 12 12:38:16 2015 -0500

    config: activate color terms in photocal

 config/hsc/processCcd.py |    2 --

 config/processCcd.py     |    2 ++

 2 files changed, 2 insertions(+), 2 deletions(-)

commit 3a2fbbe21fb35026309165a624f4b1e09ead8291

Author: Paul Price <price@astro.princeton.edu>

Date:   Wed Aug 12 12:39:21 2015 -0500

    config: set astrometry, photocal from eups

    Using new functionality in pipe_tasks to ensure these are set to

    the correct value according to the astrometry_net_data we're using.

 config/processCcd.py |    7 +++++++

 1 files changed, 7 insertions(+), 0 deletions(-)

Show

Paul Price added a comment - 12/Aug/15 10:43 AM Russell Owen , would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome. price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub commit 7bc113c086c0fac3161a9eded863804df091f8c4 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:54:52 2015 -0500 photocal: ensure essential data is present when applyColorTerms Putting the check in the Config.validate means we fail early with a clear error message. python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-) commit 4c56986bf922ceab73b42d0547ab44100c3f9047 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:57:04 2015 -0500 add functions for setting config from eups setups Both astrometry and photocal have settings that depend on which astrometry_net_data reference catalog is setup, so this provides a convenient means of setting them. This is a temporary solution until: 1. The butler includes support for reference catalogs; and 2. The configuration system supports coordinated changes. python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++ 1 files changed, 62 insertions(+), 0 deletions(-) price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub commit cc1666bca5c65225944b9eb264734db85c7906d8 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:38:16 2015 -0500 config: activate color terms in photocal config/hsc/processCcd.py | 2 -- config/processCcd.py | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-) commit 3a2fbbe21fb35026309165a624f4b1e09ead8291 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:39:21 2015 -0500 config: set astrometry, photocal from eups Using new functionality in pipe_tasks to ensure these are set to the correct value according to the astrometry_net_data we're using. config/processCcd.py | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)

Paul Price made changes - 12/Aug/15 10:43 AM

Field	Original Value	New Value
Reviewers		Russell Owen [ rowen ]
Status	To Do [ 10001 ]	In Review [ 10004 ]

Hide

Permalink

Russell Owen added a comment - 12/Aug/15 2:36 PM

This looks like a good short-term fix.

I only have two minor requests for setConfigFromEups.py:

Please document the astrometryMenu argument of setConfigFromEups.
Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.

Show

Russell Owen added a comment - 12/Aug/15 2:36 PM This looks like a good short-term fix. I only have two minor requests for setConfigFromEups.py: Please document the astrometryMenu argument of setConfigFromEups. Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.

Russell Owen made changes - 12/Aug/15 2:36 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Russell Owen made changes - 12/Aug/15 2:53 PM

Link

This issue relates to ~~DM-3445~~ [ ~~DM-3445~~ ]

John Swinbank made changes - 13/Aug/15 1:20 PM

Epic Link

~~DM-1942~~ [ 16008 ]

John Swinbank made changes - 13/Aug/15 1:20 PM

Sprint		Science Pipelines DM-S15-6 [ 165 ]
Story Points		2

Hide

Permalink

Paul Price added a comment - 18/Aug/15 9:43 AM

Thanks, Russell Owen.

As discussed on HipChat, I documented the astrometryMenu, but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary.

Merged to master.

Show

Paul Price added a comment - 18/Aug/15 9:43 AM Thanks, Russell Owen . As discussed on HipChat, I documented the astrometryMenu , but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary. Merged to master.

Paul Price made changes - 18/Aug/15 9:43 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

John Swinbank made changes - 26/Feb/16 4:23 PM

Team

Data Release Production [ 10301 ]

People

Assignee:: Paul Price

Reporter:: Paul Price

Reviewers:: Russell Owen

Watchers:: Jim Bosch, Lauren MacArthur, Paul Price, Russell Owen

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 11/Aug/15 3:37 PM

Updated:: 26/Feb/16 4:23 PM

Resolved:: 18/Aug/15 9:43 AM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins