Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3442

Processing y-band HSC data fails in loading reference sources

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_astrom, pipe_tasks
    • Labels:
      None

      Description

      processCcd.py /lsst3/HSC/data/ --output /raid/price/test --id visit=904400 ccd=50
      [...]
      processCcd.calibrate.astrometry.solver.loadAN: Loading reference objects using center (1023.5, 2091) pix = Fk5Coord(319.8934727, -0.0006943, 2000.00) sky and radius 0.111920792477 deg
      processCcd FATAL: Failed on dataId={'taiObs': '2013-11-03', 'pointing': 672, 'visit': 904400, 'dateObs': '2013-11-03', 'filter': 'HSC-Y', 'field': 'STRIPE82L', 'ccd': 50, 'expTime': 30.0}: Could not find flux field(s) y_camFlux, y_flux
      Traceback (most recent call last):
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/cmdLineTask.py", line 320, in __call__
          result = task.run(dataRef, **kwargs)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processCcd.py", line 85, in run
          result = self.process(sensorRef, postIsrExposure)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/processImage.py", line 160, in process
          calib = self.calibrate.run(inputExposure, idFactory=idFactory)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/pipe_tasks/10.1-28-gf9582e4+2/python/lsst/pipe/tasks/calibrate.py", line 457, in run
          astromRet = self.astrometry.run(exposure, sources1)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 177, in run
          results = self.astrometry(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetAstrometry.py", line 292, in astrometry
          astrom = self.solver.determineWcs(sourceCat=sourceCat, exposure=exposure, bbox=bbox)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 409, in determineWcs
          return self.determineWcs2(sourceCat=sourceCat, **margs)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 437, in determineWcs2
          astrom = self.useKnownWcs(sourceCat, wcs=wcs, **kw)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/anetBasicAstrometry.py", line 308, in useKnownWcs
          calib = None,
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 173, in loadPixelBox
          loadRes = self.loadSkyCircle(ctrCoord, maxRadius, filterName)
        File "/home/lsstsw/stack/Linux64/pipe_base/10.1-4-g6ba0cc7+15/python/lsst/pipe/base/timer.py", line 118, in wrapper
          res = func(self, *args, **keyArgs)
        File "/home/lsstsw/stack/Linux64/meas_astrom/10.1-19-g6e01b25+5/python/lsst/meas/astrom/loadAstrometryNetObjects.py", line 141, in loadSkyCircle
          fluxField = getRefFluxField(schema=refCat.schema, filterName=filterName)
        File "/home/lsstsw/stack/Linux64/meas_algorithms/10.1-15-g0d3ecf6/python/lsst/meas/algorithms/loadReferenceObjects.py", line 40, in getRefFluxField
          raise RuntimeError("Could not find flux field(s) %s" % (", ".join(fluxFieldList)))
      RuntimeError: Could not find flux field(s) y_camFlux, y_flux
      

      We should be able to fix this by setting config parameters (e.g., calibrate.astrometry.solver.defaultFilter or calibrate.astrometry.solver.filterMap), but how do we keep that synched with the choice of reference catalog? And once we get past astrometry, we also have the same problem in photocal.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            Photocal needs a different solution (at least if you want to apply color terms).

            I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?

            Show
            rowen Russell Owen added a comment - Photocal needs a different solution (at least if you want to apply color terms). I guess you could argue that the same colorterm information could be used to map camera filter to ref cat filter, but is that what we really want?
            Hide
            price Paul Price added a comment -

            Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms.

            Show
            price Paul Price added a comment - Note to my future self: calibrate.photocal.photoCatName may not be None if applyColorTerms .
            Hide
            price Paul Price added a comment -

            We want the following configuration settings:

            root.calibrate.astrometry.solver.filterMap = {'y': 'z'}
            root.calibrate.photocal.applyColorTerms = True
            root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b"
            

            As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.

            Show
            price Paul Price added a comment - We want the following configuration settings: root.calibrate.astrometry.solver.filterMap = {'y': 'z'} root.calibrate.photocal.applyColorTerms = True root.calibrate.photocal.photoCatName = "sdss-dr9-fink-v5b" As mentioned, we need some way to set calibrate.astrometry.solver.filterMap and calibrate.photocal.photoCatName based on the reference catalog. Since we are currently using astrometry_net_data and there is no other catalog support, we could add a function that sets these based on the setup version of astrometry_net_data. Beyond that, this issue is part of the more general problem of how to keep configuration data in multiple places in sync. There are some suggestions for that out there already, but it should be part of the general discussion on Task etc. I therefore propose to only put a bandaid on the problem.
            Hide
            price Paul Price added a comment -

            Russell Owen, would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome.

            price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub
            commit 7bc113c086c0fac3161a9eded863804df091f8c4
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Aug 12 10:54:52 2015 -0500
             
                photocal: ensure essential data is present when applyColorTerms
                
                Putting the check in the Config.validate means we fail early
                with a clear error message.
             
             python/lsst/pipe/tasks/photoCal.py |   16 +++++++++++-----
             1 files changed, 11 insertions(+), 5 deletions(-)
             
            commit 4c56986bf922ceab73b42d0547ab44100c3f9047
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Aug 12 10:57:04 2015 -0500
             
                add functions for setting config from eups setups
                
                Both astrometry and photocal have settings that depend on which
                astrometry_net_data reference catalog is setup, so this provides
                a convenient means of setting them.
                
                This is a temporary solution until:
                1. The butler includes support for reference catalogs; and
                2. The configuration system supports coordinated changes.
             
             python/lsst/pipe/tasks/setConfigFromEups.py |   62 +++++++++++++++++++++++++++
             1 files changed, 62 insertions(+), 0 deletions(-)
             
             
            price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub
            commit cc1666bca5c65225944b9eb264734db85c7906d8
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Aug 12 12:38:16 2015 -0500
             
                config: activate color terms in photocal
             
             config/hsc/processCcd.py |    2 --
             config/processCcd.py     |    2 ++
             2 files changed, 2 insertions(+), 2 deletions(-)
             
            commit 3a2fbbe21fb35026309165a624f4b1e09ead8291
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Aug 12 12:39:21 2015 -0500
             
                config: set astrometry, photocal from eups
                
                Using new functionality in pipe_tasks to ensure these are set to
                the correct value according to the astrometry_net_data we're using.
             
             config/processCcd.py |    7 +++++++
             1 files changed, 7 insertions(+), 0 deletions(-)
            

            Show
            price Paul Price added a comment - Russell Owen , would you please review these changes? There are two commits in each of pipe_tasks and obs_subaru. Comments from others are also welcome. price@lsst-dev:~/LSST/pipe/tasks[tickets/DM-3442] $ git sub commit 7bc113c086c0fac3161a9eded863804df091f8c4 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:54:52 2015 -0500   photocal: ensure essential data is present when applyColorTerms Putting the check in the Config.validate means we fail early with a clear error message.   python/lsst/pipe/tasks/photoCal.py | 16 +++++++++++----- 1 files changed, 11 insertions(+), 5 deletions(-)   commit 4c56986bf922ceab73b42d0547ab44100c3f9047 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 10:57:04 2015 -0500   add functions for setting config from eups setups Both astrometry and photocal have settings that depend on which astrometry_net_data reference catalog is setup, so this provides a convenient means of setting them. This is a temporary solution until: 1. The butler includes support for reference catalogs; and 2. The configuration system supports coordinated changes.   python/lsst/pipe/tasks/setConfigFromEups.py | 62 +++++++++++++++++++++++++++ 1 files changed, 62 insertions(+), 0 deletions(-)     price@lsst-dev:~/LSST/obs/subaru[tickets/DM-3442] $ git sub commit cc1666bca5c65225944b9eb264734db85c7906d8 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:38:16 2015 -0500   config: activate color terms in photocal   config/hsc/processCcd.py | 2 -- config/processCcd.py | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-)   commit 3a2fbbe21fb35026309165a624f4b1e09ead8291 Author: Paul Price <price@astro.princeton.edu> Date: Wed Aug 12 12:39:21 2015 -0500   config: set astrometry, photocal from eups Using new functionality in pipe_tasks to ensure these are set to the correct value according to the astrometry_net_data we're using.   config/processCcd.py | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
            Hide
            rowen Russell Owen added a comment -

            This looks like a good short-term fix.

            I only have two minor requests for setConfigFromEups.py:

            • Please document the astrometryMenu argument of setConfigFromEups.
            • Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.
            Show
            rowen Russell Owen added a comment - This looks like a good short-term fix. I only have two minor requests for setConfigFromEups.py: Please document the astrometryMenu argument of setConfigFromEups. Please use the same name for the "menu" argument everywhere (e.g. setConfigFromEups and setAstrometryConfigFromEups) unless there's a strong reason not to do so.
            Hide
            price Paul Price added a comment -

            Thanks, Russell Owen.

            As discussed on HipChat, I documented the astrometryMenu, but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks, Russell Owen . As discussed on HipChat, I documented the astrometryMenu , but would prefer not to rename menu in setAstrometryConfigFromEups to astrometryMenu because it's unnecessary. Merged to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Russell Owen
              Watchers:
              Jim Bosch, Lauren MacArthur, Paul Price, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.