Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3143

Fix warnings from processCcd.py on HSC

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_subaru
    • Labels:
      None

      Description

      When running processCcd.py on HSC data, I get the following warnings:

      afw.image.MaskedImage WARNING: Expected extension type not found: IMAGE
      : Empty WCS extension, using FITS header
      

      I expect these are from reading the bias/dark/flat as Exposure when they were written as Image. Probably just need to tweak the HscMapper.paf file.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          It's not just a matter of tweaking the types in the HscMapper.paf file:

          processCcd FATAL: Failed on dataId={'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata'
          Traceback (most recent call last):
            File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/cmdLineTask.py", line 316, in __call__
              result = task.run(dataRef, **kwargs)
            File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/timer.py", line 118, in wrapper
              res = func(self, *args, **keyArgs)
            File "/home/pprice/LSST/pipe/tasks/python/lsst/pipe/tasks/processCcd.py", line 80, in run
              postIsrExposure = self.isr.runDataRef(sensorRef).exposure
            File "/home/pprice/LSST/obs/subaru/python/lsst/obs/subaru/isr.py", line 219, in runDataRef
              biasExposure = self.getIsrExposure(sensorRef, "bias")
            File "/home/pprice/LSST/ip/isr/python/lsst/ip/isr/isrTask.py", line 545, in getIsrExposure
              raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, e))
          RuntimeError: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata'
          

          I think we're going to have to change some code.

          Show
          price Paul Price added a comment - It's not just a matter of tweaking the types in the HscMapper.paf file: processCcd FATAL: Failed on dataId={'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata' Traceback (most recent call last): File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/cmdLineTask.py", line 316, in __call__ result = task.run(dataRef, **kwargs) File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/timer.py", line 118, in wrapper res = func(self, *args, **keyArgs) File "/home/pprice/LSST/pipe/tasks/python/lsst/pipe/tasks/processCcd.py", line 80, in run postIsrExposure = self.isr.runDataRef(sensorRef).exposure File "/home/pprice/LSST/obs/subaru/python/lsst/obs/subaru/isr.py", line 219, in runDataRef biasExposure = self.getIsrExposure(sensorRef, "bias") File "/home/pprice/LSST/ip/isr/python/lsst/ip/isr/isrTask.py", line 545, in getIsrExposure raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, e)) RuntimeError: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata' I think we're going to have to change some code.
          Hide
          price Paul Price added a comment -

          Lauren, could you please review this? There's a single commit on tickets/DM-3143 of obs_subaru:

          pprice@tiger-sumire:~/LSST/obs/subaru (tickets/DM-3143 %=) $ git --no-pager log --stat --reverse origin/master..
          commit 922bfdd42533a81aa9e6a4a094d8ba044a89290c
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Tue Jul 14 23:14:20 2015 -0400
           
              HscMapper: fix type of calibs to silence warnings
              
              afw now warns if we read an Image as a MaskedImage or Exposure.
              To silence the warning, we read the calib as the appropriate type
              (that it is actually on disk) and then convert it to an Exposure.
           
           policy/HscMapper.paf             | 12 ++++++------
           python/lsst/obs/hsc/hscMapper.py | 40 +++++++++++++++++++++++++++++++++++-----
           2 files changed, 41 insertions(+), 11 deletions(-)
          

          Show
          price Paul Price added a comment - Lauren, could you please review this? There's a single commit on tickets/ DM-3143 of obs_subaru: pprice@tiger-sumire:~/LSST/obs/subaru (tickets/DM-3143 %=) $ git --no-pager log --stat --reverse origin/master.. commit 922bfdd42533a81aa9e6a4a094d8ba044a89290c Author: Paul Price <price@astro.princeton.edu> Date: Tue Jul 14 23:14:20 2015 -0400   HscMapper: fix type of calibs to silence warnings afw now warns if we read an Image as a MaskedImage or Exposure. To silence the warning, we read the calib as the appropriate type (that it is actually on disk) and then convert it to an Exposure.   policy/HscMapper.paf | 12 ++++++------ python/lsst/obs/hsc/hscMapper.py | 40 +++++++++++++++++++++++++++++++++++----- 2 files changed, 41 insertions(+), 11 deletions(-)
          Hide
          lauren Lauren MacArthur added a comment -

          This cetainly does the trick and you are free to merge, but I do have a few comments/querries.

          It has taken me a while to wrap my head around why your original instinct of simply "tweaking of the types in the HscMapper.paf file" didn't work. I think part of my confusion lies in the naming and actual behaviour of the exposureFromImage function in dat_butlerUtils' cameraMapper.py. The doctstring desciption is: "Generate an exposure from a DecoratedImage or similar", but it seems that "or similar" effectively translates to "or Exposure" (any thus not really useful since that would imply you've already got an Exposure!) as these are the only two with the getMetadata attribute that gets called on the "image". This function is called within _standardizeExposure if the mapper's python field contains the string "Exposure" AND the persistable field contains the string "Image". Since both MaskedImage and Image contain Image, these types could be passed into the exposureFromImage function which will then fail on the call to image.getMetadata(). It's difficult to say for sure what the specific intended use was meant to be as the @param[in,out] item (lsst.afw.image.Exposure) docstring in _standardizeExposure implies the input should already be an Exposure. I wonder if this exposureFromImage function should behave more like your standardizeCalib function (i.e. to accommodate the Image and MaskedImage types as well)? Otherwise, might we run into the case that such "custom" standardization code will be duplicated in other similar cases (e.g other obs_XXX packages)? Sorry for the sidetrack...I realize this is out of the scope of this issue, but for the lesser-initiated this is clearly a potential source of confusion.

          My other comment (actually related to your contribution!) is that I find your if/else clause a little bit misleading. You provide a raise RuntimeError("Unrecognised python type: %s" % mapping.python), but this would not catch any non-LSST defined image types that still contain the string "Image" (e.g. "bogusImage" would not be caught). However, I think any non-recognized type would be caught by lsst::daf::persistence::Formatter before this code is ever executed, so this point may be moot regardless.

          Show
          lauren Lauren MacArthur added a comment - This cetainly does the trick and you are free to merge, but I do have a few comments/querries. It has taken me a while to wrap my head around why your original instinct of simply "tweaking of the types in the HscMapper.paf file" didn't work. I think part of my confusion lies in the naming and actual behaviour of the exposureFromImage function in dat_butlerUtils ' cameraMapper.py . The doctstring desciption is: "Generate an exposure from a DecoratedImage or similar", but it seems that "or similar" effectively translates to "or Exposure" (any thus not really useful since that would imply you've already got an Exposure!) as these are the only two with the getMetadata attribute that gets called on the "image". This function is called within _standardizeExposure if the mapper's python field contains the string "Exposure" AND the persistable field contains the string "Image". Since both MaskedImage and Image contain Image, these types could be passed into the exposureFromImage function which will then fail on the call to image.getMetadata(). It's difficult to say for sure what the specific intended use was meant to be as the @param[in,out] item (lsst.afw.image.Exposure) docstring in _standardizeExposure implies the input should already be an Exposure. I wonder if this exposureFromImage function should behave more like your standardizeCalib function (i.e. to accommodate the Image and MaskedImage types as well)? Otherwise, might we run into the case that such "custom" standardization code will be duplicated in other similar cases (e.g other obs_XXX packages)? Sorry for the sidetrack...I realize this is out of the scope of this issue, but for the lesser-initiated this is clearly a potential source of confusion. My other comment (actually related to your contribution!) is that I find your if/else clause a little bit misleading. You provide a raise RuntimeError("Unrecognised python type: %s" % mapping.python) , but this would not catch any non-LSST defined image types that still contain the string "Image" (e.g. "bogusImage" would not be caught). However, I think any non-recognized type would be caught by lsst::daf::persistence::Formatter before this code is ever executed, so this point may be moot regardless.
          Hide
          lauren Lauren MacArthur added a comment -

          One more quick comment, which is just to remind you of the following statement in the Python Coding Standard page: 'super' SHOULD NOT be used unless the author really understands the implications (e.g. in a well-understood multiple inheritance hierarchy). While I have no doubt you satisfy the caveat, do you really think the use of super is required here?

          Show
          lauren Lauren MacArthur added a comment - One more quick comment, which is just to remind you of the following statement in the Python Coding Standard page: 'super' SHOULD NOT be used unless the author really understands the implications (e.g. in a well-understood multiple inheritance hierarchy). While I have no doubt you satisfy the caveat, do you really think the use of super is required here?
          Hide
          price Paul Price added a comment -

          Thanks Lauren MacArthur.

          This is certainly something that should be considered for other obs packages, but I don't think it's necessary right now because other obs packages aren't using our scripts for creating the calibration images and don't have our large existing library of calibrations, and the advent of the New Butler is on the horizon.

          Regarding super, I want an object bound to the parent class so I can check its attributes, and that's what super provides. I could look for attributes of a hard-wired parent class, but I fear the magic that goes on in the butler because some attributes of CameraMapper are only added at instantiation time.

          I propose to merge this as-is tomorrow.

          Show
          price Paul Price added a comment - Thanks Lauren MacArthur . This is certainly something that should be considered for other obs packages, but I don't think it's necessary right now because other obs packages aren't using our scripts for creating the calibration images and don't have our large existing library of calibrations, and the advent of the New Butler is on the horizon. Regarding super , I want an object bound to the parent class so I can check its attributes, and that's what super provides. I could look for attributes of a hard-wired parent class, but I fear the magic that goes on in the butler because some attributes of CameraMapper are only added at instantiation time. I propose to merge this as-is tomorrow.
          Hide
          lauren Lauren MacArthur added a comment -

          Green light from me.

          Show
          lauren Lauren MacArthur added a comment - Green light from me.
          Hide
          price Paul Price added a comment -

          Thanks Lauren MacArthur. I rebased against origin/master and merged.

          Show
          price Paul Price added a comment - Thanks Lauren MacArthur . I rebased against origin/master and merged.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Lauren MacArthur
              Watchers:
              John Swinbank, Lauren MacArthur, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel