It's not just a matter of tweaking the types in the HscMapper.paf file:

processCcd FATAL: Failed on dataId={'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata'

Traceback (most recent call last):

  File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/cmdLineTask.py", line 316, in __call__

    result = task.run(dataRef, **kwargs)

  File "/tigress/HSC/LSST/stack10_1/Linux64/pipe_base/10.1+4/python/lsst/pipe/base/timer.py", line 118, in wrapper

    res = func(self, *args, **keyArgs)

  File "/home/pprice/LSST/pipe/tasks/python/lsst/pipe/tasks/processCcd.py", line 80, in run

    postIsrExposure = self.isr.runDataRef(sensorRef).exposure

  File "/home/pprice/LSST/obs/subaru/python/lsst/obs/subaru/isr.py", line 219, in runDataRef

    biasExposure = self.getIsrExposure(sensorRef, "bias")

  File "/home/pprice/LSST/ip/isr/python/lsst/ip/isr/isrTask.py", line 545, in getIsrExposure

    raise RuntimeError("Unable to retrieve %s for %s: %s" % (datasetType, dataRef.dataId, e))

RuntimeError: Unable to retrieve bias for {'taiObs': '2013-11-02', 'pointing': 671, 'visit': 904020, 'dateObs': '2013-11-02', 'filter': 'HSC-I', 'field': 'STRIPE82L', 'ccd': 49, 'expTime': 30.0}: 'NoneType' object has no attribute 'getMetadata'

I think we're going to have to change some code.

Lauren, could you please review this? There's a single commit on tickets/DM-3143 of obs_subaru:

pprice@tiger-sumire:~/LSST/obs/subaru (tickets/DM-3143 %=) $ git --no-pager log --stat --reverse origin/master..

commit 922bfdd42533a81aa9e6a4a094d8ba044a89290c

Author: Paul Price <price@astro.princeton.edu>

Date:   Tue Jul 14 23:14:20 2015 -0400

    HscMapper: fix type of calibs to silence warnings

    afw now warns if we read an Image as a MaskedImage or Exposure.

    To silence the warning, we read the calib as the appropriate type

    (that it is actually on disk) and then convert it to an Exposure.

 policy/HscMapper.paf             | 12 ++++++------

 python/lsst/obs/hsc/hscMapper.py | 40 +++++++++++++++++++++++++++++++++++-----

 2 files changed, 41 insertions(+), 11 deletions(-)

This cetainly does the trick and you are free to merge, but I do have a few comments/querries.

It has taken me a while to wrap my head around why your original instinct of simply "tweaking of the types in the HscMapper.paf file" didn't work. I think part of my confusion lies in the naming and actual behaviour of the exposureFromImage function in dat_butlerUtils' cameraMapper.py. The doctstring desciption is: "Generate an exposure from a DecoratedImage or similar", but it seems that "or similar" effectively translates to "or Exposure" (any thus not really useful since that would imply you've already got an Exposure!) as these are the only two with the getMetadata attribute that gets called on the "image". This function is called within _standardizeExposure if the mapper's python field contains the string "Exposure" AND the persistable field contains the string "Image". Since both MaskedImage and Image contain Image, these types could be passed into the exposureFromImage function which will then fail on the call to image.getMetadata(). It's difficult to say for sure what the specific intended use was meant to be as the @param[in,out] item (lsst.afw.image.Exposure) docstring in _standardizeExposure implies the input should already be an Exposure. I wonder if this exposureFromImage function should behave more like your standardizeCalib function (i.e. to accommodate the Image and MaskedImage types as well)? Otherwise, might we run into the case that such "custom" standardization code will be duplicated in other similar cases (e.g other obs_XXX packages)? Sorry for the sidetrack...I realize this is out of the scope of this issue, but for the lesser-initiated this is clearly a potential source of confusion.

My other comment (actually related to your contribution!) is that I find your if/else clause a little bit misleading. You provide a raise RuntimeError("Unrecognised python type: %s" % mapping.python), but this would not catch any non-LSST defined image types that still contain the string "Image" (e.g. "bogusImage" would not be caught). However, I think any non-recognized type would be caught by lsst::daf::persistence::Formatter before this code is ever executed, so this point may be moot regardless.

One more quick comment, which is just to remind you of the following statement in the Python Coding Standard page: 'super' SHOULD NOT be used unless the author really understands the implications (e.g. in a well-understood multiple inheritance hierarchy). While I have no doubt you satisfy the caveat, do you really think the use of super is required here?

Thanks Lauren MacArthur.

This is certainly something that should be considered for other obs packages, but I don't think it's necessary right now because other obs packages aren't using our scripts for creating the calibration images and don't have our large existing library of calibrations, and the advent of the New Butler is on the horizon.

Regarding super, I want an object bound to the parent class so I can check its attributes, and that's what super provides. I could look for attributes of a hard-wired parent class, but I fear the magic that goes on in the butler because some attributes of CameraMapper are only added at instantiation time.

I propose to merge this as-is tomorrow.

Green light from me.

Thanks Lauren MacArthur. I rebased against origin/master and merged.