Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5502

Collect usage of header metadata

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Story Points:
      5
    • Sprint:
      Alert Production F16 - 6, Alert Production F16 - 7
    • Team:
      Alert Production

      Description

      Collect a comprehensive set of exposure oriented metadata used by science code. This should also include metadata that is not currently needed but that could be utilized in the future. In practice, I suspect this will involve looking for all calls to PropertySet.get since that is how FITS header metadata is currently passed around.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            John Parejko and I had a discussion and concluded the following:

            • visit will become exposure ID, as Robert Lupton suggested
            • camera will be added to Detector as cameraName
            • airmass will probably be a function, but if folks think a value is better (e.g. if some project does a fancy computation instead of the usual altitude->airmass) then it will be a value and we will provide the usual function, so that obs_ packages can easily compute it when raw data is loaded
            • bbox will be added to ExposureInfo. It will be updated when an exposure is sliced or when Exposure.setXY0 is called. Users can shoot themselves in the foot by getting a MaskedImage or Image and calling setXY0 on that. We can document this as a warning but there is no practical way to prevent it.
            • field center will be added to ExposureInfo. This is the nominal ICRS of the field center of the field of view. Add the following note to the documentation: do NOT try to compute this from WCS and Detector, since WCS may have high order polynomial terms and so cannot be reliably extrapolated.
            • midTime will be moved from Calib to ExposureInfo and possibly renamed to midDate to make it clearer that it is not a duration

            Note that this data is not well normalized. We may want some sort of "focal plane exposure" object that contains the camera geometry, field center, date of exposure and other shared information, with some way to retrieve individual exposures from that object. In particular this could be useful for jointcal fitting transforms that are shared across all exposures for a field. However, until and unless we come up with such a thing, we will simply expand ExposureInfo as above.

            Show
            rowen Russell Owen added a comment - John Parejko and I had a discussion and concluded the following: visit will become exposure ID, as Robert Lupton suggested camera will be added to Detector as cameraName airmass will probably be a function, but if folks think a value is better (e.g. if some project does a fancy computation instead of the usual altitude->airmass) then it will be a value and we will provide the usual function, so that obs_ packages can easily compute it when raw data is loaded bbox will be added to ExposureInfo. It will be updated when an exposure is sliced or when Exposure.setXY0 is called. Users can shoot themselves in the foot by getting a MaskedImage or Image and calling setXY0 on that. We can document this as a warning but there is no practical way to prevent it. field center will be added to ExposureInfo. This is the nominal ICRS of the field center of the field of view. Add the following note to the documentation: do NOT try to compute this from WCS and Detector, since WCS may have high order polynomial terms and so cannot be reliably extrapolated. midTime will be moved from Calib to ExposureInfo and possibly renamed to midDate to make it clearer that it is not a duration Note that this data is not well normalized. We may want some sort of "focal plane exposure" object that contains the camera geometry, field center, date of exposure and other shared information, with some way to retrieve individual exposures from that object. In particular this could be useful for jointcal fitting transforms that are shared across all exposures for a field. However, until and unless we come up with such a thing, we will simply expand ExposureInfo as above.
            Hide
            rhl Robert Lupton added a comment -

            airmass will probably be a function, but if folks think a value is better (e.g. if some project does a fancy computation instead of the usual altitude->airmass) then it will be a value and we will provide the usual function, so that obs_ packages can easily compute it when raw data is loaded

            How are you thinking of doing this? The true airmass depends on atmospheric pressure and temperature (an approximation to the pressure scale height). You were going to assume a scale height and a plane-parallel atmosphere?
            I worry about blessing something that we know is wrong; assuming a canonical atmosphere and doing the geometry right might be an option (maybe with optional arguments).

            camera will be added to Detector as cameraName

            Does this allow a user to lookup the camera given the name? In python (and given a butler) you can ask for the camera; are we providing a way to do this in C++? Or is this cameraName just for jointcal's sake before we get a chance to fix the code to not need to know its camera?

            I'm worried about the comments about the Wcs. For now that's fine, but with the new design will the same problem be present, or will the code know its limitations? How can we calculate the boresight position ourselves if all we have is a Wcs and Detector?

            Show
            rhl Robert Lupton added a comment - airmass will probably be a function, but if folks think a value is better (e.g. if some project does a fancy computation instead of the usual altitude->airmass) then it will be a value and we will provide the usual function, so that obs_ packages can easily compute it when raw data is loaded How are you thinking of doing this? The true airmass depends on atmospheric pressure and temperature (an approximation to the pressure scale height). You were going to assume a scale height and a plane-parallel atmosphere? I worry about blessing something that we know is wrong; assuming a canonical atmosphere and doing the geometry right might be an option (maybe with optional arguments). camera will be added to Detector as cameraName Does this allow a user to lookup the camera given the name? In python (and given a butler) you can ask for the camera; are we providing a way to do this in C++? Or is this cameraName just for jointcal's sake before we get a chance to fix the code to not need to know its camera? I'm worried about the comments about the Wcs. For now that's fine, but with the new design will the same problem be present, or will the code know its limitations? How can we calculate the boresight position ourselves if all we have is a Wcs and Detector?
            Hide
            krughoff Simon Krughoff added a comment -

            This seems like a reasonable plan to me. A couple of things:

            1. I would like to see the need to carry around the cameraName in the detector dropped. It seems like it is only needed to get around the fact that we treat the metadata for different instruments differently. Since the implementation ticket for this proposal will solve that, I don't think we need it.
            2. Do we want to try to carry around what the OCS thought it was doing as well as what it actually did: i.e. the requested altitude was x, but the actual observed altitude was y? I think we just add these things using the extension mechanism if needed. I just wanted to bring it up in case I'm missing a major reason to put them in by default.

            Show
            krughoff Simon Krughoff added a comment - This seems like a reasonable plan to me. A couple of things: 1. I would like to see the need to carry around the cameraName in the detector dropped. It seems like it is only needed to get around the fact that we treat the metadata for different instruments differently. Since the implementation ticket for this proposal will solve that, I don't think we need it. 2. Do we want to try to carry around what the OCS thought it was doing as well as what it actually did: i.e. the requested altitude was x, but the actual observed altitude was y? I think we just add these things using the extension mechanism if needed. I just wanted to bring it up in case I'm missing a major reason to put them in by default.
            Hide
            rowen Russell Owen added a comment -

            I summarized the results of this ticket as RFC-199 with a detailed design. It adds some weather data, deals with the question of whether airmass should be for the center of the field or the center of the CCD and removes camera name.

            Regarding Simon Krughoff's question about recording what the OSC thought it was doing: I have punted on that. I think it is out of scope for ExposureInfo.

            Show
            rowen Russell Owen added a comment - I summarized the results of this ticket as RFC-199 with a detailed design. It adds some weather data, deals with the question of whether airmass should be for the center of the field or the center of the CCD and removes camera name. Regarding Simon Krughoff 's question about recording what the OSC thought it was doing: I have punted on that. I think it is out of scope for ExposureInfo.
            Hide
            Parejkoj John Parejko added a comment -

            We've removed cameraName from detector. I think we want to deal with the OCS nominal values via extensions: downstream processing should use the actual (fitted) values.

            Thanks for the comments, everyone. Closing this: please see RFC-199 for the final proposal.

            Show
            Parejkoj John Parejko added a comment - We've removed cameraName from detector. I think we want to deal with the OCS nominal values via extensions: downstream processing should use the actual (fitted) values. Thanks for the comments, everyone. Closing this: please see RFC-199 for the final proposal.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                krughoff Simon Krughoff
                Reviewers:
                Simon Krughoff
                Watchers:
                John Parejko, Robert Lupton, Russell Owen, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel