Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23640

Unify raw and calibration dataset types in gen2 mappers

    XMLWordPrintable

    Details

    • Urgent?:
      No

      Description

      DM-22655 re-emphasized to me how inconsistent our obs packages are regarding their datasetTypes for raw and calibration exposure datasets (bias/dark/flat), with pixel types ranging from I, U, F, and high level types of Image, MaskedImage, DecoratedImage, and Exposure. At the very least, we want one consistent type for `raw` so that converted gen2 repositories all use the same registry type for `raw`. If we can also make the calibration types self-consistent, so much the better.

      I would advocate for the pixel type for raw to be float32, as the first operations we perform on raws in `ISR` are floating point (and as Merlin Fisher-Levine and Colin Slater have pointed out to me, all of the ISR corrections are dealing with things that happen before the ADC), so doing the conversion immediately may save us some trouble down the line, and saves us the trouble of ensuring that ISR calls convertIntToFloat before doing any math where the int/float difference might matter (e.g. obs_decam failing crosstalk if the raw type is non-float).

      Similarly, I would advocate for the high level type to be Exposure, which Tim Jenness attempted to fix on DM-19570. This would let us deprecate ensureExposure and possibly DecoratedImage.

      Even though we are planning to phase out the gen2 system, we want to be able to convert existing gen2 repos, and doing this will help ensure gen3 consistency as the gen3 registry type is typically pulled from the gen2 mapper type.

        Attachments

          Issue Links

            Activity

            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            As long as having the raw type be float isn't incompatible with them being ints on disk then I agree with all of the above. (I'm sure that's the case, but that's the only thing I wanted to check).

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - As long as having the raw type be float isn't incompatible with them being ints on disk then I agree with all of the above. (I'm sure that's the case, but that's the only thing I wanted to check).
            Hide
            Parejkoj John Parejko added a comment -

            Yes, this doesn't change on-disk representation, just the type you get when you e.g. butler.get('raw`).

            Show
            Parejkoj John Parejko added a comment - Yes, this doesn't change on-disk representation, just the type you get when you e.g. butler.get('raw`) .
            Hide
            tjenness Tim Jenness added a comment -

            John Parejko would you like to retarget this ticket as a Gen3 problem (I think you still had some issues with raw types) or would you like to retire the ticket as won't fix?

            Show
            tjenness Tim Jenness added a comment - John Parejko would you like to retarget this ticket as a Gen3 problem (I think you still had some issues with raw types) or would you like to retire the ticket as won't fix?
            Hide
            Parejkoj John Parejko added a comment -

            This is probably worth some more investigation: I also recall problems with int/float types in gen3, but I don't remember what tickets they were on.

            Show
            Parejkoj John Parejko added a comment - This is probably worth some more investigation: I also recall problems with int/float types in gen3, but I don't remember what tickets they were on.
            Hide
            tjenness Tim Jenness added a comment -

            Currently in gen3:

            • Default raw formatter reads with ImageU
            • Subaru explicitly forces ImageU.
            • obs_lsst uses native data type for reading a single amplifier. Assembled data are signed 32 bit int.
            • Raw CFHT data uses the native pixel data type (since raw files for CFHT can be either float or int).
            • Non raw FITS files read in whatever the native format is for the pixel data.
            Show
            tjenness Tim Jenness added a comment - Currently in gen3: Default raw formatter reads with ImageU Subaru explicitly forces ImageU. obs_lsst uses native data type for reading a single amplifier. Assembled data are signed 32 bit int. Raw CFHT data uses the native pixel data type (since raw files for CFHT can be either float or int). Non raw FITS files read in whatever the native format is for the pixel data.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              Parejkoj John Parejko
              Watchers:
              Christopher Waters, John Parejko, Lee Kelvin, Merlin Fisher-Levine, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins Builds

                  No builds found.