Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21851

Extend obs_*_data packages to include additional calibrations

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: obs_base
    • Labels:
      None
    • Team:
      Data Release Production

      Description

      RFC-595 outlined a new approach for human-curated calibration products, but its implementation tickets only covered defects.

      We should eventually also move at least cameras and transmission curves (and possibly brighter-fatter kernels) to something similar.  Each of these comes with its own complications, which will require new design work:

      • Transmission curves for non-LSST obs packages may have heterogeneous sources of truth (see e.g. https://github.com/lsst/obs_subaru/tree/master/hsc/transmission), and hence a need to (at least informally) record the history and provenance of files that may not be in a consistent format.
      • Brighter-fatter kernels are essentially binary data generated by task code, and in many respects our workflow for generating them may be much more like the workflow for "non-curated" master calibrations like biases and flats.  If we decide to put them in obs_*_data packages, we need to work out how information is transferred from task code that measures them to human-curated files (the same is true of defects that come from tasks like FindDefectsTask, actually).
      • Cameras have many different components that are updated by entirely different processes (coordinate transforms from simultaneous astrometric fitting, gains from photon transfer curves, identifiers from direct human editing of files, ...).

      I don't think all of this work is necessary for the Gen2-Gen3 transition - we have placeholders we can live with, and much of this may be easier to do after Gen2 is retired - but I'd feel better about that transition if we had a clearer vision of how curated calibrations ought to work in the future.  In particular, I'm curious about the following workflows:

      • Add one or more curated calibration products for an instrument to a Gen3 repo (this is all we have in Gen3 right now, in the form of daf.butler.Instrument.writeCuratedCalibrations).  If a particular calibration product already exists in the repo, do we need to be able to reuse it (possibly associating it with a different validity range), or will we just write duplicate files?
      • Run a Task (possibly but not necessarily a PipelineTask) that either generates curated calibration products flies directly or produces outputs that can be copied by a human into a curated calibration product file, and transfer those outputs to the obs_*_data package.
      • Run a PipelineTask that generates non-curated calibrations (e.g. flats) and save a machine-readable description of that run to the obs_*_data package.

      This is intended as an umbrella ticket for that work, with actual branches corresponding to linked spin-off tickets.  I suspect some such tickets already exist (e.g. tickets for converting more obs packages to use YAML for camera definition).  I'd appreciate it if everyone who knows about a relevant ticket could link it here somehow.

      I'm assigning this to Christopher Waters now because I think he's likely to take the lead on this sort of organization work while Merlin Fisher-Levine and Andrés Alejandro Plazas Malagón are focused on more specific Tasks.  I'm assuming Tim Jenness and Simon Krughoff won't have time to extend what they did with defects themselves, given their other responsibilities, but I wanted to make sure they have an opportunity to communicate any answers they had in mind to the questions posed above.

       

        Attachments

          Activity

          Hide
          krughoff Simon Krughoff added a comment -

          This is great to see. I certainly won't be able to do all of this myself. I will say that DM-21169 does take things a step further by generalizing the ingest step to all standardized text file representations of human curated calibration datasets.

          Show
          krughoff Simon Krughoff added a comment - This is great to see. I certainly won't be able to do all of this myself. I will say that DM-21169 does take things a step further by generalizing the ingest step to all standardized text file representations of human curated calibration datasets.

            People

            • Assignee:
              czw Christopher Waters
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Andrés Alejandro Plazas Malagón, Eli Rykoff, James Chiang, Jim Bosch, John Swinbank, Merlin Fisher-Levine, Paul Price, Robert Lupton, Simon Krughoff, Tim Jenness, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel