Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-37129

Store list of calibrations used in output header

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production

      Description

      It would be helpful for calibration debugging if some identifier for the calibrations used was written to the output header.  I'm not sure what this identifier should look like (source collection + unique file id?), but being able to clearly determine which calibrations were used needs something like this.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          You don't need the RUN if you are storing the UUID, but this does bring up some broader issues of provenance that Jim Bosch and Gregory Dubois-Felsmann may want to comment on. In particular a UUID is not necessarily useful unless you also know the butler repository it came from. In the web services side of things we are using a URI to refer to a repository + UUID combination to make it clearer where a dataset can be retrieved from (and in the longer term allow that URI to be used in a web service to return the relevant dataset).

          There is a plan for butler to fix this provenance issue more directly such that you should consider what you are doing as more of a temporary fix than something that we should be supporting longer term.

          Show
          tjenness Tim Jenness added a comment - You don't need the RUN if you are storing the UUID, but this does bring up some broader issues of provenance that Jim Bosch and Gregory Dubois-Felsmann may want to comment on. In particular a UUID is not necessarily useful unless you also know the butler repository it came from. In the web services side of things we are using a URI to refer to a repository + UUID combination to make it clearer where a dataset can be retrieved from (and in the longer term allow that URI to be used in a web service to return the relevant dataset). There is a plan for butler to fix this provenance issue more directly such that you should consider what you are doing as more of a temporary fix than something that we should be supporting longer term.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          Modulo Tim's concerns, the actual code looks good to me.

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - Modulo Tim's concerns, the actual code looks good to me.
          Hide
          czw Christopher Waters added a comment -

          I wanted to store both for different reasons;  The RUN stores a human identifiable string that should solve most cases (i.e., it's using RUN=official/standard/bias/name instead of RUN=my/new/test/bias).  Adding the UUID makes it possible to explicitly check that two ISR processes used the same calibration.  I think the way we're handling calibrations reduces the source butler repository issue, as we're generating and verifying the calibrations in one place, and then exporting/ingesting those calibrations elsewhere (unless the ingest ignores the UUID stored in the export yaml).

          I'm fine with this being a short term fix, but as more people are working with calib production, being able to point to the header to answer "which calibration was used" hopefully will prevent people from being blocked.  

          Show
          czw Christopher Waters added a comment - I wanted to store both for different reasons;  The RUN stores a human identifiable string that should solve most cases (i.e., it's using RUN=official/standard/bias/name instead of RUN=my/new/test/bias).  Adding the UUID makes it possible to explicitly check that two ISR processes used the same calibration.  I think the way we're handling calibrations reduces the source butler repository issue, as we're generating and verifying the calibrations in one place, and then exporting/ingesting those calibrations elsewhere (unless the ingest ignores the UUID stored in the export yaml). I'm fine with this being a short term fix, but as more people are working with calib production, being able to point to the header to answer "which calibration was used" hopefully will prevent people from being blocked.  

            People

            Assignee:
            czw Christopher Waters
            Reporter:
            czw Christopher Waters
            Reviewers:
            Merlin Fisher-Levine
            Watchers:
            Christopher Waters, Jim Bosch, Merlin Fisher-Levine, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.