Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-726

Design DM implementation for LVV-57/validating calibration products.

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:
      None

      Description

      LVV-57 specifies the requirement that DM verify calibration products.  We currently do not have code for this, but will want to by the end of the year.  The goal of this RFC is to get my current thoughts down, and ensure that there's a consensus on how this will work.

      The first point is that there will be no intent to write direct Gen2 middleware implementation of LVV-57.  Gen3 is usable now, and has better calibration handling than Gen2, which should make performing the validation on newly constructed calibrations possible.

      Currently, the only calibration validation code is in `cp_pipe`.  This includes the debug code in measureCrosstalk, and the PTC plots in plotPtc. This is largely diagnostic code to track down bad input data and algorithm differences.  I don't think this framework is going to work in the long term, and so I think we'll want to add a new package to the stack to handle this work, tentatively named `cp_verify`.

      `cp_verify` should supply a set of pipelineTasks that implement all the tests defined in a final version of DMTN-101.  Each task should take as input the calibration to be validated, any post-`IsrTask` processed raw frames to be used for comparison, and optionally any pre-existing calibrations to compare the input calibration against.  These `lsst.cp.verify.CalibXyVerifyTask` will perform all the tests defined, and output a yaml dataset describing the status for those tests.  This will likely need some visualization code to view those datasets (checking for failures as a function of the amplifier/detector position on the focal plane).  In addition, the items recorded in those yaml datasets should be used to create proper `lsst.verify` metrics (as is done by `ap_verify`).  These tasks will provide the step necessary to validate a proposed calibration can be certified into the main Gen3 calibration collection.

      In addition to this validation mode, the same tasks will be able to provide a way to monitor performance over time, either by passing newly taken "raw" calibration frames to compare against, or by passing newly constructed proposed calibrations to compare against the current version in the calibration collection.  Combining this with the metrics code will provide an easily trackable way to identify jumps and drifts in the residual values over time.

      `ci_cpp` has not yet merged, but as the tests are defined in `cp_verify`, they should be used as part of the unit tests used in the `ci_cpp` packages, with failures used to identify ways to improve either the `cp_pipe` code or the scope of the `ci_cpp` test data.

      DMTN-101 currently describes tests to perform for bias and dark frames (among other calibrations), but is incomplete for some calibrations (including flats).  This needs to be finalized before `cp_verify` can claim to satisfy LVV-57.

      I've set the end date for this RFC to the end of the month, as any implementation is unlikely to start before then.  This should provide enough time to identify anything else that could be a blocker to `cp_verify` implementation.

       

        Attachments

          Issue Links

            Activity

            No builds found.
            czw Christopher Waters created issue -
            czw Christopher Waters made changes -
            Field Original Value New Value
            Summary Design DM implementation for LVV-57 Design DM implementation for LVV-57/validating calibration products.
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25765 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25831 ]
            Hide
            ebellm Eric Bellm added a comment -

            This sounds good to me. I assume that the lsst.verify metrics would be uploaded to SQuaSH by ci_cpp?

            Show
            ebellm Eric Bellm added a comment - This sounds good to me. I assume that the lsst.verify metrics would be uploaded to SQuaSH by ci_cpp ?
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25905 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26003 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26083 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26170 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26285 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26511 ]
            czw Christopher Waters made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26607 ]
            czw Christopher Waters made changes -
            Link This issue FF-depends on DM-27485 [ DM-27485 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26709 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26777 ]
            Hide
            jbosch Jim Bosch added a comment -

            Christopher Waters, I think you may already have a ticket (or will soon have a ticket) for updating DMTN-101 with your latest thinking over the next few weeks. Could you link that here, as being triggered by this RFC, just to get this off of our RFC-bookkeeping radar until it's done?

            Show
            jbosch Jim Bosch added a comment - Christopher Waters , I think you may already have a ticket (or will soon have a ticket) for updating DMTN-101 with your latest thinking over the next few weeks. Could you link that here, as being triggered by this RFC, just to get this off of our RFC-bookkeeping radar until it's done?
            czw Christopher Waters made changes -
            Link This issue is triggering DM-27825 [ DM-27825 ]
            czw Christopher Waters made changes -
            Link This issue is triggering DM-27825 [ DM-27825 ]
            czw Christopher Waters made changes -
            Link This issue is triggered by DM-27825 [ DM-27825 ]
            czw Christopher Waters made changes -
            Link This issue is triggering DM-27825 [ DM-27825 ]
            czw Christopher Waters made changes -
            Link This issue is triggered by DM-27825 [ DM-27825 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 26816 ]

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Merlin Fisher-Levine, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Planned End:

                  Jenkins

                  No builds found.