Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30174

Define linearity tests for cp_verify

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 4
    • Data Release Production

    Description

      Implement tests for linearity in cp_verify, and document them in DMTN-101.

      Attachments

        Activity

          Usually, for LSSTCam, a linearizer is first produced from a PTC (that has been run with doLinearize=False because there was no linearizer to begin yet). This is done using the cpLinearityCorrected.yaml pipeline. It is called "corrected" because there is a systematic error in the photodiode data that needs to be corrected when producing the linearizer.

          Assuming that we have a linearizer that we want to test (i.e., run cp_verify on it), we need to:

          • Run again the PTC task with isr:doLinearize=True
          • Produce a second linearizer with this new PTC dataset
          • use a metric such as

          maxError = max(abs(values/binCenters))*100
          

          where

           binCenters, values = np.split(newLinearizer.linearityCoeffs[amp.getName()], 2)
          

          See, for example, https://lsstc.slack.com/archives/CQ4SMS3QQ/p1663179665228579

          plazas Andrés Alejandro Plazas Malagón added a comment - Usually, for LSSTCam, a linearizer is first produced from a PTC (that has been run with doLinearize=False because there was no linearizer to begin yet). This is done using the cpLinearityCorrected.yaml pipeline. It is called "corrected" because there is a systematic error in the photodiode data that needs to be corrected when producing the linearizer. Assuming that we have a linearizer that we want to test (i.e., run cp_verify on it), we need to: Run again the PTC task with isr:doLinearize=True Produce a second linearizer with this new PTC dataset use a metric such as maxError = max ( abs (values / binCenters)) * 100 where binCenters, values = np.split(newLinearizer.linearityCoeffs[amp.getName()], 2 ) See, for example, https://lsstc.slack.com/archives/CQ4SMS3QQ/p1663179665228579
          plazas Andrés Alejandro Plazas Malagón added a comment - DMTN-101 PR: https://github.com/lsst-dm/dmtn-101/pull/4

          I think this all looks good.  I think I'm going to ask you file another ticket for adding a notebook (or adding some plots to the PTC notebook, as they're coupled together) so we can be able to make some of the plots that cslage has presented before.  

          czw Christopher Waters added a comment - I think this all looks good.  I think I'm going to ask you file another ticket for adding a notebook (or adding some plots to the PTC notebook, as they're coupled together) so we can be able to make some of the plots that cslage  has presented before.  

          For plotting the linearizer, there's a ticket already: DM-37626

          I added a comment to include Craig's notebook with the residuals.

          plazas Andrés Alejandro Plazas Malagón added a comment - For plotting the linearizer, there's a ticket already: DM-37626 I added a comment to include Craig's notebook with the residuals.

          Test command: 

           
          pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/LsstCam/VerifyLinearity.yaml' -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/photodiode,u/cslage/bps_13144S -j 32 --register-dataset-types -d "instrument='LSSTCam' AND detector IN (55..58) AND exposure IN (3021120600575..3021120600605)" -c verifyLinearityApply:doFlat=False -c verifyLinearityApply:doCrosstalk=False -o u/plazas/DM-30174.2023FEB23.1}
          
          

           

          plazas Andrés Alejandro Plazas Malagón added a comment - Test command:    pipetask --long-log run -p '${CP_VERIFY_DIR}/pipelines/LsstCam/VerifyLinearity.yaml' -b /repo/main -i LSSTCam/raw/all,LSSTCam/calib,LSSTCam/photodiode,u/cslage/bps_13144S -j 32 --register-dataset-types -d "instrument='LSSTCam' AND detector IN (55..58) AND exposure IN (3021120600575..3021120600605)" -c verifyLinearityApply:doFlat=False -c verifyLinearityApply:doCrosstalk=False -o u/plazas/DM-30174.2023FEB23.1}  
          plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins:    https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38224/pipeline

          People

            plazas Andrés Alejandro Plazas Malagón
            czw Christopher Waters
            Christopher Waters
            Andrés Alejandro Plazas Malagón, Christopher Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.