Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-37684

Enable cp_pipe defect code to run on combined exposures

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • cp_pipe
    • 7
    • Data Release Production

    Description

      Currently, the cp_pipe defect pipeline runs on individual exposures and merges the resulting sets of defects.  To enable direct comparison with the Camera team's eotest results for bad columns and pixels, it would be useful to enable the defect pipeline to run on combined images, i.e., combined darks or combined flats.

      In addition, in order to further facilitate direct comparison with eotest results, it would be useful to be able to define the threshold for dark defects as a fraction of the expected signal level and bright defects using a threshold of e-/pixel above baseline.

      Attachments

        Issue Links

          Activity

            plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38314/  
            plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38364/pipeline

            After implementing changes suggested in first round of review comments, Jenkis has passed:

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38410/pipeline/

            plazas Andrés Alejandro Plazas Malagón added a comment - After implementing changes suggested in first round of review comments, Jenkis has passed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38410/pipeline/

            Example:

            pipetask run -j 1 -d "detector IN (54) AND instrument='LSSTCam'" -i u/cslage/dark_20211117/20211117T173016Z,u/cslage/flat_20211117/20211118T005839Z,LSSTCam/raw/all -p $CP_PIPE_DIR/pipelines/findDefectsCombined.yaml --register-dataset-types -b /sdf/group/rubin/repo/main/butler.yaml -c measureDefectsDark:thresholdType=STDEV -c measureDefectsFlat:thresholdType=STDEV -o u/plazas/DM-37684.2023APR11.1
            

            thresholdType=VALUE will work after DM-38486 is merged.

            mport lsst.daf.butler as dB
            import lsst.cp.verify.notebooks.utils as utils
            import lsst.afw.display as afwDisplay
            import lsst.afw.image as afwImage
             
            # Which calibration type to analyse.
            calibType = 'defectsCombined'
            # This cell should be edited to match the data to be inspected.
            afwDisplay.setDefaultBackend("astrowidgets")
            cameraName = 'LSSTCam'
            collections = "u/plazas/DM-37684.2023APR11.1"
             
            # Get butler
            butler = dB.Butler("/repo/main/", collections=[collections]) 
             
            display = afwDisplay.Display(dims=(1000, 1000))
            display.embed()
             
            # Defects are on disk as a list, but an image is more useful
            calib = butler.get(calibType, instrument=cameraName, detector=54)
            print(calib.getMetadata().toDict())
            realization = afwImage.MaskedImageI(4072, 4000)
            calib.maskPixels(realization)
            calibArray = realization.getMask().getArray()
             
            # Get simple stats
            q25, q50, q75 = np.percentile(calibArray.flatten(), [25, 50, 75])
            sigma = 0.74 * (q75 - q25)
            print(f"Median: {q50}   Stdev: {sigma}")
            display.mtv(realization.getMask())
            

            plazas Andrés Alejandro Plazas Malagón added a comment - Example: pipetask run -j 1 -d "detector IN (54) AND instrument='LSSTCam'" -i u/cslage/dark_20211117/20211117T173016Z,u/cslage/flat_20211117/20211118T005839Z,LSSTCam/raw/all -p $CP_PIPE_DIR/pipelines/findDefectsCombined.yaml --register-dataset-types -b /sdf/group/rubin/repo/main/butler.yaml -c measureDefectsDark:thresholdType=STDEV -c measureDefectsFlat:thresholdType=STDEV -o u/plazas/DM-37684.2023APR11.1 thresholdType=VALUE will work after DM-38486 is merged. mport lsst.daf.butler as dB import lsst.cp.verify.notebooks.utils as utils import lsst.afw.display as afwDisplay import lsst.afw.image as afwImage   # Which calibration type to analyse. calibType = 'defectsCombined' # This cell should be edited to match the data to be inspected. afwDisplay.setDefaultBackend( "astrowidgets" ) cameraName = 'LSSTCam' collections = "u/plazas/DM-37684.2023APR11.1"   # Get butler butler = dB.Butler( "/repo/main/" , collections = [collections])   display = afwDisplay.Display(dims = ( 1000 , 1000 )) display.embed()   # Defects are on disk as a list, but an image is more useful calib = butler.get(calibType, instrument = cameraName, detector = 54 ) print (calib.getMetadata().toDict()) realization = afwImage.MaskedImageI( 4072 , 4000 ) calib.maskPixels(realization) calibArray = realization.getMask().getArray()   # Get simple stats q25, q50, q75 = np.percentile(calibArray.flatten(), [ 25 , 50 , 75 ]) sigma = 0.74 * (q75 - q25) print (f "Median: {q50} Stdev: {sigma}" ) display.mtv(realization.getMask())
            plazas Andrés Alejandro Plazas Malagón added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38455/pipeline
            plazas Andrés Alejandro Plazas Malagón added a comment - Last Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38468/pipeline

            People

              plazas Andrés Alejandro Plazas Malagón
              jchiang James Chiang
              Eli Rykoff
              Aaron Roodman, Andrés Alejandro Plazas Malagón, Christopher Waters, Eli Rykoff, James Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.