Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26037

Add option to ignore nPixels from edge in PTC task

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe
    • Labels:
      None

      Description

      People are currently doing this by (ab)using defect sets to do the masking. There should be a separate config option allowing a certain border to be excluded during PTC calculation without having to make faked-up defect sets which mask otherwise-acceptable (presumably?) pixels, otherwise you have to make custom defect sets for the PTC task and other bits of the pipeline, which is clearly not a good workflow.

      There already exists some code in ip_isr to do this I think, and/or in the brighter-fatter code (though that's likely less well-made).

        Attachments

          Activity

          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited

          This function can be imported and used (pointed out by Chris): https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/isrTask.py#L2312

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - - edited This function can be imported and used (pointed out by Chris): https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/isrTask.py#L2312
          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment -

          (lsst-scipipe) [plazas@lsst-devl02 cp_pipe]$ pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado-DM-26037.3 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c ptcExtract:numEdgeSuspect=20 -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types
          

          cpPtcExtract INFO: Masking 20 pixels from the edges of all exposures as SUSPECT.
          cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.5791673212032897
          cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.5293924872998854
          cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 2.8280151653482077
          cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 4.37248816421254
          cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.214315560362933
          ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'ptcExtract' on quantum {instrument: 'LSSTCam', detector: 94} took 195.173 seconds
          ctrl.mpexec.mpGraphExecutor INFO: Executed 47 quanta, 1 remain out of total 48 quanta.
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C00: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C01: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C02: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C03: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C04: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C05: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C06: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C07: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C10: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C11: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C12: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C13: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C14: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C15: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C16: 5 out of 23
          cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C17: 5 out of 23
          
          

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - (lsst-scipipe) [plazas@lsst-devl02 cp_pipe]$ pipetask run -j 1 -d "detector=94 AND exposure IN (3020100800155,3020100800156,3020100800158,3020100800159,3020100800185,3020100800186,3020100800161,3020100800162,3020100800188,3020100800189,3020100800164,3020100800165,3020100800191,3020100800192,3020100800167,3020100800168,3020100800194,3020100800195,3020100800170,3020100800171,3020100800197,3020100800198,3020100800173,3020100800174,3020100800200,3020100800201,3020100800176,3020100800177,3020100800203,3020100800204,3020100800179,3020100800180,3020100800206,3020100800207,3020100800182,3020100800183,3020100800209,3020100800210,3020100800212,3020100800213,3020100800215,3020100800216,3020100800218,3020100800219,3020100800221,3020100800222) and instrument = 'LSSTCam' " -b /project/plazas/WORK/DM-23159/TEST_BOT/butler.yaml -i ptcTestPostIsr.02 -o ptcTestResultado-DM-26037.3 -p /home/plazas/lsst_devel/LSST/cp_pipe/pipelines/measurePhotonTransferCurve.yaml -c ptcSolve:ptcFitType=FULLCOVARIANCE -c ptcExtract:numEdgeSuspect=20 -c isr:doFlat=False -c isr:doCrosstalk=False --register-dataset-types cpPtcExtract INFO: Masking 20 pixels from the edges of all exposures as SUSPECT. cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.5791673212032897 cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.5293924872998854 cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 2.8280151653482077 cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 4.37248816421254 cpPtcExtract WARN: Absolute fractional difference between afwMatch.VARIANCECLIP and Cov[0,0] is more than 1.0%: 1.214315560362933 ctrl.mpexec.singleQuantumExecutor INFO: Execution of task 'ptcExtract' on quantum {instrument: 'LSSTCam', detector: 94} took 195.173 seconds ctrl.mpexec.mpGraphExecutor INFO: Executed 47 quanta, 1 remain out of total 48 quanta. cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C00: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C01: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C02: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C03: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C04: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C05: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C06: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C07: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C10: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C11: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C12: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C13: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C14: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C15: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C16: 5 out of 23 cpPhotonTransferCurveSolve INFO: Number of points discarded in PTC of amplifier C17: 5 out of 23
          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34001/pipeline
          Hide
          czw Christopher Waters added a comment -

          Does this improve the PTC results significantly?  In any case, it looks fine.

          Show
          czw Christopher Waters added a comment - Does this improve the PTC results significantly?  In any case, it looks fine.

            People

            Assignee:
            plazas Andrés Alejandro Plazas Malagón
            Reporter:
            mfisherlevine Merlin Fisher-Levine
            Reviewers:
            Christopher Waters
            Watchers:
            Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: