Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27598

Many images have few or no good pixels when running ptc.py

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe, ip_isr, obs_lsst
    • Labels:
      None

      Description

      While debugging DM-27458, I noticed that the reason for many of the failures was that some images had few or no good pixels, and this caused the FFT algorithm that calculates the covariances to fail.  While some amps have some defects, most pixels should be good.  I tracked down one failure to raft R11, sensor S12, detector=41, and expId=3020110200190.  The calibration data and postISRCCD images are in 

      /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12673

      I queried the number of good pixels (good meaning the mask plane is 0) by amp, and obtained the following:

      C10 0
      C11 0
      C12 0
      C13 0
      C14 0
      C15 0
      C16 0
      C17 0
      C07 3539
      C06 614
      C05 11
      C04 78
      C03 11
      C02 20657
      C01 974784
      C00 409729

      There should be about 1 million good pixels/amp, so only C01 looks normal.  I then compared mask plane and image plane in a good region between C01(good) and C06(bad):

      C06 Mask Good Region [6 6 6 6 6]
      C06 Image Good Region [99972.336 99972.336 99972.336 99972.336 99972.336]
      C01 Mask Good Region [0 0 0 0 0]
      C01 Image Good Region [104243.4   106106.586 105608.06  105036.266 104154.09 ]
       
      dict_items([('BAD', 0), ('CR', 3), ('DETECTED', 5), ('DETECTED_NEGATIVE', 6), ('EDGE', 4), ('INTRP', 2), ('NO_DATA', 8), ('SAT', 1), ('SUSPECT', 7), ('UNMASKEDNAN', 9)])

      So the bad pixels have mask plane=6, which is 'SAT', and 'INTRP', so they have been interpreted as saturated.  This isn't right, beacuse at ~100K ADU, they shouldn't be saturated.  So I looked at obs_lsst/polict/lsstCam/R11.yaml for S12, and it shows:

          S12 :
            C10 : { gain : 0.699516, readNoise : 4.745807, saturation : 135968.062500 }
            C11 : { gain : 0.699235, readNoise : 4.708802, saturation : 136083.000000 }
            C12 : { gain : 0.695590, readNoise : 4.748249, saturation : 135391.828125 }
            C13 : { gain : 0.690748, readNoise : 4.987112, saturation : 134143.984375 }
            C14 : { gain : 0.696782, readNoise : 5.019731, saturation : 135183.234375 }
            C15 : { gain : 0.689901, readNoise : 5.063357, saturation : 134983.390625 }
            C16 : { gain : 0.693390, readNoise : 4.898793, saturation : 135653.421875 }
            C17 : { gain : 0.698506, readNoise : 4.976052, saturation : 135490.921875 }
            C07 : { gain : 0.721271, readNoise : 4.937364, saturation : 135981.453125 }
            C06 : { gain : 0.715233, readNoise : 4.710922, saturation : 134819.281250 }
            C05 : { gain : 0.713144, readNoise : 4.725629, saturation : 134069.796875 }
            C04 : { gain : 0.713045, readNoise : 4.748403, saturation : 134037.093750 }
            C03 : { gain : 0.711816, readNoise : 4.717724, saturation : 133621.531250 }
            C02 : { gain : 0.717907, readNoise : 4.945924, saturation : 135765.281250 }
            C01 : { gain : 0.775595, readNoise : 6.067882, saturation : 160979.125000 }
            C00 : { gain : 0.725844, readNoise : 5.204304, saturation : 135903.109375 }

      They are being flagged as saturated, because the gain is listed as ~0.7 (too low, it is actually more like 1.1).  For C06, 134819*0.715=96395, so anything above this is flagged as saturated.  For C01, 160979*0.775=131036, so this is why C01 is not flagged as saturated.  So we need to do two things:
      (1) This will happen even with the right values, so we need to leave the test for too few pixels in ptc.py.  this is the line at: https://github.com/lsst/cp_pipe/blob/28eb519a8f97a5f2db6c21412e9172b7e0c0f463/python/lsst/cp/pipe/ptc.py#L683, but we need to increase the test to np.sum(w) < 10000 , or some reasonable number.
      (2) We need to update the gain, noise, and sat level values in the Rxx.yaml files.  i am willing to work to get the right values in there.

        Attachments

          Activity

          Hide
          plazas Andrés Alejandro Plazas Malagón added a comment -

          I agree that doCrosstalk should be True. We can leave it like True as default, and set it explicitly to False when calling the ISR.

          I'm not sure if we need isr.doFringe=False isr.doSuspect=True in the file. Christopher Waters?

          These are the values that are in ptcIsr.py:

           
          # ISR for inputs of Photon Transfer Curve task
          config.isr.doWrite = True
          config.isr.doOverscan = True
          config.isr.doAssembleCcd = True
          config.isr.doBias = True
          config.isr.doVariance = True
          config.isr.doLinearize = True
          config.isr.doCrosstalk = True
          config.isr.doBrighterFatter = False
          config.isr.doDark = True
          config.isr.doStrayLight = False
          config.isr.doFlat = False
          config.isr.doFringe = False
          config.isr.doApplyGains = False
          config.isr.doDefect = True
          config.isr.doNanMasking: True
          config.isr.doInterpolate: True
          config.isr.doSaturation=False
          config.isr.doSaturationInterpolation = False
          config.isr.growSaturationFootprintSize = 0  # We want the saturation spillover: it's good signal.
          

          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - I agree that doCrosstalk should be True. We can leave it like True as default, and set it explicitly to False when calling the ISR. I'm not sure if we need isr.doFringe=False isr.doSuspect=True in the file. Christopher Waters ? These are the values that are in ptcIsr.py:   # ISR for inputs of Photon Transfer Curve task config.isr.doWrite = True config.isr.doOverscan = True config.isr.doAssembleCcd = True config.isr.doBias = True config.isr.doVariance = True config.isr.doLinearize = True config.isr.doCrosstalk = True config.isr.doBrighterFatter = False config.isr.doDark = True config.isr.doStrayLight = False config.isr.doFlat = False config.isr.doFringe = False config.isr.doApplyGains = False config.isr.doDefect = True config.isr.doNanMasking: True config.isr.doInterpolate: True config.isr.doSaturation=False config.isr.doSaturationInterpolation = False config.isr.growSaturationFootprintSize = 0 # We want the saturation spillover: it's good signal.
          Hide
          cslage Craig Lage added a comment -

          My understanding was that we need doSuspect=True in order to have the edge masking.  I think the edge masking is important because the flux rolls off significantly near the edges.  I thought doSuspect=True, isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 were all necessary to make the edge masking happen.  Maybe I'm wrong. 

          Show
          cslage Craig Lage added a comment - My understanding was that we need doSuspect=True in order to have the edge masking.  I think the edge masking is important because the flux rolls off significantly near the edges.  I thought doSuspect=True, isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 were all necessary to make the edge masking happen.  Maybe I'm wrong. 
          Hide
          czw Christopher Waters added a comment -

          `doLinearize` can't be True if we're using the PTC dataset to remeasure the linearity.  That's probably more of a linearity issue than PTC, however, so leaving it true here makes sense.

          And I think `doDefect` enables the edge masking if the `numEdgeSuspect` configuration option is non-zero.

          Show
          czw Christopher Waters added a comment - `doLinearize` can't be True if we're using the PTC dataset to remeasure the linearity.  That's probably more of a linearity issue than PTC, however, so leaving it true here makes sense. And I think `doDefect` enables the edge masking if the `numEdgeSuspect` configuration option is non-zero.
          Hide
          mfisherlevine Merlin Fisher-Levine added a comment -

          Some minor comments on the config and that's it.

          Show
          mfisherlevine Merlin Fisher-Levine added a comment - Some minor comments on the config and that's it.
          Show
          plazas Andrés Alejandro Plazas Malagón added a comment - Latest Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33124/pipeline/

            People

            Assignee:
            plazas Andrés Alejandro Plazas Malagón
            Reporter:
            cslage Craig Lage
            Reviewers:
            Merlin Fisher-Levine
            Watchers:
            Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: