# Many images have few or no good pixels when running ptc.py

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production

#### Description

While debugging DM-27458, I noticed that the reason for many of the failures was that some images had few or no good pixels, and this caused the FFT algorithm that calculates the covariances to fail.  While some amps have some defects, most pixels should be good.  I tracked down one failure to raft R11, sensor S12, detector=41, and expId=3020110200190.  The calibration data and postISRCCD images are in

 /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12673

I queried the number of good pixels (good meaning the mask plane is 0) by amp, and obtained the following:

 C10 0 C11 0 C12 0 C13 0 C14 0 C15 0 C16 0 C17 0 C07 3539 C06 614 C05 11 C04 78 C03 11 C02 20657 C01 974784 C00 409729

There should be about 1 million good pixels/amp, so only C01 looks normal.  I then compared mask plane and image plane in a good region between C01(good) and C06(bad):

 C06 Mask Good Region [6 6 6 6 6] C06 Image Good Region [99972.336 99972.336 99972.336 99972.336 99972.336] C01 Mask Good Region [0 0 0 0 0] C01 Image Good Region [104243.4 106106.586 105608.06 105036.266 104154.09 ]   dict_items([('BAD', 0), ('CR', 3), ('DETECTED', 5), ('DETECTED_NEGATIVE', 6), ('EDGE', 4), ('INTRP', 2), ('NO_DATA', 8), ('SAT', 1), ('SUSPECT', 7), ('UNMASKEDNAN', 9)])

So the bad pixels have mask plane=6, which is 'SAT', and 'INTRP', so they have been interpreted as saturated.  This isn't right, beacuse at ~100K ADU, they shouldn't be saturated.  So I looked at obs_lsst/polict/lsstCam/R11.yaml for S12, and it shows:

  S12 :  C10 : { gain : 0.699516, readNoise : 4.745807, saturation : 135968.062500 }  C11 : { gain : 0.699235, readNoise : 4.708802, saturation : 136083.000000 }  C12 : { gain : 0.695590, readNoise : 4.748249, saturation : 135391.828125 }  C13 : { gain : 0.690748, readNoise : 4.987112, saturation : 134143.984375 }  C14 : { gain : 0.696782, readNoise : 5.019731, saturation : 135183.234375 }  C15 : { gain : 0.689901, readNoise : 5.063357, saturation : 134983.390625 }  C16 : { gain : 0.693390, readNoise : 4.898793, saturation : 135653.421875 }  C17 : { gain : 0.698506, readNoise : 4.976052, saturation : 135490.921875 }  C07 : { gain : 0.721271, readNoise : 4.937364, saturation : 135981.453125 }  C06 : { gain : 0.715233, readNoise : 4.710922, saturation : 134819.281250 }  C05 : { gain : 0.713144, readNoise : 4.725629, saturation : 134069.796875 }  C04 : { gain : 0.713045, readNoise : 4.748403, saturation : 134037.093750 }  C03 : { gain : 0.711816, readNoise : 4.717724, saturation : 133621.531250 }  C02 : { gain : 0.717907, readNoise : 4.945924, saturation : 135765.281250 }  C01 : { gain : 0.775595, readNoise : 6.067882, saturation : 160979.125000 }  C00 : { gain : 0.725844, readNoise : 5.204304, saturation : 135903.109375 }

They are being flagged as saturated, because the gain is listed as ~0.7 (too low, it is actually more like 1.1).  For C06, 134819*0.715=96395, so anything above this is flagged as saturated.  For C01, 160979*0.775=131036, so this is why C01 is not flagged as saturated.  So we need to do two things:
(1) This will happen even with the right values, so we need to leave the test for too few pixels in ptc.py.  this is the line at: https://github.com/lsst/cp_pipe/blob/28eb519a8f97a5f2db6c21412e9172b7e0c0f463/python/lsst/cp/pipe/ptc.py#L683, but we need to increase the test to np.sum(w) < 10000 , or some reasonable number.
(2) We need to update the gain, noise, and sat level values in the Rxx.yaml files.  i am willing to work to get the right values in there.

#### Activity

Hide
Andrés Alejandro Plazas Malagón added a comment -

I agree that doCrosstalk should be True. We can leave it like True as default, and set it explicitly to False when calling the ISR.

I'm not sure if we need isr.doFringe=False isr.doSuspect=True in the file. Christopher Waters?

These are the values that are in ptcIsr.py:

   # ISR for inputs of Photon Transfer Curve task config.isr.doWrite = True config.isr.doOverscan = True config.isr.doAssembleCcd = True config.isr.doBias = True config.isr.doVariance = True config.isr.doLinearize = True config.isr.doCrosstalk = True config.isr.doBrighterFatter = False config.isr.doDark = True config.isr.doStrayLight = False config.isr.doFlat = False config.isr.doFringe = False config.isr.doApplyGains = False config.isr.doDefect = True config.isr.doNanMasking: True config.isr.doInterpolate: True config.isr.doSaturation=False config.isr.doSaturationInterpolation = False config.isr.growSaturationFootprintSize = 0 # We want the saturation spillover: it's good signal. 

Show
Andrés Alejandro Plazas Malagón added a comment - I agree that doCrosstalk should be True. We can leave it like True as default, and set it explicitly to False when calling the ISR. I'm not sure if we need isr.doFringe=False isr.doSuspect=True in the file. Christopher Waters ? These are the values that are in ptcIsr.py:   # ISR for inputs of Photon Transfer Curve task config.isr.doWrite = True config.isr.doOverscan = True config.isr.doAssembleCcd = True config.isr.doBias = True config.isr.doVariance = True config.isr.doLinearize = True config.isr.doCrosstalk = True config.isr.doBrighterFatter = False config.isr.doDark = True config.isr.doStrayLight = False config.isr.doFlat = False config.isr.doFringe = False config.isr.doApplyGains = False config.isr.doDefect = True config.isr.doNanMasking: True config.isr.doInterpolate: True config.isr.doSaturation=False config.isr.doSaturationInterpolation = False config.isr.growSaturationFootprintSize = 0 # We want the saturation spillover: it's good signal.
Hide
Craig Lage added a comment -

My understanding was that we need doSuspect=True in order to have the edge masking.  I think the edge masking is important because the flux rolls off significantly near the edges.  I thought doSuspect=True, isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 were all necessary to make the edge masking happen.  Maybe I'm wrong.

Show
Craig Lage added a comment - My understanding was that we need doSuspect=True in order to have the edge masking.  I think the edge masking is important because the flux rolls off significantly near the edges.  I thought doSuspect=True, isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 were all necessary to make the edge masking happen.  Maybe I'm wrong.
Hide
Christopher Waters added a comment -

doLinearize can't be True if we're using the PTC dataset to remeasure the linearity.  That's probably more of a linearity issue than PTC, however, so leaving it true here makes sense.

And I think doDefect enables the edge masking if the numEdgeSuspect configuration option is non-zero.

Show
Christopher Waters added a comment - doLinearize can't be True if we're using the PTC dataset to remeasure the linearity.  That's probably more of a linearity issue than PTC, however, so leaving it true here makes sense. And I think doDefect enables the edge masking if the numEdgeSuspect configuration option is non-zero.
Hide
Merlin Fisher-Levine added a comment -

Some minor comments on the config and that's it.

Show
Merlin Fisher-Levine added a comment - Some minor comments on the config and that's it.
Hide
Andrés Alejandro Plazas Malagón added a comment -
Show
Andrés Alejandro Plazas Malagón added a comment - Latest Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33124/pipeline/

#### People

Assignee:
Andrés Alejandro Plazas Malagón
Reporter:
Craig Lage
Reviewers:
Merlin Fisher-Levine
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine