Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
Description
While debugging DM-27458, I noticed that the reason for many of the failures was that some images had few or no good pixels, and this caused the FFT algorithm that calculates the covariances to fail. While some amps have some defects, most pixels should be good. I tracked down one failure to raft R11, sensor S12, detector=41, and expId=3020110200190. The calibration data and postISRCCD images are in
/project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12673
|
I queried the number of good pixels (good meaning the mask plane is 0) by amp, and obtained the following:
C10 0 |
C11 0 |
C12 0 |
C13 0 |
C14 0 |
C15 0 |
C16 0 |
C17 0 |
C07 3539 |
C06 614 |
C05 11 |
C04 78 |
C03 11 |
C02 20657 |
C01 974784 |
C00 409729 |
There should be about 1 million good pixels/amp, so only C01 looks normal. I then compared mask plane and image plane in a good region between C01(good) and C06(bad):
C06 Mask Good Region [6 6 6 6 6] |
C06 Image Good Region [99972.336 99972.336 99972.336 99972.336 99972.336] |
C01 Mask Good Region [0 0 0 0 0] |
C01 Image Good Region [104243.4 106106.586 105608.06 105036.266 104154.09 ] |
|
|
dict_items([('BAD', 0), ('CR', 3), ('DETECTED', 5), ('DETECTED_NEGATIVE', 6), ('EDGE', 4), ('INTRP', 2), ('NO_DATA', 8), ('SAT', 1), ('SUSPECT', 7), ('UNMASKEDNAN', 9)]) |
So the bad pixels have mask plane=6, which is 'SAT', and 'INTRP', so they have been interpreted as saturated. This isn't right, beacuse at ~100K ADU, they shouldn't be saturated. So I looked at obs_lsst/polict/lsstCam/R11.yaml for S12, and it shows:
S12 :
|
C10 : { gain : 0.699516, readNoise : 4.745807, saturation : 135968.062500 } |
C11 : { gain : 0.699235, readNoise : 4.708802, saturation : 136083.000000 } |
C12 : { gain : 0.695590, readNoise : 4.748249, saturation : 135391.828125 } |
C13 : { gain : 0.690748, readNoise : 4.987112, saturation : 134143.984375 } |
C14 : { gain : 0.696782, readNoise : 5.019731, saturation : 135183.234375 } |
C15 : { gain : 0.689901, readNoise : 5.063357, saturation : 134983.390625 } |
C16 : { gain : 0.693390, readNoise : 4.898793, saturation : 135653.421875 } |
C17 : { gain : 0.698506, readNoise : 4.976052, saturation : 135490.921875 } |
C07 : { gain : 0.721271, readNoise : 4.937364, saturation : 135981.453125 } |
C06 : { gain : 0.715233, readNoise : 4.710922, saturation : 134819.281250 } |
C05 : { gain : 0.713144, readNoise : 4.725629, saturation : 134069.796875 } |
C04 : { gain : 0.713045, readNoise : 4.748403, saturation : 134037.093750 } |
C03 : { gain : 0.711816, readNoise : 4.717724, saturation : 133621.531250 } |
C02 : { gain : 0.717907, readNoise : 4.945924, saturation : 135765.281250 } |
C01 : { gain : 0.775595, readNoise : 6.067882, saturation : 160979.125000 } |
C00 : { gain : 0.725844, readNoise : 5.204304, saturation : 135903.109375 } |
They are being flagged as saturated, because the gain is listed as ~0.7 (too low, it is actually more like 1.1). For C06, 134819*0.715=96395, so anything above this is flagged as saturated. For C01, 160979*0.775=131036, so this is why C01 is not flagged as saturated. So we need to do two things:
(1) This will happen even with the right values, so we need to leave the test for too few pixels in ptc.py. this is the line at: https://github.com/lsst/cp_pipe/blob/28eb519a8f97a5f2db6c21412e9172b7e0c0f463/python/lsst/cp/pipe/ptc.py#L683, but we need to increase the test to np.sum(w) < 10000 , or some reasonable number.
(2) We need to update the gain, noise, and sat level values in the Rxx.yaml files. i am willing to work to get the right values in there.
Attachments
Activity
My understanding was that we need doSuspect=True in order to have the edge masking. I think the edge masking is important because the flux rolls off significantly near the edges. I thought doSuspect=True, isr.edgeMaskLevel=AMP isr.numEdgeSuspect=10 were all necessary to make the edge masking happen. Maybe I'm wrong.
`doLinearize` can't be True if we're using the PTC dataset to remeasure the linearity. That's probably more of a linearity issue than PTC, however, so leaving it true here makes sense.
And I think `doDefect` enables the edge masking if the `numEdgeSuspect` configuration option is non-zero.
Some minor comments on the config and that's it.
I agree that doCrosstalk should be True. We can leave it like True as default, and set it explicitly to False when calling the ISR.
I'm not sure if we need isr.doFringe=False isr.doSuspect=True in the file. Christopher Waters?
These are the values that are in ptcIsr.py:
# ISR for inputs of Photon Transfer Curve taskconfig.isr.doWrite = Trueconfig.isr.doOverscan = Trueconfig.isr.doAssembleCcd = Trueconfig.isr.doBias = Trueconfig.isr.doVariance = Trueconfig.isr.doLinearize = Trueconfig.isr.doCrosstalk = Trueconfig.isr.doBrighterFatter = Falseconfig.isr.doDark = Trueconfig.isr.doStrayLight = Falseconfig.isr.doFlat = Falseconfig.isr.doFringe = Falseconfig.isr.doApplyGains = Falseconfig.isr.doDefect = Trueconfig.isr.doNanMasking: Trueconfig.isr.doInterpolate: Trueconfig.isr.doSaturation=Falseconfig.isr.doSaturationInterpolation = Falseconfig.isr.growSaturationFootprintSize = 0 # We want the saturation spillover: it's good signal.