Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27598

Many images have few or no good pixels when running ptc.py

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cp_pipe, ip_isr, obs_lsst
    • Labels:
      None

      Description

      While debugging DM-27458, I noticed that the reason for many of the failures was that some images had few or no good pixels, and this caused the FFT algorithm that calculates the covariances to fail.  While some amps have some defects, most pixels should be good.  I tracked down one failure to raft R11, sensor S12, detector=41, and expId=3020110200190.  The calibration data and postISRCCD images are in 

      /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12673

      I queried the number of good pixels (good meaning the mask plane is 0) by amp, and obtained the following:

      C10 0
      C11 0
      C12 0
      C13 0
      C14 0
      C15 0
      C16 0
      C17 0
      C07 3539
      C06 614
      C05 11
      C04 78
      C03 11
      C02 20657
      C01 974784
      C00 409729

      There should be about 1 million good pixels/amp, so only C01 looks normal.  I then compared mask plane and image plane in a good region between C01(good) and C06(bad):

      C06 Mask Good Region [6 6 6 6 6]
      C06 Image Good Region [99972.336 99972.336 99972.336 99972.336 99972.336]
      C01 Mask Good Region [0 0 0 0 0]
      C01 Image Good Region [104243.4   106106.586 105608.06  105036.266 104154.09 ]
       
      dict_items([('BAD', 0), ('CR', 3), ('DETECTED', 5), ('DETECTED_NEGATIVE', 6), ('EDGE', 4), ('INTRP', 2), ('NO_DATA', 8), ('SAT', 1), ('SUSPECT', 7), ('UNMASKEDNAN', 9)])

      So the bad pixels have mask plane=6, which is 'SAT', and 'INTRP', so they have been interpreted as saturated.  This isn't right, beacuse at ~100K ADU, they shouldn't be saturated.  So I looked at obs_lsst/polict/lsstCam/R11.yaml for S12, and it shows:

          S12 :
            C10 : { gain : 0.699516, readNoise : 4.745807, saturation : 135968.062500 }
            C11 : { gain : 0.699235, readNoise : 4.708802, saturation : 136083.000000 }
            C12 : { gain : 0.695590, readNoise : 4.748249, saturation : 135391.828125 }
            C13 : { gain : 0.690748, readNoise : 4.987112, saturation : 134143.984375 }
            C14 : { gain : 0.696782, readNoise : 5.019731, saturation : 135183.234375 }
            C15 : { gain : 0.689901, readNoise : 5.063357, saturation : 134983.390625 }
            C16 : { gain : 0.693390, readNoise : 4.898793, saturation : 135653.421875 }
            C17 : { gain : 0.698506, readNoise : 4.976052, saturation : 135490.921875 }
            C07 : { gain : 0.721271, readNoise : 4.937364, saturation : 135981.453125 }
            C06 : { gain : 0.715233, readNoise : 4.710922, saturation : 134819.281250 }
            C05 : { gain : 0.713144, readNoise : 4.725629, saturation : 134069.796875 }
            C04 : { gain : 0.713045, readNoise : 4.748403, saturation : 134037.093750 }
            C03 : { gain : 0.711816, readNoise : 4.717724, saturation : 133621.531250 }
            C02 : { gain : 0.717907, readNoise : 4.945924, saturation : 135765.281250 }
            C01 : { gain : 0.775595, readNoise : 6.067882, saturation : 160979.125000 }
            C00 : { gain : 0.725844, readNoise : 5.204304, saturation : 135903.109375 }

      They are being flagged as saturated, because the gain is listed as ~0.7 (too low, it is actually more like 1.1).  For C06, 134819*0.715=96395, so anything above this is flagged as saturated.  For C01, 160979*0.775=131036, so this is why C01 is not flagged as saturated.  So we need to do two things:
      (1) This will happen even with the right values, so we need to leave the test for too few pixels in ptc.py.  this is the line at: https://github.com/lsst/cp_pipe/blob/28eb519a8f97a5f2db6c21412e9172b7e0c0f463/python/lsst/cp/pipe/ptc.py#L683, but we need to increase the test to np.sum(w) < 10000 , or some reasonable number.
      (2) We need to update the gain, noise, and sat level values in the Rxx.yaml files.  i am willing to work to get the right values in there.

        Attachments

          Activity

          No work has yet been logged on this issue.

            People

            Assignee:
            plazas Andrés Alejandro Plazas Malagón
            Reporter:
            cslage Craig Lage
            Reviewers:
            Merlin Fisher-Levine
            Watchers:
            Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage, Merlin Fisher-Levine
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.