Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27158

constructDark.py fails on LSSTCam with 'Too many CR pixels'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_drivers
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      I ran a PTC task on BOT run 12606, the first time we have the full LSSTCam.  Six CCDs failed to return results.  I am tracking down the reasons.  CCD R41_S11, detector=166 failed because it failed to create a master dark with the error "Too many CR pixels (max 100000)". Below is a command line that reproduces the error, as well as the error string itself.  I'm also attaching the dark image, which looks fine.  FWIW, I have seen this error before on several occasions, so this is not an isolated incident.

      Command line:

      constructDark.py /project/shared/BOT --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12606/CALIB --rerun /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_Test  --batch-type=smp  --cores 1  -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1  --id expId=3020100800045 detector=166

      Error:

      dark WARN: Unable to process DataId(initialdata={'expId': 3020100800045, 'detector': 166, 'dayObs': '2020-10-08', 'raftName': 'R41', 'detectorName': 'S11', 'snap': 0}, tag=set()): 
        File "src/CR.cc", line 410, in std::vector<std::shared_ptr<lsst::afw::detection::Footprint> > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::daf::base::PropertySet&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage<float>]
          Too many CR pixels (max 100000) {0}
      lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 100000)'LengthError on lsst-devl01:2455582 in reduce: 
        File "src/CR.cc", line 410, in std::vector<std::shared_ptr<lsst::afw::detection::Footprint> > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::daf::base::PropertySet&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage<float>]
          Too many CR pixels (max 100000) {0}
      lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 100000)'

        Attachments

        1. 3020100800045-R41-S11.png
          3020100800045-R41-S11.png
          315 kB
        2. Noise_Comparison_12639_23Oct20.png
          Noise_Comparison_12639_23Oct20.png
          22 kB
        3. Noise_Summary_12606_16Oct20.png
          Noise_Summary_12606_16Oct20.png
          125 kB
        4. postISR-3020100800045-R41-S11.png
          postISR-3020100800045-R41-S11.png
          863 kB
        5. PTC_R41_S11_C14.png
          PTC_R41_S11_C14.png
          10 kB
        6. R41_S11_C14.png
          R41_S11_C14.png
          56 kB

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment -

            Thinking through this again, and trying to place fixes at various different levels, I came to what is likely the best solution for now:

             constructDark.py /project/shared/BOT \
                    --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/CALIB \
                    --rerun /project/czw/DM-27158/run1016a \
                    --batch-type=smp  --cores 1 \
                    -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 \
                       isr.doEmpiricalReadNoise=True \
                    --id expId=3020100800045 detector=166

            Setting `doEmpiricalReadNoise` forces ISR to ignore the camera defined read noise levels and independently estimate it from the data.  This provides a better variance to the CR code, and it then flags a much smaller number of pixels (132 CR found on 3020100800045).

            Once DM-27010 is complete, we will be able to supply a PTC dataset to ISR, and pull gain and read noise values from there instead of from the camera definition.  I think this means that creating initial calibrations for a camera is going to need to iterate:

            1. Initial bias, dark with `doEmpiricalReadNoise=True`.
            2. Initial PTC to generate more reliable read noise estimates.
            3. Second dark with `doEmpiricalReadNoise=False` supplying the initial PTC as the source for the read noise.
            4. Possible second PTC using second dark if the empirical read noise and PTC measured read noise values are significantly different.
            Show
            czw Christopher Waters added a comment - Thinking through this again, and trying to place fixes at various different levels, I came to what is likely the best solution for now: constructDark.py /project/shared/BOT \ --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/CALIB \ --rerun /project/czw/DM- 27158 /run1016a \ --batch-type=smp --cores 1 \ -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order= 1 \ isr.doEmpiricalReadNoise=True \ --id expId= 3020100800045 detector= 166 Setting `doEmpiricalReadNoise` forces ISR to ignore the camera defined read noise levels and independently estimate it from the data.  This provides a better variance to the CR code, and it then flags a much smaller number of pixels (132 CR found on 3020100800045). Once DM-27010 is complete, we will be able to supply a PTC dataset to ISR, and pull gain and read noise values from there instead of from the camera definition.  I think this means that creating initial calibrations for a camera is going to need to iterate: Initial bias, dark with `doEmpiricalReadNoise=True`. Initial PTC to generate more reliable read noise estimates. Second dark with `doEmpiricalReadNoise=False` supplying the initial PTC as the source for the read noise. Possible second PTC using second dark if the empirical read noise and PTC measured read noise values are significantly different.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            The change itself is trivial, and it looks good if it helps constructDark. And I also see that you outline above an iterative process that is self-contained within the DM code to deal with this via doEmpiricalNoise. The more general question is, given that we have gain, full-well capacity at saturation, and noise values from the latest BOT runs (both using eoTtest and DM tools), can't we just update all these YAML files with those more realistic values? Maybe this is a more general discussion.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - The change itself is trivial, and it looks good if it helps constructDark . And I also see that you outline above an iterative process that is self-contained within the DM code to deal with this via doEmpiricalNoise . The more general question is, given that we have gain, full-well capacity at saturation, and noise values from the latest BOT runs (both using eoTtest and DM tools), can't we just update all these YAML files with those more realistic values? Maybe this is a more general discussion.
            Hide
            cslage Craig Lage added a comment -

            I ran run 12639 with doEmpiricalReadNoise=True.  It hasn't finished with all of the PTC analysis yet, but the problematic CCD 166 successfully generated the dark, flat, and defect calibration files.  I also took the resulting empirical noise values that were extracted and compared them to the noise values measured at EOtest.  They correlate well (see plot) although not with a slope of 1.  Interesting that the E2V and ITL detectors have slightly different slopes.  Both measures agree that detector 166 amp C14 is the worst one.

            Show
            cslage Craig Lage added a comment - I ran run 12639 with doEmpiricalReadNoise=True.  It hasn't finished with all of the PTC analysis yet, but the problematic CCD 166 successfully generated the dark, flat, and defect calibration files.  I also took the resulting empirical noise values that were extracted and compared them to the noise values measured at EOtest.  They correlate well (see plot) although not with a slope of 1.  Interesting that the E2V and ITL detectors have slightly different slopes.  Both measures agree that detector 166 amp C14 is the worst one.
            Hide
            czw Christopher Waters added a comment -

            I've updated the `obs_lsst` readnoise for this chip as well, using an estimate from a few exposures.  That should solve the issue without any extra effort, at least until we can fully update the gain/readnoise/saturation values.

            Show
            czw Christopher Waters added a comment - I've updated the `obs_lsst` readnoise for this chip as well, using an estimate from a few exposures.  That should solve the issue without any extra effort, at least until we can fully update the gain/readnoise/saturation values.
            Hide
            cslage Craig Lage added a comment -

            Run 12639 has now finished with the PTC analysis, and CCD 166 successfully ran all the way through.  I attached a log-log PTC plot, and you can see the high noise, but at least it ran.  I agree this issue is resolved.  We can just use the doEmpiricalReadNoise=True option until we have valid values populating the camera.

            Show
            cslage Craig Lage added a comment - Run 12639 has now finished with the PTC analysis, and CCD 166 successfully ran all the way through.  I attached a log-log PTC plot, and you can see the high noise, but at least it ran.  I agree this issue is resolved.  We can just use the doEmpiricalReadNoise=True option until we have valid values populating the camera.

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              cslage Craig Lage
              Reviewers:
              Andrés Alejandro Plazas Malagón
              Watchers:
              Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.