# constructDark.py fails on LSSTCam with 'Too many CR pixels'

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

I ran a PTC task on BOT run 12606, the first time we have the full LSSTCam.  Six CCDs failed to return results.  I am tracking down the reasons.  CCD R41_S11, detector=166 failed because it failed to create a master dark with the error "Too many CR pixels (max 100000)". Below is a command line that reproduces the error, as well as the error string itself.  I'm also attaching the dark image, which looks fine.  FWIW, I have seen this error before on several occasions, so this is not an isolated incident.

Command line:

 constructDark.py /project/shared/BOT --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_12606/CALIB --rerun /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_Test --batch-type=smp --cores 1 -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 --id expId=3020100800045 detector=166

Error:

 dark WARN: Unable to process DataId(initialdata={'expId': 3020100800045, 'detector': 166, 'dayObs': '2020-10-08', 'raftName': 'R41', 'detectorName': 'S11', 'snap': 0}, tag=set()):   File "src/CR.cc", line 410, in std::vector > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::daf::base::PropertySet&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage]  Too many CR pixels (max 100000) {0} lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 100000)'LengthError on lsst-devl01:2455582 in reduce:   File "src/CR.cc", line 410, in std::vector > lsst::meas::algorithms::findCosmicRays(MaskedImageT&, const lsst::afw::detection::Psf&, double, const lsst::daf::base::PropertySet&, bool) [with MaskedImageT = lsst::afw::image::MaskedImage]  Too many CR pixels (max 100000) {0} lsst::pex::exceptions::LengthError: 'Too many CR pixels (max 100000)'

#### Attachments

1. 3020100800045-R41-S11.png
315 kB
2. Noise_Comparison_12639_23Oct20.png
22 kB
3. Noise_Summary_12606_16Oct20.png
125 kB
4. postISR-3020100800045-R41-S11.png
863 kB
5. PTC_R41_S11_C14.png
10 kB
6. R41_S11_C14.png
56 kB

#### Activity

Hide
Christopher Waters added a comment -

Thinking through this again, and trying to place fixes at various different levels, I came to what is likely the best solution for now:

  constructDark.py /project/shared/BOT \  --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/CALIB \  --rerun /project/czw/DM-27158/run1016a \  --batch-type=smp --cores 1 \  -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order=1 \  isr.doEmpiricalReadNoise=True \  --id expId=3020100800045 detector=166

Setting doEmpiricalReadNoise forces ISR to ignore the camera defined read noise levels and independently estimate it from the data.  This provides a better variance to the CR code, and it then flags a much smaller number of pixels (132 CR found on 3020100800045).

Once DM-27010 is complete, we will be able to supply a PTC dataset to ISR, and pull gain and read noise values from there instead of from the camera definition.  I think this means that creating initial calibrations for a camera is going to need to iterate:

1. Initial bias, dark with doEmpiricalReadNoise=True.
2. Initial PTC to generate more reliable read noise estimates.
3. Second dark with doEmpiricalReadNoise=False supplying the initial PTC as the source for the read noise.
4. Possible second PTC using second dark if the empirical read noise and PTC measured read noise values are significantly different.
Show
Christopher Waters added a comment - Thinking through this again, and trying to place fixes at various different levels, I came to what is likely the best solution for now: constructDark.py /project/shared/BOT \ --calib /project/shared/BOT/rerun/cslage/PTC_LSSTCAM_New_12606/CALIB \ --rerun /project/czw/DM- 27158 /run1016a \ --batch-type=smp --cores 1 \ -c isr.doCrosstalk=False isr.overscan.fitType=MEDIAN_PER_ROW isr.overscan.order= 1 \ isr.doEmpiricalReadNoise=True \ --id expId= 3020100800045 detector= 166 Setting doEmpiricalReadNoise forces ISR to ignore the camera defined read noise levels and independently estimate it from the data.  This provides a better variance to the CR code, and it then flags a much smaller number of pixels (132 CR found on 3020100800045). Once DM-27010 is complete, we will be able to supply a PTC dataset to ISR, and pull gain and read noise values from there instead of from the camera definition.  I think this means that creating initial calibrations for a camera is going to need to iterate: Initial bias, dark with doEmpiricalReadNoise=True. Initial PTC to generate more reliable read noise estimates. Second dark with doEmpiricalReadNoise=False supplying the initial PTC as the source for the read noise. Possible second PTC using second dark if the empirical read noise and PTC measured read noise values are significantly different.
Hide
Andrés Alejandro Plazas Malagón added a comment -

The change itself is trivial, and it looks good if it helps constructDark. And I also see that you outline above an iterative process that is self-contained within the DM code to deal with this via doEmpiricalNoise. The more general question is, given that we have gain, full-well capacity at saturation, and noise values from the latest BOT runs (both using eoTtest and DM tools), can't we just update all these YAML files with those more realistic values? Maybe this is a more general discussion.

Show
Andrés Alejandro Plazas Malagón added a comment - The change itself is trivial, and it looks good if it helps constructDark . And I also see that you outline above an iterative process that is self-contained within the DM code to deal with this via doEmpiricalNoise . The more general question is, given that we have gain, full-well capacity at saturation, and noise values from the latest BOT runs (both using eoTtest and DM tools), can't we just update all these YAML files with those more realistic values? Maybe this is a more general discussion.
Hide
Craig Lage added a comment -

I ran run 12639 with doEmpiricalReadNoise=True.  It hasn't finished with all of the PTC analysis yet, but the problematic CCD 166 successfully generated the dark, flat, and defect calibration files.  I also took the resulting empirical noise values that were extracted and compared them to the noise values measured at EOtest.  They correlate well (see plot) although not with a slope of 1.  Interesting that the E2V and ITL detectors have slightly different slopes.  Both measures agree that detector 166 amp C14 is the worst one.

Show
Craig Lage added a comment - I ran run 12639 with doEmpiricalReadNoise=True.  It hasn't finished with all of the PTC analysis yet, but the problematic CCD 166 successfully generated the dark, flat, and defect calibration files.  I also took the resulting empirical noise values that were extracted and compared them to the noise values measured at EOtest.  They correlate well (see plot) although not with a slope of 1.  Interesting that the E2V and ITL detectors have slightly different slopes.  Both measures agree that detector 166 amp C14 is the worst one.
Hide
Christopher Waters added a comment -

I've updated the obs_lsst readnoise for this chip as well, using an estimate from a few exposures.  That should solve the issue without any extra effort, at least until we can fully update the gain/readnoise/saturation values.

Show
Christopher Waters added a comment - I've updated the obs_lsst readnoise for this chip as well, using an estimate from a few exposures.  That should solve the issue without any extra effort, at least until we can fully update the gain/readnoise/saturation values.
Hide
Craig Lage added a comment -

Run 12639 has now finished with the PTC analysis, and CCD 166 successfully ran all the way through.  I attached a log-log PTC plot, and you can see the high noise, but at least it ran.  I agree this issue is resolved.  We can just use the doEmpiricalReadNoise=True option until we have valid values populating the camera.

Show
Craig Lage added a comment - Run 12639 has now finished with the PTC analysis, and CCD 166 successfully ran all the way through.  I attached a log-log PTC plot, and you can see the high noise, but at least it ran.  I agree this issue is resolved.  We can just use the doEmpiricalReadNoise=True option until we have valid values populating the camera.

#### People

Assignee:
Christopher Waters
Reporter:
Craig Lage
Reviewers:
Andrés Alejandro Plazas Malagón
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, Craig Lage