# Investigate amp background level offsets in DECam postISRCCDs

XMLWordPrintable

#### Details

• Type: Bug
• Status: To Do
• Resolution: Unresolved
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Sprint:
AP F21-6 (November)
• Team:
• Urgent?:
No

#### Description

If you scroll through here, the problem is immediately apparent

https://lsst.ncsa.illinois.edu/~lskelvin/hits2014/

Hat-tip to Lee Kelvin for noticing this problem and plotting the images. This feels very familiar to when ISR order of operations was inconsistent regarding when assembleCcd happens, but I don't remember the resolution.

Possibly relevant and/or wrong: https://community.lsst.org/t/a-change-to-the-order-of-operations-for-isr/3060

#### Attachments

1. merian_bias_01.png
821 kB
2. merian_bias_35.png
909 kB
3. merian_bias_45.png
800 kB
4. merian_flat_01.png
2.97 MB
5. merian_flat_05.png
2.60 MB
6. merian_flat_25.png
3.07 MB
7. merian_flat_35.png
4.36 MB
8. merian_flat_45.png
2.66 MB
9. merian_flat_55.png
3.99 MB

#### Activity

Hide
Meredith Rawls added a comment -

Meredith, Chris, and Lee plan to work together on this during the 11am PDT June 9 pair coding.

Show
Meredith Rawls added a comment - Meredith, Chris, and Lee plan to work together on this during the 11am PDT June 9 pair coding.
Hide
Meredith Rawls added a comment -

We investigated today and were able to characterize the problem as well as come up with a potential solution.

We see discontinuities in postISRCCDs both left-right (amps) and top-bottom.

We think that if we configure isrTask to use overscan.fitType='MEDIAN_PER_ROW' (instead of the default fittype='MEDIAN') that will fix the top-bottom level mismatch. This needs to be done on ALL raws, calib and science frames, when re-running ISR, and probably in the crosstalk-prep step as well. https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/overscan.py#L48

We think that if we configure cpFlatNormalizationTask to use level='AMP' (instead of the default level='DETECTOR') this will fix the left-right amp-level mismatch. https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/cpFlatNormTask.py#L165

The next step is to try these configs in new DECam reruns entirely from scratch and see how things look.

Show
Meredith Rawls added a comment - We investigated today and were able to characterize the problem as well as come up with a potential solution. We see discontinuities in postISRCCDs both left-right (amps) and top-bottom. We think that if we configure isrTask to use overscan.fitType='MEDIAN_PER_ROW' (instead of the default fittype='MEDIAN') that will fix the top-bottom level mismatch. This needs to be done on ALL raws, calib and science frames, when re-running ISR, and probably in the crosstalk-prep step as well.  https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/overscan.py#L48 We think that if we configure cpFlatNormalizationTask to use level='AMP' (instead of the default level='DETECTOR') this will fix the left-right amp-level mismatch.  https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/cpFlatNormTask.py#L165 The next step is to try these configs in new DECam reruns entirely from scratch and see how things look.
Hide
Christopher Waters added a comment -

To add an extra note: updating the overscan fitType will make all of the crosstalkSources/pre-generated overscan subtracted images used for inter-chip crosstalk slightly wrong.  Given the scale of the crosstalk (~1e5-1e-4), and the scale of the differences (~1 ADU), this is probably not worth rerunning, but I wanted to make sure it was mentioned at least.

This may show up as slight roll downs at the top of crosstalk corrected images (run with median-per-row that eliminates the bright roll up at large y) as the old crosstalkSources (which were run with median, and so still have that bright roll up) are scaled and subtracted off.  Again, my quick calculation suggests this should still be smaller than an ADU (1e-4 * 2500 = 0.25), but I wanted to bring up the point.

Show
Christopher Waters added a comment - To add an extra note: updating the overscan fitType will make all of the crosstalkSources/pre-generated overscan subtracted images used for inter-chip crosstalk slightly wrong.  Given the scale of the crosstalk (~1e5-1e-4), and the scale of the differences (~1 ADU), this is probably not worth rerunning, but I wanted to make sure it was mentioned at least.  This may show up as slight roll downs at the top of crosstalk corrected images (run with median-per-row that eliminates the bright roll up at large y) as the old crosstalkSources (which were run with median, and so still have that bright roll up) are scaled and subtracted off.  Again, my quick calculation suggests this should still be smaller than an ADU (1e-4 * 2500 = 0.25), but I wanted to bring up the point.
Hide
Lee Kelvin added a comment - - edited

I've been able to re-run the $CP_PIPE_DIR/pipelines/cpBias.yaml pipeline to regenerate master biases for my Merian DECam data using the config option isr:overscan.fitType='MEDIAN_PER_ROW'. The plots below show a few examples of the before (left) and after (right): The biases with the new config option in place appear much healthier, resolving the upper/lower discontinuity, and also the left/right amp-to-amp discontinuity. Plots for all 62 detectors can be found linked: I'll continue rerunning my test data to see what else changes - next is generating crosstalk sources and building flat frames. Show Lee Kelvin added a comment - - edited I've been able to re-run the$CP_PIPE_DIR/pipelines/cpBias.yaml pipeline to regenerate master biases for my Merian DECam data using the config option isr:overscan.fitType='MEDIAN_PER_ROW' . The plots below show a few examples of the before (left) and after (right): The biases with the new config option in place appear much healthier, resolving the upper/lower discontinuity, and also the left/right amp-to-amp discontinuity. Plots for all 62 detectors can be found linked: current defaults: https://tigress-web.princeton.edu/~lkelvin/merian/bias-det/ new configuration: https://tigress-web.princeton.edu/~lkelvin/merian/bias-det-mpr/ I'll continue rerunning my test data to see what else changes - next is generating crosstalk sources and building flat frames.
Hide
Lee Kelvin added a comment - - edited

Following on from above, I've re-run the $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml pipeline to regenerate crosstalk sources using the MEDIAN_PER_ROW fitType, and then regenerated flats using the$CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpFlat.yaml pipeline.

Including the original (default) flats data I have available, I now have three flats outputs:

1. isr:overscan.fitType='MEDIAN' AND cpFlatNorm:level='DETECTOR' (current default)
2. isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='DETECTOR'
3. isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='AMP'

The plots attached here show comparisons for a few selected DECam detectors:

I'm unsure which of these flats configurations look the healthiest, and would be very keen to hear from others who are more experienced here. From the bias testing above, it seems that MEDIAN_PER_ROW is optimal for DECam, but the impact of switching cpFlatNormalizationTask from per-detector to per-amp varying results, with a large variation as a function of detector.

Show
Lee Kelvin added a comment - - edited Following on from above, I've re-run the $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml pipeline to regenerate crosstalk sources using the MEDIAN_PER_ROW fitType, and then regenerated flats using the$CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpFlat.yaml pipeline. Including the original (default) flats data I have available, I now have three flats outputs: isr:overscan.fitType='MEDIAN' AND cpFlatNorm:level='DETECTOR' (current default) isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='DETECTOR' isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='AMP' The plots attached here show comparisons for a few selected DECam detectors: I'm unsure which of these flats configurations look the healthiest, and would be very keen to hear from others who are more experienced here. From the bias testing above, it seems that MEDIAN_PER_ROW is optimal for DECam, but the impact of switching cpFlatNormalizationTask from per-detector to per-amp varying results, with a large variation as a function of detector.
Hide
Meredith Rawls added a comment -

Before closing this ticket, please remove references to it in the obs_decam docs, specifically the using-obs-decam.rst file written for DM-24913.

Show
Meredith Rawls added a comment - Before closing this ticket, please remove references to it in the obs_decam docs, specifically the using-obs-decam.rst file written for DM-24913 .

#### People

Assignee:
Kenneth Herner
Reporter:
Meredith Rawls
Watchers:
Christopher Waters, Dino Bektesevic, Eric Bellm, Ian Sullivan, Lee Kelvin, Meredith Rawls