Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30651

Investigate amp background level offsets in DECam postISRCCDs

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Sprint:
      AP S22-1 (December)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      If you scroll through here, the problem is immediately apparent

      https://lsst.ncsa.illinois.edu/~lskelvin/hits2014/

      Hat-tip to Lee Kelvin for noticing this problem and plotting the images. This feels very familiar to when ISR order of operations was inconsistent regarding when assembleCcd happens, but I don't remember the resolution.

      Possibly relevant and/or wrong: https://community.lsst.org/t/a-change-to-the-order-of-operations-for-isr/3060 

        Attachments

        1. merian_bias_01.png
          merian_bias_01.png
          821 kB
        2. merian_bias_35.png
          merian_bias_35.png
          909 kB
        3. merian_bias_45.png
          merian_bias_45.png
          800 kB
        4. merian_flat_01.png
          merian_flat_01.png
          2.97 MB
        5. merian_flat_05.png
          merian_flat_05.png
          2.60 MB
        6. merian_flat_25.png
          merian_flat_25.png
          3.07 MB
        7. merian_flat_35.png
          merian_flat_35.png
          4.36 MB
        8. merian_flat_45.png
          merian_flat_45.png
          2.66 MB
        9. merian_flat_55.png
          merian_flat_55.png
          3.99 MB

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            Meredith, Chris, and Lee plan to work together on this during the 11am PDT June 9 pair coding.

            Show
            mrawls Meredith Rawls added a comment - Meredith, Chris, and Lee plan to work together on this during the 11am PDT June 9 pair coding.
            Hide
            mrawls Meredith Rawls added a comment -

            We investigated today and were able to characterize the problem as well as come up with a potential solution.

            We see discontinuities in postISRCCDs both left-right (amps) and top-bottom.

            We think that if we configure isrTask to use overscan.fitType='MEDIAN_PER_ROW' (instead of the default fittype='MEDIAN') that will fix the top-bottom level mismatch. This needs to be done on ALL raws, calib and science frames, when re-running ISR, and probably in the crosstalk-prep step as well. https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/overscan.py#L48

            We think that if we configure cpFlatNormalizationTask to use level='AMP' (instead of the default level='DETECTOR') this will fix the left-right amp-level mismatch. https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/cpFlatNormTask.py#L165

            The next step is to try these configs in new DECam reruns entirely from scratch and see how things look. 

            Show
            mrawls Meredith Rawls added a comment - We investigated today and were able to characterize the problem as well as come up with a potential solution. We see discontinuities in postISRCCDs both left-right (amps) and top-bottom. We think that if we configure isrTask to use overscan.fitType='MEDIAN_PER_ROW' (instead of the default fittype='MEDIAN') that will fix the top-bottom level mismatch. This needs to be done on ALL raws, calib and science frames, when re-running ISR, and probably in the crosstalk-prep step as well.  https://github.com/lsst/ip_isr/blob/master/python/lsst/ip/isr/overscan.py#L48 We think that if we configure cpFlatNormalizationTask to use level='AMP' (instead of the default level='DETECTOR') this will fix the left-right amp-level mismatch.  https://github.com/lsst/cp_pipe/blob/master/python/lsst/cp/pipe/cpFlatNormTask.py#L165 The next step is to try these configs in new DECam reruns entirely from scratch and see how things look. 
            Hide
            czw Christopher Waters added a comment -

            To add an extra note: updating the overscan fitType will make all of the `crosstalkSources`/pre-generated overscan subtracted images used for inter-chip crosstalk slightly wrong.  Given the scale of the crosstalk (~1e5-1e-4), and the scale of the differences (~1 ADU), this is probably not worth rerunning, but I wanted to make sure it was mentioned at least. 

            This may show up as slight roll downs at the top of crosstalk corrected images (run with median-per-row that eliminates the bright roll up at large y) as the old crosstalkSources (which were run with median, and so still have that bright roll up) are scaled and subtracted off.  Again, my quick calculation suggests this should still be smaller than an ADU (1e-4 * 2500 = 0.25), but I wanted to bring up the point.

            Show
            czw Christopher Waters added a comment - To add an extra note: updating the overscan fitType will make all of the `crosstalkSources`/pre-generated overscan subtracted images used for inter-chip crosstalk slightly wrong.  Given the scale of the crosstalk (~1e5-1e-4), and the scale of the differences (~1 ADU), this is probably not worth rerunning, but I wanted to make sure it was mentioned at least.  This may show up as slight roll downs at the top of crosstalk corrected images (run with median-per-row that eliminates the bright roll up at large y) as the old crosstalkSources (which were run with median, and so still have that bright roll up) are scaled and subtracted off.  Again, my quick calculation suggests this should still be smaller than an ADU (1e-4 * 2500 = 0.25), but I wanted to bring up the point.
            Hide
            lskelvin Lee Kelvin added a comment - - edited

            I've been able to re-run the $CP_PIPE_DIR/pipelines/cpBias.yaml pipeline to regenerate master biases for my Merian DECam data using the config option isr:overscan.fitType='MEDIAN_PER_ROW'. The plots below show a few examples of the before (left) and after (right):

            The biases with the new config option in place appear much healthier, resolving the upper/lower discontinuity, and also the left/right amp-to-amp discontinuity.

            Plots for all 62 detectors can be found linked:

            I'll continue rerunning my test data to see what else changes - next is generating crosstalk sources and building flat frames.

            Show
            lskelvin Lee Kelvin added a comment - - edited I've been able to re-run the $CP_PIPE_DIR/pipelines/cpBias.yaml pipeline to regenerate master biases for my Merian DECam data using the config option isr:overscan.fitType='MEDIAN_PER_ROW' . The plots below show a few examples of the before (left) and after (right): The biases with the new config option in place appear much healthier, resolving the upper/lower discontinuity, and also the left/right amp-to-amp discontinuity. Plots for all 62 detectors can be found linked: current defaults: https://tigress-web.princeton.edu/~lkelvin/merian/bias-det/ new configuration: https://tigress-web.princeton.edu/~lkelvin/merian/bias-det-mpr/ I'll continue rerunning my test data to see what else changes - next is generating crosstalk sources and building flat frames.
            Hide
            lskelvin Lee Kelvin added a comment - - edited

            Following on from above, I've re-run the $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml pipeline to regenerate crosstalk sources using the MEDIAN_PER_ROW fitType, and then regenerated flats using the $CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpFlat.yaml pipeline.

            Including the original (default) flats data I have available, I now have three flats outputs:

            1. isr:overscan.fitType='MEDIAN' AND cpFlatNorm:level='DETECTOR' (current default)
            2. isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='DETECTOR'
            3. isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='AMP'

            The plots attached here show comparisons for a few selected DECam detectors:


            I'm unsure which of these flats configurations look the healthiest, and would be very keen to hear from others who are more experienced here. From the bias testing above, it seems that MEDIAN_PER_ROW is optimal for DECam, but the impact of switching cpFlatNormalizationTask from per-detector to per-amp varying results, with a large variation as a function of detector.

            Show
            lskelvin Lee Kelvin added a comment - - edited Following on from above, I've re-run the $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml pipeline to regenerate crosstalk sources using the MEDIAN_PER_ROW fitType, and then regenerated flats using the $CP_PIPE_DIR/pipelines/DarkEnergyCamera/cpFlat.yaml pipeline. Including the original (default) flats data I have available, I now have three flats outputs: isr:overscan.fitType='MEDIAN' AND cpFlatNorm:level='DETECTOR' (current default) isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='DETECTOR' isr:overscan.fitType='MEDIAN_PER_ROW' AND cpFlatNorm:level='AMP' The plots attached here show comparisons for a few selected DECam detectors: I'm unsure which of these flats configurations look the healthiest, and would be very keen to hear from others who are more experienced here. From the bias testing above, it seems that MEDIAN_PER_ROW is optimal for DECam, but the impact of switching cpFlatNormalizationTask from per-detector to per-amp varying results, with a large variation as a function of detector.
            Hide
            mrawls Meredith Rawls added a comment -

            Before closing this ticket, please remove references to it in the obs_decam docs, specifically the using-obs-decam.rst file written for DM-24913.

            Show
            mrawls Meredith Rawls added a comment - Before closing this ticket, please remove references to it in the obs_decam docs, specifically the using-obs-decam.rst file written for DM-24913 .

              People

              Assignee:
              kherner Kenneth Herner
              Reporter:
              mrawls Meredith Rawls
              Watchers:
              Christopher Waters, Dino Bektesevic, Eric Bellm, Ian Sullivan, Lee Kelvin, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  CI Builds

                  No builds found.