Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24389

Implement amp-to-amp offset correction task

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 36
    • Data Release Production
    • No

    Description

      HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into ip_isr, and is redirected to the primary run task in obs_subaru. This task may be switched on or off using the doAmpOffset flag, which currently defaults to False.

      In summary, this process generates an estimate of the amp offset from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp offset value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp offsets. We refer to these additive/subtractive values as pedestals in the exposure metadata.

      The method for generating an amp offset value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary DETECTION footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in background.ignoredPixelMask are not used in determining amp offsets (defaulting to BAD, SAT, INTRP, CR, EDGE, DETECTED, DETECTED_NEGATIVE, SUSPECT and NO_DATA). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by ampEdgeInset (default 5 pixels). The width of the strip is given by ampEdgeWidth (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size ampEdgeWindow (default 512 pixels). Finally, an estimate of the amp A to amp B offset is given by calculating the 3-sigma clipped mean of this rolling average data.

      During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than ampEdgeMinFrac (default 0.5) then no amp offset estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting if addition to detected source footprints can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp offset values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp offset estimate larger than ampEdgeMaxOffset (default 5 counts) will also be ignored, and no amp offset estimate will be generated for that amp boundary.

      All added/subtracted pedestals are entered into the exposure metadata as PEDESTALX (where X is the number of the amp: PEDESTAL1, PEDESTAL2...). The net sum of all pedestal corrections for a single CCD is zero.

      Full outputs for a test run on 200 visits from DM-23243 (PDR2) wide can be found on lsst-devl at /project/lskelvin/patConCorr/delta-v1 or at this link. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits. Additional information and discussion on the impact these corrections have on coadd level data can also be found in the comments below.

      Attachments

        1. v1252_calexp_vanilla.png
          v1252_calexp_vanilla.png
          522 kB
        2. v1252_calexp_ampoff.png
          v1252_calexp_ampoff.png
          525 kB
        3. v1252_calexp_diff.png
          v1252_calexp_diff.png
          190 kB
        4. v7206c14images.png
          v7206c14images.png
          184 kB
        5. v7206c14graphs.png
          v7206c14graphs.png
          481 kB
        6. v1252c003.png
          v1252c003.png
          299 kB
        7. v1252c011.png
          v1252c011.png
          365 kB
        8. v1252c019.png
          v1252c019.png
          362 kB
        9. v1252c035.png
          v1252c035.png
          374 kB
        10. v1252c036.png
          v1252c036.png
          369 kB
        11. v1252c040.png
          v1252c040.png
          366 kB
        12. v1252c067.png
          v1252c067.png
          365 kB
        13. v1252c068.png
          v1252c068.png
          365 kB
        14. v1252c043.png
          v1252c043.png
          345 kB
        15. v1252c069.png
          v1252c069.png
          340 kB
        16. rc2i_t9615_p00.png
          rc2i_t9615_p00.png
          1.57 MB
        17. rc2i_t9615_p05.png
          rc2i_t9615_p05.png
          1.31 MB
        18. rc2i_t9615_p25.png
          rc2i_t9615_p25.png
          1.56 MB
        19. rc2i_t9615_p55.png
          rc2i_t9615_p55.png
          1.31 MB
        20. pipeline_step1.pdf
          30 kB
        21. pipeline_step2.pdf
          18 kB
        22. pipeline_step3.pdf
          20 kB
        23. rc2i_iCModelMag.png
          rc2i_iCModelMag.png
          26 kB

        Issue Links

          Activity

            erykoff Eli Rykoff added a comment -

            Okay, this is all looking good. However, I realized looking through the code that there don't appear to be any unit tests to go along with the code. That is, you've done the due diligence with offline tests that everything works, and if we do run a full RC2 (or turn it on for regular RC2) then that will test the code infrequently. But if we could create at least one isr test that turns doAmpOffset on and makes sure that it doesn't explode that would be very helpful to ensure it doesn't break in the future.

            erykoff Eli Rykoff added a comment - Okay, this is all looking good. However, I realized looking through the code that there don't appear to be any unit tests to go along with the code. That is, you've done the due diligence with offline tests that everything works, and if we do run a full RC2 (or turn it on for regular RC2) then that will test the code infrequently. But if we could create at least one isr test that turns doAmpOffset on and makes sure that it doesn't explode that would be very helpful to ensure it doesn't break in the future.
            lskelvin Lee Kelvin added a comment -

            Attached is a comparison plot showing the two iCModel magnitude histograms as output in the objectTable_tract dataset type. As shown, there is little difference in these two outputs, with minor fluctuations becoming apparent at the extreme bright/faint ends (note the logarithmic y-axis).

            I'll construct additional analysis plots from this table to see if anything else looks interesting. Thanks for your feedback above Eli. I'll start construction of a basic unit-test as you suggest and get back to you shortly.

            lskelvin Lee Kelvin added a comment - Attached is a comparison plot showing the two iCModel magnitude histograms as output in the objectTable_tract dataset type. As shown, there is little difference in these two outputs, with minor fluctuations becoming apparent at the extreme bright/faint ends (note the logarithmic y-axis). I'll construct additional analysis plots from this table to see if anything else looks interesting. Thanks for your feedback above Eli. I'll start construction of a basic unit-test as you suggest and get back to you shortly.
            lskelvin Lee Kelvin added a comment -

            Thanks again for your earlier suggestion Eli. I've now added a couple of unit tests which run the amp-offset code in two different modes: once with background estimation switched off, and a second time with background estimation switched on. I hope these simple tests were what you had in mind?

            lskelvin Lee Kelvin added a comment - Thanks again for your earlier suggestion Eli. I've now added a couple of unit tests which run the amp-offset code in two different modes: once with background estimation switched off, and a second time with background estimation switched on. I hope these simple tests were what you had in mind?
            erykoff Eli Rykoff added a comment -

            Some minor comments on the PR. Otherwise looks good! Congrats!

            erykoff Eli Rykoff added a comment - Some minor comments on the PR. Otherwise looks good! Congrats!
            lskelvin Lee Kelvin added a comment -

            Thanks again Eli. Final Jenkins run passed without issue - branches merged and deleted - cheers!

            lskelvin Lee Kelvin added a comment - Thanks again Eli. Final Jenkins run passed without issue - branches merged and deleted - cheers!

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Eli Rykoff
              Christopher Waters, Eli Rykoff, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.