Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24389

Implement amp-to-amp offset correction task

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 36
    • Data Release Production
    • No

    Description

      HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into ip_isr, and is redirected to the primary run task in obs_subaru. This task may be switched on or off using the doAmpOffset flag, which currently defaults to False.

      In summary, this process generates an estimate of the amp offset from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp offset value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp offsets. We refer to these additive/subtractive values as pedestals in the exposure metadata.

      The method for generating an amp offset value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary DETECTION footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in background.ignoredPixelMask are not used in determining amp offsets (defaulting to BAD, SAT, INTRP, CR, EDGE, DETECTED, DETECTED_NEGATIVE, SUSPECT and NO_DATA). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by ampEdgeInset (default 5 pixels). The width of the strip is given by ampEdgeWidth (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size ampEdgeWindow (default 512 pixels). Finally, an estimate of the amp A to amp B offset is given by calculating the 3-sigma clipped mean of this rolling average data.

      During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than ampEdgeMinFrac (default 0.5) then no amp offset estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting if addition to detected source footprints can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp offset values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp offset estimate larger than ampEdgeMaxOffset (default 5 counts) will also be ignored, and no amp offset estimate will be generated for that amp boundary.

      All added/subtracted pedestals are entered into the exposure metadata as PEDESTALX (where X is the number of the amp: PEDESTAL1, PEDESTAL2...). The net sum of all pedestal corrections for a single CCD is zero.

      Full outputs for a test run on 200 visits from DM-23243 (PDR2) wide can be found on lsst-devl at /project/lskelvin/patConCorr/delta-v1 or at this link. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits. Additional information and discussion on the impact these corrections have on coadd level data can also be found in the comments below.

      Attachments

        1. pipeline_step1.pdf
          30 kB
        2. pipeline_step2.pdf
          18 kB
        3. pipeline_step3.pdf
          20 kB
        4. rc2i_iCModelMag.png
          rc2i_iCModelMag.png
          26 kB
        5. rc2i_t9615_p00.png
          rc2i_t9615_p00.png
          1.57 MB
        6. rc2i_t9615_p05.png
          rc2i_t9615_p05.png
          1.31 MB
        7. rc2i_t9615_p25.png
          rc2i_t9615_p25.png
          1.56 MB
        8. rc2i_t9615_p55.png
          rc2i_t9615_p55.png
          1.31 MB
        9. v1252_calexp_ampoff.png
          v1252_calexp_ampoff.png
          525 kB
        10. v1252_calexp_diff.png
          v1252_calexp_diff.png
          190 kB
        11. v1252_calexp_vanilla.png
          v1252_calexp_vanilla.png
          522 kB
        12. v1252c003.png
          v1252c003.png
          299 kB
        13. v1252c011.png
          v1252c011.png
          365 kB
        14. v1252c019.png
          v1252c019.png
          362 kB
        15. v1252c035.png
          v1252c035.png
          374 kB
        16. v1252c036.png
          v1252c036.png
          369 kB
        17. v1252c040.png
          v1252c040.png
          366 kB
        18. v1252c043.png
          v1252c043.png
          345 kB
        19. v1252c067.png
          v1252c067.png
          365 kB
        20. v1252c068.png
          v1252c068.png
          365 kB
        21. v1252c069.png
          v1252c069.png
          340 kB
        22. v7206c14graphs.png
          v7206c14graphs.png
          481 kB
        23. v7206c14images.png
          v7206c14images.png
          184 kB

        Issue Links

          Activity

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Eli Rykoff
              Christopher Waters, Eli Rykoff, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.