Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24389

Implement amp-to-amp offset correction task

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 36
    • Data Release Production
    • No

    Description

      HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into ip_isr, and is redirected to the primary run task in obs_subaru. This task may be switched on or off using the doAmpOffset flag, which currently defaults to False.

      In summary, this process generates an estimate of the amp offset from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp offset value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp offsets. We refer to these additive/subtractive values as pedestals in the exposure metadata.

      The method for generating an amp offset value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary DETECTION footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in background.ignoredPixelMask are not used in determining amp offsets (defaulting to BAD, SAT, INTRP, CR, EDGE, DETECTED, DETECTED_NEGATIVE, SUSPECT and NO_DATA). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by ampEdgeInset (default 5 pixels). The width of the strip is given by ampEdgeWidth (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size ampEdgeWindow (default 512 pixels). Finally, an estimate of the amp A to amp B offset is given by calculating the 3-sigma clipped mean of this rolling average data.

      During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than ampEdgeMinFrac (default 0.5) then no amp offset estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting if addition to detected source footprints can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp offset values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp offset estimate larger than ampEdgeMaxOffset (default 5 counts) will also be ignored, and no amp offset estimate will be generated for that amp boundary.

      All added/subtracted pedestals are entered into the exposure metadata as PEDESTALX (where X is the number of the amp: PEDESTAL1, PEDESTAL2...). The net sum of all pedestal corrections for a single CCD is zero.

      Full outputs for a test run on 200 visits from DM-23243 (PDR2) wide can be found on lsst-devl at /project/lskelvin/patConCorr/delta-v1 or at this link. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits. Additional information and discussion on the impact these corrections have on coadd level data can also be found in the comments below.

      Attachments

        1. pipeline_step1.pdf
          30 kB
        2. pipeline_step2.pdf
          18 kB
        3. pipeline_step3.pdf
          20 kB
        4. rc2i_iCModelMag.png
          rc2i_iCModelMag.png
          26 kB
        5. rc2i_t9615_p00.png
          rc2i_t9615_p00.png
          1.57 MB
        6. rc2i_t9615_p05.png
          rc2i_t9615_p05.png
          1.31 MB
        7. rc2i_t9615_p25.png
          rc2i_t9615_p25.png
          1.56 MB
        8. rc2i_t9615_p55.png
          rc2i_t9615_p55.png
          1.31 MB
        9. v1252_calexp_ampoff.png
          v1252_calexp_ampoff.png
          525 kB
        10. v1252_calexp_diff.png
          v1252_calexp_diff.png
          190 kB
        11. v1252_calexp_vanilla.png
          v1252_calexp_vanilla.png
          522 kB
        12. v1252c003.png
          v1252c003.png
          299 kB
        13. v1252c011.png
          v1252c011.png
          365 kB
        14. v1252c019.png
          v1252c019.png
          362 kB
        15. v1252c035.png
          v1252c035.png
          374 kB
        16. v1252c036.png
          v1252c036.png
          369 kB
        17. v1252c040.png
          v1252c040.png
          366 kB
        18. v1252c043.png
          v1252c043.png
          345 kB
        19. v1252c067.png
          v1252c067.png
          365 kB
        20. v1252c068.png
          v1252c068.png
          365 kB
        21. v1252c069.png
          v1252c069.png
          340 kB
        22. v7206c14graphs.png
          v7206c14graphs.png
          481 kB
        23. v7206c14images.png
          v7206c14images.png
          184 kB

        Issue Links

          Activity

            lskelvin Lee Kelvin added a comment - - edited

            Movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset corrections on a sample of visits.

            HSC-G: 30s exposures, 150s exposures (30s diffs, 150s diffs)

            HSC-R: 30s exposures, 150s exposures (30s diffs, 150s diffs)

            HSC-I: 30s exposures, 200s exposures (30s diffs, 200s diffs)

            HSC-Z: 30s exposures, 200s exposures (30s diffs, 200s diffs)

            HSC-Y: 30s exposures, 200s exposures (30s diffs, 200s diffs)

            lskelvin Lee Kelvin added a comment - - edited Movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset corrections on a sample of visits. HSC-G : 30s exposures , 150s exposures ( 30s diffs , 150s diffs ) HSC-R : 30s exposures , 150s exposures ( 30s diffs , 150s diffs ) HSC-I : 30s exposures , 200s exposures ( 30s diffs , 200s diffs ) HSC-Z : 30s exposures , 200s exposures ( 30s diffs , 200s diffs ) HSC-Y : 30s exposures , 200s exposures ( 30s diffs , 200s diffs )
            lskelvin Lee Kelvin added a comment - - edited

            An example correction for our canonical visit 1252 (HSC-I, 30s exposure).

            lskelvin Lee Kelvin added a comment - - edited An example correction for our canonical visit 1252 (HSC-I, 30s exposure).
            lskelvin Lee Kelvin added a comment - - edited

            An example showing the impact of applying a rolling average estimator to the amp edge difference values. These data are for visit 7206, CCD 14. The first figure shows the vanilla calexp (left), the amp offset corrected frame without using rolling window averaging (middle) and the amp offset corrected frame when using window averaging (right). The graphs below look at data along the amp1-amp2 interface, showing the count level along the trailing edge of amp 1 (edgeA), the leading edge of amp 2 (edgeB), and the difference between these two data (edgeDiff = edgeB - edgeA). The lower two panels show the same data, with a different scale on the y-axis.

            As seen here, this particular CCD is dominated by a bright star in the middle of amp 2. When using the original data (blue) to generate an estimate of the edgeDiff pedestal level, the resultant value is overestimated by ~0.5 counts, using either a median (original nanmedian, black) or a sigma clipped mean (original MEANCLIP, purple). Switching to use of the rolling averaged data (orange) allows us to generate improved edgeDiff pedestal levels, using either a median (rolling nanmedian, red) or a sigma clipped mean (rolling MEANCLIP, green), with the latter adopted for use on this ticket. Also note that this particular CCD does not trigger either of our safety catches (ampEdgeMinFrac or ampEdgeMaxStep) as detailed above.

            lskelvin Lee Kelvin added a comment - - edited An example showing the impact of applying a rolling average estimator to the amp edge difference values. These data are for visit 7206, CCD 14. The first figure shows the vanilla calexp (left), the amp offset corrected frame without using rolling window averaging (middle) and the amp offset corrected frame when using window averaging (right). The graphs below look at data along the amp1-amp2 interface, showing the count level along the trailing edge of amp 1 (edgeA), the leading edge of amp 2 (edgeB), and the difference between these two data (edgeDiff = edgeB - edgeA). The lower two panels show the same data, with a different scale on the y-axis. As seen here, this particular CCD is dominated by a bright star in the middle of amp 2. When using the original data (blue) to generate an estimate of the edgeDiff pedestal level, the resultant value is overestimated by ~0.5 counts, using either a median (original nanmedian, black) or a sigma clipped mean (original MEANCLIP, purple). Switching to use of the rolling averaged data (orange) allows us to generate improved edgeDiff pedestal levels, using either a median (rolling nanmedian, red) or a sigma clipped mean (rolling MEANCLIP, green), with the latter adopted for use on this ticket. Also note that this particular CCD does not trigger either of our safety catches ( ampEdgeMinFrac or ampEdgeMaxStep ) as detailed above.
            lskelvin Lee Kelvin added a comment - Successfully completed Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33622/pipeline/48
            lskelvin Lee Kelvin added a comment -

            All CCDs plotted individually for our canonical visit 1252 can be found at this link: https://lsst.ncsa.illinois.edu/~lskelvin/project/patConCorr/delta-v1/plots/v1252/. A selection of individual CCDs are attached to this ticket.

            A couple of notes on these CCDs:

            • The majority show a significant improvement, especially extreme cases such as the well-known 67/68/69 strip.
            • Only a single CCD (#3) shows a worsening - this CCD is dominated by a bright saturated star. Any attempts to catch this using any currently implemented safety checks will fail, as the amp-to-amp steps are relatively standard (~1/2 counts) and the number of available pixels along each amp edge should be sufficient.
            • Some amps (e.g. CCD 35) show evidence for requiring more than a simple flat piston correction. This isn't the case the majority of the time, but sometimes there is a slight preference for a somewhat stronger correction in the lower half of the amp than in the upper. However, I don't think we have enough information to constrain this here, so I think it still safer to stick with a flat piston-like correction.
            lskelvin Lee Kelvin added a comment - All CCDs plotted individually for our canonical visit 1252 can be found at this link: https://lsst.ncsa.illinois.edu/~lskelvin/project/patConCorr/delta-v1/plots/v1252/ . A selection of individual CCDs are attached to this ticket. A couple of notes on these CCDs: The majority show a significant improvement, especially extreme cases such as the well-known 67/68/69 strip. Only a single CCD (#3) shows a worsening - this CCD is dominated by a bright saturated star. Any attempts to catch this using any currently implemented safety checks will fail, as the amp-to-amp steps are relatively standard (~1/2 counts) and the number of available pixels along each amp edge should be sufficient. Some amps (e.g. CCD 35) show evidence for requiring more than a simple flat piston correction. This isn't the case the majority of the time, but sometimes there is a slight preference for a somewhat stronger correction in the lower half of the amp than in the upper. However, I don't think we have enough information to constrain this here, so I think it still safer to stick with a flat piston-like correction.
            lskelvin Lee Kelvin added a comment - - edited

            Thank you for your earlier comments on this ticket Eli, and apologies it has taken me some time to reply. I wanted to take the time to process a significant number of datasets using gen3 middleware, and using BPS batch processing. Thankfully, all seems to operate as expected with regards gen3, so this looks good to go.

            With regards naming conventions, following the earlier discussion on #dm-naming-things, I have decided on adopting two terms to describe the operations associated with this ticket:

            1. amp offset: the measured value (in ADU) of the offset from one amp to another;
            2. amp pedestal: the constant value (in ADU) added/subtracted from an entire amp, following amp offset measurement, in order to correct for the measured amp offsets.

            With regards testing, I have defined two distinct datasets: short and rc2i. The short visits consist of 30s exposures from all five passbands (i.e., those exposure times which have been shown to be most visibly impacted by amp offsets). The rc2i visits consist of longer exposure RC2 i-band visits, for comparison. Further details:

            short
            This dataset consists of 5 short (30s) exposures from each passband:

            • g: [1886, 6292, 6296, 6300, 6304]
            • r: [1200, 1222, 2956, 2968, 2972]
            • i: [872, 1226, 1252, 1256, 1260]
            • z: [1164, 8794, 8798, 8802, 9616]
            • y: [340, 374, 378, 382, 386]

            Two output runs exist in /repo/main, corresponding to the vanilla outputs (doAmpOffset=False) and amp offset corrected outputs (doAmpOffset=True):

            Full focal plane plots showing the vanilla, amp-offset corrected and difference calexps can be found here.

            rc2i
            All 26 visits from the GAMA_I tract 9615 are selected:
            [1258, 1262, 1270, 1274, 1278, 1280, 1282, 1286, 1288, 1290, 1294, 1300, 1302, 1306, 1308, 1310, 1314, 1316, 1324, 1326, 1330, 24494, 24504, 24522, 24536, 24538]

            Two CHAINED collections are available in /repo/main:

            Within, the outputs from step1 (single frame), step2 (coaddition) and step3 (final coadd processing) are available.

            Full focal plane plots showing the vanilla, amp-offset corrected and difference calexps can be found here.

            As discussed on GitHub, this processing has been set to False by default, and will require a future ticket to update the associated sky frames used for processing. An updated and rebased Jenkins output can be found here. Hopefully this should now be good to merge, but please do let me know if there's anything else I can look at here.

            lskelvin Lee Kelvin added a comment - - edited Thank you for your earlier comments on this ticket Eli, and apologies it has taken me some time to reply. I wanted to take the time to process a significant number of datasets using gen3 middleware, and using BPS batch processing. Thankfully, all seems to operate as expected with regards gen3, so this looks good to go. With regards naming conventions, following the earlier discussion on #dm-naming-things, I have decided on adopting two terms to describe the operations associated with this ticket: amp offset: the measured value (in ADU) of the offset from one amp to another; amp pedestal: the constant value (in ADU) added/subtracted from an entire amp, following amp offset measurement, in order to correct for the measured amp offsets. With regards testing, I have defined two distinct datasets: short and rc2i . The short visits consist of 30s exposures from all five passbands (i.e., those exposure times which have been shown to be most visibly impacted by amp offsets). The rc2i visits consist of longer exposure RC2 i-band visits, for comparison. Further details: short This dataset consists of 5 short (30s) exposures from each passband: g: [1886, 6292, 6296, 6300, 6304] r: [1200, 1222, 2956, 2968, 2972] i: [872, 1226, 1252, 1256, 1260] z: [1164, 8794, 8798, 8802, 9616] y: [340, 374, 378, 382, 386] Two output runs exist in /repo/main , corresponding to the vanilla outputs ( doAmpOffset=False ) and amp offset corrected outputs ( doAmpOffset=True ): u/lskelvin/ DM-24389 /short/vanilla u/lskelvin/ DM-24389 /short/ampoffset Full focal plane plots showing the vanilla, amp-offset corrected and difference calexps can be found here . rc2i All 26 visits from the GAMA_I tract 9615 are selected: [1258, 1262, 1270, 1274, 1278, 1280, 1282, 1286, 1288, 1290, 1294, 1300, 1302, 1306, 1308, 1310, 1314, 1316, 1324, 1326, 1330, 24494, 24504, 24522, 24536, 24538] Two CHAINED collections are available in /repo/main : u/lskelvin/ DM-24389 /rc2i/vanilla u/lskelvin/ DM-24389 /rc2i/ampoffset Within, the outputs from step1 (single frame), step2 (coaddition) and step3 (final coadd processing) are available. Full focal plane plots showing the vanilla, amp-offset corrected and difference calexps can be found here . As discussed on GitHub, this processing has been set to False by default, and will require a future ticket to update the associated sky frames used for processing. An updated and rebased Jenkins output can be found here . Hopefully this should now be good to merge, but please do let me know if there's anything else I can look at here.
            lskelvin Lee Kelvin added a comment - - edited

            Update: the above Jenkins run failed, likely due to a recent bug introduced in DM-30448 in faro. See Slack for further discussion, and see this PR for a proposed fix.

            lskelvin Lee Kelvin added a comment - - edited Update: the above Jenkins run failed, likely due to a recent bug introduced in DM-30448 in faro . See Slack for further discussion , and see this PR for a proposed fix .
            lskelvin Lee Kelvin added a comment - - edited

            Following the merge of the faro hot-fix linked above, a new Jenkins run for this ticket can be found here.

            lskelvin Lee Kelvin added a comment - - edited Following the merge of the faro hot-fix linked above, a new Jenkins run for this ticket can be found here .
            lskelvin Lee Kelvin added a comment - - edited

            A series of analysis plots comparing deepCoadd outputs from a selection of patches within tract 9615 from the rc2i runs:




            As shown, the impact of amp-offset corrections are apparent, however, at the coadd level for these long (~250s) exposures, these corrections amount to a <<1% effect.

            lskelvin Lee Kelvin added a comment - - edited A series of analysis plots comparing deepCoadd outputs from a selection of patches within tract 9615 from the rc2i runs: As shown, the impact of amp-offset corrections are apparent, however, at the coadd level for these long (~250s) exposures, these corrections amount to a <<1% effect.
            erykoff Eli Rykoff added a comment -

            Okay, this is all looking good. However, I realized looking through the code that there don't appear to be any unit tests to go along with the code. That is, you've done the due diligence with offline tests that everything works, and if we do run a full RC2 (or turn it on for regular RC2) then that will test the code infrequently. But if we could create at least one isr test that turns doAmpOffset on and makes sure that it doesn't explode that would be very helpful to ensure it doesn't break in the future.

            erykoff Eli Rykoff added a comment - Okay, this is all looking good. However, I realized looking through the code that there don't appear to be any unit tests to go along with the code. That is, you've done the due diligence with offline tests that everything works, and if we do run a full RC2 (or turn it on for regular RC2) then that will test the code infrequently. But if we could create at least one isr test that turns doAmpOffset on and makes sure that it doesn't explode that would be very helpful to ensure it doesn't break in the future.
            lskelvin Lee Kelvin added a comment -

            Attached is a comparison plot showing the two iCModel magnitude histograms as output in the objectTable_tract dataset type. As shown, there is little difference in these two outputs, with minor fluctuations becoming apparent at the extreme bright/faint ends (note the logarithmic y-axis).

            I'll construct additional analysis plots from this table to see if anything else looks interesting. Thanks for your feedback above Eli. I'll start construction of a basic unit-test as you suggest and get back to you shortly.

            lskelvin Lee Kelvin added a comment - Attached is a comparison plot showing the two iCModel magnitude histograms as output in the objectTable_tract dataset type. As shown, there is little difference in these two outputs, with minor fluctuations becoming apparent at the extreme bright/faint ends (note the logarithmic y-axis). I'll construct additional analysis plots from this table to see if anything else looks interesting. Thanks for your feedback above Eli. I'll start construction of a basic unit-test as you suggest and get back to you shortly.
            lskelvin Lee Kelvin added a comment -

            Thanks again for your earlier suggestion Eli. I've now added a couple of unit tests which run the amp-offset code in two different modes: once with background estimation switched off, and a second time with background estimation switched on. I hope these simple tests were what you had in mind?

            lskelvin Lee Kelvin added a comment - Thanks again for your earlier suggestion Eli. I've now added a couple of unit tests which run the amp-offset code in two different modes: once with background estimation switched off, and a second time with background estimation switched on. I hope these simple tests were what you had in mind?
            erykoff Eli Rykoff added a comment -

            Some minor comments on the PR. Otherwise looks good! Congrats!

            erykoff Eli Rykoff added a comment - Some minor comments on the PR. Otherwise looks good! Congrats!
            lskelvin Lee Kelvin added a comment -

            Thanks again Eli. Final Jenkins run passed without issue - branches merged and deleted - cheers!

            lskelvin Lee Kelvin added a comment - Thanks again Eli. Final Jenkins run passed without issue - branches merged and deleted - cheers!

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Eli Rykoff
              Christopher Waters, Eli Rykoff, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.