Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24389

Implement amp-to-amp offset correction task

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 36
    • Data Release Production
    • No

    Description

      HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into ip_isr, and is redirected to the primary run task in obs_subaru. This task may be switched on or off using the doAmpOffset flag, which currently defaults to False.

      In summary, this process generates an estimate of the amp offset from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp offset value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp offsets. We refer to these additive/subtractive values as pedestals in the exposure metadata.

      The method for generating an amp offset value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary DETECTION footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in background.ignoredPixelMask are not used in determining amp offsets (defaulting to BAD, SAT, INTRP, CR, EDGE, DETECTED, DETECTED_NEGATIVE, SUSPECT and NO_DATA). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by ampEdgeInset (default 5 pixels). The width of the strip is given by ampEdgeWidth (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size ampEdgeWindow (default 512 pixels). Finally, an estimate of the amp A to amp B offset is given by calculating the 3-sigma clipped mean of this rolling average data.

      During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than ampEdgeMinFrac (default 0.5) then no amp offset estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting if addition to detected source footprints can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp offset values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp offset estimate larger than ampEdgeMaxOffset (default 5 counts) will also be ignored, and no amp offset estimate will be generated for that amp boundary.

      All added/subtracted pedestals are entered into the exposure metadata as PEDESTALX (where X is the number of the amp: PEDESTAL1, PEDESTAL2...). The net sum of all pedestal corrections for a single CCD is zero.

      Full outputs for a test run on 200 visits from DM-23243 (PDR2) wide can be found on lsst-devl at /project/lskelvin/patConCorr/delta-v1 or at this link. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits. Additional information and discussion on the impact these corrections have on coadd level data can also be found in the comments below.

      Attachments

        1. pipeline_step1.pdf
          30 kB
        2. pipeline_step2.pdf
          18 kB
        3. pipeline_step3.pdf
          20 kB
        4. rc2i_iCModelMag.png
          rc2i_iCModelMag.png
          26 kB
        5. rc2i_t9615_p00.png
          rc2i_t9615_p00.png
          1.57 MB
        6. rc2i_t9615_p05.png
          rc2i_t9615_p05.png
          1.31 MB
        7. rc2i_t9615_p25.png
          rc2i_t9615_p25.png
          1.56 MB
        8. rc2i_t9615_p55.png
          rc2i_t9615_p55.png
          1.31 MB
        9. v1252_calexp_ampoff.png
          v1252_calexp_ampoff.png
          525 kB
        10. v1252_calexp_diff.png
          v1252_calexp_diff.png
          190 kB
        11. v1252_calexp_vanilla.png
          v1252_calexp_vanilla.png
          522 kB
        12. v1252c003.png
          v1252c003.png
          299 kB
        13. v1252c011.png
          v1252c011.png
          365 kB
        14. v1252c019.png
          v1252c019.png
          362 kB
        15. v1252c035.png
          v1252c035.png
          374 kB
        16. v1252c036.png
          v1252c036.png
          369 kB
        17. v1252c040.png
          v1252c040.png
          366 kB
        18. v1252c043.png
          v1252c043.png
          345 kB
        19. v1252c067.png
          v1252c067.png
          365 kB
        20. v1252c068.png
          v1252c068.png
          365 kB
        21. v1252c069.png
          v1252c069.png
          340 kB
        22. v7206c14graphs.png
          v7206c14graphs.png
          481 kB
        23. v7206c14images.png
          v7206c14images.png
          184 kB

        Issue Links

          Activity

            lskelvin Lee Kelvin created issue -
            lskelvin Lee Kelvin made changes -
            Field Original Value New Value
            Epic Link DM-23739 [ 431396 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-23794 [ DM-23794 ]
            lskelvin Lee Kelvin made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-23782 [ DM-23782 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-23739 [ 431396 ] DM-25270 [ 435629 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-25270 [ 435629 ] DM-26788 [ 439738 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-23243 [ DM-23243 ]
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp-Edge Continuity Corrections Implement Amp-Edge Pattern Continuity Corrections
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp-Edge Pattern Continuity Corrections Implement Amp Offset Pattern Continuity Corrections
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp Offset Pattern Continuity Corrections Implement Amp-Offset Pattern Continuity Corrections
            lskelvin Lee Kelvin made changes -
            Story Points 12 18
            lskelvin Lee Kelvin made changes -
            Story Points 18 24
            lskelvin Lee Kelvin made changes -
            Epic Link DM-26788 [ 439738 ] DM-27973 [ 442758 ]
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp-Offset Pattern Continuity Corrections Implement Amp-to-Amp Offset Corrections
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 ADU amp-to-amp offsets. Following on from DM-23794, here we implement amp-edge pattern continuity corrections which aim to correct for these jumps. HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp-to-Amp Offset Corrections Implement Amp-to-Amp Offset Correction Task
            lskelvin Lee Kelvin made changes -
            Summary Implement Amp-to-Amp Offset Correction Task Implement amp-to-amp offset correction task
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a
            HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). The
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). The
            HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). In order to avoid any image artifact peculiarities which are may occur along an amp edge from time to time, these strips are inset from the amp edge by
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). In order to avoid any image artifact peculiarities which are may occur along an amp edge from time to time, these strips are inset from the amp edge by
            HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). In order to avoid any image artifact peculiarities which are may occur along an amp edge from time to time, these strips are inset from the amp edge by {{ampEdgeInset}} (with a current default value of 5 pixels). The width of the strip is {{
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle 1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps and three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived determining the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a piston which is being moved up or down to correct for this amp-offset issue.

            The method for generating an amp step value is thus: first, a thin strip of pixels is defined along the trailing edge of the first amp ("A") and another along the leading edge of the second amp ("B"). In order to avoid any image artifact peculiarities which are may occur along an amp edge from time to time, these strips are inset from the amp edge by {{ampEdgeInset}} (with a current default value of 5 pixels). The width of the strip is {{
            HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}}. Comments below provide links to movies showing the before and after correction impact of amp offset correction on these visits.
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}}. Comments below provide links to movies showing the before and after correction impact of amp offset correction on these visits.
            HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}}. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits.
            lskelvin Lee Kelvin made changes -
            Attachment v1252_calexp_vanilla.png [ 47907 ]
            lskelvin Lee Kelvin made changes -
            Attachment v1252_calexp_ampoff.png [ 47908 ]
            lskelvin Lee Kelvin made changes -
            Attachment v1252_calexp_diff.png [ 47909 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14images.png [ 47910 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47911 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47911 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47912 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47912 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47913 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47913 ]
            lskelvin Lee Kelvin made changes -
            Attachment v7206c14graphs.png [ 47914 ]
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}}. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits.
            HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}} or at [this link|https://lsst.ncsa.illinois.edu/~lskelvin/project/patConCorr/delta-v1/plots/]. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits.
            lskelvin Lee Kelvin made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            Reviewers Eli Rykoff [ erykoff ]
            lskelvin Lee Kelvin made changes -
            Attachment v1252c003.png [ 48048 ]
            Attachment v1252c011.png [ 48049 ]
            Attachment v1252c019.png [ 48050 ]
            Attachment v1252c035.png [ 48051 ]
            Attachment v1252c036.png [ 48052 ]
            Attachment v1252c040.png [ 48053 ]
            Attachment v1252c043.png [ 48054 ]
            Attachment v1252c067.png [ 48055 ]
            Attachment v1252c068.png [ 48056 ]
            Attachment v1252c069.png [ 48057 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-27973 [ 442758 ] DM-29151 [ 458508 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-29151 [ 458508 ] DM-30470 [ 509195 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p00.png [ 50047 ]
            Attachment rc2i_t9615_p05.png [ 50048 ]
            Attachment rc2i_t9615_p25.png [ 50049 ]
            Attachment rc2i_t9615_p55.png [ 50050 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p00.png [ 50047 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p05.png [ 50048 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p25.png [ 50049 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p55.png [ 50050 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_t9615_p00.png [ 50051 ]
            Attachment rc2i_t9615_p05.png [ 50052 ]
            Attachment rc2i_t9615_p25.png [ 50053 ]
            Attachment rc2i_t9615_p55.png [ 50054 ]
            lskelvin Lee Kelvin made changes -
            Description HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{True}}.

            In summary, this process generates an estimate of the pedestal level step from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp step value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp steps. We refer to these additive/subtractive values as {{pistons}} in the exposure metadata, imagining that each amp lies on a flat piston which is being moved up or down to correct for any amp-offset signatures.

            The method for generating an amp step value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B step is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp step estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp step values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp step estimate larger than {{ampEdgeMaxStep}} (default 5 counts) will also be ignored, and no amp step estimate will be generated for that amp boundary.

            All added/subtracted pistons are entered into the exposure metadata as {{PISTONX}} (where {{X}} is the number of the amp: {{PISTON1}}, {{PISTON2}}...). The net sum of all piston corrections for a single CCD is zero. Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}} or at [this link|https://lsst.ncsa.illinois.edu/~lskelvin/project/patConCorr/delta-v1/plots/]. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits.
            HSC imaging is known to exhibit subtle ~1-2 count amp-to-amp offsets. Following on from DM-23794, this ticket implements an empirical amp-to-amp offset correction technique within ISR processing, removing these offset artifacts. An empty placeholder task is placed into {{ip_isr}}, and is redirected to the primary run task in {{obs_subaru}}. This task may be switched on or off using the {{doAmpOffset}} flag, which currently defaults to {{False}}.

            In summary, this process generates an estimate of the amp offset from amp to amp within an HSC CCD, looping over each amp boundary (HSC CCDs having four amps arranged linearly, and therefore three amp boundaries). Once an estimate of each amp offset value is obtained, a solution is derived to determine the singular value which must be added or subtracted from each amp in order to negate these amp offsets. We refer to these additive/subtractive values as {{pedestals}} in the exposure metadata.

            The method for generating an amp offset value is thus: first, we perform a temporary coarse background subtraction across the entire CCD. As this amp-offset correction occurs in ISR before primary background subtraction has taken place, a general background subtraction is required in order to remove any strong image gradients in the frame. Second, we perform a temporary source detection. This adds temporary {{DETECTION}} footprints to the mask plane, allowing those pixels to be ignored in all subsequent analyses if so directed. All mask planes identified in {{background.ignoredPixelMask}} are not used in determining amp offsets (defaulting to {{BAD}}, {{SAT}}, {{INTRP}}, {{CR}}, {{EDGE}}, {{DETECTED}}, {{DETECTED_NEGATIVE}}, {{SUSPECT}} and {{NO_DATA}}). For each amp pair we define a thin strip of pixels along the trailing edge of the first amp (amp "A"), and another along the leading edge of the second amp (amp "B"). In order to avoid any image artifact peculiarities which may occur along an amp edge, these two strips are inset from the amp edge by {{ampEdgeInset}} (default 5 pixels). The width of the strip is given by {{ampEdgeWidth}} (default 64 pixels). The median value is determined for each row in strip A and strip B, resulting in two one-dimensional arrays of length equal to the height of the CCD. These edge arrays are subtracted from one another, giving an estimate of the amp edge difference from A to B as a function of position along the length of the amp. In order to mitigate any remaining flux contamination from singular bright sources (e.g., the wings of bright stars which spill out beyond the source detection footprint), we calculate the rolling average of this amp edge difference array using a window of size {{ampEdgeWindow}} (default 512 pixels). Finally, an estimate of the amp A to amp B offset is given by calculating the 3-sigma clipped mean of this rolling average data.

            During testing, the procedures outlined above tend to only ever improve the fidelity of the output data, rarely (if ever) causing 'harm'. However, as additional safety checks, we have also added two failure checks to catch extreme cases. First, if the fraction of available and unmasked datapoints in the amp A to amp B difference array is lower than {{ampEdgeMinFrac}} (default 0.5) then no amp offset estimate is generated for that amp boundary. This catches those corner cases such as HSC CCD 101, where vignetting if addition to detected source footprints can significantly remove large portions of the field of view, resulting in only a small number of remaining pixels available for amp offset estimation. It also catches those cases where, for whatever reason, a large number of pixels are masked and therefore unsuitable for amp offset analysis (e.g., a strip of dead pixels caused by hardware failure along an amp edge, as is the case with HSC CCD 33). Second, testing has shown that typical amp offsets fall in the range ~1-2 counts, with larger calculated amp offset values increasingly likely to be caused by bright object contamination rather than a true amp offset signature. As such, any amp offset estimate larger than {{ampEdgeMaxOffset}} (default 5 counts) will also be ignored, and no amp offset estimate will be generated for that amp boundary.

            All added/subtracted pedestals are entered into the exposure metadata as {{PEDESTALX}} (where {{X}} is the number of the amp: {{PEDESTAL1}}, {{PEDESTAL2}}...). The net sum of all pedestal corrections for a single CCD is zero.

            Full outputs for a test run on 200 visits from {{DM-23243}} (PDR2) wide can be found on {{lsst-devl}} at {{/project/lskelvin/patConCorr/delta-v1}} or at [this link|https://lsst.ncsa.illinois.edu/~lskelvin/project/patConCorr/delta-v1/plots/]. Comments below provide examples and links to movies showing the before (vanilla) and after (amp-offset corrected) impact of amp offset correction on these visits. Additional information and discussion on the impact these corrections have on coadd level data can also be found in the comments below.
            lskelvin Lee Kelvin made changes -
            Attachment pipeline_step1.pdf [ 50059 ]
            Attachment pipeline_step2.pdf [ 50060 ]
            Attachment pipeline_step3.pdf [ 50061 ]
            lskelvin Lee Kelvin made changes -
            Attachment rc2i_iCModelMag.png [ 50062 ]
            erykoff Eli Rykoff made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            lskelvin Lee Kelvin made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lskelvin Lee Kelvin made changes -
            Story Points 24 36
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-32378 [ DM-32378 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-33629 [ DM-33629 ]
            erfan Erfan Nourbakhsh made changes -
            Link This issue relates to DM-38776 [ DM-38776 ]

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Eli Rykoff
              Christopher Waters, Eli Rykoff, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.