Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23794

Provide Simple Per-CCD Fix to Amp-to-Amp Offset Issue

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 12
    • Data Release Production
    • No

    Description

      This ticket explores two simple empirical approaches towards fixing the known amp-to-amp offset issue in HSC CCDs. Multiple attempts to provide a meaningful hardware-related solution to the amp-offset issue have been explored, with work ongoing in this regard. In the meantime, we provide these simple per-CCD fixes as a proof of concept. Should more desirable corrections not be forthcoming, this work serves as a potential future fix which can be applied to future datasets.

      Techniques explored are per-amp background subtraction and amp-edge pattern continuity corrections. Both methods have their merits, however, per-amp background subtractions remain vulnerable to stray light (ghosts, ghouls, etc) or strong flux gradients. Conversely, amp-edge pattern continuity corrections perform well across multiple scenarios, with most remaining weaknesses solveable in the short-term. Therefore, if subsequently required, we recommend amp-edge pattern continuity corrections for future use.

      Attachments below contain both example notebooks, a brief PDF presentation, and all postage stamps used within the PDF. Full processing outputs can be found at: https://github.com/leeskelvin/amp2amp

      Attachments

        1. 20200331-Princeton.pdf
          30.46 MB
        2. ampfixpatcon.ipynb
          11 kB
        3. ampfixperamp.ipynb
          9 kB
        4. patcon_v0000872c053.png
          patcon_v0000872c053.png
          2.42 MB
        5. patcon_v0001200c059.png
          patcon_v0001200c059.png
          2.27 MB
        6. patcon_v0001252c000.png
          patcon_v0001252c000.png
          972 kB
        7. patcon_v0001252c004.png
          patcon_v0001252c004.png
          2.20 MB
        8. patcon_v0001252c007.png
          patcon_v0001252c007.png
          2.41 MB
        9. patcon_v0001252c008.png
          patcon_v0001252c008.png
          2.40 MB
        10. patcon_v0001252c011.png
          patcon_v0001252c011.png
          2.28 MB
        11. patcon_v0001252c027.png
          patcon_v0001252c027.png
          2.30 MB
        12. patcon_v0001252c029.png
          patcon_v0001252c029.png
          2.04 MB
        13. patcon_v0001252c031.png
          patcon_v0001252c031.png
          2.32 MB
        14. patcon_v0001252c032.png
          patcon_v0001252c032.png
          2.45 MB
        15. patcon_v0001252c035.png
          patcon_v0001252c035.png
          2.17 MB
        16. patcon_v0001252c036.png
          patcon_v0001252c036.png
          2.37 MB
        17. patcon_v0001252c038.png
          patcon_v0001252c038.png
          2.36 MB
        18. patcon_v0001252c043.png
          patcon_v0001252c043.png
          1.16 MB
        19. patcon_v0001252c046.png
          patcon_v0001252c046.png
          2.30 MB
        20. patcon_v0001252c060.png
          patcon_v0001252c060.png
          2.30 MB
        21. patcon_v0001252c061.png
          patcon_v0001252c061.png
          2.45 MB
        22. patcon_v0001252c065.png
          patcon_v0001252c065.png
          2.22 MB
        23. patcon_v0001252c068.png
          patcon_v0001252c068.png
          2.28 MB
        24. patcon_v0001254c059.png
          patcon_v0001254c059.png
          2.36 MB
        25. patcon_v0002928c058.png
          patcon_v0002928c058.png
          1.87 MB
        26. patcon_v0002928c076.png
          patcon_v0002928c076.png
          2.43 MB
        27. peramp_v0000872c038.png
          peramp_v0000872c038.png
          2.51 MB
        28. peramp_v0000872c053.png
          peramp_v0000872c053.png
          2.41 MB
        29. peramp_v0000872c059.png
          peramp_v0000872c059.png
          2.33 MB
        30. peramp_v0001200c038.png
          peramp_v0001200c038.png
          2.37 MB
        31. peramp_v0001200c059.png
          peramp_v0001200c059.png
          2.26 MB
        32. peramp_v0001252c000.png
          peramp_v0001252c000.png
          1.73 MB
        33. peramp_v0001252c001.png
          peramp_v0001252c001.png
          2.30 MB
        34. peramp_v0001252c005.png
          peramp_v0001252c005.png
          2.32 MB
        35. peramp_v0001252c008.png
          peramp_v0001252c008.png
          2.44 MB
        36. peramp_v0001252c011.png
          peramp_v0001252c011.png
          2.28 MB
        37. peramp_v0001252c021.png
          peramp_v0001252c021.png
          2.28 MB
        38. peramp_v0001252c031.png
          peramp_v0001252c031.png
          2.31 MB
        39. peramp_v0001252c035.png
          peramp_v0001252c035.png
          2.25 MB
        40. peramp_v0001252c038.png
          peramp_v0001252c038.png
          2.36 MB
        41. peramp_v0001252c039.png
          peramp_v0001252c039.png
          2.26 MB
        42. peramp_v0001252c043.png
          peramp_v0001252c043.png
          1.91 MB
        43. peramp_v0001252c059.png
          peramp_v0001252c059.png
          2.39 MB
        44. peramp_v0001252c067.png
          peramp_v0001252c067.png
          2.31 MB
        45. peramp_v0001254c038.png
          peramp_v0001254c038.png
          2.50 MB
        46. peramp_v0001254c059.png
          peramp_v0001254c059.png
          2.12 MB
        47. peramp_v0001254c059.png
          peramp_v0001254c059.png
          2.12 MB
        48. peramp_v0002928c038.png
          peramp_v0002928c038.png
          2.46 MB
        49. peramp_v0002928c058.png
          peramp_v0002928c058.png
          2.37 MB
        50. peramp_v0002928c059.png
          peramp_v0002928c059.png
          2.38 MB
        51. peramp_v0002928c076.png
          peramp_v0002928c076.png
          2.35 MB

        Issue Links

          Activity

            erykoff Eli Rykoff added a comment -

            Here are my comments on the slides and the amp-edge notebook (since that definitely looks like the more promising approach).

            Slide 4: there is also the possibility that the non-linearities were incorrectly estimated originally. Also on this slide and others there are boxes and plots blocking some of the text. Maybe you need to export to pdf making sure all the build steps are included?

            Slide 6: With the 4-step and 2-step offsets, I was wondering if these were common amongst ccds (that is, is ccd 59 usually a two-step ccd?). Also, the way this slide is written it looks like you might be fitting a 4 step or 2 step model, but I don't think that's the case, this is just how they come out.

            Slide 12 (and relevant notebook): Have you looked at doing the median of the offset (per row) rather than the offset of the median to help with gradients?

            Slide 17: This is very promising!

            Slide 18: Missing amps are a problem. There are bad amps, and making the mask less restrictive won't help. I guess one could have a different Delta matrix when you don't have all 4 amps; you still want continuity from edge to edge (but the danger is then you mess up a horizontal gradient...)

            Slide 20: I'm still not sure why the bright star at the 3/4 amp edge is creating an issue at the 2/3 edge. Either there's some other mechanism at work (something something bleed into the overscan ... ), or I really don't understand the model, or there's a bug somewhere.

            Have you had a chance to plot how these amp offsets look across the focal plane? Are they common from visit to visit? Etc. This might require the upcoming test run on RC2.

            erykoff Eli Rykoff added a comment - Here are my comments on the slides and the amp-edge notebook (since that definitely looks like the more promising approach). Slide 4: there is also the possibility that the non-linearities were incorrectly estimated originally. Also on this slide and others there are boxes and plots blocking some of the text. Maybe you need to export to pdf making sure all the build steps are included? Slide 6: With the 4-step and 2-step offsets, I was wondering if these were common amongst ccds (that is, is ccd 59 usually a two-step ccd?). Also, the way this slide is written it looks like you might be fitting a 4 step or 2 step model, but I don't think that's the case, this is just how they come out. Slide 12 (and relevant notebook): Have you looked at doing the median of the offset (per row) rather than the offset of the median to help with gradients? Slide 17: This is very promising! Slide 18: Missing amps are a problem. There are bad amps, and making the mask less restrictive won't help. I guess one could have a different Delta matrix when you don't have all 4 amps; you still want continuity from edge to edge (but the danger is then you mess up a horizontal gradient...) Slide 20: I'm still not sure why the bright star at the 3/4 amp edge is creating an issue at the 2/3 edge. Either there's some other mechanism at work (something something bleed into the overscan ... ), or I really don't understand the model, or there's a bug somewhere. Have you had a chance to plot how these amp offsets look across the focal plane? Are they common from visit to visit? Etc. This might require the upcoming test run on RC2.
            lskelvin Lee Kelvin added a comment -

            Thanks for reviewing this Eli. I've updated the slides in response to your comments, adding extra relevant info and clearing up the formatting errors.

            Calculating the median of the offset per row rather than the offset of the medians is a good idea. At the very least, I'm convinced we should test both methods across RC2.

            With regards missing amps, I think a modified delta matrix as you suggest should work. A simple fix would be to set missing amp edges equal to their neighbouring edge value in the delta matrix, effectively setting the amp-jump to zero.

            After having spoken with you about the bright amp edge sources, I've convinced myself that this issue is actually related to the diverging 2-step ramps on the previous slide. I've updated the amp edge slide to include this conversation, and put it into the appendix section.

            Much of this work is setting the ground for a future run across RC2, which I think is the best place to look at entire focal plane plots. Potentially we'll see amp trends that will allow us to further constrain this approach.

            lskelvin Lee Kelvin added a comment - Thanks for reviewing this Eli. I've updated the slides in response to your comments, adding extra relevant info and clearing up the formatting errors. Calculating the median of the offset per row rather than the offset of the medians is a good idea. At the very least, I'm convinced we should test both methods across RC2. With regards missing amps, I think a modified delta matrix as you suggest should work. A simple fix would be to set missing amp edges equal to their neighbouring edge value in the delta matrix, effectively setting the amp-jump to zero. After having spoken with you about the bright amp edge sources, I've convinced myself that this issue is actually related to the diverging 2-step ramps on the previous slide. I've updated the amp edge slide to include this conversation, and put it into the appendix section. Much of this work is setting the ground for a future run across RC2, which I think is the best place to look at entire focal plane plots. Potentially we'll see amp trends that will allow us to further constrain this approach.

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Eli Rykoff
              Eli Rykoff, Lee Kelvin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.