Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20112

ip_isr is not handling BAD pixels as expected

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None

      Description

      As a result of the interpolation change (DM-19382), BAD pixels are now only being reset to the image median after interpolation.  However, these pixels are interpolated, and because of that, will not be set to the image median.  This shouldn't significantly change image values, but the order is not correct.

        Attachments

          Issue Links

            Activity

            No builds found.
            czw Christopher Waters created issue -
            czw Christopher Waters made changes -
            Field Original Value New Value
            Link This issue relates to DM-19382 [ DM-19382 ]
            czw Christopher Waters made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            czw Christopher Waters made changes -
            Reviewers Colin Slater [ ctslater ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            ctslater Colin Slater added a comment -

            Looks fine to me, but I'm not sure I understand why we're resetting bad pixels and then interpolating them both, and I don't know how I would find out why we're doing that. Are we collecting this ISR knowledge anywhere? (cc: Yusra AlSayyad or Jim Bosch?)

            Show
            ctslater Colin Slater added a comment - Looks fine to me, but I'm not sure I understand why we're resetting bad pixels and then interpolating them both, and I don't know how I would find out why we're doing that. Are we collecting this ISR knowledge anywhere? (cc:  Yusra AlSayyad or Jim Bosch ?)
            ctslater Colin Slater made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            yusra Yusra AlSayyad added a comment -

            I don't know. Will this fix: https://jira.lsstcorp.org/browse/DM-19841 too?

            Show
            yusra Yusra AlSayyad added a comment - I don't know. Will this fix: https://jira.lsstcorp.org/browse/DM-19841 too?
            Hide
            czw Christopher Waters added a comment -

            I don't know yet.  My suspicion that this was the case prompted me to look into this (finding if nothing else the interpolation exposure bug).  Looking at the CORR for the image cited on that ticket (visit=130334/ccd=033), the data drops from the high value in amp1 approximately linearly over the transition to amp2.  My thought was that if `BAD` pixels aren't set to a value prior to interpolating, that it might skew the one side of the transition, resulting in what's seen.  However, most of the linear transition does not appear to have the `INTRP` bit set, so I'm not sure where the data values for that region come from.

            I'd like to run this flagged exposure through `runIsr.py` myself to test, but I'm having difficulty getting the HSC archive to give it to me.  I may spot check a few other `ccd=033` images that we have on disk to see if they show a similar feature.

            Show
            czw Christopher Waters added a comment - I don't know yet.  My suspicion that this was the case prompted me to look into this (finding if nothing else the interpolation exposure bug).  Looking at the CORR for the image cited on that ticket (visit=130334/ccd=033), the data drops from the high value in amp1 approximately linearly over the transition to amp2.  My thought was that if `BAD` pixels aren't set to a value prior to interpolating, that it might skew the one side of the transition, resulting in what's seen.  However, most of the linear transition does not appear to have the `INTRP` bit set, so I'm not sure where the data values for that region come from. I'd like to run this flagged exposure through `runIsr.py` myself to test, but I'm having difficulty getting the HSC archive to give it to me.  I may spot check a few other `ccd=033` images that we have on disk to see if they show a similar feature.
            Hide
            czw Christopher Waters added a comment -

            I made a comment last week on DM-20090, but the updated version of this ticket branch should resolve DM-19841 entirely.

            We were not marking bad amplifiers as `BAD`, allowing them to be used to set the interpolation.  Marking them as such now prevents the bright amplifier edge issue, and setting the value to the image median keeps the image scaling reasonable.

            As for the choice between interpolation and median setting, the logic to me is:

            • Large contiguous areas are reset, as there is no useful information there, and values on either side of the area have little relation to what happens in between.
            • Small areas (saturation spikes, bright columns) should be interpolated, as the image data on either side of the bad pixels should be similar to the expected value in the bad region.

            I'll admit this is something of an ex-post facto justification for what we do, but I will add something to this effect into the comments of isrTask, so there's an explanation for the future.

            Show
            czw Christopher Waters added a comment - I made a comment last week on DM-20090 , but the updated version of this ticket branch should resolve DM-19841 entirely. We were not marking bad amplifiers as `BAD`, allowing them to be used to set the interpolation.  Marking them as such now prevents the bright amplifier edge issue, and setting the value to the image median keeps the image scaling reasonable. As for the choice between interpolation and median setting, the logic to me is: Large contiguous areas are reset, as there is no useful information there, and values on either side of the area have little relation to what happens in between. Small areas (saturation spikes, bright columns) should be interpolated, as the image data on either side of the bad pixels should be similar to the expected value in the bad region. I'll admit this is something of an ex-post facto justification for what we do, but I will add something to this effect into the comments of isrTask, so there's an explanation for the future.
            czw Christopher Waters made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            swinbank John Swinbank added a comment -

            Hi folks — a reminder to please make sure ticketed work has story points and epics attached. Thanks!

            Show
            swinbank John Swinbank added a comment - Hi folks — a reminder to please make sure ticketed work has story points and epics attached. Thanks!
            swinbank John Swinbank made changes -
            Epic Link DM-20164 [ 323028 ]
            swinbank John Swinbank made changes -
            Story Points 1
            czw Christopher Waters made changes -
            Story Points 1 2
            yusra Yusra AlSayyad made changes -
            Story Points 2 6

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Reviewers:
              Colin Slater
              Watchers:
              Christopher Waters, Colin Slater, John Swinbank, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.