Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31394

Overscan subtraction can create negative variance

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Edge effects in some exposures (example {exposure=36118, ccd=87}) can create a negative variance.  The process of this is:

      • Image data is unusually low (example has 0.0 values for the lower ~84 rows).
      • Overscan is fit with a spline, and cannot fit these points (which may also be excluded from the spline fit for being too discrepant).
      • Resulting overscan value in that region is higher than raw image value, resulting in a negative value prior to the variance calculation.
      • Variance calculation is just I / G + RN^2 which propagates the negative to the variance.

      These low pixels need to be masked.  One option is to propagate the overscan excluded rows back so they can be masked.  This will eliminate the majority of points, but does not guarantee that non-zero points ill fit by the overscan spline will also be removed (the points in the ramp-down region of the figure that will still result in negative variance).

      This may need a config option to enable, as we should not mask negative image values while creating biases.

        Attachments

        1. amp0_overscan.png
          27 kB
          Christopher Waters

          Issue Links

            Activity

            Hide
            czw Christopher Waters added a comment - - edited

            I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels.

            Assuming no hidden issues in `ci_hsc`, this should succeed:

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline

             

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline

            Show
            czw Christopher Waters added a comment - - edited I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD ).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels. Assuming no hidden issues in `ci_hsc`, this should succeed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline   https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline
            Hide
            csaunder Clare Saunders added a comment -

            Looks good to me!

            Show
            csaunder Clare Saunders added a comment - Looks good to me!

              People

              Assignee:
              czw Christopher Waters
              Reporter:
              czw Christopher Waters
              Reviewers:
              Clare Saunders
              Watchers:
              Christopher Waters, Clare Saunders, Eli Rykoff, Paul Price, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.