Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31394

Overscan subtraction can create negative variance

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ip_isr
    • None
    • 1
    • Data Release Production
    • No

    Description

      Edge effects in some exposures (example {exposure=36118, ccd=87}) can create a negative variance.  The process of this is:

      • Image data is unusually low (example has 0.0 values for the lower ~84 rows).
      • Overscan is fit with a spline, and cannot fit these points (which may also be excluded from the spline fit for being too discrepant).
      • Resulting overscan value in that region is higher than raw image value, resulting in a negative value prior to the variance calculation.
      • Variance calculation is just I / G + RN^2 which propagates the negative to the variance.

      These low pixels need to be masked.  One option is to propagate the overscan excluded rows back so they can be masked.  This will eliminate the majority of points, but does not guarantee that non-zero points ill fit by the overscan spline will also be removed (the points in the ramp-down region of the figure that will still result in negative variance).

      This may need a config option to enable, as we should not mask negative image values while creating biases.

      Attachments

        Issue Links

          Activity

            No builds found.
            czw Christopher Waters created issue -
            czw Christopher Waters made changes -
            Field Original Value New Value
            Link This issue blocks DM-31359 [ DM-31359 ]
            czw Christopher Waters added a comment - - edited

            I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels.

            Assuming no hidden issues in `ci_hsc`, this should succeed:

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline

             

            https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline

            czw Christopher Waters added a comment - - edited I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD ).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels. Assuming no hidden issues in `ci_hsc`, this should succeed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline   https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline
            czw Christopher Waters made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            czw Christopher Waters made changes -
            Reviewers Clare Saunders [ csaunder ]
            Status In Progress [ 3 ] In Review [ 10004 ]

            Looks good to me!

            csaunder Clare Saunders added a comment - Looks good to me!
            csaunder Clare Saunders made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            czw Christopher Waters made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            czw Christopher Waters made changes -
            Story Points 1
            csaunder Clare Saunders made changes -
            Link This issue relates to DM-31851 [ DM-31851 ]

            People

              czw Christopher Waters
              czw Christopher Waters
              Clare Saunders
              Christopher Waters, Clare Saunders, Eli Rykoff, Paul Price, Robert Lupton
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.