# Overscan subtraction can create negative variance

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1
• Team:
Data Release Production
• Urgent?:
No

#### Description

Edge effects in some exposures (example {exposure=36118, ccd=87}) can create a negative variance.  The process of this is:

• Image data is unusually low (example has 0.0 values for the lower ~84 rows).
• Overscan is fit with a spline, and cannot fit these points (which may also be excluded from the spline fit for being too discrepant).
• Resulting overscan value in that region is higher than raw image value, resulting in a negative value prior to the variance calculation.
• Variance calculation is just I / G + RN^2 which propagates the negative to the variance.

These low pixels need to be masked.  One option is to propagate the overscan excluded rows back so they can be masked.  This will eliminate the majority of points, but does not guarantee that non-zero points ill fit by the overscan spline will also be removed (the points in the ramp-down region of the figure that will still result in negative variance).

This may need a config option to enable, as we should not mask negative image values while creating biases.

#### Attachments

1. amp0_overscan.png
27 kB

#### Activity

Hide
Christopher Waters added a comment - - edited

I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels.

Assuming no hidden issues in ci_hsc, this should succeed:

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline

https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline

Show
Christopher Waters added a comment - - edited I've added code to allow pixels with negative variance to masked with a specified plane (defaults to BAD ).  This is enabled by default, as in most circumstances we wouldn't expect any negative variance pixels. Assuming no hidden issues in ci_hsc, this should succeed: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34764/pipeline   https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34765/pipeline
Hide
Clare Saunders added a comment -

Looks good to me!

Show
Clare Saunders added a comment - Looks good to me!

#### People

Assignee:
Christopher Waters
Reporter:
Christopher Waters
Reviewers:
Clare Saunders
Watchers:
Christopher Waters, Clare Saunders, Eli Rykoff, Paul Price, Robert Lupton