Details

Type: Story

Status: Done

Resolution: Done

Fix Version/s: None

Component/s: ip_isr

Labels:None

Team:DM Science
Description
The code to estimate overscan levels do not take enough care to handle low signal levels. Once the noise level is small enough a simple median does not return an estimate of population median (think about the case that the pixels have equal probability of being either 1 or 2; the population median is 1.5, but the median will be 1 or 2). With enough noise this isn't much of a problem as the noise provides dithering; even then there can be biases based on the noise's pdf.
For bias levels this is an issue; for example the biassubtracted overscan bias frames from auxTel (taken on 20180721) have amplifier medians that differ by significant fractions of a DN – and that's enough to mess up faint light measurements as some of the effect is in overscan correction used to build the bias frames.
Using a mean instead of a median "solves" this problem, but at the expense of being sensitive to CR hits (or other outliers) in the overscan.
One simple solution to the median problem is to find the mean of sets of pixels, then estimate the median of the result (this essentially raises the noise level); another — which we used in SDSS — is to form a histogram and estimate its median. This should be implemented in afwMath.Statistics as a robust median; adding a clipping option to afwMath.Statistics would also probably be fine.
The issue with a nonrobust mean can be resolved by exposing the StatisticsCtrl argument and using a clipped mean.
Jira only reports one pull request (https://github.com/lsst/ip_isr/pull/63), but there's also https://github.com/lsst/afw/pull/388.
I think that the only contentious thing is changing assertEqual to assertAlmostEqual in the tests, but I've also added Gaussian noise to the data and we're running into floatingpoint epsilon in making the tests exact. Still, you might want to look at the delta values