Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15203

The ISR overscan correction algorithms are too naive

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None
    • Team:
      DM Science

      Description

      The code to estimate overscan levels do not take enough care to handle low signal levels.  Once the noise level is small enough a simple median does not return an estimate of population median (think about the case that the pixels have equal probability of being either 1 or 2;  the population median is 1.5, but the median will be 1 or 2).  With enough noise this isn't much of a problem as the noise provides dithering;  even then there can be biases based on the noise's pdf.

      For bias levels this is an issue; for example the bias-subtracted over-scan bias frames from auxTel (taken on 2018-07-21) have amplifier medians that differ by significant fractions of a DN – and that's enough to mess up faint light measurements as some of the effect is in overscan correction used to build the bias frames.

      Using a mean instead of a median "solves" this problem, but at the expense of being sensitive to CR hits (or other outliers) in the overscan.

      One simple solution to the median problem is to find the mean of sets of pixels, then estimate the median of the result (this essentially raises the noise level);  another — which we used in SDSS — is to form a histogram and estimate its median.  This should be implemented in afwMath.Statistics as a robust median;  adding a clipping option to afwMath.Statistics would also probably be fine.

      The issue with a non-robust mean can be resolved by exposing the StatisticsCtrl argument and using a clipped mean.

        Attachments

          Activity

          Hide
          rhl Robert Lupton added a comment -

          Jira only  reports one pull request (https://github.com/lsst/ip_isr/pull/63), but there's also https://github.com/lsst/afw/pull/388.

          I think that the only contentious thing is changing assertEqual to assertAlmostEqual in the tests, but I've also added Gaussian noise to the data and we're running into floating-point epsilon in making the tests exact.  Still, you might want to look at the delta values

           

          Show
          rhl Robert Lupton added a comment - Jira only  reports one pull request ( https://github.com/lsst/ip_isr/pull/63 ), but there's also https://github.com/lsst/afw/pull/388. I think that the only contentious thing is changing assertEqual to assertAlmostEqual in the tests, but I've also added Gaussian noise to the data and we're running into floating-point epsilon in making the tests exact.  Still, you might want to look at the delta values  
          Hide
          price Paul Price added a comment -

          Some small comments on the GitHub PRs.

          Please make sure this passes Jenkins before merging. In particular, please be sure to run ci_hsc so the overscan changes get exercised.

          Show
          price Paul Price added a comment - Some small comments on the GitHub PRs. Please make sure this passes Jenkins before merging. In particular, please be sure to run ci_hsc so the overscan changes get exercised.
          Hide
          rhl Robert Lupton added a comment - - edited

          This fix to the median estimation changes the results in task_pipeline/tests_processCcd.py.

          Here is a histogram of the offsets between the two calexp files

          The changes in the values are:

          bgMean         -0.0278  0.015%
          bgStdDev        0.0157  6.526%
          numGoodPix         -36  0.002%
          imMean        0.000243  0.024%
          imStdDev      5.35e-05  0.000%
          varMean        -0.0165  0.013%
          varStdDev     0.000404  0.001%
          psfIxx        2.18e-06  0.000%
          psfIyy       -2.34e-06  0.000%
          psfIxy        1.81e-06  0.001%
          

          It is my estimate that these changes are not significant, and are consistent with small changes in the level of each amplifier due to the overscan changes.

          Show
          rhl Robert Lupton added a comment - - edited This fix to the median estimation changes the results in task_pipeline/tests_processCcd.py . Here is a histogram of the offsets between the two calexp files The changes in the values are: bgMean -0.0278 0.015% bgStdDev 0.0157 6.526% numGoodPix -36 0.002% imMean 0.000243 0.024% imStdDev 5.35e-05 0.000% varMean -0.0165 0.013% varStdDev 0.000404 0.001% psfIxx 2.18e-06 0.000% psfIyy -2.34e-06 0.000% psfIxy 1.81e-06 0.001% It is my estimate that these changes are not significant, and are consistent with small changes in the level of each amplifier due to the overscan changes.
          Hide
          rhl Robert Lupton added a comment -

          Made suitable API updates to obs_subaru and obs_decam, Jenkins passes, merged

          Show
          rhl Robert Lupton added a comment - Made suitable API updates to obs_subaru and obs_decam, Jenkins passes, merged

            People

            Assignee:
            rhl Robert Lupton
            Reporter:
            rhl Robert Lupton
            Reviewers:
            Paul Price
            Watchers:
            Paul Price, Robert Lupton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.