Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8982

Incorrect binning in overscan spline interpolation

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_isr
    • Labels:
      None

      Description

      The ordinates for the overscan spline interpolation can violate the requirement of monotonic increasing in the presence of masked rows, causing GSL to reject it and we end up raising an exception. For example (notice the third element):

      #1  0x00002aaadf6ccae4 in lsst::afw::math::InterpolateGsl::InterpolateGsl (
          this=0x6f4ef0, x=std::vector of length 30, capacity 32 = {...}, 
          y=std::vector of length 30, capacity 32 = {...}, 
          style=lsst::afw::math::Interpolate::AKIMA_SPLINE)
          at src/math/Interpolate.cc:211
      211         int const status = ::gsl_interp_init(_interp, &x[0], &y[0], _y.size());
      (gdb) p x
      $19 = std::vector of length 30, capacity 32 = {-4.6668978729026289, 
        -1.0006934865900383, -2.2712418300653598, -0.76676245210727956, 
        -0.70019157088122619, -0.63362068965517238, -0.5668103448275863, -0.5, 
        -0.43342911877394619, -0.36685823754789276, -0.30028735632183906, 
        -0.23371647509578544, -0.1669061302681992, -0.10009578544061301, 
        -0.033524904214559385, 0.033045977011494247, 0.099616858237547928, 
        0.16618773946360157, 0.23299808429118776, 0.29980842911877403, 
        0.36637931034482762, 0.43295019157088127, 0.49952107279693497, 
        0.56609195402298851, 0.6329022988505747, 0.69971264367816088, 
        0.7662835249042147, 0.83285440613026807, 0.89942528735632199, 
        0.96623563218390807}
      

      This is because the code to generate these ordinates (binning the overscan data vector) is incorrect.

        Attachments

          Activity

          Hide
          price Paul Price added a comment -

          Lauren MacArthur, would you review this change, please? I think we discussed this code at some point in the past, so you're probably the most familiar with it.

          pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-8982=) $ git sub
          commit f72963fa68f0344683b1a3c729f38f18ebda6367
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Jan 12 14:19:16 2017 -0500
           
              overscan: fix binning of spline ordinate
              
              The binning was incorrect in the presence of masked rows, which
              could cause the ordinate to be non-monotonic increasing, resulting
              in an exception. The problem was that the masking wasn't being
              applied in the binning, but it was in the scaling (the 'numBins'
              vector).
           
           python/lsst/ip/isr/isr.py | 6 ++++--
           1 file changed, 4 insertions(+), 2 deletions(-)
           
          commit da9fcc3a3e9fef3572a70ffe2ecabf15bae8d8d2
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Jan 12 14:19:31 2017 -0500
           
              overscan: plot bad data as well as good
              
              This helps us identify what was clipped.
           
           python/lsst/ip/isr/isr.py | 5 ++++-
           1 file changed, 4 insertions(+), 1 deletion(-)
          

          Show
          price Paul Price added a comment - Lauren MacArthur , would you review this change, please? I think we discussed this code at some point in the past, so you're probably the most familiar with it. pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-8982=) $ git sub commit f72963fa68f0344683b1a3c729f38f18ebda6367 Author: Paul Price <price@astro.princeton.edu> Date: Thu Jan 12 14:19:16 2017 -0500   overscan: fix binning of spline ordinate The binning was incorrect in the presence of masked rows, which could cause the ordinate to be non-monotonic increasing, resulting in an exception. The problem was that the masking wasn't being applied in the binning, but it was in the scaling (the 'numBins' vector).   python/lsst/ip/isr/isr.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)   commit da9fcc3a3e9fef3572a70ffe2ecabf15bae8d8d2 Author: Paul Price <price@astro.princeton.edu> Date: Thu Jan 12 14:19:31 2017 -0500   overscan: plot bad data as well as good This helps us identify what was clipped.   python/lsst/ip/isr/isr.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
          Hide
          lauren Lauren MacArthur added a comment -

          Wow, I'm amazed this has never bitten us (to the point of the exception) before. Nice catch!

          Out of curiosity, is it the numPerBin > 0 clause on the HSC side here:
          https://github.com/HyperSuprime-Cam/ip_isr/blob/master/python/lsst/ip/isr/isr.py#L447
          that takes care of this?

          I presume you will run Jenkins + ci_hsc...if/when it passes, feel free to merge (I suppose there is a very slim chance that some test values could get modified?)

          Show
          lauren Lauren MacArthur added a comment - Wow, I'm amazed this has never bitten us (to the point of the exception) before. Nice catch! Out of curiosity, is it the numPerBin > 0 clause on the HSC side here: https://github.com/HyperSuprime-Cam/ip_isr/blob/master/python/lsst/ip/isr/isr.py#L447 that takes care of this? I presume you will run Jenkins + ci_hsc...if/when it passes, feel free to merge (I suppose there is a very slim chance that some test values could get modified?)
          Hide
          price Paul Price added a comment -

          Thanks, Lauren.

          The code on the HSC side has a different approach (not using numpy.histogram to bin the ordinates). I believe I changed it because it doesn't account for masked rows in the ordinate.

          I'll run ci_hsc now.

          Show
          price Paul Price added a comment - Thanks, Lauren. The code on the HSC side has a different approach (not using numpy.histogram to bin the ordinates). I believe I changed it because it doesn't account for masked rows in the ordinate. I'll run ci_hsc now.
          Hide
          price Paul Price added a comment -

          This passed Jenkins.

          Merged to master.

          Show
          price Paul Price added a comment - This passed Jenkins . Merged to master.

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Lauren MacArthur
              Watchers:
              Lauren MacArthur, Paul Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel