Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8982

Incorrect binning in overscan spline interpolation

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ip_isr
    • None
    • 0.5
    • DRP S17-2
    • Data Release Production

    Description

      The ordinates for the overscan spline interpolation can violate the requirement of monotonic increasing in the presence of masked rows, causing GSL to reject it and we end up raising an exception. For example (notice the third element):

      #1  0x00002aaadf6ccae4 in lsst::afw::math::InterpolateGsl::InterpolateGsl (
          this=0x6f4ef0, x=std::vector of length 30, capacity 32 = {...}, 
          y=std::vector of length 30, capacity 32 = {...}, 
          style=lsst::afw::math::Interpolate::AKIMA_SPLINE)
          at src/math/Interpolate.cc:211
      211         int const status = ::gsl_interp_init(_interp, &x[0], &y[0], _y.size());
      (gdb) p x
      $19 = std::vector of length 30, capacity 32 = {-4.6668978729026289, 
        -1.0006934865900383, -2.2712418300653598, -0.76676245210727956, 
        -0.70019157088122619, -0.63362068965517238, -0.5668103448275863, -0.5, 
        -0.43342911877394619, -0.36685823754789276, -0.30028735632183906, 
        -0.23371647509578544, -0.1669061302681992, -0.10009578544061301, 
        -0.033524904214559385, 0.033045977011494247, 0.099616858237547928, 
        0.16618773946360157, 0.23299808429118776, 0.29980842911877403, 
        0.36637931034482762, 0.43295019157088127, 0.49952107279693497, 
        0.56609195402298851, 0.6329022988505747, 0.69971264367816088, 
        0.7662835249042147, 0.83285440613026807, 0.89942528735632199, 
        0.96623563218390807}
      

      This is because the code to generate these ordinates (binning the overscan data vector) is incorrect.

      Attachments

        Activity

          price Paul Price added a comment -

          lauren, would you review this change, please? I think we discussed this code at some point in the past, so you're probably the most familiar with it.

          pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-8982=) $ git sub
          commit f72963fa68f0344683b1a3c729f38f18ebda6367
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Jan 12 14:19:16 2017 -0500
           
              overscan: fix binning of spline ordinate
              
              The binning was incorrect in the presence of masked rows, which
              could cause the ordinate to be non-monotonic increasing, resulting
              in an exception. The problem was that the masking wasn't being
              applied in the binning, but it was in the scaling (the 'numBins'
              vector).
           
           python/lsst/ip/isr/isr.py | 6 ++++--
           1 file changed, 4 insertions(+), 2 deletions(-)
           
          commit da9fcc3a3e9fef3572a70ffe2ecabf15bae8d8d2
          Author: Paul Price <price@astro.princeton.edu>
          Date:   Thu Jan 12 14:19:31 2017 -0500
           
              overscan: plot bad data as well as good
              
              This helps us identify what was clipped.
           
           python/lsst/ip/isr/isr.py | 5 ++++-
           1 file changed, 4 insertions(+), 1 deletion(-)
          

          price Paul Price added a comment - lauren , would you review this change, please? I think we discussed this code at some point in the past, so you're probably the most familiar with it. pprice@tiger-sumire:~/LSST/ip/isr (tickets/DM-8982=) $ git sub commit f72963fa68f0344683b1a3c729f38f18ebda6367 Author: Paul Price <price@astro.princeton.edu> Date: Thu Jan 12 14:19:16 2017 -0500   overscan: fix binning of spline ordinate The binning was incorrect in the presence of masked rows, which could cause the ordinate to be non-monotonic increasing, resulting in an exception. The problem was that the masking wasn't being applied in the binning, but it was in the scaling (the 'numBins' vector).   python/lsst/ip/isr/isr.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)   commit da9fcc3a3e9fef3572a70ffe2ecabf15bae8d8d2 Author: Paul Price <price@astro.princeton.edu> Date: Thu Jan 12 14:19:31 2017 -0500   overscan: plot bad data as well as good This helps us identify what was clipped.   python/lsst/ip/isr/isr.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)

          Wow, I'm amazed this has never bitten us (to the point of the exception) before. Nice catch!

          Out of curiosity, is it the numPerBin > 0 clause on the HSC side here:
          https://github.com/HyperSuprime-Cam/ip_isr/blob/master/python/lsst/ip/isr/isr.py#L447
          that takes care of this?

          I presume you will run Jenkins + ci_hsc...if/when it passes, feel free to merge (I suppose there is a very slim chance that some test values could get modified?)

          lauren Lauren MacArthur added a comment - Wow, I'm amazed this has never bitten us (to the point of the exception) before. Nice catch! Out of curiosity, is it the numPerBin > 0 clause on the HSC side here: https://github.com/HyperSuprime-Cam/ip_isr/blob/master/python/lsst/ip/isr/isr.py#L447 that takes care of this? I presume you will run Jenkins + ci_hsc...if/when it passes, feel free to merge (I suppose there is a very slim chance that some test values could get modified?)
          price Paul Price added a comment -

          Thanks, Lauren.

          The code on the HSC side has a different approach (not using numpy.histogram to bin the ordinates). I believe I changed it because it doesn't account for masked rows in the ordinate.

          I'll run ci_hsc now.

          price Paul Price added a comment - Thanks, Lauren. The code on the HSC side has a different approach (not using numpy.histogram to bin the ordinates). I believe I changed it because it doesn't account for masked rows in the ordinate. I'll run ci_hsc now.
          price Paul Price added a comment -

          This passed Jenkins.

          Merged to master.

          price Paul Price added a comment - This passed Jenkins . Merged to master.

          People

            price Paul Price
            price Paul Price
            Lauren MacArthur
            Lauren MacArthur, Paul Price
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.