Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38973

Call to np.percentile in overscan.py leads to numpy warnings with 1.23

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ip_isr
    • 1
    • Data Release Production
    • No

    Description

      With numpy 1.23, the following call gives numpy warnings. (Lots of them). https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L620

      py.warnings WARNING: /u2/ki/erykoff/lsst-pc9_20230420/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/numpy/lib/function_base.py:4691: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
      

      This is caused by the fact that getImage gives a masked array:
      https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L599-L601

      (Masked arrays in numpy are a great idea but never work the way I think they should).

      We need to figure out how to call percentile while avoiding the mask. I think that maybe nanpercentile, filling with nans?

      Attachments

        Activity

          I've ignored the mask for this step, as anything that's masked should be outside of the middle half of the data.  This continues the current behavior, as we're simply explicitly excluding that mask from the calculation.  Doing otherwise might cause the entire calculation to fail (a test shows that if a masked value falls near one of the percentile positions, it returns that entry as "masked", which isn't helpful).

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38613/pipeline

          czw Christopher Waters added a comment - I've ignored the mask for this step, as anything that's masked should be outside of the middle half of the data.  This continues the current behavior, as we're simply explicitly excluding that mask from the calculation.  Doing otherwise might cause the entire calculation to fail (a test shows that if a masked value falls near one of the percentile positions, it returns that entry as "masked", which isn't helpful). Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38613/pipeline
          erykoff Eli Rykoff added a comment -

          Fix was even simpler than I thought!

          erykoff Eli Rykoff added a comment - Fix was even simpler than I thought!
          price Paul Price added a comment -

          In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137

          price Paul Price added a comment - In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137

          People

            czw Christopher Waters
            erykoff Eli Rykoff
            Eli Rykoff
            Christopher Waters, Eli Rykoff
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.