Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38973

Call to np.percentile in overscan.py leads to numpy warnings with 1.23

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • ip_isr
    • 1
    • Data Release Production
    • No

    Description

      With numpy 1.23, the following call gives numpy warnings. (Lots of them). https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L620

      py.warnings WARNING: /u2/ki/erykoff/lsst-pc9_20230420/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/numpy/lib/function_base.py:4691: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
      

      This is caused by the fact that getImage gives a masked array:
      https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L599-L601

      (Masked arrays in numpy are a great idea but never work the way I think they should).

      We need to figure out how to call percentile while avoiding the mask. I think that maybe nanpercentile, filling with nans?

      Attachments

        Activity

          No builds found.
          erykoff Eli Rykoff created issue -
          czw Christopher Waters made changes -
          Field Original Value New Value
          Status To Do [ 10001 ] In Progress [ 3 ]

          I've ignored the mask for this step, as anything that's masked should be outside of the middle half of the data.  This continues the current behavior, as we're simply explicitly excluding that mask from the calculation.  Doing otherwise might cause the entire calculation to fail (a test shows that if a masked value falls near one of the percentile positions, it returns that entry as "masked", which isn't helpful).

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38613/pipeline

          czw Christopher Waters added a comment - I've ignored the mask for this step, as anything that's masked should be outside of the middle half of the data.  This continues the current behavior, as we're simply explicitly excluding that mask from the calculation.  Doing otherwise might cause the entire calculation to fail (a test shows that if a masked value falls near one of the percentile positions, it returns that entry as "masked", which isn't helpful). Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38613/pipeline
          czw Christopher Waters made changes -
          Reviewers Eli Rykoff [ erykoff ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          erykoff Eli Rykoff added a comment -

          Fix was even simpler than I thought!

          erykoff Eli Rykoff added a comment - Fix was even simpler than I thought!
          erykoff Eli Rykoff made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          price Paul Price added a comment -

          In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137

          price Paul Price added a comment - In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137
          czw Christopher Waters made changes -
          Epic Link DM-32177 [ 779901 ]
          Story Points 1
          czw Christopher Waters made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

          People

            czw Christopher Waters
            erykoff Eli Rykoff
            Eli Rykoff
            Christopher Waters, Eli Rykoff
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.