Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ip_isr
-
Labels:
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
With numpy 1.23, the following call gives numpy warnings. (Lots of them). https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L620
py.warnings WARNING: /u2/ki/erykoff/lsst-pc9_20230420/conda/envs/lsst-scipipe-6.0.0/lib/python3.10/site-packages/numpy/lib/function_base.py:4691: UserWarning: Warning: 'partition' will ignore the 'mask' of the MaskedArray.
|
This is caused by the fact that getImage gives a masked array:
https://github.com/lsst/ip_isr/blob/main/python/lsst/ip/isr/overscan.py#L599-L601
(Masked arrays in numpy are a great idea but never work the way I think they should).
We need to figure out how to call percentile while avoiding the mask. I think that maybe nanpercentile, filling with nans?
Attachments
Activity
Fix was even simpler than I thought!
In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137
Paul Price
added a comment - In case it's helpful, here's a function that provides quartiles for masked data: https://github.com/Subaru-PFS/drp_stella/blob/master/include/pfs/drp/stella/math/quartiles.h#L126-L137
I've ignored the mask for this step, as anything that's masked should be outside of the middle half of the data. This continues the current behavior, as we're simply explicitly excluding that mask from the calculation. Doing otherwise might cause the entire calculation to fail (a test shows that if a masked value falls near one of the percentile positions, it returns that entry as "masked", which isn't helpful).
Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/38613/pipeline