Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13606

Suppress numpy warning from measureApCorr

    XMLWordPrintable

    Details

    • Team:
      External

      Description

      Getting lots of chatter from numpy:

      /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/6.0b6-hsc/python/lsst/meas/algorithms/measureApCorr.py:237: RuntimeWarning: invalid value encountered in greater
        fluxes > 0.0,
      

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Another one:

            /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b6-hsc/python/lsst/pipe/drivers/background.py:331: RuntimeWarning: invalid value encountered in greater
              bad = numpy.abs(residuals) > self.config.skyRej*stdev
            

            Show
            price Paul Price added a comment - Another one: /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b6-hsc/python/lsst/pipe/drivers/background.py:331: RuntimeWarning: invalid value encountered in greater bad = numpy.abs(residuals) > self.config.skyRej*stdev
            Hide
            price Paul Price added a comment -

            price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13606=) $ git sub-patch
            commit 814fdc0f163d7e7d9f03807034799adab548c9a2 (HEAD -> tickets/DM-13606, origin/tickets/DM-13606)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 21 15:50:22 2018 -0500
             
                measureApCorr: suppress numpy warnings
                
                /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/6.0b6-hsc/python/lsst/meas/algorithms/measureApCorr.py:237: RuntimeWarning: invalid value encountered in greater
                  fluxes > 0.0,
             
            diff --git a/python/lsst/meas/algorithms/measureApCorr.py b/python/lsst/meas/algorithms/measureApCorr.py
            index 9af53280..14fb6268 100644
            --- a/python/lsst/meas/algorithms/measureApCorr.py
            +++ b/python/lsst/meas/algorithms/measureApCorr.py
            @@ -231,11 +231,12 @@ class MeasureApCorrTask(Task):
                         # Create a more restricted subset with only the objects where the to-be-correct flux
                         # is not flagged.
                         fluxes = numpy.fromiter((record.get(keys.flux) for record in subset1), float)
            -            isGood = numpy.logical_and.reduce([
            -                numpy.fromiter((not record.get(keys.flag) for record in subset1), bool),
            -                numpy.isfinite(fluxes),
            -                fluxes > 0.0,
            -            ])
            +            with np.errstate(invalid="ignore"):  # suppress NAN warnings
            +                isGood = numpy.logical_and.reduce([
            +                    numpy.fromiter((not record.get(keys.flag) for record in subset1), bool),
            +                    numpy.isfinite(fluxes),
            +                    fluxes > 0.0,
            +                ])
                         subset2 = [record for record, good in zip(subset1, isGood) if good]
             
                         # Check that we have enough data points that we have at least the minimum of degrees of
             
             
            price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13606=) $ git sub-patch
            commit 0574d4cd687fc864864f6a7e9161c485b4d65859 (HEAD -> tickets/DM-13606, origin/tickets/DM-13606)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 21 15:52:37 2018 -0500
             
                background: suppress numpy warnings
                
                /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b6-hsc/python/lsst/pipe/drivers/background.py:331: RuntimeWarning: invalid value encountered in greater
                  bad = numpy.abs(residuals) > self.config.skyRej*stdev
             
            diff --git a/python/lsst/pipe/drivers/background.py b/python/lsst/pipe/drivers/background.py
            index 0e5ad0b..6036f3c 100644
            --- a/python/lsst/pipe/drivers/background.py
            +++ b/python/lsst/pipe/drivers/background.py
            @@ -328,7 +328,8 @@ class SkyMeasurementTask(Task):
                         residuals = imageSamples - solution*skySamples
                         lq, uq = numpy.percentile(residuals[mask], [25, 75])
                         stdev = 0.741*(uq - lq)  # Robust stdev from IQR
            -            bad = numpy.abs(residuals) > self.config.skyRej*stdev
            +            with np.errstate(invalid="ignore"):  # suppress NAN warnings
            +                bad = numpy.abs(residuals) > self.config.skyRej*stdev
                         mask[bad] = False
             
                     return solve(mask)
            

            Show
            price Paul Price added a comment - price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13606=) $ git sub-patch commit 814fdc0f163d7e7d9f03807034799adab548c9a2 (HEAD -> tickets/DM-13606, origin/tickets/DM-13606) Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 21 15:50:22 2018 -0500   measureApCorr: suppress numpy warnings /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/6.0b6-hsc/python/lsst/meas/algorithms/measureApCorr.py:237: RuntimeWarning: invalid value encountered in greater fluxes > 0.0,   diff --git a/python/lsst/meas/algorithms/measureApCorr.py b/python/lsst/meas/algorithms/measureApCorr.py index 9af53280..14fb6268 100644 --- a/python/lsst/meas/algorithms/measureApCorr.py +++ b/python/lsst/meas/algorithms/measureApCorr.py @@ -231,11 +231,12 @@ class MeasureApCorrTask(Task): # Create a more restricted subset with only the objects where the to-be-correct flux # is not flagged. fluxes = numpy.fromiter((record.get(keys.flux) for record in subset1), float) - isGood = numpy.logical_and.reduce([ - numpy.fromiter((not record.get(keys.flag) for record in subset1), bool), - numpy.isfinite(fluxes), - fluxes > 0.0, - ]) + with np.errstate(invalid="ignore"): # suppress NAN warnings + isGood = numpy.logical_and.reduce([ + numpy.fromiter((not record.get(keys.flag) for record in subset1), bool), + numpy.isfinite(fluxes), + fluxes > 0.0, + ]) subset2 = [record for record, good in zip(subset1, isGood) if good] # Check that we have enough data points that we have at least the minimum of degrees of     price@pap-laptop:~/LSST/pipe_drivers (tickets/DM-13606=) $ git sub-patch commit 0574d4cd687fc864864f6a7e9161c485b4d65859 (HEAD -> tickets/DM-13606, origin/tickets/DM-13606) Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 21 15:52:37 2018 -0500   background: suppress numpy warnings /tigress/HSC/stack/hsc3-perseus-20171206/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b6-hsc/python/lsst/pipe/drivers/background.py:331: RuntimeWarning: invalid value encountered in greater bad = numpy.abs(residuals) > self.config.skyRej*stdev   diff --git a/python/lsst/pipe/drivers/background.py b/python/lsst/pipe/drivers/background.py index 0e5ad0b..6036f3c 100644 --- a/python/lsst/pipe/drivers/background.py +++ b/python/lsst/pipe/drivers/background.py @@ -328,7 +328,8 @@ class SkyMeasurementTask(Task): residuals = imageSamples - solution*skySamples lq, uq = numpy.percentile(residuals[mask], [25, 75]) stdev = 0.741*(uq - lq) # Robust stdev from IQR - bad = numpy.abs(residuals) > self.config.skyRej*stdev + with np.errstate(invalid="ignore"): # suppress NAN warnings + bad = numpy.abs(residuals) > self.config.skyRej*stdev mask[bad] = False return solve(mask)
            Hide
            price Paul Price added a comment -

            Simon Krughoff, would you have a look at this little change, please?

            The patches are above, but I'm happy to open GitHub PRs if you like.

            Show
            price Paul Price added a comment - Simon Krughoff , would you have a look at this little change, please? The patches are above, but I'm happy to open GitHub PRs if you like.
            Hide
            price Paul Price added a comment -

            Jenkins is green.

            Show
            price Paul Price added a comment - Jenkins is green.
            Hide
            price Paul Price added a comment -

            Simon Krughoff is busy. Merlin Fisher-Levine, would you mind?

            Show
            price Paul Price added a comment - Simon Krughoff is busy. Merlin Fisher-Levine , would you mind?
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Looks good to me!

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Looks good to me!
            Hide
            price Paul Price added a comment -

            Thanks again, Merlin.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks again, Merlin. Merged to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Merlin Fisher-Levine
              Watchers:
              Merlin Fisher-Levine, Paul Price, Simon Krughoff
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: