Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13517

"IndexError: cannot do a non-empty take from an empty axes" in making coadd

    Details

    • Team:
      External

      Description

      This error is seen near the edge of the COSMOS tract. It used to write out a tiny corner of a coadd.

      Traceback (most recent call last):
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/ctrl_pool/14.0-1-ga2912ff+25/python/lsst/ctrl/pool/parallel.py", line 496, in logOperation
          yield
        File "/home/hchiang2/stack/pipe_drivers/python/lsst/pipe/drivers/coaddDriver.py", line 316, in coadd
          detResults = self.detectCoaddSources.runDetection(coadd, idFactory, expId=expId)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-39-g03bf09b5+1/python/lsst/pipe/tasks/multiBand.py", line 303, in runDetection
          detections = self.detection.makeSourceCatalog(table, exposure, expId=expId)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+33/python/lsst/pipe/base/timer.py", line 150, in wrapper
          res = func(self, *args, **keyArgs)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/detection.py", line 292, in run
          clearMask=clearMask, expId=expId)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/dynamicDetection.py", line 180, in detectFootprints
          factor = self.calculateThreshold(exposure, seed, sigma=sigma)
        File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/meas_algorithms/14.0-10-g59393990+2/python/lsst/meas/algorithms/dynamicDetection.py", line 107, in calculateThreshold
          lq, uq = np.percentile(fluxes, [25.0, 75.0])
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4269, in percentile
          interpolation=interpolation)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4011, in _ureduce
          r = func(a, **kwargs)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4386, in _percentile
          x1 = take(ap, indices_below, axis=axis) * weights_below
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take
          return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
        File "/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
          return getattr(obj, method)(*args, **kwds)
      IndexError: cannot do a non-empty take from an empty axes.
      

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The above error was with w_2018_04 plus a bug fix of DM-13396.

            This can also be reproduced with w_2018_05

            coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM-13256:private/user/abcd --id tract=9813 patch=1,8 filter=HSC-I  --selectId visit=30482^30502 -c makeCoaddTempExp.doApplySkyCorr=False --cores 1 
            

            `

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The above error was with w_2018_04 plus a bug fix of DM-13396 . This can also be reproduced with w_2018_05 coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_04/DM- 13256 : private /user/abcd --id tract= 9813 patch= 1 , 8 filter=HSC-I --selectId visit= 30482 ^ 30502 -c makeCoaddTempExp.doApplySkyCorr=False --cores 1 `
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In w_2018_12's RC2 run I'm getting fewer of this error, compared to w_2018_10. I'd guess DM-13741 might be related.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In w_2018_12's RC2 run I'm getting fewer of this error, compared to w_2018_10. I'd guess DM-13741  might be related.
            Hide
            price Paul Price added a comment -

            Hsin-Fang Chiang, would you be willing to review these small fixes?

            price@pap-laptop-2:~/LSST/meas_algorithms (tickets/DM-13517=) $ git sub
            commit be77d0d7aa0b4b7c271e926fc9d5cf5c12f6367a (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Apr 10 15:21:32 2018 -0400
             
                dynamicDetection: require sufficient sources for statistics
                
                Makes the minimum number of sources for statistics configurable,
                avoiding obscure numpy errors from having zero sources.
             
             python/lsst/meas/algorithms/dynamicDetection.py | 9 +++++++++
             tests/test_dynamicDetection.py                  | 4 ++++
             2 files changed, 13 insertions(+)
             
             
            price@pap-laptop-2:~/LSST/pipe_tasks (tickets/DM-13517=) $ git sub
            commit 2fd1a86b4b035d32a70ee87724f5eefb9babe4bc
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Apr 10 15:29:25 2018 -0400
             
                scaleVariance: catch NAN rescaling factor
                
                If the rescaling factor is NAN, *bad* things can happen...
             
             python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
             1 file changed, 2 insertions(+), 2 deletions(-)
             
            commit 1e3c7fecc89a3b12bbdb90018b88a6ef165fd356
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Apr 10 15:30:30 2018 -0400
             
                scaleVariance: ensure variance is postive
                
                Imaginary universes come crashing down on you when the variance is negative.
             
             python/lsst/pipe/tasks/scaleVariance.py | 4 ++--
             1 file changed, 2 insertions(+), 2 deletions(-)
             
            commit 0f5851be82104accccf88cf8ce784c225dea707c (HEAD -> tickets/DM-13517, origin/tickets/DM-13517)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Apr 10 15:31:02 2018 -0400
             
                scaleVariance: fix undefined variable
                
                maskVal wasn't defined.
             
             python/lsst/pipe/tasks/scaleVariance.py | 1 +
             1 file changed, 1 insertion(+)
            

            Show
            price Paul Price added a comment - Hsin-Fang Chiang , would you be willing to review these small fixes? price@pap-laptop-2:~/LSST/meas_algorithms (tickets/DM-13517=) $ git sub commit be77d0d7aa0b4b7c271e926fc9d5cf5c12f6367a (HEAD -> tickets/DM-13517, origin/tickets/DM-13517) Author: Paul Price <price@astro.princeton.edu> Date: Tue Apr 10 15:21:32 2018 -0400   dynamicDetection: require sufficient sources for statistics Makes the minimum number of sources for statistics configurable, avoiding obscure numpy errors from having zero sources.   python/lsst/meas/algorithms/dynamicDetection.py | 9 +++++++++ tests/test_dynamicDetection.py | 4 ++++ 2 files changed, 13 insertions(+)     price@pap-laptop-2:~/LSST/pipe_tasks (tickets/DM-13517=) $ git sub commit 2fd1a86b4b035d32a70ee87724f5eefb9babe4bc Author: Paul Price <price@astro.princeton.edu> Date: Tue Apr 10 15:29:25 2018 -0400   scaleVariance: catch NAN rescaling factor If the rescaling factor is NAN, *bad* things can happen...   python/lsst/pipe/tasks/scaleVariance.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 1e3c7fecc89a3b12bbdb90018b88a6ef165fd356 Author: Paul Price <price@astro.princeton.edu> Date: Tue Apr 10 15:30:30 2018 -0400   scaleVariance: ensure variance is postive Imaginary universes come crashing down on you when the variance is negative.   python/lsst/pipe/tasks/scaleVariance.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit 0f5851be82104accccf88cf8ce784c225dea707c (HEAD -> tickets/DM-13517, origin/tickets/DM-13517) Author: Paul Price <price@astro.princeton.edu> Date: Tue Apr 10 15:31:02 2018 -0400   scaleVariance: fix undefined variable maskVal wasn't defined.   python/lsst/pipe/tasks/scaleVariance.py | 1 + 1 file changed, 1 insertion(+)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The error was seen at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' in w_2018_14's RC2 reprocessing (DM-13890), and can be reproduced with w_2018_14:

            coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_14/DM-13890:private/hchiang2/DM-13517 --id tract=9813 patch=0,7 filter=HSC-Z  --selectId visit=17930  -c makeCoaddTempExp.doApplySkyCorr=False --cores 1
            

            With the ticket branches, the same command runs without the error. Got

            coaddDriver.detectCoaddSources.detection WARN: Insufficient good flux measurements (0 < 10) for dynamic threshold calculation
            

            as expected and finished detectCoaddSources.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The error was seen at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' in w_2018_14's RC2 reprocessing ( DM-13890 ), and can be reproduced with w_2018_14 : coaddDriver.py /datasets/hsc/repo/ --rerun RC/w_2018_14/DM- 13890 : private /hchiang2/DM- 13517 --id tract= 9813 patch= 0 , 7 filter=HSC-Z --selectId visit= 17930 -c makeCoaddTempExp.doApplySkyCorr=False --cores 1 With the ticket branches, the same command runs without the error. Got coaddDriver.detectCoaddSources.detection WARN: Insufficient good flux measurements ( 0 < 10 ) for dynamic threshold calculation as expected and finished detectCoaddSources.
            Hide
            price Paul Price added a comment -

            Jenkins seems to have passed: failed on two elements because it couldn't find the demo script. But ci_hsc passed on all, and my changes should have nothing to do with the demo.

            Show
            price Paul Price added a comment - Jenkins seems to have passed: failed on two elements because it couldn't find the demo script. But ci_hsc passed on all, and my changes should have nothing to do with the demo.
            Hide
            price Paul Price added a comment -

            Merged to master.

            Thanks Hsin-Fang Chiang!

            Show
            price Paul Price added a comment - Merged to master. Thanks Hsin-Fang Chiang !

              People

              • Assignee:
                price Paul Price
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Paul Price, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel