Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11369

ValueError from SafeClipAssembleCoaddTask.detectClip

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Team:
      External

      Description

      Dominique Boutigny reports this error:

      [lsstprod@cca010 ~]$ cd /sps/lsst/dev/lsstprod/hsc/SXDS/utils/assembleCoadd
      [lsstprod@cca010 assembleCoadd]$ source setup.sh
      [lsstprod@cca010 assembleCoadd]$ assembleCoadd.py _parent/output --output _parent/output --id filter=g tract=0 patch=9,6 @g.list --clobber-config --doraise
      [...]
      safeClipAssembleCoadd INFO: Found 7943 potential clipped objects
      Traceback (most recent call last):
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-44-g89df7c8/bin/assembleCoadd.py", line 65, in <module>
          SafeClipAssembleCoaddTask.parseAndRun(args=extra[1:])
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 514, in parseAndRun
          resultList = taskRunner.run(parsedCmd)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 223, in run
          resultList = list(mapFunc(self, targetList))
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/cmdLineTask.py", line 373, in __call__
          result = task.run(dataRef, **kwargs)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/timer.py", line 121, in wrapper
          res = func(self, *args, **keyArgs)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/assembleCoadd.py", line 344, in run
          doClip=self.config.doSigmaClip)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/assembleCoadd.py", line 1118, in assemble
          result = self.detectClip(exp, tempExpRefList)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/stack/miniconda3-4.2.12-7c8e67/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/assembleCoadd.py", line 1261, in detectClip
          if numpy.max(overlap[clipIndexComp]) < self.config.maxClipFootOverlapDouble:
        File "/sps/lsst/software/lsst_distrib/w_2017_28/python/miniconda3-4.2.12/lib/python3.5/site-packages/numpy/core/fromnumeric.py", line 2297, in amax
          out=out, **kwargs)
        File "/sps/lsst/software/lsst_distrib/w_2017_28/python/miniconda3-4.2.12/lib/python3.5/site-packages/numpy/core/_methods.py", line 26, in _amax
          return umr_maximum(a, axis, None, out, keepdims)
      ValueError: zero-size array to reduction operation maximum which has no identity
      

      On investigation, this appears to be due to a logic error in the code (< instead of <=).

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -
            Show
            price Paul Price added a comment - Passed Jenkins . Failure on CentOS7 is due to a race condition in ci_hsc .
            Hide
            price Paul Price added a comment -

            Dominique Boutigny, could you please review this simple fix:

            price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-11369=) $ git sub-patch
            commit 9a2b66ff6a3c4ecca85546caaa0da222df85d962
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Jul 25 13:29:17 2017 -0400
             
                SafeClipAssembleCoaddTask: fix logic error
                
                The opposite of '>' is '<=', not '<'. In rare situations, this can cause
                clipIndexComp to be empty, so indexing with it will cause a ValueError.
             
            diff --git a/python/lsst/pipe/tasks/assembleCoadd.py b/python/lsst/pipe/tasks/assembleCoadd.py
            index 48db48c..105ddea 100755
            --- a/python/lsst/pipe/tasks/assembleCoadd.py
            +++ b/python/lsst/pipe/tasks/assembleCoadd.py
            @@ -1263,8 +1263,8 @@ class SafeClipAssembleCoaddTask(AssembleCoaddTask):
                             # Test if there are clipped objects that overlap two different visits
                             clipIndex = numpy.where(overlap > self.config.minClipFootOverlapDouble)[0]
                             if len(clipIndex) == 2 and len(overlap) > 3:
            -                    clipIndexComp = numpy.where(overlap < self.config.minClipFootOverlapDouble)[0]
            -                    if numpy.max(overlap[clipIndexComp]) < self.config.maxClipFootOverlapDouble:
            +                    clipIndexComp = numpy.where(overlap <= self.config.minClipFootOverlapDouble)[0]
            +                    if numpy.max(overlap[clipIndexComp]) <= self.config.maxClipFootOverlapDouble:
                                     keep = True
                                     keepIndex = clipIndex
             
            

            Show
            price Paul Price added a comment - Dominique Boutigny , could you please review this simple fix: price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-11369=) $ git sub-patch commit 9a2b66ff6a3c4ecca85546caaa0da222df85d962 Author: Paul Price <price@astro.princeton.edu> Date: Tue Jul 25 13:29:17 2017 -0400   SafeClipAssembleCoaddTask: fix logic error The opposite of '>' is '<=', not '<'. In rare situations, this can cause clipIndexComp to be empty, so indexing with it will cause a ValueError.   diff --git a/python/lsst/pipe/tasks/assembleCoadd.py b/python/lsst/pipe/tasks/assembleCoadd.py index 48db48c..105ddea 100755 --- a/python/lsst/pipe/tasks/assembleCoadd.py +++ b/python/lsst/pipe/tasks/assembleCoadd.py @@ -1263,8 +1263,8 @@ class SafeClipAssembleCoaddTask(AssembleCoaddTask): # Test if there are clipped objects that overlap two different visits clipIndex = numpy.where(overlap > self.config.minClipFootOverlapDouble)[0] if len(clipIndex) == 2 and len(overlap) > 3: - clipIndexComp = numpy.where(overlap < self.config.minClipFootOverlapDouble)[0] - if numpy.max(overlap[clipIndexComp]) < self.config.maxClipFootOverlapDouble: + clipIndexComp = numpy.where(overlap <= self.config.minClipFootOverlapDouble)[0] + if numpy.max(overlap[clipIndexComp]) <= self.config.maxClipFootOverlapDouble: keep = True keepIndex = clipIndex
            Hide
            boutigny Dominique Boutigny added a comment -

            This is fine for me.

            Show
            boutigny Dominique Boutigny added a comment - This is fine for me.
            Hide
            price Paul Price added a comment -

            Thanks Dominique.

            Merged to master.

            Show
            price Paul Price added a comment - Thanks Dominique. Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Dominique Boutigny
                Watchers:
                Dominique Boutigny, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel