Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10574

Hit AssertionError in deblender

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_deblender
    • Labels:
      None
    • Story Points:
      1
    • Team:
      Data Release Production

      Description

      In a validation run for hscPipe 5.0-beta5:

      pprice@tiger-sumire:/tigress/pprice/hscPipe-5.0-beta5 $ measureCoaddSources.py /tigress/HSC/HSC --rerun hscPipe-5.0-beta5/20170517/wide --id tract=8764 patch=0,0 filter=HSC-I
      root INFO: Loading config overrride file '/tigress/HSC/stack/hsc-tiger-20170322/Linux64/obs_subaru/13.0-18-gbe0b5c0/config/measureCoaddSources.py'
      root INFO: Loading config overrride file '/tigress/HSC/stack/hsc-tiger-20170322/Linux64/obs_subaru/13.0-18-gbe0b5c0/config/hsc/measureCoaddSources.py'
      root INFO: Running: /tigress/pprice/hscPipe-5.0-beta5/pipe_tasks/bin/measureCoaddSources.py /tigress/HSC/HSC --rerun hscPipe-5.0-beta5/20170517/wide --id tract=8764 patch=0,0 filter=HSC-I
      /tigress/HSC/stack/hsc-tiger-20170322/Linux64/miniconda2/4.2.12.lsst1/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
        warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
      measureCoaddSources INFO: Read 7020 detections: DataId(initialdata={'filter': 'HSC-I', 'patch': '0,0', 'tract': 8764}, tag=set([]))
      measureCoaddSources.deblend INFO: Deblending 7020 sources
      measureCoaddSources FATAL: Failed on dataId=DataId(initialdata={'filter': 'HSC-I', 'patch': '0,0', 'tract': 8764}, tag=set([])): 
      Traceback (most recent call last):
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/pipe_base/13.0+10/python/lsst/pipe/base/cmdLineTask.py", line 347, in __call__
          result = task.run(dataRef, **kwargs)
        File "/tigress/pprice/hscPipe-5.0-beta5/pipe_tasks/python/lsst/pipe/tasks/multiBand.py", line 1086, in run
          self.deblend.run(exposure, sources)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/pipe_base/13.0+10/python/lsst/pipe/base/timer.py", line 121, in wrapper
          res = func(self, *args, **keyArgs)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/deblend.py", line 238, in run
          self.deblend(exposure, sources, psf)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/pipe_base/13.0+10/python/lsst/pipe/base/timer.py", line 121, in wrapper
          res = func(self, *args, **keyArgs)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/deblend.py", line 323, in deblend
          medianSmoothTemplate=self.config.medianSmoothTemplate
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/baseline.py", line 667, in deblend
          debResult = newDeblend(debPlugins, footprint, maskedImage, psf, psffwhm, filters, log, verbose, avgNoise)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/baseline.py", line 739, in newDeblend
          reset = debPlugins[step].run(debResult, log)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/plugins.py", line 76, in run
          reset = self.func(debResult, log, **self.kwargs)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/plugins.py", line 152, in fitPsfs
          dp.img, dp.varimg, psfChisqCut1, psfChisqCut2, psfChisqCut2b, tinyFootprintSize)
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/plugins.py", line 371, in _fitPsf
          sx1, sx2, sx3, sx4 = _overlap(xlo, xhi, obb.getMinX(), obb.getMaxX())
        File "/tigress/HSC/stack/hsc-tiger-20170322/Linux64/meas_deblender/13.0-3-gdadfd13+1/python/lsst/meas/deblender/plugins.py", line 319, in _overlap
          (xlo <= xhi) and (xmin <= xmax))
      AssertionError
      

      After a bit of poking, I believe it's due to the use of LOCAL here.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            With LOCAL removed, it works just fine.

            price@pap-laptop:~/LSST/meas_deblender (tickets/DM-10574=) $ git sub-patch
            commit dba71810921104b96f8ca10ed7d402f5abf678b3
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon May 22 14:51:21 2017 -0400
             
                fitPsf: fix non-overlapping bounding boxes
                
                Was getting an AssertionError from the _overlap internal function
                because the bounding boxes didn't overlap at all. This was surprising
                because the list of peaks should overlap by construction. On closer
                inspection, found that at construction time the LOCAL bounding box was
                checked, but the assertion uses the default frame (which is now PARENT).
                Switched to using the default (PARENT) frame for construction for
                consistency (LOCAL isn't used anywhere else in the deblender).
             
            diff --git a/python/lsst/meas/deblender/plugins.py b/python/lsst/meas/deblender/plugins.py
            index b00c8a2..ddaad87 100755
            --- a/python/lsst/meas/deblender/plugins.py
            +++ b/python/lsst/meas/deblender/plugins.py
            @@ -279,7 +279,7 @@ def _fitPsf(fp, fmask, pk, pkF, pkres, fbb, peaks, peaksF, log, psf, psffwhm,
                     if pkF.distanceSquared(pkF2) > R2**2:
                         continue
                     opsfimg = psf.computeImage(pkF2.getX(), pkF2.getY())
            -        if not opsfimg.getBBox(afwImage.LOCAL).overlaps(stampbb):
            +        if not opsfimg.getBBox().overlaps(stampbb):
                         continue
                     otherpeaks.append(opsfimg)
                     log.trace('%i other peaks within range', len(otherpeaks))
            

            Show
            price Paul Price added a comment - With LOCAL removed, it works just fine. price@pap-laptop:~/LSST/meas_deblender (tickets/DM-10574=) $ git sub-patch commit dba71810921104b96f8ca10ed7d402f5abf678b3 Author: Paul Price <price@astro.princeton.edu> Date: Mon May 22 14:51:21 2017 -0400   fitPsf: fix non-overlapping bounding boxes Was getting an AssertionError from the _overlap internal function because the bounding boxes didn't overlap at all. This was surprising because the list of peaks should overlap by construction. On closer inspection, found that at construction time the LOCAL bounding box was checked, but the assertion uses the default frame (which is now PARENT). Switched to using the default (PARENT) frame for construction for consistency (LOCAL isn't used anywhere else in the deblender).   diff --git a/python/lsst/meas/deblender/plugins.py b/python/lsst/meas/deblender/plugins.py index b00c8a2..ddaad87 100755 --- a/python/lsst/meas/deblender/plugins.py +++ b/python/lsst/meas/deblender/plugins.py @@ -279,7 +279,7 @@ def _fitPsf(fp, fmask, pk, pkF, pkres, fbb, peaks, peaksF, log, psf, psffwhm, if pkF.distanceSquared(pkF2) > R2**2: continue opsfimg = psf.computeImage(pkF2.getX(), pkF2.getY()) - if not opsfimg.getBBox(afwImage.LOCAL).overlaps(stampbb): + if not opsfimg.getBBox().overlaps(stampbb): continue otherpeaks.append(opsfimg) log.trace('%i other peaks within range', len(otherpeaks))
            Hide
            price Paul Price added a comment - - edited

            Hsin-Fang Chiang, since you said you also ran into this, would you mind having a quick look? The patch is above — real simple.

            It's currently going through Jenkins.

            Show
            price Paul Price added a comment - - edited Hsin-Fang Chiang , since you said you also ran into this, would you mind having a quick look? The patch is above — real simple. It's currently going through Jenkins.
            Hide
            price Paul Price added a comment -

            Ack! This breaks the demo, presumably because the small changes to the deblender produce small differences in the output. I'll update the demo.

            Show
            price Paul Price added a comment - Ack! This breaks the demo , presumably because the small changes to the deblender produce small differences in the output. I'll update the demo.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Looks good to me.

            I wasn't able to build this with w_2017_17. So I tried w_2017_20 and reproduced the same error with a patch in VVDS. With your fix, it successfully finished (after spending a while measuring 22091 sources!)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Looks good to me. I wasn't able to build this with w_2017_17. So I tried w_2017_20 and reproduced the same error with a patch in VVDS. With your fix, it successfully finished (after spending a while measuring 22091 sources!)
            Hide
            price Paul Price added a comment -

            Updated the demo, and now a new Jenkins is running...

            pprice@tiger-sumire:~/LSST/lsst_dm_stack_demo (tickets/DM-10574=) $ git sub
            commit 7789524c69f40bd8feefc0a0af2995f2e9e17c35
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon May 22 21:49:52 2017 -0400
             
                update expected results following deblender changes
                
                DM-10574 fixed a deblender bug that, when fixed, changes how
                the deblender works ever so slightly, requiring updated results
                for the demo.
             
             expected/DarwinX86/detected-sources.txt       | 6614 ++++++++++++-------------
             expected/DarwinX86/detected-sources_small.txt | 5032 +++++++++----------
             expected/Linux64/detected-sources.txt         | 6614 ++++++++++++-------------
             expected/Linux64/detected-sources_small.txt   | 5032 +++++++++----------
             4 files changed, 11646 insertions(+), 11646 deletions(-)
            

            Show
            price Paul Price added a comment - Updated the demo, and now a new Jenkins is running... pprice@tiger-sumire:~/LSST/lsst_dm_stack_demo (tickets/DM-10574=) $ git sub commit 7789524c69f40bd8feefc0a0af2995f2e9e17c35 Author: Paul Price <price@astro.princeton.edu> Date: Mon May 22 21:49:52 2017 -0400   update expected results following deblender changes DM-10574 fixed a deblender bug that, when fixed, changes how the deblender works ever so slightly, requiring updated results for the demo.   expected/DarwinX86/detected-sources.txt | 6614 ++++++++++++------------- expected/DarwinX86/detected-sources_small.txt | 5032 +++++++++---------- expected/Linux64/detected-sources.txt | 6614 ++++++++++++------------- expected/Linux64/detected-sources_small.txt | 5032 +++++++++---------- 4 files changed, 11646 insertions(+), 11646 deletions(-)
            Hide
            price Paul Price added a comment -

            Well, that Jenkins failed. I believe it's because it's not looking at the ticket branch. John Swinbank writes:

            Jenkins will only run the demo on master: you have to test it manually, merge to master, and stand by to fix it if it breaks.

            Having tested it manually, I'm about to merge to master and then will re-run Jenkins.

            Show
            price Paul Price added a comment - Well, that Jenkins failed. I believe it's because it's not looking at the ticket branch. John Swinbank writes: Jenkins will only run the demo on master: you have to test it manually, merge to master, and stand by to fix it if it breaks. Having tested it manually, I'm about to merge to master and then will re-run Jenkins.
            Hide
            price Paul Price added a comment -
            Show
            price Paul Price added a comment - Merged, and Jenkins is running on the new master...
            Hide
            price Paul Price added a comment -

            Jenkins passed, and all is well.

            Show
            price Paul Price added a comment - Jenkins passed, and all is well.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: