Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13553

Deal with large blends

    Details

    • Team:
      External

      Description

      With the sky correction (DM-9147, DM-13388), perhaps combined with the lower detection threshold (DM-12295), we are getting many large parent objects (> 10^6 pixels), deblending which is a burden on our compute budget. In several cases, this appears to be due to scattered light surviving the coadd process.

      Jim Bosch has proposed that we do a modest-scale background subtraction before detection, which should break up the large footprints; then we will restore the background before deblending so that the measurements have access to the original (good, large-scale) sky subtraction. There's a question about what scale to use in the background subtraction, which we can determine by running detection with different background subtraction scales, and seeing where we drop below the footprint size limits we've set in the deblender.

      Jim also noted that after detection, we grow footprints by the size of the PSF in our standard detection routine. Instead of growing them all at once and allowing them to merge as we currently do, he suggests growing them individually, allowing the footprints to overlap but not merge. This might also help keep the large chains from forming.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Using a background subtraction scale of 512 appears to remove the large footprints. Here's a list of filter, tract and a dict with the number of large footprints as a function of background scale:

            ('HSC-G', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5})
            ('HSC-G', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})
            ('HSC-G', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 1})
            ('HSC-G', 9697, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 10})
            ('HSC-G', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 4})
            ('HSC-R', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5})
            ('HSC-R', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            ('HSC-R', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1})
            ('HSC-R', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 7})
            ('HSC-R', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})
            ('HSC-I', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 6})
            ('HSC-I', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            ('HSC-I', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})
            ('HSC-I', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3})
            ('HSC-I', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3})
            ('HSC-Z', 8766, {64: 0, 128: 0, 1024: 3, 256: 0, 512: 0, 2048: 9})
            ('HSC-Z', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            ('HSC-Z', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})
            ('HSC-Z', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})
            ('HSC-Z', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            ('HSC-Y', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 5})
            ('HSC-Y', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            ('HSC-Y', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 2})
            ('HSC-Y', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1})
            ('HSC-Y', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})
            

            This was calculated from the w_2018_06 weekly run (which includes the same features as hscPipe 6.0-beta5; DM-13435), with thanks to Hsin-Fang Chiang for processing it.

            Note that when performing the background estimation it's important not to exclude DETECTED pixels, or the large areas of scattered light that are DETECTED (and hence chaining a bunch of detections together) won't be modified much.

            Show
            price Paul Price added a comment - Using a background subtraction scale of 512 appears to remove the large footprints. Here's a list of filter, tract and a dict with the number of large footprints as a function of background scale: ('HSC-G', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-G', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-G', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 1}) ('HSC-G', 9697, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 10}) ('HSC-G', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 4}) ('HSC-R', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-R', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-R', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-R', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 7}) ('HSC-R', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 6}) ('HSC-I', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-I', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-I', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-Z', 8766, {64: 0, 128: 0, 1024: 3, 256: 0, 512: 0, 2048: 9}) ('HSC-Z', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Z', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 5}) ('HSC-Y', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 2}) ('HSC-Y', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-Y', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) This was calculated from the w_2018_06 weekly run (which includes the same features as hscPipe 6.0-beta5; DM-13435 ), with thanks to Hsin-Fang Chiang for processing it. Note that when performing the background estimation it's important not to exclude DETECTED pixels, or the large areas of scattered light that are DETECTED (and hence chaining a bunch of detections together) won't be modified much.
            Hide
            price Paul Price added a comment - - edited

            checkDeblending.py: script used to measure the number of large blends as a function of background subtraction scale.

            Show
            price Paul Price added a comment - - edited checkDeblending.py : script used to measure the number of large blends as a function of background subtraction scale.
            Hide
            price Paul Price added a comment - - edited

            price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $ git sub
            ]commit 139bfe040fe65388ea92101340069ef3a43e38f8
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Feb 14 15:18:47 2018 -0500
             
                detection: add option to grow footprints independently
                
                Growing footprints collectively allows the footprints to merge, which
                may allow nearby sources to chain together, creating a large parent
                object which might overwhelm the deblender.
                
                Growing footprints individually allows the footprints to overlap, but
                they won't merge.
             
             python/lsst/meas/algorithms/detection.py | 12 +++++++++++-
             1 file changed, 11 insertions(+), 1 deletion(-)
             
            commit 726d0a50a2a94f169893c9050aa7093114bdc829
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 15 12:19:02 2018 -0500
             
                detection: add option to subtract temporary wide background
                
                Subtracts a wide (large-scale) background before the detection pass proper.
                This background is added back to the image after detection. The purpose is
                to suppress large footprints (e.g., from large artifacts, scattered light)
                that the deblender may choke on.
             
             python/lsst/meas/algorithms/detection.py        | 85 +++++++++++++++++++++----
             python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++----------
             2 files changed, 106 insertions(+), 48 deletions(-)
             
            commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 15 14:15:42 2018 -0500
             
                detection: remove NO_DATA regions after wide background subtraction
                
                The large NO_DATA regions at the edge of the field-of-view can cause
                large detections after doing the wide background subtraction because
                of extrapolation of the background model into areas with no constraints.
                
                We save the original image so as to restore it later.
             
             python/lsst/meas/algorithms/detection.py | 16 +++++++++++++---
             1 file changed, 13 insertions(+), 3 deletions(-)
             
             
             
            price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=) $ git sub
            commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 15 16:05:18 2018 -0500
             
                warpCompare: tweak config for better clipping of scattered light
                
                Some HSC fields contain significant scattered light, which somehow makes
                the warp variance rescaling inaccurate, which in turn allows the scattered
                light leak through into the coadd. Turned this off. Yusra notes:
                
                    The benefits of doScaleWarpVariance seemed minimal compared to
                    calcErrorFromInputVariance=False, and with doPreserveContainedBySource
                    it’s less important anyway
                
                Decreasing the clipping threshold for the template creation also helps
                keep (at least some) nasty features out of the template, and hence lets
                us pick them up when we clip. This value was previously 1.5, recently
                changed to 5 (too low and too high), so now set to 2.5.
             
             python/lsst/pipe/tasks/assembleCoadd.py | 4 ++--
             1 file changed, 2 insertions(+), 2 deletions(-)
             
            commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 15 16:11:53 2018 -0500
             
                detectCoaddSources: suppress large footprints
                
                Using a new option in the SourceDetectionTask, supress large footprints
                that can overwhelm the deblender.
                {
             
             python/lsst/pipe/tasks/multiBand.py | 1 +
             1 file changed, 1 insertion(+)
            

            Show
            price Paul Price added a comment - - edited price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $ git sub ]commit 139bfe040fe65388ea92101340069ef3a43e38f8 Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 14 15:18:47 2018 -0500   detection: add option to grow footprints independently Growing footprints collectively allows the footprints to merge, which may allow nearby sources to chain together, creating a large parent object which might overwhelm the deblender. Growing footprints individually allows the footprints to overlap, but they won't merge.   python/lsst/meas/algorithms/detection.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)   commit 726d0a50a2a94f169893c9050aa7093114bdc829 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 12:19:02 2018 -0500   detection: add option to subtract temporary wide background Subtracts a wide (large-scale) background before the detection pass proper. This background is added back to the image after detection. The purpose is to suppress large footprints (e.g., from large artifacts, scattered light) that the deblender may choke on.   python/lsst/meas/algorithms/detection.py | 85 +++++++++++++++++++++---- python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++---------- 2 files changed, 106 insertions(+), 48 deletions(-)   commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 14:15:42 2018 -0500   detection: remove NO_DATA regions after wide background subtraction The large NO_DATA regions at the edge of the field-of-view can cause large detections after doing the wide background subtraction because of extrapolation of the background model into areas with no constraints. We save the original image so as to restore it later.   python/lsst/meas/algorithms/detection.py | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)       price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=) $ git sub commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 16:05:18 2018 -0500   warpCompare: tweak config for better clipping of scattered light Some HSC fields contain significant scattered light, which somehow makes the warp variance rescaling inaccurate, which in turn allows the scattered light leak through into the coadd. Turned this off. Yusra notes: The benefits of doScaleWarpVariance seemed minimal compared to calcErrorFromInputVariance=False, and with doPreserveContainedBySource it’s less important anyway Decreasing the clipping threshold for the template creation also helps keep (at least some) nasty features out of the template, and hence lets us pick them up when we clip. This value was previously 1.5, recently changed to 5 (too low and too high), so now set to 2.5.   python/lsst/pipe/tasks/assembleCoadd.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 16:11:53 2018 -0500   detectCoaddSources: suppress large footprints Using a new option in the SourceDetectionTask, supress large footprints that can overwhelm the deblender. {   python/lsst/pipe/tasks/multiBand.py | 1 + 1 file changed, 1 insertion(+)
            Hide
            price Paul Price added a comment -

            Jenkins is green.

            Show
            price Paul Price added a comment - Jenkins is green.
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            I took a look and made some comments in github, mostly with respect to the variance scaling. Comments are attached to the commits.

            Show
            yusra Yusra AlSayyad added a comment - - edited I took a look and made some comments in github, mostly with respect to the variance scaling. Comments are attached to the commits.
            Hide
            price Paul Price added a comment -

            Reran Jenkins after rebasing, and it's green.

            Show
            price Paul Price added a comment - Reran Jenkins after rebasing, and it's green.
            Hide
            price Paul Price added a comment -

            Merged to master.

            Show
            price Paul Price added a comment - Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Watchers:
                Hisanori Furusawa, Hsin-Fang Chiang, Jim Bosch, Masayuki Tanaka, Paul Price, Robert Lupton, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel