# Deal with large blends

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Team:
External

#### Description

With the sky correction (DM-9147, DM-13388), perhaps combined with the lower detection threshold (DM-12295), we are getting many large parent objects (> 10^6 pixels), deblending which is a burden on our compute budget. In several cases, this appears to be due to scattered light surviving the coadd process.

Jim Bosch has proposed that we do a modest-scale background subtraction before detection, which should break up the large footprints; then we will restore the background before deblending so that the measurements have access to the original (good, large-scale) sky subtraction. There's a question about what scale to use in the background subtraction, which we can determine by running detection with different background subtraction scales, and seeing where we drop below the footprint size limits we've set in the deblender.

Jim also noted that after detection, we grow footprints by the size of the PSF in our standard detection routine. Instead of growing them all at once and allowing them to merge as we currently do, he suggests growing them individually, allowing the footprints to overlap but not merge. This might also help keep the large chains from forming.

#### Attachments

1. checkDeblending.py
3 kB

#### Activity

Hide
Paul Price added a comment -

Using a background subtraction scale of 512 appears to remove the large footprints. Here's a list of filter, tract and a dict with the number of large footprints as a function of background scale:

 ('HSC-G', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-G', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-G', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 1}) ('HSC-G', 9697, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 10}) ('HSC-G', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 4}) ('HSC-R', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-R', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-R', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-R', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 7}) ('HSC-R', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 6}) ('HSC-I', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-I', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-I', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-Z', 8766, {64: 0, 128: 0, 1024: 3, 256: 0, 512: 0, 2048: 9}) ('HSC-Z', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Z', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 5}) ('HSC-Y', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 2}) ('HSC-Y', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-Y', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) 

This was calculated from the w_2018_06 weekly run (which includes the same features as hscPipe 6.0-beta5; DM-13435), with thanks to Hsin-Fang Chiang for processing it.

Note that when performing the background estimation it's important not to exclude DETECTED pixels, or the large areas of scattered light that are DETECTED (and hence chaining a bunch of detections together) won't be modified much.

Show
Paul Price added a comment - Using a background subtraction scale of 512 appears to remove the large footprints. Here's a list of filter, tract and a dict with the number of large footprints as a function of background scale: ('HSC-G', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-G', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-G', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 1}) ('HSC-G', 9697, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 10}) ('HSC-G', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 4}) ('HSC-R', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5}) ('HSC-R', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-R', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-R', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 7}) ('HSC-R', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 6}) ('HSC-I', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-I', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-I', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-I', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3}) ('HSC-Z', 8766, {64: 0, 128: 0, 1024: 3, 256: 0, 512: 0, 2048: 9}) ('HSC-Z', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Z', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2}) ('HSC-Z', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 5}) ('HSC-Y', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) ('HSC-Y', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 2}) ('HSC-Y', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1}) ('HSC-Y', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0}) This was calculated from the w_2018_06 weekly run (which includes the same features as hscPipe 6.0-beta5; DM-13435 ), with thanks to Hsin-Fang Chiang for processing it. Note that when performing the background estimation it's important not to exclude DETECTED pixels, or the large areas of scattered light that are DETECTED (and hence chaining a bunch of detections together) won't be modified much.
Hide
Paul Price added a comment - - edited

checkDeblending.py: script used to measure the number of large blends as a function of background subtraction scale.

Show
Paul Price added a comment - - edited checkDeblending.py : script used to measure the number of large blends as a function of background subtraction scale.
Hide
Paul Price added a comment - - edited

 price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $git sub ]commit 139bfe040fe65388ea92101340069ef3a43e38f8 Author: Paul Price  Date: Wed Feb 14 15:18:47 2018 -0500    detection: add option to grow footprints independently    Growing footprints collectively allows the footprints to merge, which  may allow nearby sources to chain together, creating a large parent  object which might overwhelm the deblender.    Growing footprints individually allows the footprints to overlap, but  they won't merge.    python/lsst/meas/algorithms/detection.py | 12 +++++++++++-  1 file changed, 11 insertions(+), 1 deletion(-)   commit 726d0a50a2a94f169893c9050aa7093114bdc829 Author: Paul Price  Date: Thu Feb 15 12:19:02 2018 -0500    detection: add option to subtract temporary wide background    Subtracts a wide (large-scale) background before the detection pass proper.  This background is added back to the image after detection. The purpose is  to suppress large footprints (e.g., from large artifacts, scattered light)  that the deblender may choke on.    python/lsst/meas/algorithms/detection.py | 85 +++++++++++++++++++++----  python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++----------  2 files changed, 106 insertions(+), 48 deletions(-)   commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price  Date: Thu Feb 15 14:15:42 2018 -0500    detection: remove NO_DATA regions after wide background subtraction    The large NO_DATA regions at the edge of the field-of-view can cause  large detections after doing the wide background subtraction because  of extrapolation of the background model into areas with no constraints.    We save the original image so as to restore it later.    python/lsst/meas/algorithms/detection.py | 16 +++++++++++++---  1 file changed, 13 insertions(+), 3 deletions(-)       price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=)$ git sub commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price  Date: Thu Feb 15 16:05:18 2018 -0500    warpCompare: tweak config for better clipping of scattered light    Some HSC fields contain significant scattered light, which somehow makes  the warp variance rescaling inaccurate, which in turn allows the scattered  light leak through into the coadd. Turned this off. Yusra notes:    The benefits of doScaleWarpVariance seemed minimal compared to  calcErrorFromInputVariance=False, and with doPreserveContainedBySource  it’s less important anyway    Decreasing the clipping threshold for the template creation also helps  keep (at least some) nasty features out of the template, and hence lets  us pick them up when we clip. This value was previously 1.5, recently  changed to 5 (too low and too high), so now set to 2.5.    python/lsst/pipe/tasks/assembleCoadd.py | 4 ++--  1 file changed, 2 insertions(+), 2 deletions(-)   commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price  Date: Thu Feb 15 16:11:53 2018 -0500    detectCoaddSources: suppress large footprints    Using a new option in the SourceDetectionTask, supress large footprints  that can overwhelm the deblender.  {    python/lsst/pipe/tasks/multiBand.py | 1 +  1 file changed, 1 insertion(+) 

Show
Paul Price added a comment - - edited price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $git sub ]commit 139bfe040fe65388ea92101340069ef3a43e38f8 Author: Paul Price <price@astro.princeton.edu> Date: Wed Feb 14 15:18:47 2018 -0500 detection: add option to grow footprints independently Growing footprints collectively allows the footprints to merge, which may allow nearby sources to chain together, creating a large parent object which might overwhelm the deblender. Growing footprints individually allows the footprints to overlap, but they won't merge. python/lsst/meas/algorithms/detection.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) commit 726d0a50a2a94f169893c9050aa7093114bdc829 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 12:19:02 2018 -0500 detection: add option to subtract temporary wide background Subtracts a wide (large-scale) background before the detection pass proper. This background is added back to the image after detection. The purpose is to suppress large footprints (e.g., from large artifacts, scattered light) that the deblender may choke on. python/lsst/meas/algorithms/detection.py | 85 +++++++++++++++++++++---- python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++---------- 2 files changed, 106 insertions(+), 48 deletions(-) commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 14:15:42 2018 -0500 detection: remove NO_DATA regions after wide background subtraction The large NO_DATA regions at the edge of the field-of-view can cause large detections after doing the wide background subtraction because of extrapolation of the background model into areas with no constraints. We save the original image so as to restore it later. python/lsst/meas/algorithms/detection.py | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=)$ git sub commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 16:05:18 2018 -0500   warpCompare: tweak config for better clipping of scattered light Some HSC fields contain significant scattered light, which somehow makes the warp variance rescaling inaccurate, which in turn allows the scattered light leak through into the coadd. Turned this off. Yusra notes: The benefits of doScaleWarpVariance seemed minimal compared to calcErrorFromInputVariance=False, and with doPreserveContainedBySource it’s less important anyway Decreasing the clipping threshold for the template creation also helps keep (at least some) nasty features out of the template, and hence lets us pick them up when we clip. This value was previously 1.5, recently changed to 5 (too low and too high), so now set to 2.5.   python/lsst/pipe/tasks/assembleCoadd.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553) Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 15 16:11:53 2018 -0500   detectCoaddSources: suppress large footprints Using a new option in the SourceDetectionTask, supress large footprints that can overwhelm the deblender. {   python/lsst/pipe/tasks/multiBand.py | 1 + 1 file changed, 1 insertion(+)
Hide
Paul Price added a comment -

Jenkins is green.

Show
Paul Price added a comment - Jenkins is green.
Hide

I took a look and made some comments in github, mostly with respect to the variance scaling. Comments are attached to the commits.

Show
Yusra AlSayyad added a comment - - edited I took a look and made some comments in github, mostly with respect to the variance scaling. Comments are attached to the commits.
Hide
Paul Price added a comment -

Reran Jenkins after rebasing, and it's green.

Show
Paul Price added a comment - Reran Jenkins after rebasing, and it's green.
Hide
Paul Price added a comment -

Merged to master.

Show
Paul Price added a comment - Merged to master.

#### People

Assignee:
Paul Price
Reporter:
Paul Price
Watchers:
Hisanori Furusawa, Hsin-Fang Chiang, Jim Bosch, Masayuki Tanaka, Paul Price, Robert Lupton, Yusra AlSayyad