Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_algorithms, pipe_tasks
-
Labels:
-
Team:External
Description
With the sky correction (DM-9147, DM-13388), perhaps combined with the lower detection threshold (DM-12295), we are getting many large parent objects (> 10^6 pixels), deblending which is a burden on our compute budget. In several cases, this appears to be due to scattered light surviving the coadd process.
Jim Bosch has proposed that we do a modest-scale background subtraction before detection, which should break up the large footprints; then we will restore the background before deblending so that the measurements have access to the original (good, large-scale) sky subtraction. There's a question about what scale to use in the background subtraction, which we can determine by running detection with different background subtraction scales, and seeing where we drop below the footprint size limits we've set in the deblender.
Jim also noted that after detection, we grow footprints by the size of the PSF in our standard detection routine. Instead of growing them all at once and allowing them to merge as we currently do, he suggests growing them individually, allowing the footprints to overlap but not merge. This might also help keep the large chains from forming.
Attachments
Issue Links
Activity
checkDeblending.py: script used to measure the number of large blends as a function of background subtraction scale.
Paul Price
added a comment - - edited checkDeblending.py : script used to measure the number of large blends as a function of background subtraction scale.
price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $ git sub
|
]commit 139bfe040fe65388ea92101340069ef3a43e38f8
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Wed Feb 14 15:18:47 2018 -0500
|
|
|
detection: add option to grow footprints independently
|
|
Growing footprints collectively allows the footprints to merge, which
|
may allow nearby sources to chain together, creating a large parent
|
object which might overwhelm the deblender.
|
|
Growing footprints individually allows the footprints to overlap, but
|
they won't merge.
|
|
|
python/lsst/meas/algorithms/detection.py | 12 +++++++++++-
|
1 file changed, 11 insertions(+), 1 deletion(-)
|
|
|
commit 726d0a50a2a94f169893c9050aa7093114bdc829
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Thu Feb 15 12:19:02 2018 -0500
|
|
|
detection: add option to subtract temporary wide background
|
|
Subtracts a wide (large-scale) background before the detection pass proper.
|
This background is added back to the image after detection. The purpose is
|
to suppress large footprints (e.g., from large artifacts, scattered light)
|
that the deblender may choke on.
|
|
|
python/lsst/meas/algorithms/detection.py | 85 +++++++++++++++++++++----
|
python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++----------
|
2 files changed, 106 insertions(+), 48 deletions(-)
|
|
|
commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Thu Feb 15 14:15:42 2018 -0500
|
|
|
detection: remove NO_DATA regions after wide background subtraction
|
|
The large NO_DATA regions at the edge of the field-of-view can cause
|
large detections after doing the wide background subtraction because
|
of extrapolation of the background model into areas with no constraints.
|
|
We save the original image so as to restore it later.
|
|
|
python/lsst/meas/algorithms/detection.py | 16 +++++++++++++---
|
1 file changed, 13 insertions(+), 3 deletions(-)
|
|
|
|
|
|
|
price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=) $ git sub
|
commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Thu Feb 15 16:05:18 2018 -0500
|
|
|
warpCompare: tweak config for better clipping of scattered light
|
|
Some HSC fields contain significant scattered light, which somehow makes
|
the warp variance rescaling inaccurate, which in turn allows the scattered
|
light leak through into the coadd. Turned this off. Yusra notes:
|
|
The benefits of doScaleWarpVariance seemed minimal compared to
|
calcErrorFromInputVariance=False, and with doPreserveContainedBySource
|
it’s less important anyway
|
|
Decreasing the clipping threshold for the template creation also helps
|
keep (at least some) nasty features out of the template, and hence lets
|
us pick them up when we clip. This value was previously 1.5, recently
|
changed to 5 (too low and too high), so now set to 2.5.
|
|
|
python/lsst/pipe/tasks/assembleCoadd.py | 4 ++--
|
1 file changed, 2 insertions(+), 2 deletions(-)
|
|
|
commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
|
Author: Paul Price <price@astro.princeton.edu>
|
Date: Thu Feb 15 16:11:53 2018 -0500
|
|
|
detectCoaddSources: suppress large footprints
|
|
Using a new option in the SourceDetectionTask, supress large footprints
|
that can overwhelm the deblender.
|
{
|
|
|
python/lsst/pipe/tasks/multiBand.py | 1 +
|
1 file changed, 1 insertion(+)
|
Paul Price
added a comment - - edited
price@pap-laptop:~/LSST/meas_algorithms (tickets/DM-13553=) $ git sub
]commit 139bfe040fe65388ea92101340069ef3a43e38f8
Author: Paul Price <price@astro.princeton.edu>
Date: Wed Feb 14 15:18:47 2018 -0500
detection: add option to grow footprints independently
Growing footprints collectively allows the footprints to merge, which
may allow nearby sources to chain together, creating a large parent
object which might overwhelm the deblender.
Growing footprints individually allows the footprints to overlap, but
they won't merge.
python/lsst/meas/algorithms/detection.py | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
commit 726d0a50a2a94f169893c9050aa7093114bdc829
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Feb 15 12:19:02 2018 -0500
detection: add option to subtract temporary wide background
Subtracts a wide (large-scale) background before the detection pass proper.
This background is added back to the image after detection. The purpose is
to suppress large footprints (e.g., from large artifacts, scattered light)
that the deblender may choke on.
python/lsst/meas/algorithms/detection.py | 85 +++++++++++++++++++++----
python/lsst/meas/algorithms/dynamicDetection.py | 69 ++++++++++----------
2 files changed, 106 insertions(+), 48 deletions(-)
commit 0278e2e4481847b14f86da30975cde458dd46086 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Feb 15 14:15:42 2018 -0500
detection: remove NO_DATA regions after wide background subtraction
The large NO_DATA regions at the edge of the field-of-view can cause
large detections after doing the wide background subtraction because
of extrapolation of the background model into areas with no constraints.
We save the original image so as to restore it later.
python/lsst/meas/algorithms/detection.py | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
price@pap-laptop:~/LSST/pipe_tasks (tickets/DM-13553=) $ git sub
commit fb46c0eae125c3440302298a79409cea5571fee3 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Feb 15 16:05:18 2018 -0500
warpCompare: tweak config for better clipping of scattered light
Some HSC fields contain significant scattered light, which somehow makes
the warp variance rescaling inaccurate, which in turn allows the scattered
light leak through into the coadd. Turned this off. Yusra notes:
The benefits of doScaleWarpVariance seemed minimal compared to
calcErrorFromInputVariance=False, and with doPreserveContainedBySource
it’s less important anyway
Decreasing the clipping threshold for the template creation also helps
keep (at least some) nasty features out of the template, and hence lets
us pick them up when we clip. This value was previously 1.5, recently
changed to 5 (too low and too high), so now set to 2.5.
python/lsst/pipe/tasks/assembleCoadd.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
commit c5c807f6aff1ddc1c6679285f95dabc614e36ed7 (HEAD -> tickets/DM-13553, origin/tickets/DM-13553)
Author: Paul Price <price@astro.princeton.edu>
Date: Thu Feb 15 16:11:53 2018 -0500
detectCoaddSources: suppress large footprints
Using a new option in the SourceDetectionTask, supress large footprints
that can overwhelm the deblender.
{
python/lsst/pipe/tasks/multiBand.py | 1 +
1 file changed, 1 insertion(+)
Jenkins is green.
Paul Price
added a comment - Jenkins is green. I took a look and made some comments in github, mostly with respect to the variance scaling. Comments are attached to the commits.
Reran Jenkins after rebasing, and it's green.
Paul Price
added a comment - Reran Jenkins after rebasing, and it's green. Merged to master.
Paul Price
added a comment - Merged to master. 
Using a background subtraction scale of 512 appears to remove the large footprints. Here's a list of filter, tract and a dict with the number of large footprints as a function of background scale:
('HSC-G', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5})('HSC-G', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})('HSC-G', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 1})('HSC-G', 9697, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 10})('HSC-G', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 4})('HSC-R', 8766, {64: 0, 128: 0, 1024: 2, 256: 0, 512: 0, 2048: 5})('HSC-R', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})('HSC-R', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1})('HSC-R', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 7})('HSC-R', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})('HSC-I', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 6})('HSC-I', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})('HSC-I', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})('HSC-I', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3})('HSC-I', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 3})('HSC-Z', 8766, {64: 0, 128: 0, 1024: 3, 256: 0, 512: 0, 2048: 9})('HSC-Z', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})('HSC-Z', 9615, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})('HSC-Z', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 2})('HSC-Z', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})('HSC-Y', 8766, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 5})('HSC-Y', 8767, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})('HSC-Y', 9615, {64: 0, 128: 0, 1024: 1, 256: 0, 512: 0, 2048: 2})('HSC-Y', 9697, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 1})('HSC-Y', 9813, {64: 0, 128: 0, 1024: 0, 256: 0, 512: 0, 2048: 0})This was calculated from the w_2018_06 weekly run (which includes the same features as hscPipe 6.0-beta5;
DM-13435), with thanks to Hsin-Fang Chiang for processing it.Note that when performing the background estimation it's important not to exclude DETECTED pixels, or the large areas of scattered light that are DETECTED (and hence chaining a bunch of detections together) won't be modified much.