# Incorporate ZOGY and A&L decorrelation option into imageDifference.py

XMLWordPrintable

#### Details

• Type: Epic
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Epic Name:
Incorporate ZOGY and A&L decorrelation option into imageDifference.py
• Story Points:
40
• Sprint:
• WBS:
02C.03.04
• Team:
• Cycle:
Fall 2017

#### Description

Incorporate ZOGY and A&L decorrelation option into imageDifference.py. This is specifically regarding the spatially-varying variants of each, which rely upon the ImageMapReduce tasks. This epic includes fixing various bugs discovered after incorporation into imageDifference.py and running them on real images.

#### Stories in Epic (Custom Issue Matrix)

Key Summary Story Points Assignee Status

DM-10009 2 David Reiss Done

DM-10805 8 David Reiss Done

DM-10804 4 David Reiss Done

DM-10803 4 David Reiss Done

DM-11951 6 David Reiss Done

DM-11813 12 David Reiss Done

#### Activity

No builds found.
David Reiss created issue -
Field Original Value New Value
Epic Link DM-9633 [ 30682 ]
 Sprint Alert Production S17 - 4 [ 610 ]
 Rank Ranked higher
 Sprint Alert Production S17 - 4 [ 610 ] Alert Production S17 - 4, Alert Production S17 - 5 [ 610, 613 ]
 Rank Ranked higher
 Status To Do [ 10001 ] In Progress [ 3 ]
 Link This issue is blocked by DM-3704 [ DM-3704 ]
 Comment [ Looks like the CoaddPsf generated for spatially varying Zogy is not going to work for detection in imageDifference.py. For example, computeShape() on the CoaddPsf, which is used for detection (meas_algorithms/python/lsst/meas/algorithms/detection.py, line 347) returns NaNs. I noticed that computeImage() also returns NaNs unless you supply the method with a coordinate that is contained by one of the bounding boxes that contributed to the CoaddPsf. ]
Hide
David Reiss added a comment - - edited

Need to update ImageMapReduce to return 2 exposures (diffim and matched template, used for dipole fitting). This is currently not implemented (for spatially Zogy) but I added a check in imageDifference.py so that if it doesn't exist in the returned Struct, then it is not used.

Show
David Reiss added a comment - - edited Need to update ImageMapReduce to return 2 exposures (diffim and matched template, used for dipole fitting). This is currently not implemented (for spatially Zogy) but I added a check in imageDifference.py so that if it doesn't exist in the returned Struct, then it is not used.
 Comment [ Exposure backgrounds are not subtracted when read in by butler in imageDifference.py. This is not an issue for A&L but is one for Zogy. I have added a small bit of code to subtract the sigma-clipped mean from the exposures prior to running Zogy. ]
Hide
David Reiss added a comment - - edited

Zogy has been integrated into imageDifference.py in this ticket. The means of integration is not ideal, I admit, but this is due to the current organization of the imageDifference.py ImageDifferenceTask.run method itself. I decided not to refactor that method here, as that is subject to another ticket (DM-5295and DM-3704). After this ticket is merged, I will prioritize DM-5295 and DM-3704 to facilitate further improvements to the image differencing task.

Both the constant-across-the-image and the spatially-varying versions of Zogy are available. By default the constant one is enabled (config doSpatiallyVarying=False), but after further testing this default could be changed.

Show
David Reiss added a comment - - edited Zogy has been integrated into imageDifference.py in this ticket. The means of integration is not ideal, I admit, but this is due to the current organization of the imageDifference.py ImageDifferenceTask.run method itself. I decided not to refactor that method here, as that is subject to another ticket ( DM-5295 and DM-3704 ). After this ticket is merged, I will prioritize DM-5295 and DM-3704 to facilitate further improvements to the image differencing task. Both the constant-across-the-image and the spatially-varying versions of Zogy are available. By default the constant one is enabled (config doSpatiallyVarying=False), but after further testing this default could be changed.
Hide
David Reiss added a comment - - edited

For some reason, measurement is very slow when using the spatiallyVarying (ALdecorrelation and Zogy) options.
Possibly due to use of the CoaddPsf for those resulting diffims. To be evaluated in DM-10806.

Show
David Reiss added a comment - - edited For some reason, measurement is very slow when using the spatiallyVarying (ALdecorrelation and Zogy) options. Possibly due to use of the CoaddPsf for those resulting diffims. To be evaluated in DM-10806 .
 Comment [ Spatially varying A&L decorrelation has also been added to imageDifference.py in this ticket. Needs to be updated as a subtask rather than creating the task "manually" in the method itself. This one also uses the config doSpatiallyVarying (default False). ]
 Comment [ This might be blocked by DM-3704. ]
 Sprint Alert Production S17 - 4, Alert Production S17 - 5 [ 610, 613 ] Alert Production S17 - 4, Alert Production S17 - 5, Alert Production S17 - 6 [ 610, 613, 616 ]
 Rank Ranked higher
Hide
David Reiss added a comment - - edited

Another issue identified: Zogy seems to lead to slightly too many positive (~20%) and way too many (~2x) negative detections. Need to investigate this. It is partly due to the artifacts around bright stars (see comment below) but also seen in vicinity of image edges.

Show
David Reiss added a comment - - edited Another issue identified: Zogy seems to lead to slightly too many positive (~20%) and way too many (~2x) negative detections. Need to investigate this. It is partly due to the artifacts around bright stars (see comment below) but also seen in vicinity of image edges.
Hide
David Reiss added a comment -

Zogy diffim's have strange "fingerprint"-like artifacts around bright (but unsaturated) stars. Possibly a result of the sinc interpolation used for warping. This needs to be investigated.

Show
David Reiss added a comment - Zogy diffim's have strange "fingerprint"-like artifacts around bright (but unsaturated) stars. Possibly a result of the sinc interpolation used for warping. This needs to be investigated.
 Issue Type Story [ 10001 ] Epic [ 10000 ]
 Epic Link DM-9633 [ 30682 ]
 Description Incorporate ZOGY and A&L decorrelation option into imageDifference.py
 Epic Child DM-10009 [ 31538 ]
 Epic Name Incorporate ZOGY and A&L decorrelation option into imageDifference.py
 Description Incorporate ZOGY and A&L decorrelation option into imageDifference.py Incorporate ZOGY and A&L decorrelation option into imageDifference.py. This is specifically regarding the spatially-varying variants of each, which rely upon the ImageMapReduce tasks. This epic includes fixing various bugs discovered after incorporation into imageDifference.py and running them on real images.
 Epic Child DM-10803 [ 32777 ]
 Epic Child DM-10804 [ 32778 ]
 Epic Child DM-10805 [ 32779 ]
 Link This issue relates to DM-9633 [ DM-9633 ]
 Link This issue relates to DM-10806 [ DM-10806 ]
 Link This issue relates to DM-9441 [ DM-9441 ]
 Sprint Alert Production S17 - 4, Alert Production S17 - 5, Alert Production S17 - 6 [ 610, 613, 616 ] Alert Production S17 - 4, Alert Production S17 - 5 [ 610, 613 ]
 Epic Child DM-11656 [ 34058 ]
 Epic Child DM-11813 [ 34306 ]
 Story Points 8 40
 Cycle Fall 2017 [ 10805 ] WBS 02C.03.04
 Epic Child DM-11951 [ 34526 ]
 Epic Child DM-11656 [ 34058 ]
 Epic Child DM-11995 [ 34596 ]
 Epic Child DM-11995 [ 34596 ]
 Resolution Done [ 10000 ] Status In Progress [ 3 ] Done [ 10002 ]
 Epic Status Done [ 10103 ]

#### People

Assignee:
David Reiss
Reporter:
David Reiss
Watchers:
David Reiss
0 Vote for this issue
Watchers:
1 Start watching this issue

#### Dates

Created:
Updated:
Resolved:

#### CI Builds

No builds found.