# Change background handling defaults in imageDifference.py

XMLWordPrintable

## Details

• Type: RFC
• Status: Implemented
• Resolution: Done
• Component/s:
• Labels:
None

## Description

Image differencing tests with DECam and HSC have identified problems with background subtraction using the default ip_diffim configs (DM-20555, DM-20484).

While investigation of these issues is ongoing, the performance improvements from adjusted configuration parameters are notable: ~15% reduction in false positives in DECam, and even larger in HSC. Accordingly we suggest changing the defaults.

If accepted, this RFC would change to the following default configurations in imageDifference.py:

 config.differencer.subtract['al'].kernel.active.spatialBgOrder = 2 config.differencer.doAddCalexpBackground = False 

The current defaults are 0 and True.

## Activity

Hide
Meredith Rawls added a comment -

Yes please. While adding the calexp background back in should improve difference imaging results, in practice it doesn't right now. Changing these ip_diffim defaults to something that yields good results seems very sensible.

Show
Meredith Rawls added a comment - Yes please. While adding the calexp background back in should improve difference imaging results, in practice it doesn't right now. Changing these ip_diffim defaults to something that yields good results seems very sensible.
Hide
Zeljko Ivezic added a comment -

It's good that these config improve the performance but do you understand why? E.g. I could guess that the higher spatial order provides a better fit, but am not sure why one has freedom to add or not add background. Do you have reasons to expect that these improvements should be general and independent of dataset?

Show
Zeljko Ivezic added a comment - It's good that these config improve the performance but do you understand why? E.g. I could guess that the higher spatial order provides a better fit, but am not sure why one has freedom to add or not add background. Do you have reasons to expect that these improvements should be general and independent of dataset?
Hide
Meredith Rawls added a comment -

As I understand it, the higher spatial order fit for the difference image background is instead of using the calexp background directly. The default of 0 would be a constant background which is not an accurate representation of reality. We have shown that turning doAddCalexpBackground off significantly reduces the number of false positive sources in both HSC and DECam, but aren't sure why adding in the calexp background to the difference images doesn't work as originally intended.

Show
Meredith Rawls added a comment - As I understand it, the higher spatial order fit for the difference image background is instead of  using the calexp background directly. The default of 0 would be a constant background which is not an accurate representation of reality. We have shown that turning doAddCalexpBackground off significantly reduces the number of false positive sources in both HSC and DECam, but aren't sure why adding in the calexp background to the difference images doesn't work as originally intended.
Hide
Eric Bellm added a comment -

Zeljko Ivezic doAddCalexpBackground "undoes" the background subtraction performed by single-frame processing, so all of the background fitting is done by ip_diffim. I am told that historically SFP background fitting was worse and it made sense to redo the subtraction in ip_diffim, but that appears to no longer be true. Going forward I would continue to expect SFP to provide our best estimate of the background. I'm also not aware of significant other datasets using diffim at present except for DESC DC2, whom I will inform.

Show
Eric Bellm added a comment - Zeljko Ivezic doAddCalexpBackground "undoes" the background subtraction performed by single-frame processing, so all of the background fitting is done by ip_diffim . I am told that historically SFP background fitting was worse and it made sense to redo the subtraction in ip_diffim , but that appears to no longer be true. Going forward I would continue to expect SFP to provide our best estimate of the background. I'm also not aware of significant other datasets using diffim at present except for DESC DC2, whom I will inform.
Hide
Zeljko Ivezic added a comment -

Show
Hide
Eric Bellm added a comment -
Show
Eric Bellm added a comment - Implementation ticket is https://jira.lsstcorp.org/browse/DM-21398

## People

• Assignee:
Eric Bellm
Reporter:
Eric Bellm
Watchers:
Eric Bellm, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Zeljko Ivezic