Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:
Description
Our current ImageDifferenceTask in pipe_tasks was written long before the Gen 3 middleware was developed, and is a monolithic block of code that attempts to support a wide variety of use cases. Most of those use cases are untested and contain heavily bit-rotted code, and we have found it difficult to debug and extend to new use cases. On DM-33001 we provided a proposed new implementation of image differencing in ip_diffim which now consists of three distinct PipelineTasks:
- GetTemplateTask which constructs the warped and cropped (and possibly DCR-corrected) templateExp. This step was implemented previously as part of ImageDifferenceFromTemplateTask.
- AlardLuptonSubtractTask which PSF-matches either the template or science image, and subtracts the template from the science image. This Task can be replaced by other algorithms, such as a (yet to be written) ZogySubtractTask with the appropriate input and output connections.
- DetectAndMeasureTask which runs source detection and measurement on the difference image, including adding sky sources and forced measurement. For a future ZogySubtractTask or other image subtraction Task that produces a maximum likelihood image as an output connection we can create a new DetectAndMeasureLikelihoodTask that inherits from DetectAndMeasureTask.
As documented on DM-33001, each step of the new image differencing pipeline returns results that are within floating point precision of the old ImageDifferenceTask if forceCompatibility=True for AlardLuptonSubtractTask, except that the results of ScaleVarianceTask appear sensitive to these small differences. With compatibility mode turned off, the differences are:
- ScaleVarianceTask is applied separately to the science and template images, rather than once to the difference image. This is necessary to ensure that the relative weight of the science and template variances is correct before they are combined.
- pipe_tasks.ImageDifferenceTask has a minor bug that ignores the border pixels of the template and performs an unnecessary extra warping step. This behavior is duplicated in compatibility mode.
- The image to be convolved will be programmatically determined based on the seeing of the science and template images.
For this RFC, I am proposing that we transition to using the new code in stages:
- Immediately switch the AP pipeline to the new tasks, using forceCompatibility=True.
- After the current RC2 run of DRP is completed, also switch the DRP pipeline to the new tasks, using forceCompatibility=True.
- Add deprecation warnings to ImageDifferenceTask in pipe_tasks.
- Turn off compatibility mode for the AP pipeline.
- Turn off compatibility mode for the DRP pipeline.
- Remove deprecated code in pipe_tasks and ip_diffim.
Attachments
Issue Links
- is triggering
-
- Done
- relates to
-
- Done
-
- Done
-
- Done
Activity
I'd question whether we want forceCompatibility=True as the pipeline default. Given your remarks about ScaleVarianceTask, won't this simply mean we have to deal with two transitions rather than just one?
Krzysztof Findeisen
added a comment - I'd question whether we want forceCompatibility=True as the pipeline default. Given your remarks about ScaleVarianceTask , won't this simply mean we have to deal with two transitions rather than just one? DMCCB are fine with this migration and deprecation. We have no opinion as to whether a two stage deprecation period should be adopted or not.
I'm strongly in favor of the switch to the new one, and happy to defer to AP on exactly how to manage the deprecation and removal of the old one.
I'm obviously biased since I helped build the new code, but I think this is going to be a massive step up in our ability to validate, profile, and fix our difference imaging code.
I've argued that we should start with forceCompatibility=True because the "convolve science" part of the pipeline is not as well tested. Switching to the subtraction code in compatibility mode should give us a period where our metrics are mostly unchanged, while we finish validating the convolve science case and improve tests of the detectAndMeasure code.
I'm confused. Does forceCompatibility bypass "convolve science"? What will happen if we don't use it while you're finishing up that case?
Krzysztof Findeisen
added a comment - I'm confused. Does forceCompatibility bypass "convolve science"? What will happen if we don't use it while you're finishing up that case? Yes, forceCompatibility always convolves the template, like the current ImageDifferenceTask. We can continue testing convolving the science image separately, while keeping our regular runs reliable in compatibility mode. If we're willing to potentially have some wonky runs before convolve science is stable, we could turn off compatibility mode, but I'd prefer a period of stability.
I am excited to see the more modular ImageDifference as our stack default as I think it will speed our development and debugging.