Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33001

Design refactoring of ImageDifferenceTask

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      24
    • Sprint:
      AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May), AP F22-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      John and I sketched out some thoughts on a whiteboard on how we might refactor ImageDifferenceTask in a Gen 3-only environment. This ticket is to write a prototype new implementation to bring to a larger group for discussion. Since the new design would change the API, we would also expect to file an RFC.

      General plan:
      ImageDifferenceTask should be divided into several PipelineTasks:

      1. makeTemplate (input: calexp, skymap, coadds; output: warpedTemplate)
        1. warpPatches
        2. assembleWarps
      2. subtraction (A&L, input: calexp, warpedTemplate; output: differenceExp)
        1. makeKernel (possibly now a configurable Task)
        2. convolve image (template or science)
        3. subtraction
        4. decorrelation
      3. subtraction with preconvolution (A&L, input: calexp, warpedTemplate; output: likelihoodExp)
        1. convolve science
        2. makeKernel
        3. convolveTemplate
        4. subtraction
        5. decorrelation
      4. detection and measurement (input: calexp, warpedTemplate, likelihoodExp, output: diaSources)
        1. detection
        2. measurement
        3. dipole measurement
        4. aperture correction

        Attachments

        1. NewDiffim.yaml
          0.5 kB
        2. OldDiffim.yaml
          0.7 kB

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment - - edited

            We have a working new image differencing pipeline, which is now in ip.diffim. The new pipeline uses DM-33745 to create the warped template, and then performs image subtraction and detection and measurement of diaSources as separate PipelineTasks. The results are the same as the previous ImageDifferenceTask under a few conditions and with the following execptions:

            1. The template exposure has a BBox that is grown by several pixels. However, current image differencing treats that as an unmatched WCS and warps and crops the template. This results in small differences in the image and variance, and NaN pixels around the edges after convolving with the PSF. If both current and new image differencing are modified to trim the BBox and not warp the template, the PSF-matched template is identical.
            2. With identical PSF-matched templates, the difference images immediately after image subtraction are not identical. The mask and variance planes are pixel-wise identical, but the image plane has 10^-8^ level random pixel differences.
            3. Even with pixel-wise identical difference images loaded from temporary files, the result of ScaleVarianceTask is different at machine precision. We have not resolved this difference, but believe it is not significant.
            4. The differences from ScaleVarianceTask result in differences in the source catalogs at the ~10^-13^ level in RA and Dec, and most flux fields. We have verified that these differences disappear if the difference image is identical after scaling the diffim variance.
            5. Current image differencing performs an additional source detection step, while the new version reads in the existing calexp source catalog. This results in a slightly different selection of sources for calculating the matching kernel, but the source detection step of the current code can be enabled with a config option.
            6. We have only attempted to maintain functionality and equivalent results for the case where the Alard&Lupton algorithm is used, the template has better seeing than the science image, and the template is convolved. The new code provides functionality for pre-convolution and convolving the science image, but we are not attempting to compare those results.
            7. The new image differencing does not output a warpedExp, since the templateExp is equivalent except for a slightly larger BBox.
            Show
            sullivan Ian Sullivan added a comment - - edited We have a working new image differencing pipeline, which is now in ip.diffim . The new pipeline uses DM-33745 to create the warped template, and then performs image subtraction and detection and measurement of diaSources as separate PipelineTasks. The results are the same as the previous ImageDifferenceTask under a few conditions and with the following execptions: The template exposure has a BBox that is grown by several pixels. However, current image differencing treats that as an unmatched WCS and warps and crops the template. This results in small differences in the image and variance, and NaN pixels around the edges after convolving with the PSF. If both current and new image differencing are modified to trim the BBox and not warp the template, the PSF-matched template is identical. With identical PSF-matched templates, the difference images immediately after image subtraction are not identical. The mask and variance planes are pixel-wise identical, but the image plane has 10^-8^ level random pixel differences. Even with pixel-wise identical difference images loaded from temporary files, the result of ScaleVarianceTask is different at machine precision. We have not resolved this difference, but believe it is not significant. The differences from ScaleVarianceTask result in differences in the source catalogs at the ~10^-13^ level in RA and Dec, and most flux fields. We have verified that these differences disappear if the difference image is identical after scaling the diffim variance. Current image differencing performs an additional source detection step, while the new version reads in the existing calexp source catalog. This results in a slightly different selection of sources for calculating the matching kernel, but the source detection step of the current code can be enabled with a config option. We have only attempted to maintain functionality and equivalent results for the case where the Alard&Lupton algorithm is used, the template has better seeing than the science image, and the template is convolved. The new code provides functionality for pre-convolution and convolving the science image, but we are not attempting to compare those results. The new image differencing does not output a warpedExp , since the templateExp is equivalent except for a slightly larger BBox.
            Hide
            sullivan Ian Sullivan added a comment - - edited

            Please just review the code in ip_diffim and the two notebooks in ap_pipe-noteboks. The changes on pipe_tasks will not be merged.

            Show
            sullivan Ian Sullivan added a comment - - edited Please just review the code in ip_diffim and the two notebooks in ap_pipe-noteboks . The changes on pipe_tasks will not be merged.
            Hide
            sullivan Ian Sullivan added a comment -

            I made a lot of changes and added several more tests. Kenneth Herner, John Parejko please let me know if either of you want to take another look, and mark this ticket as 'Reviewed' when you are happy. Otherwise I think I am ready to squash the commits and merge.

            Show
            sullivan Ian Sullivan added a comment - I made a lot of changes and added several more tests. Kenneth Herner , John Parejko please let me know if either of you want to take another look, and mark this ticket as 'Reviewed' when you are happy. Otherwise I think I am ready to squash the commits and merge.
            Hide
            Parejkoj John Parejko added a comment -

            Ken and I have reviewed the tickets, and had a final closeout discussion today for lingering issues.

            For Ian Sullivan: I left a few more review comments during our discussion. Please do a doc build and look over the output to confirm that the new class/method descriptions look ok. Also, please squash the commits to have one for the subtractTask and tests, and one for the detectAndMeasureTask (tests TBD on a ticket that should be linked from here?).

            Show
            Parejkoj John Parejko added a comment - Ken and I have reviewed the tickets, and had a final closeout discussion today for lingering issues. For Ian Sullivan : I left a few more review comments during our discussion. Please do a doc build and look over the output to confirm that the new class/method descriptions look ok. Also, please squash the commits to have one for the subtractTask and tests, and one for the detectAndMeasureTask (tests TBD on a ticket that should be linked from here?).
            Hide
            sullivan Ian Sullivan added a comment -

            Example pipelines attached for running the NewDiffim.yaml compared to OldDiffim.yaml.

            Show
            sullivan Ian Sullivan added a comment - Example pipelines attached for running the NewDiffim.yaml compared to OldDiffim.yaml .

              People

              Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Kenneth Herner
              Watchers:
              Ian Sullivan, John Parejko, Kenneth Herner, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.