Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33001

Design refactoring of ImageDifferenceTask

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      24
    • Epic Link:
    • Sprint:
      AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May), AP F22-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      John and I sketched out some thoughts on a whiteboard on how we might refactor ImageDifferenceTask in a Gen 3-only environment. This ticket is to write a prototype new implementation to bring to a larger group for discussion. Since the new design would change the API, we would also expect to file an RFC.

      General plan:
      ImageDifferenceTask should be divided into several PipelineTasks:

      1. makeTemplate (input: calexp, skymap, coadds; output: warpedTemplate)
        1. warpPatches
        2. assembleWarps
      2. subtraction (A&L, input: calexp, warpedTemplate; output: differenceExp)
        1. makeKernel (possibly now a configurable Task)
        2. convolve image (template or science)
        3. subtraction
        4. decorrelation
      3. subtraction with preconvolution (A&L, input: calexp, warpedTemplate; output: likelihoodExp)
        1. convolve science
        2. makeKernel
        3. convolveTemplate
        4. subtraction
        5. decorrelation
      4. detection and measurement (input: calexp, warpedTemplate, likelihoodExp, output: diaSources)
        1. detection
        2. measurement
        3. dipole measurement
        4. aperture correction

        Attachments

        1. NewDiffim.yaml
          0.5 kB
        2. OldDiffim.yaml
          0.7 kB

          Issue Links

          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34801 Replace Diff_warpedExp with Diff_templateExp in diaPipe

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35068 New ip_diffim algorithm breaks on macOS

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-34811 Quantum graph ordering for component datasets

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33861 Process two tracts of DC2 with refactored Image Differencing

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34910 Test DCR correction with new image differencing

          • In Progress

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33391 Assist in implementation of new image differencing Task

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33745 Refactor Task to make ImageDifference template

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34217 Move ScaleVarianceTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34266 Write unit tests for the refactored ImageDifferenceTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34364 Use PSF-matched template for image decorrelation

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34566 ip_diffim tests produce many warnings

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34786 Run ci_hits with the new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34830 Increase template border size

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34909 Add preconvolution option to new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35155 Write unit tests for DetectAndMeasureTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35175 Debug convolveScience option of new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35639 Switch AP and DRP pipelines to use new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-37612 Remove unit tests of deprecated code

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34008 Test the performance of difference imaging

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34787 Run ci_cosmos and ci_hits with the new image differencing

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35172 Run the diffim sprint dataset through the refactored imageDifference code

          • Invalid

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-866 Adopt new Image Differencing task

          • Implemented

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34628 Variance underestimated in patch overlaps in templates

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35156 Prepare RFC for adoption of new image difference Task

          • Done

            Activity

            No work has yet been logged on this issue.

              People

              Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Kenneth Herner
              Watchers:
              Ian Sullivan, John Parejko, Kenneth Herner, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.