Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33001

Design refactoring of ImageDifferenceTask

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      24
    • Epic Link:
    • Sprint:
      AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May), AP F22-1 (June)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      John and I sketched out some thoughts on a whiteboard on how we might refactor ImageDifferenceTask in a Gen 3-only environment. This ticket is to write a prototype new implementation to bring to a larger group for discussion. Since the new design would change the API, we would also expect to file an RFC.

      General plan:
      ImageDifferenceTask should be divided into several PipelineTasks:

      1. makeTemplate (input: calexp, skymap, coadds; output: warpedTemplate)
        1. warpPatches
        2. assembleWarps
      2. subtraction (A&L, input: calexp, warpedTemplate; output: differenceExp)
        1. makeKernel (possibly now a configurable Task)
        2. convolve image (template or science)
        3. subtraction
        4. decorrelation
      3. subtraction with preconvolution (A&L, input: calexp, warpedTemplate; output: likelihoodExp)
        1. convolve science
        2. makeKernel
        3. convolveTemplate
        4. subtraction
        5. decorrelation
      4. detection and measurement (input: calexp, warpedTemplate, likelihoodExp, output: diaSources)
        1. detection
        2. measurement
        3. dipole measurement
        4. aperture correction

        Attachments

        1. NewDiffim.yaml
          0.5 kB
        2. OldDiffim.yaml
          0.7 kB

          Issue Links

          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34801 Replace Diff_warpedExp with Diff_templateExp in diaPipe

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35068 New ip_diffim algorithm breaks on macOS

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-34811 Quantum graph ordering for component datasets

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33861 Process two tracts of DC2 with refactored Image Differencing

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34910 Test DCR correction with new image differencing

          • In Progress

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33391 Assist in implementation of new image differencing Task

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33745 Refactor Task to make ImageDifference template

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34217 Move ScaleVarianceTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34266 Write unit tests for the refactored ImageDifferenceTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34364 Use PSF-matched template for image decorrelation

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34566 ip_diffim tests produce many warnings

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34786 Run ci_hits with the new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34830 Increase template border size

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34909 Add preconvolution option to new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35155 Write unit tests for DetectAndMeasureTask

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35175 Debug convolveScience option of new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35639 Switch AP and DRP pipelines to use new image differencing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-37612 Remove unit tests of deprecated code

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34008 Test the performance of difference imaging

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34787 Run ci_cosmos and ci_hits with the new image differencing

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35172 Run the diffim sprint dataset through the refactored imageDifference code

          • Invalid

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-866 Adopt new Image Differencing task

          • Implemented

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34628 Variance underestimated in patch overlaps in templates

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35156 Prepare RFC for adoption of new image difference Task

          • Done

            Activity

            Ascending order - Click to sort in descending order
            sullivan Ian Sullivan created issue -
            sullivan Ian Sullivan made changes -
            Field Original Value New Value
            Status To Do [ 10001 ] In Progress [ 3 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-33391 [ DM-33391 ]
            sullivan Ian Sullivan made changes -
            Sprint AP S22-1 (December) [ 1126 ] AP S22-3 (February) [ 1142 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-33745 [ DM-33745 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-33861 [ DM-33861 ]
            sullivan Ian Sullivan made changes -
            Sprint AP S22-3 (February) [ 1142 ] AP S22-3 (February), AP S22-4 (March) [ 1142, 1148 ]
            sullivan Ian Sullivan made changes -
            Epic Link DM-30504 [ 510170 ] DM-30505 [ 510171 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34008 [ DM-34008 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34217 [ DM-34217 ]
            sullivan Ian Sullivan made changes -
            Sprint AP S22-3 (February), AP S22-4 (March) [ 1142, 1148 ] AP S22-3 (February), AP S22-4 (March), AP S22-5 (April) [ 1142, 1148, 1156 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34364 [ DM-34364 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-34266 [ DM-34266 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-34566 [ DM-34566 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34566 [ DM-34566 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34566 [ DM-34566 ]
            sullivan Ian Sullivan made changes -
            Sprint AP S22-3 (February), AP S22-4 (March), AP S22-5 (April) [ 1142, 1148, 1156 ] AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May) [ 1142, 1148, 1156, 1161 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34786 [ DM-34786 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34787 [ DM-34787 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-34801 [ DM-34801 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-34811 [ DM-34811 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34830 [ DM-34830 ]
            sullivan Ian Sullivan made changes -
            Reviewers Kenneth Herner [ kherner ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34909 [ DM-34909 ]
            sullivan Ian Sullivan made changes -
            Story Points 8 24
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34910 [ DM-34910 ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-35068 [ DM-35068 ]
            sullivan Ian Sullivan made changes -
            Sprint AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May) [ 1142, 1148, 1156, 1161 ] AP S22-3 (February), AP S22-4 (March), AP S22-5 (April), AP S22-6 (May), AP F22-1 (June) [ 1142, 1148, 1156, 1161, 1166 ]
            sullivan Ian Sullivan made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            sullivan Ian Sullivan made changes -
            Attachment OldDiffim.yaml [ 60336 ]
            Attachment NewDiffim.yaml [ 60337 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-35155 [ DM-35155 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-35156 [ DM-35156 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-35172 [ DM-35172 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-35175 [ DM-35175 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to RFC-866 [ RFC-866 ]
            sullivan Ian Sullivan made changes -
            Epic Link DM-30505 [ 510171 ] DM-34935 [ 1598553 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-35639 [ DM-35639 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-34628 [ DM-34628 ]
            sullivan Ian Sullivan made changes -
            Link This issue relates to DM-37612 [ DM-37612 ]

              People

              Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Kenneth Herner
              Watchers:
              Ian Sullivan, John Parejko, Kenneth Herner, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.