Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9989

Incorporate ZOGY and A&L decorrelation option into imageDifference.py

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ip_diffim, pipe_tasks
    • Labels:
      None
    • Epic Name:
      Incorporate ZOGY and A&L decorrelation option into imageDifference.py
    • Story Points:
      40
    • Sprint:
      Alert Production S17 - 4, Alert Production S17 - 5
    • WBS:
      02C.03.04
    • Team:
      Alert Production
    • Cycle:
      Fall 2017

      Description

      Incorporate ZOGY and A&L decorrelation option into imageDifference.py. This is specifically regarding the spatially-varying variants of each, which rely upon the ImageMapReduce tasks. This epic includes fixing various bugs discovered after incorporation into imageDifference.py and running them on real images.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-3704 Refactor ImageDifferenceTask: Split into two tasks

          • Won't Fix
          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-9633 Create spatially varying Zogy and A&L decorrelation tasks

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-10806 Fix Psfs in spatially varying Zogy/decorrelation

          • Invalid

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-9441 Additional debugging of ZOGY and A&L(decorrelation)

          • Invalid

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          		DM-10805

          Spatially-varying ZOGY option

          		8 David Reiss Done
           
          		DM-10804

          Spatially-varying A&L decorrelation option

          		4 David Reiss Done
           
          		DM-10803

          Refactor grid generation in ImageMapReduceTask

          		4 David Reiss Done
           
          		DM-10009

          Ensure masks are valid from ImageMapReduceTask

          		2 David Reiss Done
           
          		DM-11813

          Summary of A&L Decorrelation and Zogy

          		12 David Reiss Done
           
          		DM-11951

          Fixes in AL Decorrelation and Zogy

          		6 David Reiss Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            reiss David Reiss added a comment - - edited

            Need to update ImageMapReduce to return 2 exposures (diffim and matched template, used for dipole fitting). This is currently not implemented (for spatially Zogy) but I added a check in imageDifference.py so that if it doesn't exist in the returned Struct, then it is not used.

            Show
            reiss David Reiss added a comment - - edited Need to update ImageMapReduce to return 2 exposures (diffim and matched template, used for dipole fitting). This is currently not implemented (for spatially Zogy) but I added a check in imageDifference.py so that if it doesn't exist in the returned Struct, then it is not used.
            Hide
            Permalink
            reiss David Reiss added a comment - - edited

            Zogy has been integrated into `imageDifference.py` in this ticket. The means of integration is not ideal, I admit, but this is due to the current organization of the `imageDifference.py` `ImageDifferenceTask.run` method itself. I decided not to refactor that method here, as that is subject to another ticket (DM-5295and DM-3704). After this ticket is merged, I will prioritize DM-5295 and DM-3704 to facilitate further improvements to the image differencing task.

            Both the constant-across-the-image and the spatially-varying versions of Zogy are available. By default the constant one is enabled (config `doSpatiallyVarying=False`), but after further testing this default could be changed.

            Show
            reiss David Reiss added a comment - - edited Zogy has been integrated into `imageDifference.py` in this ticket. The means of integration is not ideal, I admit, but this is due to the current organization of the `imageDifference.py` `ImageDifferenceTask.run` method itself. I decided not to refactor that method here, as that is subject to another ticket ( DM-5295 and DM-3704 ). After this ticket is merged, I will prioritize DM-5295 and DM-3704 to facilitate further improvements to the image differencing task. Both the constant-across-the-image and the spatially-varying versions of Zogy are available. By default the constant one is enabled (config `doSpatiallyVarying=False`), but after further testing this default could be changed.
            Hide
            Permalink
            reiss David Reiss added a comment - - edited

            For some reason, measurement is very slow when using the spatiallyVarying (ALdecorrelation and Zogy) options.
            Possibly due to use of the CoaddPsf for those resulting diffims. To be evaluated in DM-10806.

            Show
            reiss David Reiss added a comment - - edited For some reason, measurement is very slow when using the spatiallyVarying (ALdecorrelation and Zogy) options. Possibly due to use of the CoaddPsf for those resulting diffims. To be evaluated in DM-10806 .
            Hide
            Permalink
            reiss David Reiss added a comment - - edited

            Another issue identified: Zogy seems to lead to slightly too many positive (~20%) and way too many (~2x) negative detections. Need to investigate this. It is partly due to the artifacts around bright stars (see comment below) but also seen in vicinity of image edges.

            Show
            reiss David Reiss added a comment - - edited Another issue identified: Zogy seems to lead to slightly too many positive (~20%) and way too many (~2x) negative detections. Need to investigate this. It is partly due to the artifacts around bright stars (see comment below) but also seen in vicinity of image edges.
            Hide
            Permalink
            reiss David Reiss added a comment -

            Zogy diffim's have strange "fingerprint"-like artifacts around bright (but unsaturated) stars. Possibly a result of the sinc interpolation used for warping. This needs to be investigated.

            Show
            reiss David Reiss added a comment - Zogy diffim's have strange "fingerprint"-like artifacts around bright (but unsaturated) stars. Possibly a result of the sinc interpolation used for warping. This needs to be investigated.

              People

              Assignee:
              reiss David Reiss
              Reporter:
              reiss David Reiss
              Watchers:
              David Reiss
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.