Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17564

Simple efficiency improvements for DcrAssembleCoadd

    Details

    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP S19-3
    • Team:
      Alert Production

      Description

      The focus while developing the algorithm for correcting DCR in coadds and image differencing has been on the quality of the results, and not on efficiency. However, some components that were chosen for ease of initial development are unnecessarily wasteful, and could be sped up considerably with a small amount of effort. For example, the DcrModel that is passed to warpImage in each iteration is a maskedImage, but only the image plane is used even though the mask and variance planes are also warped. This ticket is to switch the DcrModel to use only the image plane throughout.

        Attachments

          Activity

          Hide
          sullivan Ian Sullivan added a comment -

          This turned out to be slightly more involved than I had originally thought, because simply switching the DcrModel to use an lsst.afw.image.Image instead of an lsst.afw.image.MaskedImage provided no speed increase. I found that at least 95% of the run time was due to using lsst.afw.math.WarpImage, so I switched the DCR shift to use numpy.ndimage.interpolation.shift, which is about 3X faster. That change had a cascading effect throughout DcrAssembleCoadd, since I needed to also replace lsst.afw.math.statisticsStack with coaddition code that could work with numpy arrays and handle masked pixels properly.

           

          This includes pull requests in both ip_diffim and pipe_tasks, though the latter doesn't show up in Jira:

          https://github.com/lsst/pipe_tasks/pull/272

          https://github.com/lsst/ip_diffim/pull/110

          Show
          sullivan Ian Sullivan added a comment - This turned out to be slightly more involved than I had originally thought, because simply switching the DcrModel to use an lsst.afw.image.Image instead of an lsst.afw.image.MaskedImage provided no speed increase. I found that at least 95% of the run time was due to using lsst.afw.math.WarpImage , so I switched the DCR shift to use numpy.ndimage.interpolation.shift , which is about 3X faster. That change had a cascading effect throughout DcrAssembleCoadd , since I needed to also replace lsst.afw.math.statisticsStack with coaddition code that could work with numpy arrays and handle masked pixels properly.   This includes pull requests in both ip_diffim and pipe_tasks , though the latter doesn't show up in Jira: https://github.com/lsst/pipe_tasks/pull/272 https://github.com/lsst/ip_diffim/pull/110
          Hide
          Parejkoj John Parejko added a comment -

          There's a couple of ongoing conversations but when those are done, you're good to merge.

          Show
          Parejkoj John Parejko added a comment - There's a couple of ongoing conversations but when those are done, you're good to merge.

            People

            • Assignee:
              sullivan Ian Sullivan
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              John Parejko
              Watchers:
              Ian Sullivan, John Parejko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel