Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20317

DCR model performance improvements

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP F19-1
    • Team:
      Alert Production

      Description

      While processing Decam data for DM-15340 I identified a few places in building the DCR model that could be made more efficient. These improvements speed up the calculation by 5 - 40%, depending on the properties of the input exposures.

        Attachments

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment -

            These two changes speed up creation of the DCR model. The first change calculates spline filtering for the DCR shifts ahead of time instead of repeatedly for each subfilter, which reduces the time to construct a typical model by ~9% (seen in the reduction in timing for spline_filter in the attached profiler outputs).

            The second change allows the splitSubfilters option that is needed for high airmass observations to be ignored for very low airmass observations. Shifting images takes ~80% of the time in building the DCR model, and this new option allows low airmass observations to use half the number of shifts.

            Show
            sullivan Ian Sullivan added a comment - These two changes speed up creation of the DCR model. The first change calculates spline filtering for the DCR shifts ahead of time instead of repeatedly for each subfilter, which reduces the time to construct a typical model by ~9% (seen in the reduction in timing for spline_filter in the attached profiler outputs). The second change allows the splitSubfilters option that is needed for high airmass observations to be ignored for very low airmass observations. Shifting images takes ~80% of the time in building the DCR model, and this new option allows low airmass observations to use half the number of shifts.
            Hide
            sullivan Ian Sullivan added a comment -

            Please note that this ticket includes two pull requests: one for ip_diffim and one for pipe_tasks, though due to a long-standing bug the pipe_tasks pull request might not show up.

            pipe_tasks: https://github.com/lsst/pipe_tasks/pull/300

            ip_diffim: https://github.com/lsst/ip_diffim/pull/127

            Show
            sullivan Ian Sullivan added a comment - Please note that this ticket includes two pull requests: one for ip_diffim and one for pipe_tasks , though due to a long-standing bug the pipe_tasks pull request might not show up. pipe_tasks : https://github.com/lsst/pipe_tasks/pull/300 ip_diffim : https://github.com/lsst/ip_diffim/pull/127
            Hide
            gkovacs Gabor Kovacs added a comment -

            Looks good. I suggest adding some info into the DcrAssembleCoaddTask class docstring about the subfilter splitting concept and mentioning the now introduced threshold speedup.

            (In the long term, I think that DMTN-037 itself can be updated with evolved implementation details from the class docstrings but this is beyond this ticket. e.g. the residual image loop, spline filtering of the residual image, splitting of the subfilters.)

            Show
            gkovacs Gabor Kovacs added a comment - Looks good. I suggest adding some info into the DcrAssembleCoaddTask class docstring about the subfilter splitting concept and mentioning the now introduced threshold speedup. (In the long term, I think that DMTN-037 itself can be updated with evolved implementation details from the class docstrings but this is beyond this ticket. e.g. the residual image loop, spline filtering of the residual image, splitting of the subfilters.)

              People

              • Assignee:
                sullivan Ian Sullivan
                Reporter:
                sullivan Ian Sullivan
                Reviewers:
                Gabor Kovacs
                Watchers:
                Gabor Kovacs, Ian Sullivan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel