Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30202

Add "online" coaddition code to meas_algorithms and pipe_tasks

    XMLWordPrintable

    Details

      Description

      For DESC, I have built an "online coaddition" class that is a replacement for afw.statisticsStack written in pure python/numpy that can do mean and clipped mean stacks using a running sum. This results in significant memory savings vs statisticsStack and does not require reading sub-regions. Tests on ci_hsc show it is 15-20% faster than statisticsStack for mean coaddition (due to reduced overhead), but it is slower for clipped means (in a way that scales with number of images) due to the increased i/o.

      This code currently lives at https://github.com/lsst/pipe_tasks/tree/u/erykoff/onlinecoadd . The AccumulatorMeanStack class will be put in meas_algorithms, a more appropriate location. Changing to the online coaddition code in AssembleCoaddTask will be configured with a doOnlineCoaddition config option that will only apply to "MEAN" and "MEANCLIP" coadds.

        Attachments

          Activity

          Show
          erykoff Eli Rykoff added a comment - PRs are https://github.com/lsst/meas_algorithms/pull/251 and https://github.com/lsst/pipe_tasks/pull/551
          Hide
          erykoff Eli Rykoff added a comment -

          Also: upon further review, when working on the tests, I realized that the MEANCLIP option would not work without an online median statistic option, and this is not trivial. Therefore, this ticket is only for straight MEAN statistics.

          Show
          erykoff Eli Rykoff added a comment - Also: upon further review, when working on the tests, I realized that the MEANCLIP option would not work without an online median statistic option, and this is not trivial. Therefore, this ticket is only for straight MEAN statistics.

            People

            Assignee:
            erykoff Eli Rykoff
            Reporter:
            erykoff Eli Rykoff
            Reviewers:
            Arun Kannawadi
            Watchers:
            Arun Kannawadi, Eli Rykoff, John Parejko
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.