Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28885

makeWarp is a memory hog for large numbers of visits

    XMLWordPrintable

Details

    • No

    Description

      When running a pipetask that includes ISR, characterize, calibrate, and makeWarp, the last step appears to use way more memory than it ought to before crashing.

      I encountered this when I fed pipetask on order 100 visits in 5 filters, i.e. some 70,000 quanta, and naively ran it with pipetask run -j 16 on lsst-devl01. I checked back a day or two later and the run had died, so I restarted it (with extend-run, skip-existing, and skip-init-writes). It had more failed quanta the second time but enough success that I let it go. Then I got into the regime of diminishing returns. It thinks there are over 20,000 quanta left to process and the first warp it tries to make takes over 1000 seconds.

      Current thinking is that this happens on patches with maximal visit overlap and that makeWarpTask needs to be more selective in opening all the input visits. See Slack discussion for more: https://lsstc.slack.com/archives/C01FBUGM2CV/p1613608286458900 

      Attachments

        Issue Links

          Activity

            No work has yet been logged on this issue.

            People

              Unassigned Unassigned
              mrawls Meredith Rawls
              Brock Brendal [X] (Inactive), Ian Sullivan, James Chiang, Jim Bosch, John Parejko, Kenneth Herner, Meredith Rawls, Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.