Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20702

Create memory usage metric

    XMLWordPrintable

Details

    • Improvement
    • Status: Done
    • Resolution: Done
    • None
    • ap_verify, verify
    • None
    • 2
    • AP F19-3 (Skipped)
    • Alert Production

    Description

      Create a MetricTask that captures the MaxResidentSetSize from timeMethod. The task can be written by cargo-culting lsst.verify.tasks.TimingMetricTask. Add metrics for ApPipeTask and its direct subtasks to the default ap_verify config. Note that astropy supports data size units.

      One outstanding question is how to define aggregation (which is still the responsibility of the MetricTask for now). For running time across multiple CCDs a sum is the natural choice; it's less natural for memory usage because the aggregated data IDs weren't necessarily run together.

      Attachments

        Issue Links

          Activity

            One suggestion I've received is that the memory usage metric, when calculated over multiple visits/ccds/patches/etc., should give the maximum memory usage across units of work rather than the sum. The reasoning is that the most expensive run is the one that will actually limit the system.

            krzys Krzysztof Findeisen added a comment - One suggestion I've received is that the memory usage metric, when calculated over multiple visits/ccds/patches/etc., should give the maximum memory usage across units of work rather than the sum. The reasoning is that the most expensive run is the one that will actually limit the system.
            krzys Krzysztof Findeisen added a comment - - edited

            Testing shows that it's not useful to define this metric for subtasks; the "maximum" in MaxResidentSetSize is over an entire processing run, so you get output like:

                                ap_pipe.ApPipeMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'})
                         pipe_tasks.ProcessCcdMemory = 909975552.0 byte ({'estimator': 'pipe.base.timeMethod'})
                    pipe_tasks.ImageDifferenceMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'})
                    ap_association.AssociationMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'})
            

            I'll keep the metric definitions, in case they're useful when running e.g. ProcessCcdTask from the command line, but will only measure ApPipeMemory for our ap_verify runs.

            (Fortunately, it does track different processes separately.)

            krzys Krzysztof Findeisen added a comment - - edited Testing shows that it's not useful to define this metric for subtasks; the "maximum" in MaxResidentSetSize is over an entire processing run, so you get output like: ap_pipe.ApPipeMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'}) pipe_tasks.ProcessCcdMemory = 909975552.0 byte ({'estimator': 'pipe.base.timeMethod'}) pipe_tasks.ImageDifferenceMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'}) ap_association.AssociationMemory = 1954844672.0 byte ({'estimator': 'pipe.base.timeMethod'}) I'll keep the metric definitions, in case they're useful when running e.g. ProcessCcdTask from the command line, but will only measure ApPipeMemory for our ap_verify runs. (Fortunately, it does track different processes separately.)
            krzys Krzysztof Findeisen added a comment - - edited

            Hello hchiang2, would you be willing to review this ticket? It's about 350 lines across three packages. Thank you!

            krzys Krzysztof Findeisen added a comment - - edited Hello hchiang2 , would you be willing to review this ticket? It's about 350 lines across three packages. Thank you!

            I'm only learning about the verify framework as I review this, but overall it looks good to me. Thanks for adding this useful metric!

            hchiang2 Hsin-Fang Chiang added a comment - I'm only learning about the verify framework as I review this, but overall it looks good to me. Thanks for adding this useful metric!

            People

              krzys Krzysztof Findeisen
              krzys Krzysztof Findeisen
              Hsin-Fang Chiang
              Hsin-Fang Chiang, John Parejko, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.