Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20702

Create memory usage metric

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify, verify
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP F19-3 (Skipped)
    • Team:
      Alert Production

      Description

      Create a MetricTask that captures the MaxResidentSetSize from timeMethod. The task can be written by cargo-culting lsst.verify.tasks.TimingMetricTask. Add metrics for ApPipeTask and its direct subtasks to the default ap_verify config. Note that astropy supports data size units.

      One outstanding question is how to define aggregation (which is still the responsibility of the MetricTask for now). For running time across multiple CCDs a sum is the natural choice; it's less natural for memory usage because the aggregated data IDs weren't necessarily run together.

        Attachments

          Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-20970 timeMethod gives platform-dependent results

          • Done

            Activity

            Ascending order - Click to sort in descending order
            krzys Krzysztof Findeisen created issue -
            krzys Krzysztof Findeisen made changes -
            Field Original Value New Value
            Epic Link DM-19979 [ 307530 ]
            krzys Krzysztof Findeisen made changes -
            Description Create a {{MetricTask}} that captures the [{{MaxResidentSetSize}} from timeMethod|https://github.com/lsst/pipe_base/blob/master/python/lsst/pipe/base/timer.py#L100]. The task can be written by cargo-culting {{lsst.verify.tasks.TimingMetricTask}}.
            Add metrics for {{ApPipeTask}} and its direct subtasks to the default {{ap_verify}} config.
            Note that astropy supports [data size units|https://docs.astropy.org/en/stable/units/standard_units.html#prefixes].

            One outstanding question is how to define aggregation (which is still the responsibility of the {{MetricTask}} for now). For running time across multiple CCDs a sum is the natural choice; it's less natural for memory usage because the aggregated data IDs weren't necessarily run together.             		Create a {{MetricTask}} that captures the [{{MaxResidentSetSize}} from timeMethod|https://github.com/lsst/pipe_base/blob/master/python/lsst/pipe/base/timer.py#L100]. The task can be written by cargo-culting {{lsst.verify.tasks.TimingMetricTask}}. Add metrics for {{ApPipeTask}} and its direct subtasks to the default {{ap_verify}} config. Note that astropy supports [data size units|https://docs.astropy.org/en/stable/units/standard_units.html#prefixes].

            One outstanding question is how to define aggregation (which is still the responsibility of the {{MetricTask}} for now). For running time across multiple CCDs a sum is the natural choice; it's less natural for memory usage because the aggregated data IDs weren't necessarily run together.
            krzys Krzysztof Findeisen made changes -
            Sprint AP F19-3 [ 941 ]
            krzys Krzysztof Findeisen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            krzys Krzysztof Findeisen made changes -
            Reviewers Hsin-Fang Chiang [ hchiang2 ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-20970 [ DM-20970 ]
            krzys Krzysztof Findeisen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, John Parejko, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.