Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10779

Implement running time metric(s)

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Sprint:
      Alert Production F17 - 7, Alert Production F17 - 8
    • Team:
      Alert Production

      Description

      By the All Hands Meeting, we should be able to demonstrate verification metrics by reporting running times (wall-clock) for AP pipeline stages. This ticket covers writing code for applying the verify framework to inline tests, profiling each stage, and supporting SQuaSH export (though the export itself is the responsibility of verify_ap).

      See DM-11118 for a list of pipeline stages. This ticket may report metrics at a finer-grained level than is presented there.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Eric Bellm, please take another look. I've renamed the files and removed the Pipeline class hierarchy.

            Show
            krzys Krzysztof Findeisen added a comment - Hi Eric Bellm , please take another look. I've renamed the files and removed the Pipeline class hierarchy.
            Hide
            ebellm Eric Bellm added a comment - - edited

            Hi Krzysztof Findeisen, I think this looks a lot better. I do wonder if we could minimize code duplication even further by using a decorator-style construction. Instead of writing a new wrapper function for each ap_pipe call, we could write a single function that takes the function to be wrapped as an argument. See the pseudo-code below. I think the key to whether this is useful is whether or not we can pass the differing call signatures in through the wrapping functions neatly.

            def metrics_decorator(callable, job, dataId=None):
               def wrapper(callable, job, **kwargs)
                   metadata = callable(**kwargs)
                   _update_metrics(metadata, job)
                   return metadata
               return wrapper
             
            _ingest_calibs = metrics_decorator(ap_pipe.ingest_calibs, job, ...)
             
            ...
            metadata.combine(_ingest_calibs(...))
            

            Let me know what you think.

            Show
            ebellm Eric Bellm added a comment - - edited Hi Krzysztof Findeisen , I think this looks a lot better. I do wonder if we could minimize code duplication even further by using a decorator-style construction. Instead of writing a new wrapper function for each ap_pipe call, we could write a single function that takes the function to be wrapped as an argument. See the pseudo-code below. I think the key to whether this is useful is whether or not we can pass the differing call signatures in through the wrapping functions neatly. def metrics_decorator( callable , job, dataId = None ): def wrapper( callable , job, * * kwargs) metadata = callable ( * * kwargs) _update_metrics(metadata, job) return metadata return wrapper   _ingest_calibs = metrics_decorator(ap_pipe.ingest_calibs, job, ...)   ... metadata.combine(_ingest_calibs(...)) Let me know what you think.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            I thought about using a decorator, but I couldn't think of an "obviously right" way to do it. (Also, they go beyond my knowledge of Python, and the official documentation is kind of vague about them.) Maybe leave it as-is for now, and create a 2-4 story point ticket for investigating and refactoring into decorators once we have an existing wrapper for ap_pipe?

            While I don't think there are any API translations that would invalidate the decorator approach (mainly we need to extract specific directories from a Dataset object and possibly append to them, which can be done in the same line), perhaps Meredith Rawls would like to weigh in here.

            Show
            krzys Krzysztof Findeisen added a comment - - edited I thought about using a decorator, but I couldn't think of an "obviously right" way to do it. (Also, they go beyond my knowledge of Python, and the official documentation is kind of vague about them.) Maybe leave it as-is for now, and create a 2-4 story point ticket for investigating and refactoring into decorators once we have an existing wrapper for ap_pipe? While I don't think there are any API translations that would invalidate the decorator approach (mainly we need to extract specific directories from a Dataset object and possibly append to them, which can be done in the same line), perhaps Meredith Rawls would like to weigh in here.
            Hide
            ebellm Eric Bellm added a comment -

            I like this idea. Closing this ticket and making a new one.

            Show
            ebellm Eric Bellm added a comment - I like this idea. Closing this ticket and making a new one.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Closed and merged. Thanks everyone for your feedback!

            Show
            krzys Krzysztof Findeisen added a comment - Closed and merged. Thanks everyone for your feedback!

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Eric Bellm
              Watchers:
              Colin Slater, David Reiss, Eric Bellm, Jonathan Sick, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: