Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34623 AP Performance sprint
  3. DM-34882

Try profiling ap_verify with pyinstrument

    XMLWordPrintable

    Details

    • Type: Technical task
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      AP S22-6 (May)
    • Team:
      Alert Production

      Description

      Eric Bellm suggested using pyinstrument to get profiling information that does not have the timing decorators causing confusing call graphs. It looks like there are quite a few other statistical profilers available (e.g. this stack overflow discussion), so if pyinstrument doesn't work, we could try another one.

      A possible output of this ticket would be a new section added to the dev guide python profiling section. Speaking of which, that could probably use a paragraph on snakeviz, and some updated notes on gprof2dot: there's a single sentence, and a reference to another tool (runsnakerun) that was last updated in 2011...

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          Parejkoj John Parejko added a comment -

          I was able to run pyinstrument, and played around with some of its options. Some comments:

          • I don't find the "-r html" output very useful: it's just text, and with the number of redirections in our call stack, it's tough to interpret.
          • It seems to take about 10-20% longer to run, compared with just turning profiling on.
          • You can't use both `-t` and `-r speedscope` together: it raises an exception when trying to write the output.
          • Putting the Speedscope output into https://www.speedscope.app results in a better flame graph than what snakeviz produces without DM-34881, but with DM-34881 (timer disabled) the standard profiler works fine, I think. I've attached a speedscope flamegraph: mouseover gives more info about each box, but I don't know that we'll learn much more from this, compared with the normal profiler.

          So, I think my preference is to just run with the timer disabled via my DM-34881 hack, since it cleans up the call graph in a way that we'd want anyway.

          It might be worth adding a couple of sentences to the dev guide section about profilers about it, so others can be aware of the option?

          Show
          Parejkoj John Parejko added a comment - I was able to run pyinstrument, and played around with some of its options. Some comments: I don't find the "-r html" output very useful: it's just text, and with the number of redirections in our call stack, it's tough to interpret. It seems to take about 10-20% longer to run, compared with just turning profiling on. You can't use both `-t` and `-r speedscope` together: it raises an exception when trying to write the output. Putting the Speedscope output into https://www.speedscope.app results in a better flame graph than what snakeviz produces without DM-34881 , but with DM-34881 (timer disabled) the standard profiler works fine, I think. I've attached a speedscope flamegraph: mouseover gives more info about each box, but I don't know that we'll learn much more from this, compared with the normal profiler. So, I think my preference is to just run with the timer disabled via my DM-34881 hack, since it cleans up the call graph in a way that we'd want anyway. It might be worth adding a couple of sentences to the dev guide section about profilers about it, so others can be aware of the option?
          Hide
          Permalink
          Parejkoj John Parejko added a comment -

          Eric Bellm: Can you please review the dev guide PR? I made several changes to the python profiling docs, which should make them more useful to others.

          The rendered output is here: https://developer.lsst.io/v/DM-34882/python/profiling.html

          Show
          Parejkoj John Parejko added a comment - Eric Bellm : Can you please review the dev guide PR? I made several changes to the python profiling docs, which should make them more useful to others. The rendered output is here: https://developer.lsst.io/v/DM-34882/python/profiling.html
          Hide
          Permalink
          ebellm Eric Bellm added a comment -

          Looks good!

          Show
          ebellm Eric Bellm added a comment - Looks good!

            People

            Assignee:
            Parejkoj John Parejko
            Reporter:
            Parejkoj John Parejko
            Reviewers:
            Eric Bellm
            Watchers:
            Eric Bellm, John Parejko, Meredith Rawls
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.