Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33331

Monitor memory usage at select steps of BPS submission

    XMLWordPrintable

    Details

      Description

      ctrl_bps times different stages of the submission process to provide more detailed insight into its overall performance. The memory usage is another important metric and it would be beneficial to record it at select time points of submission process as well.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          mkowalik Mikolaj Kowalik added a comment -

          I'm asking Tim Jenness to be the main reviewer. However,  I'd appreciate if  Michelle Gower could also take a look, especially at the changes I made in  ctrl_bps.

          Show
          mkowalik Mikolaj Kowalik added a comment - I'm asking Tim Jenness to be the main reviewer. However,  I'd appreciate if  Michelle Gower could also take a look, especially at the changes I made in  ctrl_bps.
          Hide
          Permalink
          tjenness Tim Jenness added a comment -

          I've taken a look and the main comment is to move the memory utility out of bps and into utils. I'm not sure whether "current memory" being reported is going to be overly useful rather than peak usage during the block (for time_this).

          Show
          tjenness Tim Jenness added a comment - I've taken a look and the main comment is to move the memory utility out of bps and into utils. I'm not sure whether "current memory" being reported is going to be overly useful rather than peak usage during the block (for time_this).
          Hide
          Permalink
          tjenness Tim Jenness added a comment -

          Thanks for reorganizing. Minor comments on the PR.

          Show
          tjenness Tim Jenness added a comment - Thanks for reorganizing. Minor comments on the PR.
          Hide
          Permalink
          mgower Michelle Gower added a comment -

          Since Tim Jenness spent a lot of time looking at the util code, I focused on the bps changes.   Found a typo. Ran a pipelines_check test that worked fine (got to see child process numbers). Ran a prepare test loading one of the step1 reprocessing QuantumGraphs to see larger numbers (but no child processes).  If all of Tim's comments are taken care of and the typo is fixed, I approve the merge.

          Show
          mgower Michelle Gower added a comment - Since Tim Jenness  spent a lot of time looking at the util code, I focused on the bps changes.   Found a typo. Ran a pipelines_check test that worked fine (got to see child process numbers). Ran a prepare test loading one of the step1 reprocessing QuantumGraphs to see larger numbers (but no child processes).  If all of Tim's comments are taken care of and the typo is fixed, I approve the merge.

            People

            Assignee:
            mkowalik Mikolaj Kowalik
            Reporter:
            mkowalik Mikolaj Kowalik
            Reviewers:
            Michelle Gower
            Watchers:
            Michelle Gower, Mikolaj Kowalik, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.