Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13547

Plot the node utilization of the w_2018_03 RC1 reprocessing

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Plot the node utilization versus time of the HSC RC1 reprocessing in DM-13463.
      Use Mikolaj Kowalik's existing code and improve it as needed/desired (e.g. do the plotting in python instead). Push the code to https://github.com/lsst-dm/ldf_ops_tools

      Example plots attached. 

      A future application of this is to plot the node utilization of a PDR1 run given the slurm job IDs, like this https://github.com/lsst-dm/DMTR-31/blob/master/figures/nodeUtilization.png

        Attachments

        1. time_vs_nnodes.png
          time_vs_nnodes.png
          39 kB
        2. usage-w_2017_26.png
          usage-w_2017_26.png
          43 kB
        3. usage-w_2017_28.png
          usage-w_2017_28.png
          44 kB
        4. usage-w_2017_30.png
          usage-w_2017_30.png
          45 kB
        5. usage-w_2017_32.png
          usage-w_2017_32.png
          45 kB

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13578 Plot the node utilization for RC1 Reprocessed Jobs

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13618 Find node-hour usage from usage codes

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13619 Modify usage.py for NODE_FAIL/COMPLETED failure case

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13622 Make usageplot.py output step plots

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13699 Modify usage.py and usageplot.py to allow for color-coded plots

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13783 Make color-coded node-usage plot for S17B HSC PDR1 reprocessing and find total node-hours

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13815 Find elapsed code times from usage.py/usageplot.py

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13816 Modify usage.py to allow the user to specify SLURM job names.

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13818 Modify usage.py to output node-hours that take into account significant figures

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13819 Create a top-level script to run both usage.py and usageplot.py

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14054 Add command line option for resolution in usage.py

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14111 Overhaul usage.py Readme file and delete key_len variable

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14288 Create node-hour usage plots for S18 HSC PDR1 reprocessing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13463 Reprocess RC1 with w_2018_03

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            sthrush Samantha Thrush added a comment -


            The plot is finally up to my standards, however, the code is still a bit sloppy. I will keep working on it and will hopefully be done with it tomorrow, at which point I will push the codes to github.

            Show
            sthrush Samantha Thrush added a comment - The plot is finally up to my standards, however, the code is still a bit sloppy. I will keep working on it and will hopefully be done with it tomorrow, at which point I will push the codes to github.
            Hide
            Permalink
            sthrush Samantha Thrush added a comment -

            The codes have been pushed to github, I have asked Mikolaj if he would be up for reviewing the code, and a pull request has been logged. 

            Show
            sthrush Samantha Thrush added a comment - The codes have been pushed to github, I have asked Mikolaj if he would be up for reviewing the code, and a pull request has been logged. 
            Hide
            Permalink
            sthrush Samantha Thrush added a comment - - edited

            I have addressed all of Mikolaj's comments.  However, I seem to be having a difficult time making it so that the original README is included in the "Add Mikolaj's Code" (git won't let me fixup any commits that have the old README to the "Add Mikolaj's Code" commit for some reason).  I currently have the modified README in the directory, but I'm still working on rebasing things so that the old README will be associated with the "Add Mikolaj's Code" commit.

            Show
            sthrush Samantha Thrush added a comment - - edited I have addressed all of Mikolaj's comments.  However, I seem to be having a difficult time making it so that the original README is included in the "Add Mikolaj's Code" (git won't let me fixup any commits that have the old README to the "Add Mikolaj's Code" commit for some reason).  I currently have the modified README in the directory, but I'm still working on rebasing things so that the old README will be associated with the "Add Mikolaj's Code" commit.
            Hide
            Permalink
            sthrush Samantha Thrush added a comment -

            I've just added Mikolaj's original README and then my modifications to it into the directory so that it will be clear which modifications I made to the document. 

            Show
            sthrush Samantha Thrush added a comment - I've just added Mikolaj's original README and then my modifications to it into the directory so that it will be clear which modifications I made to the document. 
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Please be sure to clean up the commit history before merging. 

             

            The Dev Guide suggests 

            During this rebase, we recommend squashing any fixup commits into the main commit implementing that feature. Git commit history should not record the iterative improvements from code review.

            and other details therein.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Please be sure to clean up the commit history before merging.    The Dev Guide suggests  During this rebase, we recommend squashing any fixup commits into the main commit implementing that feature. Git commit history should not record the iterative improvements from code review. and other details therein.
            Hide
            Permalink
            sthrush Samantha Thrush added a comment -

            I've cleaned the commit history and the ticket has been merged. 

            Show
            sthrush Samantha Thrush added a comment - I've cleaned the commit history and the ticket has been merged. 

              People

              Assignee:
              sthrush Samantha Thrush
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Mikolaj Kowalik
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik, Samantha Thrush
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.