Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13699

Modify usage.py and usageplot.py to allow for color-coded plots

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Modify the usage.py and usageplot.py codes found in https://github.com/lsst-dm/ldf_ops_tools/tree/master/usage to allow for the creation of node usage plots that are color-coded to differentiate which codes (singleFrameDriver, mosaic, coaddDriver, etc.) use the nodes at a certain time.  This will make it more clear which codes are node-greedy and which ones are not. 

        Attachments

          Issue Links

            Activity

            Hide
            sthrush Samantha Thrush added a comment -

            I've finally finished up my first draft of the code (see w_2017_42_color plot above), and although I'm happy with the results, I don't think the code modifications quite ready for public consumption as I still have one or two things coded in non-optimal ways.  I'll keep working on the code, but in the mean time, I'm planning on replacing the plots on my confluence page with color coded ones for better clarity.

            Show
            sthrush Samantha Thrush added a comment - I've finally finished up my first draft of the code (see w_2017_42_color plot above), and although I'm happy with the results, I don't think the code modifications quite ready for public consumption as I still have one or two things coded in non-optimal ways.  I'll keep working on the code, but in the mean time, I'm planning on replacing the plots on my confluence page with color coded ones for better clarity.
            Hide
            sthrush Samantha Thrush added a comment - - edited

            After an overhaul of usageplot.py, I'm finally happy with the code.  In setting up the code, I've made the following assumptions: when naming your slurm jobs, I've assumed that the job names have been assigned in the following way:

            code name starting two letters of SLURM JobName
            singleFrameDriver "Co" or "Wi"
            mosaic "mo"
            coaddDriver "co"
            multiBandDriver "mt"

            If the JobName is something else (or if the job experienced the NODE_FAIL/COMPLETED error fixed in DM-13619), the job will be marked as "unknown" on the plot. 

            I've also added hatching to the color fill so that it would be clear what codes were running concurrently (and thus would have their color fill in the same location). 

            Additionally, I noticed that while most of the reprocessing jobs had all of the same codes running at the same time, (thus, no singleFrameDriver jobs were started after coadd driver started to run, etc.), some of the reprocessing runs did (usually due to errors resulting in having to rerun a code).  Due to how the first draft of my code was structured, this caused improper color-filling in the plots (I only looked for one start and stop index for each code).  I've since fixed the bug, thus allowing usageplot.py to account for many start and stop indices for a given code which results in accurate color-filling. 

            I'll be adding the new color-fill plots to confluence tonight and will be cleaning up the code further before I have it reviewed. 

            Show
            sthrush Samantha Thrush added a comment - - edited After an overhaul of usageplot.py, I'm finally happy with the code.  In setting up the code, I've made the following assumptions: when naming your slurm jobs, I've assumed that the job names have been assigned in the following way: code name starting two letters of SLURM JobName singleFrameDriver "Co" or "Wi" mosaic "mo" coaddDriver "co" multiBandDriver "mt" If the JobName is something else (or if the job experienced the NODE_FAIL/COMPLETED error fixed in DM-13619 ), the job will be marked as "unknown" on the plot.  I've also added hatching to the color fill so that it would be clear what codes were running concurrently (and thus would have their color fill in the same location).  Additionally, I noticed that while most of the reprocessing jobs had all of the same codes running at the same time, (thus, no singleFrameDriver jobs were started after coadd driver started to run, etc.), some of the reprocessing runs did (usually due to errors resulting in having to rerun a code).  Due to how the first draft of my code was structured, this caused improper color-filling in the plots (I only looked for one start and stop index for each code).  I've since fixed the bug, thus allowing usageplot.py to account for many start and stop indices for a given code which results in accurate color-filling.  I'll be adding the new color-fill plots to confluence tonight and will be cleaning up the code further before I have it reviewed. 
            Hide
            sthrush Samantha Thrush added a comment -

            This review might take a bit longer than my other tickets as I did have to make quite a few changes to both usage.py and usageplot.py.

            Show
            sthrush Samantha Thrush added a comment - This review might take a bit longer than my other tickets as I did have to make quite a few changes to both usage.py and usageplot.py.
            Hide
            mkowalik Mikolaj Kowalik added a comment -

            Since you started using colors to show node usage by different tasks, you may consider removing the blue "envelope". It may look ugly with certain colors.

            Show
            mkowalik Mikolaj Kowalik added a comment - Since you started using colors to show node usage by different tasks, you may consider removing the blue "envelope". It may look ugly with certain colors.
            Hide
            sthrush Samantha Thrush added a comment -

            Good point.  I'll do that. 

            Show
            sthrush Samantha Thrush added a comment - Good point.  I'll do that. 

              People

              Assignee:
              sthrush Samantha Thrush
              Reporter:
              sthrush Samantha Thrush
              Reviewers:
              Mikolaj Kowalik
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik, Samantha Thrush
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.