Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:6
-
Epic Link:
-
Team:Data Facility
Description
Modify the usage.py and usageplot.py codes found in https://github.com/lsst-dm/ldf_ops_tools/tree/master/usage to allow for the creation of node usage plots that are color-coded to differentiate which codes (singleFrameDriver, mosaic, coaddDriver, etc.) use the nodes at a certain time. This will make it more clear which codes are node-greedy and which ones are not.
Attachments
Issue Links
- relates to
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
Activity
After an overhaul of usageplot.py, I'm finally happy with the code. In setting up the code, I've made the following assumptions: when naming your slurm jobs, I've assumed that the job names have been assigned in the following way:
| code name | starting two letters of SLURM JobName |
|---|---|
| singleFrameDriver | "Co" or "Wi" |
| mosaic | "mo" |
| coaddDriver | "co" |
| multiBandDriver | "mt" |
If the JobName is something else (or if the job experienced the NODE_FAIL/COMPLETED error fixed in DM-13619), the job will be marked as "unknown" on the plot.
I've also added hatching to the color fill so that it would be clear what codes were running concurrently (and thus would have their color fill in the same location).
Additionally, I noticed that while most of the reprocessing jobs had all of the same codes running at the same time, (thus, no singleFrameDriver jobs were started after coadd driver started to run, etc.), some of the reprocessing runs did (usually due to errors resulting in having to rerun a code). Due to how the first draft of my code was structured, this caused improper color-filling in the plots (I only looked for one start and stop index for each code). I've since fixed the bug, thus allowing usageplot.py to account for many start and stop indices for a given code which results in accurate color-filling.
I'll be adding the new color-fill plots to confluence tonight and will be cleaning up the code further before I have it reviewed.
Samantha Thrush
added a comment - - edited After an overhaul of usageplot.py, I'm finally happy with the code. In setting up the code, I've made the following assumptions: when naming your slurm jobs, I've assumed that the job names have been assigned in the following way:
code name
starting two letters of SLURM JobName
singleFrameDriver
"Co" or "Wi"
mosaic
"mo"
coaddDriver
"co"
multiBandDriver
"mt"
If the JobName is something else (or if the job experienced the NODE_FAIL/COMPLETED error fixed in DM-13619 ), the job will be marked as "unknown" on the plot.
I've also added hatching to the color fill so that it would be clear what codes were running concurrently (and thus would have their color fill in the same location).
Additionally, I noticed that while most of the reprocessing jobs had all of the same codes running at the same time, (thus, no singleFrameDriver jobs were started after coadd driver started to run, etc.), some of the reprocessing runs did (usually due to errors resulting in having to rerun a code). Due to how the first draft of my code was structured, this caused improper color-filling in the plots (I only looked for one start and stop index for each code). I've since fixed the bug, thus allowing usageplot.py to account for many start and stop indices for a given code which results in accurate color-filling.
I'll be adding the new color-fill plots to confluence tonight and will be cleaning up the code further before I have it reviewed. This review might take a bit longer than my other tickets as I did have to make quite a few changes to both usage.py and usageplot.py.
Samantha Thrush
added a comment - This review might take a bit longer than my other tickets as I did have to make quite a few changes to both usage.py and usageplot.py. Since you started using colors to show node usage by different tasks, you may consider removing the blue "envelope". It may look ugly with certain colors.
Mikolaj Kowalik
added a comment - Since you started using colors to show node usage by different tasks, you may consider removing the blue "envelope". It may look ugly with certain colors. Good point. I'll do that.
Samantha Thrush
added a comment - Good point. I'll do that. 
I've finally finished up my first draft of the code (see w_2017_42_color plot above), and although I'm happy with the results, I don't think the code modifications quite ready for public consumption as I still have one or two things coded in non-optimal ways. I'll keep working on the code, but in the mean time, I'm planning on replacing the plots on my confluence page with color coded ones for better clarity.