Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:5
-
Epic Link:
-
Team:DM Science
-
Urgent?:No
Description
Use w_2020_08 stack with qa_explorer at commit ab69304 and pipe_analysis at commit 09a7675. (same as DM-23579)
The command line will look like this, with all tract/filter IDs pairs, with either DEEP or WIDE accordingly
coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:DM-23243/ANALYSIS/DEEP --doraise --config doWriteParquetTables=True --id tract=08766 filter=NB0921
|
Attachments
Issue Links
Activity
For WIDE, I got 3319 jobs in total; 3319 log files are at
/datasets/hsc/repo/rerun/DM-23243/logs/coaddAnalysis/wide/
There are
- 1429 UnboundLocalError: local variable 'axes2' referenced before assignment
- 11 RuntimeError: No good data points to plot for sample labelled: star
Nonetheless, there are 3308 sets of parquet outputs:
/datasets/hsc/repo/rerun/DM-23243/ANALYSIS/WIDE/plots/*/*/*_forced.parq
/datasets/hsc/repo/rerun/DM-23243/ANALYSIS/WIDE/plots/*/*/*_unforced.parq
but more missing plots.
Hsin-Fang Chiang
added a comment - For WIDE, I got 3319 jobs in total; 3319 log files are at
/datasets/hsc/repo/rerun/ DM-23243 /logs/coaddAnalysis/wide/
There are
1429 UnboundLocalError: local variable 'axes2' referenced before assignment
11 RuntimeError: No good data points to plot for sample labelled: star
Nonetheless, there are 3308 sets of parquet outputs:
/datasets/hsc/repo/rerun/ DM-23243 /ANALYSIS/WIDE/plots/*/*/*_forced.parq
/datasets/hsc/repo/rerun/ DM-23243 /ANALYSIS/WIDE/plots/*/*/*_unforced.parq
but more missing plots. (Uploading the data IDs of the two kinds of errors for WIDE)
Hsin-Fang Chiang
added a comment - (Uploading the data IDs of the two kinds of errors for WIDE) Note that some of these jobs seem memory demanding. For WIDE, the high observed was ~75GB.
Hsin-Fang Chiang
added a comment - Note that some of these jobs seem memory demanding. For WIDE, the high observed was ~75GB. Slurm job IDs:
- DEEP: 245058,245693,245716
- WIDE: 246777,246779,246783,246785,246798,246799,246800
Commands of the jobs can be found at:
https://github.com/lsst-dm/s20-hsc-pdr2-reprocessing/tree/master/11_coaddAnalysis
I'm closing this done. See above comments for logs and noted errors. The generated parquet files allowed the subsequent task qa_explorer/matchVisits.py to be run (DM-23831). If we want a re-run post-DM-23829, please file a new ticket.
Hsin-Fang Chiang
added a comment - Slurm job IDs:
DEEP: 245058,245693,245716
WIDE: 246777,246779,246783,246785,246798,246799,246800
Commands of the jobs can be found at:
https://github.com/lsst-dm/s20-hsc-pdr2-reprocessing/tree/master/11_coaddAnalysis
I'm closing this done. See above comments for logs and noted errors. The generated parquet files allowed the subsequent task qa_explorer/matchVisits.py to be run ( DM-23831 ). If we want a re-run post- DM-23829 , please file a new ticket.
(Uploading the data IDs of the 36 UnboundLocalError in DEEP+UDEEP)