Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:3
-
Team:Data Facility
Description
As an exercise, run the single frame processing code (as-in from pipe_drivers) with the HSC COSMOS dataset on the Verification Cluster. If appropriate, develop scripts to facilitate the process.
Attachments
Issue Links
Activity
The libXext issue got fixed today, so I should run everything without clobber-config.
Hsin-Fang Chiang
added a comment - The libXext issue got fixed today, so I should run everything without clobber-config. In the last run, many failures came from the non-science CCDs. But data from those CCDs are not needed.
Submitted another batch of the 167 visits using the shared stack w_2017_8, /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and, and considering only ccd=0..103 to exclude the focus chips.
Hsin-Fang Chiang
added a comment - In the last run, many failures came from the non-science CCDs. But data from those CCDs are not needed.
Submitted another batch of the 167 visits using the shared stack w_2017_8, /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and , and considering only ccd=0..103 to exclude the focus chips. Those non-science CCDs will be ignored in the new default config, as changed in DM-9577.
Hsin-Fang Chiang
added a comment - Those non-science CCDs will be ignored in the new default config, as changed in DM-9577 . In total we have 167 visits * 104 CCDs each = 17368 subtasks of singleFrameDriver.
Out of the 17368, 65 CCDs failed; calexp and src outputs were generated for the other 17303 CCDs. Most of the failures (64 out of 65) were ccd=9
Hsin-Fang Chiang
added a comment - In total we have 167 visits * 104 CCDs each = 17368 subtasks of singleFrameDriver.
Out of the 17368, 65 CCDs failed; calexp and src outputs were generated for the other 17303 CCDs. Most of the failures (64 out of 65) were ccd=9 Output data are at /datasets/hsc/repo/rerun/private/hchiang2/DM-9053
Hsin-Fang Chiang
added a comment - Output data are at /datasets/hsc/repo/rerun/private/hchiang2/ DM-9053 
Started by using the shared stack w_2017_5 and /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and. Also using --clobber-config because of the missing libXext on the compute nodes (ncsa jira LSST-853).