Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Team:Data Facility
Description
Attachments
Issue Links
- mentioned in
-
Page Loading...
Activity
Got many 'No space left on device' errors; '/project' seems to be getting full.
Had to remove some files to continue.
Hsin-Fang Chiang
added a comment - - edited Got many 'No space left on device' errors; '/project' seems to be getting full.
Had to remove some files to continue. singleFrameDriver/prcessCcd failed in 10 CCDs at WIDE, and 36 CCDs at COSMOS. This is expected by the config changes in DM-11090. The failed dataIds are the same as in week 30 (DM-11185).
The failed data IDs are:
--id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=24308 ccd=29 --id visit=28376 ccd=69 --id visit=28380 ccd=0 --id visit=28382 ccd=101 --id visit=28392 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28396 ccd=102 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=5 --id visit=28400 ccd=10 --id visit=28400 ccd=15 --id visit=28400 ccd=23 --id visit=28400 ccd=26 --id visit=28400 ccd=40 --id visit=28400 ccd=53 --id visit=28400 ccd=55 --id visit=28400 ccd=61 --id visit=28400 ccd=68 --id visit=28400 ccd=77 --id visit=28400 ccd=84 --id visit=28400 ccd=89 --id visit=28400 ccd=93 --id visit=28400 ccd=94 --id visit=28400 ccd=95 --id visit=28400 ccd=99 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=28400 ccd=102 --id visit=29324 ccd=99 --id visit=9868 ccd=76 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=67 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58 |
Hsin-Fang Chiang
added a comment - singleFrameDriver/prcessCcd failed in 10 CCDs at WIDE, and 36 CCDs at COSMOS. This is expected by the config changes in DM-11090 . The failed dataIds are the same as in week 30 ( DM-11185 ).
The failed data IDs are:
--id visit= 278 ccd= 95 --id visit= 280 ccd= 22 --id visit= 280 ccd= 69 --id visit= 284 ccd= 61 --id visit= 1206 ccd= 77 --id visit= 24308 ccd= 29 --id visit= 28376 ccd= 69 --id visit= 28380 ccd= 0 --id visit= 28382 ccd= 101 --id visit= 28392 ccd= 101 --id visit= 28392 ccd= 102 --id visit= 28394 ccd= 93 --id visit= 28396 ccd= 102 --id visit= 28398 ccd= 95 --id visit= 28398 ccd= 101 --id visit= 28400 ccd= 5 --id visit= 28400 ccd= 10 --id visit= 28400 ccd= 15 --id visit= 28400 ccd= 23 --id visit= 28400 ccd= 26 --id visit= 28400 ccd= 40 --id visit= 28400 ccd= 53 --id visit= 28400 ccd= 55 --id visit= 28400 ccd= 61 --id visit= 28400 ccd= 68 --id visit= 28400 ccd= 77 --id visit= 28400 ccd= 84 --id visit= 28400 ccd= 89 --id visit= 28400 ccd= 93 --id visit= 28400 ccd= 94 --id visit= 28400 ccd= 95 --id visit= 28400 ccd= 99 --id visit= 28400 ccd= 100 --id visit= 28400 ccd= 101 --id visit= 28400 ccd= 102 --id visit= 29324 ccd= 99 --id visit= 9868 ccd= 76 --id visit= 11582 ccd= 76 --id visit= 7344 ccd= 67 --id visit= 19468 ccd= 69 --id visit= 6478 ccd= 99 --id visit= 6528 ccd= 24 --id visit= 6528 ccd= 67 --id visit= 9736 ccd= 67 --id visit= 17738 ccd= 69 --id visit= 17750 ccd= 58
Out of the 46 singleFrameDriver/prcessCcd fatal failures:
- 28 failed with "Unable to match sources"
- 3 failed with "No sources remaining in match list after magnitude limit cuts"
- 4 failed with "No objects passed our cuts for consideration as psf stars"
- 11 failed with "PSF star selector found [123] candidates"
A retry log with the 46 failures is attached.
In mosaic jobs, still see many warnings like afw.table WARN: Persisted match record with ID 111331498155159102 not found in catalog 1. as before (DM-11356).
In coadd and multiband, the numbers of products and the empty patches are the same as w_2017_28 (DM-11184) (as in this comment). No fatal errors were seen in the logs.
I had to remove files from /project to make space for this new run. Free space is running low.
The output repo is at /datasets/hsc/repo/rerun/RC/w_2017_32/DM-11186/
Logs are at /datasets/hsc/repo/rerun/RC/w_2017_32/DM-11186/logs
Hsin-Fang Chiang
added a comment - Out of the 46 singleFrameDriver/prcessCcd fatal failures:
28 failed with "Unable to match sources"
3 failed with "No sources remaining in match list after magnitude limit cuts"
4 failed with "No objects passed our cuts for consideration as psf stars"
11 failed with "PSF star selector found [123] candidates"
A retry log with the 46 failures is attached.
In mosaic jobs, still see many warnings like afw.table WARN: Persisted match record with ID 111331498155159102 not found in catalog 1. as before ( DM-11356 ).
In coadd and multiband, the numbers of products and the empty patches are the same as w_2017_28 ( DM-11184 ) ( as in this comment ). No fatal errors were seen in the logs.
I had to remove files from /project to make space for this new run. Free space is running low.
The output repo is at /datasets/hsc/repo/rerun/RC/w_2017_32/ DM-11186 /
Logs are at /datasets/hsc/repo/rerun/RC/w_2017_32/ DM-11186 /logs 
Started single frame processing with w_2017_32 that just became available in the shared stack an hour ago