Samantha Thrush added a comment - 04/Aug/17 1:25 PM - edited As suggested, I ran my stat code previously used for DM-10904 on /project/hsc_rc/w_2017_27/ DM-11273 . Overall, there were 356249 files present, taking up a total of 6.77 Terabytes. The average file size was 18.38 Megabytes, with the largest 2000 files having sizes from 170 Megabytes to 310 Megabytes and the smallest 2000 files having sizes from 128 kilobytes to 0 bytes. The largest files were warp fits files, found in the deepCoadd subdirectory, as well as calexp, meas, and ref fits files found within the deepCoadd-results subdirectory. However, the smallest files were almost exclusively BKGD fits files, found within one of the numbered directories, along with the following file: deepCoadd/skyMap.pickle. In another comment, I will introduce and discuss the histograms describing the file size distribution and the overall sizes of the butler data types.

Samantha Thrush added a comment - 04/Aug/17 3:21 PM - edited Below, you will see the file size distribution for all of the files contained within /project/hsc_rc/w_2017_27/ DM-11273 . Notice that it bears a resemblance to the same graphs made for SFM in DM-10904 .

Samantha Thrush added a comment - 04/Aug/17 6:43 PM - edited The organization of /project/hsc_rc/w_2017_27/ DM-11273 was more similar to the DEEP, UDEEP and WIDE directories discussed in DM-10904 , but there were also some similarities to the SFM directory. In order to better understand where each of these files are coming from, here is a table. This table is formatted like the one above, with the wildcards being bolded. To be more succinct, unlike above, please assume that all of these paths dwell within /project/hsc_rc/w_2017_27/ DM-11273 Butler File Path CORR number / filter /corr/CORR- number - number .fits SRC number / filter /output/SRC- number - number .fits SRCMATCH number / filter /output/SRCMATCH- number - number .fits SRCMATCHFULL number / filter /output/SRCMATCHFULL- number - number .fits deepCoadd-SkyMap deepCoadd/skyMap.pickle BKGD number / filter /corr/BKGD- number - number .fits boost files number / filter /singleFrameDriver_metadata/ number .boost flattened files number / filter /thumbs/flattened- number - number .png oss files number / filter /thumbs/oss- number - number .png warp deepCoadd/ filter / number / number,number /warp- filter-number - number , number - number .fits calexp deepCoadd-results/ filter / number / number,number /calexp- filter-number - number,number .fits det_bkgd deepCoadd-results/ filter / number / number,number /det_bkgd- filter - number - number,number .fits det deepCoadd-results/ filter / number / number,number /det- filter - number - number,number .fits forced_src deepCoadd-results/ filter / number / number,number /forced_src- filter - number - number,number .fits meas deepCoadd-results/ filter / number / number,number /meas- filter - number - number,number .fits srcMatchFull deepCoadd-results/ filter / number / number,number /srcMatchFull- filter - number - number,number .fits srcMatch deepCoadd-results/ filter / number / number,number /srcMatch- filter - number - number,number .fits mergeDet deepCoadd-results/merged/ number / number , number /mergeDet- number - number , number .fits ref deepCoadd-results/merged/ number / number,number /ref- filter - number - number,number .fits fcr jointcal-results/ number /fcr- number - number .fits wcs jointcal-results/ number /wcs- number - number .fits schema files schema/*

Samantha Thrush added a comment - 14/Aug/17 7:08 AM The two scripts used to gather information for these graphs can be read here: https://github.com/Samantha-Thrush/LSST_codes/blob/master/DMbutler.sh https://github.com/Samantha-Thrush/LSST_codes/blob/master/repostats.sh