Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11413

analyze the storage usage of an HSC RC reprocessing output repo

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Do a storage analysis, similar to DM-10904, for an output repo from a HSC RC reprocessing

        Attachments

          Issue Links

            Activity

            Hide
            sthrush Samantha Thrush added a comment - - edited

            As suggested, I ran my stat code previously used for DM-10904 on /project/hsc_rc/w_2017_27/DM-11273. Overall, there were 356249 files present, taking up a total of 6.77 Terabytes.

            The average file size was 18.38 Megabytes, with the largest 2000 files having sizes from 170 Megabytes to 310 Megabytes and the smallest 2000 files having sizes from 128 kilobytes to 0 bytes.

            The largest files were warp fits files, found in the deepCoadd subdirectory, as well as calexp, meas, and ref fits files found within the deepCoadd-results subdirectory. However, the smallest files were almost exclusively BKGD fits files, found within one of the numbered directories, along with the following file: deepCoadd/skyMap.pickle.

            In another comment, I will introduce and discuss the histograms describing the file size distribution and the overall sizes of the butler data types.

            Show
            sthrush Samantha Thrush added a comment - - edited As suggested, I ran my stat code previously used for DM-10904 on /project/hsc_rc/w_2017_27/ DM-11273 . Overall, there were 356249 files present, taking up a total of 6.77 Terabytes. The average file size was 18.38 Megabytes, with the largest 2000 files having sizes from 170 Megabytes to 310 Megabytes and the smallest 2000 files having sizes from 128 kilobytes to 0 bytes. The largest files were warp fits files, found in the deepCoadd subdirectory, as well as calexp, meas, and ref fits files found within the deepCoadd-results subdirectory. However, the smallest files were almost exclusively BKGD fits files, found within one of the numbered directories, along with the following file: deepCoadd/skyMap.pickle. In another comment, I will introduce and discuss the histograms describing the file size distribution and the overall sizes of the butler data types.
            Hide
            sthrush Samantha Thrush added a comment - - edited

            Below, you will see the file size distribution for all of the files contained within /project/hsc_rc/w_2017_27/DM-11273. Notice that it bears a resemblance to the same graphs made for SFM in DM-10904.

            Show
            sthrush Samantha Thrush added a comment - - edited Below, you will see the file size distribution for all of the files contained within /project/hsc_rc/w_2017_27/ DM-11273 . Notice that it bears a resemblance to the same graphs made for SFM in DM-10904 .
            Hide
            sthrush Samantha Thrush added a comment - - edited

            The organization of /project/hsc_rc/w_2017_27/DM-11273 was more similar to the DEEP, UDEEP and WIDE directories discussed in DM-10904, but there were also some similarities to the SFM directory.

            In order to better understand where each of these files are coming from, here is a table. This table is formatted like the one above, with the wildcards being bolded. To be more succinct, unlike above, please assume that all of these paths dwell within /project/hsc_rc/w_2017_27/DM-11273

            Butler File Path
            CORR number/filter/corr/CORR-number-number.fits
            SRC number/filter/output/SRC-number-number.fits
            SRCMATCH number/filter/output/SRCMATCH-number-number.fits
            SRCMATCHFULL number/filter/output/SRCMATCHFULL-number-number.fits
            deepCoadd-SkyMap deepCoadd/skyMap.pickle
            BKGD number/filter/corr/BKGD-number-number.fits
            boost files number/filter/singleFrameDriver_metadata/number.boost
            flattened files number/filter/thumbs/flattened-number-number.png
            oss files number/filter/thumbs/oss-number-number.png
            warp deepCoadd/filter/number/number,number/warp-filter-number-number,number-number.fits
            calexp deepCoadd-results/filter/number/number,number/calexp-filter-number-number,number.fits
            det_bkgd deepCoadd-results/filter/number/number,number/det_bkgd-filter-number-number,number.fits
            det deepCoadd-results/filter/number/number,number/det-filter-number-number,number.fits
            forced_src deepCoadd-results/filter/number/number,number/forced_src-filter-number-number,number.fits
            meas deepCoadd-results/filter/number/number,number/meas-filter-number-number,number.fits
            srcMatchFull deepCoadd-results/filter/number/number,number/srcMatchFull-filter-number-number,number.fits
            srcMatch deepCoadd-results/filter/number/number,number/srcMatch-filter-number-number,number.fits
            mergeDet deepCoadd-results/merged/number/number,number/mergeDet-number-number,number.fits
            ref deepCoadd-results/merged/number/number,number/ref-filter-number-number,number.fits
            fcr jointcal-results/number/fcr-number-number.fits
            wcs jointcal-results/number/wcs-number-number.fits
            schema files schema/*
            Show
            sthrush Samantha Thrush added a comment - - edited The organization of /project/hsc_rc/w_2017_27/ DM-11273 was more similar to the DEEP, UDEEP and WIDE directories discussed in DM-10904 , but there were also some similarities to the SFM directory. In order to better understand where each of these files are coming from, here is a table. This table is formatted like the one above, with the wildcards being bolded. To be more succinct, unlike above, please assume that all of these paths dwell within /project/hsc_rc/w_2017_27/ DM-11273 Butler File Path CORR number / filter /corr/CORR- number - number .fits SRC number / filter /output/SRC- number - number .fits SRCMATCH number / filter /output/SRCMATCH- number - number .fits SRCMATCHFULL number / filter /output/SRCMATCHFULL- number - number .fits deepCoadd-SkyMap deepCoadd/skyMap.pickle BKGD number / filter /corr/BKGD- number - number .fits boost files number / filter /singleFrameDriver_metadata/ number .boost flattened files number / filter /thumbs/flattened- number - number .png oss files number / filter /thumbs/oss- number - number .png warp deepCoadd/ filter / number / number,number /warp- filter-number - number , number - number .fits calexp deepCoadd-results/ filter / number / number,number /calexp- filter-number - number,number .fits det_bkgd deepCoadd-results/ filter / number / number,number /det_bkgd- filter - number - number,number .fits det deepCoadd-results/ filter / number / number,number /det- filter - number - number,number .fits forced_src deepCoadd-results/ filter / number / number,number /forced_src- filter - number - number,number .fits meas deepCoadd-results/ filter / number / number,number /meas- filter - number - number,number .fits srcMatchFull deepCoadd-results/ filter / number / number,number /srcMatchFull- filter - number - number,number .fits srcMatch deepCoadd-results/ filter / number / number,number /srcMatch- filter - number - number,number .fits mergeDet deepCoadd-results/merged/ number / number , number /mergeDet- number - number , number .fits ref deepCoadd-results/merged/ number / number,number /ref- filter - number - number,number .fits fcr jointcal-results/ number /fcr- number - number .fits wcs jointcal-results/ number /wcs- number - number .fits schema files schema/*
            Hide
            sthrush Samantha Thrush added a comment -
            Show
            sthrush Samantha Thrush added a comment - The two scripts used to gather information for these graphs can be read here: https://github.com/Samantha-Thrush/LSST_codes/blob/master/DMbutler.sh https://github.com/Samantha-Thrush/LSST_codes/blob/master/repostats.sh

              People

              Assignee:
              sthrush Samantha Thrush
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Samantha Thrush
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: