Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10596

PDAC: merge calexps and coadds for imgserv for database sdss_stripe82_01

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Merge calexp and coadd image directory trees from NCSA and IN2P3 sides of the SDSS/Stripe82 processing to match a new version of the merged catalogs in the PDAC database sdss_stripe82_01.

      The datasets shall be put into these folders:

      /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
      /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/calexp
      /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd
      

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment - - edited

            Setting up the data folders

            Set up a new data folder matching the name of the new catalog at:

            % mkdir /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
            

            Merging calibrated exposure images

            Since changes in the new catalog didn't affect calibrated exposures a collection of the corresponding images in the new catalog stayed the same as in the older one. This allowed borrowing the collection of calexps of the previous work:

            % cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
            % mkdir calexps
            % cd calexps
            % ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent
            % cp -arv /datasets/sdss/preprocessed/dr7/sdss_stripe82_00/calexps/sci-results .
            

            Run basic tests to count the number of images in the collection:

            % ls -1 sci-results/*/*/*/calexp/ | wc -l
            2514132
            

            Merging coadds

            Creating folders:

            % cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
            % mkdir coadd
            % cd coadd
            % ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent
            % for folder in deepCoadd deepCoadd-results; do \
              mkdir $folder \
              for band in u g r i z; do \
                for tract in 0 3; do \
                  mkdir $folder/$band/$tract \
                done \
              done \
            done
            

            Copy sky map:

            % cp \
              /datasets/sdss/preprocessed/dr7/sdss_stripe82_00/coadd/deepCoadd/skyMap.pickle \
              /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/
            

            The next step is to set up links under the deepCoadd folder. It's based on two collections of deepCoaddId (one from the NCSA and the other one fro the IN2P3 parts of the processing). A simple link generator will pull attributes associated with each image from a database service which was set up for the catalog merging effort (see DM-8241) using two SQL statements (for each collection) put into the corresponding files:
            ncsa_links.sql:

            SELECT
              CONCAT("ln -s /datasets/gapon/data/SDRP/data/coadd_",filterName,"_dir/deepCoadd/",filterName,"/",tract,"/",patch,
                     " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/",filterName,"/",tract,"/",patch)
            FROM
              gapon_SDRP_Stripe82.DeepCoadd
            WHERE
              deepCoaddId IN
                (SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId);
            


            in2p3_links.sql:

            SELECT
              CONCAT("ln -s /datasets/gapon/data/DC_2013/coadd/deepCoadd/",filterName,"/",tract,"/",patch,
                     " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/",filterName,"/",tract,"/",patch)
            FROM
              lsst_prod_DC_2013_2.DeepCoadd
            WHERE deepCoaddId IN
              (SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId);
            

            Each statement will be used like this:

            % cat ncsa_links.sql | mysql -N -h 141.142.211.152 > ncsa_links.source
            % wc -l ncsa_links.source
            13015 ncsa_links.source
            % source ncsa_links.source
             
            % cat in2p3_links.sql | mysql -N -h 141.142.211.152 > in2p3_links.source
            % wc -l in2p3_links.source
            13020 in2p3_links.source
            % source in2p3_links.source
            

            And the final step is to set up links under the deepCoadd-results folder:
            ncsa_results_links.sql:

            SELECT
              CONCAT("ln -s /datasets/gapon/data/SDRP/data/coadd_",filterName,"_dir/deepCoadd-results/",filterName,"/",tract,"/",patch,
                     " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/",filterName,"/",tract,"/",patch)
            FROM
              gapon_SDRP_Stripe82.DeepCoadd
            WHERE
              deepCoaddId IN
                (SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId);
            


            in2p3_results_links.sql:

            SELECT
              CONCAT("ln -s /datasets/gapon/data/DC_2013/coadd/deepCoaddDetect/deepCoadd-results/",filterName,"/",tract,"/",patch,
                     " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/",filterName,"/",tract,"/",patch)
            FROM
              lsst_prod_DC_2013_2.DeepCoadd
            WHERE deepCoaddId IN
              (SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId);
            

            Each statement will be used like this:

            % cat ncsa_results_links.sql | mysql -N -h 141.142.211.152 > ncsa_results_links.source
            % wc -l ncsa_results_links.source
            13015 ncsa_results_links.source
            % source ncsa_results_links.source
             
            % cat in2p3_results_links.sql | mysql -N -h 141.142.211.152 > in2p3_results_links.source
            % wc -l in2p3_results_links.source
            13020 in2p3_results_links.source
            % source in2p3_results_links.source
            

            Show
            gapon Igor Gaponenko added a comment - - edited Setting up the data folders Set up a new data folder matching the name of the new catalog at: % mkdir /datasets/sdss/preprocessed/dr7/sdss_stripe82_01 Merging calibrated exposure images Since changes in the new catalog didn't affect calibrated exposures a collection of the corresponding images in the new catalog stayed the same as in the older one. This allowed borrowing the collection of calexps of the previous work: % cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01 % mkdir calexps % cd calexps % ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent % cp -arv /datasets/sdss/preprocessed/dr7/sdss_stripe82_00/calexps/sci-results . Run basic tests to count the number of images in the collection: % ls -1 sci-results/*/*/* /calexp/ | wc -l 2514132 Merging coadds Creating folders: % cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01 % mkdir coadd % cd coadd % ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent % for folder in deepCoadd deepCoadd-results; do \ mkdir $folder \ for band in u g r i z; do \ for tract in 0 3; do \ mkdir $folder/$band/$tract \ done \ done \ done Copy sky map: % cp \ /datasets/sdss/preprocessed/dr7/sdss_stripe82_00/coadd/deepCoadd/skyMap .pickle \ /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/ The next step is to set up links under the deepCoadd folder. It's based on two collections of deepCoaddId (one from the NCSA and the other one fro the IN2P3 parts of the processing). A simple link generator will pull attributes associated with each image from a database service which was set up for the catalog merging effort (see DM-8241 ) using two SQL statements (for each collection) put into the corresponding files: ncsa_links.sql : SELECT CONCAT( "ln -s /datasets/gapon/data/SDRP/data/coadd_" ,filterName, "_dir/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch, " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch) FROM gapon_SDRP_Stripe82.DeepCoadd WHERE deepCoaddId IN ( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId); in2p3_links.sql : SELECT CONCAT( "ln -s /datasets/gapon/data/DC_2013/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch, " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch) FROM lsst_prod_DC_2013_2.DeepCoadd WHERE deepCoaddId IN ( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId); Each statement will be used like this: % cat ncsa_links.sql | mysql -N -h 141.142.211.152 > ncsa_links. source % wc -l ncsa_links. source 13015 ncsa_links. source % source ncsa_links. source   % cat in2p3_links.sql | mysql -N -h 141.142.211.152 > in2p3_links. source % wc -l in2p3_links. source 13020 in2p3_links. source % source in2p3_links. source And the final step is to set up links under the deepCoadd-results folder: ncsa_results_links.sql : SELECT CONCAT( "ln -s /datasets/gapon/data/SDRP/data/coadd_" ,filterName, "_dir/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch, " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch) FROM gapon_SDRP_Stripe82.DeepCoadd WHERE deepCoaddId IN ( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId); in2p3_results_links.sql : SELECT CONCAT( "ln -s /datasets/gapon/data/DC_2013/coadd/deepCoaddDetect/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch, " /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch) FROM lsst_prod_DC_2013_2.DeepCoadd WHERE deepCoaddId IN ( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId); Each statement will be used like this: % cat ncsa_results_links.sql | mysql -N -h 141.142.211.152 > ncsa_results_links. source % wc -l ncsa_results_links. source 13015 ncsa_results_links. source % source ncsa_results_links. source   % cat in2p3_results_links.sql | mysql -N -h 141.142.211.152 > in2p3_results_links. source % wc -l in2p3_results_links. source 13020 in2p3_results_links. source % source in2p3_results_links. source
            Hide
            gapon Igor Gaponenko added a comment -

            The links have been set up

            Show
            gapon Igor Gaponenko added a comment - The links have been set up

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                Fritz Mueller
                Watchers:
                Igor Gaponenko
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: