Show
added a comment - - edited Setting up the data folders
Set up a new data folder matching the name of the new catalog at:
% mkdir /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
Merging calibrated exposure images
Since changes in the new catalog didn't affect calibrated exposures a collection of the corresponding images in the new catalog stayed the same as in the older one. This allowed borrowing the collection of calexps of the previous work:
% cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
% mkdir calexps
% cd calexps
% ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent
% cp -arv /datasets/sdss/preprocessed/dr7/sdss_stripe82_00/calexps/sci-results .
Run basic tests to count the number of images in the collection:
% ls -1 sci-results/*/*/* /calexp/ | wc -l
2514132
Merging coadds
Creating folders:
% cd /datasets/sdss/preprocessed/dr7/sdss_stripe82_01
% mkdir coadd
% cd coadd
% ln -s /datasets/sdss/preprocessed/dr7/runs/ _parent
% for folder in deepCoadd deepCoadd-results; do \
mkdir $folder \
for band in u g r i z; do \
for tract in 0 3; do \
mkdir $folder/$band/$tract \
done \
done \
done
Copy sky map:
% cp \
/datasets/sdss/preprocessed/dr7/sdss_stripe82_00/coadd/deepCoadd/skyMap .pickle \
/datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/
The next step is to set up links under the deepCoadd folder. It's based on two collections of deepCoaddId (one from the NCSA and the other one fro the IN2P3 parts of the processing). A simple link generator will pull attributes associated with each image from a database service which was set up for the catalog merging effort (see DM-8241 ) using two SQL statements (for each collection) put into the corresponding files:
ncsa_links.sql :
SELECT
CONCAT( "ln -s /datasets/gapon/data/SDRP/data/coadd_" ,filterName, "_dir/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch,
" /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch)
FROM
gapon_SDRP_Stripe82.DeepCoadd
WHERE
deepCoaddId IN
( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId);
in2p3_links.sql :
SELECT
CONCAT( "ln -s /datasets/gapon/data/DC_2013/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch,
" /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd/" ,filterName, "/" ,tract, "/" ,patch)
FROM
lsst_prod_DC_2013_2.DeepCoadd
WHERE deepCoaddId IN
( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId);
Each statement will be used like this:
% cat ncsa_links.sql | mysql -N -h 141.142.211.152 > ncsa_links. source
% wc -l ncsa_links. source
13015 ncsa_links. source
% source ncsa_links. source
% cat in2p3_links.sql | mysql -N -h 141.142.211.152 > in2p3_links. source
% wc -l in2p3_links. source
13020 in2p3_links. source
% source in2p3_links. source
And the final step is to set up links under the deepCoadd-results folder:
ncsa_results_links.sql :
SELECT
CONCAT( "ln -s /datasets/gapon/data/SDRP/data/coadd_" ,filterName, "_dir/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch,
" /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch)
FROM
gapon_SDRP_Stripe82.DeepCoadd
WHERE
deepCoaddId IN
( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.ncsa_DeepCoadd_deepCoaddId);
in2p3_results_links.sql :
SELECT
CONCAT( "ln -s /datasets/gapon/data/DC_2013/coadd/deepCoaddDetect/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch,
" /datasets/sdss/preprocessed/dr7/sdss_stripe82_01/coadd/deepCoadd-results/" ,filterName, "/" ,tract, "/" ,patch)
FROM
lsst_prod_DC_2013_2.DeepCoadd
WHERE deepCoaddId IN
( SELECT deepCoaddId FROM lsst_sdss_stripe82_01.in2p3_DeepCoadd_deepCoaddId);
Each statement will be used like this:
% cat ncsa_results_links.sql | mysql -N -h 141.142.211.152 > ncsa_results_links. source
% wc -l ncsa_results_links. source
13015 ncsa_results_links. source
% source ncsa_results_links. source
% cat in2p3_results_links.sql | mysql -N -h 141.142.211.152 > in2p3_results_links. source
% wc -l in2p3_results_links. source
13020 in2p3_results_links. source
% source in2p3_results_links. source
Story
Setting up the data folders
Set up a new data folder matching the name of the new catalog at:
Merging calibrated exposure images
Since changes in the new catalog didn't affect calibrated exposures a collection of the corresponding images in the new catalog stayed the same as in the older one. This allowed borrowing the collection of calexps of the previous work:
Run basic tests to count the number of images in the collection:
2514132Merging coadds
Creating folders:
Copy sky map:
The next step is to set up links under the deepCoadd folder. It's based on two collections of deepCoaddId (one from the NCSA and the other one fro the IN2P3 parts of the processing). A simple link generator will pull attributes associated with each image from a database service which was set up for the catalog merging effort (see
DM-8241) using two SQL statements (for each collection) put into the corresponding files:ncsa_links.sql:
SELECTFROMgapon_SDRP_Stripe82.DeepCoaddWHEREin2p3_links.sql:
SELECTFROMlsst_prod_DC_2013_2.DeepCoaddEach statement will be used like this:
And the final step is to set up links under the deepCoadd-results folder:
ncsa_results_links.sql:
SELECTFROMgapon_SDRP_Stripe82.DeepCoaddWHEREin2p3_results_links.sql:
SELECTFROMlsst_prod_DC_2013_2.DeepCoaddEach statement will be used like this: