[DM-12873] Run pipe_analysis scripts on w_46-processed RC data at NCSA - Jira

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: pipe_analysis
Labels:
None

Story Points:
2
Epic Link:
s18-drp-qa
Sprint:
DRP S18-1
Team:
Data Release Production

Description

In support of the LDM-534 QA milestone, run coaddAnalysis.py, colorAnalysis.py and visitAnalysis.py on all coadds and visits from the weekly 46 RC processing run (/datasets/hsc/repo/rerun/RC/w_2017_46/~~DM-12545~~/).

Attachments

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Tim Morton [X] (Inactive) added a comment - 30/Nov/17 2:53 PM

I have just submitted the following jobs (/home/tmorton/tickets/DM-12873/make_qa_plots.sh):

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11690..11712:2^29324^29326^29336^29340^29350^29352 filter=HSC-G --tract=9813 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-R --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1202..1220:2^23692^23694^23704^23706^23716^23718 filter=HSC-R --tract=9813 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-I --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1228..1232:2^1236..1248:2^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482..30504:2 filter=HSC-I --tract=9813 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Y --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=274..302:2^306..334:2^342..370:2^1858..1862:2^1868..1882:2^11718..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2 filter=HSC-Y --tract=9813 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Z --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1166..1194:2^17900..17908:2^17926..17934:2^17944..17952:2^17962^28354..28402:2 filter=HSC-Z --tract=9813 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=NB0921 --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=23038..23056:2^23594..23606:2^24298..24310:2^25810..25816:2 filter=NB0921 --tract=9813 --clobber-versions --doraise

colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z^NB0921 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8766 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-R --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8766 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-I --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8766 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Y --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8766 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Z --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8766 --clobber-versions --doraise

colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8767 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-R --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8767 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-I --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8767 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Y --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8767 --clobber-versions --doraise

coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Z --clobber-versions --doraise

visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8767 --clobber-versions --doraise

colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise

Show

Tim Morton [X] (Inactive) added a comment - 30/Nov/17 2:53 PM I have just submitted the following jobs ( /home/tmorton/tickets/ DM-12873 /make_qa_plots.sh ): coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11690..11712:2^29324^29326^29336^29340^29350^29352 filter=HSC-G --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1202..1220:2^23692^23694^23704^23706^23716^23718 filter=HSC-R --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1228..1232:2^1236..1248:2^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482..30504:2 filter=HSC-I --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=274..302:2^306..334:2^342..370:2^1858..1862:2^1868..1882:2^11718..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2 filter=HSC-Y --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1166..1194:2^17900..17908:2^17926..17934:2^17944..17952:2^17962^28354..28402:2 filter=HSC-Z --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=NB0921 --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=23038..23056:2^23594..23606:2^24298..24310:2^25810..25816:2 filter=NB0921 --tract=9813 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z^NB0921 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8766 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8767 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise

Hide

Permalink

Tim Morton [X] (Inactive) added a comment - 01/Dec/17 8:46 AM

Due to the problems being investigated in ~~DM-12882~~, I will also be running these on the w_2017_44 weekly RC processing. The QA plots/tables will be in /project/tmorton/DM-12873/w44 and /project/tmorton/DM-12873/46.

Show

Tim Morton [X] (Inactive) added a comment - 01/Dec/17 8:46 AM Due to the problems being investigated in DM-12882 , I will also be running these on the w_2017_44 weekly RC processing. The QA plots/tables will be in /project/tmorton/ DM-12873 /w44 and /project/tmorton/ DM-12873 /46 .

Hide

Permalink

Tim Morton [X] (Inactive) added a comment - 04/Dec/17 10:46 AM

These have completed; in /project/tmorton/DM-12873/w44 and /project/tmorton/DM-12873/w46, respectively; Lauren MacArthur I'll put you as reviewer, and I suppose the review process will happen naturally as you look at plots for the QA milestone.

Show

Tim Morton [X] (Inactive) added a comment - 04/Dec/17 10:46 AM These have completed; in /project/tmorton/ DM-12873 /w44 and /project/tmorton/ DM-12873 /w46 , respectively; Lauren MacArthur I'll put you as reviewer, and I suppose the review process will happen naturally as you look at plots for the QA milestone.

Hide

Permalink

Lauren MacArthur added a comment - 24/Dec/17 1:15 PM

The outputs look good. The /project/tmorton/DM-12873/w44/plots/ directory could use some clean-up to avoid any confusion (e.g. NB9021, NB0912, cosmos, and wide sub-dirs are all empty).

Show

Lauren MacArthur added a comment - 24/Dec/17 1:15 PM The outputs look good. The /project/tmorton/ DM-12873 /w44/plots/ directory could use some clean-up to avoid any confusion (e.g. NB9021, NB0912, cosmos, and wide sub-dirs are all empty).

Data Management

Run pipe_analysis scripts on w_46-processed RC data at NCSA

Details

Description

Attachments

Activity

People

Dates

Jenkins