Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12873

Run pipe_analysis scripts on w_46-processed RC data at NCSA

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_analysis
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP S18-1
    • Team:
      Data Release Production

      Description

      In support of the LDM-534 QA milestone, run coaddAnalysis.py, colorAnalysis.py and visitAnalysis.py on all coadds and visits from the weekly 46 RC processing run (/datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/).

        Attachments

          Activity

          Hide
          tmorton Tim Morton [X] (Inactive) added a comment -

          I have just submitted the following jobs (/home/tmorton/tickets/DM-12873/make_qa_plots.sh):

          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11690..11712:2^29324^29326^29336^29340^29350^29352 filter=HSC-G --tract=9813 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-R --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1202..1220:2^23692^23694^23704^23706^23716^23718 filter=HSC-R --tract=9813 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-I --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1228..1232:2^1236..1248:2^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482..30504:2 filter=HSC-I --tract=9813 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Y --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=274..302:2^306..334:2^342..370:2^1858..1862:2^1868..1882:2^11718..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2 filter=HSC-Y --tract=9813 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Z --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=1166..1194:2^17900..17908:2^17926..17934:2^17944..17952:2^17962^28354..28402:2 filter=HSC-Z --tract=9813 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=NB0921 --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=23038..23056:2^23594..23606:2^24298..24310:2^25810..25816:2 filter=NB0921 --tract=9813 --clobber-versions --doraise
          colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z^NB0921 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8766 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-R --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8766 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-I --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8766 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Y --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8766 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Z --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8766 --clobber-versions --doraise
          colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8767 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-R --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8767 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-I --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8767 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Y --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8767 --clobber-versions --doraise
          coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Z --clobber-versions --doraise
          visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8767 --clobber-versions --doraise
          colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/  --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise
          

          Show
          tmorton Tim Morton [X] (Inactive) added a comment - I have just submitted the following jobs ( /home/tmorton/tickets/ DM-12873 /make_qa_plots.sh ): coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11690..11712:2^29324^29326^29336^29340^29350^29352 filter=HSC-G --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1202..1220:2^23692^23694^23704^23706^23716^23718 filter=HSC-R --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1228..1232:2^1236..1248:2^19658^19660^19662^19680^19682^19684^19694^19696^19698^19708^19710^19712^30482..30504:2 filter=HSC-I --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=274..302:2^306..334:2^342..370:2^1858..1862:2^1868..1882:2^11718..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2 filter=HSC-Y --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=1166..1194:2^17900..17908:2^17926..17934:2^17944..17952:2^17962^28354..28402:2 filter=HSC-Z --tract=9813 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=NB0921 --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=23038..23056:2^23594..23606:2^24298..24310:2^25810..25816:2 filter=NB0921 --tract=9813 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=9813 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z^NB0921 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8766 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8766 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8766 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9852^9856^9860^9864^9868^9870^9888^9890^9898^9900^9904^9906^9912^11568^11572^11576^11582^11588^11590^11596^11598 filter=HSC-G --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-R --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=11442^11446^11450^11470^11476^11478^11506^11508^11532^11534 filter=HSC-R --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-I --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=7300^7304^7308^7318^7322^7338^7340^7344^7348^7358^7360^7374^7384^7386^19468^19470^19482^19484^19486 filter=HSC-I --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Y --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=6478^6482^6486^6496^6498^6522^6524^6528^6532^6544^6546^6568^13152^13154 filter=HSC-Y --tract=8767 --clobber-versions --doraise coaddAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-Z --clobber-versions --doraise visitAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id visit=9708^9712^9716^9724^9726^9730^9732^9736^9740^9750^9752^9764^9772^9774^17738^17740^17750^17752^17754 filter=HSC-Z --tract=8767 --clobber-versions --doraise colorAnalysis.py /datasets/hsc/repo/rerun/RC/w_2017_46/DM-12545/ --output /project/tmorton/DM-12873/ --id tract=8767 filter=HSC-G^HSC-R^HSC-I^HSC-Y^HSC-Z --clobber-versions --doraise
          Hide
          tmorton Tim Morton [X] (Inactive) added a comment -

          Due to the problems being investigated in DM-12882, I will also be running these on the w_2017_44 weekly RC processing. The QA plots/tables will be in /project/tmorton/DM-12873/w44 and /project/tmorton/DM-12873/46.

          Show
          tmorton Tim Morton [X] (Inactive) added a comment - Due to the problems being investigated in DM-12882 , I will also be running these on the w_2017_44 weekly RC processing. The QA plots/tables will be in /project/tmorton/ DM-12873 /w44 and /project/tmorton/ DM-12873 /46 .
          Hide
          tmorton Tim Morton [X] (Inactive) added a comment -

          These have completed; in /project/tmorton/DM-12873/w44 and /project/tmorton/DM-12873/w46, respectively; Lauren MacArthur I'll put you as reviewer, and I suppose the review process will happen naturally as you look at plots for the QA milestone.

          Show
          tmorton Tim Morton [X] (Inactive) added a comment - These have completed; in /project/tmorton/ DM-12873 /w44 and /project/tmorton/ DM-12873 /w46 , respectively; Lauren MacArthur I'll put you as reviewer, and I suppose the review process will happen naturally as you look at plots for the QA milestone.
          Hide
          lauren Lauren MacArthur added a comment -

          The outputs look good. The /project/tmorton/DM-12873/w44/plots/ directory could use some clean-up to avoid any confusion (e.g. NB9021, NB0912, cosmos, and wide sub-dirs are all empty).

          Show
          lauren Lauren MacArthur added a comment - The outputs look good. The /project/tmorton/ DM-12873 /w44/plots/ directory could use some clean-up to avoid any confusion (e.g. NB9021, NB0912, cosmos, and wide sub-dirs are all empty).

            People

            Assignee:
            tmorton Tim Morton [X] (Inactive)
            Reporter:
            tmorton Tim Morton [X] (Inactive)
            Reviewers:
            Lauren MacArthur
            Watchers:
            Jim Bosch, John Swinbank, Lauren MacArthur, Tim Morton [X] (Inactive), Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.