Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12388

Reprocess RC with w_2017_44

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC dataset, as defined in DMTR-31 Sect. 4.1 , using Stack w_2017_44. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, coaddDriver.py, and multiBandDriver.py (DM-10129/DM-11020).

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            With the shared stack changes , the default stack at /software is now Python 3. This ticket is the first biweekly RC reprocessing with the Python 3 Stack.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - With the shared stack changes , the default stack at /software is now Python 3. This ticket is the first biweekly RC reprocessing with the Python 3 Stack.
            Hide
            tjenness Tim Jenness added a comment -

            We do have to remember that we know of at least two occasions where py2 and py3 disagree and we haven't looked into why yet.

            Show
            tjenness Tim Jenness added a comment - We do have to remember that we know of at least two occasions where py2 and py3 disagree and we haven't looked into why yet.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Yes, they are DM-11779 and DM-12180. I asked in Slack #dm-hsc-reprocessing and the consensus was it's okay to move on to Python 3. (tagging Jim Bosch John Swinbank )

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Yes, they are DM-11779 and DM-12180 . I asked in Slack #dm-hsc-reprocessing and the consensus was it's okay to move on to Python 3. (tagging Jim Bosch John Swinbank )
            Hide
            swinbank John Swinbank added a comment - - edited

            (Also, being totally picky, I did look into DM-11779, I just failed to achieve anything useful!)

            Show
            swinbank John Swinbank added a comment - - edited (Also, being totally picky, I did look into DM-11779 , I just failed to achieve anything useful!)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            SingleFrameDriver/prcessCcd failed in 12 CCDs at WIDE, and 17 CCDs at COSMOS. Their data IDs are:

            --id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=284 ccd=61 --id visit=28376 ccd=69 --id visit=28382 ccd=101 --id visit=28392 ccd=101 --id visit=28396 ccd=102 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=5 --id visit=28400 ccd=53 --id visit=28400 ccd=61 --id visit=28400 ccd=95 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=28400 ccd=102 --id visit=9868 ccd=76 --id visit=9868 ccd=77 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=59 --id visit=6528 ccd=67 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58
            

            Out of the 29 singleFrameDriver/prcessCcd fatal failures:

            • 14 failed with "Unable to match sources"
            • 4 failed with "No objects passed our cuts for consideration as psf stars"
            • 11 failed with "PSF star selector found [123] candidates"

            A log with the 29 processCcd failures is attached: log_w44_retryFailedSfm-99756.out

            Show
            hchiang2 Hsin-Fang Chiang added a comment - SingleFrameDriver/prcessCcd failed in 12 CCDs at WIDE, and 17 CCDs at COSMOS. Their data IDs are: --id visit= 278 ccd= 95 --id visit= 280 ccd= 22 --id visit= 280 ccd= 69 --id visit= 284 ccd= 61 --id visit= 28376 ccd= 69 --id visit= 28382 ccd= 101 --id visit= 28392 ccd= 101 --id visit= 28396 ccd= 102 --id visit= 28398 ccd= 95 --id visit= 28398 ccd= 101 --id visit= 28400 ccd= 5 --id visit= 28400 ccd= 53 --id visit= 28400 ccd= 61 --id visit= 28400 ccd= 95 --id visit= 28400 ccd= 100 --id visit= 28400 ccd= 101 --id visit= 28400 ccd= 102 --id visit= 9868 ccd= 76 --id visit= 9868 ccd= 77 --id visit= 11582 ccd= 76 --id visit= 7344 ccd= 67 --id visit= 19468 ccd= 69 --id visit= 6478 ccd= 99 --id visit= 6528 ccd= 24 --id visit= 6528 ccd= 59 --id visit= 6528 ccd= 67 --id visit= 9736 ccd= 67 --id visit= 17738 ccd= 69 --id visit= 17750 ccd= 58 Out of the 29 singleFrameDriver/prcessCcd fatal failures: 14 failed with "Unable to match sources" 4 failed with "No objects passed our cuts for consideration as psf stars" 11 failed with "PSF star selector found [123] candidates" A log with the 29 processCcd failures is attached: log_w44_retryFailedSfm-99756.out
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R
            • patch 8,8 HSC-R,NB0921

            In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813.

            This is the same since w_2017_28 (DM-11184).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R patch 8,8 HSC-R,NB0921 In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813. This is the same since w_2017_28 ( DM-11184 ).
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is available at /datasets/hsc/repo/rerun/RC/w_2017_44/DM-12388/

            Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_44/DM-12388/logs/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is available at /datasets/hsc/repo/rerun/RC/w_2017_44/ DM-12388 / Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_44/ DM-12388 /logs/

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, John Swinbank, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel