Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11020

Reprocess RC with w_2017_25

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC dataset using w_2017_25 following the same steps as in DM-10129

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In the single frame processing, among the 320 visits (103 ccds each, as ccd=9 was excluded), there were 81 reproducible fatal failures. Their dataIds are:

            • --id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=1212 ccd=103 --id visit=24308 ccd=29 --id visit=28366 ccd=101 --id visit=28372 ccd=0 --id visit=28372 ccd=100 --id visit=28374 ccd=0 --id visit=28374 ccd=101 --id visit=28376 ccd=69 --id visit=28378 ccd=10 --id visit=28378 ccd=101 --id visit=28380 ccd=0 --id visit=28380 ccd=101 --id visit=28382 ccd=98 --id visit=28382 ccd=101 --id visit=28386 ccd=52 --id visit=28386 ccd=64 --id visit=28386 ccd=100 --id visit=28388 ccd=101 --id visit=28390 ccd=102 --id visit=28392 ccd=100 --id visit=28392 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28394 ccd=101 --id visit=28396 ccd=90 --id visit=28396 ccd=102 --id visit=28398 ccd=10 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=5 --id visit=28400 ccd=10 --id visit=28400 ccd=15 --id visit=28400 ccd=18 --id visit=28400 ccd=23 --id visit=28400 ccd=26 --id visit=28400 ccd=28 --id visit=28400 ccd=40 --id visit=28400 ccd=53 --id visit=28400 ccd=55 --id visit=28400 ccd=61 --id visit=28400 ccd=68 --id visit=28400 ccd=77 --id visit=28400 ccd=80 --id visit=28400 ccd=84 --id visit=28400 ccd=89 --id visit=28400 ccd=93 --id visit=28400 ccd=94 --id visit=28400 ccd=95 --id visit=28400 ccd=99 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=28400 ccd=102 --id visit=28402 ccd=0 --id visit=28402 ccd=101 --id visit=29324 ccd=99 --id visit=28360 ccd=100 --id visit=28360 ccd=101 --id visit=9868 ccd=68 --id visit=9868 ccd=69 --id visit=9868 ccd=76 --id visit=9868 ccd=77 --id visit=9890 ccd=38 --id visit=9912 ccd=103 --id visit=11582 ccd=69 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=67 --id visit=6528 ccd=75 --id visit=9736 ccd=59 --id visit=9736 ccd=66 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17738 ccd=77 --id visit=17750 ccd=58

            Out of the 81 failures:

            • 62 failed with: "Unable to match sources"
            • 4 failed with: "No objects passed our cuts for consideration as psf stars"
            • 4 failed with: "No sources remaining in match list after magnitude limit cuts"
            • 11 failed with: "PSF star selector found [123] candidates"
            Show
            hchiang2 Hsin-Fang Chiang added a comment - In the single frame processing, among the 320 visits (103 ccds each, as ccd=9 was excluded), there were 81 reproducible fatal failures. Their dataIds are: --id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=1212 ccd=103 --id visit=24308 ccd=29 --id visit=28366 ccd=101 --id visit=28372 ccd=0 --id visit=28372 ccd=100 --id visit=28374 ccd=0 --id visit=28374 ccd=101 --id visit=28376 ccd=69 --id visit=28378 ccd=10 --id visit=28378 ccd=101 --id visit=28380 ccd=0 --id visit=28380 ccd=101 --id visit=28382 ccd=98 --id visit=28382 ccd=101 --id visit=28386 ccd=52 --id visit=28386 ccd=64 --id visit=28386 ccd=100 --id visit=28388 ccd=101 --id visit=28390 ccd=102 --id visit=28392 ccd=100 --id visit=28392 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28394 ccd=101 --id visit=28396 ccd=90 --id visit=28396 ccd=102 --id visit=28398 ccd=10 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=5 --id visit=28400 ccd=10 --id visit=28400 ccd=15 --id visit=28400 ccd=18 --id visit=28400 ccd=23 --id visit=28400 ccd=26 --id visit=28400 ccd=28 --id visit=28400 ccd=40 --id visit=28400 ccd=53 --id visit=28400 ccd=55 --id visit=28400 ccd=61 --id visit=28400 ccd=68 --id visit=28400 ccd=77 --id visit=28400 ccd=80 --id visit=28400 ccd=84 --id visit=28400 ccd=89 --id visit=28400 ccd=93 --id visit=28400 ccd=94 --id visit=28400 ccd=95 --id visit=28400 ccd=99 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=28400 ccd=102 --id visit=28402 ccd=0 --id visit=28402 ccd=101 --id visit=29324 ccd=99 --id visit=28360 ccd=100 --id visit=28360 ccd=101 --id visit=9868 ccd=68 --id visit=9868 ccd=69 --id visit=9868 ccd=76 --id visit=9868 ccd=77 --id visit=9890 ccd=38 --id visit=9912 ccd=103 --id visit=11582 ccd=69 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=67 --id visit=6528 ccd=75 --id visit=9736 ccd=59 --id visit=9736 ccd=66 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17738 ccd=77 --id visit=17750 ccd=58 Out of the 81 failures: 62 failed with: "Unable to match sources" 4 failed with: "No objects passed our cuts for consideration as psf stars" 4 failed with: "No sources remaining in match list after magnitude limit cuts" 11 failed with: "PSF star selector found [123] candidates"
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The rerun log of the 81 failed processCcd is attached.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The rerun log of the 81 failed processCcd is attached.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -
            Mosaic

            Current master of meas_mosaic (f4b19d1) built with w_2017_25 is used. All 3 tracts completed.

            Coadd

            Compared to DM-10129, not every patch has coadd products generated with w_2017_25. Now there are more calexps being removed from coaddition because "because size scatter is too large" or "because median e residual too large".

            COSMOS: In tract 9813, the coadd products have 75 patches in HSC-G, 74 in HSC-R, 79 in HSC-I, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921. The empty patches are:

            • HSC-G: 0,0^8,0^0,1^0,7^0,8^3,8
            • HSC-I: 0,0^0,8
            • HSC-R: 0,0^0,8^8,0^0,1^0,7^1,8^8,8
            • HSC-Y: 0,0^0,8
            • HSC-Z: 0,0^0,8
            • NB0921: 0,0^0,7^0,8^1,8^8,8

            Processing with stack w_2017_17 in DM-10129 did generate coadd files in 0,1 and 3,8 in the HSC-G filter. Otherwise the same.

            WIDE:
            In tract 8766 filter HSC-R, patches 7,8 and 0,8 are empty.
            In tract 8767 filter HSC-R, these patches are empty: 8,7^0,8^1,8^2,8^8,8

            MultiBand

            Due to the bug of DM-11091 from DM-9998, multiBandDriver would attempt to do coadd source detection on empty patches and fail because no coadd data exist in those patches. The error messages were like this: "NoResults: No locations for get: datasetType:deepCoadd dataId ...")

            For this ticket, I am continuing by only doing the multiband processing for patches where coadds of all filters exist. That is, specifying the safe patch IDs only in the multiBandDriver command line --id.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Mosaic Current master of meas_mosaic ( f4b19d1 ) built with w_2017_25 is used. All 3 tracts completed. Coadd Compared to DM-10129 , not every patch has coadd products generated with w_2017_25 . Now there are more calexps being removed from coaddition because "because size scatter is too large" or "because median e residual too large". COSMOS : In tract 9813, the coadd products have 75 patches in HSC-G, 74 in HSC-R, 79 in HSC-I, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921. The empty patches are: HSC-G: 0,0^8,0^0,1^0,7^0,8^3,8 HSC-I: 0,0^0,8 HSC-R: 0,0^0,8^8,0^0,1^0,7^1,8^8,8 HSC-Y: 0,0^0,8 HSC-Z: 0,0^0,8 NB0921: 0,0^0,7^0,8^1,8^8,8 Processing with stack w_2017_17 in DM-10129 did generate coadd files in 0,1 and 3,8 in the HSC-G filter. Otherwise the same. WIDE : In tract 8766 filter HSC-R, patches 7,8 and 0,8 are empty. In tract 8767 filter HSC-R, these patches are empty: 8,7^0,8^1,8^2,8^8,8 MultiBand Due to the bug of DM-11091 from DM-9998 , multiBandDriver would attempt to do coadd source detection on empty patches and fail because no coadd data exist in those patches. The error messages were like this: "NoResults: No locations for get: datasetType:deepCoadd dataId ...") For this ticket, I am continuing by only doing the multiband processing for patches where coadds of all filters exist. That is, specifying the safe patch IDs only in the multiBandDriver command line --id .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            As only patches where coadds of all filters exist, there are:

            • (COSMOS) 73 patches in tract=9813.
            • (WIDE) 79 patches in tract=8766 and 76 patches in 8767.
              and other patches are ignored in the multiband processing.

            The processed results are at this repo: /project/hsc_rc/DM-11020
            Logs are at: /project/hsc_rc/DM-11020/logs

            Show
            hchiang2 Hsin-Fang Chiang added a comment - As only patches where coadds of all filters exist, there are: (COSMOS) 73 patches in tract=9813. (WIDE) 79 patches in tract=8766 and 76 patches in 8767. and other patches are ignored in the multiband processing. The processed results are at this repo: /project/hsc_rc/ DM-11020 Logs are at: /project/hsc_rc/ DM-11020 /logs
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Everything here is moved to the new path /project/hsc_rc/w_2017_25/DM-11020 so it's easier to be found without the ticket number.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Everything here is moved to the new path /project/hsc_rc/w_2017_25/ DM-11020 so it's easier to be found without the ticket number.

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel