Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10129

Process HSC RC dataset using Stack version chosen for the full S17B reprocessing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Process the RC dataset using the stack version that the full S17B HSC reprocessing will be based on. Presumably this version will be effectively the same or comparable to the version HSC chooses for their internal data processing.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Per discussion on DM-10237, the blocking relationship there is weak — if we're pressed for time, we can go without it.

            Show
            swinbank John Swinbank added a comment - Per discussion on DM-10237 , the blocking relationship there is weak — if we're pressed for time, we can go without it.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            John Swinbank, I'll start processing the RC dataset using the stack w_2017_17; when I have the processed data, I'll inform Tim Morton so he can do QA with them.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - John Swinbank , I'll start processing the RC dataset using the stack w_2017_17 ; when I have the processed data, I'll inform Tim Morton so he can do QA with them.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In singleFrameDriver, there are 46 reproducible failures in 46 ccds from 23 visits. The failed visit/ccds are the same as those in the w_2017_14 stack (DM-10084). Their data IDs are:

            --id visit=278 ccd=95 --id visit=280 ccd=22^69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=6478 ccd=99 --id visit=6528 ccd=24^67 --id visit=7344 ccd=67 --id visit=9736 ccd=67 --id visit=9868 ccd=76 --id visit=17738 ccd=69 --id visit=17750 ccd=58 --id visit=19468 ccd=69 --id visit=24308 ccd=29 --id visit=28376 ccd=69 --id visit=28380 ccd=0 --id visit=28382 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28396 ccd=102 --id visit=28398 ccd=95^101 --id visit=28400 ccd=5^10^15^23^26^40^53^55^61^68^77^84^89^92^93^94^95^99^100^101^102 --id visit=29324 ccd=99 --id visit=29326 ccd=47

            Out of the 46 failures:

            • 27 failed with "Unable to match sources"
            • 11 failed with "PSF star selector found [123] candidates"
            • 4 failed with "No sources remaining in match list after magnitude limit cuts."
            • 4 failed with ""No objects passed our cuts for consideration as ps stars" (3 of these 4 have "Detected 0 positive sources to 50 sigma")
            Show
            hchiang2 Hsin-Fang Chiang added a comment - In singleFrameDriver, there are 46 reproducible failures in 46 ccds from 23 visits. The failed visit/ccds are the same as those in the w_2017_14 stack ( DM-10084 ). Their data IDs are: --id visit=278 ccd=95 --id visit=280 ccd=22^69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=6478 ccd=99 --id visit=6528 ccd=24^67 --id visit=7344 ccd=67 --id visit=9736 ccd=67 --id visit=9868 ccd=76 --id visit=17738 ccd=69 --id visit=17750 ccd=58 --id visit=19468 ccd=69 --id visit=24308 ccd=29 --id visit=28376 ccd=69 --id visit=28380 ccd=0 --id visit=28382 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28396 ccd=102 --id visit=28398 ccd=95^101 --id visit=28400 ccd=5^10^15^23^26^40^53^55^61^68^77^84^89^92^93^94^95^99^100^101^102 --id visit=29324 ccd=99 --id visit=29326 ccd=47 Out of the 46 failures: 27 failed with "Unable to match sources" 11 failed with "PSF star selector found [123] candidates" 4 failed with "No sources remaining in match list after magnitude limit cuts." 4 failed with ""No objects passed our cuts for consideration as ps stars" (3 of these 4 have "Detected 0 positive sources to 50 sigma")
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The processed RC data are accessible as a butler repo at
            /datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10129
            (It cannot be read with its real path because of DM-10268)

            MosaicTask was run, using meas_mosaic Apr27 master and the w_2017_17 stack. The diagnostics plots were saved in /project/hsc_rc/DM-10129_mosaic_diag.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The processed RC data are accessible as a butler repo at /datasets/hsc/repo/rerun/private/hchiang2/RC/ DM-10129 (It cannot be read with its real path because of DM-10268 ) MosaicTask was run, using meas_mosaic Apr27 master and the w_2017_17 stack. The diagnostics plots were saved in /project/hsc_rc/ DM-10129 _mosaic_diag .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            WIDE: The coadd products have all 81 patches in both tracts (8766, 8767) in 5 filters, except that there is no coadd in tract 8767 patch 1,8 in HSC-R (nothing passed the PSF quality selection there); the multiband products of all 162 patches are generated.

            COSMOS: In tract 9813, the coadd products have 77 patches in HSC-G, 74 in HSC-R, 79 in HSC-I, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; the multiband products of 79 patches are generated.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - WIDE : The coadd products have all 81 patches in both tracts (8766, 8767) in 5 filters, except that there is no coadd in tract 8767 patch 1,8 in HSC-R (nothing passed the PSF quality selection there); the multiband products of all 162 patches are generated. COSMOS : In tract 9813, the coadd products have 77 patches in HSC-G, 74 in HSC-R, 79 in HSC-I, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; the multiband products of 79 patches are generated.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Add Tim Morton as the reviewer for any basic QA or science validation the DRP team may wish to perform, or changes needed before the full S17B PDR1 reprocessing starts. Please mark as Reviewed if the output repo looks fine and the version/setup/configs are good for the full S17B PDR1 reprocessing. (More details on https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Add Tim Morton as the reviewer for any basic QA or science validation the DRP team may wish to perform, or changes needed before the full S17B PDR1 reprocessing starts. Please mark as Reviewed if the output repo looks fine and the version/setup/configs are good for the full S17B PDR1 reprocessing. (More details on https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing ).
            Hide
            tmorton Tim Morton added a comment -

            The output repo looks fine to me; in that I was able to successfully produce the full set of QA plots (DM-10044). Bob Armstrong will look at those to make sure there's nothing catastrophically wrong, but I don't expect there to be, and from our discussion at the group meeting yesterday, the consensus is that you can go ahead with the processing with your current set of configs, as the goal of this exercise is the mechanics of the large-scale exercise more than producing science-quality data results (in fact, there do seem to be some residual bugs in meas_mosaic). Thus, I will go ahead and mark this reviewed so you can get started, bugs and all.

            Show
            tmorton Tim Morton added a comment - The output repo looks fine to me; in that I was able to successfully produce the full set of QA plots ( DM-10044 ). Bob Armstrong will look at those to make sure there's nothing catastrophically wrong, but I don't expect there to be, and from our discussion at the group meeting yesterday, the consensus is that you can go ahead with the processing with your current set of configs, as the goal of this exercise is the mechanics of the large-scale exercise more than producing science-quality data results (in fact, there do seem to be some residual bugs in meas_mosaic). Thus, I will go ahead and mark this reviewed so you can get started, bugs and all.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for the review.

            If meas_mosaic has known important bugs, should I not run mosaic? Or is it worse not running mosaic?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for the review. If meas_mosaic has known important bugs, should I not run mosaic? Or is it worse not running mosaic?
            Hide
            tmorton Tim Morton added a comment -

            I can't give a good answer for this... As far as I can tell from the discussion on #subaru-hsc on slack, people aren't yet sure whether the problems are in mosaic or not. If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

            Show
            tmorton Tim Morton added a comment - I can't give a good answer for this... As far as I can tell from the discussion on #subaru-hsc on slack, people aren't yet sure whether the problems are in mosaic or not. If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?
            Hide
            swinbank John Swinbank added a comment -

            If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

            Tim guesses correctly.

            While we know there are some outstanding bugs that will affect the quality of the final results, my hope & expectation is that this is just the first of many such large-scale runs. As such, the primary aim here is to demonstrate that as much of the machinery is working as possible and to understand the issues we'll face running at scale on LSST hardware. Waiting until all the bugs are fixed will delay us hitting that goal — but after we've shown we can do this once, doing it N more times with progressively less buggy stacks should be "easy".

            Show
            swinbank John Swinbank added a comment - If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word? Tim guesses correctly. While we know there are some outstanding bugs that will affect the quality of the final results, my hope & expectation is that this is just the first of many such large-scale runs. As such, the primary aim here is to demonstrate that as much of the machinery is working as possible and to understand the issues we'll face running at scale on LSST hardware. Waiting until all the bugs are fixed will delay us hitting that goal — but after we've shown we can do this once, doing it N more times with progressively less buggy stacks should be "easy".
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'll close this ticket and continue the final tweaks in DM-9800. The repo of processed data will stay in /project until I'm ask to remove it (likely when we are short of space).
            Butler repo path: /datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10129

            Logs are stored in /project/hsc_rc/DM-10129/logs/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'll close this ticket and continue the final tweaks in DM-9800 . The repo of processed data will stay in /project until I'm ask to remove it (likely when we are short of space). Butler repo path: /datasets/hsc/repo/rerun/private/hchiang2/RC/ DM-10129 Logs are stored in /project/hsc_rc/ DM-10129 /logs/

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Tim Morton
                Watchers:
                Hsin-Fang Chiang, John Swinbank, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel