Per discussion on DM-10237, the blocking relationship there is weak — if we're pressed for time, we can go without it.

John Swinbank, I'll start processing the RC dataset using the stack w_2017_17; when I have the processed data, I'll inform Tim Morton [X] so he can do QA with them.

In singleFrameDriver, there are 46 reproducible failures in 46 ccds from 23 visits. The failed visit/ccds are the same as those in the w_2017_14 stack (DM-10084). Their data IDs are:

--id visit=278 ccd=95 --id visit=280 ccd=22^69 --id visit=284 ccd=61 --id visit=1206 ccd=77 --id visit=6478 ccd=99 --id visit=6528 ccd=24^67 --id visit=7344 ccd=67 --id visit=9736 ccd=67 --id visit=9868 ccd=76 --id visit=17738 ccd=69 --id visit=17750 ccd=58 --id visit=19468 ccd=69 --id visit=24308 ccd=29 --id visit=28376 ccd=69 --id visit=28380 ccd=0 --id visit=28382 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28396 ccd=102 --id visit=28398 ccd=95^101 --id visit=28400 ccd=5^10^15^23^26^40^53^55^61^68^77^84^89^92^93^94^95^99^100^101^102 --id visit=29324 ccd=99 --id visit=29326 ccd=47

Out of the 46 failures:

• 27 failed with "Unable to match sources"
• 11 failed with "PSF star selector found [123] candidates"
• 4 failed with "No sources remaining in match list after magnitude limit cuts."
• 4 failed with ""No objects passed our cuts for consideration as ps stars" (3 of these 4 have "Detected 0 positive sources to 50 sigma")

The processed RC data are accessible as a butler repo at
/datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10129
(It cannot be read with its real path because of DM-10268)

MosaicTask was run, using meas_mosaic Apr27 master and the w_2017_17 stack. The diagnostics plots were saved in /project/hsc_rc/DM-10129_mosaic_diag.

WIDE: The coadd products have all 81 patches in both tracts (8766, 8767) in 5 filters, except that there is no coadd in tract 8767 patch 1,8 in HSC-R (nothing passed the PSF quality selection there); the multiband products of all 162 patches are generated.

COSMOS: In tract 9813, the coadd products have 77 patches in HSC-G, 74 in HSC-R, 79 in HSC-I, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; the multiband products of 79 patches are generated.

Add Tim Morton [X] as the reviewer for any basic QA or science validation the DRP team may wish to perform, or changes needed before the full S17B PDR1 reprocessing starts. Please mark as Reviewed if the output repo looks fine and the version/setup/configs are good for the full S17B PDR1 reprocessing. (More details on https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing).

The output repo looks fine to me; in that I was able to successfully produce the full set of QA plots (DM-10044). Bob Armstrong will look at those to make sure there's nothing catastrophically wrong, but I don't expect there to be, and from our discussion at the group meeting yesterday, the consensus is that you can go ahead with the processing with your current set of configs, as the goal of this exercise is the mechanics of the large-scale exercise more than producing science-quality data results (in fact, there do seem to be some residual bugs in meas_mosaic). Thus, I will go ahead and mark this reviewed so you can get started, bugs and all.

Thanks for the review.

If meas_mosaic has known important bugs, should I not run mosaic? Or is it worse not running mosaic?

I can't give a good answer for this... As far as I can tell from the discussion on #subaru-hsc on slack, people aren't yet sure whether the problems are in mosaic or not. If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

If I had to guess, John Swinbank would probably suggest to run with mosaic at this point, but perhaps he can give the final word?

Tim guesses correctly.

While we know there are some outstanding bugs that will affect the quality of the final results, my hope & expectation is that this is just the first of many such large-scale runs. As such, the primary aim here is to demonstrate that as much of the machinery is working as possible and to understand the issues we'll face running at scale on LSST hardware. Waiting until all the bugs are fixed will delay us hitting that goal — but after we've shown we can do this once, doing it N more times with progressively less buggy stacks should be "easy".

I'll close this ticket and continue the final tweaks in DM-9800. The repo of processed data will stay in /project until I'm ask to remove it (likely when we are short of space).
Butler repo path: /datasets/hsc/repo/rerun/private/hchiang2/RC/DM-10129

Logs are stored in /project/hsc_rc/DM-10129/logs/