We believe Krzysztof Findeisen will be blocked by the need for some smoke-test data, so we created https://jira.lsstcorp.org/browse/DM-24260. The scope of this ticket is then to perform appropriate QA activities to identify a scientifically meaningful HSC CI subset (e.g., one not unduly contaminated by ghosts or bright stars), and the replace the stub dataset with those visits.

Dependent on how big we want this to be, we do have a halfway decent test data subset we've been running for the 2021 diffim sprint.

g-band
11690, 11692, 11694, 11696, 11698, 11700, 11702, 11704, 11706, 11708, 11710, 11712, 29324, 29326, 29336, 29340, 29350

r-band
1202, 1204, 1206, 1208, 1210, 1212, 1214, 1216, 1218, 1220, 23692, 23694, 23704, 23706, 23716, 23718

for each visit above, only these detectors (ccds)
49, 50, 57, 58, 65, 66

I am leaning toward adopting this dataset after we deprecate Gen 2, because it will take significant time to get it to work properly for both Gen2 and Gen3.

I'm not sure we want the full 33 visits x 6 detectors listed above? That seems a little big for CI. Meanwhile, if someone does want this, it's borderline trivial to run the AP Pipeline (though not ap_verify) by specifying the above exposures and detectors, and I'm available to assist.

I'll note the current HSC CI dataset is tiny and boring (2 visits, 1 detector each), but it has been adequate to diagnose if we have Gen2-Gen3 agreement for now.

The CI dataset needs to remain small--roughly the current size. As I commented above, the current purpose of this ticket is to determine whether the current CI dataset that Krzysztof implemented more-or-less at random is actually giving us meaningful results, or if it is dominated by edge/bright star/ghosts/other artifacts.

If the current CI dataset is indeed "boring"-ie, it's not dominated by artifacts-then I think we can just close this as Won't Fix. But let's look.

What are the extra difficulties in getting it to work in Gen 2? Note that we won't be able to avoid a Gen 2 element completely until DM-29857/DM-29543 is resolved, which won't be for a while.

Personally, I'd like to expand it to at least 3 epochs like with the HiTS dataset, just because 2-epoch association doesn't exercise all functionality.

Consensus at the May 10 AP meeting was that the current visits looked fine, we'd do as Krzysztof Findeisen suggested and add a few more epochs, but that it could wait potentially for a Gen3-only future.

Leanne Guy's team has defined an "RC2_subset" of COSMOS/HSC which they are using with faro: 6 detectors x 8 visits--might be useful to adopt some of those visits.

More details here: https://github.com/lsst-dm/rc2_subset. This dataset will be moved into github.com/lsst as per other datasets for testing. I'll get the full documentation updated in dmtn-091.