Why wouldn't the obs_*_data packages be available? They are part of lsst_distrib.

Also, can you explain how curated calibrations and skymaps interfere?

I think given the discussion on DM-29543, if you have the specific refcat already then your side would be to create the CSV ingest file and run butler ingest-files and butler register-dataset-type.

When we were developing the current convert+import system, there were some problems from the (gen 2) skymap associated with templates having a name collision with the (gen 3) skymap produced during repository initialization. I don't remember the details (perhaps the skymaps weren't quite identical?).

Following DM-31281, this ticket is to implement #3 and #4 from the list of steps needed:

1) Talk to cp_pipe folks (Chris W?) to learn best practices for running a default CP Pipeline on a new pile of raw calibs + science images for any given instrument
2) Write a "definitive" version of ApTemplate.yaml for building good-seeing templates from a pile raw science images for any given instrument, and double check the default skymap choices will work for us
3) Talk to refcat experts (John P?) to learn how to import an appropriate refcats collection to a new repository
4) Assemble the above into a script that turns a new pile of raw calibs + science images into an "ap_verify data set," presumably in the form of a single chained collection in a new repo, ready for ApVerify.yaml

In an attempt to break this up into more manageable pieces, I've moved #3 to its own issue, DM-32389. The revised scope of this issue is to combine the pieces of DM-32388, DM-31681, and DM-32389 to create a script that:

1. Creates a Gen 3 repository (the current preloaded) containing curated calibs. (As noted in the OP, we may want to remove these from the ap_verify dataset format, but that's best left for after we no longer need to sync with Gen 2.)
2. Creates and certifies master calibs for science data (in /repo/main:u/*?).
3. Imports these master calibs into preloaded.
4. Creates and certifies master calibs for template generation (in /repo/main:u/*?).
5. Creates templates (in /repo/main:u/*?).
6. Imports these templates into preloaded.
7. Imports refcats into preloaded.

The issue is considered complete once we have created and tested the script. We should not actually commit the results to ap_verify datasets while we are still supporting Gen 2, to ensure that the Gen 2 and Gen 3 repositories are exactly equivalent – reprocessing the calibs and templates with current pipelines will almost certainly change the results!

Thank you for agreeing to review this, Eric Bellm. The primary development was done in ap_verify_ci_hits2015; the scripts in ap_verify_hits2015 are nearly identical copies.

I've still got the CI generate_all_gen3.sh script running but it seems to be executing fine--I'm going to go ahead and mark this as approved for now so it is unblocked and will check back in on the off chance something goes astray.

I did wonder if there are any changes to the installation instructions needed at https://pipelines.lsst.io/modules/lsst.ap.verify/datasets-install.html to handle the gen3-only case?

Note that, as written in the comments, the CI generate_all_gen3.sh takes 10 hours to run.

Dataset installation is unaffected by this transition, as the basic format of the datasets (i.e., as EUPS packages) hasn't changed. Other documentation changes, if any, will be covered on DM-33150.