This should be pretty straightforward, though it does require running two pipelines. Note to self: now that I finished DM-29221, the necessary pipelines are all in ap_pipe/pipelines/DarkEnergyCamera.

Doing this now as part of the DM-29422 HiTS rerun. Caveat: not rebuilding calibs right now, so technically the biases and flats I'm using were built back in the day when gen3 linearity and crosstalk didn't work at all. A natural followup to coming-soon plans to add DECam data to /repo/main ought to address this for the future.

A bps run of this scale turned out to be impractical for my ad-hoc sqlite3-based DECam repo. More than half of the quanta failed due to a locked DB. Now with DM-28638 complete, the necessary HiTS datasets have landed in /repo/main! So the plan for this ticket is changed as follows:

• build new calibs for 2014 and 2015 datasets (including linearity and crosstalk!)
• run "ApTemplate.yaml" (again with linearity and crosstalk!) on 2014 dataset using the new good seeing selector ... hello DM-29842
• run AP Pipeline in bps on 2015 dataset (you guessed it, with linearity and crosstalk!)

Status update time. I built at least some HiTS2014 good seeing coadds, for use as templates. I broke it down into a 2-step process (with an intermediate "step 1.5" to do consolidateVisitSummary since some of the dataIds fail to find a bias or flat to use in ISR which is a whole other thing I haven't looked into yet). Anyway, the coadds are in /repo/main in the collection u/mrawls/
DM-29338-templates-step2/20210525T212412Z.
 
I had to be extremely careful about which input collections and data queries I used to get selectGoodSeeingVisits, makeWarp, and assembleCoadd to run without spending days in the quantum-graph-building stage with a zillion-line sql query. In particular, once I'm past single frame processing, I specified only the exact RUN collections I know I need instead of using any CHAINED collections besides refcats and skymaps. I also specified the skymap name in the data query.
 
A quick inspection of a handful of the coadds and the N-images suggests some tracts look fine while others have big regions where only 1 warp was selected. (Overall the 'hsc_rings_v1' default /repo/main skymap has much smaller tracts and patches than the skymap I am used to for HiTS, and this will undoubtedly cause problems along the lines of DM-29310 when I get to the difference imaging step since there are many tracts involved).
 
I started a first attempt at running the actual AP Pipeline this evening. There was a minor hiccup in which my postgres APDB default schema appeared to be `public` instead of `mrawls` (see IHS-4861) but I think it was an easy fix. Here's hoping the quantum graph takes less than 24 hours to build and the bps run actually kicks off. The bps submit file I used is /project/mrawls/repo-main-logs/ApPipe_bps.yaml and with any luck the output collection will be u/mrawls/DM-29338-ApPipe.

The pipeline ran, but encountered a non-negligible pile of bugs along the way. Most notably there is no APDB, but there are difference images! Nevertheless, I am ready to close this ticket out with necessary pipeline update PRs. I will investigate the various bugs that happened during the bps run on DM-30495, DM-30496, and DM-30497.

Thank you, it looks like the Gen 3 crosstalk part of the pipeline is working, and we have tickets for the remaining bugs so this can be marked Done.

The PRs are all reviewed now as well.