https://github.com/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-RC2.yaml is currently our official HSC pipeline, but its step2 subset cheats in that it is a bag of tasks that are run globally (without dataquery constraints) on our 3-tract RC2 validation runs. It contains tasks that need to be global (FGCM), tract-level (jointcal/isolatedStarAssociation) and visit-level (skyCorr, consolidateVisitSummary, finalizeCharacterization). I say that it cheats because this is only doable because it's 3-tracts.
DP0.2 processing showed us that at 100's of sq deg, we can't submit 150-tracts all at once. The execution team's solution was to submit groups of visits and groups of tracts. As a courtesy. I propose splitting step2 for large production runs into it's visit-level and tract-level components:
step2a: # visit-Level
|
- consolidatePreSourceTable
|
- consolidateVisitSummary
|
- skyCorr
|
step2b: # tract-Level
|
- isolatedStarAssociation
|
- jointcal
|
step2c: # global (can be run any time after 2a)
|
- fgcmBuildStarsTable
|
- fgcmFitCycle
|
- fgcmOutputProducts
|
step2d: # visit-Level
|
- finalizeCharacterization
|
- writeRecalibratedSourceTable
|
- transformSourceTable
|
- consolidateSourceTable
|
this has the added benefit of communicating with NAOJ the order of imageCharacterization tasks that need to be run before coaddition.