Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:3
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
https://github.com/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-RC2.yaml is currently our official HSC pipeline, but its step2 subset cheats in that it is a bag of tasks that are run globally (without dataquery constraints) on our 3-tract RC2 validation runs. It contains tasks that need to be global (FGCM), tract-level (jointcal/isolatedStarAssociation) and visit-level (skyCorr, consolidateVisitSummary, finalizeCharacterization). I say that it cheats because this is only doable because it's 3-tracts.
DP0.2 processing showed us that at 100's of sq deg, we can't submit 150-tracts all at once. The execution team's solution was to submit groups of visits and groups of tracts. As a courtesy. I propose splitting step2 for large production runs into it's visit-level and tract-level components:
step2a: # visit-Level
|
- consolidatePreSourceTable
|
- consolidateVisitSummary
|
- skyCorr
|
step2b: # tract-Level
|
- isolatedStarAssociation
|
- jointcal
|
step2c: # global (can be run any time after 2a)
|
- fgcmBuildStarsTable
|
- fgcmFitCycle
|
- fgcmOutputProducts
|
step2d: # visit-Level
|
- finalizeCharacterization
|
- writeRecalibratedSourceTable
|
- transformSourceTable
|
- consolidateSourceTable
|
this has the added benefit of communicating with NAOJ the order of imageCharacterization tasks that need to be run before coaddition.
Attachments
Issue Links
Activity
| Field | Original Value | New Value |
|---|---|---|
| Labels | hsc-pdr4 |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Reviewers | Kenneth Herner [ kherner ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
I promise to clean up the commits before merging. PR is a delta on tickets/DM-33959, will merge this to main, after I merge that to main.
Thank you Arun Kannawadi, Lee Kelvin, Kenneth Herner for the comments! I think I addressed them all and kicked off another jenkins run.
- https://ci.lsst.codes/job/stack-os-matrix/36697/display/redirect
- https://ci.lsst.codes/job/stack-os-matrix/36699/display/redirect
I'm assuming that puts this ticket into the "Reviewed" state.
If Jenkins is happy before 3am, I'll merge. If not, oh well.
...and we already know that DRP-Prod.yaml will need a reconfiguration from RC2 of FGCM to include i2 and r2.
Interesting fun fact about subsets:
DRP-RC2.yaml --> DRP-RC2+fakes.yaml --> DRP-ci_hsc+fakes.yaml
|
excl TE1, TE2, wPerp. excl faro_all (new)
|
In conclusion, because w22 gets tagged in 66 minutes, there isn't enough time for another ci_hsc on Jenkins, and I lose either way. Tomorrow I'll work on learning more about DRP-RC2+fakes.yaml and what's special about TE1,TE2,wPerp.
OK Lee Kelvin and Sophie Reed gave me their blessing to skip the faro tasks in the fakes pipeline.
New PR on ci_hsc_gen3. Jenkins running now. Ken, once you're happy, hit the reviewed button.
I'm happy once you've got a successful Jenkins run.
Kenneth Herner
added a comment - I'm happy once you've got a successful Jenkins run. Looks good modulo the pending Jenkins run.
Kenneth Herner
added a comment - Looks good modulo the pending Jenkins run. Kenneth Herner
made changes -
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
I'm calling https://ci.lsst.codes/job/stack-os-matrix/36761/display/redirect a success because both builds passed.
It was something transient the stack build: "Channel "unknown": Remote call on high_sierra-2 failed. The channel is closing down or has closed down"
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
| Epic Link |
|
|
| Story Points | 1 | 3 |
jenkins going here: https://ci.lsst.codes/job/stack-os-matrix/36687/display/redirect
This ticket only moves around subsets, doesn't change any pipeline definitions.