Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:None
-
Story Points:12
-
Epic Link:
-
Sprint:AP F21-3 (August), AP F21-4 (September), AP F21-5 (October)
-
Team:Alert Production
Description
As implemented in DM-21919, ap_verify requires a lot of in-code hacks to properly configure a dataset in Gen 3 (including specification of an instrument). Find a way to delegate as much of this as possible to something in each dataset that acts like a "pipeline config".
The description is deliberately vague because this kind of delegation is harder than it sounds; see discussion in #dm-middleware.
Attachments
Issue Links
- has to be done before
-
DM-27117 Add dummy pipeline to ap_verify testing
- Done
- is blocked by
-
DM-21919 Run ap_verify end-to-end in Gen 3
- Done
-
DM-32080 Pipelines import parameters in the wrong order
- Done
- relates to
-
DM-32726 Inconsistent overriding in ap_verify
- Invalid
-
DM-27659 Adopt new calibs and templates for ap_verify HiTS datasets
- Done
-
DM-29221 Add ApPipe.yaml pipelines to appropriate repos
- Done
-
DM-31492 Identify possible ways to unify DECam crosstalk processing into one pipeline
- To Do
-
DM-29344 Investigate the CI differences between Gen 2 and 3 in COSMOS field
- Done
Thanks for tackling this, Krzysztof, and apologies for the delay. I think this ticket realizes the vague idea for pipeline organization I had in my head, but I didn't quite have enough yaml subset pipeline-writing experience to implement from the get-go.
One question that will need a followup ticket is when we should use the new lsst.ip.diffim.getTemplate.GetMultiTractCoaddTemplateTask + lsst.pipe.tasks.imageDifference.ImageDifferenceFromTemplateTask instead of just lsst.pipe.tasks.imageDifference.ImageDifferenceTask.
That workflow is necessary for multi-tract differencing, has its own config + DM-30210 complications, etc.
The only things I'm confused about are exactly how the camera-specific ApVerify pipelines with fakes are supposed to work (see GitHub comments on the ap_verify PR).
I defer to Nate for any suggestions on making framework things better, but I'm very pleased with the ApPipe pipeline standardization, and as far as I can see, both it and the ApVerify pipelines do what they should without too many complicated import trains.