Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_pipe
-
Story Points:4
-
Epic Link:
-
Sprint:AP F19-6 (November), AP S20-6 (May), AP F20-1 (June)
-
Team:Alert Production
Description
This is a mostly-umbrella ticket for getting all current Gen2 ap_pipe functionality working in Gen3. The goal is being able to run AP on HiTS 2015 using the CmdLineActivator. It includes:
- creating scripts that ingest HiTS test data into a Gen3 repository (
DM-21862) - conversion of all remaining ApPipeTask subtasks to Gen 3 (
DM-21874,DM-21886) - creation of a YAML file configuring the pipeline (1 SP on this ticket)
ApPipeTask has a lot of code for handling the case of calexp difference imaging templates. By general agreement within the AP and Middleware groups, we will not be porting this functionality to Gen 3 (see also DM-21874), and will reimplement it from scratch should we need it after migration.
Attachments
Issue Links
- blocks
-
DM-21888 Stand up Gen3 version of ap_verify HiTS (2015) CI test
- Done
-
DM-21919 Run ap_verify end-to-end in Gen 3
- Done
- is blocked by
-
DM-23992 Cannot load refcats in Gen 3 DECam processing
- Done
-
DM-25014 Coadds converted to Gen 3 without abstract_filter
- Done
-
DM-25040 ap_association uses physical filter in Gen 3
- Done
-
DM-21862 Extend Gen3 butler support for obs_decam
- Done
-
DM-21874 Create PipelineTask version of backward compatible ImageDifferenceTask
- Done
-
DM-21886 Create PipelineTask driver for ap_pipe tasks that interact with the APDB
- Done
-
DM-22741 Remove ap_association subtasks from ap_pipe. Change to single DiaPipelineTask subtask.
- Done
-
DM-24260 Create Gen3 versions of ap_verify datasets
- Done
- relates to
-
DM-21915 Support Gen 3 ingestion of ap_verify datasets
- Done
-
DM-22599 Develop PipelineTask unit test framework
- Done
-
DM-22663 Reimplement make_apdb.py for Gen 3
- Done
-
DM-12549 ap_pipe must call AssociationTask in a reproducible order
- Done
I left straightforward comments on GitHub for the ap_pipe docs, and pending those, it looks good. I'll take the liberty of marking this as "reviewed" for all three of us.
One somewhat larger question - is the Gen 3 implementation done here kind of an intermediate step while we still need to support Gen 2? It's my vague understanding that eventually, running the AP Pipeline will just be having a yaml file with "do ISR, do characterization, do calibration, do differencing, do association, the end" and we won't need an ap_pipe package to link those steps together. Maybe something to discuss at our weekly meeting if I'm off base. For now, I'm thrilled it works!