Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ap_verify
-
Labels:
-
Story Points:8
-
Epic Link:
-
Sprint:AP S22-1 (December)
-
Team:Alert Production
-
Urgent?:No
Description
Currently, ap_verify datasets are primarily based on uningested files, which are processed in Gen 2 and converted to Gen 3 as needed. This workflow will no longer work once Gen 2 is removed; we need a pure Gen 3 workflow to maintain and create datasets.
Exact work TBD, but the following constraints apply:
- raws are still held in uningested form; no changes needed
- curated calibs are currently stored in a Gen 3 repository as a necessary artifact of the Gen 2 conversion process. It may be worth removing them from the repository and instead handling them from scratch during dataset ingestion; this would reduce dataset churn and download size (at the cost of longer ingestion). However, check whether this works in Jenkins; the obs_*_data package might not be available.
- non-curated calibs are now stored in a Gen 3 repository, as there is no such thing as "calib ingestion". Presumably we would need to run the CP pipeline when making or updating these; should they be certified into dataset/preloaded, or export-imported?
- templates would need to be created into a repository, as in Gen 2. Our Gen 2 approach of directory copying won't work; export-import should be fine, given our past experience with running the hybrid data set.
- skymaps are produced by template generation, but potentially conflict with curated calib handling. It may be worth creating a script for getting the skymap right just to avoid any pitfalls.
- Refcats are not yet supported as of April 2021.
Attachments
Issue Links
- blocks
-
DM-20572 Remove "Generation 2" Middleware
- Done
-
DM-33150 Remove Gen 2 support from ap_verify
- Done
-
DM-32975 Investigate omitting curated calibs from ap_verify dataset
- Done
-
DM-33095 Streamline updating of ap_verify CI datasets
- Invalid
- is blocked by
-
DM-31281 Determine scope of work for creating pure Gen 3 scripts
- Done
-
DM-31681 Write a "definitive" version of ApTemplate.yaml
- Done
-
DM-32388 Create calib generation script for ap_verify datasets
- Done
-
DM-32389 Create refcat management script for ap_verify datasets
- Done
- relates to
-
DM-29856 Switch ap_verify.py to Gen 3 by default
- Done
-
DM-30624 Document gen2 to gen3 refcat ingestion
- Done
-
DM-30199 Mark ap_verify dataset files as generated
- Won't Fix
Why wouldn't the obs_*_data packages be available? They are part of lsst_distrib.