Notes from discussion today with Krzysztof
- Something like ApTemplate.yaml should be run before ApVerify.yaml, and data products created as templates should be taken as inputs
- What input data products can we expect cp_pipe to provide during operations (that ap_verify can also use)?
- We need some kind of refcat solution, i.e.,
DM-29543
We should assume there exist curated calibs, master calibs, raw science images, refcats, and template images on some reasonable skymap. This is similar to how we designed ap_verify in Gen2. One challenge will be to get all these prerequisite data products into known Butler collections we can use as inputs, without accidentally grabbing extraneous data products we don't want to use as inputs.
Instrument-specific concerns must be handled with care.
- e.g., DECam will need overscan-corrected science images as a prerequisite along with everything else
These collections are currently chained together into a "defaults" input collection in ap_verify:
- instrument/calib (includes curated calibs and master calibs; could include other tags by ticket or date)
- refcats
- skymaps
- templates/deep (could be goodSeeing, or other tags by ticket or date)
The goal here is a single script that can take a pile of new images and create a proper "ap_verify data set" which may be run through ApVerify.yaml, and roughly approximates an operations-era AP pipeline.
-------
Steps to get there
1) Talk to cp_pipe folks (Chris W?) to learn best practices for running a default CP Pipeline on a new pile of raw calibs + science images for any given instrument
2) Write a "definitive" version of ApTemplate.yaml for building good-seeing templates from a pile raw science images for any given instrument, and double check the default skymap choices will work for us
3) Talk to refcat experts (John P?) to learn how to import an appropriate refcats collection to a new repository
4) Assemble the above into a script that turns a new pile of raw calibs + science images into an "ap_verify data set," presumably in the form of a single chained collection in a new repo, ready for ApVerify.yaml
Story
Notes from discussion today with Krzysztof
DM-29543We should assume there exist curated calibs, master calibs, raw science images, refcats, and template images on some reasonable skymap. This is similar to how we designed ap_verify in Gen2. One challenge will be to get all these prerequisite data products into known Butler collections we can use as inputs, without accidentally grabbing extraneous data products we don't want to use as inputs.
Instrument-specific concerns must be handled with care.
These collections are currently chained together into a "defaults" input collection in ap_verify:
The goal here is a single script that can take a pile of new images and create a proper "ap_verify data set" which may be run through ApVerify.yaml, and roughly approximates an operations-era AP pipeline.
-------
Steps to get there
1) Talk to cp_pipe folks (Chris W?) to learn best practices for running a default CP Pipeline on a new pile of raw calibs + science images for any given instrument
2) Write a "definitive" version of ApTemplate.yaml for building good-seeing templates from a pile raw science images for any given instrument, and double check the default skymap choices will work for us
3) Talk to refcat experts (John P?) to learn how to import an appropriate refcats collection to a new repository
4) Assemble the above into a script that turns a new pile of raw calibs + science images into an "ap_verify data set," presumably in the form of a single chained collection in a new repo, ready for ApVerify.yaml