Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:2
-
Epic Link:
-
Sprint:AP S23-3 (February)
-
Team:Alert Production
-
Urgent?:No
Description
Discuss the purpose of tables such as the visit summary table that are produced in postprocess.py with DRP developers. Include a discussion of versioning of the contents, and consistency and redundancy between tables.
Attachments
Issue Links
- relates to
-
DM-37077 Add versioning to VisitSummary table
- Won't Fix
-
RFC-699 Add backwards compatibility testing and associated repo(s)
- Adopted
-
DM-36200 Add new fields to VisitSummary table
- To Do
-
DM-36214 Expand documentation of MakeCcdVisitTableTask and MakeVisitTableTask
- To Do
-
DM-36219 Add task topic documentation pages for postprocess.py tasks
- To Do
Summary of my conversation with Yusra and Jim:
We should treat VisitSummary as a code interface, not a data product, because it is a temporary internal product. We do have an explicit guarantee that you can read data produced with older code with new code; we do not have a guarantee that you can process old data with new code (for example, running difference imaging with old coadds, or postprocess/analysis_tools on old calexps/tables).
In the context of this ticket, this means that developers can add fields to VisitSummary and assume in downstream code-released concurrently with or after the addition-that those fields will exist, while the removal of fields from VisitSummary or significant changes to meanings requires an RFC and potentially a deprecation period. This also means that VisitSummary is explicitly not versioned: its effective version is tied directly to the version of Science Pipelines that was used to produce it and use it.
I don't know if the above is fully captured in RFC-699: that RFC mostly discusses reading of old data, but I had interpreted some of it to refer to further processing of older intermediate data products.