Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
Description
cat (https://github.com/lsst/cat) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis (https://felis.lsst.io) format. The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the cat package to know the expected schemas of the catalogs.
We foresee more schemas to be added to cat and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: cat/yml/hsc.yaml This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the cat definitions is done in ci_hsc_gen2. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update cat/yml/hsc.yml accordingly.
As the schema definition is tied with the upstream pipelines, it would be nice if cat is versioned and gets weekly tags like other Science Pipelines packages. This also makes using ci_hsc_gen2 easier (because ci_hsc_gen2 is versioned but its dependency, cat, is not.) Including cat in lsst_distrib ensures they are tagged consistently.
(added) Following discussion in other fora, it is proposed that as part of this RFC the repository be renamed to sdm_schemas, with appropriate documentation in the repository and elsewhere that ensures that users understand what "SDM" signifies (Science Data Model) and what the role of this repo is in that regard.
Attachments
Issue Links
Activity
Hsin-Fang Chiang
added a comment - I've created DM-23614 for adding more documentation to the cat package and it blocks DM-23529 . The proposal here is still to add cat , as its new name sdm_schemas , to lsst_distrib (not just tagging it).
I'm delaying the RFC end date to give us more time to discuss this. Not entirely sure why this is the case but at the moment ci_hsc_gen2 depends on the cat package. Is that a real dependency?
The pipeline writes the object table and it's not trivial to know its schema without looking at the outputs. In ci_hsc_gen2 we want to check if the schema of the pipeline-generated object table matches the one we think it has, so we read cat/yml/hsc.yaml (the expected schema) and verify that.
Hsin-Fang Chiang
added a comment - The pipeline writes the object table and it's not trivial to know its schema without looking at the outputs. In ci_hsc_gen2 we want to check if the schema of the pipeline-generated object table matches the one we think it has, so we read cat/yml/hsc.yaml (the expected schema) and verify that. Hsin-Fang Chiang
added a comment - As there have been no new comments or objections, I'm adopting this.
We will add more documentations in the repository ( DM-23614 ), and afterwards we will add the package to lsst_distrib as sdm_schemas ( DM-23529 ). 
Qserv ingest was on my mind, but don't make this request too specific.
The uneducated user — certainly including me — looking at this package sees a bunch of YAML with no very clear idea of what it is or what it's for. Before this gets added to lsst_distrib, there needs to be a clear description of what it is we're actually looking at and why we care.