Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-672

Include cat (as sdm_schemas) in lsst_distrib

    XMLWordPrintable

Details

    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM
    • None

    Description

      cat (https://github.com/lsst/cat) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis (https://felis.lsst.io) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the cat package to know the expected schemas of the catalogs.

      We foresee more schemas to be added to cat and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: cat/yml/hsc.yaml    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the cat definitions is done in ci_hsc_gen2. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update cat/yml/hsc.yml accordingly.

      As the schema definition is tied with the upstream pipelines, it would be nice if cat is versioned and gets weekly tags like other Science Pipelines packages. This also makes using ci_hsc_gen2 easier (because ci_hsc_gen2 is versioned but its dependency, cat, is not.) Including cat in lsst_distrib ensures they are tagged consistently.

      (added) Following discussion in other fora, it is proposed that as part of this RFC the repository be renamed to sdm_schemas, with appropriate documentation in the repository and elsewhere that ensures that users understand what "SDM" signifies (Science Data Model) and what the role of this repo is in that regard.

      Attachments

        Issue Links

          Activity

            John Swinbank since "ingestion" is such an overloaded term.... which "ingestion" are you thinking in this case? Not Qserv ingest right?

            Qserv ingest was on my mind, but don't make this request too specific.

            The uneducated user — certainly including me — looking at this package sees a bunch of YAML with no very clear idea of what it is or what it's for. Before this gets added to lsst_distrib, there needs to be a clear description of what it is we're actually looking at and why we care.

            swinbank John Swinbank added a comment - John Swinbank since "ingestion" is such an overloaded term.... which "ingestion" are you thinking in this case? Not Qserv ingest right? Qserv ingest was on my mind, but don't make this request too specific. The uneducated user — certainly including me — looking at this package sees a bunch of YAML with no very clear idea of what it is or what it's for. Before this gets added to lsst_distrib, there needs to be a clear description of what it is we're actually looking at and why we care.

            I've created DM-23614 for adding more documentation to the cat package and it blocks DM-23529. The proposal here is still to add cat, as its new name sdm_schemas, to lsst_distrib (not just tagging it).

            I'm delaying the RFC end date to give us more time to discuss this.

            hchiang2 Hsin-Fang Chiang added a comment - I've created DM-23614 for adding more documentation to the cat package and it blocks DM-23529 . The proposal here is still to add cat , as its new name sdm_schemas , to lsst_distrib (not just tagging it). I'm delaying the RFC end date to give us more time to discuss this.
            tjenness Tim Jenness added a comment -

            Not entirely sure why this is the case but at the moment ci_hsc_gen2 depends on the cat package. Is that a real dependency?

            tjenness Tim Jenness added a comment - Not entirely sure why this is the case but at the moment ci_hsc_gen2 depends on the cat package. Is that a real dependency?

            The pipeline writes the object table and it's not trivial to know its schema without looking at the outputs. In ci_hsc_gen2 we want to check if the schema of the pipeline-generated object table matches the one we think it has, so we read cat/yml/hsc.yaml (the expected schema) and verify that.

            hchiang2 Hsin-Fang Chiang added a comment - The pipeline writes the object table and it's not trivial to know its schema without looking at the outputs. In ci_hsc_gen2 we want to check if the schema of the pipeline-generated object table matches the one we think it has, so we read cat/yml/hsc.yaml (the expected schema) and verify that.

            As there have been no new comments or objections, I'm adopting this.  

            We will add more documentations in the repository (DM-23614), and afterwards we will add the package to lsst_distrib as  sdm_schemas (DM-23529).  

            hchiang2 Hsin-Fang Chiang added a comment - As there have been no new comments or objections, I'm adopting this.   We will add more documentations in the repository ( DM-23614 ), and afterwards we will add the package to lsst_distrib as   sdm_schemas ( DM-23529 ).  

            People

              hchiang2 Hsin-Fang Chiang
              hchiang2 Hsin-Fang Chiang
              Colin Slater, Gregory Dubois-Felsmann, Hsin-Fang Chiang, John Parejko, John Swinbank, Tim Jenness
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.