Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-672

Include cat (as sdm_schemas) in lsst_distrib

    Details

    • Type: RFC
    • Status: Adopted
    • Resolution: Unresolved
    • Component/s: DM
    • Labels:
      None

      Description

      cat (https://github.com/lsst/cat) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis (https://felis.lsst.io) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the cat package to know the expected schemas of the catalogs.

      We foresee more schemas to be added to cat and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: cat/yml/hsc.yaml    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the cat definitions is done in ci_hsc_gen2. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update cat/yml/hsc.yml accordingly.

      As the schema definition is tied with the upstream pipelines, it would be nice if cat is versioned and gets weekly tags like other Science Pipelines packages. This also makes using ci_hsc_gen2 easier (because ci_hsc_gen2 is versioned but its dependency, cat, is not.) Including cat in lsst_distrib ensures they are tagged consistently.

      (added) Following discussion in other fora, it is proposed that as part of this RFC the repository be renamed to sdm_schemas, with appropriate documentation in the repository and elsewhere that ensures that users understand what "SDM" signifies (Science Data Model) and what the role of this repo is in that regard.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Colin Slater, Gregory Dubois-Felsmann, Hsin-Fang Chiang, John Parejko, John Swinbank, Tim Jenness
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Planned End:

                  Summary Panel