Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-672

Include cat (as sdm_schemas) in lsst_distrib

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      cat (https://github.com/lsst/cat) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis (https://felis.lsst.io) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the cat package to know the expected schemas of the catalogs.

      We foresee more schemas to be added to cat and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: cat/yml/hsc.yaml    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the cat definitions is done in ci_hsc_gen2. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update cat/yml/hsc.yml accordingly.

      As the schema definition is tied with the upstream pipelines, it would be nice if cat is versioned and gets weekly tags like other Science Pipelines packages. This also makes using ci_hsc_gen2 easier (because ci_hsc_gen2 is versioned but its dependency, cat, is not.) Including cat in lsst_distrib ensures they are tagged consistently.

      (added) Following discussion in other fora, it is proposed that as part of this RFC the repository be renamed to sdm_schemas, with appropriate documentation in the repository and elsewhere that ensures that users understand what "SDM" signifies (Science Data Model) and what the role of this repo is in that regard.

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Link This issue is triggering DM-23529 [ DM-23529 ]
            tjenness Tim Jenness made changes -
            Description {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, s/he will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.
            {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.
            hchiang2 Hsin-Fang Chiang made changes -
            Description {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.
            {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.
            hchiang2 Hsin-Fang Chiang made changes -
            Summary Include cat in lsst_distrib Include cat (sdm_schemas) in lsst_distrib
            gpdf Gregory Dubois-Felsmann made changes -
            Description {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.
            {{cat}} ([https://github.com/lsst/cat]) is a very light package and contains the schema definitions of the catalog data products. This specifies (or will specify) the Science Data Model (SDM) of the LSST data release catalogs. The definitions are in the felis ([https://felis.lsst.io|https://felis.lsst.io/]) format.    The idea is that other LSST components, such as database systems (Qserv), TAP services, validation tools, etc., can look up the {{cat}} package to know the expected schemas of the catalogs.

            We foresee more schemas to be added to {{cat}} and eventually it will cover all DPDD catalogs. As of today, there is only 1 table schema that the Science Pipelines team might care about: [cat/yml/hsc.yaml|https://github.com/lsst/cat/blob/master/yml/hsc.yaml]    This yaml file describes the Object table that is generated by the Science Pipelines (writeObjectTable.py/transformObjectCatalog.py/consolidateObjectTable.py) and that will be used in Qserv catalog loading. Currently a minimal check of the schema column names between the pipeline outputs and the {{cat}} definitions is done in {{ci_hsc_gen2}}. If somebody wants to merge a pipeline ticket that will change the column names, they will also need to update {{cat/yml/hsc.yml}} accordingly.

            As the schema definition is tied with the upstream pipelines, it would be nice if {{cat}} is versioned and gets weekly tags like other Science Pipelines packages. This also makes using {{ci_hsc_gen2}} easier (because {{ci_hsc_gen2}} is versioned but its dependency, {{cat}}, is not.) Including {{cat}} in {{lsst_distrib}} ensures they are tagged consistently.

            (added) Following discussion in other fora, it is proposed that as part of this RFC the repository be renamed to {{sdm_schemas}}, with appropriate documentation in the repository and elsewhere that ensures that users understand what "SDM" signifies (Science Data Model) and what the role of this repo is in that regard.
            gpdf Gregory Dubois-Felsmann made changes -
            Summary Include cat (sdm_schemas) in lsst_distrib Include cat (as sdm_schemas) in lsst_distrib
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggering DM-23614 [ DM-23614 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Planned End 26/Feb/20 7:00 PM 28/Feb/20 7:00 PM
            hchiang2 Hsin-Fang Chiang made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 23395 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]
            gcomoretto Gabriele Comoretto [X] (Inactive) made changes -
            Remote Link This issue links to "Page (Confluence)" [ 25155 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Colin Slater, Gregory Dubois-Felsmann, Hsin-Fang Chiang, John Parejko, John Swinbank, Tim Jenness
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.