Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23224

Cross-check the schema column names in the Object table

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None

      Description

      Now that the Object table schema of the output parquet does not depend on what filters the input data include (DM-23074), we can do checks of the schema in ci_hsc and the schema should be the same in the HSC-RC2 outputs. 

      My plan in this ticket is to add a cross-check of the schema (column names only) between the Object parquet generated in ci_hsc and the yaml ddl in cat.  

      It will be nice that the DAX team can use the yaml file in cat as the expected schema. So far nothing checks the schema of the pipeline outputs and this ticket will start adding that.  Science Pipelines can still change the schema from time to time, but that's okay, because a failed ci_hsc should prompt the developers to update the cat yaml file. 

      I plan to leave the data type checking for later (after the end-of-Feb DMLT).

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Colin Slater please review. 

            This doesn't check much, but having something to close the loop is probably better than nothing. 

            I'm also changing the yaml file name in cat to hsc.yml as now it is the default HSC Object table schema. 

            Yusra AlSayyad I'm tagging you on GH to review the Object.yaml change in obs_subaru: I am changing the column name Dec to Decl.  In SQL Dec is special so we rename it to Decl on the Qserv side.  If changing it on the Sci Pi side is inconvenient we may keep them different too.  

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Colin Slater please review.  This doesn't check much, but having something to close the loop is probably better than nothing.  I'm also changing the yaml file name in cat to hsc.yml as now it is the default HSC Object table schema.  Yusra AlSayyad I'm tagging you on GH to review the Object.yaml change in obs_subaru: I am changing the column name Dec to Decl .  In SQL Dec is special so we rename it to Decl on the Qserv side.  If changing it on the Sci Pi side is inconvenient we may keep them different too.  
            Hide
            ctslater Colin Slater added a comment -

            Validation is a good idea and the code looks good, just two suggestions on the PR.

            Do we need to do something to make sure ci_hsc versions are properly matched to versions of cat? I assume cat isn't getting the weekly tag or anything, so I don't know how one would run an older ci_hsc and get the right cat version.

            Show
            ctslater Colin Slater added a comment - Validation is a good idea and the code looks good, just two suggestions on the PR. Do we need to do something to make sure ci_hsc versions are properly matched to versions of cat? I assume cat isn't getting the weekly tag or anything, so I don't know how one would run an older ci_hsc and get the right cat version.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you for your fast review. 

            About tagging cat with stack releases, maybe I can file an RFC to get cat added to lsst_distrib?  I'll create a new ticket for that. 

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you for your fast review.  About tagging cat with stack releases, maybe I can file an RFC to get cat added to lsst_distrib?  I'll create a new ticket for that. 
            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31244/pipeline
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Merged

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Merged
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Filed DM-23529 for adding cat to lsst_distrib

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Filed DM-23529 for adding cat to lsst_distrib

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Colin Slater
                Watchers:
                Colin Slater, Hsin-Fang Chiang, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel