Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24534

Add source table pipelines to ci_hsc_gen2 & add its cat schema

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: cat, ci_hsc
    • Labels:
      None

      Description

      DM-24062 added the pipelines of generating the source tables in parquet.
      Add the pipelines to ci_hsc_gen2.

        Attachments

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Added a source table schema to cat based on the output parquet file. Added the 3 source table pipelines to ci_hsc_gen2, done per ccd for the WriteSourceTable & TransformSourceTable, per visit for ConsolidateSourceTable. The validation includes the simple check of the column names against the cat schema, just like how it's done with the object table. 

          Yusra AlSayyad may you please review it?

          Jenkins ongoing: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31714/pipeline 

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Added a source table schema to cat based on the output parquet file. Added the 3 source table pipelines to ci_hsc_gen2, done per ccd for the WriteSourceTable & TransformSourceTable, per visit for ConsolidateSourceTable. The validation includes the simple check of the column names against the cat schema, just like how it's done with the object table.  Yusra AlSayyad may you please review it? Jenkins ongoing:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31714/pipeline  
          Hide
          yusra Yusra AlSayyad added a comment -

          Looks good. Some github comments in both repos

          Show
          yusra Yusra AlSayyad added a comment - Looks good. Some github comments in both repos
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thank you for reviewing, especially for looking through the schema!

          Merged after Jenkins

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thank you for reviewing, especially for looking through the schema! Merged after Jenkins

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Hsin-Fang Chiang, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel