Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22806

Automation of end-to-end system from science pipeline outputs to Qserv ingest

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sst-qserv-ingest-automation
    • Story Points:
      85
    • Team:
      DM Science
    • Cycle:
      Spring 2020

      Description

      This epic covers all aspects of the automation of the process to take science pipelines outputs, produce parquet files that are properly configured in the SDM format, and automatically ingest them into Qserv.  This includes development of automated workflow and tooling to automatically ingest data products generated by the LSST science pipelines into Qserv. The initial implementation for automated ingest will be based around the reprocessed HSC data products given that the HSC end-to-end ingest process has already been executed manually. 

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
          DM-32022

          add index to column list

          Hsin-Fang Chiang In Review
          DM-30278

          Implement the first part of the ingest workflow in Argo

          Hsin-Fang Chiang In Review
           
          DM-29935

          For dp01_dc2_catalogs, add the flags in config json so to build the index automatically

          0.1 Hsin-Fang Chiang Done
           
          DM-29839

          Ingest dp01_dc2_catalogs on IDF qserv instances

          4 Hsin-Fang Chiang Done
           
          DM-29744

          Qserv Ingest configuration files for DP0.1 database at CC-IN2P3

          2 Hsin-Fang Chiang Done
           
          DM-29104

          Correct sdm_schemas and do not use the TEXT type in qserv schemas

          Hsin-Fang Chiang Done
           
          DM-29439

          Ingest DC2 Truth Match table into NCSA's small Qserv

          4 Hsin-Fang Chiang Done
           
          DM-30181

          Add a bucket as the Argo artifact repository in qserv-dev

          2 Hsin-Fang Chiang Done
           
          DM-28986

          Make metadata json for dc2_object_run2.2i_dr6_v2_dpdd_only

          2 Hsin-Fang Chiang Done
           
          DM-28883

          Ingest dc2_object_run2.2i_dr6_v2_dpdd_only into Qserv at NCSA

          7 Hsin-Fang Chiang Done
           
          DM-30295

          Ingest HSC-RC2 w_2021_18 Object tables into the "small" Qserv cluster at NCSA

          4 Hsin-Fang Chiang Done
           
          DM-27383

          Ingest HSC-RC2 w_2020_42 Object tables into the "small" Qserv cluster at NCSA

          4 Hsin-Fang Chiang Done
           
          DM-27172

          Ingest HSC-RC2 w_2020_38 Object tables into the "small" Qserv cluster at NCSA

          5 Hsin-Fang Chiang Done
           
          DM-28440

          Create a felis schema for the DC2 tables from IN2P3

          2 Hsin-Fang Chiang Done
           
          DM-28432

          Ingest HSC-RC2 w_2021_02 Object tables into the "small" Qserv cluster at NCSA

          2 Hsin-Fang Chiang Done
           
          DM-28692

          Make json metadata for the DC2 DR6 DPDD-only object table

          2 Hsin-Fang Chiang Done
           
          DM-27862

          Ingest a set of DC2 data into small Qserv

          4 Hsin-Fang Chiang Done
           
          DM-27770

          Write a report on Qserv Ingest of HSC-RC2 Objects

          5 Hsin-Fang Chiang Done
           
          DM-27769

          Ingest HSC-RC2 v21_0_0_rc1 Object tables into the "small" Qserv cluster at NCSA

          4 Hsin-Fang Chiang Done
           
          DM-29043

          Make the TAP schema for dp01_dc2

          2 Hsin-Fang Chiang Done
           
          DM-31055

          The length is not long enough for the "skymap" column

          Hsin-Fang Chiang Done
           
          DM-31052

          Ingest HSC-RC2 w_2021_22 & w_2021_26 Object tables into the "small" Qserv cluster at NCSA

          Hsin-Fang Chiang Done
           
          DM-31048

          Clean up un-wanted databases in NCSA Small Qserv

          Hsin-Fang Chiang Done
           
          DM-31039

          Fix the wrong method 'startwith' in parquet_tools

          0.2 Hsin-Fang Chiang Done
           
          DM-30884

          Fix schema inconsistency in hsc.yaml

          Hsin-Fang Chiang Done
           
          DM-26395

          Ingest HSC-RC2 w_2020_30 Object tables into the "small" Qserv cluster at NCSA

          2 Hsin-Fang Chiang Done
           
          DM-26068

          Ingest HSC-PDR2 Object tables into Qserv

          4 Hsin-Fang Chiang Done
           
          DM-26065

          Ingest v20_0_0_rc1 Object tables of HSC-RC2 into Qserv

          2 Hsin-Fang Chiang Done
           
          DM-22008

          Learn basic qserv ingest workflow

          5 Hsin-Fang Chiang Done
           
          DM-23763

          Ingest w_2020_07 Object tables of HSC-RC2 into Qserv

          4 Hsin-Fang Chiang Done
           
          DM-24396

          Ingest 3 DESC-generated tables into Qserv

          7 Hsin-Fang Chiang Done
           
          DM-24049

          Ingest w_2020_11 Object tables of HSC-RC2 into Qserv

          4 Hsin-Fang Chiang Done
           
          DM-23963

          Ingest a test HSC w_2020_08 Object table into Qserv

          3 Hsin-Fang Chiang Done
           
          DM-22371

          Add post-processing tasks to ci_hsc_gen2

          3 Hsin-Fang Chiang Done
           
          DM-23529

          Add cat to lsst_distrib (as sdm_schemas)

          2 Hsin-Fang Chiang Done
           
          DM-21498

          Make a set of parquet and csv Object tables of HSC-RC2 rerun

          6 Hsin-Fang Chiang Done
           
          DM-21821

          Generate Object parquet file from DC2 reprocessed data

          1 Hsin-Fang Chiang Done
           
          DM-25479

          Adopt the new python code at lsst-dm/qserv-ingest for ingesting HSC Object tables

          6 Hsin-Fang Chiang Done
           
          DM-25238

          Ingest w_2020_19 Object tables of HSC-RC2 into Qserv

          2 Hsin-Fang Chiang Done
           
          DM-24656

          Ingest w_2020_14 Object tables of HSC-RC2 into Qserv

          7 Hsin-Fang Chiang Done
           
          DM-26781

          Fix duplicate IDs in sdm_schemas/yml/hsc.yaml

          1 Hsin-Fang Chiang Done
           
          DM-26760

          Ingest HSC-PDR2 Object tables into small Qserv

          2 Hsin-Fang Chiang Done
           
          DM-24534

          Add source table pipelines to ci_hsc_gen2 & add its cat schema

          5 Hsin-Fang Chiang Done
           
          DM-23224

          Cross-check the schema column names in the Object table

          4 Hsin-Fang Chiang Done
           
          DM-23207

          Clean up the cat package

          1 Hsin-Fang Chiang Done
           
          DM-23074

          Make the schema of the output Object parquet files input-independent

          4 Hsin-Fang Chiang Done
           
          DM-22483

          Ingest w_2019_46/47 Object tables of HSC-RC2 into Qserv

          4 Hsin-Fang Chiang Done
           
          DM-20753

          Make Felis schema for HSC RC2 reprocessing

          4 Hsin-Fang Chiang Done
           
          DM-30665

          Add Source and Object schema_checks to ci_imsim

          Hsin-Fang Chiang Done
           
          DM-22809

          Learn basic Airflow

          Hsin-Fang Chiang Won't Fix

            Activity

            There are no comments yet on this issue.

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              lguy Leanne Guy
              Watchers:
              Fritz Mueller, Gregory Dubois-Felsmann, Igor Gaponenko, Jeffrey Carlin, Leanne Guy, Wil O'Mullane, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Due:
                Created:
                Updated:

                  Summary Panel

                    CI Builds

                    No builds found.