Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22806

Automation of end-to-end system from science pipeline outputs to Qserv ingest

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Epic Name:
      sst-qserv-ingest-automation
    • Story Points:
      85
    • Team:
      DM Science
    • Cycle:
      Spring 2020

      Description

      This epic covers all aspects of the automation of the process to take science pipelines outputs, produce parquet files that are properly configured in the SDM format, and automatically ingest them into Qserv.  This includes development of automated workflow and tooling to automatically ingest data products generated by the LSST science pipelines into Qserv. The initial implementation for automated ingest will be based around the reprocessed HSC data products given that the HSC end-to-end ingest process has already been executed manually. 

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

            Activity

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                lguy Leanne Guy
                Watchers:
                Fritz Mueller, Gregory Dubois-Felsmann, Igor Gaponenko, Jeffrey Carlin, Leanne Guy, Wil O'Mullane, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:

                  Summary Panel