Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14568

Transform DRP pipeline outputs to DPDD-specified format in F18

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      f18-drp-dpdd
    • Story Points:
      42
    • WBS:
      02C.04.07
    • Team:
      Data Release Production
    • Cycle:
      Spring 2019

      Description

      Post-processing DRP catalog outputs to match definitions in DPDD.

      Success criteria: qserv team can ingest HSC data and meet milestone DM-DAX-5

        Attachments

          Issue Links

          duplicates

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-2880 Create transformation task for CalibrateTask outputs

          • Invalid
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-243 Request for a regularly updated pipeline output dataset

          • Adopted

          Stories in Epic (Custom Issue Matrix)

          Key Summary Story Points Assignee Status
           
          		DM-15073

          Initial functor.yaml file for DPDD Object table

          		 10 Yusra AlSayyad Done
           
          		DM-16234

          Move qa_explorer/postprocess.py and writeObjectTable to pipe_tasks

          		 12 Yusra AlSayyad Done
           
          		DM-14289

          Update qa_explorer visualization code to use the new ParquetTable datasets

          		 16 Tim Morton [X] (Inactive) Done

            Activity

            Hide
            Permalink
            yusra Yusra AlSayyad added a comment - - edited

            Design and background presented to DMLT (2018): https://github.com/yalsayyad/dm_notebooks/blob/master/object-table/DPDD.ipynb

            Conclusion presented to DMLT (2019): https://docs.google.com/presentation/d/1kTz91UGMzxV_TA5_vXc-8vGhWifnLI2DZWBatf9qZME/edit#slide=id.g70708e1de4_0_11

            on this epic we Implemented tasks to match catalogs, write parquet tables, and transform units/quantities, concatenate full tract. This enabled the ingestion of a transformed  DRP Object Table to qserv. 

            Show
            yusra Yusra AlSayyad added a comment - - edited Design and background presented to DMLT (2018):  https://github.com/yalsayyad/dm_notebooks/blob/master/object-table/DPDD.ipynb Conclusion presented to DMLT (2019):  https://docs.google.com/presentation/d/1kTz91UGMzxV_TA5_vXc-8vGhWifnLI2DZWBatf9qZME/edit#slide=id.g70708e1de4_0_11 on this epic we Implemented tasks to match catalogs, write parquet tables, and transform units/quantities, concatenate full tract. This enabled the ingestion of a transformed  DRP Object Table to qserv. 

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Fritz Mueller, Hsin-Fang Chiang, John Swinbank, Xiuqin Wu [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.