Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10702

HSC reprocessing campaigns

    XMLWordPrintable

    Details

    • Epic Name:
      HSC reprocessing campaigns
    • Story Points:
      67
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Fall 2017

      Description

      Reprocessing of large (RC, full PDR1) datasets for service development and process improvement and to provide datasets useful for ongoing pipeline development.
      Work includes:

      • Working with Science Pipelines to define processing campaigns based on changes in the stack to be tested at scale and other goals.
      • Gathering and analyzing operational metrics, such as execution time.
      • Verifying data products generated by campaigns.
      • Monitoring compute and storage systems to ensure adequate capacity for upcoming processing runs; managing files.
      • Identifying and helping to debug issues with service elements (software, services depended on by the batch processing service, etc.).
      • Process and procedure improvement to mature the service facade, gain operational experience with pipelines and emerging framework features.
        Deliverable: output datasets.

        Attachments

          Issue Links

            Activity

            plutchak Joel Plutchak (Inactive) created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Child DM-10649 [ 32601 ]
            plutchak Joel Plutchak (Inactive) made changes -
            Description  Reprocessing of large (RC, full PDR1) datasets for service development and process improvement and to provide datasets useful for ongoing pipeline development.
            Work includes:
            - Working with Science Pipelines to define processing campaigns based on changes in the stack to be tested at scale and other goals.
            - Gathering and analyzing operational metrics, such as execution time.
            - Verifying data products generated by campaigns.
            - Monitoring compute and storage systems to ensure adequate capacity for upcoming processing runs; managing files.
            - Identifying and helping to debug issues with service elements (software, services depended on by the batch processing service, etc.).
            - Process and procedure improvement to mature the service facade, gain operational experience with pipelines and emerging framework features.
            Deliverable: output datasets.
            plutchak Joel Plutchak (Inactive) made changes -
            End date 31/Jan/17 31/Aug/17
            Start date 01/Jan/17 01/Jun/17
            plutchak Joel Plutchak (Inactive) made changes -
            Story Points 62 67
            plutchak Joel Plutchak (Inactive) made changes -
            Cycle Fall 2017 [ 10805 ]
            plutchak Joel Plutchak (Inactive) made changes -
            Labels Batch_Production_Services
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-10904 [ 32946 ]
            plutchak Joel Plutchak (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11020 [ 33119 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11165 [ 33292 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11184 [ 33324 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11185 [ 33325 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11186 [ 33326 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11187 [ 33327 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11189 [ 33329 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11189 [ 33329 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11273 [ 33446 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11290 [ 33466 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-11345 [ DM-11345 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11413 [ 33659 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11640 [ 34036 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11365 [ 33597 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Child DM-11187 [ 33327 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The next cycle of the HSC reprocessing is in DM-11839

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The next cycle of the HSC reprocessing is in DM-11839
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              plutchak Joel Plutchak (Inactive)
              Watchers:
              Dino Bektesevic, Hsin-Fang Chiang, Joel Plutchak (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Start date:
                End date:

                  Summary Panel