Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10702

HSC reprocessing campaigns

    XMLWordPrintable

    Details

    • Epic Name:
      HSC reprocessing campaigns
    • Story Points:
      67
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Fall 2017

      Description

      Reprocessing of large (RC, full PDR1) datasets for service development and process improvement and to provide datasets useful for ongoing pipeline development.
      Work includes:

      • Working with Science Pipelines to define processing campaigns based on changes in the stack to be tested at scale and other goals.
      • Gathering and analyzing operational metrics, such as execution time.
      • Verifying data products generated by campaigns.
      • Monitoring compute and storage systems to ensure adequate capacity for upcoming processing runs; managing files.
      • Identifying and helping to debug issues with service elements (software, services depended on by the batch processing service, etc.).
      • Process and procedure improvement to mature the service facade, gain operational experience with pipelines and emerging framework features.
        Deliverable: output datasets.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              plutchak Joel Plutchak (Inactive)
              Watchers:
              Dino Bektesevic, Hsin-Fang Chiang, Joel Plutchak (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Start date:
                End date:

                  Summary Panel