Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10702

HSC reprocessing campaigns

    Details

    • Templates:
    • Epic Name:
      HSC reprocessing campaigns
    • Story Points:
      67
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Fall 2017

      Description

      Reprocessing of large (RC, full PDR1) datasets for service development and process improvement and to provide datasets useful for ongoing pipeline development.
      Work includes:

      • Working with Science Pipelines to define processing campaigns based on changes in the stack to be tested at scale and other goals.
      • Gathering and analyzing operational metrics, such as execution time.
      • Verifying data products generated by campaigns.
      • Monitoring compute and storage systems to ensure adequate capacity for upcoming processing runs; managing files.
      • Identifying and helping to debug issues with service elements (software, services depended on by the batch processing service, etc.).
      • Process and procedure improvement to mature the service facade, gain operational experience with pipelines and emerging framework features.
        Deliverable: output datasets.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                plutchak Joel Plutchak (Inactive)
                Watchers:
                Hsin-Fang Chiang, Joel Plutchak (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Start date:
                  End date:

                  Summary Panel