Details
-
Type:
Epic
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Epic Name:HSC reprocessing campaigns
-
Story Points:67
-
WBS:02C.07.01
-
Team:Data Facility
-
Cycle:Fall 2017
Description
Reprocessing of large (RC, full PDR1) datasets for service development and process improvement and to provide datasets useful for ongoing pipeline development.
Work includes:
- Working with Science Pipelines to define processing campaigns based on changes in the stack to be tested at scale and other goals.
- Gathering and analyzing operational metrics, such as execution time.
- Verifying data products generated by campaigns.
- Monitoring compute and storage systems to ensure adequate capacity for upcoming processing runs; managing files.
- Identifying and helping to debug issues with service elements (software, services depended on by the batch processing service, etc.).
- Process and procedure improvement to mature the service facade, gain operational experience with pipelines and emerging framework features.
Deliverable: output datasets.
Attachments
Issue Links
- relates to
-
DM-11345 Redefine HSC "RC" dataset for bi-weeklies processing
- Done
Key | Summary | Story Points | Assignee | Status | |
---|---|---|---|---|---|
|
|
8 | Hsin-Fang Chiang | Done | |
|
|
2 | Samantha Thrush | Done | |
|
|
4 | Samantha Thrush | Done | |
|
|
4 | Hsin-Fang Chiang | Done | |
|
|
1 | Hsin-Fang Chiang | Done | |
|
|
4 | Hsin-Fang Chiang | Done | |
|
|
4 | Samantha Thrush | Done | |
|
|
4 | Hsin-Fang Chiang | Done | |
|
|
analyze the storage usage of the output butler repos from S17B reprocessing |
1 | Samantha Thrush | Done |
|
|
analyze the storage usage of an HSC RC reprocessing output repo |
1 | Samantha Thrush | Done |
|
|
4 | Hsin-Fang Chiang | Done | |
|
|
Analyze and summarize computing resource used in S17B full HSC reprocessing |
5 | Hsin-Fang Chiang | Done |