Details
-
Type:
Epic
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Epic Name:Batch Processing System Phase 1a
-
Story Points:55
-
WBS:02C.07.01
-
Team:Data Facility
-
Cycle:Fall 2017
Description
Summary: Scale-up DESDM prototype to HSC-sized processing. The prototype uses the DESDM processing and file management framework, which locates inputs via query on metadata, manages inputs and outputs, executes jobs, and collects provenance. The prototype runs pipeline payloads as command line tasks. Extending the prototype includes running real queries and submitting jobs to Slurm cluster.
Work may include:
- Setting up test database instance for querying on metadata and possibly ingesting catalog data.
- Installing DES framework software stack on Verification Cluter.
- Configuring runs, including metadata to be collected.
- Running scaling tests (e.g., RC-sized to PDR1-sized).
- Comparing outputs to parallel reprocessing efforts.
- Identifying possible modifications needed for handling LSST workloads.
Deliverable: output datasets and summary report.
Attachments
| Key | Summary | Story Points | Assignee | Status | |
|---|---|---|---|---|---|
|
|
8 | Michelle Gower | Done | ||
|
|
8 | Michelle Gower | Done | ||
|
|
Investigate ingesting a couple HSC catalogs using existing DESDM software |
4 | Michelle Gower | Done | |
|
|
1 | Michelle Gower | Done |
Activity
Deliverable: output datasets and summary report.
Where is the summary report?
Please link the indicated report to the epic closure.
I would also be interested in the output datasets/provenance databases.
Due to revised project priorities and milestone goals, this epic is on hold. The work will be revisited toward the end of FY2018.
This work has progressed to a point were it is stable under python3. Development has been halted, but can be resumed if needed as a risk mitigation path.
Initial investigation and use of DESDM code to process LSST data has been completed far enough to determine that we can and will continue to use the DESDM code for near-future work. Further work continues in active and planned epics.