Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10701

Batch Processing System Phase 1a (DES)

    Details

    • Epic Name:
      Batch Processing System Phase 1a
    • Story Points:
      55
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Fall 2017

      Description

      Summary: Scale-up DESDM prototype to HSC-sized processing. The prototype uses the DESDM processing and file management framework, which locates inputs via query on metadata, manages inputs and outputs, executes jobs, and collects provenance. The prototype runs pipeline payloads as command line tasks. Extending the prototype includes running real queries and submitting jobs to Slurm cluster.
      Work may include:

      • Setting up test database instance for querying on metadata and possibly ingesting catalog data.
      • Installing DES framework software stack on Verification Cluter.
      • Configuring runs, including metadata to be collected.
      • Running scaling tests (e.g., RC-sized to PDR1-sized).
      • Comparing outputs to parallel reprocessing efforts.
      • Identifying possible modifications needed for handling LSST workloads.
        Deliverable: output datasets and summary report.

        Attachments

        Stories in Epic (Custom Issue Matrix)

          Activity

          Hide
          plutchak Joel Plutchak (Inactive) added a comment -

          Initial investigation and use of DESDM code to process LSST data has been completed far enough to determine that we can and will continue to use the DESDM code for near-future work. Further work continues in active and planned epics.

          Show
          plutchak Joel Plutchak (Inactive) added a comment - Initial investigation and use of DESDM code to process LSST data has been completed far enough to determine that we can and will continue to use the DESDM code for near-future work. Further work continues in active and planned epics.
          Hide
          tjenness Tim Jenness added a comment -

          Deliverable: output datasets and summary report.

          Where is the summary report?

          Show
          tjenness Tim Jenness added a comment - Deliverable: output datasets and summary report. Where is the summary report?
          Hide
          womullan Wil O'Mullane added a comment -

          Please link the indicated report to the epic closure.

          Show
          womullan Wil O'Mullane added a comment - Please link the indicated report to the epic closure.
          Hide
          krughoff Simon Krughoff added a comment -

          I would also be interested in the output datasets/provenance databases.

          Show
          krughoff Simon Krughoff added a comment - I would also be interested in the output datasets/provenance databases.
          Hide
          plutchak Joel Plutchak (Inactive) added a comment - - edited

          Due to revised project priorities and milestone goals, this epic is on hold. The work will be revisited toward the end of FY2018.

          Show
          plutchak Joel Plutchak (Inactive) added a comment - - edited Due to revised project priorities and milestone goals, this epic is on hold. The work will be revisited toward the end of FY2018.
          Hide
          plutchak Joel Plutchak (Inactive) added a comment -

          This work has progressed to a point were it is stable under python3. Development has been halted, but can be resumed if needed as a risk mitigation path.

          Show
          plutchak Joel Plutchak (Inactive) added a comment - This work has progressed to a point were it is stable under python3. Development has been halted, but can be resumed if needed as a risk mitigation path.

            People

            • Assignee:
              plutchak Joel Plutchak (Inactive)
              Reporter:
              plutchak Joel Plutchak (Inactive)
              Watchers:
              Fabio Hernandez, Joel Plutchak (Inactive), Simon Krughoff, Tim Jenness, Wil O'Mullane
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Start date:
                End date:

                Summary Panel