Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8332

Run selected science pipeline using Pegasus

    Details

    • Epic Name:
      Run selected science pipeline using Pegasus
    • Story Points:
      108
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Spring 2017

      Description

      Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

      • Provide integration software needed for HTCondor + Pegasus
      • Instantiate WLMS Pegasus+HTCondor
      • Provide ability to run HTCondor glide-ins on Slurm
      • Prepare common middleware layer(s) for managed database services
      • Write Pegasus plugin in orchestration

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-8634

        Generalize DAX generator to handle arbitrary workflows

        10 Mikolaj Kowalik Done
         
        DM-8633

        Prototype a wrapper generating DAXs for sequential workflows

        10 Mikolaj Kowalik Done
         
        DM-8632

        Use Pegasus and Executor to run processCcd

        10 Mikolaj Kowalik Done
         
        DM-8618

        Clean commit history and move Executor's code to lsst-dm GitHub organization

        0.5 Mikolaj Kowalik Done
         
        DM-8603

        Expand the Pegasus-workflow-generating script to consider multiple patches

        10 Hsin-Fang Chiang Done
         
        DM-8599

        Create a Pegasus-workflow repo in GitHub lsst-dm

        0.2 Hsin-Fang Chiang Done
         
        DM-8823

        Create Pegasus plugin for Orca

        16 Steve Pietrowicz Done
         
        DM-8947

        Remove dependency on CAT package and make database setup optional

        4 Steve Pietrowicz Done
         
        DM-8339

        Make a Pegasus workflow based on ci_hsc and Fall2016 interface

        12 Hsin-Fang Chiang Done
         
        DM-8392

        Run prototype executor-less ci_hsc Pegasus workflow on Verification Cluster

        2 Hsin-Fang Chiang Done
         
        DM-8391

        Run example Pegasus workflow on Verification Cluster

        2 Hsin-Fang Chiang Done
         
        DM-9235

        Use Pegasus to do ProcessCcd with a subset of HSC COSMOS data on /datasets/hsc

        1 Hsin-Fang Chiang Done
         
        DM-9209

        Allow Executor to work with pre-exisiting dataset repositories

        1 Mikolaj Kowalik Done
         
        DM-9207

        Change Executor's data passing mechanism

        10 Mikolaj Kowalik Done
         
        DM-9010

        Investigate stack-like installation requirements for Pegasus

        3 Greg Daues Done
         
        DM-9092

        Create a simplified workflow based on the ciHsc workflow

        0.5 Hsin-Fang Chiang Done
         
        DM-9234

        Clean up the "inputData" file used in the DM-8603 dax generating script

        Hsin-Fang Chiang Won't Fix

          Activity

          Hide
          plutchak Joel Plutchak (Inactive) added a comment -

          A single workflow runs end to end with limited interaction with orchestration. Work continues this cycle under DM-9662.

          Show
          plutchak Joel Plutchak (Inactive) added a comment - A single workflow runs end to end with limited interaction with orchestration. Work continues this cycle under DM-9662 .

            People

            • Assignee:
              mkowalik Mikolaj Kowalik
              Reporter:
              plutchak Joel Plutchak (Inactive)
              Watchers:
              Joel Plutchak (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Start date:
                End date:

                Summary Panel