Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8332

Run selected science pipeline using Pegasus

    XMLWordPrintable

    Details

    • Epic Name:
      Run selected science pipeline using Pegasus
    • Story Points:
      108
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Spring 2017

      Description

      Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

      • Provide integration software needed for HTCondor + Pegasus
      • Instantiate WLMS Pegasus+HTCondor
      • Provide ability to run HTCondor glide-ins on Slurm
      • Prepare common middleware layer(s) for managed database services
      • Write Pegasus plugin in orchestration

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-8634

        Generalize DAX generator to handle arbitrary workflows

        		 10 Mikolaj Kowalik Done
         
        		DM-8633

        Prototype a wrapper generating DAXs for sequential workflows

        		 10 Mikolaj Kowalik Done
         
        		DM-8632

        Use Pegasus and Executor to run processCcd

        		 10 Mikolaj Kowalik Done
         
        		DM-8618

        Clean commit history and move Executor's code to lsst-dm GitHub organization

        		 0.5 Mikolaj Kowalik Done
         
        		DM-8603

        Expand the Pegasus-workflow-generating script to consider multiple patches

        		 10 Hsin-Fang Chiang Done
         
        		DM-8599

        Create a Pegasus-workflow repo in GitHub lsst-dm

        		 0.2 Hsin-Fang Chiang Done
         
        		DM-8947

        Remove dependency on CAT package and make database setup optional

        		 4 Steve Pietrowicz Done
         
        		DM-8823

        Create Pegasus plugin for Orca

        		 16 Steve Pietrowicz Done
         
        		DM-9010

        Investigate stack-like installation requirements for Pegasus

        		 3 Greg Daues Done
         
        		DM-9092

        Create a simplified workflow based on the ciHsc workflow

        		 0.5 Hsin-Fang Chiang Done
         
        		DM-9235

        Use Pegasus to do ProcessCcd with a subset of HSC COSMOS data on /datasets/hsc

        		 1 Hsin-Fang Chiang Done
         
        		DM-9209

        Allow Executor to work with pre-exisiting dataset repositories

        		 1 Mikolaj Kowalik Done
         
        		DM-9207

        Change Executor's data passing mechanism

        		 10 Mikolaj Kowalik Done
         
        		DM-8392

        Run prototype executor-less ci_hsc Pegasus workflow on Verification Cluster

        		 2 Hsin-Fang Chiang Done
         
        		DM-8391

        Run example Pegasus workflow on Verification Cluster

        		 2 Hsin-Fang Chiang Done
         
        		DM-8339

        Make a Pegasus workflow based on ci_hsc and Fall2016 interface

        		 12 Hsin-Fang Chiang Done
         
        		DM-9234

        Clean up the "inputData" file used in the DM-8603 dax generating script

        		 Hsin-Fang Chiang Won't Fix

          Activity

          Hide
          Permalink
          plutchak Joel Plutchak (Inactive) added a comment -

          A single workflow runs end to end with limited interaction with orchestration. Work continues this cycle under DM-9662.

          Show
          plutchak Joel Plutchak (Inactive) added a comment - A single workflow runs end to end with limited interaction with orchestration. Work continues this cycle under DM-9662 .

            People

            Assignee:
            mkowalik Mikolaj Kowalik
            Reporter:
            plutchak Joel Plutchak (Inactive)
            Watchers:
            Joel Plutchak (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Start date:
              End date:

                Jenkins

                No builds found.