Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8332

Run selected science pipeline using Pegasus

    Details

    • Epic Name:
      Run selected science pipeline using Pegasus
    • Story Points:
      108
    • WBS:
      02C.07.01
    • Team:
      Data Facility
    • Cycle:
      Spring 2017

      Description

      Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

      • Provide integration software needed for HTCondor + Pegasus
      • Instantiate WLMS Pegasus+HTCondor
      • Provide ability to run HTCondor glide-ins on Slurm
      • Prepare common middleware layer(s) for managed database services
      • Write Pegasus plugin in orchestration

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        DM-9235

        Use Pegasus to do ProcessCcd with a subset of HSC COSMOS data on /datasets/hsc

        1 Hsin-Fang Chiang Done
         
        DM-9209

        Allow Executor to work with pre-exisiting dataset repositories

        1 Mikolaj Kowalik Done
         
        DM-9207

        Change Executor's data passing mechanism

        10 Mikolaj Kowalik Done
         
        DM-9010

        Investigate stack-like installation requirements for Pegasus

        3 Greg Daues Done
         
        DM-8947

        Remove dependency on CAT package and make database setup optional

        4 Steve Pietrowicz Done
         
        DM-8823

        Create Pegasus plugin for Orca

        16 Steve Pietrowicz Done
         
        DM-9092

        Create a simplified workflow based on the ciHsc workflow

        0.5 Hsin-Fang Chiang Done
         
        DM-8634

        Generalize DAX generator to handle arbitrary workflows

        10 Mikolaj Kowalik Done
         
        DM-8633

        Prototype a wrapper generating DAXs for sequential workflows

        10 Mikolaj Kowalik Done
         
        DM-8632

        Use Pegasus and Executor to run processCcd

        10 Mikolaj Kowalik Done
         
        DM-8618

        Clean commit history and move Executor's code to lsst-dm GitHub organization

        0.5 Mikolaj Kowalik Done
         
        DM-8603

        Expand the Pegasus-workflow-generating script to consider multiple patches

        10 Hsin-Fang Chiang Done
         
        DM-8599

        Create a Pegasus-workflow repo in GitHub lsst-dm

        0.2 Hsin-Fang Chiang Done
         
        DM-8392

        Run prototype executor-less ci_hsc Pegasus workflow on Verification Cluster

        2 Hsin-Fang Chiang Done
         
        DM-8391

        Run example Pegasus workflow on Verification Cluster

        2 Hsin-Fang Chiang Done
         
        DM-8339

        Make a Pegasus workflow based on ci_hsc and Fall2016 interface

        12 Hsin-Fang Chiang Done
         
        DM-9234

        Clean up the "inputData" file used in the DM-8603 dax generating script

        Hsin-Fang Chiang Won't Fix

          Activity

          plutchak Joel Plutchak (Inactive) created issue -
          plutchak Joel Plutchak (Inactive) made changes -
          Field Original Value New Value
          Description Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only. Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.
          *
          * Provide integration software needed for HTCondor + Pegasus
          * Instantiate WLMS Pegasus+HTCondor
          * Provide ability to run HTCondor glide-ins on Slurm
          * Provide managed database services for Pegasus and HTCondor operational metadata
          * Write Pegasus plugin in orchestration



          plutchak Joel Plutchak (Inactive) made changes -
          Description Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.
          *
          * Provide integration software needed for HTCondor + Pegasus
          * Instantiate WLMS Pegasus+HTCondor
          * Provide ability to run HTCondor glide-ins on Slurm
          * Provide managed database services for Pegasus and HTCondor operational metadata
          * Write Pegasus plugin in orchestration



          Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

          * Provide integration software needed for HTCondor + Pegasus
          * Instantiate WLMS Pegasus+HTCondor
          * Provide ability to run HTCondor glide-ins on Slurm
          * Provide managed database services for Pegasus and HTCondor operational metadata
          * Write Pegasus plugin in orchestration



          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8339 [ 27940 ]
          plutchak Joel Plutchak (Inactive) made changes -
          WBS 02C.07.01
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8391 [ 28002 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8392 [ 28003 ]
          plutchak Joel Plutchak (Inactive) made changes -
          Story Points 116
          plutchak Joel Plutchak (Inactive) made changes -
          Story Points 116 108
          plutchak Joel Plutchak (Inactive) made changes -
          Cycle Spring 2017 [ 10804 ]
          plutchak Joel Plutchak (Inactive) made changes -
          Labels Batch_Production_Services
          mkowalik Mikolaj Kowalik made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8599 [ 28262 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8603 [ 28266 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-8618 [ 28283 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-8632 [ 28300 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-8633 [ 28301 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-8634 [ 28302 ]
          spietrowicz Steve Pietrowicz made changes -
          Epic Child DM-8823 [ 29035 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-9010 [ 29337 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-9092 [ 29579 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-9207 [ 29805 ]
          mkowalik Mikolaj Kowalik made changes -
          Epic Child DM-9209 [ 29808 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-9234 [ 29845 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-9235 [ 29846 ]
          spietrowicz Steve Pietrowicz made changes -
          Epic Child DM-9460 [ 30221 ]
          hchiang2 Hsin-Fang Chiang made changes -
          Epic Child DM-8947 [ 29219 ]
          plutchak Joel Plutchak (Inactive) made changes -
          Description Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

          * Provide integration software needed for HTCondor + Pegasus
          * Instantiate WLMS Pegasus+HTCondor
          * Provide ability to run HTCondor glide-ins on Slurm
          * Provide managed database services for Pegasus and HTCondor operational metadata
          * Write Pegasus plugin in orchestration



          Demonstrate batch service that is capable of running other codes besides HSC ctrl_pool framework. Support HSC camera only.

          * Provide integration software needed for HTCondor + Pegasus
          * Instantiate WLMS Pegasus+HTCondor
          * Provide ability to run HTCondor glide-ins on Slurm
          * Prepare common middleware layer(s) for managed database services
          * Write Pegasus plugin in orchestration



          plutchak Joel Plutchak (Inactive) made changes -
          Epic Child DM-9460 [ 30221 ]
          plutchak Joel Plutchak (Inactive) made changes -
          Resolution Done [ 10000 ]
          Status In Progress [ 3 ] Done [ 10002 ]
          plutchak Joel Plutchak (Inactive) made changes -
          Status Done [ 10002 ] Done [ 10002 ]
          plutchak Joel Plutchak (Inactive) made changes -
          End date 28/Feb/17 31/Aug/17
          Start date 01/Dec/16 01/Jun/17
          plutchak Joel Plutchak (Inactive) made changes -
          End date 31/Aug/17 01/Dec/16
          Start date 01/Jun/17 28/Feb/17
          frossie Frossie Economou made changes -
          Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

            People

            • Assignee:
              mkowalik Mikolaj Kowalik
              Reporter:
              plutchak Joel Plutchak (Inactive)
              Watchers:
              Joel Plutchak (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Start date:
                End date:

                Summary Panel