Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8823

Create Pegasus plugin for Orca

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ctrl_orca
    • Labels:
      None

      Description

      Write new plugin to run Pegasus via the orchestration utility "orca". This will include changes in ctrl_execute to support command line options for templates, change in the template structure, support for command line references to DAG generators, and changes in the default pipeline prefix.

      Need to consult with Hsin-Fang Chiang about how current Pegasus jobs are being executed.

        Attachments

          Activity

          Hide
          spietrowicz Steve Pietrowicz added a comment -

          That file exists:

          $ ls /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config
          /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config
          

          I can't look at your /home/hchiang2/stack directory; can you temporarily open up the path to those files so I can look around?

          Show
          spietrowicz Steve Pietrowicz added a comment - That file exists: $ ls /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config I can't look at your /home/hchiang2/stack directory; can you temporarily open up the path to those files so I can look around?
          Hide
          spietrowicz Steve Pietrowicz added a comment -

          You have to:

          $ chmod $PEGASUS_WORKFLOW_DIR/ciHsc/generateDax.py
          

          to make that script executable.

          Show
          spietrowicz Steve Pietrowicz added a comment - You have to: $ chmod $PEGASUS_WORKFLOW_DIR/ciHsc/generateDax.py to make that script executable.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thanks for figuring that out!

          I added a branch to change the generateDax.py scripts to be executable in lsst-dm/pegasus_workflow.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thanks for figuring that out! I added a branch to change the generateDax.py scripts to be executable in lsst-dm/pegasus_workflow .
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          I tested it with the shared stack with the steps as in my previous comment, and was able to run the workflow and produce outputs.

          Some comments on the PR.

          What do you suggest to modify pegasus_workflow now this plugin is in place? For example, pegasus_workflow/sites.xml is no longer needed, because Orca generates it. Anything else?

          Show
          hchiang2 Hsin-Fang Chiang added a comment - I tested it with the shared stack with the steps as in my previous comment, and was able to run the workflow and produce outputs. Some comments on the PR. What do you suggest to modify pegasus_workflow now this plugin is in place? For example, pegasus_workflow/sites.xml is no longer needed, because Orca generates it. Anything else?
          Hide
          spietrowicz Steve Pietrowicz added a comment -

          I wouldn't change the pegasus_workflow at all, so it can still be used independently of orchestration, at least for now.

          The only other thing that I can think of would be to move pegasus to a more permanent location; I'll work with Bill and Greg to get that done.

          Show
          spietrowicz Steve Pietrowicz added a comment - I wouldn't change the pegasus_workflow at all, so it can still be used independently of orchestration, at least for now. The only other thing that I can think of would be to move pegasus to a more permanent location; I'll work with Bill and Greg to get that done.

            People

            • Assignee:
              spietrowicz Steve Pietrowicz
              Reporter:
              spietrowicz Steve Pietrowicz
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Steve Pietrowicz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel