# Create Pegasus plugin for Orca

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
16
• Team:
Data Facility

## Description

Write new plugin to run Pegasus via the orchestration utility "orca". This will include changes in ctrl_execute to support command line options for templates, change in the template structure, support for command line references to DAG generators, and changes in the default pipeline prefix.

Need to consult with Hsin-Fang Chiang about how current Pegasus jobs are being executed.

## Activity

Hide
Steve Pietrowicz added a comment -

That file exists:

 $ls /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config  I can't look at your /home/hchiang2/stack directory; can you temporarily open up the path to those files so I can look around? Show Steve Pietrowicz added a comment - That file exists:$ ls /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config /scratch/hchiang2/condor_scratch/configs/hchiang2_2017_0213_093521.config I can't look at your /home/hchiang2/stack directory; can you temporarily open up the path to those files so I can look around?
Hide
Steve Pietrowicz added a comment -

You have to:

 $chmod$PEGASUS_WORKFLOW_DIR/ciHsc/generateDax.py 

to make that script executable.

Show
Steve Pietrowicz added a comment - You have to: $chmod$PEGASUS_WORKFLOW_DIR/ciHsc/generateDax.py to make that script executable.
Hide
Hsin-Fang Chiang added a comment -

Thanks for figuring that out!

I added a branch to change the generateDax.py scripts to be executable in lsst-dm/pegasus_workflow.

Show
Hsin-Fang Chiang added a comment - Thanks for figuring that out! I added a branch to change the generateDax.py scripts to be executable in lsst-dm/pegasus_workflow .
Hide
Hsin-Fang Chiang added a comment -

I tested it with the shared stack with the steps as in my previous comment, and was able to run the workflow and produce outputs.

What do you suggest to modify pegasus_workflow now this plugin is in place? For example, pegasus_workflow/sites.xml is no longer needed, because Orca generates it. Anything else?

Show
Hsin-Fang Chiang added a comment - I tested it with the shared stack with the steps as in my previous comment, and was able to run the workflow and produce outputs. Some comments on the PR. What do you suggest to modify pegasus_workflow now this plugin is in place? For example, pegasus_workflow/sites.xml is no longer needed, because Orca generates it. Anything else?
Hide
Steve Pietrowicz added a comment -

I wouldn't change the pegasus_workflow at all, so it can still be used independently of orchestration, at least for now.

The only other thing that I can think of would be to move pegasus to a more permanent location; I'll work with Bill and Greg to get that done.

Show
Steve Pietrowicz added a comment - I wouldn't change the pegasus_workflow at all, so it can still be used independently of orchestration, at least for now. The only other thing that I can think of would be to move pegasus to a more permanent location; I'll work with Bill and Greg to get that done.

## People

• Assignee:
Steve Pietrowicz
Reporter:
Steve Pietrowicz
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, Steve Pietrowicz
0 Vote for this issue
Watchers:
2 Start watching this issue

## Dates

• Created:
Updated:
Resolved: