Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8391

Run example Pegasus workflow on Verification Cluster

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Learn to run basic Pegasus examples on Verification Cluster

      Use DM-8156 to properly allocate nodes and run jobs on the worker nodes.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues has installed Pegasus on the Verification Cluster:

            export PEGASUS_HOME=/software/daues/pegasus/pegasus-4.7.2-1/usr
            export PATH=${PEGASUS_HOME}/bin:$PATH
            export PYTHONPATH=${PEGASUS_HOME}/lib64/python2.7/site-packages/:$PYTHONPATH
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues has installed Pegasus on the Verification Cluster: export PEGASUS_HOME=/software/daues/pegasus/pegasus- 4.7 . 2 - 1 /usr export PATH=${PEGASUS_HOME}/bin:$PATH export PYTHONPATH=${PEGASUS_HOME}/lib64/python2. 7 /site-packages/:$PYTHONPATH
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm able to run some simple Pegasus workflow made in DM-7981 on Verification Cluster. I did not need DM-8156 to do so. There seem 4 condor slots available (and can be seen through condor_status) without doing ctrl_execute allocateNodes.py

            I'll try a more complicated workflow in DM-8392.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm able to run some simple Pegasus workflow made in DM-7981 on Verification Cluster. I did not need DM-8156 to do so. There seem 4 condor slots available (and can be seen through condor_status) without doing ctrl_execute allocateNodes.py I'll try a more complicated workflow in DM-8392 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Putting this back to "InProgress". I was running on local only, and not on Verification Cluster worker nodes.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Putting this back to "InProgress". I was running on local only, and not on Verification Cluster worker nodes.
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            We need to move pegasus to someplace more permanent. For condor, it's under

            /software/condor/current

            which is a link to

            /software/condor/condor-8.5.9-389926-x86_64_RedHat7-stripped

            if we do the same for pegasus, we won't need to update that value in our git repos.

            Show
            spietrowicz Steve Pietrowicz added a comment - We need to move pegasus to someplace more permanent. For condor, it's under /software/condor/current which is a link to /software/condor/condor-8.5.9-389926-x86_64_RedHat7-stripped if we do the same for pegasus, we won't need to update that value in our git repos.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            With DM-8156, I am able to run some simple workflows (testPegasus, testPegasusStack, exampleTask, basicProcessCcd here) on the Verification Cluster worker nodes using the given allocated node set.

            For the Pegasus data staging configuration pegasus.data.configuration, I used condorio in one simplest workflow, and sharedfs for the other three.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - With DM-8156 , I am able to run some simple workflows (testPegasus, testPegasusStack, exampleTask, basicProcessCcd here ) on the Verification Cluster worker nodes using the given allocated node set. For the Pegasus data staging configuration pegasus.data.configuration , I used condorio in one simplest workflow, and sharedfs for the other three.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -


            I guess the current installation of Pegasus was meant to be temporary while in test. I'll leave the moving to Greg Daues.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I guess the current installation of Pegasus was meant to be temporary while in test. I'll leave the moving to Greg Daues .

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Steve Pietrowicz
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel