Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9010

Investigate stack-like installation requirements for Pegasus

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The newly added package lsst-dm/pegasus_workflow
      uses Pegasus APIs directly in the Python script. Investigate whether it will become necessary to add Pegasus as an external package / third part package and make it installable like the stack following the policy. Or do we install it on supported systems like HTCondor?

      Some questions to think about may include:

      • Running Pegasus requires a working HTCondor. Does Pegasus use HTCondor APIs directly?
      • Would HTCondor still be needed if one only runs the DAX-grnerating script, which uses the Pegasus.DAX3 Python APIs, but not tools like pegasus-run, pegasus-plan, etc?

        Attachments

          Activity

          Hide
          gdaues Greg Daues added a comment -

          I'll start investigating. I do not think we want to be in the business of installing pegasus from source on various platforms the way general packages are treated in the stack, so we will need to find what the alternative is.

          In ordinary execution of workflows, Pegasus runs the script "condor_dagman" as a job (that is the executable) , so this is not an API call per se. But it may allow for "planning" to be done on a system that does not literally have htcondor available.

          Show
          gdaues Greg Daues added a comment - I'll start investigating. I do not think we want to be in the business of installing pegasus from source on various platforms the way general packages are treated in the stack, so we will need to find what the alternative is. In ordinary execution of workflows, Pegasus runs the script "condor_dagman" as a job (that is the executable) , so this is not an API call per se. But it may allow for "planning" to be done on a system that does not literally have htcondor available.
          Hide
          gdaues Greg Daues added a comment -

          A little study shows that the package that we want to install to get the Python libraries of pegasus is
          denoted "pegasus-wms" . One way one can install this is via

          pip install http://download.pegasus.isi.edu/pegasus/4.7.2/pegasus-python-source-4.7.2.tar.gz
          

          so this install from source does not look so bad (rather pythonic.)
          A test of the pip install on Nebula shows the dependencies that we need :

           
          # yum -y install python-devel
          # yum -y install openssl-devel
           
          # pip install http://download.pegasus.isi.edu/pegasus/4.7.2/pegasus-python-source-4.7.2.tar.gz
          ...
           Werkzeug==0.9.3 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           Flask==0.10 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           Jinja2==2.7 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           Flask-SQLAlchemy==0.16 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           Flask-Cache==0.13.1 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           WTForms==1.0.3 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           requests==2.8.0 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           MarkupSafe==0.18 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           itsdangerous==0.21 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           boto==2.5.2 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           pam==0.1.4 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           SQLAlchemy==0.8.0 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2)
           setuptools in /usr/lib/python2.7/site-packages (from Flask-SQLAlchemy==0.16->pegasus-wms==4.7.2)
           six>=1.5 in /usr/lib/python2.7/site-packages (from sphinx->plex==2.0.0dev->pegasus-wms==4.7.2)
          ...
            Using cached pyOpenSSL-0.13.tar.gz
            Using cached plex-2.0.0dev.tar.gz
            Using cached nose-1.3.7-py2-none-any.whl
            Using cached Sphinx-1.5.1-py2.py3-none-any.whl
            Using cached docutils-0.13.1-py2-none-any.whl
            Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl
            Using cached alabaster-0.7.9-py2.py3-none-any.whl
            Using cached imagesize-0.7.1-py2.py3-none-any.whl
            Using cached Babel-2.3.4-py2.py3-none-any.whl
            Using cached Pygments-2.1.3-py2.py3-none-any.whl
            Using cached pytz-2016.10-py2.py3-none-any.whl
          ...
          Successfully installed   ...  pegasus-wms-4.7.2
          

          After this installation we can see that we have

          %  ls /usr/lib/python2.7/site-packages/Pegasus
          cluster.py   command.py   DAX2.py   DAX3.py   db           exitcode.pyc  init.py       init.pyc    netlogger  PDAX2.pyc    s3.py   service       submitdir.pyc  user.py
          cluster.pyc  command.pyc  DAX2.pyc  DAX3.pyc  exitcode.py  __init__.py   __init__.pyc  monitoring  PDAX2.py   plots_stats  s3.pyc  submitdir.py  tools          user.pyc
          

          However, we do not yet have "pegasus-plan" for example.

          So the question: is this package the dependency for lsst-dm/pegasus_workflow ?

          Show
          gdaues Greg Daues added a comment - A little study shows that the package that we want to install to get the Python libraries of pegasus is denoted "pegasus-wms" . One way one can install this is via pip install http://download.pegasus.isi.edu/pegasus/4.7.2/pegasus-python-source-4.7.2.tar.gz so this install from source does not look so bad (rather pythonic.) A test of the pip install on Nebula shows the dependencies that we need :   # yum -y install python-devel # yum -y install openssl-devel   # pip install http://download.pegasus.isi.edu/pegasus/4.7.2/pegasus-python-source-4.7.2.tar.gz ... Werkzeug==0.9.3 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) Flask==0.10 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) Jinja2==2.7 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) Flask-SQLAlchemy==0.16 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) Flask-Cache==0.13.1 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) WTForms==1.0.3 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) requests==2.8.0 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) MarkupSafe==0.18 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) itsdangerous==0.21 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) boto==2.5.2 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) pam==0.1.4 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) SQLAlchemy==0.8.0 in /usr/lib/python2.7/site-packages (from pegasus-wms==4.7.2) setuptools in /usr/lib/python2.7/site-packages (from Flask-SQLAlchemy==0.16->pegasus-wms==4.7.2) six>=1.5 in /usr/lib/python2.7/site-packages (from sphinx->plex==2.0.0dev->pegasus-wms==4.7.2) ... Using cached pyOpenSSL-0.13.tar.gz Using cached plex-2.0.0dev.tar.gz Using cached nose-1.3.7-py2-none-any.whl Using cached Sphinx-1.5.1-py2.py3-none-any.whl Using cached docutils-0.13.1-py2-none-any.whl Using cached snowballstemmer-1.2.1-py2.py3-none-any.whl Using cached alabaster-0.7.9-py2.py3-none-any.whl Using cached imagesize-0.7.1-py2.py3-none-any.whl Using cached Babel-2.3.4-py2.py3-none-any.whl Using cached Pygments-2.1.3-py2.py3-none-any.whl Using cached pytz-2016.10-py2.py3-none-any.whl ... Successfully installed ... pegasus-wms-4.7.2 After this installation we can see that we have % ls /usr/lib/python2.7/site-packages/Pegasus cluster.py command.py DAX2.py DAX3.py db exitcode.pyc init.py init.pyc netlogger PDAX2.pyc s3.py service submitdir.pyc user.py cluster.pyc command.pyc DAX2.pyc DAX3.pyc exitcode.py __init__.py __init__.pyc monitoring PDAX2.py plots_stats s3.pyc submitdir.py tools user.pyc However, we do not yet have "pegasus-plan" for example. So the question: is this package the dependency for lsst-dm/pegasus_workflow ?
          Hide
          gdaues Greg Daues added a comment -

          Initial investigation is accomplished, and additional work on stack integration will occur in the context of the next Epic.

          Show
          gdaues Greg Daues added a comment - Initial investigation is accomplished, and additional work on stack integration will occur in the context of the next Epic.

            People

            • Assignee:
              gdaues Greg Daues
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Greg Daues, Hsin-Fang Chiang, Steve Pietrowicz
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel