Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10049

Create representations of selected workflows in a graph format

    Details

      Description

      Generic generator of Pegasus DAXs needs to be thoroughly tested. For that a reasonable sized sample of actual workflows is required. The idea is to "encode" a few existing workflows in some commonly used graph format (e.g. JSON, GraphML) and use them to evaluate robustness and efficiency of the DAX generator.

        Attachments

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thanks for the instructions. Following them I was able to set up my environment, run your new code with a few Pegasus dax files, and create some graphml files.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thanks for the instructions. Following them I was able to set up my environment, run your new code with a few Pegasus dax files, and create some graphml files.
          Hide
          mkowalik Mikolaj Kowalik added a comment - - edited

          If you want to save some time replace the first line above with

          conda create --name myenv networkx six
          

          which install only necessary packages to test dax2graphml.py (I used your sim.dax from DM-9092).

          Show
          mkowalik Mikolaj Kowalik added a comment - - edited If you want to save some time replace the first line above with conda create --name myenv networkx six which install only necessary packages to test dax2graphml.py (I used your sim.dax from DM-9092 ).
          Hide
          mkowalik Mikolaj Kowalik added a comment - - edited

          I updated the source code getting rid of this pesky unicode character. It should work fine on lsst-dev now with either Python 2 or 3.

          To make xmltodict available on lsst-dev you need to create your own environment first. Though it may be an overkill, the easiest way to do it is to clone existing one called root. After you clone it, say as myenv, you can switch to it and install packages at will. Here are the commands to perform each step:

          conda create --name myenv --clone root
          source activate myenv
          pip install xmltodict
          

          Show
          mkowalik Mikolaj Kowalik added a comment - - edited I updated the source code getting rid of this pesky unicode character. It should work fine on lsst-dev now with either Python 2 or 3. To make xmltodict available on lsst-dev you need to create your own environment first. Though it may be an overkill, the easiest way to do it is to clone existing one called root . After you clone it, say as myenv , you can switch to it and install packages at will. Here are the commands to perform each step: conda create --name myenv --clone root source activate myenv pip install xmltodict
          Hide
          tjenness Tim Jenness added a comment -

          It imports fine for me with Python 3 but I also can't see that character in the source.

          Show
          tjenness Tim Jenness added a comment - It imports fine for me with Python 3 but I also can't see that character in the source.
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Two quick comments/questions for now:

          1. I encountered this error when trying to run it:

           File "daxgen/utils/dax2graphml.py", line 132
          SyntaxError: Non-ASCII character '\xc2' in file daxgen/utils/dax2graphml.py on line 133, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
          

          Can you fix that please?

          2. I use the Python that comes with the shared stack, and it does not have xmltodict. I cannot install new package to the shared stack. Which python do you use on lsst-dev?

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Two quick comments/questions for now: 1. I encountered this error when trying to run it: File "daxgen/utils/dax2graphml.py" , line 132 SyntaxError: Non-ASCII character '\xc2' in file daxgen/utils/dax2graphml.py on line 133 , but no encoding declared; see http: //python.org/dev/peps/pep-0263/ for details Can you fix that please? 2. I use the Python that comes with the shared stack, and it does not have xmltodict . I cannot install new package to the shared stack. Which python do you use on lsst-dev?
          Hide
          mkowalik Mikolaj Kowalik added a comment -

          I added a simple utility to my DAX generator which allows me to convert (your) existing Pegasus plans to GraphML e.g. for testing purposes.

          I would appreciate if you could take a look at the code.

          Show
          mkowalik Mikolaj Kowalik added a comment - I added a simple utility to my DAX generator which allows me to convert (your) existing Pegasus plans to GraphML e.g. for testing purposes. I would appreciate if you could take a look at the code.

            People

            • Assignee:
              mkowalik Mikolaj Kowalik
              Reporter:
              mkowalik Mikolaj Kowalik
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel