Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9092

Create a simplified workflow based on the ciHsc workflow

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Create a simplified workflow of DM-8339 for Mikolaj Kowalik

        Attachments

        1. simp.png
          simp.png
          736 kB
        2. sim.dax
          13 kB
        3. sim.dot
          11 kB

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Mikolaj Kowalik, I added a workflow in branch tickets/DM-9092. This has 4 ProcessCcdTasks + 1 MakeSkyMap + 2 MakeCoaddTempExp. The dax plot is attached. I was able to run it on lsstvc with the same steps as the ciHsc workflow, but please try it first to see if you can run it on lsstvc.

          # Set up the stack
          .  /software/lsstsw/stack/loadLSST.bash
          setup -v ctrl_platform_lsstvc 
          setup -k -v -r /scratch/hchiang2/sw/ci_hsc/ 
           
          # Allocate nodes using ctrl_execute tools
          allocateNodes.py -n 2 -s 2 -m 01:30:00 lsstvc
           
          # Set up Pegasus
          export PEGASUS_HOME=/software/daues/pegasus/pegasus-4.7.2-1/usr
          export PATH=${PEGASUS_HOME}/bin:$PATH
          export PYTHONPATH=${PEGASUS_HOME}/lib64/python2.7/site-packages/:$PYTHONPATH
           
          # I use this env var in pegasus_workflow setup
          export NODESET=${USER}_`cat ~/.lsst/node-set.seq`
           
          # Make the workflow xml file using Pegasus Python API
          python simplified/generateDax.py -i simplified/inputData.py
          # Plan and submit the xml workflow using Pegasus tools
          ./plan_dax.sh ciHsc.dax
          

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Mikolaj Kowalik , I added a workflow in branch tickets/ DM-9092 . This has 4 ProcessCcdTasks + 1 MakeSkyMap + 2 MakeCoaddTempExp. The dax plot is attached. I was able to run it on lsstvc with the same steps as the ciHsc workflow, but please try it first to see if you can run it on lsstvc. # Set up the stack . /software/lsstsw/stack/loadLSST.bash setup -v ctrl_platform_lsstvc setup -k -v -r /scratch/hchiang2/sw/ci_hsc/ # Allocate nodes using ctrl_execute tools allocateNodes.py -n 2 -s 2 -m 01 : 30 : 00 lsstvc # Set up Pegasus export PEGASUS_HOME=/software/daues/pegasus/pegasus- 4.7 . 2 - 1 /usr export PATH=${PEGASUS_HOME}/bin:$PATH export PYTHONPATH=${PEGASUS_HOME}/lib64/python2. 7 /site-packages/:$PYTHONPATH # I use this env var in pegasus_workflow setup export NODESET=${USER}_`cat ~/.lsst/node-set.seq` # Make the workflow xml file using Pegasus Python API python simplified/generateDax.py -i simplified/inputData.py # Plan and submit the xml workflow using Pegasus tools ./plan_dax.sh ciHsc.dax
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Do you think this workflow is worth merging into master? I was thinking if you plan to use this exact workflow and replace jobs with Executor jobs then maybe yes? If this is only a smaller workflow for testing then maybe no because it duplicates too much with the ciHsc workflow? What do you think?

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Do you think this workflow is worth merging into master? I was thinking if you plan to use this exact workflow and replace jobs with Executor jobs then maybe yes? If this is only a smaller workflow for testing then maybe no because it duplicates too much with the ciHsc workflow? What do you think?
          Hide
          mkowalik Mikolaj Kowalik added a comment -

          At the moment, I don't think it's worth merging into master yet (if ever). You're correct, for now I'd just like to have a simple though not completely trivial (one task) workflow I can play with. I'll just merge your ticket branch into my user branch and I should be fine for time being. Thanks a lot for preparing it for me!

          Show
          mkowalik Mikolaj Kowalik added a comment - At the moment, I don't think it's worth merging into master yet (if ever). You're correct, for now I'd just like to have a simple though not completely trivial (one task) workflow I can play with. I'll just merge your ticket branch into my user branch and I should be fine for time being. Thanks a lot for preparing it for me!
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Sounds good to me

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Sounds good to me
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Also uploading the dax xml and the dot file, in case they are useful

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Also uploading the dax xml and the dot file, in case they are useful

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Mikolaj Kowalik
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel