Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23106

Use Jenkins parallel stage functionality to thread the salgenerator jobs

    XMLWordPrintable

    Details

      Description

      In looking for ways to speed up the generation of the RPMs, I found that Jenkins pipelines can run stages in parallel. See if this functionality can work.

      https://jenkins.io/blog/2017/09/25/declarative-1/

        Attachments

          Issue Links

            Activity

            No builds found.
            rbovill Rob Bovill created issue -
            rbovill Rob Bovill made changes -
            Field Original Value New Value
            Story Points 2
            rbovill Rob Bovill made changes -
            Epic Link DM-22789 [ 428321 ]
            rbovill Rob Bovill made changes -
            Component/s ts_qa [ 16723 ]
            rbovill Rob Bovill made changes -
            Labels Build/Install CI QA jenkins
            rbovill Rob Bovill made changes -
            Summary Using Jenkins parallel stage functionality to thread the salgenerator jobs Use Jenkins parallel stage functionality to thread the salgenerator jobs
            rbovill Rob Bovill made changes -
            Sprint TSSW Sprint - Jan 21 - Feb 01 [ 993 ] TSSW Sprint - Feb3 - Feb 14 [ 996 ]
            rbovill Rob Bovill made changes -
            Rank Ranked lower
            rbovill Rob Bovill made changes -
            Description In looking for ways to speed up the generation of the RPMs, I found that Jenkins pipelines can run stages in parallel. See if this functionality can work.

            https://jenkins.io/blog/2017/09/25/declarative-1/
            aclements Andy Clements made changes -
            Sprint TSSW Sprint - Feb3 - Feb 14 [ 996 ] TSSW Sprint - Feb3 - Feb 14, TSSW Sprint - Feb 17 - Mar 2 [ 996, 1001 ]
            rbovill Rob Bovill made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            rbovill Rob Bovill added a comment -

            Got parallel staging to work! I updated the configuration in the job itself to get it working. You can see the results here: https://tssw-ci.lsst.org/job/RPM_Push_daily/252/

            Show
            rbovill Rob Bovill added a comment - Got parallel staging to work! I updated the configuration in the job itself to get it working. You can see the results here: https://tssw-ci.lsst.org/job/RPM_Push_daily/252/
            Hide
            rbovill Rob Bovill added a comment -

            I moved the pipeline script from the Jenkins UI and into the robotframework_salgenerator repo, so the configuration is under source and version control.

            https://github.com/lsst-ts/robotframework_salgenerator/commit/978d030ef58b230b9d3e605ec1716028a2f6cd59

            Show
            rbovill Rob Bovill added a comment - I moved the pipeline script from the Jenkins UI and into the robotframework_salgenerator repo, so the configuration is under source and version control. https://github.com/lsst-ts/robotframework_salgenerator/commit/978d030ef58b230b9d3e605ec1716028a2f6cd59
            Hide
            rbovill Rob Bovill added a comment -

            First job with the script from SCM: https://tssw-ci.lsst.org/job/RPM_Push_daily/258/

            Show
            rbovill Rob Bovill added a comment - First job with the script from SCM: https://tssw-ci.lsst.org/job/RPM_Push_daily/258/
            rbovill Rob Bovill made changes -
            Reviewers Michael Reuter [ mareuter ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            mareuter Michael Reuter added a comment -

            I took a look at the times on the job and it seems the parallel stages cut 2 hours off the job time. Since the file was just placed under version control, I assume it was the addition of the "parallel" sections in the "Generate the RPMs" stage that was the key here yes?

            Show
            mareuter Michael Reuter added a comment - I took a look at the times on the job and it seems the parallel stages cut 2 hours off the job time. Since the file was just placed under version control, I assume it was the addition of the "parallel" sections in the "Generate the RPMs" stage that was the key here yes?
            Hide
            rbovill Rob Bovill added a comment -

            Michael Reuter yes, your assumption is correct.

            Show
            rbovill Rob Bovill added a comment - Michael Reuter yes, your assumption is correct.
            Hide
            mareuter Michael Reuter added a comment -

            Nice improvement to the build situation.

            Show
            mareuter Michael Reuter added a comment - Nice improvement to the build situation.
            mareuter Michael Reuter made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            rbovill Rob Bovill made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            rbovill Rob Bovill made changes -
            Link This issue is cloned by DM-23925 [ DM-23925 ]

              People

              Assignee:
              rbovill Rob Bovill
              Reporter:
              rbovill Rob Bovill
              Reviewers:
              Michael Reuter
              Watchers:
              Michael Reuter, Rob Bovill
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.