Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-271 Setup the new Buildbot CI system
  3. DM-478

Allow user-triggered builds initiated from the Buildbot web form and which allow input of the git-refs to be used during the build

    Details

    • Sprint:
      DevOps Sprint 1, DevOps Sprint 2, DevOps Sprint 3
    • Team:
      SQuaRE

      Description

      Users will be allowed to initiate builds from the Buildbot web form. They will initially use an htpasswd login/password. It is anticipated the authentication method will eventually be transitioned into LDAP - but not during this pass.

      The users will be able to specify a prioritized list of git refs specifying the branches to be scanned when acquiring a package from a git repository.

      May also want to allow the user to tag the build with some user-memorable eups-tag so (1) acquiring the build manifest and/or (2) setting up the identical stack for debug, are simple operations.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            This should eventually be integrated with HipChat and a chatbot.

            Show
            ktl Kian-Tat Lim added a comment - This should eventually be integrated with HipChat and a chatbot.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            This is already implemented on the lsstsw slave under account lsstsw2. (It was working before the sprint began.)

            I just created the 'everyman' htpasswd user and set a password which I won't publish until new buildbot &co. is released and the slave is running as lsstsw on lsst-dev

            The user provision of multiple branches to sequentially search for a package hit is also operational both in the user input phase, their use by 'lsst-build prepare', and the acquisition of the named branch's git package for use in the build.

            What remains is tidying up the script effecting the build on the slave followed rapidly by its git repo update.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - This is already implemented on the lsstsw slave under account lsstsw2. (It was working before the sprint began.) I just created the 'everyman' htpasswd user and set a password which I won't publish until new buildbot &co. is released and the slave is running as lsstsw on lsst-dev The user provision of multiple branches to sequentially search for a package hit is also operational both in the user input phase, their use by 'lsst-build prepare', and the acquisition of the named branch's git package for use in the build. What remains is tidying up the script effecting the build on the slave followed rapidly by its git repo update.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Robert, This shouldn't take too much time to verify that it is working since you've been initiating rebuilds for the last week. If you decline this kind offer, let me know and I'll pass on to another.

            This task is limited to being able to initiate a build via the web interface. The default mode is to use only the master branches for the packages. If the user provides a sequenced list of alternate branches, they are used in preference to master (which is used as the repository of last resort).

            The user is required to provide their username and email address before being allowed to trigger the build. This allows Buildbot to provide both success and failure completion status to the user most interested in the build just invoked.

            If a failure occurs during the scons or unittest phase, failure artifacts are collected as appropriate for the build. For an scons build failure, the package's manifest, scons log, and lsstsw's build script are provided. For a unittest failure, the failed <test>.failed is provided. The files are copied into a directory named in the email response.

            The use of the Buildbot web-interface is described in: https://confluence.lsstcorp.org/display/LDMDG/How+to+Trigger+a+Buildbot+Build

            If you'd like to experience the use of a user-invoked ticket branch build, than I provide: 'tickets/DM-711'. You can see how the 'refs' are passed in and used. If you run it twice, you'll see how the build does not uselessly rerun the doxygen generation nor the end-to-end test (assuming no other git changes occurred in the interim that caused some package to rebuild).
            I can dig out an scons failing ticket branch, if you'd like.

            ------------------------------------------------------------------------------------------
            Future additions to this capability may include - but do not now exist:

            • allow user to specify own tag for the freshly built stack (this is actually an lsst-build update request not a buildbot update)
            • support LDAP authentication. This capability is only marginally supported by buildbot 0.8.8 at the moment.
            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Robert, This shouldn't take too much time to verify that it is working since you've been initiating rebuilds for the last week. If you decline this kind offer, let me know and I'll pass on to another. This task is limited to being able to initiate a build via the web interface. The default mode is to use only the master branches for the packages. If the user provides a sequenced list of alternate branches, they are used in preference to master (which is used as the repository of last resort). The user is required to provide their username and email address before being allowed to trigger the build. This allows Buildbot to provide both success and failure completion status to the user most interested in the build just invoked. If a failure occurs during the scons or unittest phase, failure artifacts are collected as appropriate for the build. For an scons build failure, the package's manifest, scons log, and lsstsw's build script are provided. For a unittest failure, the failed <test>.failed is provided. The files are copied into a directory named in the email response. The use of the Buildbot web-interface is described in: https://confluence.lsstcorp.org/display/LDMDG/How+to+Trigger+a+Buildbot+Build If you'd like to experience the use of a user-invoked ticket branch build, than I provide: 'tickets/ DM-711 '. You can see how the 'refs' are passed in and used. If you run it twice, you'll see how the build does not uselessly rerun the doxygen generation nor the end-to-end test (assuming no other git changes occurred in the interim that caused some package to rebuild). I can dig out an scons failing ticket branch, if you'd like. ------------------------------------------------------------------------------------------ Future additions to this capability may include - but do not now exist: allow user to specify own tag for the freshly built stack (this is actually an lsst-build update request not a buildbot update) support LDAP authentication. This capability is only marginally supported by buildbot 0.8.8 at the moment.
            Hide
            rhl Robert Lupton added a comment -

            Looks good.

            • I'd refer to http://lsst-buildx.ncsa.illinois.edu:8010/one_line_per_build more prominently (it's clearer than the waterfall page, at least to those who like a concise view of things)
            • It'd be good to accept "rhl@astro.princeton.edu" but the docs are clear!
            • The "stop this build" button says I'm not important enough. Should be noted in the docs.
            • You say, "setup -t <tag> <package>" to eups-setup <package> and its dependencies". With -t you override any dependencies; with -T you only use the tag if no version is specified. You probably want something more like "setup -t rhl -T <tag> package" (get the <tag> package, but override with my own ones). But that's more eups-foo than buildbot-foo
            • Shouldn't

              cd <your local git archive>
              git clone git@git.lsstcorp.org:LSST/DMS/afw.git
              cd afw
              setup -t b76 -r . afw

              be

              cd <favourite scratch space>
              git clone git@git.lsstcorp.org:LSST/DMS/afw.git
              cd afw
              setup -t b76 -r . afw
              scons -Q opt=3

              ? (or -T b76).

            Show
            rhl Robert Lupton added a comment - Looks good. I'd refer to http://lsst-buildx.ncsa.illinois.edu:8010/one_line_per_build more prominently (it's clearer than the waterfall page, at least to those who like a concise view of things) It'd be good to accept "rhl@astro.princeton.edu" but the docs are clear! The "stop this build" button says I'm not important enough. Should be noted in the docs. You say, "setup -t <tag> <package>" to eups-setup <package> and its dependencies". With -t you override any dependencies; with -T you only use the tag if no version is specified. You probably want something more like "setup -t rhl -T <tag> package" (get the <tag> package, but override with my own ones). But that's more eups-foo than buildbot-foo Shouldn't cd <your local git archive> git clone git@git.lsstcorp.org:LSST/DMS/afw.git cd afw setup -t b76 -r . afw be cd <favourite scratch space> git clone git@git.lsstcorp.org:LSST/DMS/afw.git cd afw setup -t b76 -r . afw scons -Q opt=3 ? (or -T b76).
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Thanks for the comments. Later reader will appreciate the upgrades, too.

            Response to reviewer's comments:
            1) I dramatically highlighted the one-line-per-build entry point and noted its value
            2) The buildbot function used to validate the email address is very rigid in its demands, as you noticed.
            3) The stop/cancel/rebuild/force user options are described and their authorization policy specified.
            4) Added your comment to the doco.
            5) Update the example as per your sample.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Thanks for the comments. Later reader will appreciate the upgrades, too. Response to reviewer's comments: 1) I dramatically highlighted the one-line-per-build entry point and noted its value 2) The buildbot function used to validate the email address is very rigid in its demands, as you noticed. 3) The stop/cancel/rebuild/force user options are described and their authorization policy specified. 4) Added your comment to the doco. 5) Update the example as per your sample.

              People

              • Assignee:
                robyn Robyn Allsman [X] (Inactive)
                Reporter:
                robyn Robyn Allsman [X] (Inactive)
                Reviewers:
                Robert Lupton
                Watchers:
                Kian-Tat Lim, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel

                    Time Tracking

                    Estimated:
                    Original Estimate - 1 week, 2 days
                    1w 2d
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 1 week, 2 days
                    1w 2d