Details

    • Sprint:
      DevOps Sprint 1, DevOps Sprint 2, DevOps Sprint 3
    • Team:
      SQuaRE

      Description

      Configure buildbot slave on lsst-dev. Initially the owner will be 'lsstsw2'.

        Attachments

          Issue Links

            Activity

            Hide
            spietrowicz Steve Pietrowicz added a comment - - edited

            Just a few minor comments. I saw changes in the code that is moving the location of lsstsw2 to lsstsw, which I think it meant to be a global change. There are a few references to the old location, some of which are in /usr/local/home. I wasn't sure if there were actually two different directories for this.

            scripts/lsstswBuild.sh - "lsst-data" should be "lsst-dm-dev"

            scripts/README_forceBuild line 8, should (I think) read:

            */10 * * * * /lsst/home/lsstsw/bin/forceBuild-cron

            master/master.cfg - I believe that the references to /usr/local/home/buildbot should be /lsst/home/lsstsw, but again, I was unsure if this was a wholesale change to a new home directory or not.

            buildslaves/lsst-dev/buildbot.tac line 15 should read:

            basedir = r'/lsst/home/lsstsw/BBSlaves/lsst-dev'

            scripts/lsstswBuild.sh line 103 (which is commented out, and should be) but it has a reference to /usr/local/home/lsstsw instead of /lsst/home/lsstsw

            scripts/create_xlinkdocs.sh: has references to /usr/local/home

            scripts/gitConstancts.sh has references to /usr/local/home/buildbot

            Show
            spietrowicz Steve Pietrowicz added a comment - - edited Just a few minor comments. I saw changes in the code that is moving the location of lsstsw2 to lsstsw, which I think it meant to be a global change. There are a few references to the old location, some of which are in /usr/local/home. I wasn't sure if there were actually two different directories for this. scripts/lsstswBuild.sh - "lsst-data" should be "lsst-dm-dev" scripts/README_forceBuild line 8, should (I think) read: */10 * * * * /lsst/home/lsstsw/bin/forceBuild-cron master/master.cfg - I believe that the references to /usr/local/home/buildbot should be /lsst/home/lsstsw, but again, I was unsure if this was a wholesale change to a new home directory or not. buildslaves/lsst-dev/buildbot.tac line 15 should read: basedir = r'/lsst/home/lsstsw/BBSlaves/lsst-dev' scripts/lsstswBuild.sh line 103 (which is commented out, and should be) but it has a reference to /usr/local/home/lsstsw instead of /lsst/home/lsstsw scripts/create_xlinkdocs.sh: has references to /usr/local/home scripts/gitConstancts.sh has references to /usr/local/home/buildbot
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            Steve,
            Thanks for the detailed review. You caught some "didn't update the internal comments when updating the code". I'll be more careful in the future.

            I'll list your findings and address them individually. The first one is unresolved.

            • master/master.cfg - I believe that the references to /usr/local/home/buildbot should be /lsst/home/lsstsw, but again, I was unsure if this was a wholesale change to a new home directory or not.
              >>> I couldn't find this in the ticket branch nor in the production code. I think you might have found an earlier commit on the ticket branch?
            • scripts/lsstswBuild.sh - "lsst-data" should be "lsst-dm-dev"
              >>> fixed comment
            • scripts/README_forceBuild line 8, should (I think) read:
              */10 * * * * /lsst/home/lsstsw/bin/forceBuild-cron
              >>> fixed comment
            • buildslaves/lsst-dev/buildbot.tac line 15 should read:
              basedir = r'/lsst/home/lsstsw/BBSlaves/lsst-dev'
              >>> Good catch. It referred to the lsstsw2@lsst-dev beta release. I don't really understand why there wasn't a complaint when the slave was reinstalled on and as lsstsw@lsst-dev and put into production. The slave has been stopped and restarted many times.
            • scripts/lsstswBuild.sh line 103 (which is commented out, and should be) but it has a reference to /usr/local/home/lsstsw instead of /lsst/home/lsstsw
              >>> The important aspect of these lines is the leading "*** " which designates an scons message. The rest of the line could be greeking since this is a test dataset exercising the recognition of errors in the buildbot status log. I changed the text to not confuse folks.
            • scripts/create_xlinkdocs.sh: has references to /usr/local/home
              >>> I updated the example.

            Again, thanks for the detailed review.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - Steve, Thanks for the detailed review. You caught some "didn't update the internal comments when updating the code". I'll be more careful in the future. I'll list your findings and address them individually. The first one is unresolved. master/master.cfg - I believe that the references to /usr/local/home/buildbot should be /lsst/home/lsstsw, but again, I was unsure if this was a wholesale change to a new home directory or not. >>> I couldn't find this in the ticket branch nor in the production code. I think you might have found an earlier commit on the ticket branch? scripts/lsstswBuild.sh - "lsst-data" should be "lsst-dm-dev" >>> fixed comment scripts/README_forceBuild line 8, should (I think) read: */10 * * * * /lsst/home/lsstsw/bin/forceBuild-cron >>> fixed comment buildslaves/lsst-dev/buildbot.tac line 15 should read: basedir = r'/lsst/home/lsstsw/BBSlaves/lsst-dev' >>> Good catch. It referred to the lsstsw2@lsst-dev beta release. I don't really understand why there wasn't a complaint when the slave was reinstalled on and as lsstsw@lsst-dev and put into production. The slave has been stopped and restarted many times. scripts/lsstswBuild.sh line 103 (which is commented out, and should be) but it has a reference to /usr/local/home/lsstsw instead of /lsst/home/lsstsw >>> The important aspect of these lines is the leading "*** " which designates an scons message. The rest of the line could be greeking since this is a test dataset exercising the recognition of errors in the buildbot status log. I changed the text to not confuse folks. scripts/create_xlinkdocs.sh: has references to /usr/local/home >>> I updated the example. scripts/gitConstants.sh has references to /usr/local/home/buildbot >>> The comment was curious to say the least. I removed the definition (it was never used anywhere) and added a new explanation of where and how the doxydoc user-facing web address is connected to actual location of the doxydoc files. This explanation is now in the Buildbot Installation manual: https://confluence.lsstcorp.org/display/LDMDG/Buildbot+Configuration+and+Setup#BuildbotConfigurationandSetup-SetupWebAddressforUserFacingDocumentation Again, thanks for the detailed review.
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            Robyn,

            I saw line 37 in master/master.cfg (and also just noticed that line 40 might need a change too.

            Steve

            Show
            spietrowicz Steve Pietrowicz added a comment - Robyn, I saw line 37 in master/master.cfg (and also just noticed that line 40 might need a change too. Steve
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited

            Ah, it all becomes clear. The players are a confused bunch in this installation.
            There is buildbot user on the Build Master - a separate host which contains the .httpasswd file. That's on an isolated system with its files maintained on the local-to-the-host directory.

            There is user buildbot who is the buildbot slave owner for the VM's on the lsst-cluster network. It usually contains the BuildSlave definitions for ALL the slaves hosted on the network but this is not required. Buildbot's home directory is on the standard user home partition. Buildbot maintains the scripts and the doxydoc files in its home directory.

            There is user lsstsw who is the buildbot slave owner on lsst-dev. This is done for the convenience of the build and distrib environment.

            The paths reflected in the master.cfg file reference these various entities and locations, e.g.,

            1. Master's Authentication file
              HTPASSWD='/usr/local/home/buildbot/.....
            2. Slaves' Primary Directories
              BUILDBOT_HOME = "/lsst/home/buildbot/RHEL6/"
              LDEV_FAILED_LOGS = "/lsst/home/lsstsw/build/FailedLogs"
            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited Ah, it all becomes clear. The players are a confused bunch in this installation. There is buildbot user on the Build Master - a separate host which contains the .httpasswd file. That's on an isolated system with its files maintained on the local-to-the-host directory. There is user buildbot who is the buildbot slave owner for the VM's on the lsst-cluster network. It usually contains the BuildSlave definitions for ALL the slaves hosted on the network but this is not required. Buildbot's home directory is on the standard user home partition. Buildbot maintains the scripts and the doxydoc files in its home directory. There is user lsstsw who is the buildbot slave owner on lsst-dev. This is done for the convenience of the build and distrib environment. The paths reflected in the master.cfg file reference these various entities and locations, e.g., Master's Authentication file HTPASSWD='/usr/local/home/buildbot/..... Slaves' Primary Directories BUILDBOT_HOME = "/lsst/home/buildbot/RHEL6/" LDEV_FAILED_LOGS = "/lsst/home/lsstsw/build/FailedLogs"
            Hide
            spietrowicz Steve Pietrowicz added a comment -

            Thanks! That clears it up.

            Show
            spietrowicz Steve Pietrowicz added a comment - Thanks! That clears it up.

              People

              • Assignee:
                robyn Robyn Allsman [X] (Inactive)
                Reporter:
                mjuric Mario Juric
                Reviewers:
                Steve Pietrowicz
                Watchers:
                Mario Juric, Steve Pietrowicz
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel

                    Time Tracking

                    Estimated:
                    Original Estimate - 10 weeks
                    10w
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 10 weeks
                    10w