Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3542

Buildbot is down

    XMLWordPrintable

    Details

      Description

      Buildbot stopped posting anything to its HipChat room on 20 August. At time of writing, the site at http://lsst-buildx.ncsa.illinois.edu:8010 is inaccessible. Looks like it needs resurrecting.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I'm investigating.

          [buildbot@lsst-buildx master]$ buildbot upgrade-master
          checking basedir
          checking for running master
          checking master.cfg
          upgrading basedir
          /usr/local/home/buildbot/master/public_html/robots.txt has old/modified contents
           writing new contents to /usr/local/home/buildbot/master/public_html/robots.txt.new
          creating /usr/local/home/buildbot/master/master.cfg.sample
          upgrading database (sqlite:///state.sqlite)
          Using SQLite Version (3, 6, 20)
          NOTE: this old version of SQLite does not support WAL journal mode; a busy master may encounter 'Database is locked' errors.  Consider upgrading.
          Traceback (most recent call last):
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 423, in errback
              self._startRunCallbacks(fail)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 490, in _startRunCallbacks
              self._runCallbacks()
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 577, in _runCallbacks
              current.result = callback(current.result, *args, **kw)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 1155, in gotResult
              _inlineCallbacks(r, g, deferred)
          --- <exception caught here> ---
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 1097, in _inlineCallbacks
              result = result.throwExceptionIntoGenerator(g)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
              return g.throw(self.type, self.value, self.tb)
            File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py", line 175, in upgradeMaster
              yield upgradeDatabase(config, master_cfg)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 1097, in _inlineCallbacks
              result = result.throwExceptionIntoGenerator(g)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
              return g.throw(self.type, self.value, self.tb)
            File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py", line 148, in upgradeDatabase
              yield db.model.upgrade()
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/threadpool.py", line 191, in _worker
              result = context.call(ctx, function, *args, **kwargs)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 118, in callWithContext
              return self.currentContext().callWithContext(ctx, func, *args, **kw)
            File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py", line 81, in callWithContext
              return func(*args,**kw)
            File "/usr/lib/python2.6/site-packages/buildbot/db/pool.py", line 184, in __thd
              rv = callable(arg, *args, **kwargs)
            File "/usr/lib/python2.6/site-packages/buildbot/db/model.py", line 525, in thd
              version_control(engine)
            File "/usr/lib/python2.6/site-packages/buildbot/db/model.py", line 490, in version_control
              self.repo_path, version)
            File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py", line 139, in create
              table = cls._create_table_version(engine, repository, version)
            File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py", line 180, in _create_table_version
              if not table.exists():
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/schema.py", line 578, in exists
              self.name, schema=self.schema)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 2425, in run_callable
              return conn.run_callable(callable_, *args, **kwargs)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1968, in run_callable
              return callable_(self, *args, **kwargs)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/sqlite/base.py", line 645, in has_table
              cursor = _pragma_cursor(connection.execute("%stable_info(%s)" % (pragma, qtable)))
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
              params)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1628, in _execute_text
              statement, parameters
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
              context)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
              context)
            File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
              cursor.execute(statement, parameters)
          sqlalchemy.exc.DatabaseError: (DatabaseError) file is encrypted or is not a database 'PRAGMA table_info("migrate_version")' ()
          
          

          Show
          jhoblitt Joshua Hoblitt added a comment - I'm investigating. [buildbot @lsst -buildx master]$ buildbot upgrade-master checking basedir checking for running master checking master.cfg upgrading basedir /usr/local/home/buildbot/master/public_html/robots.txt has old/modified contents writing new contents to /usr/local/home/buildbot/master/public_html/robots.txt. new creating /usr/local/home/buildbot/master/master.cfg.sample upgrading database (sqlite: ///state.sqlite) Using SQLite Version ( 3 , 6 , 20 ) NOTE: this old version of SQLite does not support WAL journal mode; a busy master may encounter 'Database is locked' errors. Consider upgrading. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 423 , in errback self._startRunCallbacks(fail) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 490 , in _startRunCallbacks self._runCallbacks() File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 577 , in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1155 , in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1097 , in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py" , line 389 , in throwExceptionIntoGenerator return g. throw (self.type, self.value, self.tb) File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py" , line 175 , in upgradeMaster yield upgradeDatabase(config, master_cfg) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1097 , in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py" , line 389 , in throwExceptionIntoGenerator return g. throw (self.type, self.value, self.tb) File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py" , line 148 , in upgradeDatabase yield db.model.upgrade() File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/threadpool.py" , line 191 , in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py" , line 118 , in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py" , line 81 , in callWithContext return func(*args,**kw) File "/usr/lib/python2.6/site-packages/buildbot/db/pool.py" , line 184 , in __thd rv = callable(arg, *args, **kwargs) File "/usr/lib/python2.6/site-packages/buildbot/db/model.py" , line 525 , in thd version_control(engine) File "/usr/lib/python2.6/site-packages/buildbot/db/model.py" , line 490 , in version_control self.repo_path, version) File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py" , line 139 , in create table = cls._create_table_version(engine, repository, version) File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py" , line 180 , in _create_table_version if not table.exists(): File "/usr/lib64/python2.6/site-packages/sqlalchemy/schema.py" , line 578 , in exists self.name, schema=self.schema) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 2425 , in run_callable return conn.run_callable(callable_, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1968 , in run_callable return callable_(self, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/sqlite/base.py" , line 645 , in has_table cursor = _pragma_cursor(connection.execute( "%stable_info(%s)" % (pragma, qtable))) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1449 , in execute params) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1628 , in _execute_text statement, parameters File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1698 , in _execute_context context) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1691 , in _execute_context context) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py" , line 331 , in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DatabaseError: (DatabaseError) file is encrypted or is not a database 'PRAGMA table_info("migrate_version")' ()
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I can't open the database from the cli either. My suspicion is that the database was created with sqlite 2.x which is incompatible with sqlite3.

          Show
          jhoblitt Joshua Hoblitt added a comment - I can't open the database from the cli either. My suspicion is that the database was created with sqlite 2.x which is incompatible with sqlite3.
          Hide
          frossie Frossie Economou added a comment -

          Just copy the sqlite database out of the way. It's not really needed to bring buildbot up. There's no history there we are depending on. And we can always dump it later.

          Show
          frossie Frossie Economou added a comment - Just copy the sqlite database out of the way. It's not really needed to bring buildbot up. There's no history there we are depending on. And we can always dump it later.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I was unable to open state.sqlite with sqlite2 on another host. file claims it is sqlite3.x format so I am assuming it is corrupted.

          This incantation may have resurrected it:

           mv state.sqlite state.sqlite-2015-08-27
           buildbot start # fails
           buildbot upgrade-master
           buildbot start
          

          Show
          jhoblitt Joshua Hoblitt added a comment - I was unable to open state.sqlite with sqlite2 on another host. file claims it is sqlite3.x format so I am assuming it is corrupted. This incantation may have resurrected it: mv state.sqlite state.sqlite- 2015 - 08 - 27 buildbot start # fails buildbot upgrade-master buildbot start
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The buildbot www UI (http://lsst-buildx.ncsa.illinois.edu:8010/builders/DM_stack/) has been stable since yesterday and the hourly builds have been running as expected. My expectation (without having tested it) is that this will reoccur any time the buildbot master is stopped, such as after a reboot, and the state.sqlite db will need to be manually removed.

          Show
          jhoblitt Joshua Hoblitt added a comment - The buildbot www UI ( http://lsst-buildx.ncsa.illinois.edu:8010/builders/DM_stack/ ) has been stable since yesterday and the hourly builds have been running as expected. My expectation (without having tested it) is that this will reoccur any time the buildbot master is stopped, such as after a reboot, and the state.sqlite db will need to be manually removed.

            People

            Assignee:
            jhoblitt Joshua Hoblitt
            Reporter:
            swinbank John Swinbank
            Watchers:
            Bill Glick [X] (Inactive), Frossie Economou, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.