Joshua Hoblitt added a comment - 27/Aug/15 1:08 PM I'm investigating. [buildbot @lsst -buildx master]$ buildbot upgrade-master checking basedir checking for running master checking master.cfg upgrading basedir /usr/local/home/buildbot/master/public_html/robots.txt has old/modified contents writing new contents to /usr/local/home/buildbot/master/public_html/robots.txt. new creating /usr/local/home/buildbot/master/master.cfg.sample upgrading database (sqlite: ///state.sqlite) Using SQLite Version ( 3 , 6 , 20 ) NOTE: this old version of SQLite does not support WAL journal mode; a busy master may encounter 'Database is locked' errors. Consider upgrading. Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 423 , in errback self._startRunCallbacks(fail) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 490 , in _startRunCallbacks self._runCallbacks() File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 577 , in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1155 , in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1097 , in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py" , line 389 , in throwExceptionIntoGenerator return g. throw (self.type, self.value, self.tb) File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py" , line 175 , in upgradeMaster yield upgradeDatabase(config, master_cfg) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py" , line 1097 , in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/failure.py" , line 389 , in throwExceptionIntoGenerator return g. throw (self.type, self.value, self.tb) File "/usr/lib/python2.6/site-packages/buildbot/scripts/upgrade_master.py" , line 148 , in upgradeDatabase yield db.model.upgrade() File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/threadpool.py" , line 191 , in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py" , line 118 , in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.6/site-packages/Twisted-13.2.0-py2.6-linux-x86_64.egg/twisted/python/context.py" , line 81 , in callWithContext return func(*args,**kw) File "/usr/lib/python2.6/site-packages/buildbot/db/pool.py" , line 184 , in __thd rv = callable(arg, *args, **kwargs) File "/usr/lib/python2.6/site-packages/buildbot/db/model.py" , line 525 , in thd version_control(engine) File "/usr/lib/python2.6/site-packages/buildbot/db/model.py" , line 490 , in version_control self.repo_path, version) File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py" , line 139 , in create table = cls._create_table_version(engine, repository, version) File "/usr/lib/python2.6/site-packages/migrate/versioning/schema.py" , line 180 , in _create_table_version if not table.exists(): File "/usr/lib64/python2.6/site-packages/sqlalchemy/schema.py" , line 578 , in exists self.name, schema=self.schema) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 2425 , in run_callable return conn.run_callable(callable_, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1968 , in run_callable return callable_(self, *args, **kwargs) File "/usr/lib64/python2.6/site-packages/sqlalchemy/dialects/sqlite/base.py" , line 645 , in has_table cursor = _pragma_cursor(connection.execute( "%stable_info(%s)" % (pragma, qtable))) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1449 , in execute params) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1628 , in _execute_text statement, parameters File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1698 , in _execute_context context) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py" , line 1691 , in _execute_context context) File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py" , line 331 , in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DatabaseError: (DatabaseError) file is encrypted or is not a database 'PRAGMA table_info("migrate_version")' ()

Joshua Hoblitt added a comment - 27/Aug/15 1:20 PM I can't open the database from the cli either. My suspicion is that the database was created with sqlite 2.x which is incompatible with sqlite3.

Frossie Economou added a comment - 27/Aug/15 1:25 PM Just copy the sqlite database out of the way. It's not really needed to bring buildbot up. There's no history there we are depending on. And we can always dump it later.

Joshua Hoblitt added a comment - 27/Aug/15 1:29 PM I was unable to open state.sqlite with sqlite2 on another host. file claims it is sqlite3.x format so I am assuming it is corrupted. This incantation may have resurrected it: mv state.sqlite state.sqlite- 2015 - 08 - 27 buildbot start # fails buildbot upgrade-master buildbot start

Joshua Hoblitt added a comment - 28/Aug/15 10:37 AM The buildbot www UI ( http://lsst-buildx.ncsa.illinois.edu:8010/builders/DM_stack/ ) has been stable since yesterday and the hourly builds have been running as expected. My expectation (without having tested it) is that this will reoccur any time the buildbot master is stopped, such as after a reboot, and the state.sqlite db will need to be manually removed.