Joshua Hoblitt added a comment - 07/Dec/17 7:30 AM The three failed build attemps from last nighths nightly-release build: https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/471/pipeline/ https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/472/pipeline/ https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/473/pipeline/ All attempts failed with test errors similar to: ============================= test session starts ============================== platform linux -- Python 3.6 . 2 , pytest- 3.2 . 0 , py- 1.4 . 34 , pluggy- 0.4 . 0 rootdir: /home/jenkins-slave/snowflake/release/lsstsw/build/shapelet, inifile: setup.cfg plugins: session2file- 0.1 . 9 , xdist- 1.20 . 1 , forked- 0.2 , flake8- 0.9 . 1 gw0 I / gw1 I / gw2 I / gw3 I [gw2] node down: Not properly terminated Slave restarting disabled [gw0] node down: Not properly terminated Slave restarting disabled [gw1] node down: Not properly terminated Slave restarting disabled [gw3] node down: Not properly terminated Slave restarting disabled INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py" , line 110 , in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py" , line 146 , in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 745 , in __call__ INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 339 , in _hookexec INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 334 , in <lambda> INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute() INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 614 , in execute INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py" , line 114 , in pytest_runtestloop INTERNALERROR> self.loop_once() INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py" , line 126 , in loop_once INTERNALERROR> raise RuntimeError( "Unexpectedly no active workers available" ) INTERNALERROR> RuntimeError: Unexpectedly no active workers available   ========================= no tests ran in 2.93 seconds =========================

Paul Price added a comment - 07/Dec/17 7:41 AM So this probably isn't directly related to the shapelet package, but it happens to be the first pybind or pytest user that gets built, right?

Joshua Hoblitt added a comment - 07/Dec/17 8:15 AM The current thinking, after some debugging on slack, is that this is related to the old system gcc built env still being in place. The jenkins run-rebuild and run-publish jobs are "special" in that they share a jenkins "workspace" that is in an non-standard path. Due to this fact, the env wasn't blown away in my devtoolset-6 transition cleanup yesterday. I've nuked the previous workspace and am rerunning the build. We should know in a couple of hours if that was the problem.

Pim Schellart [X] (Inactive) added a comment - 07/Dec/17 8:16 AM Are we sure all packages were rebuilt from scratch with the new compiler?

Pim Schellart [X] (Inactive) added a comment - 07/Dec/17 8:16 AM Right

Joshua Hoblitt added a comment - 07/Dec/17 11:26 AM The new build has made it past shapelet . Sorry for the noise folks!