Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12945

shapelet test failures

    Details

      Description

      shapelet is failing to build from master. The git repo has not changed since September, so it's highly probable this is related to the devtoolset-6 change over late yesterday afternoon.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The three failed build attemps from last nighths nightly-release build:

            All attempts failed with test errors similar to:

            ============================= test session starts ==============================
            platform linux -- Python 3.6.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0
            rootdir: /home/jenkins-slave/snowflake/release/lsstsw/build/shapelet, inifile: setup.cfg
            plugins: session2file-0.1.9, xdist-1.20.1, forked-0.2, flake8-0.9.1
            gw0 I / gw1 I / gw2 I / gw3 I
            [gw2] node down: Not properly terminated
            Slave restarting disabled
            [gw0] node down: Not properly terminated
            Slave restarting disabled
            [gw1] node down: Not properly terminated
            Slave restarting disabled
            [gw3] node down: Not properly terminated
            Slave restarting disabled
            INTERNALERROR> Traceback (most recent call last):
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py", line 110, in wrap_session
            INTERNALERROR>     session.exitstatus = doit(config, session) or 0
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py", line 146, in _main
            INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py", line 745, in __call__
            INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
            INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
            INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py", line 614, in execute
            INTERNALERROR>     res = hook_impl.function(*args)
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py", line 114, in pytest_runtestloop
            INTERNALERROR>     self.loop_once()
            INTERNALERROR>   File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py", line 126, in loop_once
            INTERNALERROR>     raise RuntimeError("Unexpectedly no active workers available")
            INTERNALERROR> RuntimeError: Unexpectedly no active workers available
             
            ========================= no tests ran in 2.93 seconds =========================
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The three failed build attemps from last nighths nightly-release build: https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/471/pipeline/ https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/472/pipeline/ https://ci.lsst.codes/blue/organizations/jenkins/release%2Frun-rebuild/detail/run-rebuild/473/pipeline/ All attempts failed with test errors similar to: ============================= test session starts ============================== platform linux -- Python 3.6 . 2 , pytest- 3.2 . 0 , py- 1.4 . 34 , pluggy- 0.4 . 0 rootdir: /home/jenkins-slave/snowflake/release/lsstsw/build/shapelet, inifile: setup.cfg plugins: session2file- 0.1 . 9 , xdist- 1.20 . 1 , forked- 0.2 , flake8- 0.9 . 1 gw0 I / gw1 I / gw2 I / gw3 I [gw2] node down: Not properly terminated Slave restarting disabled [gw0] node down: Not properly terminated Slave restarting disabled [gw1] node down: Not properly terminated Slave restarting disabled [gw3] node down: Not properly terminated Slave restarting disabled INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py" , line 110 , in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/main.py" , line 146 , in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 745 , in __call__ INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 339 , in _hookexec INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 334 , in <lambda> INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute() INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest/3.2.0-1-g78ee252/lib/python/pytest-3.2.0-py3.6.egg/_pytest/vendored_packages/pluggy.py" , line 614 , in execute INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py" , line 114 , in pytest_runtestloop INTERNALERROR> self.loop_once() INTERNALERROR> File "/home/jenkins-slave/snowflake/release/lsstsw/stack/Linux64/pytest_xdist/1.20.1-1-g65af3e3/lib/python/pytest_xdist-1.20.1-py3.6.egg/xdist/dsession.py" , line 126 , in loop_once INTERNALERROR> raise RuntimeError( "Unexpectedly no active workers available" ) INTERNALERROR> RuntimeError: Unexpectedly no active workers available   ========================= no tests ran in 2.93 seconds =========================
            Hide
            price Paul Price added a comment -

            So this probably isn't directly related to the shapelet package, but it happens to be the first pybind or pytest user that gets built, right?

            Show
            price Paul Price added a comment - So this probably isn't directly related to the shapelet package, but it happens to be the first pybind or pytest user that gets built, right?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The current thinking, after some debugging on slack, is that this is related to the old system gcc built env still being in place. The jenkins run-rebuild and run-publish jobs are "special" in that they share a jenkins "workspace" that is in an non-standard path. Due to this fact, the env wasn't blown away in my devtoolset-6 transition cleanup yesterday. I've nuked the previous workspace and am rerunning the build. We should know in a couple of hours if that was the problem.

            Show
            jhoblitt Joshua Hoblitt added a comment - The current thinking, after some debugging on slack, is that this is related to the old system gcc built env still being in place. The jenkins run-rebuild and run-publish jobs are "special" in that they share a jenkins "workspace" that is in an non-standard path. Due to this fact, the env wasn't blown away in my devtoolset-6 transition cleanup yesterday. I've nuked the previous workspace and am rerunning the build. We should know in a couple of hours if that was the problem.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Are we sure all packages were rebuilt from scratch with the new compiler?

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Are we sure all packages were rebuilt from scratch with the new compiler?
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Right

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Right
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The new build has made it past shapelet. Sorry for the noise folks!

            Show
            jhoblitt Joshua Hoblitt added a comment - The new build has made it past shapelet . Sorry for the noise folks!

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Jim Bosch, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Paul Price, Pim Schellart [X] (Inactive), Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel