Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12201

pipe_base test failure -- error creating .eups cache sub-dir

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      pipe_base is failing to eups distrib install due to what appears to be an eups failure creating a cache dir. This might be a sign of file-system trouble (full or unwritable) or some sort of eups cache corruption. Either way, the .eups dir should not be under ~ to prevent cross contamination between jobs and to ensure it gets cleaned up.

      [build] 
       
      		 
      ***** error: from /build/stack/miniconda2-4.3.21-10a4fa6/EupsBuildDir/Linux64/pipe_base-13.0-14-g8b3bf66+32/build.log:
      		 
          self._setProductStack_fromCache(dataDir, [self.flavor])
      		 
        File "/build/eups/2.1.4/python/eups/Eups.py", line 1488, in _setProductStack_fromCache
      		 
          userCacheDir = self._makeUserCacheDir(dataDir)
      		 
        File "/build/eups/2.1.4/python/eups/Eups.py", line 516, in _makeUserCacheDir
      		 
          os.makedirs(cachedir)
      		 
        File "/build/python/miniconda2-4.3.21/lib/python2.7/os.py", line 157, in makedirs
      		 
          mkdir(name, mode)
      		 
      OSError: [Errno 17] File exists: '/home/jenkins-slave/.eups/_caches_/home/jenkins-slave/.eups'
      		 
      CameraMapper INFO: Loading exposure registry from /build/stack/miniconda2-4.3.21-10a4fa6/Linux64/obs_test/13.0-14-g7685640+2/data/input/registry.sqlite3
      		 
      root INFO: Running: lsst.pipe.base.cmdLineTask.ExampleTask.parseAndRun(u'/build/stack/miniconda2-4.3.21-10a4fa6/Linux64/obs_test/13.0-14-g7685640+2/data/input', '--output', '/tmp/tmpOHmFFP', '--id', 'visit=3', 'filter=r', '--config', 'floatField=-99.9', '--clobber-config')
      		 
      ===================== 2 failed, 50 passed in 10.15 seconds =====================
      		 
      Global pytest run: failed

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12199 nightly-release d_2017_10_11 tarball build failed

          • Done

            Activity

            Ascending order - Click to sort in descending order
            15 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            That's because base defines lsst.base.getPythonPackages(). That is only used for provenance.

            Show
            tjenness Tim Jenness added a comment - That's because base defines lsst.base.getPythonPackages() . That is only used for provenance.
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            I understand the scope of the problem. Note that lsst.base.getEnvironmentPackages() is the culprit, not lsst.base.getPythonPackages(). I've opened a small PR on sconUtils to pass through EUPS_USERDATA.

            This does not solve the race condition. Cache directories should be removable by a user without causing programs that are not currently running to malfunction.
            We still either need to serialize all tests that that ultimately use Packages, make eups reentrant, or remove the usage of eups to essentially collect env vars. Eg.

            $ env | grep SETUP
            		 
            SETUP_PYTEST_FORKED=pytest_forked 0.2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYFLAKES=pyflakes 1.5.0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYCODESTYLE=pycodestyle 2.3.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTHON=python 0.0.6 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTHON_EXECNET=python_execnet 1.4.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTEST_XDIST=pytest_xdist 1.19.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTHON_MCCABE=python_mccabe 0.6.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_DOXYGEN=doxygen 1.8.13.lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYBIND11=pybind11 2.1.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_LSST_BUILD=lsst_build LOCAL:/home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/lsst_build -f Linux64 -Z (none)
            		 
            SETUP_FLAKE8=flake8 3.3.0.lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTEST=pytest 3.2.0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_BASE=base 13.0-12-g6586b4b+2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_SCONSUTILS=sconsUtils tickets.DM-12201-eups-cache-gd0f3dd09ae -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_EUPS=eups LOCAL:/home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/eups/2.1.4 -f (none) -Z (none)
            		 
            SETUP_PEP8_NAMING=pep8_naming 0.4.1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTHON_FUTURE=python_future 0.16.0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTEST_SESSION2FILE=pytest_session2file 0.1.9 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_SCONS=scons 3.0.0.lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack
            		 
            SETUP_PYTEST_FLAKE8=pytest_flake8 0.8.1+2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos-7.py3/lsstsw/stack

            Show
            jhoblitt Joshua Hoblitt added a comment - I understand the scope of the problem. Note that lsst.base.getEnvironmentPackages() is the culprit, not lsst.base.getPythonPackages() . I've opened a small PR on sconUtils to pass through EUPS_USERDATA . This does not solve the race condition. Cache directories should be removable by a user without causing programs that are not currently running to malfunction. We still either need to serialize all tests that that ultimately use Packages , make eups reentrant, or remove the usage of eups to essentially collect env vars. Eg. $ env | grep SETUP SETUP_PYTEST_FORKED=pytest_forked 0.2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYFLAKES=pyflakes 1.5 . 0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYCODESTYLE=pycodestyle 2.3 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTHON=python 0.0 . 6 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTHON_EXECNET=python_execnet 1.4 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTEST_XDIST=pytest_xdist 1.19 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTHON_MCCABE=python_mccabe 0.6 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_DOXYGEN=doxygen 1.8 . 13 .lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYBIND11=pybind11 2.1 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_LSST_BUILD=lsst_build LOCAL:/home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/lsst_build -f Linux64 -Z (none) SETUP_FLAKE8=flake8 3.3 . 0 .lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTEST=pytest 3.2 . 0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_BASE=base 13.0 - 12 -g6586b4b+ 2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_SCONSUTILS=sconsUtils tickets.DM- 12201 -eups-cache-gd0f3dd09ae -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_EUPS=eups LOCAL:/home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/eups/ 2.1 . 4 -f (none) -Z (none) SETUP_PEP8_NAMING=pep8_naming 0.4 . 1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTHON_FUTURE=python_future 0.16 . 0 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTEST_SESSION2FILE=pytest_session2file 0.1 . 9 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_SCONS=scons 3.0 . 0 .lsst1 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack SETUP_PYTEST_FLAKE8=pytest_flake8 0.8 . 1 + 2 -f Linux64 -Z /home/jenkins-slave/workspace/stack-os-matrix/centos- 7 .py3/lsstsw/stack
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            3 configurations failed 3 times in a row with last night's build. Interestingly, they were all centos 6 – for some reason the race seems more likely to be triggered on that platform.

            Show
            jhoblitt Joshua Hoblitt added a comment - 3 configurations failed 3 times in a row with last night's build. Interestingly, they were all centos 6 – for some reason the race seems more likely to be triggered on that platform.
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            I have addressed review comments and merged the sconUtils PR to preserve EUPS_USERDATA in the env.

            Show
            jhoblitt Joshua Hoblitt added a comment - I have addressed review comments and merged the sconUtils PR to preserve EUPS_USERDATA in the env.
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            This problem appears to be resolved in the sense that the CI system is no longer triggering the race condition.

            Show
            jhoblitt Joshua Hoblitt added a comment - This problem appears to be resolved in the sense that the CI system is no longer triggering the race condition.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: