Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11613

ctrl_execute fails with pytest-xdist

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ctrl_execute
    • Labels:
      None
    • Templates:
    • Story Points:
      0.5
    • Team:
      Data Facility

      Description

      Testing ctrl_execute with pytest-xdist (eg using ticket branches tickets/DM-11514 and tickets/DM-11514-base, results in the following error:

      ### ctrl_execute - pytest-ctrl_execute.xml.failed
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      ============================= test session starts ==============================
      platform linux -- Python 3.5.2, pytest-3.2.0, py-1.4.34, pluggy-0.4.0
      rootdir: /home/jenkins-slave/workspace/stack-os-matrix/label/centos-7/python/py3/lsstsw/build/ctrl_execute, inifile:
      plugins: session2file-0.1.9, xdist-1.19.2.dev0+g459d52e.d20170817, forked-0.3.dev0+g1dd93f6.d20170817, flake8-0.8.1
      gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I
      gw0 [37] / gw1 [37] / gw2 [37] / gw3 [37] / gw4 [37] / gw5 [37] / gw6 [37] / gw7 [37]
       
      scheduling tests via LoadScheduling
      .......................F.............
       generated xml file: /home/jenkins-slave/workspace/stack-os-matrix/label/centos-7/python/py3/lsstsw/build/ctrl_execute/tests/.tests/pytest-ctrl_execute.xml 
      =================================== FAILURES ===================================
      _____________________________ TestAllocator.test3 ______________________________
      [gw7] linux -- Python 3.5.2 /home/jenkins-slave/workspace/stack-os-matrix/label/centos-7/python/py3/lsstsw/miniconda/bin/python
       
      self = <test_allocator.TestAllocator testMethod=test3>
       
          def test3(self):
              sys.argv = self.regularArgs()
              al = self.subSetup("config_condor.py")
          
              fileName = os.path.join("tests", "testfiles", "config_allocation.py")
      >       al.loadPbs(fileName)
       
      tests/test_allocator.py:116: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      python/lsst/ctrl/execute/pbsPlugin.py:109: in loadPbs
          configuration = self.loadAllocationConfig(name, "pbs")
      python/lsst/ctrl/execute/allocator.py:202: in loadAllocationConfig
          os.makedirs(self.configDir)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
       
      name = './tests/condor_scratch/configs', mode = 511, exist_ok = False
       
          def makedirs(name, mode=0o777, exist_ok=False):
              """makedirs(name [, mode=0o777][, exist_ok=False])
          
              Super-mkdir; create a leaf directory and all intermediate ones.  Works like
              mkdir, except that any intermediate path segment (not just the rightmost)
              will be created if it does not exist. If the target directory already
              exists, raise an OSError if exist_ok is False. Otherwise no exception is
              raised.  This is recursive.
          
              """
              head, tail = path.split(name)
              if not tail:
                  head, tail = path.split(head)
              if head and tail and not path.exists(head):
                  try:
                      makedirs(head, mode, exist_ok)
                  except FileExistsError:
                      # Defeats race condition when another thread created the path
                      pass
                  cdir = curdir
                  if isinstance(tail, bytes):
                      cdir = bytes(curdir, 'ASCII')
                  if tail == cdir:           # xxx/newdir/. exists if xxx/newdir exists
                      return
              try:
      >           mkdir(name, mode)
      E           FileExistsError: [Errno 17] File exists: './tests/condor_scratch/configs'
       
      ../../miniconda/lib/python3.5/os.py:241: FileExistsError
      =============================== warnings summary ===============================
      tests/test_configurator.py::TestConfigurator::test4
        /home/jenkins-slave/workspace/stack-os-matrix/label/centos-7/python/py3/lsstsw/eups/2.1.3/python/eups/hooks.py:253: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/jenkins-slave/workspace/stack-os-matrix/label/centos-7/python/py3/lsstsw/stack/site/startup.py' mode='r' encoding='ANSI_X3.4-1968'>
          exec(compile(open(startupFile).read(), startupFile, 'exec'), _globals, locals())
       
      -- Docs: http://doc.pytest.org/en/latest/warnings.html
      =============== 1 failed, 36 passed, 1 warnings in 3.60 seconds ================
      

      The problem seems to be that tests are assuming they are running sequentially but with pytest -n 8 individual tests can run in different processes and since the tests use the same config directory they all race with each other.

      Steve Pietrowicz before I look deeply at the code, is there an obvious way for the tests to use different temp directories or config files?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                spietrowicz Steve Pietrowicz
                Reporter:
                tjenness Tim Jenness
                Reviewers:
                Tim Jenness
                Watchers:
                John Swinbank, Steve Pietrowicz, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel