Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11691

Test failure with butler in obs_sdss

    XMLWordPrintable

Details

    • 1
    • DB_S17_8
    • Data Access and Database

    Description

      In a Jenkins run last night the Mac failed with an error in obs_sdss:

      RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/Users/square/jenkins/workspace/stack-os-matrix/osx.py3/lsstsw/build/obs_sdss', mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x141d71320>, mapperArgs={}, parents=[], policy=None), RepositoryArgs:RepositoryArgs(root='/Users/square/jenkins/workspace/stack-os-matrix/osx.py3/lsstsw/build/obs_sdss', cfgRoot=None, mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x141d4d320>, mapperArgs=None, tags=set(), mode='rw', policy=None)
      Stacktrace
      self = <testGetId.GetIdTestCase testMethod=testId>
          def setUp(self):
              self.bf = dafPersist.ButlerFactory(mapper=SdssMapper(root="."))
      >       self.butler = self.bf.create()
      tests/testGetId.py:37: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      ../../stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butlerFactory.py:79: in create
          return Butler(root=root, mapper=self.mapper)
      ../../stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py:521: in __init__
          self._getCfgs(repoDataList)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      self = <[AttributeError("'Butler' object has no attribute '_repos'") raised in repr()] Butler object at 0x141d4d6d8>
      repoDataList = [RepoData(id=5399557680,repoArgs=RepositoryArgs(root='/Users/square/jenkins/workspace/stack-os-matrix/osx.py3/lsstsw/b...fg=None,cfgOrigin=None,cfgRoot=None,repo=None,parentRepoDatas=[],isV1Repository=False,role=output,parentRegistry=None)]
          def _getCfgs(self, repoDataList):
              """Get or make a RepositoryCfg for each RepoData, and add the cfg to the RepoData.
                  If the cfg exists, compare values. If values match then use the cfg as an "existing" cfg. If the
                  values do not match, use the cfg as a "nested" cfg.
                  If the cfg does not exist, the RepositoryArgs must be for a writable repository.
          
                  Parameters
                  ----------
                  repoDataList : list of RepoData
                      The RepoData that are output and inputs of this Butler
          
                  Raises
                  ------
                  RuntimeError
                      If the passed-in RepositoryArgs indicate an existing repository but other cfg parameters in those
                      RepositoryArgs don't
                      match the existing repository's cfg a RuntimeError will be raised.
                  """
              def cfgMatchesArgs(args, cfg):
                  """Test if there are any values in an RepositoryArgs that conflict with the values in a cfg"""
                  if args.mapper is not None and cfg.mapper != args.mapper:
                      return False
                  if args.mapperArgs is not None and cfg.mapperArgs != args.mapperArgs:
                      return False
                  if args.policy is not None and cfg.policy != args.policy:
                      return False
                  return True
          
              for repoData in repoDataList:
                  cfg, isOldButlerRepository = self._getRepositoryCfg(repoData.repoArgs)
                  if cfg is None:
                      if 'w' not in repoData.repoArgs.mode:
                          raise RuntimeError(
                              "No cfg found for read-only input repository at {}".format(repoData.repoArgs.cfgRoot))
                      repoData.setCfg(cfg=RepositoryCfg.makeFromArgs(repoData.repoArgs),
                                      origin='new',
                                      root=repoData.repoArgs.cfgRoot,
                                      isV1Repository=isOldButlerRepository)
                  else:
                      if 'w' in repoData.repoArgs.mode:
                          # if it's an output repository, the RepositoryArgs must match the existing cfg.
                          if not cfgMatchesArgs(repoData.repoArgs, cfg):
                              raise RuntimeError(("The RepositoryArgs and RepositoryCfg must match for writable " +
                                                  "repositories, RepositoryCfg:{}, RepositoryArgs:{}").format(
      >                                               cfg, repoData.repoArgs))
      E                       RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/Users/square/jenkins/workspace/stack-os-matrix/osx.py3/lsstsw/build/obs_sdss', mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x141d71320>, mapperArgs={}, parents=[], policy=None), RepositoryArgs:RepositoryArgs(root='/Users/square/jenkins/workspace/stack-os-matrix/osx.py3/lsstsw/build/obs_sdss', cfgRoot=None, mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x141d4d320>, mapperArgs=None, tags=set(), mode='rw', policy=None)
      ../../stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py:799: RuntimeError
      

      Attachments

        Issue Links

          Activity

            also, when Butler init fails and raises an exception, _repr_ gets called but raises because self.repos has not been created yet. __repr_ needs to be fixed to be more defensive.

            npease Nate Pease [X] (Inactive) added a comment - also, when Butler init fails and raises an exception, _ repr _ gets called but raises because self. repos has not been created yet. __repr _ needs to be fixed to be more defensive.
            tjenness Tim Jenness added a comment -

            Not sure if you've worked this out yet, but just in case. The problem with the test is that it leaves context behind that is not cleared up. In a clean directory tests/testGetId.py runs fine.

            $ python tests/testGetId.py 
            CameraMapper INFO: Loading Posix exposure registry from .
            daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object.
            daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object.
            ...
            ----------------------------------------------------------------------
            Ran 3 tests in 0.535s
             
            OK
            

            but after it runs it leaves behind a repositoryCfg.yaml file which prevents it from running again:

            $ python tests/testGetId.py 
            CameraMapper INFO: Loading Posix exposure registry from .
            daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object.
            daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object.
            CameraMapper INFO: Loading Posix exposure registry from .
            E.
            336 Objects leaked:
            ...
            1029: 0x7f81e4f4cec8 lsst::afw::table::BaseTable
            1027: 0x7f81e4f44b00 lsst::daf::base::PropertySet
            1026: 0x7f81e4f39a38 lsst::afw::table::BaseRecord
            1025: 0x7f81e4f49058 lsst::afw::table::BaseRecord
            1024: 0x7f81e4f44a58 lsst::afw::table::detail::SchemaImpl
            1023: 0x7f81e4f39e38 lsst::afw::table::BaseTable
            1021: 0x7f81e4f47940 lsst::daf::base::PropertySet
            1020: 0x7f81e4f39d88 lsst::afw::table::BaseRecord
            1019: 0x7f81e4f42c48 lsst::afw::table::BaseRecord
            1018: 0x7f81e4f47b98 lsst::afw::table::detail::SchemaImpl
            1017: 0x7f81e4f486c8 lsst::afw::table::BaseTable
            1015: 0x7f81e4f3a810 lsst::daf::base::PropertySet
            1014: 0x7f81e4f3c928 lsst::afw::table::BaseRecord
            1013: 0x7f81e4f3c838 lsst::afw::table::BaseRecord
            1012: 0x7f81e4f38728 lsst::afw::table::detail::SchemaImpl
            1011: 0x7f81e4f386b8 lsst::afw::table::BaseTable
            1008: 0x7f81e4adfae0 lsst::daf::base::PropertySet
            366: 0x7f81e4eaa068 lsst::daf::persistence::Persistence
            365: 0x7f81e4e5c0a8 lsst::daf::base::PropertySet
            364: 0x7f81e4e67e88 lsst::pex::policy::Policy
            F
            ======================================================================
            ERROR: testId (__main__.GetIdTestCase)
            Test retrieval of exposure ids
            ----------------------------------------------------------------------
            Traceback (most recent call last):
              File "tests/testGetId.py", line 37, in setUp
                self.butler = self.bf.create()
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butlerFactory.py", line 79, in create
                return Butler(root=root, mapper=self.mapper)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py", line 521, in __init__
                self._getCfgs(repoDataList)
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py", line 799, in _getCfgs
                cfg, repoData.repoArgs))
            RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/Users/timj/work/lsstsw3/build/obs_sdss', mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x1483848d0>, mapperArgs={}, parents=[], policy=None), RepositoryArgs:RepositoryArgs(root='/Users/timj/work/lsstsw3/build/obs_sdss', cfgRoot=None, mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x148365080>, mapperArgs=None, tags=set(), mode='rw', policy=None)
             
            ======================================================================
            FAIL: testLeaks (__main__.TestMemory)
            !Check for memory leaks in the preceding tests
            ----------------------------------------------------------------------
            Traceback (most recent call last):
              File "/Users/timj/work/lsstsw3/stack/DarwinX86/utils/13.0-9-gf29e843/python/lsst/utils/tests.py", line 161, in testLeaks
                self.fail("Leaked %d block%s" % (nleak, plural))
            AssertionError: Leaked 336 blocks
             
            ----------------------------------------------------------------------
            Ran 3 tests in 0.867s
             
            FAILED (failures=1, errors=1)
            

            So the test should probably use a temporary directory.

            tjenness Tim Jenness added a comment - Not sure if you've worked this out yet, but just in case. The problem with the test is that it leaves context behind that is not cleared up. In a clean directory tests/testGetId.py runs fine. $ python tests/testGetId.py CameraMapper INFO: Loading Posix exposure registry from . daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object. daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object. ... ---------------------------------------------------------------------- Ran 3 tests in 0.535s   OK but after it runs it leaves behind a repositoryCfg.yaml file which prevents it from running again: $ python tests/testGetId.py CameraMapper INFO: Loading Posix exposure registry from . daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object. daf.persistence.butler WARN: Passing an instantiated mapper into Butler.__init__ will prevent Butler from passing parentRegistry or repositoryCfg information to the mapper, which is done only at init time. It is better to pass a importable string or class object. CameraMapper INFO: Loading Posix exposure registry from . E. 336 Objects leaked: ... 1029: 0x7f81e4f4cec8 lsst::afw::table::BaseTable 1027: 0x7f81e4f44b00 lsst::daf::base::PropertySet 1026: 0x7f81e4f39a38 lsst::afw::table::BaseRecord 1025: 0x7f81e4f49058 lsst::afw::table::BaseRecord 1024: 0x7f81e4f44a58 lsst::afw::table::detail::SchemaImpl 1023: 0x7f81e4f39e38 lsst::afw::table::BaseTable 1021: 0x7f81e4f47940 lsst::daf::base::PropertySet 1020: 0x7f81e4f39d88 lsst::afw::table::BaseRecord 1019: 0x7f81e4f42c48 lsst::afw::table::BaseRecord 1018: 0x7f81e4f47b98 lsst::afw::table::detail::SchemaImpl 1017: 0x7f81e4f486c8 lsst::afw::table::BaseTable 1015: 0x7f81e4f3a810 lsst::daf::base::PropertySet 1014: 0x7f81e4f3c928 lsst::afw::table::BaseRecord 1013: 0x7f81e4f3c838 lsst::afw::table::BaseRecord 1012: 0x7f81e4f38728 lsst::afw::table::detail::SchemaImpl 1011: 0x7f81e4f386b8 lsst::afw::table::BaseTable 1008: 0x7f81e4adfae0 lsst::daf::base::PropertySet 366: 0x7f81e4eaa068 lsst::daf::persistence::Persistence 365: 0x7f81e4e5c0a8 lsst::daf::base::PropertySet 364: 0x7f81e4e67e88 lsst::pex::policy::Policy F ====================================================================== ERROR: testId (__main__.GetIdTestCase) Test retrieval of exposure ids ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/testGetId.py", line 37, in setUp self.butler = self.bf.create() File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butlerFactory.py", line 79, in create return Butler(root=root, mapper=self.mapper) File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py", line 521, in __init__ self._getCfgs(repoDataList) File "/Users/timj/work/lsstsw3/stack/DarwinX86/daf_persistence/13.0-32-g7b14ddd/python/lsst/daf/persistence/butler.py", line 799, in _getCfgs cfg, repoData.repoArgs)) RuntimeError: The RepositoryArgs and RepositoryCfg must match for writable repositories, RepositoryCfg:RepositoryCfg(root='/Users/timj/work/lsstsw3/build/obs_sdss', mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x1483848d0>, mapperArgs={}, parents=[], policy=None), RepositoryArgs:RepositoryArgs(root='/Users/timj/work/lsstsw3/build/obs_sdss', cfgRoot=None, mapper=<lsst.obs.sdss.sdssMapper.SdssMapper object at 0x148365080>, mapperArgs=None, tags=set(), mode='rw', policy=None)   ====================================================================== FAIL: testLeaks (__main__.TestMemory) !Check for memory leaks in the preceding tests ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/timj/work/lsstsw3/stack/DarwinX86/utils/13.0-9-gf29e843/python/lsst/utils/tests.py", line 161, in testLeaks self.fail("Leaked %d block%s" % (nleak, plural)) AssertionError: Leaked 336 blocks   ---------------------------------------------------------------------- Ran 3 tests in 0.867s   FAILED (failures=1, errors=1) So the test should probably use a temporary directory.

            I haven't tested in Jenkins yet, I'm waiting on the non-ascii character fix in pipe_base before I can do that. However, it builds & passes through multiple iterations on my machine.

            npease Nate Pease [X] (Inactive) added a comment - I haven't tested in Jenkins yet, I'm waiting on the non-ascii character fix in pipe_base before I can do that. However, it builds & passes through multiple iterations on my machine.
            tjenness Tim Jenness added a comment -

            obs_decam has a similar test but handles it differently by having a test directory with a yaml file in it.

            tjenness Tim Jenness added a comment - obs_decam has a similar test but handles it differently by having a test directory with a yaml file in it.
            tjenness Tim Jenness added a comment -

            Looks great. Thank you.

            tjenness Tim Jenness added a comment - Looks great. Thank you.

            People

              npease Nate Pease [X] (Inactive)
              tjenness Tim Jenness
              Tim Jenness
              Nate Pease [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.