Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7457

daf_persistence 3e6e15d fails build

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I ran rebuild this morning and got a failure in daf_persistence:

      Error message below. Full build log and failure logs attached.

      [2016-08-31T13:40:10.381424Z] ======================================================================
      [2016-08-31T13:40:10.381449Z] ERROR: testPickle (__main__.ButlerPickleTestCase)
      [2016-08-31T13:40:10.381482Z] ----------------------------------------------------------------------
      [2016-08-31T13:40:10.381500Z] Traceback (most recent call last):
      [2016-08-31T13:40:10.381524Z]   File "tests/butlerPickle.py", line 50, in setUp
      [2016-08-31T13:40:10.381555Z]     self.butler = dafPersist.Butler(root='.', mapper=MinMapper())
      [2016-08-31T13:40:10.381604Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 285, in __init__
      [2016-08-31T13:40:10.381648Z]     self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)
      [2016-08-31T13:40:10.381697Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 345, in _addRepo
      [2016-08-31T13:40:10.381734Z]     cfg = Storage.getRepositoryCfg(args.cfgRoot)
      [2016-08-31T13:40:10.381785Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg
      [2016-08-31T13:40:10.381815Z]     ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)
      [2016-08-31T13:40:10.381868Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 78, in getRepositoryCfg
      [2016-08-31T13:40:10.381894Z]     repositoryCfg = PosixStorage._getRepositoryCfg(uri)
      [2016-08-31T13:40:10.381948Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 71, in _getRepositoryCfg
      [2016-08-31T13:40:10.381965Z]     repositoryCfg = yaml.load(f)
      [2016-08-31T13:40:10.382013Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/__init__.py", line 71, in load
      [2016-08-31T13:40:10.382031Z]     return loader.get_single_data()
      [2016-08-31T13:40:10.382083Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 39, in get_single_data
      [2016-08-31T13:40:10.382104Z]     return self.construct_document(node)
      [2016-08-31T13:40:10.382158Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 43, in construct_document
      [2016-08-31T13:40:10.382177Z]     data = self.construct_object(node)
      [2016-08-31T13:40:10.382230Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 88, in construct_object
      [2016-08-31T13:40:10.382248Z]     data = constructor(self, node)
      [2016-08-31T13:40:10.382300Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 61, in v1Constructor
      [2016-08-31T13:40:10.382320Z]     d = loader.construct_mapping(node)
      [2016-08-31T13:40:10.382374Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 208, in construct_mapping
      [2016-08-31T13:40:10.382404Z]     return BaseConstructor.construct_mapping(self, node, deep=deep)
      [2016-08-31T13:40:10.382458Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 133, in construct_mapping
      [2016-08-31T13:40:10.382495Z]     value = self.construct_object(value_node, deep=deep)
      [2016-08-31T13:40:10.382549Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 90, in construct_object
      [2016-08-31T13:40:10.382572Z]     data = constructor(self, tag_suffix, node)
      [2016-08-31T13:40:10.382630Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 610, in construct_python_object_new
      [2016-08-31T13:40:10.382662Z]     return self.construct_python_object_apply(suffix, node, newobj=True)
      [2016-08-31T13:40:10.382721Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply
      [2016-08-31T13:40:10.382754Z]     instance = self.make_python_instance(suffix, node, args, kwds, newobj)
      [2016-08-31T13:40:10.382809Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instance
      [2016-08-31T13:40:10.382836Z]     cls = self.find_python_name(suffix, node.start_mark)
      [2016-08-31T13:40:10.382889Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name
      [2016-08-31T13:40:10.382905Z]     module.__name__), mark)
      [2016-08-31T13:40:10.382929Z] ConstructorError: while constructing a Python object
      [2016-08-31T13:40:10.382953Z] cannot find 'ImgMapper' in the module '__main__'
      [2016-08-31T13:40:10.382975Z]   in "./repositoryCfg.yaml", line 3, column 10
      [2016-08-31T13:40:10.382980Z]
      [2016-08-31T13:40:10.383012Z] ----------------------------------------------------------------------
      [2016-08-31T13:40:10.383025Z] Ran 4 tests in 0.064s
      [2016-08-31T13:40:10.383029Z]
      [2016-08-31T13:40:10.383041Z] FAILED (errors=2)
      [2016-08-31T13:40:10.383056Z] The following tests failed:
      [2016-08-31T13:40:10.387426Z] /Volumes/PS1/lsstsw/build/daf_persistence/tests/.tests/butlerPickle.py.failed
      [2016-08-31T13:40:10.387545Z] 1 tests failed
      [2016-08-31T13:40:10.388400Z] scons: *** [checkTestStatus] Error 1
      [2016-08-31T13:40:10.391647Z] scons: building terminated because of errors.
      

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Works for me on python 2 and 3 on macOS Sierra. Jenkins seems to be okay as well.

            daf_persistence: 12.0-8-ge6890fd+1 ok (36.2 sec).
            

            Show
            tjenness Tim Jenness added a comment - Works for me on python 2 and 3 on macOS Sierra. Jenkins seems to be okay as well. daf_persistence: 12.0-8-ge6890fd+1 ok (36.2 sec).
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            @timj Thanks for the feedback. I'll spend some time trying to debug locally.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - @timj Thanks for the feedback. I'll spend some time trying to debug locally.
            Hide
            npease Nate Pease added a comment -

            should this ticket be assigned to Michael Wood-Vasey for now then?

            Show
            npease Nate Pease added a comment - should this ticket be assigned to Michael Wood-Vasey for now then?
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Just build successfully on my laptop. The error was on my desktop.

            I've taken this ticket back.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Just build successfully on my laptop. The error was on my desktop. I've taken this ticket back.
            Hide
            tjenness Tim Jenness added a comment -

            Nate Pease yes. Isn't the real problem here that Michael Wood-Vasey is building a non-standard branch? The commit in question is from branch tickets/DM-6985.

            Show
            tjenness Tim Jenness added a comment - Nate Pease yes. Isn't the real problem here that Michael Wood-Vasey is building a non-standard branch? The commit in question is from branch tickets/ DM-6985 .
            Hide
            tjenness Tim Jenness added a comment -

            Although 3e6e15d also builds for me:

             daf_persistence: tickets.DM-6985-g3e6e15dc72 .............ok (80.3 sec).
            

            Show
            tjenness Tim Jenness added a comment - Although 3e6e15d also builds for me: daf_persistence: tickets.DM-6985-g3e6e15dc72 .............ok (80.3 sec).
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            A contributor is that Michael Wood-Vasey can't successfully copy and paste. I meant e6890fd. Apologies for the confusion.

            It's not a strong day for Michael Wood-Vasey ...

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - A contributor is that Michael Wood-Vasey can't successfully copy and paste. I meant e6890fd . Apologies for the confusion. It's not a strong day for Michael Wood-Vasey ...
            Hide
            npease Nate Pease added a comment -

            Michael Wood-Vasey should hang in there (eups+git branches+less-than-clear-head can make for a rough day) and let Nate Pease know if there's anything that needs fixing in daf_persistence.

            Show
            npease Nate Pease added a comment - Michael Wood-Vasey should hang in there (eups+git branches+less-than-clear-head can make for a rough day) and let Nate Pease know if there's anything that needs fixing in daf_persistence.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            And now it works...

            [serenity ~] rebuild -u daf_persistence
                 daf_persistence:  ok (3.8 sec).
                   mariadbclient:  ok (1.3 sec).
                     pex_logging:  ok (2.2 sec).
                  pex_exceptions:  ok (0.7 sec).
                            base:  ok (0.6 sec).
                      sconsUtils:  ok (0.7 sec).
                           scons:  ok (0.5 sec).
                          python:  ok (0.4 sec).
                         doxygen:  ok (0.4 sec).
                            swig:  ok (0.5 sec).
                   python_future:  ok (0.3 sec).
                           boost:  ok (0.5 sec).
                        daf_base:  ok (1.2 sec).
                           utils:  ok (1.3 sec).
                             log:  ok (0.5 sec).
                         log4cxx:  ok (0.4 sec).
                             apr:  ok (0.4 sec).
                        apr_util:  ok (0.3 sec).
                           numpy:  ok (0.4 sec).
                   python_psutil:  ok (0.3 sec).
                      pex_policy:  ok (2.0 sec).
                         astropy:  ok (0.3 sec).
                          pyyaml:  ok (1.2 sec).
            # BUILD ID: b2005
                         doxygen: 1.8.5.lsst1 (already installed).
                             apr: 1.5.2 (already installed).
                            swig: 3.0.10 (already installed).
                          python: 0.0.5 (already installed).
                         astropy: 0.0.1.lsst2+1 (already installed).
                   mariadbclient: 10.1.11.lsst3 (already installed).
                        apr_util: 1.5.4 (already installed).
                   python_future: 0.15.2+1 (already installed).
                   python_psutil: 4.1.0+2 (already installed).
                           numpy: 0.0.3 (already installed).
                          pyyaml: 3.11.lsst1+2 (already installed).
                           boost: 1.60.lsst1 (already installed).
                           scons: 2.5.0.lsst2 (already installed).
                      sconsUtils: 12.0-5-g67b2f9e (already installed).
                         log4cxx: 0.10.0.lsst6+1 (already installed).
                            base: 12.0-5-g19372bd+1 (already installed).
                  pex_exceptions: 12.0-1-g4e7a85b+1 (already installed).
                             log: 12.0-4-gbdaee9d (already installed).
                           utils: 12.0-10-g3b62a7d (already installed).
                        daf_base: 12.0-4-g3a3e718+1 (already installed).
                     pex_logging: 12.0-1-gc6c1df4+1 (already installed).
                      pex_policy: 12.0-3-g3e90007+1 (already installed).
                 daf_persistence: 12.0-8-ge6890fd+1 .ok (36.5 sec).
            # BUILD b2005 completed.
            #
            

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - And now it works... [serenity ~] rebuild -u daf_persistence daf_persistence: ok (3.8 sec). mariadbclient: ok (1.3 sec). pex_logging: ok (2.2 sec). pex_exceptions: ok (0.7 sec). base: ok (0.6 sec). sconsUtils: ok (0.7 sec). scons: ok (0.5 sec). python: ok (0.4 sec). doxygen: ok (0.4 sec). swig: ok (0.5 sec). python_future: ok (0.3 sec). boost: ok (0.5 sec). daf_base: ok (1.2 sec). utils: ok (1.3 sec). log: ok (0.5 sec). log4cxx: ok (0.4 sec). apr: ok (0.4 sec). apr_util: ok (0.3 sec). numpy: ok (0.4 sec). python_psutil: ok (0.3 sec). pex_policy: ok (2.0 sec). astropy: ok (0.3 sec). pyyaml: ok (1.2 sec). # BUILD ID: b2005 doxygen: 1.8.5.lsst1 (already installed). apr: 1.5.2 (already installed). swig: 3.0.10 (already installed). python: 0.0.5 (already installed). astropy: 0.0.1.lsst2+1 (already installed). mariadbclient: 10.1.11.lsst3 (already installed). apr_util: 1.5.4 (already installed). python_future: 0.15.2+1 (already installed). python_psutil: 4.1.0+2 (already installed). numpy: 0.0.3 (already installed). pyyaml: 3.11.lsst1+2 (already installed). boost: 1.60.lsst1 (already installed). scons: 2.5.0.lsst2 (already installed). sconsUtils: 12.0-5-g67b2f9e (already installed). log4cxx: 0.10.0.lsst6+1 (already installed). base: 12.0-5-g19372bd+1 (already installed). pex_exceptions: 12.0-1-g4e7a85b+1 (already installed). log: 12.0-4-gbdaee9d (already installed). utils: 12.0-10-g3b62a7d (already installed). daf_base: 12.0-4-g3a3e718+1 (already installed). pex_logging: 12.0-1-gc6c1df4+1 (already installed). pex_policy: 12.0-3-g3e90007+1 (already installed). daf_persistence: 12.0-8-ge6890fd+1 .ok (36.5 sec). # BUILD b2005 completed. #
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            No idea what happened. I think I'll just go off in a corner and try to reboot the day.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - No idea what happened. I think I'll just go off in a corner and try to reboot the day.
            Hide
            swinbank John Swinbank added a comment -

            For what it's worth, Michael Wood-Vasey is not crazy. Or, at least, if he is, this ticket is not evidence for it. I just hit the same:

            tests/butlerPickle.py
             
            /tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to O
            SError:Could not find unix home directory to search for astropy config dir on None
              warn(ConfigurationMissingWarning(msg))
            .E..
            ======================================================================
            ERROR: testPickle (__main__.ButlerPickleTestCase)
            ----------------------------------------------------------------------
            Traceback (most recent call last):
              File "tests/butlerPickle.py", line 67, in testPickle
                butler = pickle.loads(pickledButler)
              File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1388, in loads
                return Unpickler(file).load()
              File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 864, in load
                dispatch[key](self)
              File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1139, in load_reduce
                value = func(*args)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 943, in _unreduce
                butler = Butler(**initArgs)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 304, in __init__
                self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 369, in _addRepo
                cfg = Storage.getRepositoryCfg(args.cfgRoot)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg
                ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 79, in getRepositoryCfg
                repositoryCfg = PosixStorage._getRepositoryCfg(uri)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 72, in _getRepositoryCfg
                repositoryCfg = yaml.load(f)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/__init__.py", line 71, in load
                return loader.get_single_data()
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 39, in get_single_data
                return self.construct_document(node)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 43, in construct_document
                data = self.construct_object(node)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 88, in construct_object
                data = constructor(self, node)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 63, in v1Constructor
                d = loader.construct_mapping(node)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 208, in construct_mapping
                return BaseConstructor.construct_mapping(self, node, deep=deep)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 133, in construct_mapping
                value = self.construct_object(value_node, deep=deep)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 90, in construct_object
                data = constructor(self, tag_suffix, node)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 610, in construct_python_object_new
                return self.construct_python_object_apply(suffix, node, newobj=True)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply
                instance = self.make_python_instance(suffix, node, args, kwds, newobj)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 540, in make_python_instance
                cls = self.find_python_name(suffix, node.start_mark)
              File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 513, in find_python_name
                module.__name__), mark)
            ConstructorError: while constructing a Python object
            cannot find 'ImgMapper' in the module '__main__'
              in "./repositoryCfg.yaml", line 3, column 10
            

            A couple of things to note:

            • This happened when I was trying to build old weeklies (I noticed it with w_2017_5 and w_2017_7). I've not reproduced it with the latest weekly (but I haven't seriously tried yet).
            • I've only noticed it when building with a lot of jobs (scons -j 28). That suggests that it might be a race. On the other hand, it might just be that I've not tried often enough with a single job (-j 1) because that takes much longer.

            I'll poke at this a bit harder in the morning.

            Show
            swinbank John Swinbank added a comment - For what it's worth, Michael Wood-Vasey is not crazy. Or, at least, if he is, this ticket is not evidence for it. I just hit the same: tests/butlerPickle.py   /tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to O SError:Could not find unix home directory to search for astropy config dir on None warn(ConfigurationMissingWarning(msg)) .E.. ====================================================================== ERROR: testPickle (__main__.ButlerPickleTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/butlerPickle.py", line 67, in testPickle butler = pickle.loads(pickledButler) File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1388, in loads return Unpickler(file).load() File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1139, in load_reduce value = func(*args) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 943, in _unreduce butler = Butler(**initArgs) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 304, in __init__ self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 369, in _addRepo cfg = Storage.getRepositoryCfg(args.cfgRoot) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 79, in getRepositoryCfg repositoryCfg = PosixStorage._getRepositoryCfg(uri) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 72, in _getRepositoryCfg repositoryCfg = yaml.load(f) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/__init__.py", line 71, in load return loader.get_single_data() File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 39, in get_single_data return self.construct_document(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 43, in construct_document data = self.construct_object(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 88, in construct_object data = constructor(self, node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 63, in v1Constructor d = loader.construct_mapping(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 208, in construct_mapping return BaseConstructor.construct_mapping(self, node, deep=deep) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 133, in construct_mapping value = self.construct_object(value_node, deep=deep) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 90, in construct_object data = constructor(self, tag_suffix, node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 610, in construct_python_object_new return self.construct_python_object_apply(suffix, node, newobj=True) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply instance = self.make_python_instance(suffix, node, args, kwds, newobj) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 540, in make_python_instance cls = self.find_python_name(suffix, node.start_mark) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 513, in find_python_name module.__name__), mark) ConstructorError: while constructing a Python object cannot find 'ImgMapper' in the module '__main__' in "./repositoryCfg.yaml", line 3, column 10 A couple of things to note: This happened when I was trying to build old weeklies (I noticed it with w_2017_5 and w_2017_7 ). I've not reproduced it with the latest weekly (but I haven't seriously tried yet). I've only noticed it when building with a lot of jobs ( scons -j 28 ). That suggests that it might be a race. On the other hand, it might just be that I've not tried often enough with a single job ( -j 1 ) because that takes much longer. I'll poke at this a bit harder in the morning.
            Hide
            swinbank John Swinbank added a comment -

            (Reopening the ticket at least until I can convince myself that there's nothing here worth worrying about.)

            Show
            swinbank John Swinbank added a comment - (Reopening the ticket at least until I can convince myself that there's nothing here worth worrying about.)
            Hide
            swinbank John Swinbank added a comment - - edited

            Ok, found the problem.

            ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests. When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper, which is only defined within butlerSubset.py.

            If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time. It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase, doesn't know anything about ImgMapper, and so falls over.

            Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.

            Show
            swinbank John Swinbank added a comment - - edited Ok, found the problem. ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests . When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper , which is only defined within butlerSubset.py . If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time . It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase , doesn't know anything about ImgMapper , and so falls over. Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.
            Hide
            swinbank John Swinbank added a comment -

            See also the daf_persistence error reported on DM-11488, which is almost the same as the problem here. When fixing this, check if they could be related.

            Show
            swinbank John Swinbank added a comment - See also the daf_persistence error reported on DM-11488 , which is almost the same as the problem here. When fixing this, check if they could be related.
            Hide
            swinbank John Swinbank added a comment - - edited

            Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address DM-11488.

            Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/

            Show
            swinbank John Swinbank added a comment - - edited Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address DM-11488 . Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/
            Hide
            npease Nate Pease added a comment -

            looks good.

            Show
            npease Nate Pease added a comment - looks good.
            Hide
            swinbank John Swinbank added a comment -

            Thanks! Merged & pushed.

            Show
            swinbank John Swinbank added a comment - Thanks! Merged & pushed.

              People

              • Assignee:
                swinbank John Swinbank
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Reviewers:
                Nate Pease
                Watchers:
                John Swinbank, Michael Wood-Vasey, Nate Pease, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: