Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7457

daf_persistence 3e6e15d fails build

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I ran rebuild this morning and got a failure in daf_persistence:

      Error message below. Full build log and failure logs attached.

      [2016-08-31T13:40:10.381424Z] ======================================================================
      [2016-08-31T13:40:10.381449Z] ERROR: testPickle (__main__.ButlerPickleTestCase)
      [2016-08-31T13:40:10.381482Z] ----------------------------------------------------------------------
      [2016-08-31T13:40:10.381500Z] Traceback (most recent call last):
      [2016-08-31T13:40:10.381524Z]   File "tests/butlerPickle.py", line 50, in setUp
      [2016-08-31T13:40:10.381555Z]     self.butler = dafPersist.Butler(root='.', mapper=MinMapper())
      [2016-08-31T13:40:10.381604Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 285, in __init__
      [2016-08-31T13:40:10.381648Z]     self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)
      [2016-08-31T13:40:10.381697Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 345, in _addRepo
      [2016-08-31T13:40:10.381734Z]     cfg = Storage.getRepositoryCfg(args.cfgRoot)
      [2016-08-31T13:40:10.381785Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg
      [2016-08-31T13:40:10.381815Z]     ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)
      [2016-08-31T13:40:10.381868Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 78, in getRepositoryCfg
      [2016-08-31T13:40:10.381894Z]     repositoryCfg = PosixStorage._getRepositoryCfg(uri)
      [2016-08-31T13:40:10.381948Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 71, in _getRepositoryCfg
      [2016-08-31T13:40:10.381965Z]     repositoryCfg = yaml.load(f)
      [2016-08-31T13:40:10.382013Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/__init__.py", line 71, in load
      [2016-08-31T13:40:10.382031Z]     return loader.get_single_data()
      [2016-08-31T13:40:10.382083Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 39, in get_single_data
      [2016-08-31T13:40:10.382104Z]     return self.construct_document(node)
      [2016-08-31T13:40:10.382158Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 43, in construct_document
      [2016-08-31T13:40:10.382177Z]     data = self.construct_object(node)
      [2016-08-31T13:40:10.382230Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 88, in construct_object
      [2016-08-31T13:40:10.382248Z]     data = constructor(self, node)
      [2016-08-31T13:40:10.382300Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 61, in v1Constructor
      [2016-08-31T13:40:10.382320Z]     d = loader.construct_mapping(node)
      [2016-08-31T13:40:10.382374Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 208, in construct_mapping
      [2016-08-31T13:40:10.382404Z]     return BaseConstructor.construct_mapping(self, node, deep=deep)
      [2016-08-31T13:40:10.382458Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 133, in construct_mapping
      [2016-08-31T13:40:10.382495Z]     value = self.construct_object(value_node, deep=deep)
      [2016-08-31T13:40:10.382549Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 90, in construct_object
      [2016-08-31T13:40:10.382572Z]     data = constructor(self, tag_suffix, node)
      [2016-08-31T13:40:10.382630Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 610, in construct_python_object_new
      [2016-08-31T13:40:10.382662Z]     return self.construct_python_object_apply(suffix, node, newobj=True)
      [2016-08-31T13:40:10.382721Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply
      [2016-08-31T13:40:10.382754Z]     instance = self.make_python_instance(suffix, node, args, kwds, newobj)
      [2016-08-31T13:40:10.382809Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instance
      [2016-08-31T13:40:10.382836Z]     cls = self.find_python_name(suffix, node.start_mark)
      [2016-08-31T13:40:10.382889Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name
      [2016-08-31T13:40:10.382905Z]     module.__name__), mark)
      [2016-08-31T13:40:10.382929Z] ConstructorError: while constructing a Python object
      [2016-08-31T13:40:10.382953Z] cannot find 'ImgMapper' in the module '__main__'
      [2016-08-31T13:40:10.382975Z]   in "./repositoryCfg.yaml", line 3, column 10
      [2016-08-31T13:40:10.382980Z]
      [2016-08-31T13:40:10.383012Z] ----------------------------------------------------------------------
      [2016-08-31T13:40:10.383025Z] Ran 4 tests in 0.064s
      [2016-08-31T13:40:10.383029Z]
      [2016-08-31T13:40:10.383041Z] FAILED (errors=2)
      [2016-08-31T13:40:10.383056Z] The following tests failed:
      [2016-08-31T13:40:10.387426Z] /Volumes/PS1/lsstsw/build/daf_persistence/tests/.tests/butlerPickle.py.failed
      [2016-08-31T13:40:10.387545Z] 1 tests failed
      [2016-08-31T13:40:10.388400Z] scons: *** [checkTestStatus] Error 1
      [2016-08-31T13:40:10.391647Z] scons: building terminated because of errors.
      

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Thanks! Merged & pushed.

            Show
            swinbank John Swinbank added a comment - Thanks! Merged & pushed.
            Hide
            npease Nate Pease added a comment -

            looks good.

            Show
            npease Nate Pease added a comment - looks good.
            Hide
            swinbank John Swinbank added a comment - - edited

            Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address DM-11488.

            Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/

            Show
            swinbank John Swinbank added a comment - - edited Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address DM-11488 . Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/
            Hide
            swinbank John Swinbank added a comment -

            See also the daf_persistence error reported on DM-11488, which is almost the same as the problem here. When fixing this, check if they could be related.

            Show
            swinbank John Swinbank added a comment - See also the daf_persistence error reported on DM-11488 , which is almost the same as the problem here. When fixing this, check if they could be related.
            Hide
            swinbank John Swinbank added a comment - - edited

            Ok, found the problem.

            ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests. When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper, which is only defined within butlerSubset.py.

            If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time. It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase, doesn't know anything about ImgMapper, and so falls over.

            Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.

            Show
            swinbank John Swinbank added a comment - - edited Ok, found the problem. ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests . When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper , which is only defined within butlerSubset.py . If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time . It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase , doesn't know anything about ImgMapper , and so falls over. Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.

              People

              • Assignee:
                swinbank John Swinbank
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Reviewers:
                Nate Pease
                Watchers:
                John Swinbank, Michael Wood-Vasey, Nate Pease, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel