[DM-7457] daf_persistence 3e6e15d fails build - Jira

XML

Word

Printable

Details

Type: Bug
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: None
Labels:
None

Story Points:
1
Epic Link:
f17-drp-soft-maint
Sprint:
DRP F17-3
Team:
Data Release Production

Description

I ran rebuild this morning and got a failure in daf_persistence:

Error message below. Full build log and failure logs attached.

[2016-08-31T13:40:10.381424Z] ======================================================================

[2016-08-31T13:40:10.381449Z] ERROR: testPickle (__main__.ButlerPickleTestCase)

[2016-08-31T13:40:10.381482Z] ----------------------------------------------------------------------

[2016-08-31T13:40:10.381500Z] Traceback (most recent call last):

[2016-08-31T13:40:10.381524Z]   File "tests/butlerPickle.py", line 50, in setUp

[2016-08-31T13:40:10.381555Z]     self.butler = dafPersist.Butler(root='.', mapper=MinMapper())

[2016-08-31T13:40:10.381604Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 285, in __init__

[2016-08-31T13:40:10.381648Z]     self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)

[2016-08-31T13:40:10.381697Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 345, in _addRepo

[2016-08-31T13:40:10.381734Z]     cfg = Storage.getRepositoryCfg(args.cfgRoot)

[2016-08-31T13:40:10.381785Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg

[2016-08-31T13:40:10.381815Z]     ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)

[2016-08-31T13:40:10.381868Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 78, in getRepositoryCfg

[2016-08-31T13:40:10.381894Z]     repositoryCfg = PosixStorage._getRepositoryCfg(uri)

[2016-08-31T13:40:10.381948Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 71, in _getRepositoryCfg

[2016-08-31T13:40:10.381965Z]     repositoryCfg = yaml.load(f)

[2016-08-31T13:40:10.382013Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/__init__.py", line 71, in load

[2016-08-31T13:40:10.382031Z]     return loader.get_single_data()

[2016-08-31T13:40:10.382083Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 39, in get_single_data

[2016-08-31T13:40:10.382104Z]     return self.construct_document(node)

[2016-08-31T13:40:10.382158Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 43, in construct_document

[2016-08-31T13:40:10.382177Z]     data = self.construct_object(node)

[2016-08-31T13:40:10.382230Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 88, in construct_object

[2016-08-31T13:40:10.382248Z]     data = constructor(self, node)

[2016-08-31T13:40:10.382300Z]   File "/Volumes/PS1/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 61, in v1Constructor

[2016-08-31T13:40:10.382320Z]     d = loader.construct_mapping(node)

[2016-08-31T13:40:10.382374Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 208, in construct_mapping

[2016-08-31T13:40:10.382404Z]     return BaseConstructor.construct_mapping(self, node, deep=deep)

[2016-08-31T13:40:10.382458Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 133, in construct_mapping

[2016-08-31T13:40:10.382495Z]     value = self.construct_object(value_node, deep=deep)

[2016-08-31T13:40:10.382549Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 90, in construct_object

[2016-08-31T13:40:10.382572Z]     data = constructor(self, tag_suffix, node)

[2016-08-31T13:40:10.382630Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 610, in construct_python_object_new

[2016-08-31T13:40:10.382662Z]     return self.construct_python_object_apply(suffix, node, newobj=True)

[2016-08-31T13:40:10.382721Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply

[2016-08-31T13:40:10.382754Z]     instance = self.make_python_instance(suffix, node, args, kwds, newobj)

[2016-08-31T13:40:10.382809Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 540, in make_python_instance

[2016-08-31T13:40:10.382836Z]     cls = self.find_python_name(suffix, node.start_mark)

[2016-08-31T13:40:10.382889Z]   File "/Users/wmwv/lsstsw/stack/DarwinX86/pyyaml/3.11.lsst1+2/lib/python/yaml/constructor.py", line 513, in find_python_name

[2016-08-31T13:40:10.382905Z]     module.__name__), mark)

[2016-08-31T13:40:10.382929Z] ConstructorError: while constructing a Python object

[2016-08-31T13:40:10.382953Z] cannot find 'ImgMapper' in the module '__main__'

[2016-08-31T13:40:10.382975Z]   in "./repositoryCfg.yaml", line 3, column 10

[2016-08-31T13:40:10.382980Z]

[2016-08-31T13:40:10.383012Z] ----------------------------------------------------------------------

[2016-08-31T13:40:10.383025Z] Ran 4 tests in 0.064s

[2016-08-31T13:40:10.383029Z]

[2016-08-31T13:40:10.383041Z] FAILED (errors=2)

[2016-08-31T13:40:10.383056Z] The following tests failed:

[2016-08-31T13:40:10.387426Z] /Volumes/PS1/lsstsw/build/daf_persistence/tests/.tests/butlerPickle.py.failed

[2016-08-31T13:40:10.387545Z] 1 tests failed

[2016-08-31T13:40:10.388400Z] scons: *** [checkTestStatus] Error 1

[2016-08-31T13:40:10.391647Z] scons: building terminated because of errors.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

_build.log
94 kB
31/Aug/16 6:48 AM
butlerPickle.py.failed
7 kB
31/Aug/16 6:48 AM

Issue Links

relates to

DM-11488 weekly release w_2017_31 failed

Done

Activity

Ascending order - Click to sort in descending order

Michael Wood-Vasey created issue - 31/Aug/16 6:50 AM

Tim Jenness made changes - 31/Aug/16 8:18 AM

Field	Original Value	New Value
Watchers	Michael Wood-Vasey [ Michael Wood-Vasey ]	Michael Wood-Vasey, Nate Pease [ Michael Wood-Vasey, Nate Pease ]

Hide

Permalink

Tim Jenness added a comment - 31/Aug/16 8:18 AM

Works for me on python 2 and 3 on macOS Sierra. Jenkins seems to be okay as well.

daf_persistence: 12.0-8-ge6890fd+1 ok (36.2 sec).

Show

Tim Jenness added a comment - 31/Aug/16 8:18 AM Works for me on python 2 and 3 on macOS Sierra. Jenkins seems to be okay as well. daf_persistence: 12.0-8-ge6890fd+1 ok (36.2 sec).

Hide

Permalink

Michael Wood-Vasey added a comment - 31/Aug/16 8:21 AM

@timj Thanks for the feedback. I'll spend some time trying to debug locally.

Show

Michael Wood-Vasey added a comment - 31/Aug/16 8:21 AM @timj Thanks for the feedback. I'll spend some time trying to debug locally.

Hide

Permalink

Nate Pease added a comment - 31/Aug/16 8:27 AM

should this ticket be assigned to Michael Wood-Vasey for now then?

Show

Nate Pease added a comment - 31/Aug/16 8:27 AM should this ticket be assigned to Michael Wood-Vasey for now then?

Michael Wood-Vasey made changes - 31/Aug/16 8:31 AM

Assignee

Michael Wood-Vasey [ wmwood-vasey ]

Hide

Permalink

Michael Wood-Vasey added a comment - 31/Aug/16 8:31 AM

Just build successfully on my laptop. The error was on my desktop.

I've taken this ticket back.

Show

Michael Wood-Vasey added a comment - 31/Aug/16 8:31 AM Just build successfully on my laptop. The error was on my desktop. I've taken this ticket back.

Hide

Permalink

Tim Jenness added a comment - 31/Aug/16 8:32 AM

Nate Pease yes. Isn't the real problem here that Michael Wood-Vasey is building a non-standard branch? The commit in question is from branch tickets/DM-6985.

Show

Tim Jenness added a comment - 31/Aug/16 8:32 AM Nate Pease yes. Isn't the real problem here that Michael Wood-Vasey is building a non-standard branch? The commit in question is from branch tickets/ DM-6985 .

Hide

Permalink

Tim Jenness added a comment - 31/Aug/16 8:38 AM

Although 3e6e15d also builds for me:

 daf_persistence: tickets.DM-6985-g3e6e15dc72 .............ok (80.3 sec).

Show

Tim Jenness added a comment - 31/Aug/16 8:38 AM Although 3e6e15d also builds for me: daf_persistence: tickets.DM-6985-g3e6e15dc72 .............ok (80.3 sec).

Hide

Permalink

Michael Wood-Vasey added a comment - 31/Aug/16 8:46 AM

A contributor is that Michael Wood-Vasey can't successfully copy and paste. I meant e6890fd. Apologies for the confusion.

It's not a strong day for Michael Wood-Vasey ...

Show

Michael Wood-Vasey added a comment - 31/Aug/16 8:46 AM A contributor is that Michael Wood-Vasey can't successfully copy and paste. I meant e6890fd . Apologies for the confusion. It's not a strong day for Michael Wood-Vasey ...

Hide

Permalink

Nate Pease added a comment - 31/Aug/16 9:06 AM

Michael Wood-Vasey should hang in there (eups+git branches+less-than-clear-head can make for a rough day) and let Nate Pease know if there's anything that needs fixing in daf_persistence.

Show

Nate Pease added a comment - 31/Aug/16 9:06 AM Michael Wood-Vasey should hang in there (eups+git branches+less-than-clear-head can make for a rough day) and let Nate Pease know if there's anything that needs fixing in daf_persistence.

Hide

Permalink

Michael Wood-Vasey added a comment - 31/Aug/16 10:39 AM

And now it works...

[serenity ~] rebuild -u daf_persistence

     daf_persistence:  ok (3.8 sec).

       mariadbclient:  ok (1.3 sec).

         pex_logging:  ok (2.2 sec).

      pex_exceptions:  ok (0.7 sec).

                base:  ok (0.6 sec).

          sconsUtils:  ok (0.7 sec).

               scons:  ok (0.5 sec).

              python:  ok (0.4 sec).

             doxygen:  ok (0.4 sec).

                swig:  ok (0.5 sec).

       python_future:  ok (0.3 sec).

               boost:  ok (0.5 sec).

            daf_base:  ok (1.2 sec).

               utils:  ok (1.3 sec).

                 log:  ok (0.5 sec).

             log4cxx:  ok (0.4 sec).

                 apr:  ok (0.4 sec).

            apr_util:  ok (0.3 sec).

               numpy:  ok (0.4 sec).

       python_psutil:  ok (0.3 sec).

          pex_policy:  ok (2.0 sec).

             astropy:  ok (0.3 sec).

              pyyaml:  ok (1.2 sec).

# BUILD ID: b2005

             doxygen: 1.8.5.lsst1 (already installed).

                 apr: 1.5.2 (already installed).

                swig: 3.0.10 (already installed).

              python: 0.0.5 (already installed).

             astropy: 0.0.1.lsst2+1 (already installed).

       mariadbclient: 10.1.11.lsst3 (already installed).

            apr_util: 1.5.4 (already installed).

       python_future: 0.15.2+1 (already installed).

       python_psutil: 4.1.0+2 (already installed).

               numpy: 0.0.3 (already installed).

              pyyaml: 3.11.lsst1+2 (already installed).

               boost: 1.60.lsst1 (already installed).

               scons: 2.5.0.lsst2 (already installed).

          sconsUtils: 12.0-5-g67b2f9e (already installed).

             log4cxx: 0.10.0.lsst6+1 (already installed).

                base: 12.0-5-g19372bd+1 (already installed).

      pex_exceptions: 12.0-1-g4e7a85b+1 (already installed).

                 log: 12.0-4-gbdaee9d (already installed).

               utils: 12.0-10-g3b62a7d (already installed).

            daf_base: 12.0-4-g3a3e718+1 (already installed).

         pex_logging: 12.0-1-gc6c1df4+1 (already installed).

          pex_policy: 12.0-3-g3e90007+1 (already installed).

     daf_persistence: 12.0-8-ge6890fd+1 .ok (36.5 sec).

# BUILD b2005 completed.

Show

Michael Wood-Vasey added a comment - 31/Aug/16 10:39 AM And now it works... [serenity ~] rebuild -u daf_persistence daf_persistence: ok (3.8 sec). mariadbclient: ok (1.3 sec). pex_logging: ok (2.2 sec). pex_exceptions: ok (0.7 sec). base: ok (0.6 sec). sconsUtils: ok (0.7 sec). scons: ok (0.5 sec). python: ok (0.4 sec). doxygen: ok (0.4 sec). swig: ok (0.5 sec). python_future: ok (0.3 sec). boost: ok (0.5 sec). daf_base: ok (1.2 sec). utils: ok (1.3 sec). log: ok (0.5 sec). log4cxx: ok (0.4 sec). apr: ok (0.4 sec). apr_util: ok (0.3 sec). numpy: ok (0.4 sec). python_psutil: ok (0.3 sec). pex_policy: ok (2.0 sec). astropy: ok (0.3 sec). pyyaml: ok (1.2 sec). # BUILD ID: b2005 doxygen: 1.8.5.lsst1 (already installed). apr: 1.5.2 (already installed). swig: 3.0.10 (already installed). python: 0.0.5 (already installed). astropy: 0.0.1.lsst2+1 (already installed). mariadbclient: 10.1.11.lsst3 (already installed). apr_util: 1.5.4 (already installed). python_future: 0.15.2+1 (already installed). python_psutil: 4.1.0+2 (already installed). numpy: 0.0.3 (already installed). pyyaml: 3.11.lsst1+2 (already installed). boost: 1.60.lsst1 (already installed). scons: 2.5.0.lsst2 (already installed). sconsUtils: 12.0-5-g67b2f9e (already installed). log4cxx: 0.10.0.lsst6+1 (already installed). base: 12.0-5-g19372bd+1 (already installed). pex_exceptions: 12.0-1-g4e7a85b+1 (already installed). log: 12.0-4-gbdaee9d (already installed). utils: 12.0-10-g3b62a7d (already installed). daf_base: 12.0-4-g3a3e718+1 (already installed). pex_logging: 12.0-1-gc6c1df4+1 (already installed). pex_policy: 12.0-3-g3e90007+1 (already installed). daf_persistence: 12.0-8-ge6890fd+1 .ok (36.5 sec). # BUILD b2005 completed. #

Hide

Permalink

Michael Wood-Vasey added a comment - 31/Aug/16 10:40 AM

No idea what happened. I think I'll just go off in a corner and try to reboot the day.

Show

Michael Wood-Vasey added a comment - 31/Aug/16 10:40 AM No idea what happened. I think I'll just go off in a corner and try to reboot the day.

Michael Wood-Vasey made changes - 31/Aug/16 10:40 AM

Resolution		Done [ 10000 ]
Status	To Do [ 10001 ]	Invalid [ 11005 ]

John Swinbank made changes - 05/Sep/16 5:35 AM

Team

Data Access and Database [ 10204 ]

Hide

Permalink

John Swinbank added a comment - 01/Aug/17 7:33 PM

For what it's worth, Michael Wood-Vasey is not crazy. Or, at least, if he is, this ticket is not evidence for it. I just hit the same:

tests/butlerPickle.py

/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to O

SError:Could not find unix home directory to search for astropy config dir on None

  warn(ConfigurationMissingWarning(msg))

.E..

======================================================================

ERROR: testPickle (__main__.ButlerPickleTestCase)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "tests/butlerPickle.py", line 67, in testPickle

    butler = pickle.loads(pickledButler)

  File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1388, in loads

    return Unpickler(file).load()

  File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 864, in load

    dispatch[key](self)

  File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1139, in load_reduce

    value = func(*args)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 943, in _unreduce

    butler = Butler(**initArgs)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 304, in __init__

    self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 369, in _addRepo

    cfg = Storage.getRepositoryCfg(args.cfgRoot)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg

    ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 79, in getRepositoryCfg

    repositoryCfg = PosixStorage._getRepositoryCfg(uri)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 72, in _getRepositoryCfg

    repositoryCfg = yaml.load(f)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/__init__.py", line 71, in load

    return loader.get_single_data()

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 39, in get_single_data

    return self.construct_document(node)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 43, in construct_document

    data = self.construct_object(node)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 88, in construct_object

    data = constructor(self, node)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 63, in v1Constructor

    d = loader.construct_mapping(node)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 208, in construct_mapping

    return BaseConstructor.construct_mapping(self, node, deep=deep)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 133, in construct_mapping

    value = self.construct_object(value_node, deep=deep)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 90, in construct_object

    data = constructor(self, tag_suffix, node)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 610, in construct_python_object_new

    return self.construct_python_object_apply(suffix, node, newobj=True)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply

    instance = self.make_python_instance(suffix, node, args, kwds, newobj)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 540, in make_python_instance

    cls = self.find_python_name(suffix, node.start_mark)

  File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 513, in find_python_name

    module.__name__), mark)

ConstructorError: while constructing a Python object

cannot find 'ImgMapper' in the module '__main__'

  in "./repositoryCfg.yaml", line 3, column 10

A couple of things to note:

This happened when I was trying to build old weeklies (I noticed it with w_2017_5 and w_2017_7). I've not reproduced it with the latest weekly (but I haven't seriously tried yet).
I've only noticed it when building with a lot of jobs (scons -j 28). That suggests that it might be a race. On the other hand, it might just be that I've not tried often enough with a single job (-j 1) because that takes much longer.

I'll poke at this a bit harder in the morning.

Show

John Swinbank added a comment - 01/Aug/17 7:33 PM For what it's worth, Michael Wood-Vasey is not crazy. Or, at least, if he is, this ticket is not evidence for it. I just hit the same: tests/butlerPickle.py /tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to O SError:Could not find unix home directory to search for astropy config dir on None warn(ConfigurationMissingWarning(msg)) .E.. ====================================================================== ERROR: testPickle (__main__.ButlerPickleTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/butlerPickle.py", line 67, in testPickle butler = pickle.loads(pickledButler) File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1388, in loads return Unpickler(file).load() File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 864, in load dispatch[key](self) File "/tigress/HSC/LSST/stack_perseus_20170801/python/miniconda2-4.2.12/lib/python2.7/pickle.py", line 1139, in load_reduce value = func(*args) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 943, in _unreduce butler = Butler(**initArgs) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 304, in __init__ self._addRepo(args, inout='out', defaultMapper=defaultMapper, butlerIOParents=butlerIOParents) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/butler.py", line 369, in _addRepo cfg = Storage.getRepositoryCfg(args.cfgRoot) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/storage.py", line 75, in getRepositoryCfg ret = Storage.storages[parseRes.scheme].getRepositoryCfg(uri) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 79, in getRepositoryCfg repositoryCfg = PosixStorage._getRepositoryCfg(uri) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 72, in _getRepositoryCfg repositoryCfg = yaml.load(f) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/__init__.py", line 71, in load return loader.get_single_data() File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 39, in get_single_data return self.construct_document(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 43, in construct_document data = self.construct_object(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 88, in construct_object data = constructor(self, node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/EupsBuildDir/Linux64/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 63, in v1Constructor d = loader.construct_mapping(node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 208, in construct_mapping return BaseConstructor.construct_mapping(self, node, deep=deep) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 133, in construct_mapping value = self.construct_object(value_node, deep=deep) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 90, in construct_object data = constructor(self, tag_suffix, node) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 610, in construct_python_object_new return self.construct_python_object_apply(suffix, node, newobj=True) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 599, in construct_python_object_apply instance = self.make_python_instance(suffix, node, args, kwds, newobj) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 540, in make_python_instance cls = self.find_python_name(suffix, node.start_mark) File "/tigress/HSC/LSST/stack_perseus_20170801/stack/miniconda2-4.2.12-7c8e67/Linux64/pyyaml/3.11.lsst1+3/lib/python/yaml/constructor.py", line 513, in find_python_name module.__name__), mark) ConstructorError: while constructing a Python object cannot find 'ImgMapper' in the module '__main__' in "./repositoryCfg.yaml", line 3, column 10 A couple of things to note: This happened when I was trying to build old weeklies (I noticed it with w_2017_5 and w_2017_7 ). I've not reproduced it with the latest weekly (but I haven't seriously tried yet). I've only noticed it when building with a lot of jobs ( scons -j 28 ). That suggests that it might be a race. On the other hand, it might just be that I've not tried often enough with a single job ( -j 1 ) because that takes much longer. I'll poke at this a bit harder in the morning.

John Swinbank made changes - 01/Aug/17 7:33 PM

Assignee

Michael Wood-Vasey [ wmwood-vasey ]

John Swinbank [ swinbank ]

John Swinbank made changes - 01/Aug/17 7:33 PM

Resolution	Done [ 10000 ]
Status	Invalid [ 11005 ]	In Progress [ 3 ]

Hide

Permalink

John Swinbank added a comment - 01/Aug/17 7:34 PM

(Reopening the ticket at least until I can convince myself that there's nothing here worth worrying about.)

Show

John Swinbank added a comment - 01/Aug/17 7:34 PM (Reopening the ticket at least until I can convince myself that there's nothing here worth worrying about.)

Hide

Permalink

John Swinbank added a comment - 01/Aug/17 7:44 PM - edited

Ok, found the problem.

ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests. When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper, which is only defined within butlerSubset.py.

If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time. It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase, doesn't know anything about ImgMapper, and so falls over.

Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.

Show

John Swinbank added a comment - 01/Aug/17 7:44 PM - edited Ok, found the problem. ButlerSubsetTestCase writes a repositoryCfg.yaml into daf_persistence/tests . When the test case is finished, it removes it again (in the tearDown method). That repositoryCfg.yaml contains a reference to ImgMapper , which is only defined within butlerSubset.py . If we're multi-processing, we can run ButlerPickleTestCase from butlerPickle.py at the same time . It attempts to open daf_persistence.tests as a Butler repository, finds the repositoryCfg.yaml from ButlerSubsetTestCase , doesn't know anything about ImgMapper , and so falls over. Still need to check if this problem exists in current master or if it has been solved sometime in the last ~6 months.

Hide

Permalink

John Swinbank added a comment - 05/Aug/17 3:55 PM

See also the daf_persistence error reported on ~~DM-11488~~, which is almost the same as the problem here. When fixing this, check if they could be related.

Show

John Swinbank added a comment - 05/Aug/17 3:55 PM See also the daf_persistence error reported on DM-11488 , which is almost the same as the problem here. When fixing this, check if they could be related.

John Swinbank made changes - 05/Aug/17 3:55 PM

Link

This issue relates to ~~DM-11488~~ [ ~~DM-11488~~ ]

Hide

Permalink

John Swinbank added a comment - 05/Aug/17 7:56 PM - edited

Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address ~~DM-11488~~.

Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/

Show

John Swinbank added a comment - 05/Aug/17 7:56 PM - edited Hi Nate Pease — could you take a look at this please? I think the PR on daf_persistence will both avoid the issue reported in this ticket and also address DM-11488 . Jenkins: https://ci.lsst.codes/job/lsst_distrib/1164/

John Swinbank made changes - 05/Aug/17 7:56 PM

Reviewers		Nate Pease [ npease ]
Status	In Progress [ 3 ]	In Review [ 10004 ]

John Swinbank made changes - 07/Aug/17 8:45 AM

Epic Link

~~DM-10383~~ [ 32102 ]

John Swinbank made changes - 07/Aug/17 8:45 AM

Sprint		DRP F17-3 [ 622 ]
Story Points		1
Team	Data Access and Database [ 10204 ]	Data Release Production [ 10301 ]

Hide

Permalink

Nate Pease added a comment - 07/Aug/17 12:07 PM

looks good.

Show

Nate Pease added a comment - 07/Aug/17 12:07 PM looks good.

Nate Pease made changes - 07/Aug/17 12:07 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Hide

Permalink

John Swinbank added a comment - 08/Aug/17 6:38 AM

Thanks! Merged & pushed.

Show

John Swinbank added a comment - 08/Aug/17 6:38 AM Thanks! Merged & pushed.

John Swinbank made changes - 08/Aug/17 6:38 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

People

Assignee:

John Swinbank

Reporter:

Michael Wood-Vasey

Reviewers:

Nate Pease

Watchers:

John Swinbank, Michael Wood-Vasey, Nate Pease, Tim Jenness

Votes:

0 Vote for this issue

Watchers:

4 Start watching this issue

Dates

Created:

31/Aug/16 6:50 AM

Updated:

08/Aug/17 6:38 AM

Resolved:

08/Aug/17 6:38 AM

Summary Panel