Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Continuous Integration, Developer Infrastructure
-
Labels:None
-
Story Points:0.5
-
Epic Link:
-
Team:SQuaRE
Description
The w_2017_31 weekly release failed building master due to what appears to be a repeatable git-lfs error, possibly triggered by DM-11095.
testdata_jointcal: Traceback (most recent call last):
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/bin/lsst-build", line 51, in <module> |
args.func(args)
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 762, in run |
manifest = p.construct(args.products)
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 725, in construct |
self._add_product_tree(products, name)
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 713, in _add_product_tree |
dependencies.append(self._add_product_tree(products, dprod.name))
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 713, in _add_product_tree |
dependencies.append(self._add_product_tree(products, dprod.name))
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 696, in _add_product_tree |
ref, sha1 = self.product_fetcher.fetch(product_name)
|
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 333, in fetch |
git.checkout("--force", ref) |
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/git.py", line 49, in checkout |
return self('checkout', *args, **kwargs) |
File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/git.py", line 44, in __call__ |
raise GitError(retcode, cmd, stdout, stderr)
|
lsst.ci.git.GitError: Command '['git', 'checkout', '--force', 'master']' returned non-zero exit status 128. |
stdout:
|
|
|
stderr:
|
Downloading cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits (75.94 KB) |
Error downloading object: cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits (5071e4548d33b407c0438b3b7fcbf4fab390674f30ccfd5f7492e21b689e6653) |
|
|
Errors logged to /home/lsstsw/jenkins/release/lsstsw/build/testdata_jointcal/.git/lfs/objects/logs/20170805T020850.70624876.log |
Use `git lfs logs last` to view the log.
|
error: external filter git-lfs smudge %f failed 2 |
error: external filter git-lfs smudge %f failed
|
fatal: cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits: smudge filter lfs failed |
Attachments
Issue Links
Activity
On to problem #2, A test failure in daf_persistence/:
|
|
/home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None |
warn(ConfigurationMissingWarning(msg))
|
.E.....
|
======================================================================
|
ERROR: testDoubleIteration (__main__.ButlerSubsetTestCase)
|
----------------------------------------------------------------------
|
Traceback (most recent call last):
|
File "tests/test_butlerSubset.py", line 190, in testDoubleIteration |
'mapper': ImgMapper}) |
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 527, in __init__ |
self._getCfgs(repoDataList)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 790, in _getCfgs |
cfg, isOldButlerRepository = self._getRepositoryCfg(repoData.repoArgs)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 753, in _getRepositoryCfg |
cfg = self.storage.getRepositoryCfg(repositoryArgs.cfgRoot)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py", line 88, in getRepositoryCfg |
cfg = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 152, in getRepositoryCfg |
datasetType=None))
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 114, in read |
repositoryCfg = RepositoryCfgPosixFormatter._read(f, loc)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 101, in _read |
repositoryCfg = yaml.load(fileObject)
|
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/__init__.py", line 71, in load |
return loader.get_single_data() |
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 39, in get_single_data |
return self.construct_document(node) |
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 43, in construct_document |
data = self.construct_object(node)
|
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 88, in construct_object |
data = constructor(self, node)
|
File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 82, in v1Constructor |
d = loader.construct_mapping(node)
|
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 208, in construct_mapping |
return BaseConstructor.construct_mapping(self, node, deep=deep) |
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 133, in construct_mapping |
value = self.construct_object(value_node, deep=deep)
|
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 90, in construct_object |
data = constructor(self, tag_suffix, node)
|
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 522, in construct_python_name |
return self.find_python_name(suffix, node.start_mark) |
File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 513, in find_python_name |
module.__name__), mark)
|
ConstructorError: while constructing a Python object |
cannot find 'MinMapper' in the module '__main__' |
in "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/repositoryCfg.yaml", line 2, column 10 |
|
|
----------------------------------------------------------------------
|
Ran 7 tests in 0.132s |
|
|
FAILED (errors=1) |
Looks like a non-repeatable test failure: daf_persistence: 13.0-29-g32a3f02 ...........ok (54.9 sec).
The daf_persistence error looks suspiciously similar to (albeit not quite the same as) DM-7457, which I stumbled over the other day. When I address that ticket, I'll see if I can do this one at the same time.
The qserv container build failed as per usual (this is no longer fatal). However, weekly problem #3 is an exception from the pipeline script itself after building the stacktest base image for the jupyterlabdemo container images.
Undefined symbol ‘choiceParam’
|
The jupyterlabdemo container images were manually triggered and built without error. w_2017_31 is officially done.
The issue with fetching testadata_jointcall was two fold. The git-lfs version was 1.2.0 (should be 1.5.5) as lsstsw/bin/deploy does not update the installed version and the lsst_build checkout was not current. The trigger was a change in testdata_jointcal, which I suspect was the first change to an lfs repo in several months.