Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11488

weekly release w_2017_31 failed

    Details

      Description

      The w_2017_31 weekly release failed building master due to what appears to be a repeatable git-lfs error, possibly triggered by DM-11095.

         testdata_jointcal: Traceback (most recent call last):
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/bin/lsst-build", line 51, in <module>
          args.func(args)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 762, in run
          manifest = p.construct(args.products)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 725, in construct
          self._add_product_tree(products, name)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 713, in _add_product_tree
          dependencies.append(self._add_product_tree(products, dprod.name))
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 713, in _add_product_tree
          dependencies.append(self._add_product_tree(products, dprod.name))
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 696, in _add_product_tree
          ref, sha1 = self.product_fetcher.fetch(product_name)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/prepare.py", line 333, in fetch
          git.checkout("--force", ref)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/git.py", line 49, in checkout
          return self('checkout', *args, **kwargs)
        File "/home/lsstsw/jenkins/release/lsstsw/lsst_build/python/lsst/ci/git.py", line 44, in __call__
          raise GitError(retcode, cmd, stdout, stderr)
      lsst.ci.git.GitError: Command '['git', 'checkout', '--force', 'master']' returned non-zero exit status 128.
      stdout:
       
      stderr:
      Downloading cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits (75.94 KB)
      Error downloading object: cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits (5071e4548d33b407c0438b3b7fcbf4fab390674f30ccfd5f7492e21b689e6653)
       
      Errors logged to /home/lsstsw/jenkins/release/lsstsw/build/testdata_jointcal/.git/lfs/objects/logs/20170805T020850.70624876.log
      Use `git lfs logs last` to view the log.
      error: external filter git-lfs smudge %f failed 2
      error: external filter git-lfs smudge %f failed
      fatal: cfht_minimal/src/06AL01/D3/2006-05-20/r/SRC-849375-12.fits: smudge filter lfs failed
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The issue with fetching testadata_jointcall was two fold. The git-lfs version was 1.2.0 (should be 1.5.5) as lsstsw/bin/deploy does not update the installed version and the lsst_build checkout was not current. The trigger was a change in testdata_jointcal, which I suspect was the first change to an lfs repo in several months.

            Show
            jhoblitt Joshua Hoblitt added a comment - The issue with fetching testadata_jointcall was two fold. The git-lfs version was 1.2.0 (should be 1.5.5) as lsstsw/bin/deploy does not update the installed version and the lsst_build checkout was not current. The trigger was a change in testdata_jointcal , which I suspect was the first change to an lfs repo in several months.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            On to problem #2, A test failure in daf_persistence/:

             
            /home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2.7/site-packages/astropy/config/configuration.py:541: ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None
              warn(ConfigurationMissingWarning(msg))
            .E.....
            ======================================================================
            ERROR: testDoubleIteration (__main__.ButlerSubsetTestCase)
            ----------------------------------------------------------------------
            Traceback (most recent call last):
              File "tests/test_butlerSubset.py", line 190, in testDoubleIteration
                'mapper': ImgMapper})
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 527, in __init__
                self._getCfgs(repoDataList)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 790, in _getCfgs
                cfg, isOldButlerRepository = self._getRepositoryCfg(repoData.repoArgs)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py", line 753, in _getRepositoryCfg
                cfg = self.storage.getRepositoryCfg(repositoryArgs.cfgRoot)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py", line 88, in getRepositoryCfg
                cfg = Storage.storages[parseRes.scheme].getRepositoryCfg(uri)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py", line 152, in getRepositoryCfg
                datasetType=None))
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 114, in read
                repositoryCfg = RepositoryCfgPosixFormatter._read(f, loc)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py", line 101, in _read
                repositoryCfg = yaml.load(fileObject)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/__init__.py", line 71, in load
                return loader.get_single_data()
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 39, in get_single_data
                return self.construct_document(node)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 43, in construct_document
                data = self.construct_object(node)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 88, in construct_object
                data = constructor(self, node)
              File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py", line 82, in v1Constructor
                d = loader.construct_mapping(node)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 208, in construct_mapping
                return BaseConstructor.construct_mapping(self, node, deep=deep)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 133, in construct_mapping
                value = self.construct_object(value_node, deep=deep)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 90, in construct_object
                data = constructor(self, tag_suffix, node)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 522, in construct_python_name
                return self.find_python_name(suffix, node.start_mark)
              File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py", line 513, in find_python_name
                module.__name__), mark)
            ConstructorError: while constructing a Python object
            cannot find 'MinMapper' in the module '__main__'
              in "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/repositoryCfg.yaml", line 2, column 10
             
            ----------------------------------------------------------------------
            Ran 7 tests in 0.132s
             
            FAILED (errors=1)
            

            Show
            jhoblitt Joshua Hoblitt added a comment - On to problem #2, A test failure in daf_persistence/ :   /home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2. 7 /site-packages/astropy/config/configuration.py: 541 : ConfigurationMissingWarning: Configuration defaults will be used due to OSError:Could not find unix home directory to search for astropy config dir on None warn(ConfigurationMissingWarning(msg)) .E..... ====================================================================== ERROR: testDoubleIteration (__main__.ButlerSubsetTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/test_butlerSubset.py" , line 190 , in testDoubleIteration 'mapper' : ImgMapper}) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py" , line 527 , in __init__ self._getCfgs(repoDataList) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py" , line 790 , in _getCfgs cfg, isOldButlerRepository = self._getRepositoryCfg(repoData.repoArgs) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/butler.py" , line 753 , in _getRepositoryCfg cfg = self.storage.getRepositoryCfg(repositoryArgs.cfgRoot) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/storage.py" , line 88 , in getRepositoryCfg cfg = Storage.storages[parseRes.scheme].getRepositoryCfg(uri) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/posixStorage.py" , line 152 , in getRepositoryCfg datasetType=None)) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py" , line 114 , in read repositoryCfg = RepositoryCfgPosixFormatter._read(f, loc) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/fmtPosixRepositoryCfg.py" , line 101 , in _read repositoryCfg = yaml.load(fileObject) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/__init__.py" , line 71 , in load return loader.get_single_data() File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 39 , in get_single_data return self.construct_document(node) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 43 , in construct_document data = self.construct_object(node) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 88 , in construct_object data = constructor(self, node) File "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/python/lsst/daf/persistence/repositoryCfg.py" , line 82 , in v1Constructor d = loader.construct_mapping(node) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 208 , in construct_mapping return BaseConstructor.construct_mapping(self, node, deep=deep) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 133 , in construct_mapping value = self.construct_object(value_node, deep=deep) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 90 , in construct_object data = constructor(self, tag_suffix, node) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 522 , in construct_python_name return self.find_python_name(suffix, node.start_mark) File "/home/lsstsw/jenkins/release/lsstsw/stack/Linux64/pyyaml/3.11.lsst1-2-g6ae2b7a/lib/python/yaml/constructor.py" , line 513 , in find_python_name module.__name__), mark) ConstructorError: while constructing a Python object cannot find 'MinMapper' in the module '__main__' in "/home/lsstsw/jenkins/release/lsstsw/build/daf_persistence/repositoryCfg.yaml" , line 2 , column 10   ---------------------------------------------------------------------- Ran 7 tests in 0 .132s   FAILED (errors= 1 )
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Looks like a non-repeatable test failure: daf_persistence: 13.0-29-g32a3f02 ...........ok (54.9 sec).

            Show
            jhoblitt Joshua Hoblitt added a comment - Looks like a non-repeatable test failure: daf_persistence: 13.0-29-g32a3f02 ...........ok (54.9 sec).
            Hide
            swinbank John Swinbank added a comment -

            The daf_persistence error looks suspiciously similar to (albeit not quite the same as) DM-7457, which I stumbled over the other day. When I address that ticket, I'll see if I can do this one at the same time.

            Show
            swinbank John Swinbank added a comment - The daf_persistence error looks suspiciously similar to (albeit not quite the same as) DM-7457 , which I stumbled over the other day. When I address that ticket, I'll see if I can do this one at the same time.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The qserv container build failed as per usual (this is no longer fatal). However, weekly problem #3 is an exception from the pipeline script itself after building the stacktest base image for the jupyterlabdemo container images.

             Undefined symbol ‘choiceParam’
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The qserv container build failed as per usual (this is no longer fatal). However, weekly problem #3 is an exception from the pipeline script itself after building the stacktest base image for the jupyterlabdemo container images. Undefined symbol ‘choiceParam’
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The jupyterlabdemo container images were manually triggered and built without error. w_2017_31 is officially done.

            Show
            jhoblitt Joshua Hoblitt added a comment - The jupyterlabdemo container images were manually triggered and built without error. w_2017_31 is officially done.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Frossie Economou, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel