Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11791

weekly release w_2017_35 failed

    Details

      Description

      _35 failed in the canonical build on lsst-dev failed due to a full disk or quota limit.

                       afw: 13.0-94-ga1c4440+6 ....................................................................................:::::  [2017-09-02T07:30:59.486911Z] /home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
      :::::  [2017-09-02T07:30:59.486972Z] /home/ilan/minonda/conda-bld/work/Python-2.7.12/./Modules/posixmodule.c:7578: warning: the use of `tempnam' is dangerous, better use `mkstemp'
      :::::  [2017-09-02T07:30:59.874220Z] /home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tmpnam':
      :::::  [2017-09-02T07:30:59.874323Z] /home/ilan/minonda/conda-bld/work/Python-2.7.12/./Modules/posixmodule.c:7631: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
      :::::  [2017-09-02T07:30:59.874362Z] /home/lsstsw/jenkins/release/lsstsw/miniconda/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
      :::::  [2017-09-02T07:30:59.874401Z] /home/ilan/minonda/conda-bld/work/Python-2.7.12/./Modules/posixmodule.c:7578: warning: the use of `tempnam' is dangerous, better use `mkstemp'
      :::::  [2017-09-02T07:31:10.858727Z] /tmp/cck8pdsX.s: Assembler messages:
      :::::  [2017-09-02T07:31:10.858909Z] /tmp/cck8pdsX.s: Fatal error: can't close tests/ellipse.o: No space left on device
      :::::  [2017-09-02T07:31:10.935937Z] scons: *** [tests/ellipse.o] Error 1
      :::::  [2017-09-02T07:31:10.941966Z] scons: building terminated because of errors.
      ERROR (206 sec).
      *** error building product afw.
      *** exit code = 2
      *** log is in /home/lsstsw/jenkins/release/lsstsw/build/afw/_build.log
      *** last few lines:
      

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            The ~lsstsw home dir on lsst-dev was at ~1TiB and presumably was at the quota limit. I was able to free up ~350GiB by removing an old copy of installed eups products. This got the release past the build/publish steps but failed tagging the git repos due to the eups distrib tag and versiondb being out of sync. Rerunning the pipeline got past this step. My assumption is this was due to s3's eventual consistency. Newly created objects tend to be visible immediately but it can occasionally take minutes for the new version of a replaced object to become visible. This can likely be worked around in the future by adding a sleep step into the release pipeline.

            The eups distrib, git tag, and tarballs have been published but the pipeline failed building the jupyterlab demo.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited The ~lsstsw home dir on lsst-dev was at ~1TiB and presumably was at the quota limit. I was able to free up ~350GiB by removing an old copy of installed eups products. This got the release past the build/publish steps but failed tagging the git repos due to the eups distrib tag and versiondb being out of sync. Rerunning the pipeline got past this step. My assumption is this was due to s3's eventual consistency. Newly created objects tend to be visible immediately but it can occasionally take minutes for the new version of a replaced object to become visible. This can likely be worked around in the future by adding a sleep step into the release pipeline. The eups distrib, git tag, and tarballs have been published but the pipeline failed building the jupyterlab demo.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Frossie Economou, John Swinbank, Joshua Hoblitt, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: