Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14914

ccutils from ci-scripts appears to be unsetting error exit

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      I've seen build failures a few times over the last several month that should have died earlier then then did. This was observed again last night with a tarball build and my theory is that ccutils from ci-scripts is unsetting errexit as an accidental side effect (the tarball build would have failed either way from what appears to be a broken eups install).

      From https://ci.lsst.codes/blue/organizations/jenkins/release%2Ftarball/detail/tarball/2958/pipeline :

      + . ./loadLSST.bash
       
      ++ export PATH=/build/python/miniconda3-4.3.21/bin:/opt/rh/devtoolset-6/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
       
      ++ PATH=/build/python/miniconda3-4.3.21/bin:/opt/rh/devtoolset-6/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
       
      ++++ dirname ./loadLSST.bash
       
      +++ cd .
       
      +++ pwd
       
      ++ LSST_HOME=/build
       
      ++ EUPS_DIR=/build/eups/2.1.4
       
      ++ source /build/eups/2.1.4/bin/setups.sh
       
      +++ export EUPS_SHELL=sh
       
      +++ EUPS_SHELL=sh
       
      +++ export EUPS_DIR=/build/eups/2.1.4
       
      +++ EUPS_DIR=/build/eups/2.1.4
       
      ++++ echo /build/eups/2.1.4
       
      ++++ sed -e 's/ /-+-/g'
       
      +++ eupslocalpath=/build/eups/2.1.4
       
      ++++ /build/python/miniconda3-4.3.21/bin/python -E -c '
       
      from __future__ import print_function
       
      import sys
       
      pp = []
       
      for d in sys.argv[1].split(":"):
       
          if d and d not in pp:
       
              pp += [d]
       
      if not sys.argv[2] in pp:
       
          pp = [sys.argv[2]] + pp
       
      print(":".join(pp))' '' /build/stack/miniconda3-4.3.21-10a4fa6
       
      +++ export EUPS_PATH=/build/stack/miniconda3-4.3.21-10a4fa6
       
      +++ EUPS_PATH=/build/stack/miniconda3-4.3.21-10a4fa6
       
      +++ _eups_path=/build/stack/miniconda3-4.3.21-10a4fa6
       
      ++++ /build/eups/2.1.4/bin/eups_setup DYLD_LIBRARY_PATH= eups -r /build/eups/2.1.4
       
      setup: Unable to take shared lock on /build/stack/miniconda3-4.3.21-10a4fa6: an exclusive lock is held by [user=jenkins-slave, pid=590]
       
      +++ eval false
       
      ++++ false
       
      +++ export 'SETUP_EUPS=eups LOCAL:/build/eups/2.1.4 -f (none) -Z (none)'
       
      +++ SETUP_EUPS='eups LOCAL:/build/eups/2.1.4 -f (none) -Z (none)'
       
      +++ export EUPS_PATH=/build/stack/miniconda3-4.3.21-10a4fa6
       
      +++ EUPS_PATH=/build/stack/miniconda3-4.3.21-10a4fa6
       
      +++ unset eupslocalpath _eups_path
       
      +++ '[' X '!=' X -a -f /build/eups/2.1.4/etc/bash_completion.d/eups ']'
       
      ++ export -f setup
       
      ./loadLSST.bash: line 11: export: setup: not a function
       
      ++ export -f unsetup
       
      ./loadLSST.bash: line 12: export: unsetup: not a function
       
      ++ export 'EUPS_PKGROOT=https://****/stack/redhat/el6/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://****/stack/src'
       
      ++ EUPS_PKGROOT='https://****/stack/redhat/el6/devtoolset-6/miniconda3-4.3.21-10a4fa6|https://****/stack/src'
       
      + for prod in lsst_distrib
       
      + eups distrib install lsst_distrib -t d_2018_06_27 -vvv
       
      /build/scripts/run.sh: line 36: eups: command not found
       
      + export EUPS_PKGROOT=/distrib
       
      + EUPS_PKGROOT=/distrib
       
      + [[ -e /distrib ]]
       
      + rm -f '/distrib/*.list'
       
      + for prod in lsst_distrib
       
      + eups distrib create --server-dir /distrib -d tarball lsst_distrib -t d_2018_06_27 -vvv
       
      /build/scripts/run.sh: line 50: eups: command not found
       
      + eups distrib declare --server-dir /distrib -t d_2018_06_27 -vvv
       
      /build/scripts/run.sh: line 52: eups: command not found
       
      script returned exit code 127
      

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment - - edited

          Similar problems were found in lsstsw and newinstall.sh. Additionally, pipefile and nounset were enabled where it was reasonably low risk to do so. errtrace was enabled in newinstall.sh as trap is in use. The changes to ci-scripts/lsstsw were tested in a jenkins dev env and merged. newinstall.sh passed travis, which does run the scripts, but needs review prior to being merged.

          Show
          jhoblitt Joshua Hoblitt added a comment - - edited Similar problems were found in lsstsw and newinstall.sh . Additionally, pipefile and nounset were enabled where it was reasonably low risk to do so. errtrace was enabled in newinstall.sh as trap is in use. The changes to ci-scripts / lsstsw were tested in a jenkins dev env and merged. newinstall.sh passed travis, which does run the scripts, but needs review prior to being merged.
          Hide
          tjenness Tim Jenness added a comment -

          I think all the PRs have been reviewed.

          Show
          tjenness Tim Jenness added a comment - I think all the PRs have been reviewed.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Tim Jenness thank you.

          All PRs have been merged.

          Show
          jhoblitt Joshua Hoblitt added a comment - Tim Jenness thank you. All PRs have been merged.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Tim Jenness
              Watchers:
              Gabriele Comoretto, Joshua Hoblitt, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel