Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10140

lsstsw deploy failing with conda lock file errors

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      This failure mode has been observed several times now under jenkins when starting from a clean workspace.

      + ./buildbot-scripts/jenkins_wrapper.sh
      if grep -q -i "CentOS release 6" /etc/redhat-release 2>/dev/null; then
          # shellcheck disable=SC1091
        . /opt/rh/devtoolset-3/enable
      fi
      set +o verbose
      add_anaconda_token: True
      add_pip_as_python_dependency: True
      allow_softlinks: True
      always_copy: False
      always_yes: False
      auto_update_conda: True
      binstar_upload: None
      changeps1: True
      channel_alias: https://conda.anaconda.org
      channel_priority: True
      channels:
        - http://conda-mirror.lsst.codes/pkgs/free
      client_tls_cert: 
      client_tls_cert_key: 
      create_default_packages: []
      debug: False
      default_channels:
        - https://repo.continuum.io/pkgs/free
        - https://repo.continuum.io/pkgs/pro
      disallow: []
      envs_dirs:
        - /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda/envs
      json: False
      offline: False
      proxy_servers: {}
      quiet: False
      shortcuts: True
      show_channel_urls: None
      ssl_verify: True
      track_features: []
      update_dependencies: True
      use_pip: True
      verbosity: 0
      Fetching package metadata .....
      .Solving package specifications: ..........
       
      Package plan for installation in environment /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda:
       
      The following packages will be downloaded:
       
          package                    |            build
          ---------------------------|-----------------
          mkl-11.3.3                 |                0       101.4 MB
          libpng-1.6.22              |                0         225 KB
          cython-0.25.2              |           py27_0         2.6 MB
          freetype-2.5.5             |                1         734 KB
          future-0.16.0              |           py27_0         618 KB
          numpy-1.11.2               |           py27_0         3.1 MB
          pyparsing-2.1.4            |           py27_0          70 KB
          pytz-2016.10               |           py27_0         181 KB
          pyyaml-3.12                |           py27_0         153 KB
          requests-2.12.4            |           py27_0         752 KB
          sip-4.18                   |           py27_0         245 KB
          six-1.10.0                 |           py27_0          16 KB
          sqlalchemy-1.1.4           |           py27_0         1.3 MB
          astropy-1.2.1              |      np111py27_0         6.1 MB
          bottleneck-1.2.0           |      np111py27_0         106 KB
          conda-4.2.13               |           py27_0         375 KB
          cycler-0.10.0              |           py27_0          11 KB
          numexpr-2.6.0              |      np111py27_0         111 KB
          python-dateutil-2.6.0      |           py27_0         232 KB
          qt-4.8.7                   |                4        37.1 MB
          scipy-0.18.1               |      np111py27_0        13.4 MB
          pandas-0.19.1              |      np111py27_0         7.8 MB
          pyqt-4.11.4                |           py27_4         4.0 MB
          matplotlib-1.5.1           |      np111py27_0         6.2 MB
          ------------------------------------------------------------
                                                 Total:       186.8 MB
       
      The following NEW packages will be INSTALLED:
       
          astropy:         1.2.1-np111py27_0 
          bottleneck:      1.2.0-np111py27_0 
          cycler:          0.10.0-py27_0     
          cython:          0.25.2-py27_0     
          freetype:        2.5.5-1           
          future:          0.16.0-py27_0     
          libpng:          1.6.22-0          
          matplotlib:      1.5.1-np111py27_0 
          mkl:             11.3.3-0          
          numexpr:         2.6.0-np111py27_0 
          numpy:           1.11.2-py27_0     
          pandas:          0.19.1-np111py27_0
          pyparsing:       2.1.4-py27_0      
          pyqt:            4.11.4-py27_4     
          python-dateutil: 2.6.0-py27_0      
          pytz:            2016.10-py27_0    
          pyyaml:          3.12-py27_0       
          qt:              4.8.7-4           
          scipy:           0.18.1-np111py27_0
          sip:             4.18-py27_0       
          six:             1.10.0-py27_0     
          sqlalchemy:      1.1.4-py27_0      
       
      The following packages will be UPDATED:
       
          conda:           4.2.12-py27_0      --> 4.2.13-py27_0
          requests:        2.11.1-py27_0      --> 2.12.4-py27_0
       
      Exceeded max retries, giving up
       
      LockError: Lock error: Already locked: Lock error: 
      LOCKERROR: It looks like conda is already doing something.
      The lock [u'/Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda/pkgs/mkl-11.3.3-0.tar.bz2.pid86953.conda_lock'] was found. Wait for it to finish before continuing.
      If you are sure that conda is not running, remove it and try again.
      You can also use: $ conda clean --lock
       
       
       
      Build step 'Execute shell' marked build as failure
      

        Attachments

          Activity

          jhoblitt Joshua Hoblitt created issue -
          jhoblitt Joshua Hoblitt made changes -
          Field Original Value New Value
          Epic Link DM-10099 [ 31683 ]
          jhoblitt Joshua Hoblitt made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          jhoblitt Joshua Hoblitt made changes -
          Reviewers Adam Thornton [ athornton ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The fix has been confirmed as working in production on the sierra-2 node, which was previously failing.

          + ./buildbot-scripts/jenkins_wrapper.sh
          if grep -q -i "CentOS release 6" /etc/redhat-release 2>/dev/null; then
              # shellcheck disable=SC1091
            . /opt/rh/devtoolset-3/enable
          fi
          set +o verbose
          add_anaconda_token: True
          add_pip_as_python_dependency: True
          allow_softlinks: True
          always_copy: False
          always_yes: False
          auto_update_conda: True
          binstar_upload: None
          changeps1: True
          channel_alias: https://conda.anaconda.org
          channel_priority: True
          channels:
            - http://conda-mirror.lsst.codes/pkgs/free
          client_tls_cert: 
          client_tls_cert_key: 
          create_default_packages: []
          debug: False
          default_channels:
            - https://repo.continuum.io/pkgs/free
            - https://repo.continuum.io/pkgs/pro
          disallow: []
          envs_dirs:
            - /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda/envs
          json: False
          offline: False
          proxy_servers: {}
          quiet: False
          shortcuts: True
          show_channel_urls: None
          ssl_verify: True
          track_features: []
          update_dependencies: True
          use_pip: True
          verbosity: 0
          Fetching package metadata .....
          .Solving package specifications: ..........
          Package plan for installation in environment /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda:
           
          The following packages will be downloaded:
           
              package                    |            build
              ---------------------------|-----------------
              mkl-11.3.3                 |                0       101.4 MB
              libpng-1.6.22              |                0         225 KB
              cython-0.25.2              |           py27_0         2.6 MB
              freetype-2.5.5             |                1         734 KB
              future-0.16.0              |           py27_0         618 KB
              numpy-1.11.2               |           py27_0         3.1 MB
              pyparsing-2.1.4            |           py27_0          70 KB
              pytz-2016.10               |           py27_0         181 KB
              pyyaml-3.12                |           py27_0         153 KB
              requests-2.12.4            |           py27_0         752 KB
              sip-4.18                   |           py27_0         245 KB
              six-1.10.0                 |           py27_0          16 KB
              sqlalchemy-1.1.4           |           py27_0         1.3 MB
              astropy-1.2.1              |      np111py27_0         6.1 MB
              bottleneck-1.2.0           |      np111py27_0         106 KB
              conda-4.2.13               |           py27_0         375 KB
              cycler-0.10.0              |           py27_0          11 KB
              numexpr-2.6.0              |      np111py27_0         111 KB
              python-dateutil-2.6.0      |           py27_0         232 KB
              qt-4.8.7                   |                4        37.1 MB
              scipy-0.18.1               |      np111py27_0        13.4 MB
              pandas-0.19.1              |      np111py27_0         7.8 MB
              pyqt-4.11.4                |           py27_4         4.0 MB
              matplotlib-1.5.1           |      np111py27_0         6.2 MB
              ------------------------------------------------------------
                                                     Total:       186.8 MB
           
          The following NEW packages will be INSTALLED:
           
              astropy:         1.2.1-np111py27_0 
              bottleneck:      1.2.0-np111py27_0 
              cycler:          0.10.0-py27_0     
              cython:          0.25.2-py27_0     
              freetype:        2.5.5-1           
              future:          0.16.0-py27_0     
              libpng:          1.6.22-0          
              matplotlib:      1.5.1-np111py27_0 
              mkl:             11.3.3-0          
              numexpr:         2.6.0-np111py27_0 
              numpy:           1.11.2-py27_0     
              pandas:          0.19.1-np111py27_0
              pyparsing:       2.1.4-py27_0      
              pyqt:            4.11.4-py27_4     
              python-dateutil: 2.6.0-py27_0      
              pytz:            2016.10-py27_0    
              pyyaml:          3.12-py27_0       
              qt:              4.8.7-4           
              scipy:           0.18.1-np111py27_0
              sip:             4.18-py27_0       
              six:             1.10.0-py27_0     
              sqlalchemy:      1.1.4-py27_0      
           
          The following packages will be UPDATED:
           
              conda:           4.2.12-py27_0      --> 4.2.13-py27_0
              requests:        2.11.1-py27_0      --> 2.12.4-py27_0
           
          ::: Using installed git
          

          Show
          jhoblitt Joshua Hoblitt added a comment - The fix has been confirmed as working in production on the sierra-2 node, which was previously failing. + ./buildbot-scripts/jenkins_wrapper.sh if grep -q -i "CentOS release 6" /etc/redhat-release 2 >/dev/ null ; then # shellcheck disable=SC1091 . /opt/rh/devtoolset- 3 /enable fi set +o verbose add_anaconda_token: True add_pip_as_python_dependency: True allow_softlinks: True always_copy: False always_yes: False auto_update_conda: True binstar_upload: None changeps1: True channel_alias: https: //conda.anaconda.org channel_priority: True channels: - http: //conda-mirror.lsst.codes/pkgs/free client_tls_cert: client_tls_cert_key: create_default_packages: [] debug: False default_channels: - https: //repo.continuum.io/pkgs/free - https: //repo.continuum.io/pkgs/pro disallow: [] envs_dirs: - /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda/envs json: False offline: False proxy_servers: {} quiet: False shortcuts: True show_channel_urls: None ssl_verify: True track_features: [] update_dependencies: True use_pip: True verbosity: 0 Fetching package metadata ..... .Solving package specifications: .......... Package plan for installation in environment /Users/square/jenkins/workspace/stack-os-matrix/label/osx/python/py2/lsstsw/miniconda:   The following packages will be downloaded:   package | build ---------------------------|----------------- mkl- 11.3 . 3 | 0 101.4 MB libpng- 1.6 . 22 | 0 225 KB cython- 0.25 . 2 | py27_0 2.6 MB freetype- 2.5 . 5 | 1 734 KB future- 0.16 . 0 | py27_0 618 KB numpy- 1.11 . 2 | py27_0 3.1 MB pyparsing- 2.1 . 4 | py27_0 70 KB pytz- 2016.10 | py27_0 181 KB pyyaml- 3.12 | py27_0 153 KB requests- 2.12 . 4 | py27_0 752 KB sip- 4.18 | py27_0 245 KB six- 1.10 . 0 | py27_0 16 KB sqlalchemy- 1.1 . 4 | py27_0 1.3 MB astropy- 1.2 . 1 | np111py27_0 6.1 MB bottleneck- 1.2 . 0 | np111py27_0 106 KB conda- 4.2 . 13 | py27_0 375 KB cycler- 0.10 . 0 | py27_0 11 KB numexpr- 2.6 . 0 | np111py27_0 111 KB python-dateutil- 2.6 . 0 | py27_0 232 KB qt- 4.8 . 7 | 4 37.1 MB scipy- 0.18 . 1 | np111py27_0 13.4 MB pandas- 0.19 . 1 | np111py27_0 7.8 MB pyqt- 4.11 . 4 | py27_4 4.0 MB matplotlib- 1.5 . 1 | np111py27_0 6.2 MB ------------------------------------------------------------ Total: 186.8 MB   The following NEW packages will be INSTALLED:   astropy: 1.2 . 1 -np111py27_0 bottleneck: 1.2 . 0 -np111py27_0 cycler: 0.10 . 0 -py27_0 cython: 0.25 . 2 -py27_0 freetype: 2.5 . 5 - 1 future: 0.16 . 0 -py27_0 libpng: 1.6 . 22 - 0 matplotlib: 1.5 . 1 -np111py27_0 mkl: 11.3 . 3 - 0 numexpr: 2.6 . 0 -np111py27_0 numpy: 1.11 . 2 -py27_0 pandas: 0.19 . 1 -np111py27_0 pyparsing: 2.1 . 4 -py27_0 pyqt: 4.11 . 4 -py27_4 python-dateutil: 2.6 . 0 -py27_0 pytz: 2016.10 -py27_0 pyyaml: 3.12 -py27_0 qt: 4.8 . 7 - 4 scipy: 0.18 . 1 -np111py27_0 sip: 4.18 -py27_0 six: 1.10 . 0 -py27_0 sqlalchemy: 1.1 . 4 -py27_0   The following packages will be UPDATED:   conda: 4.2 . 12 -py27_0 --> 4.2 . 13 -py27_0 requests: 2.11 . 1 -py27_0 --> 2.12 . 4 -py27_0   ::: Using installed git
          jhoblitt Joshua Hoblitt made changes -
          Resolution Done [ 10000 ]
          Status In Review [ 10004 ] Done [ 10002 ]
          jhoblitt Joshua Hoblitt made changes -
          Story Points 0.25

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Adam Thornton
              Watchers:
              Adam Thornton, Joshua Hoblitt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel