Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23492

Jenkins stack-os-matrix jobs failing in conda environment build.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jenkins
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Debug causes of CentOS failures.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            I think the difference is that the pytest and astropy tweaks to the end were entirely standalone and didn't impact other packages (you can do `conda install pytest=x` without having to change numpy). My new conda env changes numpy/astropy/scipy in ways that mean that you can't downgrade without changing everything and since there are packages in the new env that aren't in the old one (they split numpy and matplotlib into two) I think trying to install an old env on top causes real problems trying to resolve exactly which numpy is being used etc. You can't downgrade numpy, for example, without forcing every package to downgrade and clearly the conda solver couldn't handle it.

            Show
            tjenness Tim Jenness added a comment - I think the difference is that the pytest and astropy tweaks to the end were entirely standalone and didn't impact other packages (you can do `conda install pytest=x` without having to change numpy). My new conda env changes numpy/astropy/scipy in ways that mean that you can't downgrade without changing everything and since there are packages in the new env that aren't in the old one (they split numpy and matplotlib into two) I think trying to install an old env on top causes real problems trying to resolve exactly which numpy is being used etc. You can't downgrade numpy, for example, without forcing every package to downgrade and clearly the conda solver couldn't handle it.
            Hide
            ktl Kian-Tat Lim added a comment -

            I have implemented the first bullet and the LSST_CONDA_ENV_NAME fix in https://github.com/lsst-sqre/ci-scripts/pull/65

            I will discuss DM-23098 with Leanne later this afternoon.

            Show
            ktl Kian-Tat Lim added a comment - I have implemented the first bullet and the LSST_CONDA_ENV_NAME fix in https://github.com/lsst-sqre/ci-scripts/pull/65 I will discuss DM-23098 with Leanne later this afternoon.
            Hide
            mbutler Michelle Butler added a comment -

            @mlong executed:  
            cd /project/jenkins/prod
            rm -rf agent-ldfc-ws-0/ws/stack-os-matrix/centos-7.devtoolset-8.py3 agent-ldfc-ws-2/ws/stack-os-matrix/centos-7.devtoolset-8.py3 agent-ldfc-ws-4/ws/stack-os-matrix/centos-6.devtoolset-8.py3

            on direction from KTL

            Show
            mbutler Michelle Butler added a comment - @mlong executed:   cd /project/jenkins/prod rm -rf agent-ldfc-ws-0/ws/stack-os-matrix/centos-7.devtoolset-8.py3 agent-ldfc-ws-2/ws/stack-os-matrix/centos-7.devtoolset-8.py3 agent-ldfc-ws-4/ws/stack-os-matrix/centos-6.devtoolset-8.py3 on direction from KTL
            Hide
            gcomoretto Gabriele Comoretto added a comment -
            Show
            gcomoretto Gabriele Comoretto added a comment - The PR  https://github.com/lsst-sqre/ci-scripts/pull/65  seems OK to me.
            Hide
            mtlong2 Matthew Thomas Long added a comment -

            rm completed.

            Let me know if anything else needs done from our side.

            Show
            mtlong2 Matthew Thomas Long added a comment - rm completed. Let me know if anything else needs done from our side.

              People

              • Assignee:
                ktl Kian-Tat Lim
                Reporter:
                ktl Kian-Tat Lim
                Reviewers:
                Gabriele Comoretto
                Watchers:
                Gabriele Comoretto, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Matthew Thomas Long, Michelle Butler, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel