Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5105

new conda 'mkl' dependent packages break meas_base tests

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Continuum release/rebuilt a number of packages last friday to depend on the the Intel MKL library.

      https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations

      There are [new feature named] versions that continue to use openblas but the MKL versions appear to be installed by default. This causes at least multiple meas_base tests to fail.After extensive testing, I have confirmed that the meas_base tests do not fail with the equivalent 'nomkl' package.
      In addition, mkl is closed source software that requires you to accept and download a license file or it is time-bombed to stop working after a trial period.

          docker-centos-7: [ 36/36 ]  meas_base 2015_10.0-9-g6daf04b+7 ...
          docker-centos-7:
          docker-centos-7: ***** error: from /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/build.log:
          docker-centos-7: tests/sincPhotSums.py
          docker-centos-7:
          docker-centos-7: tests/measureSources.py
          docker-centos-7:
          docker-centos-7: tests/testApertureFlux.py
          docker-centos-7:
          docker-centos-7: tests/testJacobian.py
          docker-centos-7:
          docker-centos-7: tests/testScaledApertureFlux.py
          docker-centos-7:
          docker-centos-7: The following tests failed:
          docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/sincPhotSums.py.failed
          docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/measureSources.py.failed
          docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testApertureFlux.py.failed
          docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testJacobian.py.failed
          docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testScaledApertureFlux.py.failed
          docker-centos-7: 5 tests failed
      

      (the exact cause of the test failures was not investigated as this should not have happened)

      This change has also broken the ability to import an existing conda env from 2016-02-05 or earlier that uses scipy due to some sort of package version resolution problem. Explicit declaring it as the scipy package without mkl fixes the resolution problem.

      There is a new 'nomkl' package, when installed, any subsequent package installations will default to versions without mkl. However, this does not fix any already installed packages.

      I am traumatized by the lack of reproducible build envs even within a few days of each other. After discussion in the Tucson office, I'm going to pin the lsstsw and newinstall.sh conda package versions with a commitment from square to update them on a monthly basis. I already have a test version of lsstsw/bin/deploy that defaults to a bundled package but with a option flag to use bleeding edge.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm going to leave this ticket open until a sanity check build of lsst_distrib via newinstall.sh from master has completed.

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm going to leave this ticket open until a sanity check build of lsst_distrib via newinstall.sh from master has completed.
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            I've opened a PR to bump the version of miniconda2 in newinstall.sh; will self merge as soon as travis finishes.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited I've opened a PR to bump the version of miniconda2 in newinstall.sh ; will self merge as soon as travis finishes.
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            Grrrrr. Looks like I broke the PATH construction in miniconda2. (I have no idea why this worked in the pre-merge testing)

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited Grrrrr. Looks like I broke the PATH construction in miniconda2 . (I have no idea why this worked in the pre-merge testing)
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Published miniconda2-3.19.0.lsst2 from b1914 with a one liner fix for the path construction.

            Show
            jhoblitt Joshua Hoblitt added a comment - Published miniconda2-3.19.0.lsst2 from b1914 with a one liner fix for the path construction.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            It's confirmed, w_2016_05 now builds from a clean sandbox with newinstall.sh.

            Show
            jhoblitt Joshua Hoblitt added a comment - It's confirmed, w_2016_05 now builds from a clean sandbox with newinstall.sh .

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Mario Juric, Tim Jenness
                Watchers:
                Frossie Economou, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Mario Juric, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel