# new conda 'mkl' dependent packages break meas_base tests

XMLWordPrintable

#### Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
4
• Team:
SQuaRE

#### Description

Continuum release/rebuilt a number of packages last friday to depend on the the Intel MKL library.

https://www.continuum.io/blog/developer-blog/anaconda-25-release-now-mkl-optimizations

There are [new feature named] versions that continue to use openblas but the MKL versions appear to be installed by default. This causes at least multiple meas_base tests to fail.After extensive testing, I have confirmed that the meas_base tests do not fail with the equivalent 'nomkl' package.
In addition, mkl is closed source software that requires you to accept and download a license file or it is time-bombed to stop working after a trial period.

  docker-centos-7: [ 36/36 ] meas_base 2015_10.0-9-g6daf04b+7 ...  docker-centos-7:  docker-centos-7: ***** error: from /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/build.log:  docker-centos-7: tests/sincPhotSums.py  docker-centos-7:  docker-centos-7: tests/measureSources.py  docker-centos-7:  docker-centos-7: tests/testApertureFlux.py  docker-centos-7:  docker-centos-7: tests/testJacobian.py  docker-centos-7:  docker-centos-7: tests/testScaledApertureFlux.py  docker-centos-7:  docker-centos-7: The following tests failed:  docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/sincPhotSums.py.failed  docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/measureSources.py.failed  docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testApertureFlux.py.failed  docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testJacobian.py.failed  docker-centos-7: /opt/lsst/software/stack/EupsBuildDir/Linux64/meas_base-2015_10.0-9-g6daf04b+7/meas_base-2015_10.0-9-g6daf04b+7/tests/.tests/testScaledApertureFlux.py.failed  docker-centos-7: 5 tests failed 

(the exact cause of the test failures was not investigated as this should not have happened)

This change has also broken the ability to import an existing conda env from 2016-02-05 or earlier that uses scipy due to some sort of package version resolution problem. Explicit declaring it as the scipy package without mkl fixes the resolution problem.

There is a new 'nomkl' package, when installed, any subsequent package installations will default to versions without mkl. However, this does not fix any already installed packages.

I am traumatized by the lack of reproducible build envs even within a few days of each other. After discussion in the Tucson office, I'm going to pin the lsstsw and newinstall.sh conda package versions with a commitment from square to update them on a monthly basis. I already have a test version of lsstsw/bin/deploy that defaults to a bundled package but with a option flag to use bleeding edge.

#### Activity

Hide
Joshua Hoblitt added a comment -

I'm going to leave this ticket open until a sanity check build of lsst_distrib via newinstall.sh from master has completed.

Show
Joshua Hoblitt added a comment - I'm going to leave this ticket open until a sanity check build of lsst_distrib via newinstall.sh from master has completed.
Hide
Joshua Hoblitt added a comment - - edited

I've opened a PR to bump the version of miniconda2 in newinstall.sh; will self merge as soon as travis finishes.

Show
Joshua Hoblitt added a comment - - edited I've opened a PR to bump the version of miniconda2 in newinstall.sh ; will self merge as soon as travis finishes.
Hide
Joshua Hoblitt added a comment - - edited

Grrrrr. Looks like I broke the PATH construction in miniconda2. (I have no idea why this worked in the pre-merge testing)

Show
Joshua Hoblitt added a comment - - edited Grrrrr. Looks like I broke the PATH construction in miniconda2 . (I have no idea why this worked in the pre-merge testing)
Hide
Joshua Hoblitt added a comment -

Published miniconda2-3.19.0.lsst2 from b1914 with a one liner fix for the path construction.

Show
Joshua Hoblitt added a comment - Published miniconda2-3.19.0.lsst2 from b1914 with a one liner fix for the path construction.
Hide
Joshua Hoblitt added a comment -

It's confirmed, w_2016_05 now builds from a clean sandbox with newinstall.sh.

Show
Joshua Hoblitt added a comment - It's confirmed, w_2016_05 now builds from a clean sandbox with newinstall.sh .

#### People

Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Reviewers:
Mario Juric, Tim Jenness
Watchers:
Frossie Economou, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Mario Juric, Tim Jenness