conda-lsst mpi4py build failure.

XMLWordPrintable

Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
1
• Team:
SQuaRE

Description

The conda-lsst build log for the build failure can be found here:

https://gist.github.com/jmatt/ea91899f050b5bc82acbb98cc8138ce3

 + ./_build.sh build Traceback (most recent call last):  File "setup.py", line 593, in   main()  File "setup.py", line 590, in main  run_setup()  File "setup.py", line 469, in run_setup  **setup_args)  File "/Users/square/dev/conda-lsst/miniconda/conda-bld/work/conf/mpidistutils.py", line 478, in setup  return fcn_setup(**attrs)  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/distutils/core.py", line 111, in setup  _setup_distribution = dist = klass(attrs)  File "/Users/square/dev/conda-lsst/miniconda/conda-bld/work/conf/mpidistutils.py", line 422, in __init__  cls_Distribution.__init__(self, attrs)  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 315, in __init__  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 361, in fetch_build_eggs  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 851, in resolve  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1123, in best_match  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1135, in obtain  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 428, in fetch_build_egg  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/easy_install.py", line 652, in easy_install  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 634, in fetch_distribution  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 469, in find_packages  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 807, in scan_url  File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 307, in process_url RuntimeError: Setuptools downloading is disabled in conda build. Be sure to add all dependencies in the meta.yaml url=https://pypi.python.org/simple/Cython/r 

Since mpi4py is stable and a relatively common external dependency the fix will be to attempt to move to the Conda version. Similar to astropy, numpy, pandas.

Related to this it may also make sense to do the same with lsst-mpi which is currently mpich (but I believe could be other similar packages).

Activity

Hide
Tim Jenness added a comment -

Presumably, an alternative fix is to make sure that the dependencies are installed (cython?)

Show
Tim Jenness added a comment - Presumably, an alternative fix is to make sure that the dependencies are installed (cython?)
Hide
J Matt Peterson [X] (Inactive) added a comment -

I should try that first.

Show
J Matt Peterson [X] (Inactive) added a comment - I should try that first.
Hide
J Matt Peterson [X] (Inactive) added a comment -

Ok so this is the log for the install. It's concerning because it's installing openmpi-1.6.3. Shouldn't it be using mpich?

Show
J Matt Peterson [X] (Inactive) added a comment - Ok so this is the log for the install. It's concerning because it's installing openmpi-1.6.3. Shouldn't it be using mpich? https://gist.github.com/jmatt/a3dc282d1e0960b9ead57f98fd13ee9b
Hide
Tim Jenness added a comment - - edited

openmpi is fine. The mpi EUPS package was explicitly designed for this eventuality: https://github.com/lsst/mpi/blob/master/ups/mpi.table

We only use MPI from mpi4py so if mpi4py is installed and working, then it doesn't actually matter which MPI is behind the scenes.

Show
Tim Jenness added a comment - - edited openmpi is fine. The mpi EUPS package was explicitly designed for this eventuality: https://github.com/lsst/mpi/blob/master/ups/mpi.table We only use MPI from mpi4py so if mpi4py is installed and working, then it doesn't actually matter which MPI is behind the scenes.
Hide
J Matt Peterson [X] (Inactive) added a comment -

Appears to work on Mac OS X. Need to test it and build it on Linux still.

Show
J Matt Peterson [X] (Inactive) added a comment - Appears to work on Mac OS X. Need to test it and build it on Linux still.
Hide
J Matt Peterson [X] (Inactive) added a comment -

Fixed in a commit on the v12.1 branch that will be PRed when Conda binary creation is complete.

https://github.com/jmatt/conda-lsst/commit/4a7dab3ba0b8a2034248ed4abfd152a226e1a199

Show
J Matt Peterson [X] (Inactive) added a comment - Fixed in a commit on the v12.1 branch that will be PRed when Conda binary creation is complete. https://github.com/jmatt/conda-lsst/commit/4a7dab3ba0b8a2034248ed4abfd152a226e1a199
Hide
Tim Jenness added a comment -

Assuming it works, it looks fine by inspection.

Show
Tim Jenness added a comment - Assuming it works, it looks fine by inspection.

People

• Assignee:
J Matt Peterson [X] (Inactive)
Reporter:
J Matt Peterson [X] (Inactive)
Reviewers:
Tim Jenness
Watchers:
J Matt Peterson [X] (Inactive), Tim Jenness