Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7747

conda-lsst mpi4py build failure.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The conda-lsst build log for the build failure can be found here:

      https://gist.github.com/jmatt/ea91899f050b5bc82acbb98cc8138ce3

      + ./_build.sh build
      Traceback (most recent call last):
        File "setup.py", line 593, in <module>
          main()
        File "setup.py", line 590, in main
          run_setup()
        File "setup.py", line 469, in run_setup
          **setup_args)
        File "/Users/square/dev/conda-lsst/miniconda/conda-bld/work/conf/mpidistutils.py", line 478, in setup
          return fcn_setup(**attrs)
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/distutils/core.py", line 111, in setup
          _setup_distribution = dist = klass(attrs)
        File "/Users/square/dev/conda-lsst/miniconda/conda-bld/work/conf/mpidistutils.py", line 422, in __init__
          cls_Distribution.__init__(self, attrs)
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 315, in __init__
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 361, in fetch_build_eggs
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 851, in resolve
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1123, in best_match
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/pkg_resources/__init__.py", line 1135, in obtain
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/dist.py", line 428, in fetch_build_egg
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/command/easy_install.py", line 652, in easy_install
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 634, in fetch_distribution
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 469, in find_packages
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 807, in scan_url
        File "/Users/square/dev/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/setuptools-27.2.0-py2.7.egg/setuptools/package_index.py", line 307, in process_url
      RuntimeError: Setuptools downloading is disabled in conda build. Be sure to add all dependencies in the meta.yaml  url=https://pypi.python.org/simple/Cython/r
      

      Since mpi4py is stable and a relatively common external dependency the fix will be to attempt to move to the Conda version. Similar to astropy, numpy, pandas.

      Related to this it may also make sense to do the same with lsst-mpi which is currently mpich (but I believe could be other similar packages).

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Presumably, an alternative fix is to make sure that the dependencies are installed (cython?)

          Show
          tjenness Tim Jenness added a comment - Presumably, an alternative fix is to make sure that the dependencies are installed (cython?)
          Hide
          jmatt J Matt Peterson [X] (Inactive) added a comment -

          I should try that first.

          Show
          jmatt J Matt Peterson [X] (Inactive) added a comment - I should try that first.
          Hide
          jmatt J Matt Peterson [X] (Inactive) added a comment -

          Ok so this is the log for the install. It's concerning because it's installing openmpi-1.6.3. Shouldn't it be using mpich?

          https://gist.github.com/jmatt/a3dc282d1e0960b9ead57f98fd13ee9b

          Show
          jmatt J Matt Peterson [X] (Inactive) added a comment - Ok so this is the log for the install. It's concerning because it's installing openmpi-1.6.3. Shouldn't it be using mpich? https://gist.github.com/jmatt/a3dc282d1e0960b9ead57f98fd13ee9b
          Hide
          tjenness Tim Jenness added a comment - - edited

          openmpi is fine. The mpi EUPS package was explicitly designed for this eventuality: https://github.com/lsst/mpi/blob/master/ups/mpi.table

          We only use MPI from mpi4py so if mpi4py is installed and working, then it doesn't actually matter which MPI is behind the scenes.

          Show
          tjenness Tim Jenness added a comment - - edited openmpi is fine. The mpi EUPS package was explicitly designed for this eventuality: https://github.com/lsst/mpi/blob/master/ups/mpi.table We only use MPI from mpi4py so if mpi4py is installed and working, then it doesn't actually matter which MPI is behind the scenes.
          Hide
          jmatt J Matt Peterson [X] (Inactive) added a comment -

          Appears to work on Mac OS X. Need to test it and build it on Linux still.

          Show
          jmatt J Matt Peterson [X] (Inactive) added a comment - Appears to work on Mac OS X. Need to test it and build it on Linux still.
          Hide
          jmatt J Matt Peterson [X] (Inactive) added a comment -

          Fixed in a commit on the v12.1 branch that will be PRed when Conda binary creation is complete.

          https://github.com/jmatt/conda-lsst/commit/4a7dab3ba0b8a2034248ed4abfd152a226e1a199

          Show
          jmatt J Matt Peterson [X] (Inactive) added a comment - Fixed in a commit on the v12.1 branch that will be PRed when Conda binary creation is complete. https://github.com/jmatt/conda-lsst/commit/4a7dab3ba0b8a2034248ed4abfd152a226e1a199
          Hide
          tjenness Tim Jenness added a comment -

          Assuming it works, it looks fine by inspection.

          Show
          tjenness Tim Jenness added a comment - Assuming it works, it looks fine by inspection.

            People

            • Assignee:
              jmatt J Matt Peterson [X] (Inactive)
              Reporter:
              jmatt J Matt Peterson [X] (Inactive)
              Reviewers:
              Tim Jenness
              Watchers:
              J Matt Peterson [X] (Inactive), Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel