Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12693

Galsim does not build on macOS anaconda python3.6.3

    XMLWordPrintable

Details

    • Architecture

    Description

      It seems that using a stack with python3.6.3 on macOS by updating python from a previous version, breaks Galsim. The configuration tests fail because python gets confused about thread states. A cursory glance at the error indicates that this might be because the newest python from anaconda no longer links against a shared libpython and the configuration tests seem to link against a static libpython and I postulate that this is leading to interpreter state confusion when the tests run.

      Anaconda 3.6.2 seems to have a libpython and galsim builds fine.

      Attachments

        Activity

          tjenness Tim Jenness added a comment -

          I have tested with GalSim 1.5 (SIM-2550) and that also fails.

          tjenness Tim Jenness added a comment - I have tested with GalSim 1.5 (SIM-2550) and that also fails.

          For the record, the build is fine on macOS 10.13.1 / Python 3.6.3 when Anaconda isn't in the loop:

          [jds@magpie ~]$ eups distrib install galsim 1.4.3.lsst1+2
             [  1/10 ]  fftw 3.3.4.lsst2 (already installed)                       done.
             [  2/10 ]  python 0.0.7 (already installed)                           done.
             [  3/10 ]  boost 1.60.lsst1+2 (already installed)                     done.
             [  4/10 ]  numpy 1.13.1 (already installed)                           done.
             [  5/10 ]  python_d2to1 0.2.12.lsst2+2 (already installed)            done.
             [  6/10 ]  scons 3.0.0.lsst1+1 (already installed)                    done.
             [  7/10 ]  stsci_distutils 0.3.7.lsst1+3 (already installed)          done.
             [  8/10 ]  tmv 0.73.lsst1+2 (already installed)                       done.
             [  9/10 ]  pyfits 3.4.0+9 (already installed)                         done.
             [ 10/10 ]  galsim 1.4.3.lsst1+2                                       done.
          [jds@magpie ~]$ python --version
          Python 3.6.3
          [jds@magpie ~]$ sw_vers 
          ProductName:    Mac OS X
          ProductVersion: 10.13.1
          

          Note that this also does not link against a shared libpython:

          [jds@magpie ~]$ otool -L $(which python)
          /opt/local/bin/python:
                  /opt/local/Library/Frameworks/Python.framework/Versions/3.6/Python (compatibility version 3.6.0, current version 3.6.0)
                  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
          

          swinbank John Swinbank added a comment - For the record, the build is fine on macOS 10.13.1 / Python 3.6.3 when Anaconda isn't in the loop: [jds@magpie ~]$ eups distrib install galsim 1.4.3.lsst1+2 [ 1/10 ] fftw 3.3.4.lsst2 (already installed) done. [ 2/10 ] python 0.0.7 (already installed) done. [ 3/10 ] boost 1.60.lsst1+2 (already installed) done. [ 4/10 ] numpy 1.13.1 (already installed) done. [ 5/10 ] python_d2to1 0.2.12.lsst2+2 (already installed) done. [ 6/10 ] scons 3.0.0.lsst1+1 (already installed) done. [ 7/10 ] stsci_distutils 0.3.7.lsst1+3 (already installed) done. [ 8/10 ] tmv 0.73.lsst1+2 (already installed) done. [ 9/10 ] pyfits 3.4.0+9 (already installed) done. [ 10/10 ] galsim 1.4.3.lsst1+2 done. [jds@magpie ~]$ python --version Python 3.6.3 [jds@magpie ~]$ sw_vers ProductName: Mac OS X ProductVersion: 10.13.1 Note that this also does not link against a shared libpython : [jds@magpie ~]$ otool -L $(which python) /opt/local/bin/python: /opt/local/Library/Frameworks/Python.framework/Versions/3.6/Python (compatibility version 3.6.0, current version 3.6.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
          tjenness Tim Jenness added a comment -

          The problem seems to be that GalSim doesn't build native modules like setuptools does. When using setuptools modules are built with -bundle -undefined dynamic_lookup and libpython is not explicitly linked. GalSim is using -bundle on its own and explicitly linking against libpython. It also links against a .a library because in python3.6.3 doesn't put the .so in the architecture specific library directory. This all becomes an issue because Anaconda 3.6.3 does not use a shared python library at all and all the python symbols are in the binary itself. If you try to link your module against either the .a or .so you end up with two interpreter states clashing and the one from the module has not been initialized properly. If I compile the test files using dynamic lookup and no libpython they build fine and python 3.6.3 can import them.

          I need to work out how setuptools decides whther it needs to link against libpython or not.

          tjenness Tim Jenness added a comment - The problem seems to be that GalSim doesn't build native modules like setuptools does. When using setuptools modules are built with -bundle -undefined dynamic_lookup and libpython is not explicitly linked. GalSim is using -bundle on its own and explicitly linking against libpython . It also links against a .a library because in python3.6.3 doesn't put the .so in the architecture specific library directory. This all becomes an issue because Anaconda 3.6.3 does not use a shared python library at all and all the python symbols are in the binary itself. If you try to link your module against either the .a or .so you end up with two interpreter states clashing and the one from the module has not been initialized properly. If I compile the test files using dynamic lookup and no libpython they build fine and python 3.6.3 can import them. I need to work out how setuptools decides whther it needs to link against libpython or not.
          tjenness Tim Jenness added a comment -

          If I look at https://docs.python.org/3/extending/building.html#building-c-and-c-extensions-with-distutils it seems pretty clear that libpython is not meant to be included in the build when making a shared library module for python. Also, distutils seems to want to create that shared library using sysconfig.get_config_var("LDSHARED").

          tjenness Tim Jenness added a comment - If I look at https://docs.python.org/3/extending/building.html#building-c-and-c-extensions-with-distutils it seems pretty clear that libpython is not meant to be included in the build when making a shared library module for python. Also, distutils seems to want to create that shared library using sysconfig.get_config_var("LDSHARED") .
          tjenness Tim Jenness added a comment - I've filed a ticket upstream: https://github.com/GalSim-developers/GalSim/issues/924
          tjenness Tim Jenness added a comment -

          Parejkoj I'm opening this for review in case you are in a position to test it. Otherwise, since it's from upstream I'll do a quick Jenkins test and merge it.

          tjenness Tim Jenness added a comment - Parejkoj I'm opening this for review in case you are in a position to test it. Otherwise, since it's from upstream I'll do a quick Jenkins test and merge it.
          tjenness Tim Jenness added a comment -

          I've merged this since it passes Jenkins, works on my laptop where it didn't work before and upstream has merged it.

          tjenness Tim Jenness added a comment - I've merged this since it passes Jenkins, works on my laptop where it didn't work before and upstream has merged it.

          People

            Unassigned Unassigned
            tjenness Tim Jenness
            John Parejko
            John Parejko, John Swinbank, Scott Daniel, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.