Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3200

Base package fails to build under OS X El Capitan

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • base
    • None
    • Architecture

    Description

      I tried to build the stack on a Mac running El Capitan Public beta 2, and it gave three errors in base.

       % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100 12671  100 12671    0     0  18798      0 --:--:-- --:--:-- --:--:-- 18799
       
      LSST Software Stack Builder
      =======================================================================
       
      Detected git version 2.3.2 (Apple Git-55). OK.
       
       
      In addition to Python 2.7, some LSST packages depend on recent versions of numpy,
      matplotlib, and scipy. If you don't have all of these, the installation may fail.
      Using the Anaconda Python distribution will ensure all these are set up.
       
      Anaconda Python installed by this installer will be managed by LSST's EUPS
      package manager, and will not replace or modify your system python.
       
      Would you like us to install Anaconda Python distribution (if unsure, say yes)? yes
       
      Installing EUPS (v1.5.9)...  done.
      Installing Anaconda Python Distribution ... 
        [  1/1  ]  anaconda 2.2.0                                             done. 
      Installing the basic environment ... 
        [  1/5  ]  doxygen 1.8.5                                              done. 
        [  2/5  ]  python 0.0.3 ...
                   Using externally provided Python 2.7.9 :: Anaconda 2.2.0 (x86_64).
        [  2/5  ]  python 0.0.3                                               done. 
        [  3/5  ]  scons 2.3.4                                                done. 
        [  4/5  ]  sconsUtils 10.1-4-gce72cec+1                               done. 
        [  5/5  ]  lsst 10.1-1-g84e9557+13                                    done. 
      Creating startup scripts (bash) ... done.
      Creating startup scripts (ksh) ... done.
      Creating startup scripts (csh) ... done.
      Creating startup scripts (zsh) ... done.
       
      Bootstrap complete. To continue installing (and to use) the LSST stack
      type one of:
       
      source "/Users/srp/stack/lsst/loadLSST.bash"  # for bash
      source "/Users/srp/stack/lsst/loadLSST.csh"   # for csh
      source "/Users/srp/stack/lsst/loadLSST.ksh"   # for ksh
      source "/Users/srp/stack/lsst/loadLSST.zsh"   # for zsh
       
      Individual LSST packages may now be installed with the usual `eups
      distrib install' command.  For example, to install the science pipeline
      elements of the LSST stack, use:
       
      eups distrib install lsst_apps
       
      Next, read the documentation at:
       
          https://confluence.lsstcorp.org/display/LSWUG/LSST+Software+User+Guide
       
      and feel free to ask any questions via our mailing list at:
       
          https://lists.lsst.org/mailman/listinfo/dm-users
       
                                             Thanks!
                                                      -- The LSST Software Teams
                                                             http://dm.lsst.org/
       
      srps-Mac-Pro:stack srp$ pwd
      /Users/srp/stack
      srps-Mac-Pro:stack srp$ ls
      install.sh	lsst
      srps-Mac-Pro:stack srp$ cd lsst
      srps-Mac-Pro:lsst srp$ ls
      DarwinX86	eups		loadLSST.csh	newinstall.sh
      EupsBuildDir	eupsbuild.log	loadLSST.ksh	site
      _build		loadLSST.bash	loadLSST.zsh	ups_db
      srps-Mac-Pro:lsst srp$ source loadLSST.bash
      srps-Mac-Pro:lsst srp$ eups distrib install lsst_apps
        [  1/51 ]  cfitsio 3360.lsst1                                         done. 
        [  2/51 ]  doxygen 1.8.5 (already installed)                          done. 
        [  3/51 ]  eigen 3.2.0                                                done. 
        [  4/51 ]  fftw 3.3.3                                                 done. 
        [  5/51 ]  gsl 1.16.lsst1                                             done. 
        [  6/51 ]  minuit2 5.28.00                                            done. 
        [  7/51 ]  mysqlclient 5.1.73.lsst1-1-gb8bcc43                        done. 
        [  8/51 ]  python 0.0.3 (already installed)                           done. 
        [  9/51 ]  swig 3.0.2.lsst1                                           done. 
        [ 10/51 ]  xpa 2.1.15.lsst2                                           done. 
        [ 11/51 ]  boost 1.55.0.1.lsst2+3                                     done. 
        [ 12/51 ]  mysqlpython 1.2.3+17                                       done. 
        [ 13/51 ]  numpy 0.0.1+5 ...
                   Using externally provided numpy v1.9.2.
        [ 13/51 ]  numpy 0.0.1+5                                              done. 
        [ 14/51 ]  scons 2.3.4 (already installed)                            done. 
        [ 15/51 ]  wcslib 4.14+7                                              done. 
        [ 16/51 ]  astrometry_net 0.50.1+6                                    done. 
        [ 17/51 ]  matplotlib 0.0.1+5 ...
                   Using externally provided matplotlib v1.4.3.
        [ 17/51 ]  matplotlib 0.0.1+5                                         done. 
        [ 18/51 ]  pyfits 3.2.4.lsst1+3                                       done. 
        [ 19/51 ]  sconsUtils 10.1-4-gce72cec+1 (already installed)           done. 
        [ 20/51 ]  astrometry_net_data 10.0+23                                done. 
        [ 21/51 ]  base 10.1-1-g8080078+14 ...
       
      ***** error: from /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/build.log:
      c++ -o lib/libbase.dylib -dynamiclib -Wl,-install_name -Wl,libbase.dylib src/ModuleImporter.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lpthread
      swig -o tests/testModuleImporterLib_wrap.cc -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 -c++ -python tests/testModuleImporterLib.i
      c++ -o python/lsstcppimport_wrap.os -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -fPIC -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 python/lsstcppimport_wrap.cc
      c++ -o tests/ptr tests/ptr.o -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread
      running tests/ptr... c++ -o tests/testModuleImporter1 tests/testModuleImporter1.o -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread
      c++ -o tests/testModuleImporterLib_wrap.os -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -fPIC -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 tests/testModuleImporterLib_wrap.cc
      sh: line 1: 70182 Trace/BPT trap: 5       "tests/ptr" >> "tests/.tests/ptr" 2>&1
      failed
      buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
      running tests/testModuleImporter1... sh: line 1: 70192 Trace/BPT trap: 5       "tests/testModuleImporter1" >> "tests/.tests/testModuleImporter1" 2>&1
      failed
      doxygen /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/doc/doxygen.conf
      /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/doc/mainpage.dox:61: warning: unable to resolve link to `lsst.pipe.base.cmdLineTask.CmdLineTask' for \link command
      c++ -o python/_lsstcppimport.so -bundle -F/ -undefined suppress -flat_namespace python/lsstcppimport_wrap.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread -ldl -lpython2.7
      c++ -o tests/_testModuleImporterLib.so -bundle -F/ -undefined suppress -flat_namespace tests/testModuleImporterLib_wrap.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread -ldl -lpython2.7
      running tests/testModuleImporter2.py... failed
      3 tests failed
      scons: *** [checkTestStatus] Error 1
      scons: building terminated because of errors.
      + exit -4
      eups distrib: Failed to build base-10.1-1-g8080078+14.eupspkg: Command:
      	source /Users/srp/stack/lsst/eups/bin/setups.sh; export EUPS_PATH=/Users/srp/stack/lsst; (/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/build.sh) >> /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/build.log 2>&1 4>/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/build.msg 
      exited with code 252
       
      srps-Mac-Pro:lsst srp$ 
      

      In the file /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/build.log, near the end, we have:

      Checking whether the C++ compiler works... yes
      C++11 supported with '-std=c++11'
      Checking for C++ header file tr1/unordered_map... no
      Setting up environment to build package 'base'.
      Checking whether int64_t is long ... no
      Determining RTLD values...ok
      scons: done reading SConscript files.
      scons: Building targets ...
      c++ -o src/ModuleImporter.os -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -fPIC -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 src/ModuleImporter.cc
      Cannot guess fingerprint without .git directory; will be set to '0x0'.
      swig -o python/lsstcppimport_wrap.cc -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 -c++ -python python/lsstcppimport.i
      c++ -o tests/ptr.o -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 tests/ptr.cc
      cd python && m4 -Dm4_RTLD_GLOBAL=8 -Dm4_RTLD_NOW=2 < lsst64defs.py.m4 > /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/python/lsst64defs.py
      c++ -o tests/testModuleImporter1.o -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 tests/testModuleImporter1.cc
      c++ -o lib/libbase.dylib -dynamiclib -Wl,-install_name -Wl,libbase.dylib src/ModuleImporter.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lpthread
      swig -o tests/testModuleImporterLib_wrap.cc -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 -c++ -python tests/testModuleImporterLib.i
      c++ -o python/lsstcppimport_wrap.os -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -fPIC -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 python/lsstcppimport_wrap.cc
      c++ -o tests/ptr tests/ptr.o -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread
      running tests/ptr... c++ -o tests/testModuleImporter1 tests/testModuleImporter1.o -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread
      c++ -o tests/testModuleImporterLib_wrap.os -c -std=c++11 -g -DLSST_HAVE_TR1=0 -DLSST_LITTLE_ENDIAN=1 -O3 -Wall -Wno-unused-function -fPIC -Iinclude -I/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/include -I/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/include/python2.7 tests/testModuleImporterLib_wrap.cc
      sh: line 1: 70182 Trace/BPT trap: 5       "tests/ptr" >> "tests/.tests/ptr" 2>&1
      failed
      buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
      running tests/testModuleImporter1... sh: line 1: 70192 Trace/BPT trap: 5       "tests/testModuleImporter1" >> "tests/.tests/testModuleImporter1" 2>&1
      failed
      doxygen /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/doc/doxygen.conf
      /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/doc/mainpage.dox:61: warning: unable to resolve link to `lsst.pipe.base.cmdLineTask.CmdLineTask' for \link command
      c++ -o python/_lsstcppimport.so -bundle -F/ -undefined suppress -flat_namespace python/lsstcppimport_wrap.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread -ldl -lpython2.7
      c++ -o tests/_testModuleImporterLib.so -bundle -F/ -undefined suppress -flat_namespace tests/testModuleImporterLib_wrap.os -Llib -L/Users/srp/stack/lsst/DarwinX86/boost/1.55.0.1.lsst2+3/lib -L/Users/srp/stack/lsst/DarwinX86/anaconda/2.2.0/lib/python2.7/config -lbase -lpthread -ldl -lpython2.7
      running tests/testModuleImporter2.py... failed
      3 tests failed
      scons: *** [checkTestStatus] Error 1
      scons: building terminated because of errors.
      + exit -4
      

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            Can you attach the outputs of the tests/.tests/*.failed files?

            jbosch Jim Bosch added a comment - Can you attach the outputs of the tests/.tests/*.failed files?

            ptr.failed
            tests/ptr

            dyld: Library not loaded: libbase.dylib
            Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/ptr
            Reason: image not found
            -------
            testModuleImporter1.failed
            tests/testModuleImporter1

            dyld: Library not loaded: libbase.dylib
            Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporter1
            Reason: image not found
            -------
            testModuleImporter2.py.failed
            tests/testModuleImporter2.py

            Traceback (most recent call last):
            File "tests/testModuleImporter2.py", line 25, in <module>
            import testModuleImporterLib
            File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module>
            _testModuleImporterLib = swig_import_helper()
            File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper
            _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description)
            ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib
            Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so
            Reason: image not found
            -------

            spietrowicz Steve Pietrowicz added a comment - ptr.failed tests/ptr dyld: Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/ptr Reason: image not found ------- testModuleImporter1.failed tests/testModuleImporter1 dyld: Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporter1 Reason: image not found ------- testModuleImporter2.py.failed tests/testModuleImporter2.py Traceback (most recent call last): File "tests/testModuleImporter2.py", line 25, in <module> import testModuleImporterLib File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module> _testModuleImporterLib = swig_import_helper() File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description) ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so Reason: image not found -------
            jbosch Jim Bosch added a comment -

            Looks like something has changed in how compiled Python modules need to be built; I think these tests are the first ones in the dependency tree that try to import a compiled module we build ourselves.

            jbosch Jim Bosch added a comment - Looks like something has changed in how compiled Python modules need to be built; I think these tests are the first ones in the dependency tree that try to import a compiled module we build ourselves.

            Not sure if this helps at all or not, but I'll add this anyway. I went into /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14, did a "setup -r ." and then executed the failed tests:

            srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/ptr
            srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter1
            ModuleImporter test succeeded.
            srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter2.py
            Traceback (most recent call last):
              File "tests/testModuleImporter2.py", line 25, in <module>
                import testModuleImporterLib
              File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module>
                _testModuleImporterLib = swig_import_helper()
              File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper
                _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description)
            ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib
              Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so
              Reason: image not found
            

            spietrowicz Steve Pietrowicz added a comment - Not sure if this helps at all or not, but I'll add this anyway. I went into /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14, did a "setup -r ." and then executed the failed tests: srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/ptr srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter1 ModuleImporter test succeeded. srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ tests/testModuleImporter2.py Traceback (most recent call last): File "tests/testModuleImporter2.py", line 25, in <module> import testModuleImporterLib File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module> _testModuleImporterLib = swig_import_helper() File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description) ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so Reason: image not found

            $ tests/testModuleImporter2.py
            Traceback (most recent call last):
              File "tests/testModuleImporter2.py", line 25, in <module>
                import testModuleImporterLib
              File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module>
                _testModuleImporterLib = swig_import_helper()
              File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper
                _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description)
            ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib
              Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so
              Reason: image not found
            srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ python tests/testModuleImporter2.py
            .
            ----------------------------------------------------------------------
            Ran 1 test in 0.174s
             
            OK
            $  cat tests/testModuleImporter2.py
            #!/usr/bin/env python
             
            [ stuff deleted ]
            

            spietrowicz Steve Pietrowicz added a comment - $ tests/testModuleImporter2.py Traceback (most recent call last): File "tests/testModuleImporter2.py", line 25, in <module> import testModuleImporterLib File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 28, in <module> _testModuleImporterLib = swig_import_helper() File "/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/testModuleImporterLib.py", line 24, in swig_import_helper _mod = imp.load_module('_testModuleImporterLib', fp, pathname, description) ImportError: dlopen(/Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so, 2): Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/lsst/EupsBuildDir/DarwinX86/base-10.1-1-g8080078+14/base-10.1-1-g8080078+14/tests/_testModuleImporterLib.so Reason: image not found srps-Mac-Pro:base-10.1-1-g8080078+14 srp$ python tests/testModuleImporter2.py . ---------------------------------------------------------------------- Ran 1 test in 0.174s   OK $ cat tests/testModuleImporter2.py #!/usr/bin/env python   [ stuff deleted ]

            SQuaRE takes the ticket but we don't have El Capitain yet so it won't be imminent.

            frossie Frossie Economou added a comment - SQuaRE takes the ticket but we don't have El Capitain yet so it won't be imminent.

            In investigating this further this afternoon, El Capitan disables the inclusion of LD_LIBRARY_PATH and DYLD_LIBRARY_PATH when a new shell is invoked. (This was reported by other developers on other projects). I ran a quick test:

            $ cat boo.sh
            #!/bin/bash
            tests/ptr
            $ ./boo.sh
            dyld: Library not loaded: libbase.dylib
              Referenced from: /Users/srp/stack/EupsBuildDir/DarwinX86/base-11.0/base-11.0/tests/ptr
              Reason: image not found
            ./boo.sh: line 2: 47242 Trace/BPT trap: 5       tests/ptr
            $ cat foo.sh
            #!/bin/bash
            export DYLD_LIBRARY_PATH=$PWD/lib
            tests/ptr
            $ ./foo.sh
            $ 
            

            spietrowicz Steve Pietrowicz added a comment - In investigating this further this afternoon, El Capitan disables the inclusion of LD_LIBRARY_PATH and DYLD_LIBRARY_PATH when a new shell is invoked. (This was reported by other developers on other projects). I ran a quick test: $ cat boo.sh #!/bin/bash tests/ptr $ ./boo.sh dyld: Library not loaded: libbase.dylib Referenced from: /Users/srp/stack/EupsBuildDir/DarwinX86/base-11.0/base-11.0/tests/ptr Reason: image not found ./boo.sh: line 2: 47242 Trace/BPT trap: 5 tests/ptr $ cat foo.sh #!/bin/bash export DYLD_LIBRARY_PATH=$PWD/lib tests/ptr $ ./foo.sh $

            spietrowicz - thanks, very helpful. From looking around, did you get an idea of whether this is a bug or deliberate behaviour (i.e. some kind of new security sandboxing feature?)

            frossie Frossie Economou added a comment - spietrowicz - thanks, very helpful. From looking around, did you get an idea of whether this is a bug or deliberate behaviour (i.e. some kind of new security sandboxing feature?)
            tjenness Tim Jenness added a comment -

            Seems to be a deliberate security feature. If you reboot, twiddle with your boot prom and disable the new security features you can use the stack. That doesn't seem to be a good solution though. Some thought is required. If you run a binary that binary does inherit the library path, if you run something with a #! the library path is sanitized.

            tjenness Tim Jenness added a comment - Seems to be a deliberate security feature. If you reboot, twiddle with your boot prom and disable the new security features you can use the stack. That doesn't seem to be a good solution though. Some thought is required. If you run a binary that binary does inherit the library path, if you run something with a #! the library path is sanitized.

            Yes, definitely deliberate. For the python programs, using "python whatever.py" works. The "#!/usr/bin/env python" at the top is broken, so no more just invoking "whatever.py".

            OS X 10.11.1 beta 2 is already out for developers. I haven't looked it yet to see if they're reverted the behavior, but I suspect they haven't and won't.

            spietrowicz Steve Pietrowicz added a comment - Yes, definitely deliberate. For the python programs, using "python whatever.py" works. The "#!/usr/bin/env python" at the top is broken, so no more just invoking "whatever.py". OS X 10.11.1 beta 2 is already out for developers. I haven't looked it yet to see if they're reverted the behavior, but I suspect they haven't and won't.
            swinbank John Swinbank added a comment - - edited

            According to the System Integrity Protection documentation this is a feature, not a bug: I don't think there's any serious possibility that it will be reverted.

            However. To quote that documentation:

            When a process is started, the kernel checks to see whether the main executable is protected on disk or is signed with an special system entitlement. If either is true, then a flag is set to denote that it is protected against modification...
            Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

            (my emphasis). It looks as though spietrowicz has demonstrated above that /bin/bash is a protected process. However, I'm speculating (I don't have a 10.11 system to check) that end-user installed shells, Python interpreters, etc are not. Worth checking.

            swinbank John Swinbank added a comment - - edited According to the System Integrity Protection documentation this is a feature, not a bug: I don't think there's any serious possibility that it will be reverted. However. To quote that documentation: When a process is started, the kernel checks to see whether the main executable is protected on disk or is signed with an special system entitlement. If either is true, then a flag is set to denote that it is protected against modification... Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes . (my emphasis). It looks as though spietrowicz has demonstrated above that /bin/bash is a protected process. However, I'm speculating (I don't have a 10.11 system to check) that end-user installed shells, Python interpreters, etc are not. Worth checking.

            swinbank is correct, but it's a little complicated because it depends on how that interpreter is invoked:

            $ cat test.py
            #!/usr/bin/env python
            import os, sys
            for i in os.environ:
                if i == "DYLD_LIBRARY_PATH":
                    print "DYLD_LIBRARY_PATH found"
                    sys.exit(0)
             
            print "DYLD_LIBRARY_PATH not found"
            $ python test.py
            DYLD_LIBRARY_PATH found
            $ ./test.py
            DYLD_LIBRARY_PATH not found
            

            But if you hardcode the path to python, you get different results.

            $ cat test2.py
            #!/Users/srp/stack/DarwinX86/anaconda/2.2.0/bin/python
            import os, sys
            for i in os.environ:
                if i == "DYLD_LIBRARY_PATH":
                    print "DYLD_LIBRARY_PATH found"
                    sys.exit(0)
             
            print "DYLD_LIBRARY_PATH not found"
            $ python test2.py
            DYLD_LIBRARY_PATH found
            $ ./test2.py
            DYLD_LIBRARY_PATH found
            $
            

            spietrowicz Steve Pietrowicz added a comment - swinbank is correct, but it's a little complicated because it depends on how that interpreter is invoked: $ cat test.py #!/usr/bin/env python import os, sys for i in os.environ: if i == "DYLD_LIBRARY_PATH": print "DYLD_LIBRARY_PATH found" sys.exit(0)   print "DYLD_LIBRARY_PATH not found" $ python test.py DYLD_LIBRARY_PATH found $ ./test.py DYLD_LIBRARY_PATH not found But if you hardcode the path to python, you get different results. $ cat test2.py #!/Users/srp/stack/DarwinX86/anaconda/2.2.0/bin/python import os, sys for i in os.environ: if i == "DYLD_LIBRARY_PATH": print "DYLD_LIBRARY_PATH found" sys.exit(0)   print "DYLD_LIBRARY_PATH not found" $ python test2.py DYLD_LIBRARY_PATH found $ ./test2.py DYLD_LIBRARY_PATH found $
            price Paul Price added a comment -

            It's not python that's blocking it — it's env.

            price Paul Price added a comment - It's not python that's blocking it — it's env .

            It's actually the location of env.

            If you copy env to /usr/local/bin/env in change to #!/usr/local/bin/env python

            it works.

            spietrowicz Steve Pietrowicz added a comment - It's actually the location of env. If you copy env to /usr/local/bin/env in change to #!/usr/local/bin/env python it works.
            spietrowicz Steve Pietrowicz added a comment - - edited

            I believe we're not going to be able to make Python 2.7 an optional install from now on. On 10.11.x Python 2.7 is installed in /usr/bin, and since it's in that location, it doesn't get passed DYLD_LIBRARY_PATH. (Version is 2.7.10)

            spietrowicz Steve Pietrowicz added a comment - - edited I believe we're not going to be able to make Python 2.7 an optional install from now on. On 10.11.x Python 2.7 is installed in /usr/bin, and since it's in that location, it doesn't get passed DYLD_LIBRARY_PATH. (Version is 2.7.10)
            tjenness Tim Jenness added a comment -

            I updated the summary title of this ticket to reflect the reality of El Capitan not working regardless of beta status.

            tjenness Tim Jenness added a comment - I updated the summary title of this ticket to reflect the reality of El Capitan not working regardless of beta status.
            tjenness Tim Jenness added a comment -

            I've finally had a chance to look at this and I propose the following (which I have proof-of-concepted on El Capitan):

            • Modify the scons build so that the path to the selected python is burned in to the shebang.
            • Change the test driver in sconsUtils to set DYLD_LIBRARY_PATH in the same line of shell script that runs the test. This is made a bit more complicated by the inability to simply set the environment variable in scons and have it appear in the subprocess. (similar to how DM-3856 was implemented).
            • If we do not want to modify scons we have to get the library path in through another mechanism: I suggest we just add a line to all the EUPS table files to define a LSST_LIBRARY_PATH and then copy that for test execution.
            • Executable python scripts will need to have the python path baked in rather than using env. People didn't like the idea of using a autoconf-style .in suffix for the pre-modified file so I suggest we put them in a scripts directory as is, and process each file in their to the bin directory (which conceptually is little different to how the lib directory is populated at the moment during a build.

            It is important to realize that none of this works if people try to build their stack with the system OS X python!

            I have some sconsUtils patches that are sufficient to handle the test target (and only trigger for El Capitan), the shebang processing will need to be added and the rest is just moving things around and (possibly) tweaking all the table files.

            tjenness Tim Jenness added a comment - I've finally had a chance to look at this and I propose the following (which I have proof-of-concepted on El Capitan): Modify the scons build so that the path to the selected python is burned in to the shebang. Change the test driver in sconsUtils to set DYLD_LIBRARY_PATH in the same line of shell script that runs the test. This is made a bit more complicated by the inability to simply set the environment variable in scons and have it appear in the subprocess. (similar to how DM-3856 was implemented). If we do not want to modify scons we have to get the library path in through another mechanism: I suggest we just add a line to all the EUPS table files to define a LSST_LIBRARY_PATH and then copy that for test execution. Executable python scripts will need to have the python path baked in rather than using env . People didn't like the idea of using a autoconf-style .in suffix for the pre-modified file so I suggest we put them in a scripts directory as is, and process each file in their to the bin directory (which conceptually is little different to how the lib directory is populated at the moment during a build. It is important to realize that none of this works if people try to build their stack with the system OS X python! I have some sconsUtils patches that are sufficient to handle the test target (and only trigger for El Capitan), the shebang processing will need to be added and the rest is just moving things around and (possibly) tweaking all the table files.
            tjenness Tim Jenness added a comment -

            ktl rowen I've made initial changes for El Capitan support. The remaining changes are all rote moves of executables to bin.src and modifications of the table files. I think it would make sense to get these changes reviewed before I embark on the bulk changes. After that I can incrementally do the changes and merges without further review. Changing 40 packages and then being told that I've done it wrong is not where I want to go.

            tjenness Tim Jenness added a comment - ktl rowen I've made initial changes for El Capitan support. The remaining changes are all rote moves of executables to bin.src and modifications of the table files. I think it would make sense to get these changes reviewed before I embark on the bulk changes. After that I can incrementally do the changes and merges without further review. Changing 40 packages and then being told that I've done it wrong is not where I want to go.
            rowen Russell Owen added a comment -

            This looks like a good approach to me. In the long run I hope we can move away from using environment variables to setup packages. But we don't have time for that right now, and your solution looks like a good fix that we can have now.

            I like the way you are handling python files in bin/ and tests/. For bin/ rewriting the #! line makes sense, and I am grateful to you for not insisting on .in files. For tests/ the savings of not having to modify the files outweighs the minor inconvenience of making the tests not runnable.

            rowen Russell Owen added a comment - This looks like a good approach to me. In the long run I hope we can move away from using environment variables to setup packages. But we don't have time for that right now, and your solution looks like a good fix that we can have now. I like the way you are handling python files in bin/ and tests/ . For bin/ rewriting the #! line makes sense, and I am grateful to you for not insisting on .in files. For tests/ the savings of not having to modify the files outweighs the minor inconvenience of making the tests not runnable.

            Just an FYI, Xcode 7.1 was released yesterday, and 10.11.1 a few days before that. Won't change what's needed to be done to fix the stack, but I thought I'd mention it.

            spietrowicz Steve Pietrowicz added a comment - Just an FYI, Xcode 7.1 was released yesterday, and 10.11.1 a few days before that. Won't change what's needed to be done to fix the stack, but I thought I'd mention it.
            tjenness Tim Jenness added a comment -

            John, would you be able to have a quick look at this? There are 3 parts to this work:

            1. Modify sconsUtils
            2. Add the new LSST_LIBRARY_PATH variable to EUPS table files everywhere
            3. Move all the .py executables in bin directories to bin.src

            I have examples of all these in branches linked from this ticket but I'd like to get the all clear before I do the big push for steps 2 and 3 above. The main item for review is the sconsUtils changes that I will merge before starting on incrementally merging each package in turn (it will be much quicker if I can merge as I go without having to get each bit reviewed and then having to rebase everything).

            ktl and rowen have agreed to the approach.

            tjenness Tim Jenness added a comment - John, would you be able to have a quick look at this? There are 3 parts to this work: 1. Modify sconsUtils 2. Add the new LSST_LIBRARY_PATH variable to EUPS table files everywhere 3. Move all the .py executables in bin directories to bin.src I have examples of all these in branches linked from this ticket but I'd like to get the all clear before I do the big push for steps 2 and 3 above. The main item for review is the sconsUtils changes that I will merge before starting on incrementally merging each package in turn (it will be much quicker if I can merge as I go without having to get each bit reviewed and then having to rebase everything). ktl and rowen have agreed to the approach.

            I am happy – indeed, keen – to look, but it'll probably take me a couple of days (I want to get another WBS-and-a-half of planning for the DMLT in first). If you're happy to wait for that, then this is on my todo list.

            swinbank John Swinbank added a comment - I am happy – indeed, keen – to look, but it'll probably take me a couple of days (I want to get another WBS-and-a-half of planning for the DMLT in first). If you're happy to wait for that, then this is on my todo list.
            tjenness Tim Jenness added a comment -

            A couple of days is fine.

            tjenness Tim Jenness added a comment - A couple of days is fine.
            rowen Russell Owen added a comment -

            My review of the changes in sconsUtils (I'm working may way through the packages one at a time).

            scripts.py: defaultTargets=("lib", "python", "tests", "examples", "doc", "shebang") is used in two places and had to be modified in two places. I suggest the default value be made a module-level constant.

            scripts.py: if src is None then it appears that rewrite_shebang tries to re-match the first line of every file, even binary executables. This worries me for several reasons:

            • No warning is logged if it runs on a python file and there is no match for the shebang line
            • It seems odd and a bit unwise to read a binary executable in a line-oriented fashion
              I suggest if the file is a .py file then run the match-and-change code and warn if no match and use shutil.copyfile on all non-python files.

            Several files use from X import *, where X is "eups" or "Scons.Script". Please either fix or file a ticket. These is forbidden and prevents pyflakes from running properly. Please fix or file a ticket. I am offering to do the work.

            I appreciate many of the small cleanups you made, such as removing silly line breaks such as {{if }} and expanding one-line conditionals into two lines. Nonetheless, I am a bit grumpy about the many changes to spacing, such as:

            • Changing {{ = }} to = in argument lists. Yes the lack of spaces is recommended in argument lists, but {{ = }} is allowed and is usually more readable unless there are many arguments, so this change made the code harder to read, in my opinion.
            • Adding an extra space before the # in in-line comments.
            • Wrapping a line that was already less than 110 chars
              These changes made the differences harder to understand without materially enhancing readability (and in the first case, degrading it, in my opinion). I suggest undoing those changes before committing.

            installation.py: please consider removing locks = on line 338. The variable is not used, which needlessly upsets pyflakes.

            The following minor annoyances are probably out of scope for this ticket, but if so, I request a new low-priority ticket:

            Many files use % formatting with a bare object as the right hand side, instead of a tuple. This works but is mildly dangerous. In some cases the object is enclosed in paranthesis, but of course that has no effect without a comma after the object.priority ticket.

            The doc strings are non-standard and so will not be seen by python's "help" function.

            rowen Russell Owen added a comment - My review of the changes in sconsUtils (I'm working may way through the packages one at a time). scripts.py: defaultTargets=("lib", "python", "tests", "examples", "doc", "shebang") is used in two places and had to be modified in two places. I suggest the default value be made a module-level constant. scripts.py: if src is None then it appears that rewrite_shebang tries to re-match the first line of every file, even binary executables. This worries me for several reasons: No warning is logged if it runs on a python file and there is no match for the shebang line It seems odd and a bit unwise to read a binary executable in a line-oriented fashion I suggest if the file is a .py file then run the match-and-change code and warn if no match and use shutil.copyfile on all non-python files. Several files use from X import * , where X is "eups" or "Scons.Script". Please either fix or file a ticket. These is forbidden and prevents pyflakes from running properly. Please fix or file a ticket. I am offering to do the work. I appreciate many of the small cleanups you made, such as removing silly line breaks such as {{if }} and expanding one-line conditionals into two lines. Nonetheless, I am a bit grumpy about the many changes to spacing, such as: Changing {{ = }} to = in argument lists. Yes the lack of spaces is recommended in argument lists, but {{ = }} is allowed and is usually more readable unless there are many arguments, so this change made the code harder to read, in my opinion. Adding an extra space before the # in in-line comments. Wrapping a line that was already less than 110 chars These changes made the differences harder to understand without materially enhancing readability (and in the first case, degrading it, in my opinion). I suggest undoing those changes before committing. installation.py: please consider removing locks = on line 338. The variable is not used, which needlessly upsets pyflakes. The following minor annoyances are probably out of scope for this ticket, but if so, I request a new low-priority ticket: Many files use % formatting with a bare object as the right hand side, instead of a tuple. This works but is mildly dangerous. In some cases the object is enclosed in paranthesis, but of course that has no effect without a comma after the object.priority ticket. The doc strings are non-standard and so will not be seen by python's "help" function.
            rowen Russell Owen added a comment -

            The other packages look great. So...overall it's fine. I had one or two substantive requested changes and some picky requests. If you want help with any of it let me know.

            rowen Russell Owen added a comment - The other packages look great. So...overall it's fine. I had one or two substantive requested changes and some picky requests. If you want help with any of it let me know.
            tjenness Tim Jenness added a comment -

            Thanks for the review.

            • I will have a look at removing that defaultTargets duplication. I didn't put that code there to start with so I was trying not to change unrelated logic in this ticket.
            • bin.src != bin. I am only expecting items to be placed in bin.src that need to have their shebang rewritten. It is a bug to put a binary executable in the bin.src directory.
            • I'll see about adding a warning if no shebang is found. If a file is in bin.src it is there to be modified and if it is not being modified that is a bug. I should probably trigger that warning even when not on El Capitan.
            • I do not want to worry about changing the import semantics in this ticket. I will file a ticket indicating the "import *" issue.
            • Regarding the PEP-8 clean ups:
              • I am inclined to stick with the removal of spaces in keyword arguments (I don't think the LSST coding standard expresses an opinion on keyword arguments other than when defining defaults).
              • Two spaces for inline comment is the PEP-8 standard and the LSST coding standard.
              • I did not intend to wrap a line that didn't need wrapping. Scanning through the patch it's not entirely obvious to me which line is being referred to.
            • Regarding readability of the changes, the reformatting was isolated to a single commit and the rest of the commits are nicely self-contained. I think this is one thing where GitHub PRs get in the way of clarity as they prefer you to view the entire change set as one item. I guess they expect a separate cleanup ticket. I can create an entirely new ticket for the sconsUtils PEP-8 work if people prefer that approach. @ktl ?

            Fixing formatting with % is out of scope of this ticket.

            The doc strings are documented to be non-standard. The readme file explicitly states that they are set up for doxygen because it is impossible to import this file outside of the context of scons and therefore help() will never work (and pydoc doesn't work either because pydoc relies on the ability to import the code). sconsUtils might challenge jsick.

            tjenness Tim Jenness added a comment - Thanks for the review. I will have a look at removing that defaultTargets duplication. I didn't put that code there to start with so I was trying not to change unrelated logic in this ticket. bin.src != bin . I am only expecting items to be placed in bin.src that need to have their shebang rewritten. It is a bug to put a binary executable in the bin.src directory. I'll see about adding a warning if no shebang is found. If a file is in bin.src it is there to be modified and if it is not being modified that is a bug. I should probably trigger that warning even when not on El Capitan. I do not want to worry about changing the import semantics in this ticket. I will file a ticket indicating the "import *" issue. Regarding the PEP-8 clean ups: I am inclined to stick with the removal of spaces in keyword arguments (I don't think the LSST coding standard expresses an opinion on keyword arguments other than when defining defaults). Two spaces for inline comment is the PEP-8 standard and the LSST coding standard. I did not intend to wrap a line that didn't need wrapping. Scanning through the patch it's not entirely obvious to me which line is being referred to. Regarding readability of the changes, the reformatting was isolated to a single commit and the rest of the commits are nicely self-contained. I think this is one thing where GitHub PRs get in the way of clarity as they prefer you to view the entire change set as one item. I guess they expect a separate cleanup ticket. I can create an entirely new ticket for the sconsUtils PEP-8 work if people prefer that approach. @ktl ? Fixing formatting with % is out of scope of this ticket. The doc strings are documented to be non-standard. The readme file explicitly states that they are set up for doxygen because it is impossible to import this file outside of the context of scons and therefore help() will never work (and pydoc doesn't work either because pydoc relies on the ability to import the code). sconsUtils might challenge jsick .

            Thank you for the explanations.

            The extra line break is in scripts.py at line 394:

                state.env.SwigLoadableModule("_" + name, src,
                                             LIBS=state.env.getLibs("main python"))
            

            I'm glad to know that the doc strings cannot be found, but I still think using standard doc strings would make the code easier to read. And Doxygen will parse standard doc strings provided they begin with """!. But that is clearly out of scope for this ticket.

            Regarding processing files in bin.src. I had not realize that only python files were even expected to appear in this directory, but I'm glad you will add a warning. A code comment might clarify things a bit. As I recall you prefer that python executables not a ".py" suffix, so the obvious filter won't work. I do wonder what other sort of executables we can expect. C++ and shell scripts both seem plausible. Other languages seem less likely in the DM stack, though I wonder if java can be expected in orchestration. Do we need to munge the shebang for shell scripts?

            rowen Russell Owen added a comment - Thank you for the explanations. The extra line break is in scripts.py at line 394: state.env.SwigLoadableModule("_" + name, src, LIBS=state.env.getLibs("main python")) I'm glad to know that the doc strings cannot be found, but I still think using standard doc strings would make the code easier to read. And Doxygen will parse standard doc strings provided they begin with """! . But that is clearly out of scope for this ticket. Regarding processing files in bin.src. I had not realize that only python files were even expected to appear in this directory, but I'm glad you will add a warning. A code comment might clarify things a bit. As I recall you prefer that python executables not a ".py" suffix, so the obvious filter won't work. I do wonder what other sort of executables we can expect. C++ and shell scripts both seem plausible. Other languages seem less likely in the DM stack, though I wonder if java can be expected in orchestration. Do we need to munge the shebang for shell scripts?
            tjenness Tim Jenness added a comment -

            We might need to be able to spot other shebangs but I'll cross that bridge when we come across the problem. Shell scripts may be a bit of a problem as /bin/sh will not allow DYLD_LIBRARY_PATH to be passed in.

            I do think the documentation in this package will need to be revisited. I'll save that for jsick.

            tjenness Tim Jenness added a comment - We might need to be able to spot other shebangs but I'll cross that bridge when we come across the problem. Shell scripts may be a bit of a problem as /bin/sh will not allow DYLD_LIBRARY_PATH to be passed in. I do think the documentation in this package will need to be revisited. I'll save that for jsick .

            No plans that I'm aware of for Java in orchestration. The only thing in the github DM structure that I'm aware of that uses Java is the complex event processing software in ctrl_evmon, and that's not currently in use or even being distributed.

            spietrowicz Steve Pietrowicz added a comment - No plans that I'm aware of for Java in orchestration. The only thing in the github DM structure that I'm aware of that uses Java is the complex event processing software in ctrl_evmon, and that's not currently in use or even being distributed.
            tjenness Tim Jenness added a comment -

            Closing this now that lsst_distrib builds on El Capitan. I will write up a tech note covering the details.

            Qserv does not build because of issues with partition and qserv.

            tjenness Tim Jenness added a comment - Closing this now that lsst_distrib builds on El Capitan. I will write up a tech note covering the details. Qserv does not build because of issues with partition and qserv.

            Review of technical note http://dmtn-001.lsst.codes/en/master/

            This is a very clear and helpful document. Nonetheless, I do have a few suggested changes.

            "The Summer 2015 version of the LSST software stack does not work with OS X El Capitan". That is true, but I encourage you to highlight the fact that no version of the stack up to and including Summer 2015. Also, the incompatibility extends beyond that point, and I am wondering if we can explicitly tag a new release soon that is compatible with El Capitan, to give a clearer marker, e.g. Winter 2016. In which case you could say something like "OS X 10.11 (El Capitan) is not compatible with any version of the LSST software stack prior to Winter 2016".

            "scons runs the tests and so the tests do not have the correct environment; applications such as processCcd.py will also not function properly." I found this slightly confusing. I suggest you break it into two sentences and explain what you mean by "applications" or even use a different term (I don't think of our bin scripts as applications, but I am not sure what our official documentation calls them), e.g. "Executables in <package>/bin, such as processCcd.py, will also not function properly."

            "2. /usr/bin/env can no longer be used to run python scripts from the command line." I suggest omitting the word "python" because this applies to any executable found with /usr/bin/env. I admit that will make the rest of the paragraph a bit messier, as you then need to point out that we only fix python scripts in bin.src because that is all we have, so far. I think it is worth the work to increase clarity and highlight a possible need to modify other kinds of scripts in the future.

            "One additional complication on El Capitan is that Apple no longer distribute" is the start of an interesting paragraph. Clearly you view "Apple" as a collective noun, whereas I do not. This may be a British vs. American viewpoint or I may just be exposing my ignorance. Perhaps consider wording that makes the distinction less obvious, such as: "One additional complication on El Capitan is that Apple no longer distributes OpenSSL include files. Apple deprecated the use of OpenSSL in OS X 10.7 (Lion) and removed the include files in El Capitan (though the libraries remain for binary compatibility)." (where I used "distributes" but you would probably prefer "distribute").

            In addition, it would help to briefly explain why we felt we could get away with simply removing SSL support. The tickets you refer to later explain this, but a brief note here would be comforting, e.g. "We did not need SSL support in those packages (see DM-4327 and DM-4334) so we disabled it".

            I do not know what this means: "Executable shell scripts should ensure they run setup rather than relying on the setup of the parent shell." Could you please say more, or provide an example?

            "If a package requires OpenSSL consider optionally using Apple CommonCrypto" I suggest a comma after OpenSSL. Also, I had to read it twice to understand it. I think by "optionally using Apple CommonCrypto" you mean as an alternative to OpenSSL, rather than making Apple CommonCrypto optional in the sense that it can be omitted. Perhaps it would be clearer to say "If a package requires OpenSSL, consider supporting Apple CommonCrypto as an alternative" or "If a package requires SSL, consider supporting both Apple CommonCrypto and OpenSSL".

            The hyphen in "pre-requisite" is unnecessary.

            rowen Russell Owen added a comment - Review of technical note http://dmtn-001.lsst.codes/en/master/ This is a very clear and helpful document. Nonetheless, I do have a few suggested changes. "The Summer 2015 version of the LSST software stack does not work with OS X El Capitan". That is true, but I encourage you to highlight the fact that no version of the stack up to and including Summer 2015. Also, the incompatibility extends beyond that point, and I am wondering if we can explicitly tag a new release soon that is compatible with El Capitan, to give a clearer marker, e.g. Winter 2016. In which case you could say something like "OS X 10.11 (El Capitan) is not compatible with any version of the LSST software stack prior to Winter 2016". "scons runs the tests and so the tests do not have the correct environment; applications such as processCcd.py will also not function properly." I found this slightly confusing. I suggest you break it into two sentences and explain what you mean by "applications" or even use a different term (I don't think of our bin scripts as applications, but I am not sure what our official documentation calls them), e.g. "Executables in <package>/bin, such as processCcd.py, will also not function properly." "2. /usr/bin/env can no longer be used to run python scripts from the command line." I suggest omitting the word "python" because this applies to any executable found with /usr/bin/env. I admit that will make the rest of the paragraph a bit messier, as you then need to point out that we only fix python scripts in bin.src because that is all we have, so far. I think it is worth the work to increase clarity and highlight a possible need to modify other kinds of scripts in the future. "One additional complication on El Capitan is that Apple no longer distribute" is the start of an interesting paragraph. Clearly you view "Apple" as a collective noun, whereas I do not. This may be a British vs. American viewpoint or I may just be exposing my ignorance. Perhaps consider wording that makes the distinction less obvious, such as: "One additional complication on El Capitan is that Apple no longer distributes OpenSSL include files. Apple deprecated the use of OpenSSL in OS X 10.7 (Lion) and removed the include files in El Capitan (though the libraries remain for binary compatibility)." (where I used "distributes" but you would probably prefer "distribute"). In addition, it would help to briefly explain why we felt we could get away with simply removing SSL support. The tickets you refer to later explain this, but a brief note here would be comforting, e.g. "We did not need SSL support in those packages (see DM-4327 and DM-4334 ) so we disabled it". I do not know what this means: "Executable shell scripts should ensure they run setup rather than relying on the setup of the parent shell." Could you please say more, or provide an example? "If a package requires OpenSSL consider optionally using Apple CommonCrypto" I suggest a comma after OpenSSL. Also, I had to read it twice to understand it. I think by "optionally using Apple CommonCrypto" you mean as an alternative to OpenSSL, rather than making Apple CommonCrypto optional in the sense that it can be omitted. Perhaps it would be clearer to say "If a package requires OpenSSL, consider supporting Apple CommonCrypto as an alternative" or "If a package requires SSL, consider supporting both Apple CommonCrypto and OpenSSL". The hyphen in "pre-requisite" is unnecessary.

            People

              tjenness Tim Jenness
              spietrowicz Steve Pietrowicz
              Russell Owen
              Chris Walter, David Shupe, Frossie Economou, Jim Bosch, John Swinbank, Paul Price, Russell Owen, Steve Pietrowicz, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.