Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4931

Qserv build fails on El Capitan with missing OpenSSL

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None
    • Story Points:
      2
    • Team:
      Architecture

      Description

      Qserv does not build on OS X El Capitan due to the absence of OpenSSL include files. Apple now only ship the OpenSSL library (for backwards compatibility reasons). Qserv only uses SSL in two places to calculate digests (MD5 and SHA). This functionality is available in the Apple CommonCrypto library. Qserv digest code needs to be taught how to use CommonCrypto.

        Attachments

          Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. DM-5030 Tests fail on Qserv on OS X El Capitan because of SIP

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-5595 daf_persistence build failure on OSX

          • Done

            Activity

            Ascending order - Click to sort in descending order
            6 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Yes, tried adding the -L/usr/lib explicitly but the linker converts that to the SDK location with the table files:

            "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -v -L/usr/lib -demangle -dynamic -dylib -arch x86_64 -flat_namespace -headerpad_max_install_names -macosx_version_min 10.11.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -undefined suppress -undefined suppress -o build/libqserv_common.dylib -Lbuild -Lcore/modules -L/Users/timj/work/lsstsw/stack/DarwinX86/xrootd/4.3.0.rc4a.lsst1+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/protobuf/2.6.1.lsst3+fbf04ba888/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/sphgeom/master-gb10b2753f2+fca56db4f7/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/antlr/2.7.7.lsst1+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/mariadb/tickets.DM-4929-g342f8ac52e+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/log4cxx/0.10.0.lsst6+ea1ce4b7f0/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/log/2015_10.0-2-g81df675+ae92025211/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/boost/1.59.lsst5+fbf04ba888/lib -L/usr/lib -install_name libqserv_common.dylib build/global/ResourceUnit.os build/global/sqltoken.os build/global/debugUtil.os build/proto/ProtoHeaderWrap.os build/proto/TaskMsgDigest.os build/proto/worker.pb.os build/mysql/SchemaFactory.os build/mysql/RowBuffer.os build/mysql/LocalInfile.os build/mysql/MySqlConfig.os build/mysql/MySqlConnection.os build/sql/SqlConnection.os build/sql/MockSql.os build/sql/SqlErrorObject.os build/sql/SqlResults.os build/sql/SqlTransaction.os build/sql/statement.os build/sql/SqlFragmenter.os build/util/Command.os build/util/WorkQueue.os build/util/MultiError.os build/util/Thread.os build/util/Substitution.os build/util/Issue.os build/util/EventThread.os build/util/xrootd.os build/util/StringHash.os build/util/DynamicWorkQueue.os build/util/Error.os build/util/Timer.os build/util/MmapFile.os -llog -lprotobuf -lmysqlclient_r -lboost_thread /usr/lib/libz.dylib -lc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin/libclang_rt.osx.a
            		 
            @(#)PROGRAM:ld  PROJECT:ld64-253.9
            		 
            configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
            		 
            Library search paths:
            		 
            	/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
            		 
            	build
            		 
            	core/modules
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/xrootd/4.3.0.rc4a.lsst1+da39a3ee5e/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/protobuf/2.6.1.lsst3+fbf04ba888/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/sphgeom/master-gb10b2753f2+fca56db4f7/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/antlr/2.7.7.lsst1+da39a3ee5e/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/mariadb/tickets.DM-4929-g342f8ac52e+da39a3ee5e/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/log4cxx/0.10.0.lsst6+ea1ce4b7f0/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/log/2015_10.0-2-g81df675+ae92025211/lib
            		 
            	/Users/timj/work/lsstsw/stack/DarwinX86/boost/1.59.lsst5+fbf04ba888/lib
            		 
            	/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
            		 
            	/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
            		 
            Framework search paths:
            		 
            	/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/
            		 
            ld: file not found: libz.1.dylib for architecture x86_64

            Note the doubling up of the MacOSX10.11.sdk line. It's all a bit weird.

            In other news when I put the libz.1.dylib soft link in the build works but all the tests fail. This is the problem I was expecting from the beginning and is caused by SIP disabling the DYLD_LIBRARY_PATH variable inside scons. The fix will be to copy the logic over from sconsUtils that handles copying the environment from LSST_LIBRARY_PATH. That's for a separate ticket.

            Show
            tjenness Tim Jenness added a comment - Yes, tried adding the -L/usr/lib explicitly but the linker converts that to the SDK location with the table files: "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -v -L/usr/lib -demangle -dynamic -dylib -arch x86_64 -flat_namespace -headerpad_max_install_names -macosx_version_min 10.11.0 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -undefined suppress -undefined suppress -o build/libqserv_common.dylib -Lbuild -Lcore/modules -L/Users/timj/work/lsstsw/stack/DarwinX86/xrootd/4.3.0.rc4a.lsst1+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/protobuf/2.6.1.lsst3+fbf04ba888/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/sphgeom/master-gb10b2753f2+fca56db4f7/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/antlr/2.7.7.lsst1+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/mariadb/tickets.DM-4929-g342f8ac52e+da39a3ee5e/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/log4cxx/0.10.0.lsst6+ea1ce4b7f0/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/log/2015_10.0-2-g81df675+ae92025211/lib -L/Users/timj/work/lsstsw/stack/DarwinX86/boost/1.59.lsst5+fbf04ba888/lib -L/usr/lib -install_name libqserv_common.dylib build/global/ResourceUnit.os build/global/sqltoken.os build/global/debugUtil.os build/proto/ProtoHeaderWrap.os build/proto/TaskMsgDigest.os build/proto/worker.pb.os build/mysql/SchemaFactory.os build/mysql/RowBuffer.os build/mysql/LocalInfile.os build/mysql/MySqlConfig.os build/mysql/MySqlConnection.os build/sql/SqlConnection.os build/sql/MockSql.os build/sql/SqlErrorObject.os build/sql/SqlResults.os build/sql/SqlTransaction.os build/sql/statement.os build/sql/SqlFragmenter.os build/util/Command.os build/util/WorkQueue.os build/util/MultiError.os build/util/Thread.os build/util/Substitution.os build/util/Issue.os build/util/EventThread.os build/util/xrootd.os build/util/StringHash.os build/util/DynamicWorkQueue.os build/util/Error.os build/util/Timer.os build/util/MmapFile.os -llog -lprotobuf -lmysqlclient_r -lboost_thread /usr/lib/libz.dylib -lc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.2/lib/darwin/libclang_rt.osx.a @(#)PROGRAM:ld PROJECT:ld64-253.9 configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS) Library search paths: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib build core/modules /Users/timj/work/lsstsw/stack/DarwinX86/xrootd/4.3.0.rc4a.lsst1+da39a3ee5e/lib /Users/timj/work/lsstsw/stack/DarwinX86/protobuf/2.6.1.lsst3+fbf04ba888/lib /Users/timj/work/lsstsw/stack/DarwinX86/sphgeom/master-gb10b2753f2+fca56db4f7/lib /Users/timj/work/lsstsw/stack/DarwinX86/antlr/2.7.7.lsst1+da39a3ee5e/lib /Users/timj/work/lsstsw/stack/DarwinX86/mariadb/tickets.DM-4929-g342f8ac52e+da39a3ee5e/lib /Users/timj/work/lsstsw/stack/DarwinX86/log4cxx/0.10.0.lsst6+ea1ce4b7f0/lib /Users/timj/work/lsstsw/stack/DarwinX86/log/2015_10.0-2-g81df675+ae92025211/lib /Users/timj/work/lsstsw/stack/DarwinX86/boost/1.59.lsst5+fbf04ba888/lib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib Framework search paths: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/ ld: file not found: libz.1.dylib for architecture x86_64 Note the doubling up of the MacOSX10.11.sdk line. It's all a bit weird. In other news when I put the libz.1.dylib soft link in the build works but all the tests fail. This is the problem I was expecting from the beginning and is caused by SIP disabling the DYLD_LIBRARY_PATH variable inside scons . The fix will be to copy the logic over from sconsUtils that handles copying the environment from LSST_LIBRARY_PATH . That's for a separate ticket.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            It seems I had a bad XCode installation. John Swinbank was able to build this branch and link okay (tests failed but that is a known problem). I have deleted my XCode and the linking is now fine for me.

            Show
            tjenness Tim Jenness added a comment - It seems I had a bad XCode installation. John Swinbank was able to build this branch and link okay (tests failed but that is a known problem). I have deleted my XCode and the linking is now fine for me.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Pull request created. In particular I would like advice on my implementation of the fix in StringHash.cc.

            Show
            tjenness Tim Jenness added a comment - Pull request created. In particular I would like advice on my implementation of the fix in StringHash.cc .
            Hide
            Permalink
            fritzm Fritz Mueller added a comment -

            Hey Tim – pushed some suggested cleanup for StringHash.cc onto the PR for your consideration. Thanks!

            Show
            fritzm Fritz Mueller added a comment - Hey Tim – pushed some suggested cleanup for StringHash.cc onto the PR for your consideration. Thanks!
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Thanks for the fixes. Builds for me so I intend to merge and push.

            Show
            tjenness Tim Jenness added a comment - Thanks for the fixes. Builds for me so I intend to merge and push.

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Fritz Mueller
              Watchers:
              Andy Salnikov, Fritz Mueller, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.