Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5170

mariadbclient libssl linking problems on Linux

    XMLWordPrintable

Details

    • Data Access and Database

    Description

      I have a build failure in daf_persistence due to mariadbclient not linking to libssl/libcrypto on a linux machine. Can we make the ssloptions fix for OSX apply generically, or would that break something else?

      Here's the daf_persistence error that I received:

      daf_persistence: 2016_01.0-1-gf47bb69 ERROR (33 sec).
      *** error building product daf_persistence.
      *** exit code = 2
      *** log is in /astro/apps6/opt/lsstStacks/lsstsw/build/daf_persistence/_build.log
      *** last few lines:
      :::::  [2016-02-17T23:03:47.312613Z] /astro/apps6/opt/lsstStacks/lsstsw/stack/Linux64/mariadbclient/10.1.11/lib/libmysqlclient_r.so: undefined reference to `SSL_CTX_ctrl'
      :::::  [2016-02-17T23:03:47.312663Z] /astro/apps6/opt/lsstStacks/lsstsw/stack/Linux64/mariadbclient/10.1.11/lib/libmysqlclient_r.so: undefined reference to `ERR_get_error'
      :::::  [2016-02-17T23:03:47.312765Z] /astro/apps6/opt/lsstStacks/lsstsw/stack/Linux64/mariadbclient/10.1.11/lib/libmysqlclient_r.so: undefined reference to `SSL_new'
      :::::  [2016-02-17T23:03:47.312815Z] collect2: error: ld returned 1 exit status
      :::::  [2016-02-17T23:03:47.312834Z] scons: *** [tests/DbStorage_1] Error 1
      :::::  [2016-02-17T23:03:47.735155Z] /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tmpnam':
      :::::  [2016-02-17T23:03:47.735289Z] -------src-dir-------/Python-2.7.11/./Modules/posixmodule.c:7631: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
      :::::  [2016-02-17T23:03:47.735345Z] /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/python2.7/config/libpython2.7.a(posixmodule.o): In function `posix_tempnam':
      :::::  [2016-02-17T23:03:47.735400Z] -------src-dir-------/Python-2.7.11/./Modules/posixmodule.c:7578: warning: the use of `tempnam' is dangerous, better use `mkstemp'
      :::::  [2016-02-17T23:03:47.764830Z] scons: building terminated because of errors.
      

      Here's what my mariadb lib linkage looks like, with the missing libs:

      [16:32:53 parejkoj@magneto: /astro/apps6/opt/lsstStacks/lsstsw/stack/Linux64/mariadbclient/10.1.11/lib]
      $ ldd libmysqlclient.so.18.0.0
      linux-vdso.so.1 => (0x00007ffdb63b4000)
      libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f088ab10000)
      libz.so.1 => /lib64/libz.so.1 (0x00007f088a8f9000)
      libssl.so.1.0.0 => not found
      libcrypto.so.1.0.0 => not found
      libdl.so.2 => /lib64/libdl.so.2 (0x00007f088a6f4000)
      librt.so.1 => /lib64/librt.so.1 (0x00007f088a4ec000)
      libstdc+.so.6 => /usr/lib64/libstdc+.so.6 (0x00007f088a1e6000)
      libm.so.6 => /lib64/libm.so.6 (0x00007f0889f61000)
      libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0889d4b000)
      libc.so.6 => /lib64/libc.so.6 (0x00007f08899b7000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f088b2af000)

      Attachments

        Issue Links

          Activity

            No builds found.
            Parejkoj John Parejko created issue -
            Parejkoj John Parejko added a comment -

            Additional information: mariadbclient found miniconda's ssl files, but then didn't link to them, probably because miniconda isn't in LD_LIBRARY_PATH:

            [2016-02-17T22:46:16.110867Z] -- OPENSSL_INCLUDE_DIR = /astro/apps6/opt/lsstStac
            ks/lsstsw/miniconda/include
            [2016-02-17T22:46:16.110926Z] -- OPENSSL_LIBRARIES = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libssl.so
            [2016-02-17T22:46:16.110968Z] -- CRYPTO_LIBRARY = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libcrypto.so
            [2016-02-17T22:46:16.110986Z] -- OPENSSL_MAJOR_VERSION =
            [2016-02-17T22:46:16.111049Z] -- SSL_LIBRARIES = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libssl.so;/astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libcrypto.so;dl

            Parejkoj John Parejko added a comment - Additional information: mariadbclient found miniconda's ssl files, but then didn't link to them, probably because miniconda isn't in LD_LIBRARY_PATH: [2016-02-17T22:46:16.110867Z] -- OPENSSL_INCLUDE_DIR = /astro/apps6/opt/lsstStac ks/lsstsw/miniconda/include [2016-02-17T22:46:16.110926Z] -- OPENSSL_LIBRARIES = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libssl.so [2016-02-17T22:46:16.110968Z] -- CRYPTO_LIBRARY = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libcrypto.so [2016-02-17T22:46:16.110986Z] -- OPENSSL_MAJOR_VERSION = [2016-02-17T22:46:16.111049Z] -- SSL_LIBRARIES = /astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libssl.so;/astro/apps6/opt/lsstStacks/lsstsw/miniconda/lib/libcrypto.so;dl
            tjenness Tim Jenness added a comment -

            On Linux we let SSL discovery happen as part of the mariadbclient build rather than forcing the internal implementation (as is done on OS X). This leads to the mariadbclient build discovering the miniconda version of libssl so everything works fine. That is until daf_persistence tries to use it and discovers that the miniconda library path is not in the LD_LIBRARY_PATH variable. Hence the link failure. Fix is to tell people they have to set the library path themselves, or else to disable SSL as we do for OS X.

            tjenness Tim Jenness added a comment - On Linux we let SSL discovery happen as part of the mariadbclient build rather than forcing the internal implementation (as is done on OS X). This leads to the mariadbclient build discovering the miniconda version of libssl so everything works fine. That is until daf_persistence tries to use it and discovers that the miniconda library path is not in the LD_LIBRARY_PATH variable. Hence the link failure. Fix is to tell people they have to set the library path themselves, or else to disable SSL as we do for OS X.
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue relates to DM-4929 [ DM-4929 ]
            Parejkoj John Parejko made changes -
            Issue Type Story [ 10001 ] Bug [ 1 ]
            Parejkoj John Parejko made changes -
            Story Points 4
            tjenness Tim Jenness made changes -
            Watchers John Parejko, Tim Jenness [ John Parejko, Tim Jenness ] John Parejko, Lynne Jones, Tim Jenness [ John Parejko, Lynne Jones, Tim Jenness ]
            tjenness Tim Jenness added a comment -

            fritzm jammes is there any objection in making mariadbclient use the internal SSL implementation on Linux as well?

            tjenness Tim Jenness added a comment - fritzm jammes is there any objection in making mariadbclient use the internal SSL implementation on Linux as well?

            Is this a dup of DM-5587?

            swinbank John Swinbank added a comment - Is this a dup of DM-5587 ?
            tjenness Tim Jenness added a comment -

            I think I'd say it's the other way round. I'll fix it. I thought I saw a ticket from ljones last week but then I couldn't find it...

            tjenness Tim Jenness added a comment - I think I'd say it's the other way round. I'll fix it. I thought I saw a ticket from ljones last week but then I couldn't find it...
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-5587 [ DM-5587 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-5595 [ DM-5595 ]
            ljones Lynne Jones added a comment -

            The ticket swinbank linked to (https://jira.lsstcorp.org/browse/DM-5587) is the ticket I filed.
            My ticket just complained that things weren't working.

            ljones Lynne Jones added a comment - The ticket swinbank linked to ( https://jira.lsstcorp.org/browse/DM-5587 ) is the ticket I filed. My ticket just complained that things weren't working.
            jammes Fabrice Jammes added a comment - - edited

            No objection on my side to use Linux SSL lib.

            jammes Fabrice Jammes added a comment - - edited No objection on my side to use Linux SSL lib.

            tjenness Could you clarify what is meant by "internal SSL implementation" above (specifically, internal to what?)

            fritzm Fritz Mueller added a comment - tjenness Could you clarify what is meant by "internal SSL implementation" above (specifically, internal to what?)
            tjenness Tim Jenness added a comment -

            I mean the SSL implementation provided in the MariaDB source. This is what is used on Mac.

            tjenness Tim Jenness added a comment - I mean the SSL implementation provided in the MariaDB source. This is what is used on Mac.

            Well, the only issue I see is that the system SSL libraries are updated frequently and in a timely fashion in response to CVE alerts, etc. What is the vulnerability of MariaDB's internal SSL implementation? I'd be interested in the point of view of somebody more security-conscious here (maybe jhoblitt?)

            I haven't really cared too much to date about the OSX builds of qserv, since they aren't currently intended to be supported for production use. So, whatever works for the convenience of devs on OSX is okay for me, in order to get the bonus of having clang run regularly over our code.

            fritzm Fritz Mueller added a comment - Well, the only issue I see is that the system SSL libraries are updated frequently and in a timely fashion in response to CVE alerts, etc. What is the vulnerability of MariaDB's internal SSL implementation? I'd be interested in the point of view of somebody more security-conscious here (maybe jhoblitt ?) I haven't really cared too much to date about the OSX builds of qserv, since they aren't currently intended to be supported for production use. So, whatever works for the convenience of devs on OSX is okay for me, in order to get the bonus of having clang run regularly over our code.
            tjenness Tim Jenness added a comment -

            Now that DM-5595 has been fixed the impetus to use internal SSL is lessened (Although an inability to install openssl on the particular machine would be a problem). My main question is whether Qserv use the SSL connectivity at the moment.

            tjenness Tim Jenness added a comment - Now that DM-5595 has been fixed the impetus to use internal SSL is lessened (Although an inability to install openssl on the particular machine would be a problem). My main question is whether Qserv use the SSL connectivity at the moment.

            Qserv itself only makes use of the hashing functions. But I don't know if mariadbclient might make use of SSL proper (e.g. talking to the mysqlproxy instance, where the qserv czar is now in-process).

            fritzm Fritz Mueller added a comment - Qserv itself only makes use of the hashing functions. But I don't know if mariadbclient might make use of SSL proper (e.g. talking to the mysqlproxy instance, where the qserv czar is now in-process).
            tjenness Tim Jenness added a comment -

            I'll close this as "Won't Fix" given that the confusion from Anaconda libraries has been fixed in DM-5595 and OpenSSL is still listed as a requirement for Linux.

            tjenness Tim Jenness added a comment - I'll close this as "Won't Fix" given that the confusion from Anaconda libraries has been fixed in DM-5595 and OpenSSL is still listed as a requirement for Linux.
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Won't Fix [ 10405 ]

            I don't know anything about maria's internal TLS libs so I can't really add much to this conversation. My hope is that we'll be able to link against the anaconda openssl package on OSX in the future.

            jhoblitt Joshua Hoblitt added a comment - I don't know anything about maria's internal TLS libs so I can't really add much to this conversation. My hope is that we'll be able to link against the anaconda openssl package on OSX in the future.
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-5802 [ DM-5802 ]

            People

              Unassigned Unassigned
              Parejkoj John Parejko
              Fabrice Jammes, Fritz Mueller, John Parejko, John Swinbank, Joshua Hoblitt, Lynne Jones, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.