Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-449

Add GnuTLS as a macOS platform prerequisite

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Retired
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Recent versions of MariaDB (10.2 onwards) require use of an unbundled SSL library. While these versions will build just fine on our CentOS supported platforms, they cannot be built on macOS without adding an unbundled SSL library as a platform prereq.

      GnuTLS is preferred by MariaDB, and seems the path of least resistance (indeed, for those running homebrew a brew install gnutls is all that is needed to make these versions of MariaDB buildable.)

      There is recent pressure to modernize our MariaDB components because of platform compatibility issues concerning the DAX service containers that are being built for use on clusters at CC-IN2P3 and within the PDAC and NCSA. This RFC proposes that GnuTLS be added as a platform prereq on macOS, so that MariaDB components may be upgraded.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            In case it's not clear, this has an impact on the ability to build lsst_apps because the mariadbclient eups package will need to be updated and that is used by daf_persistence.

            Show
            tjenness Tim Jenness added a comment - In case it's not clear, this has an impact on the ability to build lsst_apps because the mariadbclient eups package will need to be updated and that is used by daf_persistence .
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            We are already installing openssl 1.0.x as a conda package. I've searched through the mariadb docs and don't see any mention of gnutls being preferred over openssl. What are the specific reasons for not wanting to use openssl?

            Show
            jhoblitt Joshua Hoblitt added a comment - We are already installing openssl 1.0.x as a conda package. I've searched through the mariadb docs and don't see any mention of gnutls being preferred over openssl . What are the specific reasons for not wanting to use openssl ?
            Hide
            tjenness Tim Jenness added a comment -

            Historically we've had real problems building cmake packages against conda libraries, partly because our build system doesn't know about the conda library location at runtime (so the linker gets confused).

            Show
            tjenness Tim Jenness added a comment - Historically we've had real problems building cmake packages against conda libraries, partly because our build system doesn't know about the conda library location at runtime (so the linker gets confused).
            Hide
            fritzm Fritz Mueller added a comment - - edited

            Joshua Hoblitt you are a better man than I if you can derive a patch which convinces MariaDB's cmake to succeed with the openssl conda package instead of failing on lack of GnuTLS

            (Though seriously, if you can derive such a patch, I can unblock my group today instead of waiting on this RFC, so I would be quite happy! Branches tickets/DM-12397 of mariadb and mariadbclient, with upgraded upstream tarballs and patches, are at your disposal if you wish to try!)

            Show
            fritzm Fritz Mueller added a comment - - edited Joshua Hoblitt you are a better man than I if you can derive a patch which convinces MariaDB's cmake to succeed with the openssl conda package instead of failing on lack of GnuTLS (Though seriously, if you can derive such a patch, I can unblock my group today instead of waiting on this RFC, so I would be quite happy! Branches tickets/ DM-12397 of mariadb and mariadbclient, with upgraded upstream tarballs and patches, are at your disposal if you wish to try!)
            Hide
            tjenness Tim Jenness added a comment -

            I think the problem is that EUPS can't know it needs to add the miniconda lib directory from conda to the DYLD_LIBRARY_PATH (and LSST_LIBRARY_PATH) because table files rely on environment variables and now that miniconda is not an EUPS package you don't have one to use.

            Show
            tjenness Tim Jenness added a comment - I think the problem is that EUPS can't know it needs to add the miniconda lib directory from conda to the DYLD_LIBRARY_PATH (and LSST_LIBRARY_PATH ) because table files rely on environment variables and now that miniconda is not an EUPS package you don't have one to use.
            Hide
            tjenness Tim Jenness added a comment -

            Kian-Tat Lim can you comment on this before we adopt it?

            Show
            tjenness Tim Jenness added a comment - Kian-Tat Lim can you comment on this before we adopt it?
            Hide
            ktl Kian-Tat Lim added a comment - - edited

            Aside from the difficulties with eups and cmake, I'd rather not have any C++ code depend on conda-installed packages, as the stack and other products are intended to be usable with non-conda Python installations.

            Accordingly, the options appear to be making SSL/TLS a system dependency as requested here and making it an eups package.

            Since this is only for macOS, the eups package route looks less desirable.

            We currently have only one system dependency for macOS: cmake itself. But having one means that having two is not as difficult.

            (In the longer term, it may be possible to remove the need for mariadbclient by increasing usage of SQLAlchemy; the Consolidated Database is unlikely to be MySQL or MariaDB in any case.)

            In the end, it seems like the proposal is the best available alternative in the short term.

            Show
            ktl Kian-Tat Lim added a comment - - edited Aside from the difficulties with eups and cmake , I'd rather not have any C++ code depend on conda -installed packages, as the stack and other products are intended to be usable with non- conda Python installations. Accordingly, the options appear to be making SSL/TLS a system dependency as requested here and making it an eups package. Since this is only for macOS, the eups package route looks less desirable. We currently have only one system dependency for macOS: cmake itself. But having one means that having two is not as difficult. (In the longer term, it may be possible to remove the need for mariadbclient by increasing usage of SQLAlchemy; the Consolidated Database is unlikely to be MySQL or MariaDB in any case.) In the end, it seems like the proposal is the best available alternative in the short term.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            For the record, all of the mariadb binaries I have checked are linked against openssl or neither.  That includes the vendored RHEL/centos 7 binaries, Fedora 27 binaries, Debian 9.3 (mariadb 10.1 and it isn't linked against either. The dpkg metadata isn't pulling in either as a dep either – possibly using a bundled tls lib.) and the official Mariadb published debian based docker images. The later actually has gnutls installed, as the included version of libcurl is linked against it, but the Mariadb binaries are not linked against it.

            $ docker run -ti --entrypoint=/bin/bash mariadb:10.2.13 ldd /usr/sbin/mysqld
                linux-vdso.so.1 (0x00007ffe799d1000)
                libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f117bc96000)
                libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f117ba94000)
                libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f117b889000)
                libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f117b66e000)
                libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f117b437000)
                libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f117b213000)
                libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f117afb2000)
                libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f117abb6000)
                libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f117a9b2000)
                libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f117a6a7000)
                libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f117a3a6000)
                libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1179ffb000)
                /lib64/ld-linux-x86-64.so.2 (0x00007f117dc1c000)
                librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1179df3000)
                liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1179bd0000)
                libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f11798ef000)
                libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f11796d8000)
                libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f11794c2000)
                libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f11792b0000)
            

            Show
            jhoblitt Joshua Hoblitt added a comment - For the record, all of the mariadb binaries I have checked are linked against openssl or neither.  That includes the vendored RHEL/centos 7 binaries, Fedora 27 binaries, Debian 9.3 (mariadb 10.1 and it isn't linked against either. The dpkg metadata isn't pulling in either as a dep either – possibly using a bundled tls lib.) and the official Mariadb published debian based docker images. The later actually has gnutls installed, as the included version of libcurl is linked against it, but the Mariadb binaries are not linked against it. $ docker run -ti --entrypoint=/bin/bash mariadb: 10.2 . 13 ldd /usr/sbin/mysqld     linux-vdso.so. 1 ( 0x00007ffe799d1000 )     libpthread.so. 0 => /lib/x86_64-linux-gnu/libpthread.so. 0 ( 0x00007f117bc96000 )     libaio.so. 1 => /lib/x86_64-linux-gnu/libaio.so. 1 ( 0x00007f117ba94000 )     libnuma.so. 1 => /usr/lib/x86_64-linux-gnu/libnuma.so. 1 ( 0x00007f117b889000 )     libz.so. 1 => /lib/x86_64-linux-gnu/libz.so. 1 ( 0x00007f117b66e000 )     libcrypt.so. 1 => /lib/x86_64-linux-gnu/libcrypt.so. 1 ( 0x00007f117b437000 )     libsystemd.so. 0 => /lib/x86_64-linux-gnu/libsystemd.so. 0 ( 0x00007f117b213000 )     libssl.so. 1.0 . 0 => /usr/lib/x86_64-linux-gnu/libssl.so. 1.0 . 0 ( 0x00007f117afb2000 )     libcrypto.so. 1.0 . 0 => /usr/lib/x86_64-linux-gnu/libcrypto.so. 1.0 . 0 ( 0x00007f117abb6000 )     libdl.so. 2 => /lib/x86_64-linux-gnu/libdl.so. 2 ( 0x00007f117a9b2000 )     libstdc++.so. 6 => /usr/lib/x86_64-linux-gnu/libstdc++.so. 6 ( 0x00007f117a6a7000 )     libm.so. 6 => /lib/x86_64-linux-gnu/libm.so. 6 ( 0x00007f117a3a6000 )     libc.so. 6 => /lib/x86_64-linux-gnu/libc.so. 6 ( 0x00007f1179ffb000 )     /lib64/ld-linux-x86- 64 .so. 2 ( 0x00007f117dc1c000 )     librt.so. 1 => /lib/x86_64-linux-gnu/librt.so. 1 ( 0x00007f1179df3000 )     liblzma.so. 5 => /lib/x86_64-linux-gnu/liblzma.so. 5 ( 0x00007f1179bd0000 )     libgcrypt.so. 20 => /lib/x86_64-linux-gnu/libgcrypt.so. 20 ( 0x00007f11798ef000 )     libresolv.so. 2 => /lib/x86_64-linux-gnu/libresolv.so. 2 ( 0x00007f11796d8000 )     libgcc_s.so. 1 => /lib/x86_64-linux-gnu/libgcc_s.so. 1 ( 0x00007f11794c2000 )     libgpg-error.so. 0 => /lib/x86_64-linux-gnu/libgpg-error.so. 0 ( 0x00007f11792b0000 )
            Hide
            fritzm Fritz Mueller added a comment -

            Joshua Hoblitt: right!  Not needed at all for Linux-based builds.  Only needed in the MacOS build environment.

            Show
            fritzm Fritz Mueller added a comment - Joshua Hoblitt : right!  Not needed at all for Linux-based builds.  Only  needed in the MacOS build environment.
            Hide
            fritzm Fritz Mueller added a comment -

            [Joshua Hoblitt: Again, please do feel free to pull branches tickets/DM-12397 of mariadb and mariadbclient if you would like to see if you can cajole the MacOS build into building either ssl-free or against conda openssl in our build enironment?  I suppose MacOS users would be fine with un-encrypted connections here.  I only need the new MariaDB releases to be buildable in eups without breaking our MacOS users and CI!]

            Show
            fritzm Fritz Mueller added a comment - [ Joshua Hoblitt : Again, please do feel free to pull branches tickets/ DM-12397  of mariadb and mariadbclient if you would like to see if you can cajole the MacOS build into building either ssl-free or against conda openssl in our build enironment?  I suppose MacOS users would be fine with un-encrypted connections here.  I only  need the new MariaDB releases to be buildable in eups without breaking our MacOS users and CI!]
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The brew bottle for mariadb 10.2.13 is also linked against openssl....

            $ otool -L /usr/local/bin/mysqld
            /usr/local/bin/mysqld:
            	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.13.0)
            	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
            	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
            	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
            	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
            	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
            	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The brew bottle for mariadb 10.2.13 is also linked against openssl .... $ otool -L /usr/local/bin/mysqld /usr/local/bin/mysqld: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0 . 0 , current version 728.13 . 0 ) /usr/lib/libbz2. 1.0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 5 ) /usr/lib/libz. 1 .dylib (compatibility version 1.0 . 0 , current version 1.2 . 5 ) /usr/local/opt/openssl/lib/libssl. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/local/opt/openssl/lib/libcrypto. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/lib/libc++. 1 .dylib (compatibility version 1.0 . 0 , current version 120.1 . 0 ) /usr/lib/libSystem.B.dylib (compatibility version 1.0 . 0 , current version 1226.10 . 1 )
            Hide
            fritzm Fritz Mueller added a comment - - edited

            Awesome!  All we have to do then is figure out how to do that in the context of eups + miniconda...  Please dig in!

            Show
            fritzm Fritz Mueller added a comment - - edited Awesome!  All we have to do then is figure out how to do that in the context of eups + miniconda...  Please dig in!
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            -WITH_SSL=OFF is silently ignored under 10.2https://jira.mariadb.org/browse/CONC-224 so disablnig TLS isn't an option.

            Show
            jhoblitt Joshua Hoblitt added a comment - -WITH_SSL=OFF is silently ignored under 10.2 – https://jira.mariadb.org/browse/CONC-224 so disablnig TLS isn't an option.
            Hide
            fritzm Fritz Mueller added a comment -

            Yes, sadly – I tried that too.  (It's probably the wrong thing, anyway – all our endpoints, even the internal ones, should really be using TLS.)

            Show
            fritzm Fritz Mueller added a comment - Yes, sadly – I tried that too.  (It's probably the wrong thing, anyway – all our endpoints, even the internal ones, should really be using TLS.)
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            It looks like PKG_CONFIG_LIBDIR needs to be setup for pkg-config. There's several ways this could be accomplished but it will require simultaneous changes to lsstsw and newinstall.sh.

            mac-os-x-el-capitan-build-4:lsstsw square$ cat miniconda/etc/conda/activate.d/env_vars.sh 
            #!/bin/sh
             
            export MINICONDA_ROOT="/Users/square/jhoblitt/lsstsw/miniconda"
            export PKG_CONFIG_LIBDIR=${LD_LIBRARY_PATH}/pkgconfig
            # probably not necessary...
            export LD_LIBRARY_PATH="${MINICONDA_ROOT}/lib"
            export CPATH="${MINICONDA_ROOT}/include"
            mac-os-x-el-capitan-build-4:lsstsw square$ source activate root
            (root) mac-os-x-el-capitan-build-4:lsstsw square$ 
            (root) mac-os-x-el-capitan-build-4:lsstsw square$ rebuild -r tickets/DM-12397-openssl mariadbclient
                   mariadbclient:  ok (238.8 sec).
            # BUILD ID: b3447
                   mariadbclient: tickets.DM-12397-openssl-g993eea619b ok (72.6 sec).
            # BUILD b3447 completed.
            (root) mac-os-x-el-capitan-build-4:lsstsw square$ otool -L stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql
            stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql:
            	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
            	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
            	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
            	@rpath/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
            	@rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
            	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
            	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
            

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited It looks like PKG_CONFIG_LIBDIR needs to be setup for pkg-config . There's several ways this could be accomplished but it will require simultaneous changes to lsstsw and newinstall.sh . mac-os-x-el-capitan-build- 4 :lsstsw square$ cat miniconda/etc/conda/activate.d/env_vars.sh #!/bin/sh   export MINICONDA_ROOT= "/Users/square/jhoblitt/lsstsw/miniconda" export PKG_CONFIG_LIBDIR=${LD_LIBRARY_PATH}/pkgconfig # probably not necessary... export LD_LIBRARY_PATH= "${MINICONDA_ROOT}/lib" export CPATH= "${MINICONDA_ROOT}/include" mac-os-x-el-capitan-build- 4 :lsstsw square$ source activate root (root) mac-os-x-el-capitan-build- 4 :lsstsw square$ (root) mac-os-x-el-capitan-build- 4 :lsstsw square$ rebuild -r tickets/DM- 12397 -openssl mariadbclient mariadbclient: ok ( 238.8 sec). # BUILD ID: b3447 mariadbclient: tickets.DM- 12397 -openssl-g993eea619b ok ( 72.6 sec). # BUILD b3447 completed. (root) mac-os-x-el-capitan-build- 4 :lsstsw square$ otool -L stack/DarwinX86/mariadbclient/tickets.DM- 12397 -openssl-g993eea619b/bin/mysql stack/DarwinX86/mariadbclient/tickets.DM- 12397 -openssl-g993eea619b/bin/mysql: /usr/lib/libedit. 3 .dylib (compatibility version 2.0 . 0 , current version 3.0 . 0 ) /usr/lib/libncurses. 5.4 .dylib (compatibility version 5.4 . 0 , current version 5.4 . 0 ) /usr/lib/libiconv. 2 .dylib (compatibility version 7.0 . 0 , current version 7.0 . 0 ) @rpath /libssl. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) @rpath /libcrypto. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/lib/libc++. 1 .dylib (compatibility version 1.0 . 0 , current version 120.1 . 0 ) /usr/lib/libSystem.B.dylib (compatibility version 1.0 . 0 , current version 1226.10 . 1 )
            Hide
            fritzm Fritz Mueller added a comment -

            Ah – nice detective work there! Are the proposed changes at all controversial, or can we just move ahead with them?

            Show
            fritzm Fritz Mueller added a comment - Ah – nice detective work there! Are the proposed changes at all controversial, or can we just move ahead with them?
            Hide
            ktl Kian-Tat Lim added a comment -

            Can you provide a recipe that works if someone is not using our miniconda (or any other conda-based Python)?

            Show
            ktl Kian-Tat Lim added a comment - Can you provide a recipe that works if someone is not using our miniconda (or any other conda-based Python)?
            Hide
            danielsf Scott Daniel added a comment -

            Sorry to come late to the party.  Is the conclusion of Josh's investigation that there is a way to support Mac users without forcing them to run homebrew?  The sims stack will be affected by this.  It is not obvious that we can jettison our mariadbclient dependence without losing support MySQL database access.  I am trying to figure out what is the best way for us to accommodate this RFC.  Thanks (and, again, sorry for my tardiness).

            Show
            danielsf Scott Daniel added a comment - Sorry to come late to the party.  Is the conclusion of Josh's investigation that there is a way to support Mac users without forcing them to run homebrew?  The sims stack will be affected by this.  It is not obvious that we can jettison our  mariadbclient dependence without losing support MySQL database access.  I am trying to figure out what is the best way for us to accommodate this RFC.  Thanks (and, again, sorry for my tardiness).
            Hide
            fritzm Fritz Mueller added a comment -

            Hi Scott – we're trying pretty hard for the lowest-impact solution. The best proposal right now looks to be removing the stack's dependency on mariadbclient. We would then upgrade the mariadb/mariadbclient packages in eups. This will "break" builds of those packages on MacOS, but within LSST-distributed software at that point only Qserv will depend upon those packages, and Qserv is neither expected nor required to be buildable under MacOS.

            If sims also requires some sort of mysql client, one option would be to have sims users/devs install it as a sims prereq external to the LSST stack? Thanks for chiming in here – if this is not going to be an acceptable solution we should decide that soon, as we already have a developer devoting effort to breaking the stack/mariadbclient dependency!

            Show
            fritzm Fritz Mueller added a comment - Hi Scott – we're trying pretty hard for the lowest-impact solution. The best proposal right now looks to be removing the stack's dependency on mariadbclient. We would then upgrade the mariadb/mariadbclient packages in eups. This will "break" builds of those packages on MacOS, but within LSST-distributed software at that point only Qserv will depend upon those packages, and Qserv is neither expected nor required to be buildable under MacOS. If sims also requires some sort of mysql client, one option would be to have sims users/devs install it as a sims prereq external to the LSST stack? Thanks for chiming in here – if this is not going to be an acceptable solution we should decide that soon, as we already have a developer devoting effort to breaking the stack/mariadbclient dependency!
            Hide
            tjenness Tim Jenness added a comment -

            To clarify. No-one uses the mysql support in daf_persistence so we can remove that and science pipelines would get a faster build. Sims use sqlalchemy and mysqlpython directly so still use mariadbclient. Scott Daniel my understanding is that the fix we have on DM-12397 requires home brew. The problem is that Mac openssl is old and so you need to find a different implementation. Homebrew openssl does not install itself into the normal visible location because they are worried that people would pick up the old openssl by mistake. I don't actually know what macports does. We can't use conda openssl without starting to add conda's lib path to DYLD_LIBRARY_PATH. We could try adding that in loadLSST.xxx to see if that fixes the problem (and might help with iconv problem).

            Show
            tjenness Tim Jenness added a comment - To clarify. No-one uses the mysql support in daf_persistence so we can remove that and science pipelines would get a faster build. Sims use sqlalchemy and mysqlpython directly so still use mariadbclient. Scott Daniel my understanding is that the fix we have on DM-12397 requires home brew. The problem is that Mac openssl is old and so you need to find a different implementation. Homebrew openssl does not install itself into the normal visible location because they are worried that people would pick up the old openssl by mistake. I don't actually know what macports does. We can't use conda openssl without starting to add conda's lib path to DYLD_LIBRARY_PATH . We could try adding that in loadLSST.xxx to see if that fixes the problem (and might help with iconv problem).
            Hide
            danielsf Scott Daniel added a comment - - edited

            It is possible that sims doesn't care about MySQL support, either, so we might be able to get rid of our explicit dependence on `python_mysqlclient` (I am polling our developers, now).

            sqlalchemy does not bring in a mariadbclient dependence itself, so we should be fine there (unless sqlalchemy is supposed to depend on mariadbclient or python_mysqlclient, in which case, that is another problem).

            obs_lsstSim currently depends on python_myssqlclient.  We need obs_lsstSim.  Is that going to change as a part of the plan to remove }}{{mariadb from DM, or is obs_lsstSim considered to be outside of DM?

            Show
            danielsf Scott Daniel added a comment - - edited It is possible that sims doesn't care about MySQL support, either, so we might be able to get rid of our explicit dependence on `python_mysqlclient` (I am polling our developers, now). sqlalchemy does not bring in a mariadbclient dependence itself, so we should be fine there (unless sqlalchemy is supposed to depend on mariadbclient or python_mysqlclient , in which case, that is another problem). obs_lsstSim currently depends on python_myssqlclient .  We need obs_lsstSim .  Is that going to change as a part of the plan to remove }}{{mariadb from DM, or is obs_lsstSim considered to be outside of DM?
            Hide
            danielsf Scott Daniel added a comment -

            Simon informs me that obs_lsstSim doesn't actually use python_mysqlclient, so we may be safe, there

            Show
            danielsf Scott Daniel added a comment - Simon informs me that obs_lsstSim doesn't actually use python_mysqlclient , so we may be safe, there
            Hide
            tjenness Tim Jenness added a comment - - edited

            I see that python_mysqlclient is a dependency for ctrl_stats, obs_sdss, db, obs_lsstSim, scisql, cat and sims_catalogs. Fixing daf_persistence won't help us for those then since they will depend on mariadbclient.

            sqlalchemy does a runtime import of database backend. I can't see any code that explicitly asks for mysql backend though (but I may be looking in the wrong place).

            obs_lsstSim and obs_sdss use mysqlpython to query lsst-db. No idea if that is important or not or whether sqlalchemy could be used instead.

            Show
            tjenness Tim Jenness added a comment - - edited I see that python_mysqlclient is a dependency for ctrl_stats , obs_sdss , db , obs_lsstSim , scisql , cat and sims_catalogs . Fixing daf_persistence won't help us for those then since they will depend on mariadbclient. sqlalchemy does a runtime import of database backend. I can't see any code that explicitly asks for mysql backend though (but I may be looking in the wrong place). obs_lsstSim and obs_sdss use mysqlpython to query lsst-db. No idea if that is important or not or whether sqlalchemy could be used instead.
            Hide
            krughoff Simon Krughoff added a comment -

            I'm not aware of currently used code that uses the SQL DB image selection routines in obs_sdss and obs_lsstSim. The need for these will go away with the Gen3 data abstraction system, so it may be OK to just remove those methods with the expectation that data discovery will change fundamentally in the next few months.

            Show
            krughoff Simon Krughoff added a comment - I'm not aware of currently used code that uses the SQL DB image selection routines in obs_sdss and obs_lsstSim . The need for these will go away with the Gen3 data abstraction system, so it may be OK to just remove those methods with the expectation that data discovery will change fundamentally in the next few months.
            Hide
            tjenness Tim Jenness added a comment -

            I should add that cat, scisql and db are not relevant for lsst_distrib (assuming we finally remove datarel which we have wanted to do forever) so the three that matter are the two obs packages and sims_catalogs. Sounds like the obs packages don't really need it at the moment.

            Show
            tjenness Tim Jenness added a comment - I should add that cat , scisql and db are not relevant for lsst_distrib (assuming we finally remove datarel which we have wanted to do forever) so the three that matter are the two obs packages and sims_catalogs . Sounds like the obs packages don't really need it at the moment.
            Hide
            tjenness Tim Jenness added a comment - - edited

            We are adjusting the plan that was adopted in this RFC.

            • We will document that Gnutls is required for mac users who want to install the mariadb/mariadbclient packages, but we will clean up other dependencies to minimize the impact of this on end users (most users of Qserv and dax servers are linux).
            • We will remove the mariadbclient dependency from daf_persistence (DM-13788)
            • Scott Daniel will remove the explicit python_mysqlclient dependency on sims_catalogs. It uses sqlalchemy and users can install additional backends in the normal python way as needed.
            • We will finally remove datarel from lsst_distrib (DM-2948).
            • Simon Krughoff will file an RFC for removal of the database query code from obs_lsstSim and obs_sdss.

            Once this is done we believe that neither sims nor lsst_distrib will require mariadbclient.

            Show
            tjenness Tim Jenness added a comment - - edited We are adjusting the plan that was adopted in this RFC. We will document that Gnutls is required for mac users who want to install the mariadb/mariadbclient packages, but we will clean up other dependencies to minimize the impact of this on end users (most users of Qserv and dax servers are linux). We will remove the mariadbclient dependency from daf_persistence ( DM-13788 ) Scott Daniel will remove the explicit python_mysqlclient dependency on sims_catalogs . It uses sqlalchemy and users can install additional backends in the normal python way as needed. We will finally remove datarel from lsst_distrib ( DM-2948 ). Simon Krughoff will file an RFC for removal of the database query code from obs_lsstSim and obs_sdss. Once this is done we believe that neither sims nor lsst_distrib will require mariadbclient.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Documenting that lib X is required is not sufficient, it must be present at the exact same path as the eups tarball builds were linked against.

            Show
            jhoblitt Joshua Hoblitt added a comment - Documenting that lib X is required is not sufficient, it must be present at the exact same path as the eups tarball builds were linked against.
            Hide
            tjenness Tim Jenness added a comment -

            For the few people on a mac using the binary eups releases of mariadbclient I have no problem stating that they must use home brew. This will only affect people using dax or qserv on a mac and that means only a few people once we remove the stated dependencies on lsst_Distrib.

            Show
            tjenness Tim Jenness added a comment - For the few people on a mac using the binary eups releases of mariadbclient I have no problem stating that they must use home brew. This will only affect people using dax or qserv on a mac and that means only a few people once we remove the stated dependencies on lsst_Distrib.
            Hide
            fritzm Fritz Mueller added a comment -

            (All the DAX and Qserv devs who do any work outside vms are homebrew users, in any case)

            Show
            fritzm Fritz Mueller added a comment - (All the DAX and Qserv devs who do any work outside vms are homebrew users, in any case)
            Hide
            tjenness Tim Jenness added a comment -

            Fritz Mueller it looks like lsst_distrib and lsst_sims no longer need mariadbclient. I think that means you are cleared to update so long as mariadb builds on the jenkins instances and you update the docs to say that the few people building the eups package of mariadb[client] need to do something extra on Mac.

            Show
            tjenness Tim Jenness added a comment - Fritz Mueller it looks like lsst_distrib and lsst_sims no longer need mariadbclient. I think that means you are cleared to update so long as mariadb builds on the jenkins instances and you update the docs to say that the few people building the eups package of mariadb [client] need to do something extra on Mac.
            Hide
            tjenness Tim Jenness added a comment -

            All triggered work has been completed but I'm unhappy that we did not document the macOS GnuTLS requirement on Qserv. DM-13636 was closed as INVALID.

            Show
            tjenness Tim Jenness added a comment - All triggered work has been completed but I'm unhappy that we did not document the macOS GnuTLS requirement on Qserv. DM-13636 was closed as INVALID.
            Hide
            tjenness Tim Jenness added a comment -

            Fritz Mueller should we be documenting this or should we say that we aren't even trying to build Qserv on macOS? (although in RFC-521 there is an attempt to bring mariadb back into the Mac Jenkins builds).

            Show
            tjenness Tim Jenness added a comment - Fritz Mueller should we be documenting this or should we say that we aren't even trying to build Qserv on macOS? (although in RFC-521 there is an attempt to bring mariadb back into the Mac Jenkins builds).
            Hide
            fritzm Fritz Mueller added a comment -

            I got tired of working the fine points of this, and am quite satisfied to claim that we will no longer even try to build Qserv on macOS (where it is, after all, useless for any practical purpose anyway).

            The rest of the stack may have to take this back up again if mariadb re-enters the stack, but that will be between the stack developers and the CI maintainers and need not involve Qserv.

            Show
            fritzm Fritz Mueller added a comment - I got tired of working the fine points of this, and am quite satisfied to claim that we will no longer even try to build Qserv on macOS (where it is, after all, useless for any practical purpose anyway). The rest of the stack may have to take this back up again if mariadb re-enters the stack, but that will be between the stack developers and the CI maintainers and need not involve Qserv.
            Hide
            womullan Wil O'Mullane added a comment -

            This was not implemented in the way intended -  though the  end result was acceptable.

            Show
            womullan Wil O'Mullane added a comment - This was not implemented in the way intended -  though the  end result was acceptable.

              People

              Assignee:
              fritzm Fritz Mueller
              Reporter:
              fritzm Fritz Mueller
              Watchers:
              Fritz Mueller, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness, Wil O'Mullane
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.