# Add GnuTLS as a macOS platform prerequisite

XMLWordPrintable

#### Details

• Type: RFC
• Status: Retired
• Resolution: Done
• Component/s:
• Labels:
None

#### Description

Recent versions of MariaDB (10.2 onwards) require use of an unbundled SSL library. While these versions will build just fine on our CentOS supported platforms, they cannot be built on macOS without adding an unbundled SSL library as a platform prereq.

GnuTLS is preferred by MariaDB, and seems the path of least resistance (indeed, for those running homebrew a brew install gnutls is all that is needed to make these versions of MariaDB buildable.)

There is recent pressure to modernize our MariaDB components because of platform compatibility issues concerning the DAX service containers that are being built for use on clusters at CC-IN2P3 and within the PDAC and NCSA. This RFC proposes that GnuTLS be added as a platform prereq on macOS, so that MariaDB components may be upgraded.

#### Activity

Hide
Tim Jenness added a comment -

In case it's not clear, this has an impact on the ability to build lsst_apps because the mariadbclient eups package will need to be updated and that is used by daf_persistence.

Show
Tim Jenness added a comment - In case it's not clear, this has an impact on the ability to build lsst_apps because the mariadbclient eups package will need to be updated and that is used by daf_persistence .
Hide
Joshua Hoblitt added a comment -

We are already installing openssl 1.0.x as a conda package. I've searched through the mariadb docs and don't see any mention of gnutls being preferred over openssl. What are the specific reasons for not wanting to use openssl?

Show
Joshua Hoblitt added a comment - We are already installing openssl 1.0.x as a conda package. I've searched through the mariadb docs and don't see any mention of gnutls being preferred over openssl . What are the specific reasons for not wanting to use openssl ?
Hide
Tim Jenness added a comment -

Historically we've had real problems building cmake packages against conda libraries, partly because our build system doesn't know about the conda library location at runtime (so the linker gets confused).

Show
Tim Jenness added a comment - Historically we've had real problems building cmake packages against conda libraries, partly because our build system doesn't know about the conda library location at runtime (so the linker gets confused).
Hide
Fritz Mueller added a comment - - edited

Joshua Hoblitt you are a better man than I if you can derive a patch which convinces MariaDB's cmake to succeed with the openssl conda package instead of failing on lack of GnuTLS

(Though seriously, if you can derive such a patch, I can unblock my group today instead of waiting on this RFC, so I would be quite happy! Branches tickets/DM-12397 of mariadb and mariadbclient, with upgraded upstream tarballs and patches, are at your disposal if you wish to try!)

Show
Fritz Mueller added a comment - - edited Joshua Hoblitt you are a better man than I if you can derive a patch which convinces MariaDB's cmake to succeed with the openssl conda package instead of failing on lack of GnuTLS (Though seriously, if you can derive such a patch, I can unblock my group today instead of waiting on this RFC, so I would be quite happy! Branches tickets/ DM-12397 of mariadb and mariadbclient, with upgraded upstream tarballs and patches, are at your disposal if you wish to try!)
Hide
Tim Jenness added a comment -

I think the problem is that EUPS can't know it needs to add the miniconda lib directory from conda to the DYLD_LIBRARY_PATH (and LSST_LIBRARY_PATH) because table files rely on environment variables and now that miniconda is not an EUPS package you don't have one to use.

Show
Tim Jenness added a comment - I think the problem is that EUPS can't know it needs to add the miniconda lib directory from conda to the DYLD_LIBRARY_PATH (and LSST_LIBRARY_PATH ) because table files rely on environment variables and now that miniconda is not an EUPS package you don't have one to use.
Hide
Tim Jenness added a comment -

Kian-Tat Lim can you comment on this before we adopt it?

Show
Tim Jenness added a comment - Kian-Tat Lim can you comment on this before we adopt it?
Hide
Kian-Tat Lim added a comment - - edited

Aside from the difficulties with eups and cmake, I'd rather not have any C++ code depend on conda-installed packages, as the stack and other products are intended to be usable with non-conda Python installations.

Accordingly, the options appear to be making SSL/TLS a system dependency as requested here and making it an eups package.

Since this is only for macOS, the eups package route looks less desirable.

We currently have only one system dependency for macOS: cmake itself. But having one means that having two is not as difficult.

(In the longer term, it may be possible to remove the need for mariadbclient by increasing usage of SQLAlchemy; the Consolidated Database is unlikely to be MySQL or MariaDB in any case.)

In the end, it seems like the proposal is the best available alternative in the short term.

Show
Kian-Tat Lim added a comment - - edited Aside from the difficulties with eups and cmake , I'd rather not have any C++ code depend on conda -installed packages, as the stack and other products are intended to be usable with non- conda Python installations. Accordingly, the options appear to be making SSL/TLS a system dependency as requested here and making it an eups package. Since this is only for macOS, the eups package route looks less desirable. We currently have only one system dependency for macOS: cmake itself. But having one means that having two is not as difficult. (In the longer term, it may be possible to remove the need for mariadbclient by increasing usage of SQLAlchemy; the Consolidated Database is unlikely to be MySQL or MariaDB in any case.) In the end, it seems like the proposal is the best available alternative in the short term.
Hide
Joshua Hoblitt added a comment -

For the record, all of the mariadb binaries I have checked are linked against openssl or neither.  That includes the vendored RHEL/centos 7 binaries, Fedora 27 binaries, Debian 9.3 (mariadb 10.1 and it isn't linked against either. The dpkg metadata isn't pulling in either as a dep either – possibly using a bundled tls lib.) and the official Mariadb published debian based docker images. The later actually has gnutls installed, as the included version of libcurl is linked against it, but the Mariadb binaries are not linked against it.

 $docker run -ti --entrypoint=/bin/bash mariadb:10.2.13 ldd /usr/sbin/mysqld  linux-vdso.so.1 (0x00007ffe799d1000)  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f117bc96000)  libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f117ba94000)  libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f117b889000)  libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f117b66e000)  libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f117b437000)  libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f117b213000)  libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f117afb2000)  libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f117abb6000)  libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f117a9b2000)  libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f117a6a7000)  libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f117a3a6000)  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1179ffb000)  /lib64/ld-linux-x86-64.so.2 (0x00007f117dc1c000)  librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1179df3000)  liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1179bd0000)  libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f11798ef000)  libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f11796d8000)  libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f11794c2000)  libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f11792b0000)  Show Joshua Hoblitt added a comment - For the record, all of the mariadb binaries I have checked are linked against openssl or neither. That includes the vendored RHEL/centos 7 binaries, Fedora 27 binaries, Debian 9.3 (mariadb 10.1 and it isn't linked against either. The dpkg metadata isn't pulling in either as a dep either – possibly using a bundled tls lib.) and the official Mariadb published debian based docker images. The later actually has gnutls installed, as the included version of libcurl is linked against it, but the Mariadb binaries are not linked against it.$ docker run -ti --entrypoint=/bin/bash mariadb: 10.2 . 13 ldd /usr/sbin/mysqld     linux-vdso.so. 1 ( 0x00007ffe799d1000 )     libpthread.so. 0 => /lib/x86_64-linux-gnu/libpthread.so. 0 ( 0x00007f117bc96000 )     libaio.so. 1 => /lib/x86_64-linux-gnu/libaio.so. 1 ( 0x00007f117ba94000 )     libnuma.so. 1 => /usr/lib/x86_64-linux-gnu/libnuma.so. 1 ( 0x00007f117b889000 )     libz.so. 1 => /lib/x86_64-linux-gnu/libz.so. 1 ( 0x00007f117b66e000 )     libcrypt.so. 1 => /lib/x86_64-linux-gnu/libcrypt.so. 1 ( 0x00007f117b437000 )     libsystemd.so. 0 => /lib/x86_64-linux-gnu/libsystemd.so. 0 ( 0x00007f117b213000 )     libssl.so. 1.0 . 0 => /usr/lib/x86_64-linux-gnu/libssl.so. 1.0 . 0 ( 0x00007f117afb2000 )     libcrypto.so. 1.0 . 0 => /usr/lib/x86_64-linux-gnu/libcrypto.so. 1.0 . 0 ( 0x00007f117abb6000 )     libdl.so. 2 => /lib/x86_64-linux-gnu/libdl.so. 2 ( 0x00007f117a9b2000 )     libstdc++.so. 6 => /usr/lib/x86_64-linux-gnu/libstdc++.so. 6 ( 0x00007f117a6a7000 )     libm.so. 6 => /lib/x86_64-linux-gnu/libm.so. 6 ( 0x00007f117a3a6000 )     libc.so. 6 => /lib/x86_64-linux-gnu/libc.so. 6 ( 0x00007f1179ffb000 )     /lib64/ld-linux-x86- 64 .so. 2 ( 0x00007f117dc1c000 )     librt.so. 1 => /lib/x86_64-linux-gnu/librt.so. 1 ( 0x00007f1179df3000 )     liblzma.so. 5 => /lib/x86_64-linux-gnu/liblzma.so. 5 ( 0x00007f1179bd0000 )     libgcrypt.so. 20 => /lib/x86_64-linux-gnu/libgcrypt.so. 20 ( 0x00007f11798ef000 )     libresolv.so. 2 => /lib/x86_64-linux-gnu/libresolv.so. 2 ( 0x00007f11796d8000 )     libgcc_s.so. 1 => /lib/x86_64-linux-gnu/libgcc_s.so. 1 ( 0x00007f11794c2000 )     libgpg-error.so. 0 => /lib/x86_64-linux-gnu/libgpg-error.so. 0 ( 0x00007f11792b0000 )
Hide
Fritz Mueller added a comment -

Joshua Hoblitt: right!  Not needed at all for Linux-based builds.  Only needed in the MacOS build environment.

Show
Fritz Mueller added a comment - Joshua Hoblitt : right!  Not needed at all for Linux-based builds.  Only  needed in the MacOS build environment.
Hide
Fritz Mueller added a comment -

[Joshua Hoblitt: Again, please do feel free to pull branches tickets/DM-12397 of mariadb and mariadbclient if you would like to see if you can cajole the MacOS build into building either ssl-free or against conda openssl in our build enironment?  I suppose MacOS users would be fine with un-encrypted connections here.  I only need the new MariaDB releases to be buildable in eups without breaking our MacOS users and CI!]

Show
Fritz Mueller added a comment - [ Joshua Hoblitt : Again, please do feel free to pull branches tickets/ DM-12397  of mariadb and mariadbclient if you would like to see if you can cajole the MacOS build into building either ssl-free or against conda openssl in our build enironment?  I suppose MacOS users would be fine with un-encrypted connections here.  I only  need the new MariaDB releases to be buildable in eups without breaking our MacOS users and CI!]
Hide
Joshua Hoblitt added a comment -

The brew bottle for mariadb 10.2.13 is also linked against openssl....

 $otool -L /usr/local/bin/mysqld /usr/local/bin/mysqld:  /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.13.0)  /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)  /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)  /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)  Show Joshua Hoblitt added a comment - The brew bottle for mariadb 10.2.13 is also linked against openssl ....$ otool -L /usr/local/bin/mysqld /usr/local/bin/mysqld: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0 . 0 , current version 728.13 . 0 ) /usr/lib/libbz2. 1.0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 5 ) /usr/lib/libz. 1 .dylib (compatibility version 1.0 . 0 , current version 1.2 . 5 ) /usr/local/opt/openssl/lib/libssl. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/local/opt/openssl/lib/libcrypto. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/lib/libc++. 1 .dylib (compatibility version 1.0 . 0 , current version 120.1 . 0 ) /usr/lib/libSystem.B.dylib (compatibility version 1.0 . 0 , current version 1226.10 . 1 )
Hide
Fritz Mueller added a comment - - edited

Awesome!  All we have to do then is figure out how to do that in the context of eups + miniconda...  Please dig in!

Show
Fritz Mueller added a comment - - edited Awesome!  All we have to do then is figure out how to do that in the context of eups + miniconda...  Please dig in!
Hide
Joshua Hoblitt added a comment -

-WITH_SSL=OFF is silently ignored under 10.2https://jira.mariadb.org/browse/CONC-224 so disablnig TLS isn't an option.

Show
Joshua Hoblitt added a comment - -WITH_SSL=OFF is silently ignored under 10.2 – https://jira.mariadb.org/browse/CONC-224 so disablnig TLS isn't an option.
Hide
Fritz Mueller added a comment -

Yes, sadly – I tried that too.  (It's probably the wrong thing, anyway – all our endpoints, even the internal ones, should really be using TLS.)

Show
Fritz Mueller added a comment - Yes, sadly – I tried that too.  (It's probably the wrong thing, anyway – all our endpoints, even the internal ones, should really be using TLS.)
Hide
Joshua Hoblitt added a comment - - edited

It looks like PKG_CONFIG_LIBDIR needs to be setup for pkg-config. There's several ways this could be accomplished but it will require simultaneous changes to lsstsw and newinstall.sh.

 mac-os-x-el-capitan-build-4:lsstsw square$cat miniconda/etc/conda/activate.d/env_vars.sh  #!/bin/sh   export MINICONDA_ROOT="/Users/square/jhoblitt/lsstsw/miniconda" export PKG_CONFIG_LIBDIR=${LD_LIBRARY_PATH}/pkgconfig # probably not necessary... export LD_LIBRARY_PATH="${MINICONDA_ROOT}/lib" export CPATH="${MINICONDA_ROOT}/include" mac-os-x-el-capitan-build-4:lsstsw square$source activate root (root) mac-os-x-el-capitan-build-4:lsstsw square$  (root) mac-os-x-el-capitan-build-4:lsstsw square$rebuild -r tickets/DM-12397-openssl mariadbclient  mariadbclient: ok (238.8 sec). # BUILD ID: b3447  mariadbclient: tickets.DM-12397-openssl-g993eea619b ok (72.6 sec). # BUILD b3447 completed. (root) mac-os-x-el-capitan-build-4:lsstsw square$ otool -L stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql:  /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)  /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)  /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)  @rpath/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)  @rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1) 

Show
Joshua Hoblitt added a comment - - edited It looks like PKG_CONFIG_LIBDIR needs to be setup for pkg-config . There's several ways this could be accomplished but it will require simultaneous changes to lsstsw and newinstall.sh . mac-os-x-el-capitan-build- 4 :lsstsw square$cat miniconda/etc/conda/activate.d/env_vars.sh #!/bin/sh export MINICONDA_ROOT= "/Users/square/jhoblitt/lsstsw/miniconda" export PKG_CONFIG_LIBDIR=${LD_LIBRARY_PATH}/pkgconfig # probably not necessary... export LD_LIBRARY_PATH= "${MINICONDA_ROOT}/lib" export CPATH= "${MINICONDA_ROOT}/include" mac-os-x-el-capitan-build- 4 :lsstsw square$source activate root (root) mac-os-x-el-capitan-build- 4 :lsstsw square$ (root) mac-os-x-el-capitan-build- 4 :lsstsw square$rebuild -r tickets/DM- 12397 -openssl mariadbclient mariadbclient: ok ( 238.8 sec). # BUILD ID: b3447 mariadbclient: tickets.DM- 12397 -openssl-g993eea619b ok ( 72.6 sec). # BUILD b3447 completed. (root) mac-os-x-el-capitan-build- 4 :lsstsw square$ otool -L stack/DarwinX86/mariadbclient/tickets.DM- 12397 -openssl-g993eea619b/bin/mysql stack/DarwinX86/mariadbclient/tickets.DM- 12397 -openssl-g993eea619b/bin/mysql: /usr/lib/libedit. 3 .dylib (compatibility version 2.0 . 0 , current version 3.0 . 0 ) /usr/lib/libncurses. 5.4 .dylib (compatibility version 5.4 . 0 , current version 5.4 . 0 ) /usr/lib/libiconv. 2 .dylib (compatibility version 7.0 . 0 , current version 7.0 . 0 ) @rpath /libssl. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) @rpath /libcrypto. 1.0 . 0 .dylib (compatibility version 1.0 . 0 , current version 1.0 . 0 ) /usr/lib/libc++. 1 .dylib (compatibility version 1.0 . 0 , current version 120.1 . 0 ) /usr/lib/libSystem.B.dylib (compatibility version 1.0 . 0 , current version 1226.10 . 1 )
Hide
Fritz Mueller added a comment -

Ah – nice detective work there! Are the proposed changes at all controversial, or can we just move ahead with them?

Show
Fritz Mueller added a comment - Ah – nice detective work there! Are the proposed changes at all controversial, or can we just move ahead with them?
Hide
Kian-Tat Lim added a comment -

Can you provide a recipe that works if someone is not using our miniconda (or any other conda-based Python)?

Show
Kian-Tat Lim added a comment - Can you provide a recipe that works if someone is not using our miniconda (or any other conda-based Python)?
Hide
Scott Daniel added a comment -

Sorry to come late to the party.  Is the conclusion of Josh's investigation that there is a way to support Mac users without forcing them to run homebrew?  The sims stack will be affected by this.  It is not obvious that we can jettison our mariadbclient dependence without losing support MySQL database access.  I am trying to figure out what is the best way for us to accommodate this RFC.  Thanks (and, again, sorry for my tardiness).

Show
Scott Daniel added a comment - Sorry to come late to the party.  Is the conclusion of Josh's investigation that there is a way to support Mac users without forcing them to run homebrew?  The sims stack will be affected by this.  It is not obvious that we can jettison our  mariadbclient dependence without losing support MySQL database access.  I am trying to figure out what is the best way for us to accommodate this RFC.  Thanks (and, again, sorry for my tardiness).
Hide
Fritz Mueller added a comment -

Hi Scott – we're trying pretty hard for the lowest-impact solution. The best proposal right now looks to be removing the stack's dependency on mariadbclient. We would then upgrade the mariadb/mariadbclient packages in eups. This will "break" builds of those packages on MacOS, but within LSST-distributed software at that point only Qserv will depend upon those packages, and Qserv is neither expected nor required to be buildable under MacOS.

If sims also requires some sort of mysql client, one option would be to have sims users/devs install it as a sims prereq external to the LSST stack? Thanks for chiming in here – if this is not going to be an acceptable solution we should decide that soon, as we already have a developer devoting effort to breaking the stack/mariadbclient dependency!

Show
Fritz Mueller added a comment - Hi Scott – we're trying pretty hard for the lowest-impact solution. The best proposal right now looks to be removing the stack's dependency on mariadbclient. We would then upgrade the mariadb/mariadbclient packages in eups. This will "break" builds of those packages on MacOS, but within LSST-distributed software at that point only Qserv will depend upon those packages, and Qserv is neither expected nor required to be buildable under MacOS. If sims also requires some sort of mysql client, one option would be to have sims users/devs install it as a sims prereq external to the LSST stack? Thanks for chiming in here – if this is not going to be an acceptable solution we should decide that soon, as we already have a developer devoting effort to breaking the stack/mariadbclient dependency!
Hide
Tim Jenness added a comment -

To clarify. No-one uses the mysql support in daf_persistence so we can remove that and science pipelines would get a faster build. Sims use sqlalchemy and mysqlpython directly so still use mariadbclient. Scott Daniel my understanding is that the fix we have on DM-12397 requires home brew. The problem is that Mac openssl is old and so you need to find a different implementation. Homebrew openssl does not install itself into the normal visible location because they are worried that people would pick up the old openssl by mistake. I don't actually know what macports does. We can't use conda openssl without starting to add conda's lib path to DYLD_LIBRARY_PATH. We could try adding that in loadLSST.xxx to see if that fixes the problem (and might help with iconv problem).

Show
Tim Jenness added a comment - To clarify. No-one uses the mysql support in daf_persistence so we can remove that and science pipelines would get a faster build. Sims use sqlalchemy and mysqlpython directly so still use mariadbclient. Scott Daniel my understanding is that the fix we have on DM-12397 requires home brew. The problem is that Mac openssl is old and so you need to find a different implementation. Homebrew openssl does not install itself into the normal visible location because they are worried that people would pick up the old openssl by mistake. I don't actually know what macports does. We can't use conda openssl without starting to add conda's lib path to DYLD_LIBRARY_PATH . We could try adding that in loadLSST.xxx to see if that fixes the problem (and might help with iconv problem).
Hide
Scott Daniel added a comment - - edited

It is possible that sims doesn't care about MySQL support, either, so we might be able to get rid of our explicit dependence on python_mysqlclient (I am polling our developers, now).

sqlalchemy does not bring in a mariadbclient dependence itself, so we should be fine there (unless sqlalchemy is supposed to depend on mariadbclient or python_mysqlclient, in which case, that is another problem).

obs_lsstSim currently depends on python_myssqlclient.  We need obs_lsstSim.  Is that going to change as a part of the plan to remove }}{{mariadb from DM, or is obs_lsstSim considered to be outside of DM?

Show
Scott Daniel added a comment - - edited It is possible that sims doesn't care about MySQL support, either, so we might be able to get rid of our explicit dependence on python_mysqlclient (I am polling our developers, now). sqlalchemy does not bring in a mariadbclient dependence itself, so we should be fine there (unless sqlalchemy is supposed to depend on mariadbclient or python_mysqlclient , in which case, that is another problem). obs_lsstSim currently depends on python_myssqlclient .  We need obs_lsstSim .  Is that going to change as a part of the plan to remove }}{{mariadb from DM, or is obs_lsstSim considered to be outside of DM?
Hide
Scott Daniel added a comment -

Simon informs me that obs_lsstSim doesn't actually use python_mysqlclient, so we may be safe, there

Show
Scott Daniel added a comment - Simon informs me that obs_lsstSim doesn't actually use python_mysqlclient , so we may be safe, there
Hide
Tim Jenness added a comment - - edited

I see that python_mysqlclient is a dependency for ctrl_stats, obs_sdss, db, obs_lsstSim, scisql, cat and sims_catalogs. Fixing daf_persistence won't help us for those then since they will depend on mariadbclient.

sqlalchemy does a runtime import of database backend. I can't see any code that explicitly asks for mysql backend though (but I may be looking in the wrong place).

obs_lsstSim and obs_sdss use mysqlpython to query lsst-db. No idea if that is important or not or whether sqlalchemy could be used instead.

Show
Tim Jenness added a comment - - edited I see that python_mysqlclient is a dependency for ctrl_stats , obs_sdss , db , obs_lsstSim , scisql , cat and sims_catalogs . Fixing daf_persistence won't help us for those then since they will depend on mariadbclient. sqlalchemy does a runtime import of database backend. I can't see any code that explicitly asks for mysql backend though (but I may be looking in the wrong place). obs_lsstSim and obs_sdss use mysqlpython to query lsst-db. No idea if that is important or not or whether sqlalchemy could be used instead.
Hide
Simon Krughoff added a comment -

I'm not aware of currently used code that uses the SQL DB image selection routines in obs_sdss and obs_lsstSim. The need for these will go away with the Gen3 data abstraction system, so it may be OK to just remove those methods with the expectation that data discovery will change fundamentally in the next few months.

Show
Simon Krughoff added a comment - I'm not aware of currently used code that uses the SQL DB image selection routines in obs_sdss and obs_lsstSim . The need for these will go away with the Gen3 data abstraction system, so it may be OK to just remove those methods with the expectation that data discovery will change fundamentally in the next few months.
Hide
Tim Jenness added a comment -

I should add that cat, scisql and db are not relevant for lsst_distrib (assuming we finally remove datarel which we have wanted to do forever) so the three that matter are the two obs packages and sims_catalogs. Sounds like the obs packages don't really need it at the moment.

Show
Tim Jenness added a comment - I should add that cat , scisql and db are not relevant for lsst_distrib (assuming we finally remove datarel which we have wanted to do forever) so the three that matter are the two obs packages and sims_catalogs . Sounds like the obs packages don't really need it at the moment.
Hide
Tim Jenness added a comment - - edited

We are adjusting the plan that was adopted in this RFC.

• We will document that Gnutls is required for mac users who want to install the mariadb/mariadbclient packages, but we will clean up other dependencies to minimize the impact of this on end users (most users of Qserv and dax servers are linux).
• We will remove the mariadbclient dependency from daf_persistence (DM-13788)
• Scott Daniel will remove the explicit python_mysqlclient dependency on sims_catalogs. It uses sqlalchemy and users can install additional backends in the normal python way as needed.
• We will finally remove datarel from lsst_distrib (DM-2948).
• Simon Krughoff will file an RFC for removal of the database query code from obs_lsstSim and obs_sdss.

Once this is done we believe that neither sims nor lsst_distrib will require mariadbclient.

Show
Tim Jenness added a comment - - edited We are adjusting the plan that was adopted in this RFC. We will document that Gnutls is required for mac users who want to install the mariadb/mariadbclient packages, but we will clean up other dependencies to minimize the impact of this on end users (most users of Qserv and dax servers are linux). We will remove the mariadbclient dependency from daf_persistence ( DM-13788 ) Scott Daniel will remove the explicit python_mysqlclient dependency on sims_catalogs . It uses sqlalchemy and users can install additional backends in the normal python way as needed. We will finally remove datarel from lsst_distrib ( DM-2948 ). Simon Krughoff will file an RFC for removal of the database query code from obs_lsstSim and obs_sdss. Once this is done we believe that neither sims nor lsst_distrib will require mariadbclient.
Hide
Joshua Hoblitt added a comment -

Documenting that lib X is required is not sufficient, it must be present at the exact same path as the eups tarball builds were linked against.

Show
Joshua Hoblitt added a comment - Documenting that lib X is required is not sufficient, it must be present at the exact same path as the eups tarball builds were linked against.
Hide
Tim Jenness added a comment -

For the few people on a mac using the binary eups releases of mariadbclient I have no problem stating that they must use home brew. This will only affect people using dax or qserv on a mac and that means only a few people once we remove the stated dependencies on lsst_Distrib.

Show
Tim Jenness added a comment - For the few people on a mac using the binary eups releases of mariadbclient I have no problem stating that they must use home brew. This will only affect people using dax or qserv on a mac and that means only a few people once we remove the stated dependencies on lsst_Distrib.
Hide
Fritz Mueller added a comment -

(All the DAX and Qserv devs who do any work outside vms are homebrew users, in any case)

Show
Fritz Mueller added a comment - (All the DAX and Qserv devs who do any work outside vms are homebrew users, in any case)
Hide
Tim Jenness added a comment -

Fritz Mueller it looks like lsst_distrib and lsst_sims no longer need mariadbclient. I think that means you are cleared to update so long as mariadb builds on the jenkins instances and you update the docs to say that the few people building the eups package of mariadb[client] need to do something extra on Mac.

Show
Tim Jenness added a comment - Fritz Mueller it looks like lsst_distrib and lsst_sims no longer need mariadbclient. I think that means you are cleared to update so long as mariadb builds on the jenkins instances and you update the docs to say that the few people building the eups package of mariadb [client] need to do something extra on Mac.
Hide
Tim Jenness added a comment -

All triggered work has been completed but I'm unhappy that we did not document the macOS GnuTLS requirement on Qserv. DM-13636 was closed as INVALID.

Show
Tim Jenness added a comment - All triggered work has been completed but I'm unhappy that we did not document the macOS GnuTLS requirement on Qserv. DM-13636 was closed as INVALID.
Hide
Tim Jenness added a comment -

Fritz Mueller should we be documenting this or should we say that we aren't even trying to build Qserv on macOS? (although in RFC-521 there is an attempt to bring mariadb back into the Mac Jenkins builds).

Show
Tim Jenness added a comment - Fritz Mueller should we be documenting this or should we say that we aren't even trying to build Qserv on macOS? (although in RFC-521 there is an attempt to bring mariadb back into the Mac Jenkins builds).
Hide
Fritz Mueller added a comment -

I got tired of working the fine points of this, and am quite satisfied to claim that we will no longer even try to build Qserv on macOS (where it is, after all, useless for any practical purpose anyway).

The rest of the stack may have to take this back up again if mariadb re-enters the stack, but that will be between the stack developers and the CI maintainers and need not involve Qserv.

Show
Fritz Mueller added a comment - I got tired of working the fine points of this, and am quite satisfied to claim that we will no longer even try to build Qserv on macOS (where it is, after all, useless for any practical purpose anyway). The rest of the stack may have to take this back up again if mariadb re-enters the stack, but that will be between the stack developers and the CI maintainers and need not involve Qserv.
Hide
Wil O'Mullane added a comment -

This was not implemented in the way intended -  though the  end result was acceptable.

Show
Wil O'Mullane added a comment - This was not implemented in the way intended -  though the  end result was acceptable.

#### People

Assignee:
Fritz Mueller
Reporter:
Fritz Mueller
Watchers:
Fritz Mueller, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness, Wil O'Mullane
0 Vote for this issue
Watchers:
8 Start watching this issue

#### Dates

Created:
Updated:
Resolved:
Planned End:

#### Jenkins

No builds found.