In case it's not clear, this has an impact on the ability to build lsst_apps because the mariadbclient eups package will need to be updated and that is used by daf_persistence.

We are already installing openssl 1.0.x as a conda package. I've searched through the mariadb docs and don't see any mention of gnutls being preferred over openssl. What are the specific reasons for not wanting to use openssl?

Historically we've had real problems building cmake packages against conda libraries, partly because our build system doesn't know about the conda library location at runtime (so the linker gets confused).

Joshua Hoblitt you are a better man than I if you can derive a patch which convinces MariaDB's cmake to succeed with the openssl conda package instead of failing on lack of GnuTLS

(Though seriously, if you can derive such a patch, I can unblock my group today instead of waiting on this RFC, so I would be quite happy! Branches tickets/DM-12397 of mariadb and mariadbclient, with upgraded upstream tarballs and patches, are at your disposal if you wish to try!)

I think the problem is that EUPS can't know it needs to add the miniconda lib directory from conda to the DYLD_LIBRARY_PATH (and LSST_LIBRARY_PATH) because table files rely on environment variables and now that miniconda is not an EUPS package you don't have one to use.

Kian-Tat Lim can you comment on this before we adopt it?

Aside from the difficulties with eups and cmake, I'd rather not have any C++ code depend on conda-installed packages, as the stack and other products are intended to be usable with non-conda Python installations.

Accordingly, the options appear to be making SSL/TLS a system dependency as requested here and making it an eups package.

Since this is only for macOS, the eups package route looks less desirable.

We currently have only one system dependency for macOS: cmake itself. But having one means that having two is not as difficult.

(In the longer term, it may be possible to remove the need for mariadbclient by increasing usage of SQLAlchemy; the Consolidated Database is unlikely to be MySQL or MariaDB in any case.)

In the end, it seems like the proposal is the best available alternative in the short term.

For the record, all of the mariadb binaries I have checked are linked against openssl or neither.  That includes the vendored RHEL/centos 7 binaries, Fedora 27 binaries, Debian 9.3 (mariadb 10.1 and it isn't linked against either. The dpkg metadata isn't pulling in either as a dep either – possibly using a bundled tls lib.) and the official Mariadb published debian based docker images. The later actually has gnutls installed, as the included version of libcurl is linked against it, but the Mariadb binaries are not linked against it.

$ docker run -ti --entrypoint=/bin/bash mariadb:10.2.13 ldd /usr/sbin/mysqld

    linux-vdso.so.1 (0x00007ffe799d1000)

    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f117bc96000)

    libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f117ba94000)

    libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f117b889000)

    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f117b66e000)

    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f117b437000)

    libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f117b213000)

    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f117afb2000)

    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f117abb6000)

    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f117a9b2000)

    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f117a6a7000)

    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f117a3a6000)

    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1179ffb000)

    /lib64/ld-linux-x86-64.so.2 (0x00007f117dc1c000)

    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1179df3000)

    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f1179bd0000)

    libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f11798ef000)

    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f11796d8000)

    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f11794c2000)

    libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f11792b0000)

Joshua Hoblitt: right!  Not needed at all for Linux-based builds.  Only needed in the MacOS build environment.

[Joshua Hoblitt: Again, please do feel free to pull branches tickets/DM-12397 of mariadb and mariadbclient if you would like to see if you can cajole the MacOS build into building either ssl-free or against conda openssl in our build enironment?  I suppose MacOS users would be fine with un-encrypted connections here.  I only need the new MariaDB releases to be buildable in eups without breaking our MacOS users and CI!]

The brew bottle for mariadb 10.2.13 is also linked against openssl....

$ otool -L /usr/local/bin/mysqld

/usr/local/bin/mysqld:

	/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 728.13.0)

	/usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)

	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)

	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Awesome!  All we have to do then is figure out how to do that in the context of eups + miniconda...  Please dig in!

-WITH_SSL=OFF is silently ignored under 10.2 – https://jira.mariadb.org/browse/CONC-224 so disablnig TLS isn't an option.

Yes, sadly – I tried that too.  (It's probably the wrong thing, anyway – all our endpoints, even the internal ones, should really be using TLS.)

It looks like PKG_CONFIG_LIBDIR needs to be setup for pkg-config. There's several ways this could be accomplished but it will require simultaneous changes to lsstsw and newinstall.sh.

mac-os-x-el-capitan-build-4:lsstsw square$ cat miniconda/etc/conda/activate.d/env_vars.sh 

#!/bin/sh

export MINICONDA_ROOT="/Users/square/jhoblitt/lsstsw/miniconda"

export PKG_CONFIG_LIBDIR=${LD_LIBRARY_PATH}/pkgconfig

# probably not necessary...

export LD_LIBRARY_PATH="${MINICONDA_ROOT}/lib"

export CPATH="${MINICONDA_ROOT}/include"

mac-os-x-el-capitan-build-4:lsstsw square$ source activate root

(root) mac-os-x-el-capitan-build-4:lsstsw square$ 

(root) mac-os-x-el-capitan-build-4:lsstsw square$ rebuild -r tickets/DM-12397-openssl mariadbclient

       mariadbclient:  ok (238.8 sec).

# BUILD ID: b3447

       mariadbclient: tickets.DM-12397-openssl-g993eea619b ok (72.6 sec).

# BUILD b3447 completed.

(root) mac-os-x-el-capitan-build-4:lsstsw square$ otool -L stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql

stack/DarwinX86/mariadbclient/tickets.DM-12397-openssl-g993eea619b/bin/mysql:

	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)

	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

	@rpath/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

	@rpath/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)

	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Ah – nice detective work there! Are the proposed changes at all controversial, or can we just move ahead with them?

Can you provide a recipe that works if someone is not using our miniconda (or any other conda-based Python)?

Sorry to come late to the party.  Is the conclusion of Josh's investigation that there is a way to support Mac users without forcing them to run homebrew?  The sims stack will be affected by this.  It is not obvious that we can jettison our mariadbclient dependence without losing support MySQL database access.  I am trying to figure out what is the best way for us to accommodate this RFC.  Thanks (and, again, sorry for my tardiness).

Hi Scott – we're trying pretty hard for the lowest-impact solution. The best proposal right now looks to be removing the stack's dependency on mariadbclient. We would then upgrade the mariadb/mariadbclient packages in eups. This will "break" builds of those packages on MacOS, but within LSST-distributed software at that point only Qserv will depend upon those packages, and Qserv is neither expected nor required to be buildable under MacOS.

If sims also requires some sort of mysql client, one option would be to have sims users/devs install it as a sims prereq external to the LSST stack? Thanks for chiming in here – if this is not going to be an acceptable solution we should decide that soon, as we already have a developer devoting effort to breaking the stack/mariadbclient dependency!

To clarify. No-one uses the mysql support in daf_persistence so we can remove that and science pipelines would get a faster build. Sims use sqlalchemy and mysqlpython directly so still use mariadbclient. Scott Daniel my understanding is that the fix we have on DM-12397 requires home brew. The problem is that Mac openssl is old and so you need to find a different implementation. Homebrew openssl does not install itself into the normal visible location because they are worried that people would pick up the old openssl by mistake. I don't actually know what macports does. We can't use conda openssl without starting to add conda's lib path to DYLD_LIBRARY_PATH. We could try adding that in loadLSST.xxx to see if that fixes the problem (and might help with iconv problem).

It is possible that sims doesn't care about MySQL support, either, so we might be able to get rid of our explicit dependence on `python_mysqlclient` (I am polling our developers, now).

sqlalchemy does not bring in a mariadbclient dependence itself, so we should be fine there (unless sqlalchemy is supposed to depend on mariadbclient or python_mysqlclient, in which case, that is another problem).

obs_lsstSim currently depends on python_myssqlclient.  We need obs_lsstSim.  Is that going to change as a part of the plan to remove }}{{mariadb from DM, or is obs_lsstSim considered to be outside of DM?

Simon informs me that obs_lsstSim doesn't actually use python_mysqlclient, so we may be safe, there

I see that python_mysqlclient is a dependency for ctrl_stats, obs_sdss, db, obs_lsstSim, scisql, cat and sims_catalogs. Fixing daf_persistence won't help us for those then since they will depend on mariadbclient.

sqlalchemy does a runtime import of database backend. I can't see any code that explicitly asks for mysql backend though (but I may be looking in the wrong place).

obs_lsstSim and obs_sdss use mysqlpython to query lsst-db. No idea if that is important or not or whether sqlalchemy could be used instead.

I'm not aware of currently used code that uses the SQL DB image selection routines in obs_sdss and obs_lsstSim. The need for these will go away with the Gen3 data abstraction system, so it may be OK to just remove those methods with the expectation that data discovery will change fundamentally in the next few months.

I should add that cat, scisql and db are not relevant for lsst_distrib (assuming we finally remove datarel which we have wanted to do forever) so the three that matter are the two obs packages and sims_catalogs. Sounds like the obs packages don't really need it at the moment.

We are adjusting the plan that was adopted in this RFC.

• We will document that Gnutls is required for mac users who want to install the mariadb/mariadbclient packages, but we will clean up other dependencies to minimize the impact of this on end users (most users of Qserv and dax servers are linux).
• We will remove the mariadbclient dependency from daf_persistence (DM-13788)
• Scott Daniel will remove the explicit python_mysqlclient dependency on sims_catalogs. It uses sqlalchemy and users can install additional backends in the normal python way as needed.
• We will finally remove datarel from lsst_distrib (DM-2948).
• Simon Krughoff will file an RFC for removal of the database query code from obs_lsstSim and obs_sdss.

Once this is done we believe that neither sims nor lsst_distrib will require mariadbclient.

Documenting that lib X is required is not sufficient, it must be present at the exact same path as the eups tarball builds were linked against.

For the few people on a mac using the binary eups releases of mariadbclient I have no problem stating that they must use home brew. This will only affect people using dax or qserv on a mac and that means only a few people once we remove the stated dependencies on lsst_Distrib.

(All the DAX and Qserv devs who do any work outside vms are homebrew users, in any case)

Fritz Mueller it looks like lsst_distrib and lsst_sims no longer need mariadbclient. I think that means you are cleared to update so long as mariadb builds on the jenkins instances and you update the docs to say that the few people building the eups package of mariadb[client] need to do something extra on Mac.

All triggered work has been completed but I'm unhappy that we did not document the macOS GnuTLS requirement on Qserv. DM-13636 was closed as INVALID.

Fritz Mueller should we be documenting this or should we say that we aren't even trying to build Qserv on macOS? (although in RFC-521 there is an attempt to bring mariadb back into the Mac Jenkins builds).

I got tired of working the fine points of this, and am quite satisfied to claim that we will no longer even try to build Qserv on macOS (where it is, after all, useless for any practical purpose anyway).

The rest of the stack may have to take this back up again if mariadb re-enters the stack, but that will be between the stack developers and the CI maintainers and need not involve Qserv.

This was not implemented in the way intended -  though the  end result was acceptable.