It is not so clear that server-side tools like webserv and the like must not depend on anaconda.

In this case, I believe this is a substitute for requiring the packaging (or specification as system dependencies) of several other Python packages.

Mario, we are in the process of moving away from anaconda, but it will require packaging few things like sqlalchemy and flask dependencies (and nobody likes dealing with that in eups...). But as K-T said, this is server side and we should relax the rules a bit.

p.s. please don't just push your change to master until we package appropriate things, or you will break webserv.

OK, I see.

That's somewhat unusual (I think), because for other modules that appear anaconda I don't think we've made explicit dependencies on the anaconda EUPS product (e.g., python itself, numpy, matplotlib, etc., though there are dummy packages (that are mostly a relic of the past)). Is there a reason you need an explicit dependency on anaconda (e.g., would builds fail, etc.)?

The main reason, as far as I know, is because we need to add 4 libraries to eups in order to get this to build (sqlalchemy, flask, jinja2, werkzeug), but those all exist in anaconda. I think Fabrice worked on it a bit but had a few issues making sure all the libraries would install fine. I believe there was a decision made that, at least in the mean time, it was much simpler to just depend on anaconda.

That's fine – I'm just asking whether you need to make it explicitly depend on the anaconda EUPS product (or implicitly require the user to have it (or to supply the libraries otherwise))?

Because once you depend on the anaconda EUPS package, if you want to use your own custom Anaconda you need to do some pretty nasty EUPS magic (the manifest.remap stuff).

I'm a bit late to this ticket but I agree with Mario Juric that packages should not depend on anaconda. We want to do CI run with different python builds. Per RFC-52, we should not be punting on dependency management.

Sorry, I should have referenced RFC-52. Corrected the previous comment.

After the merge of DM-4347, qserv is the last package [that I'm aware of] that has a dep on anaconda.

https://github.com/lsst/qserv/pull/186

passes a jenkins build

https://ci.lsst.codes/job/qserv-os-matrix/3889/

I've opened a PR on qserv_testdata to add the missing dep on pyyaml. It passes jenkins:

https://ci.lsst.codes/job/qserv-os-matrix/3913/

I'm going to self merge as this is a trivial fix.

My understanding from Frossie Economou is that qserv team is OK with considering requests a "system dep", in the context for qserv_distrib, is that correct?

If so, there's a couple of different ways of fulfilling that. I suspect that eventually, the best course will be some sort of virtualenv-ish setup that makes it easy to add new python deps. In the mean time, I suggest adding something along the lines of apt-get install -qq -y python python-dev python-setuptools python-yaml python-requests python-sqlalchemy to the qserv docker files. Fabrice Jammes, do you have any thoughts on how python deps should be handled?

@fritz requested on HC that the version of the requests package be updated and that it be added to the qserv eups table file in place of anaconda.

The requests product has been updated to 2.9.1 and added to repos.yaml. The PR has been updated to add requests as a dep and CI passes:

https://ci.lsst.codes/job/qserv-os-matrix/3944/

merged

Hi Joshua Hoblitt,

It's difficult to be sure that all "supported" distribution will provide correct version of our python deps, on the other hand, using eups might lead to use of deprecated and non supported deps. virtualenv might not be compliant with eups... So for now I'll stick with the current way we manage dependencies, but I think in production we'll need to use systems deps or at least up-to-date deps.

Cheers,