Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2993

Products must not depend on anaconda

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: webservcommon
    • Labels:
      None

      Description

      setupRequired(anaconda) should be removed from webservcommon.table.

      We want to keep the stack buildable with any python 2.7, and should not explicitly depend on anaconda.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            It is not so clear that server-side tools like webserv and the like must not depend on anaconda.

            In this case, I believe this is a substitute for requiring the packaging (or specification as system dependencies) of several other Python packages.

            Show
            ktl Kian-Tat Lim added a comment - It is not so clear that server-side tools like webserv and the like must not depend on anaconda . In this case, I believe this is a substitute for requiring the packaging (or specification as system dependencies) of several other Python packages.
            Hide
            jbecla Jacek Becla added a comment -

            Mario, we are in the process of moving away from anaconda, but it will require packaging few things like sqlalchemy and flask dependencies (and nobody likes dealing with that in eups...). But as K-T said, this is server side and we should relax the rules a bit.

            Show
            jbecla Jacek Becla added a comment - Mario, we are in the process of moving away from anaconda, but it will require packaging few things like sqlalchemy and flask dependencies (and nobody likes dealing with that in eups...). But as K-T said, this is server side and we should relax the rules a bit.
            Hide
            jbecla Jacek Becla added a comment -

            p.s. please don't just push your change to master until we package appropriate things, or you will break webserv.

            Show
            jbecla Jacek Becla added a comment - p.s. please don't just push your change to master until we package appropriate things, or you will break webserv.
            Hide
            mjuric Mario Juric added a comment -

            OK, I see.

            That's somewhat unusual (I think), because for other modules that appear anaconda I don't think we've made explicit dependencies on the anaconda EUPS product (e.g., python itself, numpy, matplotlib, etc., though there are dummy packages (that are mostly a relic of the past)). Is there a reason you need an explicit dependency on anaconda (e.g., would builds fail, etc.)?

            Show
            mjuric Mario Juric added a comment - OK, I see. That's somewhat unusual (I think), because for other modules that appear anaconda I don't think we've made explicit dependencies on the anaconda EUPS product (e.g., python itself, numpy, matplotlib, etc., though there are dummy packages (that are mostly a relic of the past)). Is there a reason you need an explicit dependency on anaconda (e.g., would builds fail, etc.)?
            Hide
            bvan Brian Van Klaveren added a comment -

            The main reason, as far as I know, is because we need to add 4 libraries to eups in order to get this to build (sqlalchemy, flask, jinja2, werkzeug), but those all exist in anaconda. I think Fabrice worked on it a bit but had a few issues making sure all the libraries would install fine. I believe there was a decision made that, at least in the mean time, it was much simpler to just depend on anaconda.

            Show
            bvan Brian Van Klaveren added a comment - The main reason, as far as I know, is because we need to add 4 libraries to eups in order to get this to build (sqlalchemy, flask, jinja2, werkzeug), but those all exist in anaconda. I think Fabrice worked on it a bit but had a few issues making sure all the libraries would install fine. I believe there was a decision made that, at least in the mean time, it was much simpler to just depend on anaconda.
            Hide
            mjuric Mario Juric added a comment -

            That's fine – I'm just asking whether you need to make it explicitly depend on the anaconda EUPS product (or implicitly require the user to have it (or to supply the libraries otherwise))?

            Because once you depend on the anaconda EUPS package, if you want to use your own custom Anaconda you need to do some pretty nasty EUPS magic (the manifest.remap stuff).

            Show
            mjuric Mario Juric added a comment - That's fine – I'm just asking whether you need to make it explicitly depend on the anaconda EUPS product (or implicitly require the user to have it (or to supply the libraries otherwise))? Because once you depend on the anaconda EUPS package, if you want to use your own custom Anaconda you need to do some pretty nasty EUPS magic (the manifest.remap stuff).
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            I'm a bit late to this ticket but I agree with Mario Juric that packages should not depend on anaconda. We want to do CI run with different python builds. Per RFC-52, we should not be punting on dependency management.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited I'm a bit late to this ticket but I agree with Mario Juric that packages should not depend on anaconda. We want to do CI run with different python builds. Per RFC-52 , we should not be punting on dependency management.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Sorry, I should have referenced RFC-52. Corrected the previous comment.

            Show
            jhoblitt Joshua Hoblitt added a comment - Sorry, I should have referenced RFC-52 . Corrected the previous comment.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            After the merge of DM-4347, qserv is the last package [that I'm aware of] that has a dep on anaconda.

            Show
            jhoblitt Joshua Hoblitt added a comment - After the merge of DM-4347 , qserv is the last package [that I'm aware of] that has a dep on anaconda.
            Show
            jhoblitt Joshua Hoblitt added a comment - https://github.com/lsst/qserv/pull/186 passes a jenkins build https://ci.lsst.codes/job/qserv-os-matrix/3889/
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've opened a PR on qserv_testdata to add the missing dep on pyyaml. It passes jenkins:

            https://ci.lsst.codes/job/qserv-os-matrix/3913/

            I'm going to self merge as this is a trivial fix.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've opened a PR on qserv_testdata to add the missing dep on pyyaml . It passes jenkins: https://ci.lsst.codes/job/qserv-os-matrix/3913/ I'm going to self merge as this is a trivial fix.
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            My understanding from Frossie Economou is that qserv team is OK with considering requests a "system dep", in the context for qserv_distrib, is that correct?

            If so, there's a couple of different ways of fulfilling that. I suspect that eventually, the best course will be some sort of virtualenv-ish setup that makes it easy to add new python deps. In the mean time, I suggest adding something along the lines of apt-get install -qq -y python python-dev python-setuptools python-yaml python-requests python-sqlalchemy to the qserv docker files. Fabrice Jammes, do you have any thoughts on how python deps should be handled?

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited My understanding from Frossie Economou is that qserv team is OK with considering requests a "system dep", in the context for qserv_distrib, is that correct? If so, there's a couple of different ways of fulfilling that. I suspect that eventually, the best course will be some sort of virtualenv-ish setup that makes it easy to add new python deps. In the mean time, I suggest adding something along the lines of apt-get install -qq -y python python-dev python-setuptools python-yaml python-requests python-sqlalchemy to the qserv docker files. Fabrice Jammes , do you have any thoughts on how python deps should be handled?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            @fritz requested on HC that the version of the requests package be updated and that it be added to the qserv eups table file in place of anaconda.

            Show
            jhoblitt Joshua Hoblitt added a comment - @fritz requested on HC that the version of the requests package be updated and that it be added to the qserv eups table file in place of anaconda.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The requests product has been updated to 2.9.1 and added to repos.yaml. The PR has been updated to add requests as a dep and CI passes:

            https://ci.lsst.codes/job/qserv-os-matrix/3944/

            Show
            jhoblitt Joshua Hoblitt added a comment - The requests product has been updated to 2.9.1 and added to repos.yaml . The PR has been updated to add requests as a dep and CI passes: https://ci.lsst.codes/job/qserv-os-matrix/3944/
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            merged

            Show
            jhoblitt Joshua Hoblitt added a comment - merged
            Hide
            jammes Fabrice Jammes added a comment -

            Hi Joshua Hoblitt,

            It's difficult to be sure that all "supported" distribution will provide correct version of our python deps, on the other hand, using eups might lead to use of deprecated and non supported deps. virtualenv might not be compliant with eups... So for now I'll stick with the current way we manage dependencies, but I think in production we'll need to use systems deps or at least up-to-date deps.

            Cheers,

            Show
            jammes Fabrice Jammes added a comment - Hi Joshua Hoblitt , It's difficult to be sure that all "supported" distribution will provide correct version of our python deps, on the other hand, using eups might lead to use of deprecated and non supported deps. virtualenv might not be compliant with eups... So for now I'll stick with the current way we manage dependencies, but I think in production we'll need to use systems deps or at least up-to-date deps. Cheers,

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              mjuric Mario Juric
              Reviewers:
              Fritz Mueller
              Watchers:
              Brian Van Klaveren, Fabrice Jammes, Fritz Mueller, Jacek Becla, Joshua Hoblitt, Kian-Tat Lim, Mario Juric
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: