Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-679

Switch to conda-forge for all conda third parties and use packaged conda compilers

    XMLWordPrintable

Details

    • RFC
    • Status: Implemented
    • Resolution: Done
    • DM
    • None

    Description

      In order to better deal with third party packages, both in conda today and those we repackage in eups, we plan to move to the conda-forge channel as the provider for third party packages, moving most third party packages from eups into conda-forge.

      This RFC does not fully implement changes described in community.lsst.org or the draft tech note regarding development environment, but it is a step towards that.

      In order to ensure compatibility with those packages provided in the conda-forge channel, we also plan to switch to the bundled compilers available in that channel.

      Currently, lsst_distrib builds just fine on macOS and linux with these changes, as tested out in DM-23835. We plan on importing this into Jenkins and building eups package binaries as well as testing out with the other CI jobs in there.

      Some small changes in sconsUtils were required to enable compiling/linking to third party libraries for C++ code, and a small change also was required to work smoothly with the conda-forge provided compiler.

      Some small changes to unit tests were required to deal with the case handling for a newer cfitsio, regarding the handling of PropertySet. We expect some modifications of PropertySet to better support the stricter standard of FITS and CFITSIO in the future, though it is not necessary to implement this RFC.

      An effort to pin a few of the third party packages (e.g. astropy, numpy, scipy, etc...) to be similar to the current baseline will be enabled for the initial implementation of this RFC.

      One third party which is required to also be in eups for the foreseeable future is eigen - to support jointcal. We will have two eigens until either a 3.4 release of eigen is made or the conda-forge maintainers for the eigen package accept patches that are in the unreleased master branch of the eigen repository. In testing, this did not have an impact to building lsst_distrib/jointcal. We hope to test the other CI jobs.

      In addition to removing the need to install a compiler, this change can also reduce or eliminate the need for system installed libraries or utilities. As an example, the only required dependency to bootstrap a bare centos 7 docker container is to install git, then run the instructions for installing lsstsw.

      It is possible to do a test right now. The following code should work on mac or linux. It is recommended such a test be done in a sandbox, but do note this would trigger git lfs downloads:

      git clone https://github.com/lsst/lsstsw.git
      cd lsstsw
      ./bin/deploy -r tickets/DM-23835
      source bin/setup.sh
      export SCONSUTILS_USE_CONDA_COMPILERS=1
      . bin/envconfig -n lsst-scipipe-tickets-DM-23835.726a682
      rebuild -r tickets/DM-23835 lsst_distrib
      

      The export of the SCONSUTILS_USE_CONDA_COMPILERS environment variable will likely be rendered unnecessary in the near future, but it is required for the this example.

      Attachments

        Issue Links

          Activity

            jsick Jonathan Sick added a comment - Links to posts/documents mentioned in the RFC: Community.lsst.org post: https://community.lsst.org/t/using-conda-compilers-and-conda-forge-third-parties/3950 Technote: https://dmtn-138.lsst.io/v/u-bvan-v1/index.html
            price Paul Price added a comment -

            How do we test new versions (or patches) of third-party packages using this system?

            price Paul Price added a comment - How do we test new versions (or patches) of third-party packages using this system?
            bvan Brian Van Klaveren added a comment - - edited

            There’s a few scenarios there.

            In general, conda-forge has many libraries and many versions of libraries. In the case that the library you want to test already exists, you could test locally by just installing the version you want. Officially, you would create a new conda environment in the scipipe_conda_env repo and test that out, like the instructions above.

            If it’s a version not available in conda-forge, we would encourage a PR to the conda-forge feedstock repo. The community is responsive. This would be followed by the steps just mentioned above.

            If it’s a patch, those can be submitted as PRs too. We will also provide instructions to perform a local build (with patching) of a conda recipe and how to install it, although most of those instructions are available already on the conda-forge docs.

            If a patch is not accepted, we will maintain an extra conda channel for a forked version. We haven't started this yet.

            If those all fail, we can package the third party in eups as a fallback, as we will do with eigen for the time being. We are doing this with eigen right now, but that’s mostly because it is already in eups.

            In our experience, the last two haven’t been necessary. The issue with eigen is unique and not representative.

            bvan Brian Van Klaveren added a comment - - edited There’s a few scenarios there. In general, conda-forge has many libraries and many versions of libraries. In the case that the library you want to test already exists, you could test locally by just installing the version you want. Officially, you would create a new conda environment in the scipipe_conda_env repo and test that out, like the instructions above. If it’s a version not available in conda-forge, we would encourage a PR to the conda-forge feedstock repo. The community is responsive. This would be followed by the steps just mentioned above. If it’s a patch, those can be submitted as PRs too. We will also provide instructions to perform a local build (with patching) of a conda recipe and how to install it, although most of those instructions are available already on the conda-forge docs. If a patch is not accepted, we will maintain an extra conda channel for a forked version. We haven't started this yet. If those all fail, we can package the third party in eups as a fallback, as we will do with eigen for the time being. We are doing this with eigen right now, but that’s mostly because it is already in eups. In our experience, the last two haven’t been necessary. The issue with eigen is unique and not representative.
            tjenness Tim Jenness added a comment -

            pytest-flake8 is more worrying since the PR we made a year ago has been ignored completely.

            tjenness Tim Jenness added a comment - pytest-flake8 is more worrying since the PR we made a year ago has been ignored completely.

            This RFC close a bit over a week ago without much comments.

            While we are still testing tarballs from a test release and haven't rebased our branches in the last two weeks, we are confident in what we see so far and we think there are some good improvements in build time thanks to these changes.

            There are two triggered issues, one to provide documentation on testing third parties (and patches), and another to perform the switchover.

            bvan Brian Van Klaveren added a comment - This RFC close a bit over a week ago without much comments. While we are still testing tarballs from a test release and haven't rebased our branches in the last two weeks, we are confident in what we see so far and we think there are some good improvements in build time thanks to these changes. There are two triggered issues, one to provide documentation on testing third parties (and patches), and another to perform the switchover.

            Assuming this switch to conda-forge would also apply to installations done via newinstall.sh, is there already a way to test your proposal or should I wait a bit ?

            I'm asking because for distributing lsst_distrib via cvmfs we currently build (from sources) using newinstall.sh. So, if that helps, I volunteer to test when you think it is the right time, hopefully before the decision to switch is taken.

            FabioHernandez Fabio Hernandez added a comment - Assuming this switch to conda-forge would also apply to installations done via newinstall.sh,  is there already a way to test your proposal or should I wait a bit ? I'm asking because for distributing lsst_distrib via cvmfs we currently build (from sources) using newinstall.sh . So, if that helps, I volunteer to test when you think it is the right time, hopefully before the decision to switch is taken.

            People

              bvan Brian Van Klaveren
              bvan Brian Van Klaveren
              Brian Van Klaveren, Fabio Hernandez, Gabriele Comoretto [X] (Inactive), John Parejko, John Swinbank, Jonathan Sick, Paul Price, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                Jenkins

                  No builds found.