Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-679

Switch to conda-forge for all conda third parties and use packaged conda compilers

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      In order to better deal with third party packages, both in conda today and those we repackage in eups, we plan to move to the conda-forge channel as the provider for third party packages, moving most third party packages from eups into conda-forge.

      This RFC does not fully implement changes described in community.lsst.org or the draft tech note regarding development environment, but it is a step towards that.

      In order to ensure compatibility with those packages provided in the conda-forge channel, we also plan to switch to the bundled compilers available in that channel.

      Currently, lsst_distrib builds just fine on macOS and linux with these changes, as tested out in DM-23835. We plan on importing this into Jenkins and building eups package binaries as well as testing out with the other CI jobs in there.

      Some small changes in sconsUtils were required to enable compiling/linking to third party libraries for C++ code, and a small change also was required to work smoothly with the conda-forge provided compiler.

      Some small changes to unit tests were required to deal with the case handling for a newer cfitsio, regarding the handling of PropertySet. We expect some modifications of PropertySet to better support the stricter standard of FITS and CFITSIO in the future, though it is not necessary to implement this RFC.

      An effort to pin a few of the third party packages (e.g. astropy, numpy, scipy, etc...) to be similar to the current baseline will be enabled for the initial implementation of this RFC.

      One third party which is required to also be in eups for the foreseeable future is eigen - to support jointcal. We will have two eigens until either a 3.4 release of eigen is made or the conda-forge maintainers for the eigen package accept patches that are in the unreleased master branch of the eigen repository. In testing, this did not have an impact to building lsst_distrib/jointcal. We hope to test the other CI jobs.

      In addition to removing the need to install a compiler, this change can also reduce or eliminate the need for system installed libraries or utilities. As an example, the only required dependency to bootstrap a bare centos 7 docker container is to install git, then run the instructions for installing lsstsw.

      It is possible to do a test right now. The following code should work on mac or linux. It is recommended such a test be done in a sandbox, but do note this would trigger git lfs downloads:

      git clone https://github.com/lsst/lsstsw.git
      cd lsstsw
      ./bin/deploy -r tickets/DM-23835
      source bin/setup.sh
      export SCONSUTILS_USE_CONDA_COMPILERS=1
      . bin/envconfig -n lsst-scipipe-tickets-DM-23835.726a682
      rebuild -r tickets/DM-23835 lsst_distrib
      

      The export of the SCONSUTILS_USE_CONDA_COMPILERS environment variable will likely be rendered unnecessary in the near future, but it is required for the this example.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bvan Brian Van Klaveren
                Reporter:
                bvan Brian Van Klaveren
                Watchers:
                Brian Van Klaveren, Fabio Hernandez, Gabriele Comoretto, John Parejko, John Swinbank, Jonathan Sick, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel