Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23835

Update environment with conda-forge third parties and conda compilers

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • lsstsw
    • None
    • Architecture
    • No

    Description

      Update environment and dependencies for the stack, and include conda compilers.

      Revives DM-20864

      Attachments

        Issue Links

          Activity

            I think this is quite an ambitious plan, and unlikely we will manage to have it completed by March 31st.

            I suggest we just try to move third party libraries out of the science pipeline stack, to start.

            From our conversations in slack, in order to do that, we need to include conda compilers in the environment and ensure that the rebuild of lsst_distrib is working.

            I think this is already a big change, and probably should be documented in a different Jira issue.

             

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I think this is quite an ambitious plan, and unlikely we will manage to have it completed by March 31st. I suggest we just try to move third party libraries out of the science pipeline stack, to start. From our conversations in slack, in order to do that, we need to include conda compilers in the environment and ensure that the rebuild of lsst_distrib is working. I think this is already a big change, and probably should be documented in a different Jira issue.  
            bvan Brian Van Klaveren added a comment - - edited

            builds of lsst_distrib on OS X and centos 7 with conda compilers are working well - I also did a deploy+build on a raw centos 7 container with only git installed - I had to add `patch` to get the eups eigen to build

            To get this in jenkins, I need to either modify sconsUtils to be smarter about auto-selecting conda compilers if available, or explicitly set `SCONSUTILS_USE_CONDA_COMPILERS=1` somewhere

            this script should work on any system:

            git clone https://github.com/lsst/lsstsw.git
            cd lsstsw
            ./bin/deploy -r tickets/DM-23835
            source bin/setup.sh
            export SCONSUTILS_USE_CONDA_COMPILERS=1
            . /lsstsw/bin/envconfig -n lsst-scipipe-tickets-DM-23835.726a682
            # conda install patch (necessary on bare centos7 docker image)
            rebuild -r tickets/DM-23835 lsst_distrib
            

            Note: this ticket's branches do have a few patched unit tests to make this work properly which will go away once we fix capitalization

            bvan Brian Van Klaveren added a comment - - edited builds of lsst_distrib on OS X and centos 7 with conda compilers are working well - I also did a deploy+build on a raw centos 7 container with only git installed - I had to add `patch` to get the eups eigen to build To get this in jenkins, I need to either modify sconsUtils to be smarter about auto-selecting conda compilers if available, or explicitly set `SCONSUTILS_USE_CONDA_COMPILERS=1` somewhere this script should work on any system: git clone https://github.com/lsst/lsstsw.git cd lsstsw ./bin/deploy -r tickets/DM-23835 source bin/setup.sh export SCONSUTILS_USE_CONDA_COMPILERS=1 . /lsstsw/bin/envconfig -n lsst-scipipe-tickets-DM-23835.726a682 # conda install patch (necessary on bare centos7 docker image) rebuild -r tickets/DM-23835 lsst_distrib Note: this ticket's branches do have a few patched unit tests to make this work properly which will go away once we fix capitalization

            There is a current build (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31686/pipeline/47) which I expect to pass, as ci_hsc_gen2 and ci_hsc_gen3 have passed on different executors.

            That build is running to verify some fixes that were merged into cfitiso this morning on conda-forge which caused a segfault in ci_hsc_gen2 on linux (but not on mac).

            That build does not include sconsUtils changes. I have a new sconsUtils build running to verify some changes for the open pull requests:
            https://ci.lsst.codes/job/stack-os-matrix/31693/

            I believe these two builds should resolve any last issues in the code preventing merging, but I can execute one final ci_hsc build after the sconsUtils changes are verified.

            bvan Brian Van Klaveren added a comment - There is a current build ( https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31686/pipeline/47 ) which I expect to pass, as ci_hsc_gen2 and ci_hsc_gen3 have passed on different executors. That build is running to verify some fixes that were merged into cfitiso this morning on conda-forge which caused a segfault in ci_hsc_gen2 on linux (but not on mac). That build does not include sconsUtils changes. I have a new sconsUtils build running to verify some changes for the open pull requests: https://ci.lsst.codes/job/stack-os-matrix/31693/ I believe these two builds should resolve any last issues in the code preventing merging, but I can execute one final ci_hsc build after the sconsUtils changes are verified.

            The scipipe_conda_env PR has been reviewed and approved.

            The sconsUtils PR has been reviewed by tjenness and I think it's ready for merge.\

            All other repos, except for psfex, have changes to their eups table files or .cfg -in almost all cases just removing the eups third parties, supporting the new third parties. gcomoretto if you wouldn't mind looking at that. I will post a list of those repos.

            The psfex changes are an update to the autotools configurations, based on upstream psfex changes and this conda-forge patch, to more easily compile against the conda-forge provided libraries and headers. This is the reason for the lapack inclusion in the bleed file.

            bvan Brian Van Klaveren added a comment - The scipipe_conda_env PR has been reviewed and approved. The sconsUtils PR has been reviewed by tjenness and I think it's ready for merge.\ All other repos, except for psfex, have changes to their eups table files or .cfg -in almost all cases just removing the eups third parties, supporting the new third parties. gcomoretto if you wouldn't mind looking at that. I will post a list of those repos. The psfex changes are an update to the autotools configurations, based on upstream psfex changes and this conda-forge patch , to more easily compile against the conda-forge provided libraries and headers. This is the reason for the lapack inclusion in the bleed file.

            the build mentioned above https://ci.lsst.codes/job/stack-os-matrix/31693/ has failed. Did a new one succeed?

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - the build mentioned above https://ci.lsst.codes/job/stack-os-matrix/31693/  has failed. Did a new one succeed?

            I have reviewed the PRs as requested. 

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I have reviewed the PRs as requested. 

            People

              bvan Brian Van Klaveren
              bvan Brian Van Klaveren
              Gabriele Comoretto [X] (Inactive), Tim Jenness
              Brian Van Klaveren, Gabriele Comoretto [X] (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.