Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24232

Perform switch to conda-forge channel for third party software and conda environment update

    XMLWordPrintable

Details

    • Architecture
    • No

    Description

      This is the ticket to actually perform the switch to the conda environments based on the conda-forge channel as well as the conda compilers.

      This involves a few changes across a few repos to fully implement. Those changes are equivalent to adding a new compiler and a new conda environment, since that is what we are doing.

      • A rebase of all branches in DM-23835, a new conda test tag
      • Small changes to ci-scripts (specifically ccutils for conda compiler)
      • A final verification with ci jobs and stack demo
      • Verification the tarballs work
      • Updates to jenkins-dm-jobs (specifically the YAML configuration, and some jobs)
      • A publishing of a new daily release with the new conda environment (including tarballs)

      This change implicitly also means updating the versions of several third party packages, such as cfitsio.

      Attachments

        Issue Links

          Activity

            tarball_matrix failed because I did not run_rebuild with lsst_dm_stack_demo;

            jenkins-dm-scripts/pipeline/release/tarball_matrix.groovy ignores the "products" field, so it populates the tarball.groovy job with lsst_distrib and lsst_dm_stack_demo (from the etc/build_matrix.yaml)

            I was not able to rebuild the individual tarball jobs via Jenkins UI; jenkins-dm-scripts/jobs/tarball.groovy has a hard choice for miniconda version which is not updated for 4.7.12. When tarball matrix launches that job, it is bypassing that check so it works fine.

            I have re-ran rebuild to include lsst_dm_stack_demo on the tag for now, then retry tarball_matrix.

            bvan Brian Van Klaveren added a comment - tarball_matrix failed because I did not run_rebuild with lsst_dm_stack_demo; jenkins-dm-scripts/pipeline/release/tarball_matrix.groovy ignores the "products" field, so it populates the tarball.groovy job with lsst_distrib and lsst_dm_stack_demo (from the etc/build_matrix.yaml) I was not able to rebuild the individual tarball jobs via Jenkins UI; jenkins-dm-scripts/jobs/tarball.groovy has a hard choice for miniconda version which is not updated for 4.7.12. When tarball matrix launches that job, it is bypassing that check so it works fine. I have re-ran rebuild to include lsst_dm_stack_demo on the tag for now, then retry tarball_matrix.

            I have launched a stack-os-matrix job with lsst_sims out of curiosity.

            bvan Brian Van Klaveren added a comment - I have launched a stack-os-matrix job with lsst_sims out of curiosity.

            The code review is for the two open PRs for newinstall/lsstsw.

            Those two PRs fix the conda channels to be based on conda-forge after installation.

            These changes are expected to be valid even if we modify how the environment is installed.

            bvan Brian Van Klaveren added a comment - The code review is for the two open PRs for newinstall/lsstsw. Those two PRs fix the conda channels to be based on conda-forge after installation. These changes are expected to be valid even if we modify how the environment is installed.
            tjenness Tim Jenness added a comment -

            I haven't really been following the discussion on the specific options here, but it seems reasonable in general.

            tjenness Tim Jenness added a comment - I haven't really been following the discussion on the specific options here, but it seems reasonable in general.

            I try lsstsw and newinstall and both environments seems to be setup correctly. I started a build and everything seems fine.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - I try lsstsw and newinstall and both environments seems to be setup correctly. I started a build and everything seems fine.

            People

              bvan Brian Van Klaveren
              bvan Brian Van Klaveren
              Gabriele Comoretto [X] (Inactive), Tim Jenness
              Brian Van Klaveren, Gabriele Comoretto [X] (Inactive), John Parejko, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.