Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24972

Switch to using an explicit export of conda packages

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: conda
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Using conda env export is susceptible to not being resolvable if a package is marked as broken. We should switch to using conda list --explicit to avoid this issue.

      This triggers a few different modifications:
      1. newinstall/lsstsw need to be modified to install using conda install. This is a different workflow than the existing workflow.
      2. newinstall/lsstsw should probably be backwards compatible with previous methods
      3. pip packages should be moved into conda-forge (pytest-subtests, peigen) as conda list --explicit does not seem to spit out packages installed by pip
      4. Some additional work with the conda-forge team is necessary to ensure broken packages are NOT removed for the CDN the way the currently are, but are removed from the metadata. This ensure broken packages are still installable

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Please remember to update the README.md in scipipe_conda_env to describe how to generate these lockfiles. (Or remove that readme, and have the whole process documented somewhere central.)

            Show
            swinbank John Swinbank added a comment - Please remember to update the README.md in scipipe_conda_env to describe how to generate these lockfiles. (Or remove that readme, and have the whole process documented somewhere central.)
            Hide
            ktl Kian-Tat Lim added a comment -

            I was worried about removing the python version option (because we might need to go to Py4 in the future), but Brian convinced me that this was only for the base miniconda environment and that it didn't affect our environment.

            I was also questioning the conda clean commands, but they seem to be quite useful to minimize space utilization, so they can stay.

            There are a couple of other trivial comments; the rest looks great.

            Show
            ktl Kian-Tat Lim added a comment - I was worried about removing the python version option (because we might need to go to Py4 in the future), but Brian convinced me that this was only for the base miniconda environment and that it didn't affect our environment. I was also questioning the conda clean commands, but they seem to be quite useful to minimize space utilization, so they can stay. There are a couple of other trivial comments; the rest looks great.
            Hide
            ktl Kian-Tat Lim added a comment -

            Brian will use the same lock files that were generated a while back, as that is the environment we actually want to freeze. He will update the _packages file to correspond, even though we won't actually use it anymore. He will also add new versions of the lock, bleed, and packages files to the qserv branch for building that product.

            Show
            ktl Kian-Tat Lim added a comment - Brian will use the same lock files that were generated a while back, as that is the environment we actually want to freeze. He will update the _packages file to correspond, even though we won't actually use it anymore. He will also add new versions of the lock, bleed, and packages files to the qserv branch for building that product.
            Hide
            ktl Kian-Tat Lim added a comment -

            Since the DMCCB has decided to implement this for the release to ensure reproducibility, this ticket should block the release.

            Show
            ktl Kian-Tat Lim added a comment - Since the DMCCB has decided to implement this for the release to ensure reproducibility, this ticket should block the release.
            Hide
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment -

            If everything has been merge, can the status of this issue be changed to DONE, so I can start with the 20.0.0 release candidate 1?

            Show
            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - If everything has been merge, can the status of this issue be changed to DONE, so I can start with the 20.0.0 release candidate 1?

              People

              Assignee:
              bvan Brian Van Klaveren
              Reporter:
              bvan Brian Van Klaveren
              Reviewers:
              Gabriele Comoretto [X] (Inactive), Kian-Tat Lim
              Watchers:
              Brian Van Klaveren, Chris Walter, Gabriele Comoretto [X] (Inactive), John Swinbank, Kian-Tat Lim
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.