Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25434

Restructure tarball paths with goal to remove miniconda version

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: conda, lsst, lsstsw
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      At a minimum, the miniconda reference in the tarball path is not useful, so it should be removed.

      Currently, the paths look like these:
      https://eups.lsst.codes/stack/osx/10.9/conda-system/miniconda3-4.7.12-1a1d771/

      Which corresponds to the following template:

      stack/{os_family}/{kernel_equivalent}/{compiler}/{miniconda_version}-{environment hash}/
      

      Since an environment effectively encodes kernel equivalent, compiler and the miniconda version is irrelevant, I'd suggest dropping those from the paths to simplify things, although at least dropping miniconda_version is desirable, as that version has no impact on binary or API compatibility.

      Further restructuring

      If we were to simplify this to rely more on the environment repo and hash, we may reconsider the string to be of the following format:

      stack/{arch}/{environment_repo}@{environment_hash}/
      

      e.g.:

      stack/osx-64/github.com/lsst/scipipe_conda_env@1234abc/
      

      This form is a bit more explicit, showing a bit more information about an environment in the URL, such as the repo where that environment is located.

      Using {arch} (linux-64, osx-64) is more consistent with how conda/conda-forge describes systems as well.

      Restructuring according to system

      If we would like to proceed further, it should be noted we conda-forge will be packaging newer versions of glibc. If we want to consider that in the path, we would expect kernel equivalent should also be changed to be either 10.9 (OS X), cos6 (centos 6), cos7 (centos 7), and roughly refers to the system/glibc features that are available to packages. Like the compiler, this can effectively be namespaced by the environment hash as well.

      Currently, the compiler is currently just "conda-system", which is also fine if we are leaving in the environment hash.

      Based on this, If we were to reimagine the path so that binaries might be reused across conda environments, we might choose something similar to the following:

      /stack/{arch}/{kernel_equivalent}/{compiler_equivalent}/
      

        Attachments

          Issue Links

            Activity

            No work has yet been logged on this issue.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bvan Brian Van Klaveren
              Watchers:
              Brian Van Klaveren, Chris Walter, Gabriele Comoretto [X] (Inactive), Heather Kelly, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.