Details
-
Type:
Improvement
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Team:Data Release Production
Description
It would be good to know the update schedule for base packages on lsst-dev, lsstsw, and Jenkins (e.g. numpy, scipy), and/or have a clear procedure for how/when they get updated. We need to know that they will updated to the latest version N-weeks after its release, or have an obvious place to go to to request updates (and a way to be informed of updates and report problems post-update).
The motivation in this case is that astropy is currently at 1.1.1, which has a bug preventing me from generating plots that is fixed in 1.1.2. It's a sub-minor version bump, but I had to install my own stack to get what I wanted.
This is the one disadvantage of having pip-installed packages not eups controlled: one can only update them system-wide, thus affecting everyone on the system.
Attachments
Issue Links
Activity
Hmm. The shared stack on lsst-dev pulls in Python when it's first set up by installing our miniconda2 package & hence the list of packages from https://github.com/lsst/lsstsw/blob/master/etc/conda_packages-linux-64.txt. It won't reinstall miniconda2 (except if the version of that package is bumped), so it won't pick up any changes to that list.
Even if we did do something to track that list (say, by bumping the miniconda2 version every time the list of packages is updated, or by periodically syncing against that list), I'd be worried that we could pull in ABI-incompatible packages and break the installed stack. (Naively, this seems like a generic problem with managing some subset of our dependencies with conda and some with EUPS; is there a solution?)
In short, it's not obvious to me that doing anything to lsstsw will resolve the issue for shared stack users.
I'm removing myself as the assignee of this ticket, as, while I have a direct interest in the shared stack, there's nothing I can do about lsstsw or Jenkins.
I'm not saying that updating the package list in lsstsw will automatically fix the shared stack. It can't do that because you effectively need to re-deploy lsstsw (or the EUPS miniconda2/3 packages to pick up the changes. There is a chain of issues that need to be solved here:
- How often should the pinned list be updated in lsstsw. This list is the one used by minicondaN.
- Should the shared stack versions have any relationship to the pinned version list?
- How many shared stack builds are guaranteed to work? If you update numpy in conda is it okay to break any of the older builds?
- If the pinned versions are updated in the shared stack should the most recent release and some of the weeklies be rebuilt? What happens if a tag no longer builds? (say an Astropy API changed?).
I don't really know what the requirements are for the shared stack. Maybe people just want the most recent weekly? If they want the most recent official release then that raises supportability issues. We may have to face the possibility of patching a 12.0 release to make it work with newer versions of conda packages.
I imagine that at minimum we should update the pinned versions in lsstsw prior to each major release.
I don't think I'm scoped to answer those questions. I've opened to DM-7398 in the hope of finding out who is.
Superseded by work elsewhere. See LDM-294, the DM-CCB, etc.
In some sense this should be related to decisions on when to update the reference packages for lsstsw itself (which also triggers newinstall). Once we solve that plan the lsst-dev plan should drop out automatically.