Description
We have relied on CentOS 7 as a stable, reliable base operating system for all of our Linux needs, both in development and in operations (especially on the Summit).
The CentOS project has changed directions and is releasing new distributions as CentOS Stream, which is now upstream from Red Hat Enterprise Linux (RHEL) instead of downstream from it. Accordingly, while still well-tested, CentOS Stream is not as stable as CentOS used to be.
Alternatives have sprung up in the Open Source community. The two major ones are AlmaLinux and Rocky Linux. Both are rebuilds of RHEL 8, fulfilling the same role as CentOS used to, but with different tooling. AlmaLinux uses build tooling originally from a commercial provider (CloudLinux), while Rocky Linux built theirs separately. Both released seemingly identical 8.5 versions within days of each other (Nov. 12 for Alma, Nov. 15 for Rocky).
Switching between distributions is basically trivial. I believe this involves only changing the base container name in Dockerfiles or the execution of a migration script that changes out distribution sources and branding.
It is unclear which version will be best supported in the future. There are differences in the organizational structures between the two projects, but it is not clear that these are significant. Google Cloud is an official sponsor of Rocky Linux but not AlmaLinux; as a result, Google makes Rocky (but not Alma) virtual machine images directly available on Google Compute Engine. On the other hand, we're not likely to use such VMs, as the vast majority of our cloud-based services should run in containers on Kubernetes (which generally uses a container-optimized underlying OS).
Nevertheless, given our existing relationship with Google and the fact that AWS and Microsoft Azure sponsor both distributions, I suggested that we use this very slight preference to break the tie in favor of Rocky.
Later comments indicated slight preferences for AlmaLinux due to possibly-faster release timing and a small difference in the default root user configuration for Raspberry Pi. So we will go with AlmaLinux.
SLAC's current plan is to stick with RHEL8, though we do need some more negotiation on server licences. They are open to running clones on the client side (eg compute nodes). There is no issue for which clone runs in containers run at SLAC.