Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Continuous Integration
-
Labels:None
-
Story Points:1.25
-
Epic Link:
-
Team:SQuaRE
Description
There are frequently breakages / regressions that occur in the normal course of development between runs of weekly-release, which for various reason only turn up during the weekly release cycle, and these problems are preventing its on time production. As the "weekly" is now on the critical path for many developers, this results in unplanned / reactive ops works nearly every week.
In hopes of beating down the rate of failure, a more limited per day or "nightly" release is proposed. At least initially, this would treated as a shake down or quasi-canary build and not considered as an "official" product.
The "nightly" would differ from the weekly in the following ways:
- no git tag (initially) – a moving "nightly" tag might be useful at some point while avoiding having to eventually prune git tags
- only a single eups distrib tag would be created
- tarballs limited to py3 (reducing runtime / cpu consumption by half)
- no docker images (initially – this will also reduce resource consumption)
- eups distrib tag format would be d_<year><mon><day>. Where the d prefix is for "day" and year/mon/day would be the Gregorian calendar date in project standard time.
Attachments
Issue Links
- is triggered by
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
-
- Done
- is triggering
-
- To Do
-
- Done
Activity
There were successful builds yesterday afternoon / just after midnight of d_2017_08_09 and d_2017_08_10:
https://ci.lsst.codes/job/release/job/nightly-release/9/
https://ci.lsst.codes/job/release/job/nightly-release/10/
The first build was ~6:10 and the second ~3:20, which is indeed shorter than the current weekly release job. Of course, the "nightly" will need to become more like the "weekly" in order to shake down the same code paths.
Kian-Tat Lim What is your concern with moving git tags? I added support to sqre-codekit for moving a tag as part of DM-11430.
What we are missing is an automated way to remove git tags, which presumably should be happening for old "weeklies". We also a lack an automated way of cleaning up eups distrib tags – this has gotten a bit more complicated now that there is a src tag + 1 per tarball repo.
The main issue with moving tags is that there can be disagreement as to what a tag means/meant. This occurs even if the tag is solely local, but it is even worse if a tag is distributed to other clones.
Still, this is to some extent a social phenomenon rather than necessarily a technical one, so I am not set against using tags for this function.
I've triggered an issue to add git tags to the "nightly" but blocked it on another ticket to setup some sort of git tag purging.
My understanding is that moving git tags is fraught with issues and should generally be avoided. We could perhaps use a branch instead. If we did have such a tag, it might be best for it to be a lightweight tag rather than an annotated one. In any case, if we're having a moving tag, it might be useful to have two: one for the last successful build and one for the next pending build.
Would it be better to not create distinct eups distrib tags but instead to reuse one over and over?