Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11549

generate "nightly" releases

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      There are frequently breakages / regressions that occur in the normal course of development between runs of weekly-release, which for various reason only turn up during the weekly release cycle, and these problems are preventing its on time production. As the "weekly" is now on the critical path for many developers, this results in unplanned / reactive ops works nearly every week.

      In hopes of beating down the rate of failure, a more limited per day or "nightly" release is proposed. At least initially, this would treated as a shake down or quasi-canary build and not considered as an "official" product.

      The "nightly" would differ from the weekly in the following ways:

      • no git tag (initially) – a moving "nightly" tag might be useful at some point while avoiding having to eventually prune git tags
      • only a single eups distrib tag would be created
      • tarballs limited to py3 (reducing runtime / cpu consumption by half)
      • no docker images (initially – this will also reduce resource consumption)
      • eups distrib tag format would be d_<year><mon><day>. Where the d prefix is for "day" and year/mon/day would be the Gregorian calendar date in project standard time.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            My understanding is that moving git tags is fraught with issues and should generally be avoided. We could perhaps use a branch instead. If we did have such a tag, it might be best for it to be a lightweight tag rather than an annotated one. In any case, if we're having a moving tag, it might be useful to have two: one for the last successful build and one for the next pending build.

            Would it be better to not create distinct eups distrib tags but instead to reuse one over and over?

            Show
            ktl Kian-Tat Lim added a comment - My understanding is that moving git tags is fraught with issues and should generally be avoided. We could perhaps use a branch instead. If we did have such a tag, it might be best for it to be a lightweight tag rather than an annotated one. In any case, if we're having a moving tag, it might be useful to have two: one for the last successful build and one for the next pending build. Would it be better to not create distinct eups distrib tags but instead to reuse one over and over?
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            There were successful builds yesterday afternoon / just after midnight of d_2017_08_09 and d_2017_08_10:

            https://ci.lsst.codes/job/release/job/nightly-release/9/
            https://ci.lsst.codes/job/release/job/nightly-release/10/

            The first build was ~6:10 and the second ~3:20, which is indeed shorter than the current weekly release job. Of course, the "nightly" will need to become more like the "weekly" in order to shake down the same code paths.

            Show
            jhoblitt Joshua Hoblitt added a comment - There were successful builds yesterday afternoon / just after midnight of d_2017_08_09 and d_2017_08_10 : https://ci.lsst.codes/job/release/job/nightly-release/9/ https://ci.lsst.codes/job/release/job/nightly-release/10/ The first build was ~6:10 and the second ~3:20, which is indeed shorter than the current weekly release job. Of course, the "nightly" will need to become more like the "weekly" in order to shake down the same code paths.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Kian-Tat Lim What is your concern with moving git tags? I added support to sqre-codekit for moving a tag as part of DM-11430.

            What we are missing is an automated way to remove git tags, which presumably should be happening for old "weeklies". We also a lack an automated way of cleaning up eups distrib tags – this has gotten a bit more complicated now that there is a src tag + 1 per tarball repo.

            Show
            jhoblitt Joshua Hoblitt added a comment - Kian-Tat Lim What is your concern with moving git tags? I added support to sqre-codekit for moving a tag as part of DM-11430 . What we are missing is an automated way to remove git tags, which presumably should be happening for old "weeklies". We also a lack an automated way of cleaning up eups distrib tags – this has gotten a bit more complicated now that there is a src tag + 1 per tarball repo.
            Hide
            ktl Kian-Tat Lim added a comment -

            The main issue with moving tags is that there can be disagreement as to what a tag means/meant. This occurs even if the tag is solely local, but it is even worse if a tag is distributed to other clones.

            Still, this is to some extent a social phenomenon rather than necessarily a technical one, so I am not set against using tags for this function.

            Show
            ktl Kian-Tat Lim added a comment - The main issue with moving tags is that there can be disagreement as to what a tag means/meant. This occurs even if the tag is solely local, but it is even worse if a tag is distributed to other clones. Still, this is to some extent a social phenomenon rather than necessarily a technical one, so I am not set against using tags for this function.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've triggered an issue to add git tags to the "nightly" but blocked it on another ticket to setup some sort of git tag purging.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've triggered an issue to add git tags to the "nightly" but blocked it on another ticket to setup some sort of git tag purging.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Fritz Mueller, Frossie Economou, John Swinbank, Joshua Hoblitt, Kian-Tat Lim, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel