Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12772

weekly release tags are not reliably rebuild-able from git

    Details

      Description

      The w.2017.45 weekly release git tag already can not be built via lsstsw/lsst-build due to a change in pytest-flake8. This is a major problem for being able to reproduce a build from even a week ago.

      The easiest solution should be to add the weekly release tags to the external repos.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            If you want older weeklies to be buildable you absolutely need to add the `w` tags to the thirdparty packages.

            Show
            tjenness Tim Jenness added a comment - If you want older weeklies to be buildable you absolutely need to add the `w` tags to the thirdparty packages.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Agreed. I'm going implement this immediately (reproducible builds are of critical importance IMHO).

            Show
            jhoblitt Joshua Hoblitt added a comment - Agreed. I'm going implement this immediately (reproducible builds are of critical importance IMHO).
            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            Weekly (and nightly "dry run") tagging of externals has been implemented and merged to production.

            Summary:

            • A simple docker image was constructed around sqre-codekit to allow the github-tag-version script to be used without having to bootstrap a [minimal] python env to use it: https://github.com/lsst-sqre/sqre-codekit/tree/master/docker
            • A jenkins job was added to build + publish the sqre-codekit image: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Fbuild-codekit/activity
              and the image is published to the docker hub registry as: https://hub.docker.com/r/lsstsqre/codekit/
            • A new utility method named {util.githubTagVersion() was added to invoke the docker.io/lsstsqre/codekit image with arbitrary CLI parameters, which are merged with an internal set of defaults. Eg.

              util.githubTagVersion(
                params.GIT_TAG,
                params.BUILD_ID,
                  [
                    '--dry-run': false,
                    '--team': ['Data Management', 'External'],
                  ]
              ) 
              

            • The existing release/tag-git-repos job was reimplemented as a simple call to to util.githubTagVersion().
            • The nightly-releaes and weekly-release jenkins jobs were updated to use the new util.githubTagVersion() method rather than triggering the release/tag-git-repos and added External to the list of teams names to be tagged.

            I'm placing this ticket into self review to follow up on the next builds of the nightly and weekly releases.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited Weekly (and nightly "dry run") tagging of externals has been implemented and merged to production. Summary: A simple docker image was constructed around sqre-codekit to allow the github-tag-version script to be used without having to bootstrap a [minimal] python env to use it: https://github.com/lsst-sqre/sqre-codekit/tree/master/docker A jenkins job was added to build + publish the sqre-codekit image: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Fbuild-codekit/activity and the image is published to the docker hub registry as: https://hub.docker.com/r/lsstsqre/codekit/ A new utility method named { util.githubTagVersion() was added to invoke the docker.io/lsstsqre/codekit image with arbitrary CLI parameters, which are merged with an internal set of defaults. Eg. util.githubTagVersion( params.GIT_TAG, params.BUILD_ID, [ '--dry-run' : false , '--team' : [ 'Data Management' , 'External' ], ] ) The existing release/tag-git-repos job was reimplemented as a simple call to to util.githubTagVersion() . The nightly-releaes and weekly-release jenkins jobs were updated to use the new util.githubTagVersion() method rather than triggering the release/tag-git-repos and added External to the list of teams names to be tagged. I'm placing this ticket into self review to follow up on the next builds of the nightly and weekly releases.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Looks like I broke nightly-release as isstanceof needed to be white-listed in the production groovy sandbox. That has been fixed but I may have found another possible problem:
            util.githubTagVersion() needs to be invoked from a node where docker is available.

            Show
            jhoblitt Joshua Hoblitt added a comment - Looks like I broke nightly-release as isstanceof needed to be white-listed in the production groovy sandbox. That has been fixed but I may have found another possible problem: util.githubTagVersion() needs to be invoked from a node where docker is available.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The above listed issues have been resolved and the nightly-release is being re-run.

            Show
            jhoblitt Joshua Hoblitt added a comment - The above listed issues have been resolved and the nightly-release is being re-run.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            There was another problem small problem with the calling parameter names, which were fixed, and a second re-run attempt passed. However, analyzing the console output, it appears my memory was faulty and the github team name for external dependencies is DM Externals rather than External. This isn't a big deal since the nightly is a "dry run" for git tagging. This gaff has been fixed and will be verified tomorrow after tonight's release build finishes.

            Show
            jhoblitt Joshua Hoblitt added a comment - There was another problem small problem with the calling parameter names, which were fixed, and a second re-run attempt passed. However, analyzing the console output, it appears my memory was faulty and the github team name for external dependencies is DM Externals rather than External . This isn't a big deal since the nightly is a "dry run" for git tagging. This gaff has been fixed and will be verified tomorrow after tonight's release build finishes.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The w.2017.47 tag has appeared in the external repos.
            Eg.: https://github.com/lsst/cfitsio/releases/tag/w.2017.47

            External tagging upon weekly release is now working as expected.

            Show
            jhoblitt Joshua Hoblitt added a comment - The w.2017.47 tag has appeared in the external repos. Eg.: https://github.com/lsst/cfitsio/releases/tag/w.2017.47 External tagging upon weekly release is now working as expected.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Joshua Hoblitt
                Watchers:
                Frossie Economou, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: