# weekly release tags are not reliably rebuild-able from git

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
1.75
• Team:
SQuaRE

## Description

The w.2017.45 weekly release git tag already can not be built via lsstsw/lsst-build due to a change in pytest-flake8. This is a major problem for being able to reproduce a build from even a week ago.

The easiest solution should be to add the weekly release tags to the external repos.

## Activity

Hide
Tim Jenness added a comment -

If you want older weeklies to be buildable you absolutely need to add the w tags to the thirdparty packages.

Show
Tim Jenness added a comment - If you want older weeklies to be buildable you absolutely need to add the w tags to the thirdparty packages.
Hide
Joshua Hoblitt added a comment -

Agreed. I'm going implement this immediately (reproducible builds are of critical importance IMHO).

Show
Joshua Hoblitt added a comment - Agreed. I'm going implement this immediately (reproducible builds are of critical importance IMHO).
Hide
Joshua Hoblitt added a comment - - edited

Weekly (and nightly "dry run") tagging of externals has been implemented and merged to production.

Summary:

• A simple docker image was constructed around sqre-codekit to allow the github-tag-version script to be used without having to bootstrap a [minimal] python env to use it: https://github.com/lsst-sqre/sqre-codekit/tree/master/docker
• A jenkins job was added to build + publish the sqre-codekit image: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Fbuild-codekit/activity
and the image is published to the docker hub registry as: https://hub.docker.com/r/lsstsqre/codekit/
• A new utility method named {util.githubTagVersion() was added to invoke the docker.io/lsstsqre/codekit image with arbitrary CLI parameters, which are merged with an internal set of defaults. Eg.

 util.githubTagVersion(  params.GIT_TAG,  params.BUILD_ID,  [  '--dry-run': false,  '--team': ['Data Management', 'External'],  ] ) 

• The existing release/tag-git-repos job was reimplemented as a simple call to to util.githubTagVersion().
• The nightly-releaes and weekly-release jenkins jobs were updated to use the new util.githubTagVersion() method rather than triggering the release/tag-git-repos and added External to the list of teams names to be tagged.

I'm placing this ticket into self review to follow up on the next builds of the nightly and weekly releases.

Show
Joshua Hoblitt added a comment - - edited Weekly (and nightly "dry run") tagging of externals has been implemented and merged to production. Summary: A simple docker image was constructed around sqre-codekit to allow the github-tag-version script to be used without having to bootstrap a [minimal] python env to use it: https://github.com/lsst-sqre/sqre-codekit/tree/master/docker A jenkins job was added to build + publish the sqre-codekit image: https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Finfrastructure%2Fbuild-codekit/activity and the image is published to the docker hub registry as: https://hub.docker.com/r/lsstsqre/codekit/ A new utility method named { util.githubTagVersion() was added to invoke the docker.io/lsstsqre/codekit image with arbitrary CLI parameters, which are merged with an internal set of defaults. Eg. util.githubTagVersion( params.GIT_TAG, params.BUILD_ID, [ '--dry-run' : false , '--team' : [ 'Data Management' , 'External' ], ] ) The existing release/tag-git-repos job was reimplemented as a simple call to to util.githubTagVersion() . The nightly-releaes and weekly-release jenkins jobs were updated to use the new util.githubTagVersion() method rather than triggering the release/tag-git-repos and added External to the list of teams names to be tagged. I'm placing this ticket into self review to follow up on the next builds of the nightly and weekly releases.
Hide
Joshua Hoblitt added a comment -

Looks like I broke nightly-release as isstanceof needed to be white-listed in the production groovy sandbox. That has been fixed but I may have found another possible problem:
util.githubTagVersion() needs to be invoked from a node where docker is available.

Show
Joshua Hoblitt added a comment - Looks like I broke nightly-release as isstanceof needed to be white-listed in the production groovy sandbox. That has been fixed but I may have found another possible problem: util.githubTagVersion() needs to be invoked from a node where docker is available.
Hide
Joshua Hoblitt added a comment -

The above listed issues have been resolved and the nightly-release is being re-run.

Show
Joshua Hoblitt added a comment - The above listed issues have been resolved and the nightly-release is being re-run.
Hide
Joshua Hoblitt added a comment -

There was another problem small problem with the calling parameter names, which were fixed, and a second re-run attempt passed. However, analyzing the console output, it appears my memory was faulty and the github team name for external dependencies is DM Externals rather than External. This isn't a big deal since the nightly is a "dry run" for git tagging. This gaff has been fixed and will be verified tomorrow after tonight's release build finishes.

Show
Joshua Hoblitt added a comment - There was another problem small problem with the calling parameter names, which were fixed, and a second re-run attempt passed. However, analyzing the console output, it appears my memory was faulty and the github team name for external dependencies is DM Externals rather than External . This isn't a big deal since the nightly is a "dry run" for git tagging. This gaff has been fixed and will be verified tomorrow after tonight's release build finishes.
Hide
Joshua Hoblitt added a comment -

The w.2017.47 tag has appeared in the external repos.
Eg.: https://github.com/lsst/cfitsio/releases/tag/w.2017.47

External tagging upon weekly release is now working as expected.

Show
Joshua Hoblitt added a comment - The w.2017.47 tag has appeared in the external repos. Eg.: https://github.com/lsst/cfitsio/releases/tag/w.2017.47 External tagging upon weekly release is now working as expected.

## People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Reviewers:
Joshua Hoblitt
Watchers:
Frossie Economou, Joshua Hoblitt, Kian-Tat Lim, Scott Daniel, Simon Krughoff, Tim Jenness