weekly release w_2017_38 delayed/required manual intervention

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2.375
• Team:
SQuaRE

Description

The w_2017_38 weekly-release is still in progress. In appears that there may have been some sort of network lurch over the weekend that interrupted the ssh connections between the master and the osx build nodes. Jenkins reconnected via ssh and was displaying the tarball build pipeline script fragment as running on an osx node, but not progress was being made. The tarball builds needed to be manually aborted so the weekly-release pipeline would retry.

I suspect that setting timeouts on the tarball builds would avoid this. Also, splitting the tarball builds up into one build per python major version/os/conda env, would make retrying vastly more efficient.

Activity

Hide
Joshua Hoblitt added a comment -

It appears that only waiting for some of the tarball jobs has a drawback. Jenkins will immediately mark the parallel branch for the triggered job as successful. While it is still possible to 'wait' on these builds to complete before exiting the pipeline, it makes the stage status display misleading. It might be possible to setup another set of parallel branches to wait for completion, which would help with the display... but I think that would also make a mess of job retrying. I'm not going to implement this at this time.

Show
Joshua Hoblitt added a comment - It appears that only waiting for some of the tarball jobs has a drawback. Jenkins will immediately mark the parallel branch for the triggered job as successful. While it is still possible to 'wait' on these builds to complete before exiting the pipeline, it makes the stage status display misleading. It might be possible to setup another set of parallel branches to wait for completion, which would help with the display... but I think that would also make a mess of job retrying. I'm not going to implement this at this time.
Hide
Joshua Hoblitt added a comment -

A new job named release/tarball-matrix has been added to (more than) replace the preivous behavior of release/tarball. Archiving on error has been fixed up and additional timeout blocks have been added.

Show
Joshua Hoblitt added a comment - A new job named release/tarball-matrix has been added to (more than) replace the preivous behavior of release/tarball . Archiving on error has been fixed up and additional timeout blocks have been added.
Hide
Joshua Hoblitt added a comment -

The nightly-release pipeline failed last night due to problems with tarball production and an exception is being thrown from the construction of results.json.

 groovy.lang.MissingPropertyException: No such property: bx for class: groovy.lang.Binding  

Show
Joshua Hoblitt added a comment - The nightly-release pipeline failed last night due to problems with tarball production and an exception is being thrown from the construction of results.json . groovy.lang.MissingPropertyException: No such property: bx for class : groovy.lang.Binding
Hide
Joshua Hoblitt added a comment -

The MPE was caused by a simply variable scoping problem due to the archiving code being moved into a finally block.

Show
Joshua Hoblitt added a comment - The MPE was caused by a simply variable scoping problem due to the archiving code being moved into a finally block.
Hide
Joshua Hoblitt added a comment -

A fix has been merged.

Show
Joshua Hoblitt added a comment - A fix has been merged.

People

Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Watchers: