# jenkins pipeline retry{} block not repeating upon failure

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
0.625
• Team:
SQuaRE

## Description

This build of build-layercake failed on the first try of an image build without retrying despite the failing section being wrapped with a retry block. Eg.,

https://ci.lsst.codes/job/sqre/job/infra/job/build-layercake/44/console

  stage(tsTag) {  retry(retries) {  tsBuild = packIt('centos_devtoolset.json', [  "-var base_image=${baseTag}",  "-var build_repository=${buildRepo}",  "-var build_name=${tsName}",  "-var scl_compiler=${scl}",  ])  } // retry  images << [(tsTag): tsBuild]  } // stage  

## Activity

Hide
Joshua Hoblitt added a comment -

I have been unable to replicate this failure in a test env. Eg.,

 node {  retry(3) {  packIt()  } }   def String packIt() {  docker.image("alpine").inside {  sh "exit 1"  } } 

However, the test env I had already provisioned has updated plugins installed, so it's possible that a plugin update has fixed some sort of problem related to retrying... I didn't see anything obvious in the change logs for the pipeline basic steps or pipeline groovy plugins. This may have to be tested in the prod env.

Show
Joshua Hoblitt added a comment - I have been unable to replicate this failure in a test env. Eg., node { retry( 3 ) { packIt() } }   def String packIt() { docker.image( "alpine" ).inside { sh "exit 1" } } However, the test env I had already provisioned has updated plugins installed, so it's possible that a plugin update has fixed some sort of problem related to retrying... I didn't see anything obvious in the change logs for the pipeline basic steps or pipeline groovy plugins. This may have to be tested in the prod env.
Hide
Joshua Hoblitt added a comment -

The test snippet above also works as expected in current prod, so the problem doesn't appear to be a plugin or core bug.

Show
Joshua Hoblitt added a comment - The test snippet above also works as expected in current prod, so the problem doesn't appear to be a plugin or core bug.
Hide
Joshua Hoblitt added a comment -

Mixing in timeout blocks is also unable to replicate the failure to retry. Eg.

 timeout(time: 30, unit: 'MINUTES'){  node {   timeout(time: 30, unit: 'MINUTES'){  retry(3) {  packIt()  }  }  } }   def String packIt() {  docker.image("alpine").inside {   sh "exit 6"  }  } 

Show
Joshua Hoblitt added a comment - Mixing in timeout blocks is also unable to replicate the failure to retry. Eg. timeout(time: 30 , unit: 'MINUTES' ){ node { timeout(time: 30 , unit: 'MINUTES' ){ retry( 3 ) { packIt() } } } }   def String packIt() { docker.image( "alpine" ).inside { sh "exit 6" } }
Hide
Joshua Hoblitt added a comment -

Arrrgh, I see the problem now... it is a logic error. There are two different loops (base + w/ compiler) for building images and the base loop doesn't have retry blocks.

Show
Joshua Hoblitt added a comment - Arrrgh, I see the problem now... it is a logic error. There are two different loops (base + w/ compiler) for building images and the base loop doesn't have retry blocks.
Hide
Joshua Hoblitt added a comment -

A fix has been merged.

Show
Joshua Hoblitt added a comment - A fix has been merged.
Hide
Joshua Hoblitt added a comment -

I'm reopening this ticket as a test run has turned up a groovy error:

 groovy.lang.MissingPropertyException: No such property: baseBuild for class: groovy.lang.Binding  at groovy.lang.Binding.getVariable(Binding.java:63) 

Show
Joshua Hoblitt added a comment - I'm reopening this ticket as a test run has turned up a groovy error: groovy.lang.MissingPropertyException: No such property: baseBuild for class : groovy.lang.Binding at groovy.lang.Binding.getVariable(Binding.java: 63 )
Hide
Joshua Hoblitt added a comment - - edited

It looks like a simple scoping problem caused by the addition of the retry block. The fix should be simple:

 --- a/pipelines/sqre/infra/build_layercake.groovy +++ b/pipelines/sqre/infra/build_layercake.groovy @@ -68,8 +68,8 @@ notify.wrap {  "-var build_repository=${buildRepo}",  "-var build_name=${baseName}",  ]) + images << [(baseTag): baseBuild]  } // retry - images << [(baseTag): baseBuild]  } // stage  } // majrelease 

Show
Joshua Hoblitt added a comment - - edited It looks like a simple scoping problem caused by the addition of the retry block. The fix should be simple: --- a/pipelines/sqre/infra/build_layercake.groovy +++ b/pipelines/sqre/infra/build_layercake.groovy @@ - 68 , 8 + 68 , 8 @@ notify.wrap { "-var build_repository=${buildRepo}" , "-var build_name=${baseName}" , ]) + images << [(baseTag): baseBuild] } // retry - images << [(baseTag): baseBuild] } // stage } // majrelease
Hide
Joshua Hoblitt added a comment -

The scope fix has been tested as working in prod.

Show
Joshua Hoblitt added a comment - The scope fix has been tested as working in prod.

## People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Watchers:
Joshua Hoblitt