Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20444

jenkins pipeline retry{} block not repeating upon failure

    Details

      Description

      This build of build-layercake failed on the first try of an image build without retrying despite the failing section being wrapped with a retry block. Eg.,

      https://ci.lsst.codes/job/sqre/job/infra/job/build-layercake/44/console

                stage(tsTag) {
                  retry(retries) {
                    tsBuild = packIt('centos_devtoolset.json', [
                      "-var base_image=${baseTag}",
                      "-var build_repository=${buildRepo}",
                      "-var build_name=${tsName}",
                      "-var scl_compiler=${scl}",
                    ])
                  } // retry
                  images << [(tsTag): tsBuild]
                } // stage
      
      

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I have been unable to replicate this failure in a test env. Eg.,

          node {
              retry(3) {
                  packIt()
              }
          }
           
          def String packIt() {
              docker.image("alpine").inside {
                    sh "exit 1"
              }
          }
          

          However, the test env I had already provisioned has updated plugins installed, so it's possible that a plugin update has fixed some sort of problem related to retrying... I didn't see anything obvious in the change logs for the pipeline basic steps or pipeline groovy plugins. This may have to be tested in the prod env.

          Show
          jhoblitt Joshua Hoblitt added a comment - I have been unable to replicate this failure in a test env. Eg., node { retry( 3 ) { packIt() } }   def String packIt() { docker.image( "alpine" ).inside { sh "exit 1" } } However, the test env I had already provisioned has updated plugins installed, so it's possible that a plugin update has fixed some sort of problem related to retrying... I didn't see anything obvious in the change logs for the pipeline basic steps or pipeline groovy plugins. This may have to be tested in the prod env.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The test snippet above also works as expected in current prod, so the problem doesn't appear to be a plugin or core bug.

          Show
          jhoblitt Joshua Hoblitt added a comment - The test snippet above also works as expected in current prod, so the problem doesn't appear to be a plugin or core bug.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Mixing in timeout blocks is also unable to replicate the failure to retry. Eg.

          timeout(time: 30, unit: 'MINUTES'){
            node {                                 
              timeout(time: 30, unit: 'MINUTES'){
                retry(3) {
                  packIt()
                }
              }
            }
          }
           
          def String packIt() {
            docker.image("alpine").inside {  
              sh "exit 6"
            }  
          }
          

          Show
          jhoblitt Joshua Hoblitt added a comment - Mixing in timeout blocks is also unable to replicate the failure to retry. Eg. timeout(time: 30 , unit: 'MINUTES' ){ node { timeout(time: 30 , unit: 'MINUTES' ){ retry( 3 ) { packIt() } } } }   def String packIt() { docker.image( "alpine" ).inside { sh "exit 6" } }
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Arrrgh, I see the problem now... it is a logic error. There are two different loops (base + w/ compiler) for building images and the base loop doesn't have retry blocks.

          Show
          jhoblitt Joshua Hoblitt added a comment - Arrrgh, I see the problem now... it is a logic error. There are two different loops (base + w/ compiler) for building images and the base loop doesn't have retry blocks.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          A fix has been merged.

          Show
          jhoblitt Joshua Hoblitt added a comment - A fix has been merged.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I'm reopening this ticket as a test run has turned up a groovy error:

          groovy.lang.MissingPropertyException: No such property: baseBuild for class: groovy.lang.Binding
          	at groovy.lang.Binding.getVariable(Binding.java:63)
          

          Show
          jhoblitt Joshua Hoblitt added a comment - I'm reopening this ticket as a test run has turned up a groovy error: groovy.lang.MissingPropertyException: No such property: baseBuild for class : groovy.lang.Binding at groovy.lang.Binding.getVariable(Binding.java: 63 )
          Hide
          jhoblitt Joshua Hoblitt added a comment - - edited

          It looks like a simple scoping problem caused by the addition of the retry block. The fix should be simple:

          --- a/pipelines/sqre/infra/build_layercake.groovy
          +++ b/pipelines/sqre/infra/build_layercake.groovy
          @@ -68,8 +68,8 @@ notify.wrap {
                         "-var build_repository=${buildRepo}",
                         "-var build_name=${baseName}",
                       ])
          +            images << [(baseTag): baseBuild]
                     } // retry
          -          images << [(baseTag): baseBuild]
                   } // stage
                 } // majrelease
          

          Show
          jhoblitt Joshua Hoblitt added a comment - - edited It looks like a simple scoping problem caused by the addition of the retry block. The fix should be simple: --- a/pipelines/sqre/infra/build_layercake.groovy +++ b/pipelines/sqre/infra/build_layercake.groovy @@ - 68 , 8 + 68 , 8 @@ notify.wrap { "-var build_repository=${buildRepo}" , "-var build_name=${baseName}" , ]) + images << [(baseTag): baseBuild] } // retry - images << [(baseTag): baseBuild] } // stage } // majrelease
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The scope fix has been tested as working in prod.

          Show
          jhoblitt Joshua Hoblitt added a comment - The scope fix has been tested as working in prod.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Joshua Hoblitt
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: