Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12010

weekly release w_2017_38 delayed/required manual intervention

    XMLWordPrintable

    Details

      Description

      The w_2017_38 weekly-release is still in progress. In appears that there may have been some sort of network lurch over the weekend that interrupted the ssh connections between the master and the osx build nodes. Jenkins reconnected via ssh and was displaying the tarball build pipeline script fragment as running on an osx node, but not progress was being made. The tarball builds needed to be manually aborted so the weekly-release pipeline would retry.

      I suspect that setting timeouts on the tarball builds would avoid this. Also, splitting the tarball builds up into one build per python major version/os/conda env, would make retrying vastly more efficient.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          It appears that only waiting for some of the tarball jobs has a drawback. Jenkins will immediately mark the parallel branch for the triggered job as successful. While it is still possible to 'wait' on these builds to complete before exiting the pipeline, it makes the stage status display misleading. It might be possible to setup another set of parallel branches to wait for completion, which would help with the display... but I think that would also make a mess of job retrying. I'm not going to implement this at this time.

          Show
          jhoblitt Joshua Hoblitt added a comment - It appears that only waiting for some of the tarball jobs has a drawback. Jenkins will immediately mark the parallel branch for the triggered job as successful. While it is still possible to 'wait' on these builds to complete before exiting the pipeline, it makes the stage status display misleading. It might be possible to setup another set of parallel branches to wait for completion, which would help with the display... but I think that would also make a mess of job retrying. I'm not going to implement this at this time.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          A new job named release/tarball-matrix has been added to (more than) replace the preivous behavior of release/tarball. Archiving on error has been fixed up and additional timeout blocks have been added.

          Show
          jhoblitt Joshua Hoblitt added a comment - A new job named release/tarball-matrix has been added to (more than) replace the preivous behavior of release/tarball . Archiving on error has been fixed up and additional timeout blocks have been added.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The nightly-release pipeline failed last night due to problems with tarball production and an exception is being thrown from the construction of results.json.

          groovy.lang.MissingPropertyException: No such property: bx for class: groovy.lang.Binding
          
          

          Show
          jhoblitt Joshua Hoblitt added a comment - The nightly-release pipeline failed last night due to problems with tarball production and an exception is being thrown from the construction of results.json . groovy.lang.MissingPropertyException: No such property: bx for class : groovy.lang.Binding
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The MPE was caused by a simply variable scoping problem due to the archiving code being moved into a finally block.

          Show
          jhoblitt Joshua Hoblitt added a comment - The MPE was caused by a simply variable scoping problem due to the archiving code being moved into a finally block.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          A fix has been merged.

          Show
          jhoblitt Joshua Hoblitt added a comment - A fix has been merged.

            People

            Assignee:
            jhoblitt Joshua Hoblitt
            Reporter:
            jhoblitt Joshua Hoblitt
            Watchers:
            Adam Thornton, Joshua Hoblitt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.