Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5821

Intermittent fault building ci_hsc through Jenkins

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc
    • Labels:
      None
    • Story Points:
      3
    • Sprint:
      DRP X16-3, DRP F16-1
    • Team:
      Data Release Production

      Description

      Occasionally (see e.g. here and here) the ci_hsc job in Jenkins fails, reporting:

      RuntimeError: dictionary changed size during iteration
      

      The fault seems to be intermittent. Please fix it.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            I've now eyeballed the code. Looks good to go. Thank you, Paul Price!

            Show
            swinbank John Swinbank added a comment - I've now eyeballed the code. Looks good to go. Thank you, Paul Price !
            Hide
            swinbank John Swinbank added a comment -

            Oh, with the caveat that the PR on GitHub currently claims "All checks have failed" and has a ugly red cross next to it. I think that's just an artefact of ongoing maintenance – Joshua Hoblitt might be able to confirm – so it's probably safe to merge, but it's a bit scary.

            Show
            swinbank John Swinbank added a comment - Oh, with the caveat that the PR on GitHub currently claims "All checks have failed" and has a ugly red cross next to it. I think that's just an artefact of ongoing maintenance – Joshua Hoblitt might be able to confirm – so it's probably safe to merge, but it's a bit scary.
            Hide
            price Paul Price added a comment -

            I did multiple pushes of ci_hsc to github (push, rebase, push) and the automatically triggered Jenkins runs were holding up my manually-triggered Jenkins run (which I reported above passed), so I asked Joshua Hoblitt to kill the automatically triggered runs. The log for the checks that failed says "Aborted by user Joshua Hoblitt", which is consistent with my story. I therefore believe this is safe to merge, and will go ahead and do so now.

            Show
            price Paul Price added a comment - I did multiple pushes of ci_hsc to github (push, rebase, push) and the automatically triggered Jenkins runs were holding up my manually-triggered Jenkins run (which I reported above passed), so I asked Joshua Hoblitt to kill the automatically triggered runs. The log for the checks that failed says "Aborted by user Joshua Hoblitt", which is consistent with my story. I therefore believe this is safe to merge, and will go ahead and do so now.
            Hide
            price Paul Price added a comment -

            Merged to master.

            Show
            price Paul Price added a comment - Merged to master.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            (I thought I had answered this... maybe it was a parallel inquiry on HC)

            I concur with Paul's assessment. At the time of the incident, I looked at the configuration of the jenkins plugin that handles this integration with github (github pull request builder) and this behavior is not configurable. If this issue comes up again the plugin should be extended to either set the `error` status on the commit (https://developer.github.com/v3/repos/statuses/) or remove the commit status completely.

            Show
            jhoblitt Joshua Hoblitt added a comment - (I thought I had answered this... maybe it was a parallel inquiry on HC) I concur with Paul's assessment. At the time of the incident, I looked at the configuration of the jenkins plugin that handles this integration with github (github pull request builder) and this behavior is not configurable. If this issue comes up again the plugin should be extended to either set the `error` status on the commit ( https://developer.github.com/v3/repos/statuses/ ) or remove the commit status completely.

              People

              • Assignee:
                price Paul Price
                Reporter:
                swinbank John Swinbank
                Reviewers:
                John Swinbank
                Watchers:
                John Swinbank, Joshua Hoblitt, Paul Price, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel