Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10071

add timeouts to lsstsw based jenkins jobs

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      Occasionally, we observe lsstsw driven builds hanging for extreme periods of time (> 8 hours). This could be prevented by implemented a per job build timeout.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment - - edited

            Just killed a build that was stuck / running for over 2 days.

            Show
            jhoblitt Joshua Hoblitt added a comment - - edited Just killed a build that was stuck / running for over 2 days.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            A 5 hour timeout, due to a from scratch build currently taking almost 4 hours, has been added to stack-os-matrix and release/run-rebuild.

            Show
            jhoblitt Joshua Hoblitt added a comment - A 5 hour timeout, due to a from scratch build currently taking almost 4 hours, has been added to stack-os-matrix and release/run-rebuild .
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            It appears that an overall timer maybe be needed on stack-os-matrix despite there being a timeout block on each build configuration.

            Show
            jhoblitt Joshua Hoblitt added a comment - It appears that an overall timer maybe be needed on stack-os-matrix despite there being a timeout block on each build configuration.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            This seems to be working reasonably well. It would be nice to have a timer that starts as soon as any one parallel branch starts running but I don't think there is a supported way to do IPC between branches.

            Show
            jhoblitt Joshua Hoblitt added a comment - This seems to be working reasonably well. It would be nice to have a timer that starts as soon as any one parallel branch starts running but I don't think there is a supported way to do IPC between branches.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt, Scott Daniel
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel