Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16495

TSSW Jenkins crashing

    Details

      Description

      The TSSW Jenkins service, https://ts-ci.lsst.codes/blue/organizations/jenkins, had a 502 Bad Gateway blip again, but recovered very quickly.  I know this because I saw the error in a browser cached console output.  When the browser refreshed, the build was gone, just gone.  This episode cost several hours of work and has delayed a critical software release.  This is most likely related to DM-16180 and DM-16459.

      The missing build in question, can not be found in the https://ts-ci.lsst.codes/job/ts_sal/ project. Notice there is a missing build #642.

        Attachments

          Issue Links

            Activity

            Hide
            rbovill Rob Bovill added a comment -

            Thank you Joshua Hoblitt !!!

            Show
            rbovill Rob Bovill added a comment - Thank you Joshua Hoblitt !!!
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The jenkins master has restarted in the last day, at least. The suspected robot related faults were occurring days or weeks apart, so there likely isn't much more action to take on this ticket at this time other than periodic observation over the the next month or so.

            Show
            jhoblitt Joshua Hoblitt added a comment - The jenkins master has restarted in the last day, at least. The suspected robot related faults were occurring days or weeks apart, so there likely isn't much more action to take on this ticket at this time other than periodic observation over the the next month or so.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The jenkins master still appears to be running normally:

            ● jenkins.service - Jenkins master service
               Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: disabled)
               Active: active (running) since Wed 2019-01-30 21:04:30 PST; 2 weeks 1 days ago
            

            Show
            jhoblitt Joshua Hoblitt added a comment - The jenkins master still appears to be running normally: ● jenkins.service - Jenkins master service Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019 - 01 - 30 21 : 04 : 30 PST; 2 weeks 1 days ago
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Note that the master process uptime was reset last night by the deployment of changes for DM-17540.

            Show
            jhoblitt Joshua Hoblitt added a comment - Note that the master process uptime was reset last night by the deployment of changes for DM-17540 .
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm going to go ahead and close this ticket as my "gut" feel is it is resolved and there's another core update coming, which will reset the uptime count again.

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm going to go ahead and close this ticket as my "gut" feel is it is resolved and there's another core update coming, which will reset the uptime count again.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                rbovill Rob Bovill
                Reviewers:
                Rob Bovill
                Watchers:
                Frossie Economou, Joshua Hoblitt, Kevin Robison [X] (Inactive), Rob Bovill, Wil O'Mullane
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel