Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21456

ts-ci jenkins master low disk space

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None
    • Templates:
    • Story Points:
      0.625
    • Epic Link:
    • Team:
      SQuaRE

      Description

      Tiago Ribeiro reported via slack that he was unable to launch new builds and the "master" jenkins agent was offline. Upon investigation, it was determined that the "master" agent was taken offline as it had hit the 50GiB minimum disk space threshold. Reducing the limit to 30GiB brought the "master" agent back online. However, the UI was showing a large number (probably > 50) executors on the "master" agent even though it was zero to 0. Changing the value 0 -> 1 -> 0, didn't change this, so the master agent ws restarted. At this point the "el7-1" jenkins agent would connect then immediately drop every 10s or. Restarting the master and the "el7-1" agent didn't resolve this. It was also noted that the jenkins master was hanging upon startup and not producing log files. A general OS update was done, which did not resolve the issue, but turned up some duplicate RPM packages installed up on the master VM. Eventually, a few selinux related errors turned up in the dmesg/logs. relabeling did not fix jenkins intermittently freezing at start up. Completely disabling selinux and restarting the VM resolved the freezing at start up problem and the "el7-1" constantly reconnecting. My theory is that a security update patch (which are automatically applied) some how messed up selinux labels related to java/jenkins/nginx but the problems wasn't exposed until the jenkins master was restarted.

      The low disk situation still needs to be resolved. The proposal is to remove the disabled jobs which has since been migrated to the "LSST T&S" org folder. Per Andy Clements, Rob Bovill has until ~EOB tomorrow to veto.

        Attachments

          Activity

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Andy Clements, Joshua Hoblitt, Rob Bovill, Tiago Ribeiro
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel