Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21502

DM jenkins master low disk space

    XMLWordPrintable

    Details

      Description

      Per Fritz Mueller via slack

      Oh, clicking on "master" it says "Disk space is too low.  Only 50.408 GB left on /var/lib/jenkins" in scary red type...

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          jhoblitt Joshua Hoblitt added a comment -

          I've lowered the free disk threshold to 40GiB to get the system quickly running again. Per job build retention values should be set as the long term fix.

          Show
          jhoblitt Joshua Hoblitt added a comment - I've lowered the free disk threshold to 40GiB to get the system quickly running again. Per job build retention values should be set as the long term fix.
          Hide
          Permalink
          jhoblitt Joshua Hoblitt added a comment -

          A default "log rotation" policy has been set. Artifacts are to be kept for 365 days while the build log (console output, etc.) is to be purged after 730 days. The cleanup will be triggered as jobs are running. Launching stack-os-matrix along was able to increase the free disk space to 67GiB.

          This ticket should be left open for at least a day or two to follow up on the disk space situation after more jobs have had a chance to run.

          Show
          jhoblitt Joshua Hoblitt added a comment - A default "log rotation" policy has been set. Artifacts are to be kept for 365 days while the build log (console output, etc.) is to be purged after 730 days. The cleanup will be triggered as jobs are running. Launching stack-os-matrix along was able to increase the free disk space to 67GiB. This ticket should be left open for at least a day or two to follow up on the disk space situation after more jobs have had a chance to run.
          Hide
          Permalink
          jhoblitt Joshua Hoblitt added a comment -

          Free disk space on the master has increased to 126.12 GB overnight. That should be a sufficient margin to hold us until the volume is expanded as part of DM-20229.

          Show
          jhoblitt Joshua Hoblitt added a comment - Free disk space on the master has increased to 126.12 GB overnight. That should be a sufficient margin to hold us until the volume is expanded as part of DM-20229 .

            People

            Assignee:
            jhoblitt Joshua Hoblitt
            Reporter:
            jhoblitt Joshua Hoblitt
            Watchers:
            Adam Thornton, Fritz Mueller, Frossie Economou, Joshua Hoblitt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: