Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18113

TSSW jenkins el7-1 agent out of disk space

    Details

      Description

      Per email from Te-Wei Tsai:

      Hi Joshua,
       
      This is Te-Wei Tsai. I have the problem for the "container size" of Jenkins server now.
       
      {color:#ff0000}It looks like the container size is much smaller than before after the maintenance and update recently.
       {color}
       
      You can see the following two console outputs ([https://ts-ci.lsst.codes/job/ts_tcs_ofcPython/job/develop/]):
       
      #28 has no built problem and the console output is here:
       
      [https://ts-ci.lsst.codes/job/ts_tcs_ofcPython/job/develop/28/console]
       
      #29 has no difference compared with #28. But I have the space error now:
       
      [https://ts-ci.lsst.codes/job/ts_tcs_ofcPython/job/develop/29/console]
       
      The following is the error from the console (#29):
      failed to register layer: Error processing tar file(exit status 1): write /opt/lsst/stack/miniconda3-4.5.4-fcd27eb/Linux64/pipe_tasks/16.0-51-gbbe9c988/tests/.tests/testCoadds-data/deepCoadd_psfMatchedWarp-00-0,0-0010.fits: no space left on device
      Could you help for this?
       
      Thank you!
       
      Te-Wei Tsai
       
       
      

      However, the jenkins-el7-1 node has simply run out of free disk space, which is unrelated to DM-17540.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've purged all of the workspaces on jenkins-el7-1 and set a 50GiB free disk space limit for jenkins to automatically take a node offline.

            Show
            jhoblitt Joshua Hoblitt added a comment - I've purged all of the workspaces on jenkins-el7-1 and set a 50GiB free disk space limit for jenkins to automatically take a node offline.
            Hide
            ttsai Te-Wei Tsai added a comment -

            The disk size has been updated in server and my Jenkins job can rerun now:

            https://ts-ci.lsst.codes/job/ts_tcs_ofcPython/job/develop/30/

            Thanks!

            Show
            ttsai Te-Wei Tsai added a comment - The disk size has been updated in server and my Jenkins job can rerun now: https://ts-ci.lsst.codes/job/ts_tcs_ofcPython/job/develop/30/ Thanks!
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm glad it was an easy fix.

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm glad it was an easy fix.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Te-Wei Tsai
                Watchers:
                Andy Clements, Eric Coughlin, Joshua Hoblitt, Rob Bovill, Te-Wei Tsai
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: