Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9437

automate jenkins workspace cleanup

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      All jenkins jobs that use lsstsw consume couscous amounts of disk space and require periodic manual workspace purging. This should either be moved to a jenkins plugin or an automated script.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-12942 add a job to flush all workspaces

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            https://gist.github.com/rb2k/8372402

            Show
            jhoblitt Joshua Hoblitt added a comment - https://gist.github.com/rb2k/8372402
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            https://julienprog.wordpress.com/2016/04/14/clean-up-jenkins-workspaces/

            Show
            jhoblitt Joshua Hoblitt added a comment - https://julienprog.wordpress.com/2016/04/14/clean-up-jenkins-workspaces/
            Hide
            Permalink
            jhoblitt Joshua Hoblitt added a comment -

            This is the basic list of requirements I've formulated for a cleanup script:

            * cleanup the workspace of jobs on a node when free disk space drops below
            		 
              threshold X
            		 
             
            		 
            * must work with jobs nested in cloudbee's folders to an arbitrary depth.  This
            		 
              appears to be a limitation of many of the idenitied existing cleanup scripts.
            		 
             
            		 
            * node marked as offline during cleanup to prevent new builds of a job from
            		 
              being started while the workspace cleanup is in progress
            		 
             
            		 
            * must not operate on nodes marked as offline by the operator
            		 
             
            		 
            * must not interfer with running builds/pipelines
            		 
             
            		 
            * must be possible to exclude certain jobs, nodes, or job+node combinations
            		 
              from cleanup. Eg, `run-rebuild` on lsst-dev

            After looking a few existing cleanup scripts (there were too many options to do a comprehensive survey), I ended using https://gist.github.com/akomakom/481507c0dd79ec52a395 as a starting point and doing some modifications / cleanup work.

            This was deployed to production yesterday after testing in a sandbox env and appears to be working as expected, thus far.

            Show
            jhoblitt Joshua Hoblitt added a comment - This is the basic list of requirements I've formulated for a cleanup script: * cleanup the workspace of jobs on a node when free disk space drops below threshold X   * must work with jobs nested in cloudbee's folders to an arbitrary depth. This appears to be a limitation of many of the idenitied existing cleanup scripts.   * node marked as offline during cleanup to prevent new builds of a job from being started while the workspace cleanup is in progress   * must not operate on nodes marked as offline by the operator   * must not interfer with running builds/pipelines   * must be possible to exclude certain jobs, nodes, or job+node combinations from cleanup. Eg, `run-rebuild` on lsst-dev After looking a few existing cleanup scripts (there were too many options to do a comprehensive survey), I ended using https://gist.github.com/akomakom/481507c0dd79ec52a395 as a starting point and doing some modifications / cleanup work. This was deployed to production yesterday after testing in a sandbox env and appears to be working as expected, thus far.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Watchers:
              Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.