Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9437

automate jenkins workspace cleanup

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      All jenkins jobs that use lsstsw consume couscous amounts of disk space and require periodic manual workspace purging. This should either be moved to a jenkins plugin or an automated script.

        Attachments

          Issue Links

            Activity

            Show
            jhoblitt Joshua Hoblitt added a comment - https://gist.github.com/rb2k/8372402
            Show
            jhoblitt Joshua Hoblitt added a comment - https://julienprog.wordpress.com/2016/04/14/clean-up-jenkins-workspaces/
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            This is the basic list of requirements I've formulated for a cleanup script:

            * cleanup the workspace of jobs on a node when free disk space drops below
              threshold X
             
            * must work with jobs nested in cloudbee's folders to an arbitrary depth.  This
              appears to be a limitation of many of the idenitied existing cleanup scripts.
             
            * node marked as offline during cleanup to prevent new builds of a job from
              being started while the workspace cleanup is in progress
             
            * must not operate on nodes marked as offline by the operator
             
            * must not interfer with running builds/pipelines
             
            * must be possible to exclude certain jobs, nodes, or job+node combinations
              from cleanup. Eg, `run-rebuild` on lsst-dev
            
            

            After looking a few existing cleanup scripts (there were too many options to do a comprehensive survey), I ended using https://gist.github.com/akomakom/481507c0dd79ec52a395 as a starting point and doing some modifications / cleanup work.

            This was deployed to production yesterday after testing in a sandbox env and appears to be working as expected, thus far.

            Show
            jhoblitt Joshua Hoblitt added a comment - This is the basic list of requirements I've formulated for a cleanup script: * cleanup the workspace of jobs on a node when free disk space drops below threshold X   * must work with jobs nested in cloudbee's folders to an arbitrary depth. This appears to be a limitation of many of the idenitied existing cleanup scripts.   * node marked as offline during cleanup to prevent new builds of a job from being started while the workspace cleanup is in progress   * must not operate on nodes marked as offline by the operator   * must not interfer with running builds/pipelines   * must be possible to exclude certain jobs, nodes, or job+node combinations from cleanup. Eg, `run-rebuild` on lsst-dev After looking a few existing cleanup scripts (there were too many options to do a comprehensive survey), I ended using https://gist.github.com/akomakom/481507c0dd79ec52a395 as a starting point and doing some modifications / cleanup work. This was deployed to production yesterday after testing in a sandbox env and appears to be working as expected, thus far.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel