Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12780

prune eups.lsst.code s3 backups

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None
    • Templates:
    • Story Points:
      2
    • Epic Link:
    • Team:
      SQuaRE

      Description

      The daily backups for s3://eups.lsst.codes were growing unbounded. Backups have been prune down to retain only the 1st day of each month prior to the current month. Eg.

      #!/bin/bash
       
      #set -e
      set -o xtrace
       
      for m in {4..10}; do
        mon=$(printf "%02d" $m)
        for d in {2..30}; do
          day=$(printf "%02d" $d)
          aws s3 rm --recursive "s3://eups.lsst.codes-backups/2017/${mon}/${day}" &
        done
      done
      

      Either a rotation script is needed or s3 object expiration needs to be set. Ie., backups on every day but the first of the month have a 30 day expiration set on the objects.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel