Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11486

Automate backups for squash-db

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          Adopting a solution from https://github.com/whereisaaron/kube-backup

          After a few changes I managed to make it work with squash-deployment

          $ kubectl run --attach --rm --restart=Never kube-backup --image lsstsqre/kube-backup:latest -- --task=backup-mysql-exec --namespace=demo --pod=squash-db-1790786895-cx70z --container=mariadb
          Fetched AWS credientials from 'kube-backup' secret
          No database specified, getting database name from environment of container 'mariadb'
          Backing up MySQL database 'qadb' from container 'mariadb' in pod 'squash-db-1790786895-cx70z' to 's3://jenkins-prod-qadb.lsst.codes-backups/demo/20171004-2116/squash-db-mariadb-qadb-20171004-2116.gz'
          Backed up MySQL database 'qadb' from container 'mariadb' in pod 'squash-db-1790786895-cx70z' to 's3://jenkins-prod-qadb.lsst.codes-backups/demo/20171004-2116/squash-db-mariadb-qadb-20171004-2116.gz'
          Done
          

          Show
          afausti Angelo Fausti added a comment - Adopting a solution from https://github.com/whereisaaron/kube-backup After a few changes I managed to make it work with squash-deployment $ kubectl run --attach --rm --restart=Never kube-backup --image lsstsqre/kube-backup:latest -- --task=backup-mysql-exec --namespace=demo --pod=squash-db-1790786895-cx70z --container=mariadb Fetched AWS credientials from 'kube-backup' secret No database specified, getting database name from environment of container 'mariadb' Backing up MySQL database 'qadb' from container 'mariadb' in pod 'squash-db-1790786895-cx70z' to 's3://jenkins-prod-qadb.lsst.codes-backups/demo/20171004-2116/squash-db-mariadb-qadb-20171004-2116.gz' Backed up MySQL database 'qadb' from container 'mariadb' in pod 'squash-db-1790786895-cx70z' to 's3://jenkins-prod-qadb.lsst.codes-backups/demo/20171004-2116/squash-db-mariadb-qadb-20171004-2116.gz' Done
          Hide
          afausti Angelo Fausti added a comment - - edited

          kube-backup provides a Dockerfile which installs aws cli, kubectl, mysqldump and other dependencies and a well structured bash script where tasks to backup files or db dumps are defined.

          A new image was built and pushed to lsstsqre/kube-backup:latest

          In order to use it we just needed to create a kubernetes secret to store AWS credentials, the S3 bucket URI where we want to store the backups and a web hook URL for Slack Integration.

          Show
          afausti Angelo Fausti added a comment - - edited kube-backup provides a Dockerfile which installs aws cli , kubectl , mysqldump and other dependencies and a well structured bash script where tasks to backup files or db dumps are defined. A new image was built and pushed to lsstsqre/kube-backup:latest In order to use it we just needed to create a kubernetes secret to store AWS credentials, the S3 bucket URI where we want to store the backups and a web hook URL for Slack Integration.
          Hide
          afausti Angelo Fausti added a comment - - edited

          Automated the creation of the squash-db-backup secret and the cronjob to schedule database backups:

           
          $ make squash-db-backup
          Creating backup secret
          kubectl delete --ignore-not-found=true secrets squash-db-backup
          secret "squash-db-backup" deleted
          kubectl create secret generic squash-db-backup \
                  --from-literal=AWS_ACCESS_KEY_ID=*******
                  --from-literal=AWS_SECRET_ACCESS_KEY=******* \
                  --from-literal=S3_BUCKET=******* \
                  --from-literal=SLACK_WEBHOOK=******** 
          secret "squash-db-backup" created
          Schedule periodic backups for squash-db
          kubectl delete --ignore-not-found=true cronjob squash-db-backup
          cronjob "squash-db-backup" deleted
          kubectl create -f kubernetes/squash-db-backup.yaml
          cronjob "squash-db-backup" created
          

          Show
          afausti Angelo Fausti added a comment - - edited Automated the creation of the squash-db-backup secret and the cronjob to schedule database backups:   $ make squash-db-backup Creating backup secret kubectl delete --ignore-not-found=true secrets squash-db-backup secret "squash-db-backup" deleted kubectl create secret generic squash-db-backup \ --from-literal=AWS_ACCESS_KEY_ID=******* --from-literal=AWS_SECRET_ACCESS_KEY=******* \ --from-literal=S3_BUCKET=******* \ --from-literal=SLACK_WEBHOOK=******** secret "squash-db-backup" created Schedule periodic backups for squash-db kubectl delete --ignore-not-found=true cronjob squash-db-backup cronjob "squash-db-backup" deleted kubectl create -f kubernetes/squash-db-backup.yaml cronjob "squash-db-backup" created
          Hide
          afausti Angelo Fausti added a comment - - edited

          Schedule squash-db backups each 10 min for testing

          $ kubectl get cronjob
          NAME               SCHEDULE       SUSPEND   ACTIVE    LAST-SCHEDULE
          squash-db-backup   */10 * * * *   False     1         Thu, 05 Oct 2017 10:20:00 -0700
          

          Show
          afausti Angelo Fausti added a comment - - edited Schedule squash-db backups each 10 min for testing $ kubectl get cronjob NAME SCHEDULE SUSPEND ACTIVE LAST-SCHEDULE squash-db-backup */10 * * * * False 1 Thu, 05 Oct 2017 10:20:00 -0700
          Hide
          afausti Angelo Fausti added a comment -

          Automate backups for squash-db

          https://github.com/lsst-sqre/squash-db/pull/4

          and minimal changes required in kube-backup utility to work with the squash deployment:

          https://github.com/lsst-sqre/kube-backup/pull/1

          Show
          afausti Angelo Fausti added a comment - Automate backups for squash-db https://github.com/lsst-sqre/squash-db/pull/4 and minimal changes required in kube-backup utility to work with the squash deployment: https://github.com/lsst-sqre/kube-backup/pull/1
          Hide
          afausti Angelo Fausti added a comment -

          Tested in `squash-prod` deployment, slack integration is working too.

          Show
          afausti Angelo Fausti added a comment - Tested in `squash-prod` deployment, slack integration is working too.

            People

            Assignee:
            afausti Angelo Fausti
            Reporter:
            afausti Angelo Fausti
            Reviewers:
            Jonathan Sick, Joshua Hoblitt
            Watchers:
            Angelo Fausti, Jonathan Sick, Joshua Hoblitt
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.