[s3backup-eups] Running shell script
+ docker run -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** -e S3_SRC_BUCKET=**** -e S3_BACKUP_BUCKET=**** lsstsqre/s3backup:latest
Missing optional env variable: S3_SRC_PREFIX
Missing optional env variable: S3_BACKUP_PREFIX
Missing optional env variable: DRYRUN
fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
I suspect that the aws iam credentials were inadvertently cycled as part of a redeployment.
point in time backups for eups.lsst.codes s3 bucket
I'm unable to locate tf code to setup the backup bucket/iam user. My best guess is that for some reason, it was never committed to git and has since been lost. I also noticed that the jenkins build-s3backup job hadn't been run in months, so I added it to the the monthly container rebuild job (infra-monthly).
I found a local branch in a clone of lsst-sqre/sandbox-eups that should have been part of DM-10038, but appears to have never been pushed/PR'd – my best guess is this was an accidental omission.
Fixes have been merged, the aws iam credentials have been recreated but the new credentials haven't been deployed to the production jenkins instance yet as this would result in the jenkins master being restarted due to some parts of the puppet code not being properly idempotent.
Backups have been functioning as expected for several days now (since the production credentials were updated).