# eups.lsst.codes s3 backups failing

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
0.75
• Team:
SQuaRE

## Description

 [s3backup-eups] Running shell script + docker run -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** -e S3_SRC_BUCKET=**** -e S3_BACKUP_BUCKET=**** lsstsqre/s3backup:latest Missing optional env variable: S3_SRC_PREFIX Missing optional env variable: S3_BACKUP_PREFIX Missing optional env variable: DRYRUN fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records. 

https://ci.lsst.codes/job/sqre/job/backup/job/s3backup-eups/279/console

I suspect that the aws iam credentials were inadvertently cycled as part of a redeployment.

## Activity

Hide
Joshua Hoblitt added a comment -

I'm unable to locate tf code to setup the backup bucket/iam user. My best guess is that for some reason, it was never committed to git and has since been lost. I also noticed that the jenkins build-s3backup job hadn't been run in months, so I added it to the the monthly container rebuild job (infra-monthly).

Show
Joshua Hoblitt added a comment - I'm unable to locate tf code to setup the backup bucket/iam user. My best guess is that for some reason, it was never committed to git and has since been lost. I also noticed that the jenkins build-s3backup job hadn't been run in months, so I added it to the the monthly container rebuild job ( infra-monthly ).
Hide
Joshua Hoblitt added a comment -

I found a local branch in a clone of lsst-sqre/sandbox-eups that should have been part of DM-10038, but appears to have never been pushed/PR'd – my best guess is this was an accidental omission.

Show
Joshua Hoblitt added a comment - I found a local branch in a clone of lsst-sqre/sandbox-eups that should have been part of DM-10038 , but appears to have never been pushed/PR'd – my best guess is this was an accidental omission.
Hide
Joshua Hoblitt added a comment -

Fixes have been merged, the aws iam credentials have been recreated but the new credentials haven't been deployed to the production jenkins instance yet as this would result in the jenkins master being restarted due to some parts of the puppet code not being properly idempotent.

Show
Joshua Hoblitt added a comment - Fixes have been merged, the aws iam credentials have been recreated but the new credentials haven't been deployed to the production jenkins instance yet as this would result in the jenkins master being restarted due to some parts of the puppet code not being properly idempotent.
Hide
Joshua Hoblitt added a comment -

Backups have been functioning as expected for several days now (since the production credentials were updated).

https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Fbackup%2Fs3backup-eups/detail/s3backup-eups/296/pipeline

Show
Joshua Hoblitt added a comment - Backups have been functioning as expected for several days now (since the production credentials were updated). https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Fbackup%2Fs3backup-eups/detail/s3backup-eups/296/pipeline

## People

• Assignee:
Joshua Hoblitt
Reporter:
Joshua Hoblitt
Watchers:
Joshua Hoblitt