Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13433

eups.lsst.codes s3 backups failing

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      [s3backup-eups] Running shell script
      + docker run -e AWS_ACCESS_KEY_ID=**** -e AWS_SECRET_ACCESS_KEY=**** -e S3_SRC_BUCKET=**** -e S3_BACKUP_BUCKET=**** lsstsqre/s3backup:latest
      Missing optional env variable: S3_SRC_PREFIX
      Missing optional env variable: S3_BACKUP_PREFIX
      Missing optional env variable: DRYRUN
      fatal error: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.
      

      https://ci.lsst.codes/job/sqre/job/backup/job/s3backup-eups/279/console

      I suspect that the aws iam credentials were inadvertently cycled as part of a redeployment.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm unable to locate tf code to setup the backup bucket/iam user. My best guess is that for some reason, it was never committed to git and has since been lost. I also noticed that the jenkins build-s3backup job hadn't been run in months, so I added it to the the monthly container rebuild job (infra-monthly).

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm unable to locate tf code to setup the backup bucket/iam user. My best guess is that for some reason, it was never committed to git and has since been lost. I also noticed that the jenkins build-s3backup job hadn't been run in months, so I added it to the the monthly container rebuild job ( infra-monthly ).
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I found a local branch in a clone of lsst-sqre/sandbox-eups that should have been part of DM-10038, but appears to have never been pushed/PR'd – my best guess is this was an accidental omission.

            Show
            jhoblitt Joshua Hoblitt added a comment - I found a local branch in a clone of lsst-sqre/sandbox-eups that should have been part of DM-10038 , but appears to have never been pushed/PR'd – my best guess is this was an accidental omission.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Fixes have been merged, the aws iam credentials have been recreated but the new credentials haven't been deployed to the production jenkins instance yet as this would result in the jenkins master being restarted due to some parts of the puppet code not being properly idempotent.

            Show
            jhoblitt Joshua Hoblitt added a comment - Fixes have been merged, the aws iam credentials have been recreated but the new credentials haven't been deployed to the production jenkins instance yet as this would result in the jenkins master being restarted due to some parts of the puppet code not being properly idempotent.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Backups have been functioning as expected for several days now (since the production credentials were updated).

            https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Fbackup%2Fs3backup-eups/detail/s3backup-eups/296/pipeline

            Show
            jhoblitt Joshua Hoblitt added a comment - Backups have been functioning as expected for several days now (since the production credentials were updated). https://ci.lsst.codes/blue/organizations/jenkins/sqre%2Fbackup%2Fs3backup-eups/detail/s3backup-eups/296/pipeline

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel