Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12174

community.lsst.org backups not being sent to S3

    Details

      Description

      The daily backups on community.lsst.org should be uploaded to S3 (builtin Discourse functionality), but they aren't. We are doing weekly DigitalOcean snapshots that mitigate some of the data loss risk, but it would be better if we copied those daily backups.

      Check if it's a permissions issue that can be fixed. Alternatively, set up a cron service that {{scp}}s the backup to S3.

        Attachments

          Issue Links

            Activity

            Hide
            jsick Jonathan Sick added a comment -

            See https://medium.freecodecamp.org/how-to-set-up-an-internal-team-forum-in-half-a-day-using-discourse-b13588d907fe

            They recommend creating an IAM user with the following inline policy:

            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "Stmt1506240388000",
                        "Effect": "Allow",
                        "Action": [
                            "s3:*"
                        ],
                        "Resource": [
                            "arn:aws:s3:::**discourse-upload**",
                            "arn:aws:s3:::**discourse-upload**/*"
                        ]
                    },
                    {
                        "Sid": "Stmt1506240479000",
                        "Effect": "Allow",
                        "Action": [
                            "s3:*"
                        ],
                        "Resource": [
                            "arn:aws:s3:::**discourse-backup**",
                            "arn:aws:s3:::**discourse-backup**/*"
                        ]
                    }
                ]
            }
            

            Show
            jsick Jonathan Sick added a comment - See https://medium.freecodecamp.org/how-to-set-up-an-internal-team-forum-in-half-a-day-using-discourse-b13588d907fe They recommend creating an IAM user with the following inline policy: { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1506240388000", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::**discourse-upload**", "arn:aws:s3:::**discourse-upload**/*" ] }, { "Sid": "Stmt1506240479000", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::**discourse-backup**", "arn:aws:s3:::**discourse-backup**/*" ] } ] }
            Hide
            jsick Jonathan Sick added a comment -
            • Created a brand new IAM user with the policy documented in https://meta.discourse.org/t/s3-backup-permissions/63257/6
            • Turned off all public access to the backups bucket. This prevents any possibility of a backup object becoming public.
            • Ended up having to reboot the Discourse server itself to resolve a timeclock drift that was creating an issue with the S3 client in Discourse.
            • I still want to upgrade the Ubuntu LTS version of the Discourse server. I think the best way to do that is to assign a floating IP so that I can run multiple instances of Discourse based on cloned snapshots. That way I can do the OS upgrade on the non-live version, and then do a cutover by switching the floating IP.
            Show
            jsick Jonathan Sick added a comment - Created a brand new IAM user with the policy documented in https://meta.discourse.org/t/s3-backup-permissions/63257/6 Turned off all public access to the backups bucket. This prevents any possibility of a backup object becoming public. Ended up having to reboot the Discourse server itself to resolve a timeclock drift that was creating an issue with the S3 client in Discourse. I still want to upgrade the Ubuntu LTS version of the Discourse server. I think the best way to do that is to assign a floating IP so that I can run multiple instances of Discourse based on cloned snapshots. That way I can do the OS upgrade on the non-live version, and then do a cutover by switching the floating IP.

              People

              • Assignee:
                jsick Jonathan Sick
                Reporter:
                jsick Jonathan Sick
                Watchers:
                Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel