Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12174

community.lsst.org backups not being sent to S3

    XMLWordPrintable

    Details

      Description

      The daily backups on community.lsst.org should be uploaded to S3 (builtin Discourse functionality), but they aren't. We are doing weekly DigitalOcean snapshots that mitigate some of the data loss risk, but it would be better if we copied those daily backups.

      Check if it's a permissions issue that can be fixed. Alternatively, set up a cron service that {{scp}}s the backup to S3.

        Attachments

          Issue Links

            Activity

            No builds found.
            jsick Jonathan Sick created issue -
            jsick Jonathan Sick made changes -
            Field Original Value New Value
            Epic Link DM-11833 [ 34328 ]
            jsick Jonathan Sick made changes -
            Epic Link DM-11833 [ 34328 ] DM-8448 [ 28064 ]
            jsick Jonathan Sick made changes -
            Summary community.lsst.org backups not being send to S3 community.lsst.org backups not being sent to S3
            Hide
            jsick Jonathan Sick added a comment -

            See https://medium.freecodecamp.org/how-to-set-up-an-internal-team-forum-in-half-a-day-using-discourse-b13588d907fe

            They recommend creating an IAM user with the following inline policy:

            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "Stmt1506240388000",
                        "Effect": "Allow",
                        "Action": [
                            "s3:*"
                        ],
                        "Resource": [
                            "arn:aws:s3:::**discourse-upload**",
                            "arn:aws:s3:::**discourse-upload**/*"
                        ]
                    },
                    {
                        "Sid": "Stmt1506240479000",
                        "Effect": "Allow",
                        "Action": [
                            "s3:*"
                        ],
                        "Resource": [
                            "arn:aws:s3:::**discourse-backup**",
                            "arn:aws:s3:::**discourse-backup**/*"
                        ]
                    }
                ]
            }
            

            Show
            jsick Jonathan Sick added a comment - See https://medium.freecodecamp.org/how-to-set-up-an-internal-team-forum-in-half-a-day-using-discourse-b13588d907fe They recommend creating an IAM user with the following inline policy: { "Version": "2012-10-17", "Statement": [ { "Sid": "Stmt1506240388000", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::**discourse-upload**", "arn:aws:s3:::**discourse-upload**/*" ] }, { "Sid": "Stmt1506240479000", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::**discourse-backup**", "arn:aws:s3:::**discourse-backup**/*" ] } ] }
            jsick Jonathan Sick made changes -
            Epic Link DM-8448 [ 28064 ] DM-25248 [ 435569 ]
            jsick Jonathan Sick made changes -
            Link This issue relates to DM-25405 [ DM-25405 ]
            jsick Jonathan Sick made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            jsick Jonathan Sick added a comment -
            • Created a brand new IAM user with the policy documented in https://meta.discourse.org/t/s3-backup-permissions/63257/6
            • Turned off all public access to the backups bucket. This prevents any possibility of a backup object becoming public.
            • Ended up having to reboot the Discourse server itself to resolve a timeclock drift that was creating an issue with the S3 client in Discourse.
            • I still want to upgrade the Ubuntu LTS version of the Discourse server. I think the best way to do that is to assign a floating IP so that I can run multiple instances of Discourse based on cloned snapshots. That way I can do the OS upgrade on the non-live version, and then do a cutover by switching the floating IP.
            Show
            jsick Jonathan Sick added a comment - Created a brand new IAM user with the policy documented in https://meta.discourse.org/t/s3-backup-permissions/63257/6 Turned off all public access to the backups bucket. This prevents any possibility of a backup object becoming public. Ended up having to reboot the Discourse server itself to resolve a timeclock drift that was creating an issue with the S3 client in Discourse. I still want to upgrade the Ubuntu LTS version of the Discourse server. I think the best way to do that is to assign a floating IP so that I can run multiple instances of Discourse based on cloned snapshots. That way I can do the OS upgrade on the non-live version, and then do a cutover by switching the floating IP.
            jsick Jonathan Sick made changes -
            Story Points 1
            Urgent? off
            jsick Jonathan Sick made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            jsick Jonathan Sick made changes -
            Epic Link DM-25248 [ 435569 ] DM-25259 [ 435603 ]

              People

              Assignee:
              jsick Jonathan Sick
              Reporter:
              jsick Jonathan Sick
              Watchers:
              Jonathan Sick
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.