Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3921

Create and deploy a git-lfs prototype

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Create and deploy a git-lfs-s3 server.

      High level requirements:

      • The server should use github to authenticate users. Any github user who is a member of the lsst organization has write access. This means they can push objects to the git-lfs server.
      • The server should allow for anonymous read access. This means anyone can clone, pull, fetch, etc.
      • Use an S3 compliant API to store objects.
      • Simple, well defined method to redeploy.
      • Uses https.
      • Backs up to Amazon Glacier (or similar) periodically. The data should be "slow" so backing up approximately once a day is OK.

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            If ceph isn't used as a back end, Aws::config shouldn't be called directly.

            if GitLfsS3::Application.settings.ceph_s3
              Aws.config.update(
                endpoint: ENV['LFS_CEPH_ENDPOINT'],
                access_key_id: ENV['AWS_ACCESS_KEY_ID'],
                secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
                force_path_style: true,
                region: 'us-east-1',
                # ssl_ca_bundle: '/usr/local/etc/openssl/cert.pem' # Required for brew install on a mac.
              )     
            end
            

            Show
            jhoblitt Joshua Hoblitt added a comment - If ceph isn't used as a back end, Aws::config shouldn't be called directly. if GitLfsS3::Application.settings.ceph_s3 Aws.config.update( endpoint: ENV[ 'LFS_CEPH_ENDPOINT' ], access_key_id: ENV[ 'AWS_ACCESS_KEY_ID' ], secret_access_key: ENV[ 'AWS_SECRET_ACCESS_KEY' ], force_path_style: true , region: 'us-east-1' , # ssl_ca_bundle: '/usr/local/etc/openssl/cert.pem' # Required for brew install on a mac. ) end
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            s3 objects are created without public read permissions. The only "grantee" is the username the aws credentials are associated with. This may have been caused by my comment on acl setting here https://github.com/lsst-sqre/git-lfs-s3/pull/1#discussion-diff-42070724

            Show
            jhoblitt Joshua Hoblitt added a comment - s3 objects are created without public read permissions. The only "grantee" is the username the aws credentials are associated with. This may have been caused by my comment on acl setting here https://github.com/lsst-sqre/git-lfs-s3/pull/1#discussion-diff-42070724
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've seen really odd client behavior with the s3 bucket is missing.

            Username for 'https://gitlfsblobs.s3-us-west-2.amazonaws.com': Password for 'htt(196 of 147 files) 122.94 MB / 92.04 MB                                        U(197 of 147 files) 124.14 MB / 92.04 MB                                        U(198 of 147 files) 124.14 MB / 92.04 MB      
            

            Show
            jhoblitt Joshua Hoblitt added a comment - I've seen really odd client behavior with the s3 bucket is missing. Username for 'https://gitlfsblobs.s3-us-west-2.amazonaws.com' : Password for 'htt( 196 of 147 files) 122.94 MB / 92.04 MB U( 197 of 147 files) 124.14 MB / 92.04 MB U( 198 of 147 files) 124.14 MB / 92.04 MB
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I've now run into the github api request rate limit from the lfs server.

            App 24366 stderr: 2015-10-16 13:24:48 - Octokit::TooManyRequests - GET https://api.github.com/user: 403 - API rate limit exceeded for jhoblitt. // See: https://developer.github.com/v3/#rate-limiting:
            

            Show
            jhoblitt Joshua Hoblitt added a comment - I've now run into the github api request rate limit from the lfs server. App 24366 stderr: 2015 - 10 - 16 13 : 24 : 48 - Octokit::TooManyRequests - GET https: //api.github.com/user: 403 - API rate limit exceeded for jhoblitt. // See: https://developer.github.com/v3/#rate-limiting:
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            Done.

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - Done.

              People

              • Assignee:
                jmatt J Matt Peterson [X] (Inactive)
                Reporter:
                jmatt J Matt Peterson [X] (Inactive)
                Reviewers:
                Frossie Economou, Jonathan Sick, Joshua Hoblitt
                Watchers:
                Frossie Economou, J Matt Peterson [X] (Inactive), Jonathan Sick, Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Summary Panel