Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13086

reduce s3 bandwidth charges

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      There are non-trivial aws region to region bandwidth charges on this months bill. I presume this is a consequence of the s3 bucket used for git-lfs object being in us-west-2 while the jenkins instances are all running is us-east-1 (the plan was to move most everything to us-west-2 but it never seems to be a convenient time to do it when the annual reserved instances expire). This traffic has greatly increased now that we are deleting the jenkins validate_drp job workspace(s) before each build.

      Intra-aws-region bandwidth is free, so replicating all the git-lfs objects to a bucket in the us-east-1 region and running a second git-lfs server instance will eliminate these costs.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I have enabled versioning on the s3://lsst-sqre-prod-git-lfs bucket and setup a replication policy to copy new objects to s3://lsst-sqre-prod-git-lfs-us-east-1 (which, as the name implies, is in the us-east-1 region). I have also started an object sync with the aws cli on a jenkins test env node.

            Show
            jhoblitt Joshua Hoblitt added a comment - I have enabled versioning on the s3://lsst-sqre-prod-git-lfs bucket and setup a replication policy to copy new objects to s3://lsst-sqre-prod-git-lfs-us-east-1 (which, as the name implies, is in the us-east-1 region). I have also started an object sync with the aws cli on a jenkins test env node.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The goal of ticket was done on DM-13745.

            Show
            jhoblitt Joshua Hoblitt added a comment - The goal of ticket was done on DM-13745 .

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                jhoblitt Joshua Hoblitt
                Watchers:
                Frossie Economou, Joshua Hoblitt, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel