Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12178

jenkins run-rebuild s3 doc upload takes > 10 hours

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None
    • Templates:
    • Story Points:
      0.375
    • Epic Link:
    • Team:
      SQuaRE

      Description

      The s3 doc push is done using the pypi awscli package command aws s3 sync. This operation has become extremely slow, and is taking over 10 hours. I suspect it is a reflection of the 10's of millions of objects in the s3 bucket that is being uploaded to and some sort of "bad" behavior of aws s3 sync, such as enumerating every object in the bucket.

        Attachments

          Activity

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Fabrice Jammes, Joshua Hoblitt, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel