Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22201

S3 "Eventual Consistency" model can interfere with ingest functionality.

    XMLWordPrintable

    Details

    • Team:
      External

      Description

      Hsin-Fang reports encountering an FileNotFound error during a run even though s3CheckFileExists returns True and manually navigating to the faulty Key in the Bucket clearly shows it existing.

      Two potential issues at fault here can be:

      • transient network/connectivity issue
      • S3 Eventual Consistency model

      The error appeared in Butler.ingest functionality following a Butler.put of the same exact dataset in question. It is not a far fetched idea that due to S3 consistency model the Bucket has not yet been "updated" with the newly inserted Key in the time between the key was placed there, in Butler.put, and the time the key was checked for existence, in Butler.ingest.
      In this case a waiting loop allowing the eventual consistency model to catch up would fix the problem.

      In the case of network/connectivity issue it would be nicer to have a more specific error.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dinob Dino Bektesevic
              Reporter:
              dinob Dino Bektesevic
              Reviewers:
              Kian-Tat Lim, Tim Jenness
              Watchers:
              Dino Bektesevic, Hsin-Fang Chiang, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.