Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25818

S3Datastore tests existence before writing

    XMLWordPrintable

    Details

    • Story Points:
      4
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      https://github.com/lsst/daf_butler/blob/master/python/lsst/daf/butler/datastores/s3Datastore.py#L199-L205 checks to see if an object exists before writing it. With AWS and likely other object stores, relaxed consistency guarantees mean that there could be a race condition here if multiple simultaneous jobs are trying to write. Furthermore, the additional API call is pretty wasteful as it is almost always going to return false, and it may help trigger rate limiting.

      We should consider whether we can live with overwriting of a dataset.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-24247 butler validation error in ci_hsc_gen3

          • Done
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26278 Investigate situation of deleting and adding to the same butler collection simultaneously

          • To Do
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26131 Pipetask should handle SIGINT and back out of database transactions

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-22201 S3 "Eventual Consistency" model can interfere with ingest functionality.

          • To Do

            Activity

            Ascending order - Click to sort in descending order
            ktl Kian-Tat Lim created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Labels gen3-middleware
            tjenness Tim Jenness made changes -
            Watchers Kian-Tat Lim [ Kian-Tat Lim ] Dino Bektesevic, Kian-Tat Lim [ Dino Bektesevic, Kian-Tat Lim ]
            tjenness Tim Jenness made changes -
            Watchers Dino Bektesevic, Kian-Tat Lim [ Dino Bektesevic, Kian-Tat Lim ] Dino Bektesevic, Jim Bosch, Kian-Tat Lim [ Dino Bektesevic, Jim Bosch, Kian-Tat Lim ]
            swinbank John Swinbank made changes -
            Team Architecture [ 10304 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-22201 [ DM-22201 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-26131 [ DM-26131 ]
            tjenness Tim Jenness made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Reviewers Kian-Tat Lim [ ktl ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ktl Kian-Tat Lim made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            tjenness Tim Jenness made changes -
            Link This issue is triggering DM-26278 [ DM-26278 ]
            tjenness Tim Jenness made changes -
            Assignee Tim Jenness [ tjenness ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-24247 [ DM-24247 ]
            tjenness Tim Jenness made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            tjenness Tim Jenness made changes -
            Story Points 4

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              ktl Kian-Tat Lim
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Dino Bektesevic, Hsin-Fang Chiang, Jim Bosch, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.