Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21203

Research cross-database approach to bulk inserts returning autoincrement values

    XMLWordPrintable

    Details

      Description

      Using SQLAlchemy with SQLite, Oracle, and PostgreSQL backends, write a Python snippet that bulk-inserts into a table with an autoincrement field and returns to Python the values of the autoincrement field.  This operation must be safe in the present of concurrent inserts into the same table.

      My understanding is that:

      • PostgreSQL can do this via INSERT ... RETURNING, and SQLAlchemy already supports that syntax.
      • Oracle can do this via INSERT ... RETURNING, but the way those results are returned is different from how the PostgreSQL native Python driver does it, and SQLAlchemy does not support that syntax, at least for bulk inserts.
      • SQLite cannot do this for bulk inserts at all, unless you either:
        • rely on exclusive locking and do arithmetic on the last inserted ID and the number of rows (scary, to say the least - and I'd really like to stop using exclusive locking)
        • select from the sequence in advance and then use those values explicitly in the insert (at least, I think this ought to work - but it definitely needs to be tested).

      I don't think this operation needs to also include the conflict-resolution options of DM-21201, but it's possible we'll identify a need for that in the future.

      I do think we'll want a guarantee that the order of returned IDs match the order of the rows provided, or have some other way to connect them if that's not true.

       

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Team Data Access and Database [ 10204 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-21231 [ DM-21231 ]
            salnikov Andy Salnikov made changes -
            Assignee Andy Salnikov [ salnikov ]
            salnikov Andy Salnikov made changes -
            Epic Link DM-20082 [ 318933 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-21770 [ DM-21770 ]
            salnikov Andy Salnikov made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            salnikov Andy Salnikov made changes -
            Story Points 4
            salnikov Andy Salnikov made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Andy Salnikov, Christopher Stephens [X] (Inactive), Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.