Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19807

Avoid savepoints in Gen3 ingest and bulk conversions

    XMLWordPrintable

    Details

      Description

      The savepoints used to implement nested transactions are taking the bulk of the time in raw and calibration in Oracle.  See if we can implement lighter-weight, Python-side nested transactions for at least some operations.

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21768 Vectorize dataset insert API

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17419 Resolve upsert order in SQLite Registry

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            It looks like it will be tricky to remove savepoints without first reimplementing operations that involve catching and ignoring certain constraint violations.   For SQLite, that involves DM-17419, and for Oracle, it involves using MERGE, I believe.

            Show
            jbosch Jim Bosch added a comment - It looks like it will be tricky to remove savepoints without first reimplementing operations that involve catching and ignoring certain constraint violations.   For SQLite, that involves DM-17419 , and for Oracle, it involves using MERGE, I believe.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I've decided that while this ticket is a good description of a problem, it doesn't reflect how we'll likely address that problem.  I'm zeroing the story points and blocking it on DM-21768, which will probably take care of it (if not, DM-21770 will).  We can then just mark this ticket as done when its blocker is.

            Show
            jbosch Jim Bosch added a comment - I've decided that while this ticket is a good description of a problem, it doesn't reflect how we'll likely address that problem.  I'm zeroing the story points and blocking it on DM-21768 , which will probably take care of it (if not, DM-21770 will).  We can then just mark this ticket as done when its blocker is.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.