Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19807

Avoid savepoints in Gen3 ingest and bulk conversions

    XMLWordPrintable

    Details

      Description

      The savepoints used to implement nested transactions are taking the bulk of the time in raw and calibration in Oracle.  See if we can implement lighter-weight, Python-side nested transactions for at least some operations.

        Attachments

          Issue Links

            Activity

            No builds found.
            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Epic Link DM-16771 [ 235477 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            jbosch Jim Bosch added a comment -

            It looks like it will be tricky to remove savepoints without first reimplementing operations that involve catching and ignoring certain constraint violations.   For SQLite, that involves DM-17419, and for Oracle, it involves using MERGE, I believe.

            Show
            jbosch Jim Bosch added a comment - It looks like it will be tricky to remove savepoints without first reimplementing operations that involve catching and ignoring certain constraint violations.   For SQLite, that involves DM-17419 , and for Oracle, it involves using MERGE, I believe.
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-17419 [ DM-17419 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-16771 [ 235477 ] DM-20166 [ 323037 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-20166 [ 323037 ] DM-21254 [ 414685 ]
            Hide
            jbosch Jim Bosch added a comment -

            I've decided that while this ticket is a good description of a problem, it doesn't reflect how we'll likely address that problem.  I'm zeroing the story points and blocking it on DM-21768, which will probably take care of it (if not, DM-21770 will).  We can then just mark this ticket as done when its blocker is.

            Show
            jbosch Jim Bosch added a comment - I've decided that while this ticket is a good description of a problem, it doesn't reflect how we'll likely address that problem.  I'm zeroing the story points and blocking it on DM-21768 , which will probably take care of it (if not, DM-21770 will).  We can then just mark this ticket as done when its blocker is.
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-21768 [ DM-21768 ]
            jbosch Jim Bosch made changes -
            Story Points 4 0
            jbosch Jim Bosch made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            yusra Yusra AlSayyad made changes -
            Epic Link DM-21254 [ 414685 ] DM-22586 [ 427653 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-22586 [ 427653 ] DM-23737 [ 431393 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.