Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18839

Remove explicit registry close in the butler

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: daf_butler
    • Labels:
      None

      Description

      In working with oracle, exceptions were being thrown related to connections not being closed properly, leading to the addition of the _del_ method in the butler to make sure the connections were properly cleaned up in registry held by the butler.

      This however had the side effect of creating an exception when a butler was used in a single threaded way. The temporary solution to that was to add an explicit close method to the butler to make sure everything was finalized before garbage collection (as _del_ is not predicable in the timing or ordering when it is evoked.

      This ticket will explore why this is all necessary, and address what is causing the unexpected behavior.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment -

            After spending many many hours reading through the code in daf_butler, python multiprocessing, and sqlalchemy I was able to trace this down to an issue with the connection pool in sqlalchemy. We were using the NullPool to handle connections, as we only expected to need one connection per task. However this has an issue where the connection is closed after use, and then the pool attempts to do a rollback and close on all connections that the pool created when the pool is closed. This lead to the Exceptions related to attempting to do work on closed connections.

            The best solution seems to be using the default QueuePool in the Oracle registry, and restricting the pool of connections to one. This lets the pool handle the lifetime on the connection completely.

            Show
            nlust Nate Lust added a comment - After spending many many hours reading through the code in daf_butler, python multiprocessing, and sqlalchemy I was able to trace this down to an issue with the connection pool in sqlalchemy. We were using the NullPool to handle connections, as we only expected to need one connection per task. However this has an issue where the connection is closed after use, and then the pool attempts to do a rollback and close on all connections that the pool created when the pool is closed. This lead to the Exceptions related to attempting to do work on closed connections. The best solution seems to be using the default QueuePool in the Oracle registry, and restricting the pool of connections to one. This lets the pool handle the lifetime on the connection completely.
            Hide
            jbosch Jim Bosch added a comment -

            Please fix the daf_butler commit message.  Otherwise looks great; thanks for tracking this down!

            Show
            jbosch Jim Bosch added a comment - Please fix the daf_butler commit message.  Otherwise looks great; thanks for tracking this down!

              People

              • Assignee:
                nlust Nate Lust
                Reporter:
                nlust Nate Lust
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, Nate Lust
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel