Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21314

Investigate GC problems with Storable

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Story Points:
      6
    • Sprint:
      AP F19-5 (October)
    • Team:
      Alert Production

      Description

      DM-19582 revealed that a Python subclass of lsst.afw.typehandling.Storable may be garbage collected while C++ code still has a pointer to it (for example, if the Python object is constructed as a "temporary" that is never bound to a name). When this happens, the C++ Storable does not segfault, but instead behaves as if none of Storable's methods were overridden.

      This is a known bug in pybind11, but two years of discussion have failed to converge to a solution. There are several workarounds posted on GitHub; I briefly tried the one listed in pybind11#1546 (which most closely matches our case) but wasn't able to fix the problem.

      Investigate this issue more systematically and determine if there is any workaround that works in our case. This problem is unlikely to be fixed upstream, and being unable to use Python subclasses of Storable from C++ will significantly impair our persistence plans.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            The unit test test_Storable.py:PythonStorableTestSuite.testGarbageCollection reproduces this bug reliably, at least on my computer.

            Show
            krzys Krzysztof Findeisen added a comment - The unit test test_Storable.py:PythonStorableTestSuite.testGarbageCollection reproduces this bug reliably, at least on my computer.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Progress report, by workaround:

            • pybind11#1546: no effect; can't figure out why.
            • pybind11#1389, modified "trampoline" class: cannot be done in our code because our trampoline needs to support C++ subclasses of Storable.
            • pybind11#1389, custom shared pointer: works for Python classes that inherit directly from Storable. Inheriting from C++ subclasses of Storable not yet tested.
            • pybind11#1333: not yet tested (I have only a rough description to go by).
            • pybind11#1145: cannot be done in our code because most Storable classes need multiple inheritance on the C++ side.
            Show
            krzys Krzysztof Findeisen added a comment - - edited Progress report, by workaround: pybind11#1546 : no effect; can't figure out why. pybind11#1389 , modified "trampoline" class: cannot be done in our code because our trampoline needs to support C++ subclasses of Storable . pybind11#1389 , custom shared pointer: works for Python classes that inherit directly from Storable . Inheriting from C++ subclasses of Storable not yet tested . pybind11#1333 : not yet tested (I have only a rough description to go by). pybind11#1145 : cannot be done in our code because most Storable classes need multiple inheritance on the C++ side.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Jim Bosch, sorry for the timing, but would you be willing to review this? It's 340 lines, but requires an in-depth understanding of pybind11.

            Show
            krzys Krzysztof Findeisen added a comment - Hi Jim Bosch , sorry for the timing, but would you be willing to review this? It's 340 lines, but requires an in-depth understanding of pybind11.
            Hide
            jbosch Jim Bosch added a comment -

            Looks great!  Only minor comments on the PRs.

            Show
            jbosch Jim Bosch added a comment - Looks great!  Only minor comments on the PRs.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.