Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20286

Modernize pybind11 wrappers for afw.table and afw.table.io

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Templates:
    • Story Points:
      6
    • Sprint:
      AP F19-1
    • Team:
      Alert Production

      Description

      DM-17982 is blocked on circular import problems in afw (which, strictly speaking, should have been caught in DM-19575). They are only detectable if each unit test script is run by a separate pytest instance (pytest --forked does not work), but are likely to appear unexpectedly for real users. I've identified the following cycles so far:

      python/lsst/afw/typehandling/__init__.py:23: in <module>
          from ._typehandling import Storable
      python/lsst/afw/table/io/__init__.py:23: in <module>
          from .persistable import *
      python/lsst/afw/fits/__init__.py:2: in <module>
          from .pickleFits import reduceToFits, unreduceFromFits
      python/lsst/afw/fits/pickleFits.py:4: in <module>
          import lsst.afw.image
      python/lsst/afw/image/__init__.py:25: in <module>
          from .apCorrMap import *
      python/lsst/afw/image/apCorrMap/__init__.py:23: in <module>
          from .apCorrMap import *
      E   ImportError: generic_type: type "ApCorrMap" referenced unknown base type "lsst::afw::typehandling::Storable"
      

      python/lsst/afw/typehandling/__init__.py:23: in <module>
          from ._typehandling import Storable
      python/lsst/afw/table/__init__.py:28: in <module>
          from .aggregates import *
      python/lsst/afw/table/aggregates/__init__.py:23: in <module>
          from .aggregates import *
      python/lsst/afw/geom/__init__.py:30: in <module>
          from .polygon import *
      python/lsst/afw/geom/polygon/__init__.py:25: in <module>
          from .polygon import *
      E   ImportError: generic_type: type "Polygon" referenced unknown base type "lsst::afw::typehandling::Storable"
      

      After discussing on #dm-pybind-11, I suspect that the most productive way to deal with the cycles is to migrate the lsst.afw.table.io package and its dependency lsst.afw.table to the lsst::utils::python::WrapperCollection framework. Hopefully doing so will fix the dependency problems associated with inheriting from Storable without requiring complete conversion of all of afw.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                John Parejko
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel