This is a proposal to change our approach to organizing pybind11 modules, intended to address three issues:
- Circular dependencies between types (especially in afw) are not handled well in our current approach, making the current codebase extremely fragile. (This is currently blocking DM-15749.)
- The total size of our module binaries is unnecessarily large, almost entirely because we currently build multiple modules per package with significant duplication. This may contribute to slow link times though that has not been demonstrated.
- repr and pickle (in particular) currently see module strings that are too long because they include implementation details (e.g. "lsst.daf.base.propertyContainer.propertyList.PropertyList").
The proposal addresses these issues by:
- deferring the execution of signature-wrapping code in a module until after all type-wrapping code has been executed (effectively the pybind11 equivalent of forward-declaring all types);
- building one module per package (from multiple source files linked together at build time, allowing duplicate symbols to be discarded);
- updating the __module__ attributes of all wrapped type objects.
These are facilitated by the new WrapperCollection helper class in utils, which should be used in all new pybind11 code.
These changes (the one-module-per-package convention in particular) necessitate a change to our pybind11 file/directory layout. But they also make the original motivation for that layout (encapsulation of individual wrappers) impossible to achieve, so I'm proposing a larger-than-necessary change to those layout conventions, with different motivations:
- adhere to the general Python convention that private modules have leading underscores;
- maximize the similarity in filenames between related headers, C++ source files, and pybind11 source files, including avoiding unnecessary differences in case.
The details of the proposal are probably best seen in action, via the updated developer guide pybind11 how-to section and the accompanying pybind11 example repo that provides full source code and directory structure for it.
The new WrapperCollection class that makes all of this work can be seen in the tickets/DM-10384 branch of utils.
A much more extensive, real-world example can be seen in the same branch of geom.