Pim Schellart [X] recently spent some time exploring a few alternatives to Swig, by building wrappers for some example C++ interfaces I put together. The results for Cython and Pybind11 have been written up in technical reports (DMTN-13 and DMTN-14, respectively), and CFFI was rejected early due to the need to write pure C wrappers for every C++ interface.
I encourage everyone to read those notes and form their own opinions, but I think they make a very strong case for switching to Pybind11 and essentially reject Cython. In particular:
- Pybind11 is essentially a full rewrite of Boost.Python, but as a dependency-free header-only library. It's got all the nice support for edge cases and careful memory management that Boost.Python has, better support for the C++ standard library, and the extensibility that comes from being able to just write customization code in C++ without going through a code generator. I don't have a good sense for how widely adopted it is (the fact that it's been around less than a year puts a pretty strong upper bound on that), but the main developer is very active and responsive, and it has excellent documentation.
- Cython has a ton of market share (mostly, I think, because it's very good at adding a small amount of compiled code to Python), but its C++ support is immature and they've made some architectural decisions that make me doubt it will ever really be any good. I'd put their ceiling (for wrapping C++) as perhaps only slightly better than Swig, and it's really not anywhere close to Swig now. This is a disappointment - Cython is what Astropy uses, as well as a significant fraction of the scientific community. But I'd rather use Swig or even the raw Python C API to wrap C++ at this point.
With that in mind, I think our choices come down to switching to Pybind11 or rewriting much of our Swig to improve our dependency handling. Given that even the latter would still be a significant amount of work, and I think Pybind11 is the better choice from a (preliminary) technical standpoint, I'd like to propose that we have Pim Schellart [X] spend some fraction of his time over the next few months actually converting Science Pipelines code (from the bottom up) to use Pybind11 on a branch. The intent is that this would inform a later decision around the time of the AHM on whether to convert the rest of the stack or throw away the branch.
All I'm proposing right now is that we devote some of Pim's time to this project; I'd like to allocate enough effort that we have a reasonable shot at getting through much or all of afw, but his actual pace will tell us quite a bit about the cost of a more complete conversion.
One reason I'm attracted to Pybind11 is that we do want to spend more effort defining Pythonic interfaces - this is easier to do in Pybind11, I think, and wanting custom-crafted interfaces negates much of the automatic-interface-generation advantages of Swig. But I'm not proposing that we make any such changes while converting to Pybind11; I think it's much easier if we try to maintain the same Python interfaces whenever possible at this point, and deal with making them more Pythonic in the future.