Andy Salnikov, the good news is that I think pybind11 addresses your first two issues completely; I think the documentation for pybind11 is a vast improvement over Boost.Python, and the dependency obviously is.

The bad news is that I think that the rest of your criticisms still apply. But I personally disagree with a few of them, not because I disagree with your downsides but because they're tradeoffs where I like the upside. In particular, I much prefer a complex, RTTI/template-driven pure C++ library to a code generator, because I'd prefer to step through library logic in a debugger than be forced to guess about the code generator's logic (or build the generator from source and step through that). I also find that having a library in this role gives you essentially unlimited extensibility, while relying on a code generator can close off solutions to problems the code generator authors did not anticipate.

I think you're also right that RTTI-based type dispatch should have slightly higher overheard than custom dispatch code. I suspect this is small compared with the overall overheads involved in converting from C++ to Python, but I don't actually know that. However, I think using RTTI for the type conversion system makes it much more likely that the binding approach will scale up to a large number of modules, because it puts the responsibility for conveying that type information between dependent modules on the linker. Without RTTI, the bindings for a dependent package need to include some information from the bindings for all of its dependencies, and while I think it's possible that a bindings generator could make that information lightweight enough to not significantly impact the build time of the dependent module, I think most binding generators don't do this well. (Whether Swig does this well enough depends on your definition of "significant"; the way we were using it meant that we were including much more than we needed, and that piled up - but with an RTTI-based system there's actually nothing to pile up, so the problem we had with Swig is virtually impossible with pybind11 or Boost.Python).

Using RTTI also makes it much easier to define custom wrapper code; if I have a complex C++ template type that I want to convert to Python and then manipulate with the Python C API, it's easy to invoke the to-Python converter for that type since the compiler can look it up using RTTI. If the bindings generator doesn't use RTTI, looking up that converter is much harder (in Swig, it basically requires some fragile reverse engineering).

Finally, the problem that converter failures only happen at runtime is an important one. I think the degree to which this can happen could increase with pybind11, as it has will have less information at compile time to catch such errors (this is the flip side using the linker to transmit all type information). And some type conversion failures can of course only be caught at runtime with any bindings generator, because Python is a dynamically typed language. But I think this will practically happen less in Pybind11 than with Swig, but only because Swig has the unfortunate philosophical stance that it should be possible to pass unwrapped objects to Python as opaque objects. As a result, returning an unwrapped object in Swig leads to a failure only after the returned object is used in downstream code; with Pybind11, the error will happen still happen at runtime (because the type converter may be defined in another module), but it will happen immediately when the object is returned from C++.

Jim Bosch, I'm happy that you understand all RTTI-related complexities and are prepared to handle them. To me it's still feels more complex than necessary, akin to only programming with __getattr__ in Python and ignoring regular syntax. I guess I'm OK with whatever decision works best for afw, we don't have too much code to wrap in qserv and we can even probably keep swig in qserv until we have time to learn how to do the same in pybind11.

At the risk of sounding like I'm pulling a bait-and-switch, this is probably a good time (we're about to adopt this, once we have an implementation epic to link to) for me to mention my own concerns with Pybind11. I think they're all quite surmountable, but I don't want to give the impression that there will be no speed-bumps at all:

• We haven't yet checked how exception translation will work in Pybind11. It should work essentially the same way as in Boost.Python, but it's possible it will interact with the linker in a slightly different way in a header-only library.
• Pybind11 may not publicly expose (or at least doesn't document) the kind of typemap-like functionality we'd need to convert ndarray objects to. It's still possible (you have to instantiate an undocumented traits class), but we'd like to get that (or something equivalent) blessed by the owner of Pybind11.
• Pybind11 prefers to convert STL containers to pure Python containers, rather than providing wrappers for the STL containers, while Swig does the opposite. There are advantages to both approaches (converting to pure Python is safer and more Pythonic, but less efficient especially for large containers). If we want to continue with returning wrapped STL containers, we may have to do a bit more work in providing those.

Again, all of this is included in my claim that this conversion is not hard to do, but I don't want to give the impression that it will be completely trivial. I expect that investigating these will be the first issues we go after in the implementation epic.

Implementation epic will be DM-6168.

Trial conversion in DM-6168 is completed. For full stack conversion, see RFC-216 & DM-7717.