Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-40308

Evaluate nanobind as an alternative to pybind11 and a fix for DM-38772



    • Architecture
    • No


      The nanobind project is a successor to pybind11 from the same original author, with the goal of emphasizing simplicity and performance while intentionally limiting functionality.

      It has the smart-pointer support we need to fix DM-38772, but it's not a drop-in replacement (even aside from having a different namespace and build/link architecture); every pybind11 module we currently have would require at least some changes.

      • The big change that looks troublesome for porting to me is that nanobind's pointer arguments reject None by default instead of turning into nullptr.
      • Another big change is that wrapped classes no longer declare a holder type, but this was already true on the pybind11 smart_holder branch, and hence the tickets/DM-38772 branches of all (or nearly all?) packages with C++ already have this change, and would be a good starting point for a nanobind conversion.

      In terms of whether nanobind is suitable, we also need to see if we can live without support for multiple inheritance (or at least whether we need to declare multiple base classes to Python) and whether nanobind's custom type-converter API is compatible with ndarray without major changes.

      I think the work plan here should look roughly like this:

      1. Set up a libnanobind EUPS package to build a shared library from the conda-forge nanobind headers.  We might be able to get this replaced by a conda-forge shared library eventually, but we only if we can convince upstream that it's a good idea (I think it is, but I concede that there are downsides), and this doesn't need to be a priority.
      2. Add a nanobind header to ndarray with tests, and cut a new conda release of ndarray.  This can be done in parallel with (1).
      3. Convert sphgeom, using libnanobind from EUPS in the SCons build but not the pip packaging.
      4. Convert base and pex_exceptions, using libnanobind from EUPS.
      5. Convert cpputils, geom, daf_base, and afw adding porting support code (maybe even a full looks-like-pybind11 shim layer) to cpputils as we learn what would be useful (packages in previous steps don't depend on cpputils and couldn't make use of this anyway).
      6. RFC the plan to convert the rest of the stack if all has gone well so far.
      7. Finish the job.




          jbosch Jim Bosch created issue -
          tjenness Tim Jenness made changes -
          Field Original Value New Value
          Team Architecture [ 10304 ]
          Labels SciencePipelines
          wittgen Matthias Wittgen made changes -
          Assignee Matthias Wittgen [ wittgen ]
          wittgen Matthias Wittgen made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]


            wittgen Matthias Wittgen
            jbosch Jim Bosch
            Eli Rykoff, Jim Bosch
            0 Vote for this issue
            2 Start watching this issue




                No builds found.