pybind11 upstream only supports non-public dtors via a unique_ptr no-op deleter.  That doesn't work for types that have a protected destructor because they're used as nonpolymorphic base classes (e.g. ImageBaseFitsReader), because pybind11 also requires that subclasses use the same holder type as their base class.

pybind11 really shouldn't be requiring any holder type for classes that are never instantiated.

Something at least closely related has been reported as https://github.com/pybind/pybind11/issues/1178.