Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7115

Enable pybind11 wrapped functions with container arguments to accept any sequence type

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      DRP F16-5
    • Team:
      Data Release Production

      Description

      Currently pybind11 uses an explicit mapping:

      • std::vector, std::list <-> list
      • std::tuple(, std::pair) <-> tuple

      The problem of that lies in the two-way conversion. This means that wrapped C++ functions with vector or list arguments cannot accept a Python tuple and functions with tuple (or pair) parameters cannot accept a Python list. The same holds true for any other sequence type.

      This ticket aims to fix this in upstream pybind11.

        Attachments

          Activity

          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Turned out to be easy for std::vector but potentially harder for the other C++ types because they are deeper integrated in the internals of pybind11.
          So, proposing to limit this ticket to std::vector. This is also by far the most used case in our codebase.
          This means that functions that accept a pair or tuple continue to only be callable with a Python tuple, but functions that accept a vector can accept any sequence type.
          If needed later we can always revisit.

          The PR is at https://github.com/lsst-dm/pybind11/pull/4 and is ready for submitting to upstream after internal review (but before closing this ticket).

          The main possible stumbling block I see, for acceptance upstream, is that when functions are overloaded on vector and tuple (or pair) now the vector one will shadow it if it comes first in the overload list. But I don't see a way out here. And in any case this is arguably bad design. Plus the same problem already exists between tuple and pair.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Turned out to be easy for std::vector but potentially harder for the other C++ types because they are deeper integrated in the internals of pybind11. So, proposing to limit this ticket to std::vector . This is also by far the most used case in our codebase. This means that functions that accept a pair or tuple continue to only be callable with a Python tuple, but functions that accept a vector can accept any sequence type. If needed later we can always revisit. The PR is at https://github.com/lsst-dm/pybind11/pull/4 and is ready for submitting to upstream after internal review (but before closing this ticket). The main possible stumbling block I see, for acceptance upstream, is that when functions are overloaded on vector and tuple (or pair) now the vector one will shadow it if it comes first in the overload list. But I don't see a way out here. And in any case this is arguably bad design. Plus the same problem already exists between tuple and pair.
          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Forgot to mention that this also works for std::list, not just std::vector.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Forgot to mention that this also works for std::list , not just std::vector .
          Hide
          swinbank John Swinbank added a comment -

          I have no objection to the aims of this issue or, at a superficial level, to the code changes. I don't have the expertise — or the opportunity to acquire it in the near future — to understand the details of the implementation, but I assume that the upstream pybind11 project will only accept it if they're happy with it.

          I was surprised by the lack of documentation, but it seems that upstream's attitude to this is basically "see the test suite for examples", so maybe this is adequate.

          Happy for you to go ahead and submit this upstream.

          Show
          swinbank John Swinbank added a comment - I have no objection to the aims of this issue or, at a superficial level, to the code changes. I don't have the expertise — or the opportunity to acquire it in the near future — to understand the details of the implementation, but I assume that the upstream pybind11 project will only accept it if they're happy with it. I was surprised by the lack of documentation, but it seems that upstream's attitude to this is basically "see the test suite for examples" , so maybe this is adequate. Happy for you to go ahead and submit this upstream.
          Hide
          pschella Pim Schellart [X] (Inactive) added a comment -

          Patch accepted upstream.

          Show
          pschella Pim Schellart [X] (Inactive) added a comment - Patch accepted upstream.

            People

            Assignee:
            pschella Pim Schellart [X] (Inactive)
            Reporter:
            pschella Pim Schellart [X] (Inactive)
            Reviewers:
            John Swinbank
            Watchers:
            John Swinbank, Pim Schellart [X] (Inactive), Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.