Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21268

GenericMap can't distinguish int/float sizes in Python

    Details

    • Type: Bug
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: afw
    • Labels:

      Description

      If a GenericMap is given a value created using one of Numpy's fixed-size numeric types (e.g., int32, int64, float32, float64), pybind11 will insert these using the first integer or floating-point wrapper it has. This is not a problem if the value is read in using Python, but it causes values to have unexpected types in C++.

      I've spent a day investigating this, and have learned the following:

      • In C++, GenericMap treats different-sized primitive numbers as non-convertible types. This is correct behavior, because, like all C++ containers, GenericMap supports value mutation by returning references, and a float& cannot be safely used to refer to a double (or vice versa).
      • The behavior is not affected by pybind11's noconvert flag: it behaves as if np.int32, np.int64, std::int32_t, and std::int64_t were all equivalent types. Possibly related, error messages for other failures triggered by noconvert list all integer overloads as int and all floating-point overloads as float.
      • Pybind11 provides some special APIs for dealing with numpy arrays in C++, but none for scalar types.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            https://ci.lsst.codes/job/stack-os-matrix/30447/artifact/osx.clang.py3/lsstsw/build/afw/_build.log is another example of boost::variant having trouble with primitives (though it's a bit odd that long is not considered equivalent to either std::int32_t or std::int64_t).

            The non-equivalence has been explained – int, long, and long long are always considered distinct types (something carried over from C?), so on a system where int32_t is aliased to int and int64_t is aliased to long long, long will not match either despite being either 32 or 64 bits long.

            Show
            krzys Krzysztof Findeisen added a comment - - edited https://ci.lsst.codes/job/stack-os-matrix/30447/artifact/osx.clang.py3/lsstsw/build/afw/_build.log is another example of boost::variant having trouble with primitives (though it's a bit odd that long is not considered equivalent to either std::int32_t or std::int64_t ) . The non-equivalence has been explained – int , long , and long long are always considered distinct types (something carried over from C?), so on a system where int32_t is aliased to int and int64_t is aliased to long long , long will not match either despite being either 32 or 64 bits long.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: