Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8591

Follow-up pybind11 behavior with numpy.int64s as indices in Python 3

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None

      Description

      DM-8557 introduced a workaround for an issue seen in afw's multiMatch module where a numpy.int64 could not be used as an index by SWIG under Python 3. This ticket is to see if pybind11 has a similar issue under Python 3, and introduce a fix if so.

      Suggested procedure to diagnose the numpy.int64 as index issue is:

      1. Revert commit in afw from DM-8557 called "Cast catalog indices to int as SWIG Py3 workaround"
      2. See if the afw test tests/testTableMultiMatch.py runs.

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            Pim Schellart [X] — I gave this an off-the-top-of-the-head SP estimate. Please check & refine.

            Show
            swinbank John Swinbank added a comment - Pim Schellart [X] — I gave this an off-the-top-of-the-head SP estimate. Please check & refine.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Seems about right.

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Seems about right.
            Hide
            swinbank John Swinbank added a comment -

            [swinbank@lsst-dev03 tests (master *)]$ git diff
            diff --git a/python/lsst/afw/table/multiMatch.py b/python/lsst/afw/table/multiMatch.py
            index 4dfce6614..1bf0b75cc 100644
            --- a/python/lsst/afw/table/multiMatch.py
            +++ b/python/lsst/afw/table/multiMatch.py
            @@ -244,8 +244,7 @@ class GroupView(collections.abc.Mapping):
                     groups = numpy.zeros(len(ids), dtype=object)
                     ends = list(indices[1:]) + [len(catalog)]
                     for n, (i1, i2) in enumerate(zip(indices, ends)):
            -            # casts are a work-around for DM-8557
            -            groups[n] = catalog[int(i1):int(i2)]
            +            groups[n] = catalog[i1:i2]
                         assert (groups[n].get(groupKey) == ids[n]).all()
                     return cls(catalog.schema, ids, groups)
             
            [swinbank@lsst-dev03 tests (master *)]$ python test_tableMultiMatch.py
            ..
            ----------------------------------------------------------------------
            Ran 2 tests in 6.126s
             
            OK
            

            I think we can close this one down. I'll make a tiny PR to remove the casting and the comment.

            Show
            swinbank John Swinbank added a comment - [swinbank@lsst-dev03 tests (master *)]$ git diff diff --git a/python/lsst/afw/table/multiMatch.py b/python/lsst/afw/table/multiMatch.py index 4dfce6614..1bf0b75cc 100644 --- a/python/lsst/afw/table/multiMatch.py +++ b/python/lsst/afw/table/multiMatch.py @@ -244,8 +244,7 @@ class GroupView(collections.abc.Mapping): groups = numpy.zeros(len(ids), dtype=object) ends = list(indices[1:]) + [len(catalog)] for n, (i1, i2) in enumerate(zip(indices, ends)): - # casts are a work-around for DM-8557 - groups[n] = catalog[int(i1):int(i2)] + groups[n] = catalog[i1:i2] assert (groups[n].get(groupKey) == ids[n]).all() return cls(catalog.schema, ids, groups)   [swinbank@lsst-dev03 tests (master *)]$ python test_tableMultiMatch.py .. ---------------------------------------------------------------------- Ran 2 tests in 6.126s   OK I think we can close this one down. I'll make a tiny PR to remove the casting and the comment.
            Hide
            swinbank John Swinbank added a comment -

            Hey Nate Lust, do you have time for this tiny review?

            PR: https://github.com/lsst/afw/pull/533
            Jenkins: https://ci.lsst.codes/job/stack-os-matrix/32159/ (in progress at time of writing)

            Show
            swinbank John Swinbank added a comment - Hey Nate Lust , do you have time for this tiny review? PR: https://github.com/lsst/afw/pull/533 Jenkins: https://ci.lsst.codes/job/stack-os-matrix/32159/ (in progress at time of writing)
            Hide
            nlust Nate Lust added a comment -

            Looks fine

            Show
            nlust Nate Lust added a comment - Looks fine
            Hide
            swinbank John Swinbank added a comment -

            Thanks Nate!

            Show
            swinbank John Swinbank added a comment - Thanks Nate!

              People

              Assignee:
              swinbank John Swinbank
              Reporter:
              jsick Jonathan Sick
              Reviewers:
              Nate Lust
              Watchers:
              John Swinbank, Jonathan Sick, Nate Lust, Pim Schellart [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.