Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8557

SWIG in Python 3 does not accept numpy.int64 types as an index

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: afw
    • Labels:
      None
    • Templates:
    • Story Points:
      0.6
    • Team:
      SQuaRE

      Description

      Running validateDrp.py under python 3:

      Traceback (most recent call last):                                                                                                                             │  129 ¬                                                                                                                        |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/validate_drp/tickets.DM-7328-g3a1b5eb624/bin/validateDrp.py", line 95, in <module>                          │S>130     A GroupView provides access to a catalog of grouped objects, in which the grouping is indicated by¬                  |   ~
          validate.run(args.repo, **kwargs)                                                                                                                          │S>131     a field for which all records in a group have the same value.  Once constructed, it allows operations¬               |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/validate_drp/tickets.DM-7328-g3a1b5eb624/python/lsst/validate/drp/validate.py", line 104, in run            │S>132     similar to those supported by SQL "GROUP BY", such as filtering and aggregate calculation.¬                          |   ~
          **kwargs)                                                                                                                                                  │  133     """¬                                                                                                                 |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/validate_drp/tickets.DM-7328-g3a1b5eb624/python/lsst/validate/drp/validate.py", line 204, in runOneFilter   │  134 ¬                                                                                                                        |   ~
          verbose=verbose)                                                                                                                                           │  135     @classmethod¬                                                                                                        |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/validate_drp/tickets.DM-7328-g3a1b5eb624/python/lsst/validate/drp/matchreduce.py", line 147, in __init__    │  136     def build(cls, catalog, groupField="object"):¬                                                                       |   ~
          repo, dataIds, matchRadius)                                                                                                                                │  137         """!Construct a GroupView from a concatenated catalog.¬                                                          |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/validate_drp/tickets.DM-7328-g3a1b5eb624/python/lsst/validate/drp/matchreduce.py", line 253, in _loadAndMatc│  138 ¬                                                                                                                        |   ~
      hCatalogs                                                                                                                                                      │S>139         @param[in]  catalog     Input catalog, containing records grouped by a field in which all records¬               |   ~
          allMatches = GroupView.build(matchCat)                                                                                                                     │S>140                                 in the same group have the same value.  Must be sorted by the group field.¬              |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/afw/12.1-26-g5d40821/python/lsst/afw/table/multiMatch.py", line 149, in build                               │S>141         @param[in]  groupField  Name or Key for the field that indicates groups.¬                                        |   ~
          groups[n] = catalog[i1:i2]                                                                                                                                 │  142         """¬                                                                                                             |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/afw/12.1-26-g5d40821/python/lsst/afw/table/tableLib.py", line 9595, in __getitem__                          │  143         groupKey = catalog.schema.find(groupField).key¬                                                                  |   ~
          return self.subset(start, stop, step)                                                                                                                      │  144         ids, indices = numpy.unique(catalog.get(groupKey), return_index=True)¬                                           |   ~
        File "/mnt/validate_drp/py3/lsstsw/stack/Linux64/afw/12.1-26-g5d40821/python/lsst/afw/table/tableLib.py", line 9890, in subset                               │  145         groups = numpy.zeros(len(ids), dtype=object)¬                                                                    |   ~
          return _tableLib.SimpleCatalog_subset(self, *args)                                                                                                         │  146         ends = list(indices[1:]) + [len(catalog)]¬                                                                       |   ~
      NotImplementedError: Wrong number or type of arguments for overloaded function 'SimpleCatalog_subset'.                                                         │  147         for n, (i1, i2) in enumerate(zip(indices, ends)):¬                                                               |   ~
        Possible C/C++ prototypes are:                                                                                                                               │  148             print(i1, i2, type(i1), type(i2))¬                                                                           |   ~
          lsst::afw::table::SortedCatalogT< lsst::afw::table::SimpleRecord >::subset(ndarray::Array< bool const,1 > const &) const                                   │  149             groups[n] = catalog[i1:i2]¬                                                                                  |   ~
          lsst::afw::table::SortedCatalogT< lsst::afw::table::SimpleRecord >::subset(std::ptrdiff_t,std::ptrdiff_t,std::ptrdiff_t) const
      

      The issue is that in lsst.afw.table.multiMatch line 148,

      groups[n] = catalog[i1:i2]
      

      the i1 and i2 types are numpy.int64 types. In Python 2, these can successfully be used as indices, but that does work in Python 3.

      I've confirmed that the problem is solved if I cast i1 and i2 to int.

      The solution is either

      1. Fix multiMatch to always use int types here, or more generally
      2. Fix SWIG's treatment of numpy.int64 types in Python 3.

      I'm not sure how pervasive this problem is elsewhere in the Stack; if it is then option 2 might be preferred.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  jsick Jonathan Sick
                  Reporter:
                  jsick Jonathan Sick
                  Reviewers:
                  Jim Bosch
                  Watchers:
                  Jim Bosch, Jonathan Sick, Pim Schellart [X] (Inactive), Tim Jenness
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel