Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9796

esutil HTM module broken with latest Python/NumPy


    • Type: Bug
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Templates:
    • Story Points:
    • Team:
      Data Release Production


      On my Arch box with too-new versions of Python (3.6) and NumPy (1.12), I'm seeing Python exception-handling and numpy import errors in meas_algorithms (and downstream) tests that use the HTM indexing from esutil. Error message is this:

      ImportError: numpy.core.multiarray failed to import
      The above exception was the direct cause of the following exception:
      Traceback (most recent call last):
        File "tests/testHtmIndex.py", line 125, in setUpClass
          cls.indexer = IndexerRegistry['HTM'](config)
        File "/home/jbosch/LSST/sw/build/meas_algorithms/python/lsst/meas/algorithms/indexerRegistry.py", line 46, in makeHtmIndexer
          return HtmIndexer(depth=config.depth)
        File "/home/jbosch/LSST/sw/build/meas_algorithms/python/lsst/meas/algorithms/htmIndexer.py", line 35, in __init__
          self.htm = esutil.htm.HTM(depth)
        File "/home/jbosch/LSST/sw/stack/Linux64/esutil/0.6.0+1/lib/python/esutil/htm/htmc.py", line 152, in __init__
          this = _htmc.new_HTMC(depth)
      SystemError: <built-in function new_HTMC> returned a result with an error set

      I've tracked down the file where the error occurs, and it's a Swig Python module (esutil/htm/_htmc) whose C source is actually directly included in the esutils tarball, not generated on the fly. I originally thought that was the source of the trouble, but regenerating that those source files in the middle of the build (using the runscript.sh script in the same directory) doesn't fix the problem for me. But I'm still a bit worried about the fact that the Swig-generated code is not always regenerated.

      I'm going to build a new stack that doesn't use my too-new OS Python and NumPy now (something I've regretted for a while now, but hoped not to have to take the time to fix), so this doesn't need to be a super high priority, but I believe this is a blocker for upgrading either our supported Python version or supported NumPy version (I'm not sure which). It also makes me want to drop esutils in favor of sphgeom's HTM implementation at the earliest possible opportunity.


          Container Issues



              • Assignee:
                jbosch Jim Bosch
                jbosch Jim Bosch
                Simon Krughoff
                Jim Bosch, John Swinbank, Kenny Lo, Tim Jenness
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created:

                  Summary Panel