Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11401

testHtmIndex.py takes over 6 minutes to run

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_algorithms
    • Labels:
      None
    • Templates:
    • Team:
      External

      Description

      6 minutes is a bit much for a simple unit test like this — it's usually running when everything else is done by the other threads. I believe most of the time (~80%) is coming from about a million butler lookups, because the search radii are large (several degrees) and the HTM is going deep (8 or 10) so there are a lot of files to read. Some small tuning of the test could make it run much faster.

               322918914 function calls (321817486 primitive calls) in 372.179 seconds
       
         Ordered by: cumulative time
         List reduced from 2386 to 30 due to restriction <30>
       
         ncalls  tottime  percall  cumtime  percall filename:lineno(function)
              1    0.000    0.000  372.212  372.212 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/main.py:72(__init__)
              1    0.000    0.000  372.210  372.210 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/main.py:216(runTests)
              1    0.000    0.000  372.210  372.210 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/runner.py:140(run)
            2/1    0.000    0.000  372.210  372.210 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/suite.py:69(__call__)
            2/1    0.000    0.000  372.210  372.210 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/suite.py:88(run)
              9    0.000    0.000  364.679   40.520 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/case.py:392(__call__)
              9    0.000    0.000  364.679   40.520 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/unittest/case.py:295(run)
         124451    1.243    0.000  355.346    0.003 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py:1229(datasetExists)
         129771    2.344    0.000  353.332    0.003 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/butler.py:1267(_locate)
         129/87    0.004    0.000  346.570    3.984 /Users/price/LSST/stack/DarwinX86/pipe_base/13.0-9-g1c7d9c5+11/python/lsst/pipe/base/timer.py:117(wrapper)
             87    0.009    0.000  346.474    3.982 /Users/price/LSST/meas_algorithms/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py:57(loadSkyCircle)
             87    0.525    0.006  346.279    3.980 /Users/price/LSST/meas_algorithms/python/lsst/meas/algorithms/loadIndexedReferenceObjects.py:110(get_shards)
         992156    0.991    0.000  298.809    0.000 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py:491(instanceSearch)
         992156   10.769    0.000  297.817    0.000 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py:511(search)
              1    0.011    0.011  267.106  267.106 tests/testHtmIndex.py:282(testLoadPixelBox)
             42    0.002    0.000  267.061    6.359 /Users/price/LSST/meas_algorithms/python/lsst/meas/algorithms/loadReferenceObjects.py:187(loadPixelBox)
         252994    0.644    0.000  266.623    0.001 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/repository.py:231(map)
         252994    0.527    0.000  265.902    0.001 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/mapper.py:139(map)
         252994    0.421    0.000  264.518    0.001 /Users/price/LSST/stack/DarwinX86/obs_base/13.0-33-g9f98c6c/python/lsst/obs/base/cameraMapper.py:377(mapClosure)
         252994    4.381    0.000  264.097    0.001 /Users/price/LSST/stack/DarwinX86/obs_base/13.0-33-g9f98c6c/python/lsst/obs/base/mapping.py:119(map)
        3968673    4.449    0.000  258.211    0.000 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py:372(realpath)
      4967125/3968673   43.991    0.000  181.569    0.000 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py:380(_joinrealpath)
       20837338   22.907    0.000   86.845    0.000 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py:132(islink)
         250222    0.355    0.000   83.882    0.000 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/repository.py:290(exists)
         250230    0.291    0.000   83.467    0.000 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py:404(exists)
         250222    5.390    0.000   83.145    0.000 /Users/price/LSST/stack/DarwinX86/daf_persistence/13.0-25-g49e493d/python/lsst/daf/persistence/posixStorage.py:390(butlerLocationExists)
        3973000    6.696    0.000   72.274    0.000 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py:358(abspath)
       21832518   59.184    0.000   59.184    0.000 {posix.lstat}
              1    0.004    0.004   56.698   56.698 tests/testHtmIndex.py:264(testLoadSkyCircle)
       23826673   33.082    0.000   44.602    0.000 /Users/price/LSST/stack/DarwinX86/miniconda2/3.19.0.lsst4/lib/python2.7/posixpath.py:61(join)
      

        Attachments

          Activity

            People

            • Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Russell Owen
              Watchers:
              Paul Price, Russell Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel