Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7701

conda-lsst ndarray test failure

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ndarray
    • Labels:
      None

      Description

      The conda-lsst build log for the test failure can be found here:

      https://gist.github.com/jmatt/dcb6ce8312aac1ff81cbb923703d8430

      Running tests...
      Test project /Users/square/dev/conda-lsst/miniconda/conda-bld/work/build
          Start 1: test_ndarray
      1/5 Test #1: test_ndarray .....................   Passed    0.01 sec
          Start 2: test_views
      2/5 Test #2: test_views .......................   Passed    0.01 sec
          Start 3: test_ndarray-fft
      3/5 Test #3: test_ndarray-fft .................***Exception: Other  0.00 sec
          Start 4: test_ndarray_eigen
      4/5 Test #4: test_ndarray_eigen ...............   Passed    0.01 sec
          Start 5: swig_test
      5/5 Test #5: swig_test ........................   Passed    0.11 sec
       
      80% tests passed, 1 tests failed out of 5
       
      Total Test time (real) =   0.14 sec
       
      The following tests FAILED:
      	  3 - test_ndarray-fft (OTHER_FAULT)
      Errors while running CTest
      make: *** [test] Error 8
      

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment - - edited

            Which commit is this in the ndarray git repo?

            Show
            tjenness Tim Jenness added a comment - - edited Which commit is this in the ndarray git repo?
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment - - edited

            To reproduce this problem with a far smaller build footprint:

            git clone https://github.com/jmatt/conda-lsst.git
            cd conda-lsst
            git checkout v12.1
            ./bin/bootstrap.sh
            export PATH="${PWD}/bin:${PWD}/miniconda/bin:${PATH}"
            conda lsst make-recipes build:b2299 ndarray —build
            

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - - edited To reproduce this problem with a far smaller build footprint: git clone https://github.com/jmatt/conda-lsst.git cd conda-lsst git checkout v12.1 ./bin/bootstrap.sh export PATH="${PWD}/bin:${PWD}/miniconda/bin:${PATH}" conda lsst make-recipes build:b2299 ndarray —build
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            This is happening on Mac OS X El Capitan.

            There is a different failure on Linux. I'll create a separate ticket.

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - This is happening on Mac OS X El Capitan. There is a different failure on Linux. I'll create a separate ticket.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            Just tried it but it doesn't seem to work.

            conda-lsst $ conda lsst make-recipes build:b2299 ndarray —build
            updating built package cache [from file:///Users/pschella/Development/lsst/code/conda-lsst/miniconda/conda-bld/osx-64] + done.
            https://raw.githubusercontent.com/lsst/versiondb/master/manifests/b2299.txt
            Traceback (most recent call last):
              File "/Users/pschella/Development/lsst/code/conda-lsst/bin/conda-lsst", line 149, in <module>
                args.func(config, args)
              File "/Users/pschella/Development/lsst/code/conda-lsst/bin/conda-lsst", line 23, in main_make_recipes
                manifest, tags = build_manifest_for_products(args.products, args.manifest)
              File "/Users/pschella/Development/lsst/code/conda-lsst/conda_lsst/utils.py", line 81, in build_manifest_for_products
                bottom_up_add_to_manifest(product)
              File "/Users/pschella/Development/lsst/code/conda-lsst/conda_lsst/utils.py", line 74, in bottom_up_add_to_manifest
                (product, sha, version, deps) = products[product]
            KeyError: '\xe2\x80\x94build'
            

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - Just tried it but it doesn't seem to work. conda-lsst $ conda lsst make -recipes build:b2299 ndarray —build updating built package cache [from file : ///Users/pschella/Development/lsst/code/conda-lsst/miniconda/conda-bld/osx-64 ] + done . https: //raw .githubusercontent.com /lsst/versiondb/master/manifests/b2299 .txt Traceback (most recent call last): File "/Users/pschella/Development/lsst/code/conda-lsst/bin/conda-lsst" , line 149, in <module> args.func(config, args) File "/Users/pschella/Development/lsst/code/conda-lsst/bin/conda-lsst" , line 23, in main_make_recipes manifest, tags = build_manifest_for_products(args.products, args.manifest) File "/Users/pschella/Development/lsst/code/conda-lsst/conda_lsst/utils.py" , line 81, in build_manifest_for_products bottom_up_add_to_manifest(product) File "/Users/pschella/Development/lsst/code/conda-lsst/conda_lsst/utils.py" , line 74, in bottom_up_add_to_manifest (product, sha, version, deps) = products[product] KeyError: '\xe2\x80\x94build'
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            Pim Schellart [X] it should be --build but it was converted to some Unicode character.

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - Pim Schellart [X] it should be --build but it was converted to some Unicode character.
            Hide
            rhl Robert Lupton added a comment -

            I see this on os/x 10.11.6 too.

            I asked Jim Bosch about this, and he tells me that we aren't using the ndarray/fft integration and it's only tested if enabled in ndarray. Can we turn it off? We should fix it, but this is an unnecessary dependency that might come back and bite us again.

            Show
            rhl Robert Lupton added a comment - I see this on os/x 10.11.6 too. I asked Jim Bosch about this, and he tells me that we aren't using the ndarray/fft integration and it's only tested if enabled in ndarray. Can we turn it off? We should fix it, but this is an unnecessary dependency that might come back and bite us again.
            Hide
            tjenness Tim Jenness added a comment -

            Looks like a library loading issue:

            $ otool -L ndarray-fft 
            ndarray-fft:
            	@rpath/libboost_unit_test_framework.dylib (compatibility version 0.0.0, current version 0.0.0)
            	@rpath/../opt/lsst/fftw/lib/libfftw3.3.dylib (compatibility version 8.0.0, current version 8.4.0)
            	@rpath/../opt/lsst/fftw/lib/libfftw3f.3.dylib (compatibility version 8.0.0, current version 8.4.0)
            	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0)
            	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
            

            That path to libfftw is only valid for installed packages, not for test binaries. I don't think DM-5753 is relevant because the initial link is working to make the binary, but the runtime loading is broken.

            If I make sure that the relevant library path is added to my DYLD_LIBRARY_PATH the test file runs fine. It seems to me that we will get this problem with all of our test binaries (eg those in afw) unless we fix the rpaths in the test files to be correct or else try to work out how to fixup DYLD_LIBRARY_PATH.

            Show
            tjenness Tim Jenness added a comment - Looks like a library loading issue: $ otool -L ndarray-fft ndarray-fft: @rpath/libboost_unit_test_framework.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/../opt/lsst/fftw/lib/libfftw3.3.dylib (compatibility version 8.0.0, current version 8.4.0) @rpath/../opt/lsst/fftw/lib/libfftw3f.3.dylib (compatibility version 8.0.0, current version 8.4.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.4.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0) That path to libfftw is only valid for installed packages, not for test binaries. I don't think DM-5753 is relevant because the initial link is working to make the binary, but the runtime loading is broken. If I make sure that the relevant library path is added to my DYLD_LIBRARY_PATH the test file runs fine. It seems to me that we will get this problem with all of our test binaries (eg those in afw) unless we fix the rpaths in the test files to be correct or else try to work out how to fixup DYLD_LIBRARY_PATH.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            I cannot reproduce this.

            conda-lsst $ conda lsst make-recipes build:b2299 ndarray --build
            updating built package cache [from file:///Users/pschella/Development/lsst/code/conda-lsst/miniconda/conda-bld/osx-64] . done.
            https://raw.githubusercontent.com/lsst/versiondb/master/manifests/b2299.txt
            generating recipes: 
              lsst-python-eups-configs-0.0.5...  
              eups-2.0.2...  
              lsst-boost-1.60.1...  
              lsst-numpy-eups-configs-0.0.3...  
              lsst-eigen-3.2.5.2...  
              lsst-swig-eups-configs-3.0.10...  
              lsst-fftw-3.3.4.2...  
              lsst-ndarray-1.2.0.1...  
            done.
            generating rebuild script:
              will build:    eups-2.0.2-0
              will build:    lsst-python-eups-configs-0.0.5-0
              will build:    lsst-boost-1.60.1-0
              will build:    lsst-numpy-eups-configs-0.0.3-0
              will build:    lsst-eigen-3.2.5.2-0
              will build:    lsst-swig-eups-configs-3.0.10-0
              will build:    lsst-fftw-3.3.4.2-0
              will build:    lsst-ndarray-1.2.0.1-0
            done.
            calling /Users/pschella/Development/lsst/code/conda-lsst/recipes/rebuild.sh to build:
              eups (ver 2.0.2-0)   [from static recipe] ... OK
              lsst-python-eups-configs (ver 0.0.5-0)   [from python-0.0.5] ... OK
              lsst-boost (ver 1.60.1-0)   [from boost-1.60.lsst1] ... gOK
              lsst-numpy-eups-configs (ver 0.0.3-0)   [from numpy-0.0.3] ... OK
              lsst-eigen (ver 3.2.5.2-0)   [from eigen-3.2.5.lsst2] ... OK
              lsst-swig-eups-configs (ver 3.0.10-0)   [from swig-3.0.10] ... OK
              lsst-fftw (ver 3.3.4.2-0)   [from fftw-3.3.4.lsst2] ... OK
              lsst-ndarray (ver 1.2.0.1-0)   [from ndarray-1.2.0.lsst1] ... OK
            done.
            

            I'm on OSX 10.11.6 (the latest El Cap version).
            As per the earlier question, yes it is easy to disable the FFTW dependency for ndarray. Should I do that in our package?

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - I cannot reproduce this. conda-lsst $ conda lsst make -recipes build:b2299 ndarray --build updating built package cache [from file : ///Users/pschella/Development/lsst/code/conda-lsst/miniconda/conda-bld/osx-64 ] . done . https: //raw .githubusercontent.com /lsst/versiondb/master/manifests/b2299 .txt generating recipes: lsst-python-eups-configs-0.0.5... eups-2.0.2... lsst-boost-1.60.1... lsst-numpy-eups-configs-0.0.3... lsst-eigen-3.2.5.2... lsst-swig-eups-configs-3.0.10... lsst-fftw-3.3.4.2... lsst-ndarray-1.2.0.1... done . generating rebuild script: will build: eups-2.0.2-0 will build: lsst-python-eups-configs-0.0.5-0 will build: lsst-boost-1.60.1-0 will build: lsst-numpy-eups-configs-0.0.3-0 will build: lsst-eigen-3.2.5.2-0 will build: lsst-swig-eups-configs-3.0.10-0 will build: lsst-fftw-3.3.4.2-0 will build: lsst-ndarray-1.2.0.1-0 done . calling /Users/pschella/Development/lsst/code/conda-lsst/recipes/rebuild .sh to build: eups (ver 2.0.2-0) [from static recipe] ... OK lsst-python-eups-configs (ver 0.0.5-0) [from python-0.0.5] ... OK lsst-boost (ver 1.60.1-0) [from boost-1.60.lsst1] ... gOK lsst-numpy-eups-configs (ver 0.0.3-0) [from numpy-0.0.3] ... OK lsst-eigen (ver 3.2.5.2-0) [from eigen-3.2.5.lsst2] ... OK lsst-swig-eups-configs (ver 3.0.10-0) [from swig-3.0.10] ... OK lsst-fftw (ver 3.3.4.2-0) [from fftw-3.3.4.lsst2] ... OK lsst-ndarray (ver 1.2.0.1-0) [from ndarray-1.2.0.lsst1] ... OK done . I'm on OSX 10.11.6 (the latest El Cap version). As per the earlier question, yes it is easy to disable the FFTW dependency for ndarray. Should I do that in our package?
            Hide
            tjenness Tim Jenness added a comment -

            That's a bit strange. I was thinking about this last night and it occurs to me that (1) we never usually run tests of third party packages so ndarray running tests is anomalous, and (2) the reason why this works in an lsstsw build but not (at least for most of us) in conda is that make is a SIP program so strips DYLD_LIBRARY_PATH but when we build fftw normally with lsstsw it has a full path burned in to the library so that the location is available to the test binary. scons builds know about SIP and forward on the LSST_LIBRARY_PATH when running tests. In conda builds it's @rpath as we've stated above. This is also discussed in DM-5753.

            Show
            tjenness Tim Jenness added a comment - That's a bit strange. I was thinking about this last night and it occurs to me that (1) we never usually run tests of third party packages so ndarray running tests is anomalous, and (2) the reason why this works in an lsstsw build but not (at least for most of us) in conda is that make is a SIP program so strips DYLD_LIBRARY_PATH but when we build fftw normally with lsstsw it has a full path burned in to the library so that the location is available to the test binary. scons builds know about SIP and forward on the LSST_LIBRARY_PATH when running tests. In conda builds it's @rpath as we've stated above. This is also discussed in DM-5753 .
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            I used Conda's patch system to create an eups patch to remove the fft test that was failing on Mac OS X in ndarry.

            https://github.com/jmatt/conda-lsst/commit/9dbadf0fcfa6dd7fc369a6682a8c87a7cb55ab14

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - I used Conda's patch system to create an eups patch to remove the fft test that was failing on Mac OS X in ndarry. https://github.com/jmatt/conda-lsst/commit/9dbadf0fcfa6dd7fc369a6682a8c87a7cb55ab14
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            I created a patch to create a patch to remove the fft ndarray test that was failing due to a missing path. This is a workaround to finish v12.1

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - I created a patch to create a patch to remove the fft ndarray test that was failing due to a missing path. This is a workaround to finish v12.1
            Hide
            tjenness Tim Jenness added a comment -

            This seems to be a reasonable short term hack.

            Show
            tjenness Tim Jenness added a comment - This seems to be a reasonable short term hack.
            Hide
            jmatt J Matt Peterson [X] (Inactive) added a comment -

            This is finally pushed upstream in in DM-8135

            Show
            jmatt J Matt Peterson [X] (Inactive) added a comment - This is finally pushed upstream in in DM-8135

              People

              • Assignee:
                jmatt J Matt Peterson [X] (Inactive)
                Reporter:
                jmatt J Matt Peterson [X] (Inactive)
                Reviewers:
                Tim Jenness
                Watchers:
                J Matt Peterson [X] (Inactive), Pim Schellart [X] (Inactive), Robert Lupton, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel