Then that's a bug, isn't it? PRNGs should be fully deterministic.

Raising priority to 'Blocker' as this currently breaks OS X builds.

Then that's a bug, isn't it? PRNGs should be fully deterministic.

It may be a NumPy version difference, not just a platform difference, as I'm using Ubuntu 12.04's system Python/NumPy (1.6.1). In any case, it's not our bug, as the RNG is explicitly seeded in the test, and I think having tolerances that work with different random numbers here is an improvement regardless.

Proposed fix is on u/jbosch/DM-1305 of shapelet. Please verify that the new tolerances work.

Reassigning review to Russell, since this is blocking him and he's free to do it now.

I'm afraid your changes did not fix the problem. I'm not sure if the new values are being ignored, or if they don't override all the rtol and atol values that they need to override. Here is output from one unit test:

Traceback (most recent call last):

  File "tests/testMatrixBuilder.py", line 310, in testConvolvedCompoundMatrixBuilder

    self.assertClose(numpy.dot(matrix1D, coefficients), checkVector, rtol=1E-14)

  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/utils/9.2+10/python/lsst/utils/tests.py", line 328, in assertClose

    testCase.assertFalse(failed, msg="\n".join(msg))

AssertionError: 1/50 elements differ with rtol=1e-14, atol=2.22044604925e-16

0.175869366369 != 0.175869366369 (diff=1.99840144433e-15/0.175869366369=1.13629876856e-14)

The line in the traceback you pasted does not match the code I see on branch u/jbosch/DM-1305; line 310 of testMatrixBuilder.py should be:

self.assertClose(numpy.dot(matrix1D, coefficients), checkVector, rtol=1E-13, atol=1E-14)

Sorry, I missed that change to testMatrixBuilder (it was in a previous commit and I was being sloppy). That test now passes. Unfortunately testMultiShapelet.py still fails and I'm not sure if the failure is significant:

FAIL: testConvolveGaussians (__main__.MultiShapeletTestCase)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "tests/testMultiShapelet.py", line 87, in testConvolveGaussians

    self.compareMultiShapeletFunctions(msf3a, msf3b)

  File "/Users/rowen/LSST/lsstsw/build/shapelet/python/lsst/shapelet/tests.py", line 107, in compareMultiShapeletFunctions

    self.compareShapeletFunctions(sa, sb, rtolEllipse=rtolEllipse, rtolCoeff=rtolCoeff)

  File "/Users/rowen/LSST/lsstsw/build/shapelet/python/lsst/shapelet/tests.py", line 86, in compareShapeletFunctions

    rtol=rtolEllipse)

  File "/Users/rowen/LSST/lsstsw/stack/DarwinX86/utils/9.2+10/python/lsst/utils/tests.py", line 328, in assertClose

    testCase.assertFalse(failed, msg="\n".join(msg))

AssertionError: 1/5 elements differ with rtol=1e-13, atol=2.22044604925e-16

2.44929359829e-16 != 1.13310777953e-15 (diff=8.881784197e-16/1.13310777953e-15=0.783842839795)

It's not significant. Feel free to adjust rtol and atol as needed to get it to pass. In general, I wouldn't consider any failure above 1E-8 a serious problem for any of these tests, but I'd rather not set the tolerances any lower than they need to be in order to protect against future regressions.

Ah, actually it looks like the problem is that we need to set atol here (rtol is useless because the numbers are so close to zero) and there's no obvious code path to do that, since the actual comparison is delegated to some test utility code and it doesn't have an atol argument that's forwarded. I'll be able to return to working on this later this evening, but feel free to look into it yourself earlier if you'd like.

Added one more commit to this branch with absolute tolerances in the test utility functions. Hopefully it will solve the most recent problem.

Looks good (other than a few typos that I fixed). I rebased that fix and merged to master.

Thank you very much for the fixes.