# fgcmcal failure Ubuntu

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
1
• Urgent?:
No

## Description

fgcmcal fails on master with the latest conda environment on Ubuntu 18.04.4. The failure is in one of the new tests comparing the photoCalib.getCalibrationErr() with the fgcmZptGrayErr:

 tests/test_fgcmcal_hsc.py:154:  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _  tests/fgcmcalTestBase.py:407: in _testFgcmOutputProducts  (np.log(10.0)/2.5)*testCal.getCalibrationMean()*fgcmZptGrayErr) ../../stack/Linux64/utils/19.0.0-7-g686a884+2/python/lsst/utils/tests.py:656: in assertFloatsAlmostEqual  testCase.assertFalse(failed, msg="\n".join(errMsg)) E AssertionError: True is not false : 1/1 elements differ with rtol=2.220446049250313e-16, atol=2.220446049250313e-16 E 0.00223844323512219 != 0.0022384434 (diff=2.3283064e-10/0.0022384434=1.0401453e-07) 

We are not seeing this on other machines (CentOS, macOS), so there's probably a "fun" interaction with system libraries going on here. Adding rtol=2e-7, atol=None to the above test prevents the error, but may not be the actual solution we want. This suggests to me that there's a float32 calculation going on under the hood somewhere.

I can provide an Ubuntu account to test on.

## Activity

Hide
Eli Rykoff added a comment -

I've updated the persisted zeropoint table to all float64 which appears to fix the problem. (These values were stored internally as float64, and for some reason weren't persisted with the butler as float64).

Show
Eli Rykoff added a comment - I've updated the persisted zeropoint table to all float64 which appears to fix the problem. (These values were stored internally as float64, and for some reason weren't persisted with the butler as float64).
Hide
John Parejko added a comment -

Thanks for the quick fix. The tests passed on my Ubuntu 18.04 machine.

Show
John Parejko added a comment - Thanks for the quick fix. The tests passed on my Ubuntu 18.04 machine.
Hide
Eli Rykoff added a comment -

And jenkins passed in 23 minutes. I guess this is what happens when there aren't any merges to master since the last weekly? https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31283/pipeline

Show
Eli Rykoff added a comment - And jenkins passed in 23 minutes. I guess this is what happens when there aren't any merges to master since the last weekly? https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31283/pipeline

## People

• Assignee:
Eli Rykoff
Reporter:
John Parejko
Reviewers:
John Parejko
Watchers:
Eli Rykoff, John Parejko, John Swinbank, Krzysztof Findeisen