Status: Won't Fix
Fix Version/s: None
The package lsst_dm_stack_demo is used by buildbot as an integration test by comparing an output file to an existing expected output data file. Unfortunately there are several problems with the test:
- There is no statistical summary of the measured change, and in particular there is no report of the maximum error. When this test breaks we usually see tiny changes in many values. But if one or a few values have big changes (large enough to be really scary) we may not see this. In other words it is far too easy for minor innocuous changes to hide an crucial large change.
- The data is written out using a number of digits that varies. I am not sure of all the causes of variability, but we have observed different files created on different systems. A recent commit (
DM-2719) apparently caused the number of digits to change on buildbot, though I find it very hard to believe. The numbers match out to the number of digits written, but the number changed, resulting in a failed integration test.
- Even if the data was written out to a consistent number of digits, the test should probably rely on fewer digits, as meaningless minor changes can cause failures.
The result of is that we have many false negatives, which makes us take failures less seriously and wastes our time trying to figure out what is wrong when nothing is wrong.