Details
Description
The output of demo2012 results in an output file which is compared against a benchmarked file. Currently the comparison allows a deviation from the benchmark based "on the the number of digits in the significands used in multiple precision arithmetic"; that number is currently set to 11.
An example using that setting is:
@ Absolute error = 5.9973406400e-1, Relative error = 9.1865836000e-4
##2566 #:25 <== 29.7751550737
##2566 #:25 ==> 29.7478269835
Additionally, the current use returns: a 'Fatal' error if the code itself fails to execute correctly; a "Warning' error if the any of the benchmarked quantities do not meet the comparison criteria; 'Success' if the comparison meets all criteria.
It is noted that the buildbot 'Warning' color indicator is not currently being displayed when the comparison fails. That is a coding error.
This Issue will:
- ensure that BUILDBOT_WARNING(S) causes the correct display color when appropriate.
This Ticket has been split into two parts. This is part 1. Part 2 is DM-1379
Attachments
Issue Links
- is triggering
-
DM-1379 End-to-end demo should fail on inexact comparison to benchmark file.
- Invalid
This ticket was split into two. This ticket addresses a bug regarding the failure to set the correct status result on completion of the demo2012 end-to-end test. The new ticket:
DM-1379, will address the comparison .The change to EQUALS will coerce the developer to validate and update the benchmark concurrently with the updated code base. So demo runs against that code base should remain consistent.
There will be a problem with folks not using the same host system as the one on which the benchmark file was constructed. E.g., users checking their MacOsX generated FP values against the benchmark RHEL6 FP values. In this case, the comparison should probably not be EQUALS but some appropriate benchmark +- epsilon. The comparison tool does have this facility but someone will need to specify the appropriate epsilon either individually for each 'column' or globally for all 'columns'. This could be wrapped for the non-buildbot user.
Only current problem slowing this addition is fact that the demo package is not acquired using branch specifications - it's always master.