Details
Description
The output of demo2012 results in an output file which is compared against a benchmarked file. Currently the comparison allows a deviation from the benchmark based "on the the number of digits in the significands used in multiple precision arithmetic"; that number is currently set to 11.
An example using that setting is:
@ Absolute error = 5.9973406400e-1, Relative error = 9.1865836000e-4
##2566 #:25 <== 29.7751550737
##2566 #:25 ==> 29.7478269835
Additionally, the current use returns: a 'Fatal' error if the code itself fails to execute correctly; a "Warning' error if the any of the benchmarked quantities do not meet the comparison criteria; 'Success' if the comparison meets all criteria.
It is noted that the buildbot 'Warning' color indicator is not currently being displayed when the comparison fails. That is a coding error.
This Issue will:
- ensure that BUILDBOT_WARNING(S) causes the correct display color when appropriate.
This Ticket has been split into two parts. This is part 1. Part 2 is DM-1379
Attachments
Issue Links
- is triggering
-
DM-1379 End-to-end demo should fail on inexact comparison to benchmark file.
- Invalid
Activity
I agree that EQUALS is inappropriate if different systems give slightly different results. This is part of what we need to discover by forcing people to do the comparison. (And having multiple systems available as buildslaves will help.)
The failure was found to be the use of an incorrect name for the global BUILDBOT_WARNING variable.
This update fixes the incorrect exit occurring on buildbot WARNING exits; The branch was committed last week but not put into review.
In the interim, the eups 1.5.4 requirement to source the eups config in each script using eups (DM-1394) resulted in an emergency fix in buildbot production scripts. That now needs to be reflected in the master branch. I am adding that fix to this Issue to bring master and the production into sync using one review instead of two.
Code looks OK to me.
This ticket was split into two. This ticket addresses a bug regarding the failure to set the correct status result on completion of the demo2012 end-to-end test. The new ticket:
DM-1379, will address the comparison .The change to EQUALS will coerce the developer to validate and update the benchmark concurrently with the updated code base. So demo runs against that code base should remain consistent.
There will be a problem with folks not using the same host system as the one on which the benchmark file was constructed. E.g., users checking their MacOsX generated FP values against the benchmark RHEL6 FP values. In this case, the comparison should probably not be EQUALS but some appropriate benchmark +- epsilon. The comparison tool does have this facility but someone will need to specify the appropriate epsilon either individually for each 'column' or globally for all 'columns'. This could be wrapped for the non-buildbot user.
Only current problem slowing this addition is fact that the demo package is not acquired using branch specifications - it's always master.