Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1358

End-to-end demo fails to exit with the correct status when Warning would be correct.

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: buildbot
    • Labels:
    • Story Points:
      2
    • Team:
      SQuaRE

      Description

      The output of demo2012 results in an output file which is compared against a benchmarked file. Currently the comparison allows a deviation from the benchmark based "on the the number of digits in the significands used in multiple precision arithmetic"; that number is currently set to 11.

      An example using that setting is:
      @ Absolute error = 5.9973406400e-1, Relative error = 9.1865836000e-4
      ##2566 #:25 <== 29.7751550737
      ##2566 #:25 ==> 29.7478269835

      Additionally, the current use returns: a 'Fatal' error if the code itself fails to execute correctly; a "Warning' error if the any of the benchmarked quantities do not meet the comparison criteria; 'Success' if the comparison meets all criteria.

      It is noted that the buildbot 'Warning' color indicator is not currently being displayed when the comparison fails. That is a coding error.

      This Issue will:

      • ensure that BUILDBOT_WARNING(S) causes the correct display color when appropriate.

      This Ticket has been split into two parts. This is part 1. Part 2 is DM-1379

        Attachments

          Issue Links

            Activity

            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited

            This ticket was split into two. This ticket addresses a bug regarding the failure to set the correct status result on completion of the demo2012 end-to-end test. The new ticket: DM-1379, will address the comparison .

            The change to EQUALS will coerce the developer to validate and update the benchmark concurrently with the updated code base. So demo runs against that code base should remain consistent.

            There will be a problem with folks not using the same host system as the one on which the benchmark file was constructed. E.g., users checking their MacOsX generated FP values against the benchmark RHEL6 FP values. In this case, the comparison should probably not be EQUALS but some appropriate benchmark +- epsilon. The comparison tool does have this facility but someone will need to specify the appropriate epsilon either individually for each 'column' or globally for all 'columns'. This could be wrapped for the non-buildbot user.

            Only current problem slowing this addition is fact that the demo package is not acquired using branch specifications - it's always master.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - - edited This ticket was split into two. This ticket addresses a bug regarding the failure to set the correct status result on completion of the demo2012 end-to-end test. The new ticket: DM-1379 , will address the comparison . The change to EQUALS will coerce the developer to validate and update the benchmark concurrently with the updated code base. So demo runs against that code base should remain consistent. There will be a problem with folks not using the same host system as the one on which the benchmark file was constructed. E.g., users checking their MacOsX generated FP values against the benchmark RHEL6 FP values. In this case, the comparison should probably not be EQUALS but some appropriate benchmark +- epsilon. The comparison tool does have this facility but someone will need to specify the appropriate epsilon either individually for each 'column' or globally for all 'columns'. This could be wrapped for the non-buildbot user. Only current problem slowing this addition is fact that the demo package is not acquired using branch specifications - it's always master.
            Hide
            ktl Kian-Tat Lim added a comment -

            I agree that EQUALS is inappropriate if different systems give slightly different results. This is part of what we need to discover by forcing people to do the comparison. (And having multiple systems available as buildslaves will help.)

            Show
            ktl Kian-Tat Lim added a comment - I agree that EQUALS is inappropriate if different systems give slightly different results. This is part of what we need to discover by forcing people to do the comparison. (And having multiple systems available as buildslaves will help.)
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            The failure was found to be the use of an incorrect name for the global BUILDBOT_WARNING variable.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - The failure was found to be the use of an incorrect name for the global BUILDBOT_WARNING variable.
            Hide
            robyn Robyn Allsman [X] (Inactive) added a comment -

            This update fixes the incorrect exit occurring on buildbot WARNING exits; The branch was committed last week but not put into review.

            In the interim, the eups 1.5.4 requirement to source the eups config in each script using eups (DM-1394) resulted in an emergency fix in buildbot production scripts. That now needs to be reflected in the master branch. I am adding that fix to this Issue to bring master and the production into sync using one review instead of two.

            Show
            robyn Robyn Allsman [X] (Inactive) added a comment - This update fixes the incorrect exit occurring on buildbot WARNING exits; The branch was committed last week but not put into review. In the interim, the eups 1.5.4 requirement to source the eups config in each script using eups ( DM-1394 ) resulted in an emergency fix in buildbot production scripts. That now needs to be reflected in the master branch. I am adding that fix to this Issue to bring master and the production into sync using one review instead of two.
            Hide
            ktl Kian-Tat Lim added a comment -

            Code looks OK to me.

            Show
            ktl Kian-Tat Lim added a comment - Code looks OK to me.

              People

              • Assignee:
                robyn Robyn Allsman [X] (Inactive)
                Reporter:
                robyn Robyn Allsman [X] (Inactive)
                Reviewers:
                Kian-Tat Lim
                Watchers:
                Kian-Tat Lim, Perry Gee
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel

                    Time Tracking

                    Estimated:
                    Original Estimate - 1 minute
                    1m
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 1 day
                    1d