Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1075

Message about failed buildbot runs points to deleted log files

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: buildbot
    • Labels:
      None
    • Team:
      SQuaRE

      Description

      At the end of the buildbot failed message these lines appear:

          • error building product afw.
          • exit code = 1
          • log is in /lsst/home/lsstsw/build/afw/_build.log
          • last few lines:
            ::::: + echo '*** Failed unit tests.'
            ::::: *** Failed unit tests.

      This is not true. As Robyn points out, the correct location of these files is implied by statements at the top of the message:

      Failed Package Info at lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/328/
      .......Log */_build.log

      by which is meant

      dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/328/afw/_build.log

      It'd be nice if this were a little more explicit.

        Attachments

          Activity

          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          Help me with some hints on what clarification you need.
          Perhaps:
          Failed Package log and status files are on the cluster system
          path ~lsstsw/build/FailedLogs/328/
          _build.log the build's log file
          _build.sh script used by BB to build info only
          _build.txt manifest for build
          *.failed the failed unit test log

          Do you want me to put a disclaimer somewhere that: "The build directories are transient and overwritten on the next build; important debug logs are moved to: ~lsstsw/buidl.FailedLogs/<BB#>/* "

          But I do not want to rewrite the stdio log to alter the embedded directory paths.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - Help me with some hints on what clarification you need. Perhaps: Failed Package log and status files are on the cluster system path ~lsstsw/build/FailedLogs/328/ _build.log the build's log file _build.sh script used by BB to build info only _build.txt manifest for build *.failed the failed unit test log Do you want me to put a disclaimer somewhere that: "The build directories are transient and overwritten on the next build; important debug logs are moved to: ~lsstsw/buidl.FailedLogs/<BB#>/* " But I do not want to rewrite the stdio log to alter the embedded directory paths.
          Hide
          rhl Robert Lupton added a comment -

          Is there a reason not to write the files to ~lsstsw/build.FailedLogs/<BB#> in the first place and delete when they become stale if the build succeeded? Then you wouldn't need to change stdout. You could have a link pointing to the latest run if that's why you call them ~lsstsw/build/afw/_build.log now.

          Failing that, I think that:

          Do you want me to put a disclaimer somewhere that: "The build directories are transient and overwritten on the next build; important debug logs are moved to: ~lsstsw/buidl.FailedLogs/<BB#>/* "

          would be helpful for people like me who seldom look at these logs, maybe at the bottom just after the quote from stdout?

          Show
          rhl Robert Lupton added a comment - Is there a reason not to write the files to ~lsstsw/build.FailedLogs/<BB#> in the first place and delete when they become stale if the build succeeded? Then you wouldn't need to change stdout. You could have a link pointing to the latest run if that's why you call them ~lsstsw/build/afw/_build.log now. Failing that, I think that: Do you want me to put a disclaimer somewhere that: "The build directories are transient and overwritten on the next build; important debug logs are moved to: ~lsstsw/buidl.FailedLogs/<BB#>/* " would be helpful for people like me who seldom look at these logs, maybe at the bottom just after the quote from stdout?
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          BB is a not-so-thin wrapper around the build tools that Mario wrote - primarily lsstsw which itself wraps lsst-build. BB initiates the stack build and stands back until it exits. Then BB peers into the collected build directories to determine if there were failed unit tests or if the _build.log has any text indicating a failed 'scons'. In the past, BB didn't know what packages had failed until it scanned the build logs. Your new fix for actually causing an exit on unit test fail will help simplify this process. Now the error exit should occur on the last package built which can be deduced without checking every single package in the stack-build directory.

          However, back to your original suggestion: sure most anything is possible. This particular suggestion would involve modification to lsstsw. Mario has another implementation in his back pocket so I'm not interested in spending time altering this version to have it changed out.

          To understand whre you alternate solution would be placed. You want it after the list of lines which were identified as some sort of error indication? So essentially, the last line of stdout?

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - BB is a not-so-thin wrapper around the build tools that Mario wrote - primarily lsstsw which itself wraps lsst-build. BB initiates the stack build and stands back until it exits. Then BB peers into the collected build directories to determine if there were failed unit tests or if the _build.log has any text indicating a failed 'scons'. In the past, BB didn't know what packages had failed until it scanned the build logs. Your new fix for actually causing an exit on unit test fail will help simplify this process. Now the error exit should occur on the last package built which can be deduced without checking every single package in the stack-build directory. However, back to your original suggestion: sure most anything is possible. This particular suggestion would involve modification to lsstsw. Mario has another implementation in his back pocket so I'm not interested in spending time altering this version to have it changed out. To understand whre you alternate solution would be placed. You want it after the list of lines which were identified as some sort of error indication? So essentially, the last line of stdout?
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          The final statement on the emailed status report will be the following with the appr BB# used:
          'Build directories are transient and overwritten on next build; failed-run logs were moved to: "lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2487/" '

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - The final statement on the emailed status report will be the following with the appr BB# used: 'Build directories are transient and overwritten on next build; failed-run logs were moved to: "lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2487/" '
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          Following is the tail end of the email from a failed build; last line is addition.
          ==========================================================
          Failed Package Info at lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2609/
          .......Log */_build.log
          .......Manifest */_build.tags
          .......Bot Script */_build.sh – for info, not use
          .......Unittest /.failed
          Build Reason A build was forced by 'raa ': force build
          Built on behalf of robynallsman robynallsman@gmail.com

          Build Log

          Failure: showing error messages in: stdio, Status: 2

          ctrl_events: u.rallsman.TestBBFailInScons-g1031c5f8f2+2 ERROR (62 sec).

              • error building product ctrl_events.
              • exit code = 2
              • log is in /lsst/home/lsstsw/build/ctrl_events/_build.log
              • last few lines:
                ::::: scons: *** [src/Event.os] Error 1

          A directory where a build failure occurred is overwritten next build; its failed-build logs are moved to: "lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2609/"

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - Following is the tail end of the email from a failed build; last line is addition. ========================================================== Failed Package Info at lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2609/ .......Log */_build.log .......Manifest */_build.tags .......Bot Script */_build.sh – for info, not use .......Unittest / .failed Build Reason A build was forced by 'raa ': force build Built on behalf of robynallsman robynallsman@gmail.com Build Log Failure: showing error messages in: stdio, Status: 2 ctrl_events: u.rallsman.TestBBFailInScons-g1031c5f8f2+2 ERROR (62 sec). error building product ctrl_events. exit code = 2 log is in /lsst/home/lsstsw/build/ctrl_events/_build.log last few lines: ::::: scons: *** [src/Event.os] Error 1 A directory where a build failure occurred is overwritten next build; its failed-build logs are moved to: "lsst-dev.ncsa.illinois.edu:/lsst/home/lsstsw/build/FailedLogs/2609/"
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          Changes are in: https://github.com/frossie/buildmaster.git (master) and on buildbot server (lsst-buildx)

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - Changes are in: https://github.com/frossie/buildmaster.git (master) and on buildbot server (lsst-buildx)
          Hide
          tjenness Tim Jenness added a comment -

          Following through to that repo indicates that the changes have been merged so should this ticket be closed?

          Show
          tjenness Tim Jenness added a comment - Following through to that repo indicates that the changes have been merged so should this ticket be closed?

            People

            Assignee:
            frossie Frossie Economou
            Reporter:
            rhl Robert Lupton
            Reviewers:
            Frossie Economou
            Watchers:
            Frossie Economou, Robert Lupton, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.