Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-702

Buildbot CI needs to save manifest file of failed build for later user debug

    Details

      Description

      The manifest file created during a build instance is transient and removed as soon as the next build commences. Due to that volatility, it's important to save the manifest to some well-known location so that the developer responsible for debug and repair can easily setup the failing environment. The location of the manifest file will be provided to the developer(s) in the failure notification.

        Attachments

          Activity

          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          This just requires that the failed package's: _build.sh is saved somewhere less-transient.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - This just requires that the failed package's: _build.sh is saved somewhere less-transient.
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          This was implemented during the initial round of Buildbot release. It needs to go into review.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - This was implemented during the initial round of Buildbot release. It needs to go into review.
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          Perry: If you are unable to review this ticket, then please let me know so it can be redirected.

          The directory containing the results of a failed buildbot run is found at:
          /lsst/home/lsstsw/build/FailedLogs/. The structure is as follows:
          @lsstdev:~lsstsw/build/FailedLogs/<BB Build#>/<Package>/<logs from failure>

          For example:
          /lsst/home/lsstsw/build/FailedLogs/303/meas_base/ contains:
          _build.log testPixelFlagsBasic.py.failed
          _build.sh testPsfFluxBasic.py.failed
          _build.tags testPsfFlux.py.failed
          testClassificationBasic.py.failed testResultMappers.py.failed
          testForced.py.failed testSdssCentroidBasic.py.failed
          testGaussianCentroidBasic.py.failed testSdssShapeBasic.py.failed
          testGaussianFluxBasic.py.failed testSincFluxBasic.py.failed
          testNaiveCentroidBasic.py.failed testSingleFrameMeasurement.py.failed
          testNaiveFluxBasic.py.failed testSkyCoordBasic.py.failed

          _build.log is the build log from the package
          _build.tags is the list of dependencies for that package
          _build.sh is the lsst_build script autogenerated to build and install the package within the sytsem stack. Do NOT use this yourself; but use it as a guide describing the events leading up to the failure.
          The rest of the files are the output resulting from unittest failures.

          When a failure is received, email is sent to either the [lsst-dm-dev] maillist or, if the build was user-invoked, to the user directly. That email contains the BB build number used to find the Failure artifacts.

          The software containing this generation of the FailedLogs archive is at:
          https://dev.lsstcorp.org/cgit/LSST/DMS/devenv/buildbot.git/tree/scripts/lsstswBuild.sh?h=tickets/DM-271
          lines 118-152.

          This version of the script is used by the production system and has been since the initial release of the new Buildbot.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - Perry: If you are unable to review this ticket, then please let me know so it can be redirected. The directory containing the results of a failed buildbot run is found at: /lsst/home/lsstsw/build/FailedLogs/. The structure is as follows: @lsstdev:~lsstsw/build/FailedLogs/<BB Build#>/<Package>/<logs from failure> For example: /lsst/home/lsstsw/build/FailedLogs/303/meas_base/ contains: _build.log testPixelFlagsBasic.py.failed _build.sh testPsfFluxBasic.py.failed _build.tags testPsfFlux.py.failed testClassificationBasic.py.failed testResultMappers.py.failed testForced.py.failed testSdssCentroidBasic.py.failed testGaussianCentroidBasic.py.failed testSdssShapeBasic.py.failed testGaussianFluxBasic.py.failed testSincFluxBasic.py.failed testNaiveCentroidBasic.py.failed testSingleFrameMeasurement.py.failed testNaiveFluxBasic.py.failed testSkyCoordBasic.py.failed _build.log is the build log from the package _build.tags is the list of dependencies for that package _build.sh is the lsst_build script autogenerated to build and install the package within the sytsem stack. Do NOT use this yourself; but use it as a guide describing the events leading up to the failure. The rest of the files are the output resulting from unittest failures. When a failure is received, email is sent to either the [lsst-dm-dev] maillist or, if the build was user-invoked, to the user directly. That email contains the BB build number used to find the Failure artifacts. The software containing this generation of the FailedLogs archive is at: https://dev.lsstcorp.org/cgit/LSST/DMS/devenv/buildbot.git/tree/scripts/lsstswBuild.sh?h=tickets/DM-271 lines 118-152. This version of the script is used by the production system and has been since the initial release of the new Buildbot.
          Hide
          robyn Robyn Allsman [X] (Inactive) added a comment -

          This was reviewed at the same time as tickets/DM-477.

          Show
          robyn Robyn Allsman [X] (Inactive) added a comment - This was reviewed at the same time as tickets/ DM-477 .

            People

            • Assignee:
              robyn Robyn Allsman [X] (Inactive)
              Reporter:
              robyn Robyn Allsman [X] (Inactive)
              Reviewers:
              Perry Gee
              Watchers:
              Perry Gee
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel

                  Time Tracking

                  Estimated:
                  Original Estimate - 2 days
                  2d
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d