Last week I was able to demonstrate working email notifications in a test env using aws ses for delivery. It appeared that the extra features of the jenkins email-ext and emailext-template plugins weren't being used and that the more basic mailer plugin would be sufficient. email-ext requires duplicate configuration with mailer, which it also depends on,and doesn't provider a "test" feature either. I suspect this duplicate configuration has contributed to the email problems in the past.

The configuration of aws-ses has been terraform-erized and a PR is open to update the jenkins deployment to fully configure the mailer plugin, remove the unneeded mail related plugins, and change the github oauth scopes to allow user email addresses to be retrieved. I prefer to do a restart when an auth change is made to ensure there aren't any lurking issues with cached metadata and a maintenance period needs to be scheduled to roll out these changes.

Maintenance for next Monday has been announced on #ts-software.

@here https://ts-ci.lsst.codes/ will be undergoing maintenance this coming Monday 02/18 (President's day) at 1600 US/Arizona (Tucson local time).

The deployment didn't go as smoothly as planned and the global credentials (those not tied to "folders") were removed by the CASC plugin as due to a configuration error on my part. Instead of immediately putting the credentials back in place, I went down a several hour rabbit hole of auditing how credentials were being used. In the vast majority of cases, credentials were configured on clones of git repos under the github lsst-ts org that were public, and thus credentials weren't required. I believe I removed most if not all of these cases. I also converted a number of uses of rbovill github credentials to uses those of the lssttsadmin github "robot" user that I've created. The two examples that needed existing credentials were the setup of ssh agent by the ts_scheduler job and for access to https://gitlab.tekniker.es/sai/projects/3151-LSST/lsst.git by the TMA_Simulator job. In the end, I left my removal/conversion of github credentials in place and restored all of the original global credentials. However, I believe the majority of the global credentials are unused may be removed in the future.

Rob Bovill Some action is required to configure email notifications when a build fails. For existing "free style" jobs, the "Set jenkins user build variables" binding needs to be set to make the BUILD_USER_EMAIL env var available. A Email Notification post build action is also necessary. Including {$BUILD_USER_EMAIL}} in the Recipients list will cause email to be sent to the github registered email address of the individual that triggered the build. Also note that all users will be prompted to re-auth due to the change in github oauth scopes necessary to lookup email addresses.

For Declarative pipelines, a mail step needs to be added to a post block. Here is an approximation of the behavior of the freestyle post build. Note that adding the console output requires another plugin or punching holes in the groovy security sandbox. I'd like to try to avoid the later as much as possible for TSSW due to our experience with DM.

pipeline {

  agent any

  stages {

    stage('build user') {

      steps {

        sh "exit 1"

      }

    }

  }

  post {

    failure {

      wrap([$class: 'BuildUser']) {

        mail(

          to:      env.BUILD_USER_EMAIL,

          subject: "Build failed in Jenkins: ${currentBuild.fullDisplayName}",

          body:    "See ${env.RUN_DISPLAY_URL}",

        )

      }

    }

  }

}

Please give it a try and let me know how it goes.

Email notifications seem to working once again.