Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17540

TSSW Jenkins email notifications not working

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Test Data
    • Labels:
      None

      Description

      The email plugin was installed and configured to send out email notifications after a build.  This was setup for the ts_xml and ts_sal jobs.  Emails stopped working a couple of months ago.  The build logs show they are sent, but the recipients never get the messages.  I asked IT, as I thought the spam filter might be catching them, but they confirmed that no traffic from Jenkins is being blocked.  My next thought is that the messages are being blocked at the source, AWS.  Since this is not an area I can access, I am filing a ticket.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Last week I was able to demonstrate working email notifications in a test env using aws ses for delivery. It appeared that the extra features of the jenkins email-ext and emailext-template plugins weren't being used and that the more basic mailer plugin would be sufficient. email-ext requires duplicate configuration with mailer, which it also depends on,and doesn't provider a "test" feature either. I suspect this duplicate configuration has contributed to the email problems in the past.

            The configuration of aws-ses has been terraform-erized and a PR is open to update the jenkins deployment to fully configure the mailer plugin, remove the unneeded mail related plugins, and change the github oauth scopes to allow user email addresses to be retrieved. I prefer to do a restart when an auth change is made to ensure there aren't any lurking issues with cached metadata and a maintenance period needs to be scheduled to roll out these changes.

            Show
            jhoblitt Joshua Hoblitt added a comment - Last week I was able to demonstrate working email notifications in a test env using aws ses for delivery. It appeared that the extra features of the jenkins email-ext and emailext-template plugins weren't being used and that the more basic mailer plugin would be sufficient. email-ext requires duplicate configuration with mailer , which it also depends on,and doesn't provider a "test" feature either. I suspect this duplicate configuration has contributed to the email problems in the past. The configuration of aws-ses has been terraform-erized and a PR is open to update the jenkins deployment to fully configure the mailer plugin, remove the unneeded mail related plugins, and change the github oauth scopes to allow user email addresses to be retrieved. I prefer to do a restart when an auth change is made to ensure there aren't any lurking issues with cached metadata and a maintenance period needs to be scheduled to roll out these changes.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Maintenance for next Monday has been announced on #ts-software.

            @here https://ts-ci.lsst.codes/ will be undergoing maintenance this coming Monday 02/18 (President's day) at 1600 US/Arizona (Tucson local time).
            

            Show
            jhoblitt Joshua Hoblitt added a comment - Maintenance for next Monday has been announced on #ts-software . @here https: //ts-ci.lsst.codes/ will be undergoing maintenance this coming Monday 02/18 (President's day) at 1600 US/Arizona (Tucson local time).
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The deployment didn't go as smoothly as planned and the global credentials (those not tied to "folders") were removed by the CASC plugin as due to a configuration error on my part. Instead of immediately putting the credentials back in place, I went down a several hour rabbit hole of auditing how credentials were being used. In the vast majority of cases, credentials were configured on clones of git repos under the github lsst-ts org that were public, and thus credentials weren't required. I believe I removed most if not all of these cases. I also converted a number of uses of rbovill github credentials to uses those of the lssttsadmin github "robot" user that I've created. The two examples that needed existing credentials were the setup of ssh agent by the ts_scheduler job and for access to https://gitlab.tekniker.es/sai/projects/3151-LSST/lsst.git by the TMA_Simulator job. In the end, I left my removal/conversion of github credentials in place and restored all of the original global credentials. However, I believe the majority of the global credentials are unused may be removed in the future.

            Show
            jhoblitt Joshua Hoblitt added a comment - The deployment didn't go as smoothly as planned and the global credentials (those not tied to "folders") were removed by the CASC plugin as due to a configuration error on my part. Instead of immediately putting the credentials back in place, I went down a several hour rabbit hole of auditing how credentials were being used. In the vast majority of cases, credentials were configured on clones of git repos under the github lsst-ts org that were public, and thus credentials weren't required. I believe I removed most if not all of these cases. I also converted a number of uses of rbovill github credentials to uses those of the lssttsadmin github "robot" user that I've created. The two examples that needed existing credentials were the setup of ssh agent by the ts_scheduler job and for access to https://gitlab.tekniker.es/sai/projects/3151-LSST/lsst.git by the TMA_Simulator job. In the end, I left my removal/conversion of github credentials in place and restored all of the original global credentials. However, I believe the majority of the global credentials are unused may be removed in the future.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Rob Bovill Some action is required to configure email notifications when a build fails. For existing "free style" jobs, the "Set jenkins user build variables" binding needs to be set to make the BUILD_USER_EMAIL env var available. A Email Notification post build action is also necessary. Including {$BUILD_USER_EMAIL}} in the Recipients list will cause email to be sent to the github registered email address of the individual that triggered the build. Also note that all users will be prompted to re-auth due to the change in github oauth scopes necessary to lookup email addresses.

            For Declarative pipelines, a mail step needs to be added to a post block. Here is an approximation of the behavior of the freestyle post build. Note that adding the console output requires another plugin or punching holes in the groovy security sandbox. I'd like to try to avoid the later as much as possible for TSSW due to our experience with DM.

            pipeline {
              agent any
             
              stages {
                stage('build user') {
                  steps {
                    sh "exit 1"
                  }
                }
              }
             
              post {
                failure {
                  wrap([$class: 'BuildUser']) {
                    mail(
                      to:      env.BUILD_USER_EMAIL,
                      subject: "Build failed in Jenkins: ${currentBuild.fullDisplayName}",
                      body:    "See ${env.RUN_DISPLAY_URL}",
                    )
                  }
                }
              }
            }
            

            Please give it a try and let me know how it goes.

            Show
            jhoblitt Joshua Hoblitt added a comment - Rob Bovill Some action is required to configure email notifications when a build fails. For existing "free style" jobs, the "Set jenkins user build variables" binding needs to be set to make the BUILD_USER_EMAIL env var available. A Email Notification post build action is also necessary. Including {$BUILD_USER_EMAIL}} in the Recipients list will cause email to be sent to the github registered email address of the individual that triggered the build. Also note that all users will be prompted to re-auth due to the change in github oauth scopes necessary to lookup email addresses. For Declarative pipelines, a mail step needs to be added to a post block. Here is an approximation of the behavior of the freestyle post build. Note that adding the console output requires another plugin or punching holes in the groovy security sandbox. I'd like to try to avoid the later as much as possible for TSSW due to our experience with DM. pipeline { agent any   stages { stage( 'build user' ) { steps { sh "exit 1" } } }   post { failure { wrap([$ class : 'BuildUser' ]) { mail( to: env.BUILD_USER_EMAIL, subject: "Build failed in Jenkins: ${currentBuild.fullDisplayName}" , body: "See ${env.RUN_DISPLAY_URL}" , ) } } } } Please give it a try and let me know how it goes.
            Hide
            rbovill Rob Bovill added a comment -

            Email notifications seem to working once again.

            Show
            rbovill Rob Bovill added a comment - Email notifications seem to working once again.

              People

              • Assignee:
                jhoblitt Joshua Hoblitt
                Reporter:
                rbovill Rob Bovill
                Reviewers:
                Rob Bovill
                Watchers:
                Andy Clements, Frossie Economou, Iain Goodenow, Joshua Hoblitt, Rob Bovill
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel