Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14934

jenkins agent aws instances need manual intervention after reboot

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Continuous Integration
    • Labels:
      None

      Description

      The jenkins "swarm" client must use DNS based discovery of the master as multicast is not support on aws (or most public clouds). Presently, an /etc/hosts entry is injected by puppet during configuration but this is lost as a reboot because it is overwritten by cloud-init. Forcing the hostname via cloud-init is needed as aws seems to lack a way to set a hostname via dhcp options (custom magic could probably use the metadata service).

      A local /etc/hosts entry could be avoided by using a vpc private route53 zone, which is straight forward to setup via tf.

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I've merged tf/vagrant/cloud-init/puppet changes to switch to using private DNS for the jenkins master discovery but have not deployed to production as there is another change that I'd like to make at the same time. This ticket will be left in self-review until its has been deployed.

          Show
          jhoblitt Joshua Hoblitt added a comment - I've merged tf/vagrant/cloud-init/puppet changes to switch to using private DNS for the jenkins master discovery but have not deployed to production as there is another change that I'd like to make at the same time. This ticket will be left in self-review until its has been deployed.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I've reconsidered and want to roll this out now as the other change that was planned may have some performance impact.

          Show
          jhoblitt Joshua Hoblitt added a comment - I've reconsidered and want to roll this out now as the other change that was planned may have some performance impact.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          The prod jenkins tf deployment has been updated and all of the linux agent instances have been rebooted with the exception of el7-5 which is busy. All nodes were able to reconnect to the jenkins master without intervention.

          Show
          jhoblitt Joshua Hoblitt added a comment - The prod jenkins tf deployment has been updated and all of the linux agent instances have been rebooted with the exception of el7-5 which is busy. All nodes were able to reconnect to the jenkins master without intervention.

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              jhoblitt Joshua Hoblitt
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Adam Thornton, Frossie Economou, Joshua Hoblitt
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel