Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27399

Send Kafka JMX metrics to InfluxDB using the Telegraf Prometheus input plugin

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Confluent Platform exposes Kafka metrics using the Prometheus Java Management Extensions (JMX) exporter.

      In this ticket we demonstrate that it is possible to use the Telegraf Prometheus input plugin to collect the JMX metrics and ship them to InfluxDB. Essentially we need to implement the following workflow:

      JMX metrics > Prometheus JMX exporter > Telegraf Prometheus input plugin > InfluxDB

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-28873 Send MirrorMaker 2 metrics to InfluxDB for monitoring

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27274 Add Kafdrop to argocd-efd deployment

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-27376 Kafka monitoring sprint presentation and kafdrop demo

          • Done

            Activity

            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            From our EFD deployment (Confluent Kafka) we first verified that we can scrape the Prometheus endpoint from the Kafka pods.

            kubectl exec -n cp-helm-charts -c cp-schema-registry-server cp-helm-charts-cp-schema-registry-6bc49b7878-p9p5v  curl http://localhost:5556/metrics
            		 
             
            		 
            kubectl exec -n cp-helm-charts -c cp-kafka-connect-server cp-helm-charts-cp-kafka-connect-64899d5484-cfck5 curl http://localhost:5556/metrics

            Then we added Telegraf to the EFD deployment:

            https://github.com/lsst-sqre/argocd-efd/tree/master/apps/telegraf

            We also need a service to access the Prometheus endpoint from the Telegraf pod, for example:

            https://github.com/lsst-sqre/argocd-efd/blob/master/apps/telegraf/templates/service-cp-kafka-connect-jmx.yaml

            and here's the Telegraf configuration, for example for the summit environment:

            https://github.com/lsst-sqre/argocd-efd/blob/master/apps/telegraf/values-summit.yaml

            Show
            afausti Angelo Fausti added a comment - - edited From our EFD deployment (Confluent Kafka) we first verified that we can scrape the Prometheus endpoint from the Kafka pods. kubectl exec -n cp-helm-charts -c cp-schema-registry-server cp-helm-charts-cp-schema-registry-6bc49b7878-p9p5v curl http://localhost:5556/metrics   kubectl exec -n cp-helm-charts -c cp-kafka-connect-server cp-helm-charts-cp-kafka-connect-64899d5484-cfck5 curl http://localhost:5556/metrics Then we added Telegraf to the EFD deployment: https://github.com/lsst-sqre/argocd-efd/tree/master/apps/telegraf We also need a service to access the Prometheus endpoint from the Telegraf pod, for example: https://github.com/lsst-sqre/argocd-efd/blob/master/apps/telegraf/templates/service-cp-kafka-connect-jmx.yaml and here's the Telegraf configuration, for example for the summit environment: https://github.com/lsst-sqre/argocd-efd/blob/master/apps/telegraf/values-summit.yaml

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.