Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20516

Add Kafka Replicator Connector to kafka-connect-manager Helm chart

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      kafka-connect-manager is the project that manages the different connectors used in the EFD https://kafka-connect-manager.lsst.io/

      It is deployed via the kafka-connect-manager Helm Chart.

      In DM-23575 we add support to the Replicator connector in kafka-connect-manager. In this ticket, we add the necessary configuration to deploy the Replicator connector using the kafka-connect-manager Helm chart.

        Attachments

          Issue Links

            Activity

            afausti Angelo Fausti created issue -
            afausti Angelo Fausti made changes -
            Field Original Value New Value
            Epic Link DM-20445 [ 342674 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-20517 [ DM-20517 ]
            afausti Angelo Fausti made changes -
            Epic Link DM-20445 [ 342674 ] DM-21713 [ 424613 ]
            afausti Angelo Fausti made changes -
            Epic Link DM-21713 [ 424613 ] DM-5504 [ 23337 ]
            afausti Angelo Fausti made changes -
            Epic Link DM-5504 [ 23337 ] DM-22721 [ 427980 ]
            afausti Angelo Fausti made changes -
            Summary Create a Helm chart to deploy the Kafka Replicator Connector Add Kafka Replicator Connector to kafka-connect-manager helm chart
            afausti Angelo Fausti made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-23575 [ DM-23575 ]
            afausti Angelo Fausti made changes -
            Description Replication between two Kafka clusters is done through a special connector called Replicator.

            We can test replication using the DM-EFD instances running at the Tucson lab and at GKE. In this scenario, we want to run the SAL transform step just once. That can be accomplished but replicating only the transformed topics across the two deployments.

            This can be implemented as a Helm chart following what we did for the InfluxDB Sink Connector Helm chart.

            See [https://docs.confluent.io/current/multi-dc-replicator/replicator-install.html]

             
            {{kafka-connect-manager}} is the project that manages the different connectors used in the EFD https://kafka-connect-manager.lsst.io/

            It is deployed via the {{kafka-connect-manager}} Helm Chart.

            In DM-23575 we add support to the Replicator connector in {{kafka-connect-manager}}. In this ticket, we add the necessary configuration to deploy the Replicator connector using the {{kafka-connect-manager}} Helm chart.
            afausti Angelo Fausti made changes -
            Summary Add Kafka Replicator Connector to kafka-connect-manager helm chart Add Kafka Replicator Connector to kafka-connect-manager Helm chart
            Hide
            afausti Angelo Fausti added a comment -

            Update default values to use checkInterval in milliseconds in the Kafka Connect Helm chart.

            Show
            afausti Angelo Fausti added a comment - Update default values to use checkInterval in milliseconds in the Kafka Connect Helm chart.
            afausti Angelo Fausti made changes -
            Epic Link DM-22721 [ 427980 ] DM-5504 [ 23337 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-23973 [ DM-23973 ]
            afausti Angelo Fausti made changes -
            Epic Link DM-5504 [ 23337 ] DM-23917 [ 432182 ]
            Hide
            afausti Angelo Fausti added a comment - - edited

            This is done in version 0.4.0 of the kafka-connect-manager Helm chart and it is being used by the replicator app in Argo CD.

            Show
            afausti Angelo Fausti added a comment - - edited This is done in version 0.4.0 of the kafka-connect-manager Helm chart and it is being used by the replicator app in Argo CD.
            afausti Angelo Fausti made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel