Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24774

Plan to replace the Confluent replicator connector and other enterprise components of Confluent Kafka

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      After negotiating pricing with Confluent the conclusion was that we cannot afford their enterprise license. The license is all-inclusive and it costs 10k per node/yr.

      We are currently managing a minimum of 3 nodes * 6 instances of the EFD which means 180k/yr. Also, they consider each component of the Confluent platform running on a different node, so we cannot deploy the Schema Registry and Zookeeper in the same node to reduce costs, for example.

      The decision was to replace the Replicator connector with an opensource alternative, Mirus seems like a good option for kafka replication, it also uses Kafka connect and is superior to Mirror Maker, see this blog post.

      The other enterprise product we were using is the Confluent Control center, its been very useful to monitor and diagnose the Kafka clusters. An interesting alternative for the control center is the Lenses dashboards

      We plan on investigating these alternatives in the next epic for the EFD.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Plan for replacing the enterprise components of Confluent Kafka (3-4 weeks of work):

            • Get familiar with Strimzi, read docs, k8s deployment broker + zookeeper : 2 days
            • k8s deployment broker + Kafka Connect + Schema Registry: 2 days
            • Make the InfluxDB Sink connector work with Strimzi Kafka connect, build a kafka connect image with the connector plugin, deploy and test: 2 days
            • Refresh Kafka-connect-manager - hopefully, it will just work with Strimzi kafka connect, i.e. I expect it has the same API as the Confluent Kafka connect: 3 days
            • Argo CD deployment: broker + zookeeper + kafka connect + kafka connect manager + schema registry: 2 days
            • Investigate Mirus open source alternative for the Replicator connector, add Mirus plugin to kafka connect and test with Strimzi:  4 days
            • Add Mirus to Argo CD deployment and make a full replication test between two kafka clusters: 2 days

             

            Show
            afausti Angelo Fausti added a comment - - edited Plan for replacing the enterprise components of Confluent Kafka (3-4 weeks of work): Get familiar with Strimzi, read docs, k8s deployment broker + zookeeper : 2 days k8s deployment broker + Kafka Connect + Schema Registry: 2 days Make the InfluxDB Sink connector work with Strimzi Kafka connect, build a kafka connect image with the connector plugin, deploy and test: 2 days Refresh Kafka-connect-manager - hopefully, it will just work with Strimzi kafka connect, i.e. I expect it has the same API as the Confluent Kafka connect: 3 days Argo CD deployment: broker + zookeeper + kafka connect + kafka connect manager + schema registry: 2 days Investigate Mirus open source alternative for the Replicator connector, add Mirus plugin to kafka connect and test with Strimzi:  4 days Add Mirus to Argo CD deployment and make a full replication test between two kafka clusters: 2 days  

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel