Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:10
-
Epic Link:
-
Team:SQuaRE
Description
As described in sqr-034.lsst.io we'll have EFD data replication from the Summit and test stands to the LDF. The natural way to implement that is by using the Confluent Replicator source connector.
https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html
In this ticket, we'll test topic replication from the Tucson test stand and from the Summit EFDs (source) to the EFD instance deployed at Google Cloud (destination)
We'll also need to test schema migration. In our architecture we deploy the Kafka Schema Registry in both the source and destination clusters.
https://docs.confluent.io/current/schema-registry/multidc.html#schemaregistry-mirroring
Attachments
Activity
This PR https://github.com/lsst-sqre/kafka-efd-demo/pull/26 adds the Replicator connector plugin to our Kafka Connect container image and provides an example of a configuration that includes topic replication, continuous schema migration, and schema translation.
This PR https://github.com/lsst-sqre/sqr-034/pull/3 updates SQR-34 adding the concept of "source EFD" for the Summit and test stand deployments and "aggregator EFD" for the LDF deployment.
There's also a new section on how to configure the Replicator source connector.
A requirement for configuring the connector is connectivity from the cluster that runs the connector to the source cluster.
Usually, the connector is deployed at the destination cluster and pulls topics from the source cluster. For the Tucson lab EFD -> GKE EFD replication test we cannot deploy the connector at GKE because of the NOAO VPN. Thus, we'll try deploying the replicator connector at the source cluster.