Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21897

Test the Confluent Replicator connector for EFD data replication

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As described in sqr-034.lsst.io we'll have EFD data replication from the Summit and test stands to the LDF. The natural way to implement that is by using the Confluent Replicator source connector.

      https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html

      In this ticket, we'll test topic replication from the Tucson test stand and from the Summit EFDs (source) to the EFD instance deployed at Google Cloud (destination)

      We'll also need to test schema migration. In our architecture we deploy the Kafka Schema Registry in both the source and destination clusters.

      https://docs.confluent.io/current/schema-registry/multidc.html#schemaregistry-mirroring

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          A requirement for configuring the connector is connectivity from the cluster that runs the connector to the source cluster.

          Usually, the connector is deployed at the destination cluster and pulls topics from the source cluster. For the Tucson lab EFD -> GKE EFD replication test we cannot deploy the connector at GKE because of the NOAO VPN. Thus, we'll try deploying the replicator connector at the source cluster.

          Show
          afausti Angelo Fausti added a comment - A requirement for configuring the connector is connectivity from the cluster that runs the connector to the source cluster. Usually, the connector is deployed at the destination cluster and pulls topics from the source cluster. For the Tucson lab EFD -> GKE EFD replication test we cannot deploy the connector at GKE because of the NOAO VPN. Thus, we'll try deploying the replicator connector at the source cluster.
          Hide
          afausti Angelo Fausti added a comment -

          This PR https://github.com/lsst-sqre/kafka-efd-demo/pull/26 adds the Replicator connector plugin to our Kafka Connect container image and provides an example of a configuration that includes topic replication, continuous schema migration, and schema translation.

          This PR https://github.com/lsst-sqre/sqr-034/pull/3 updates SQR-34 adding the concept of "source EFD" for the Summit and test stand deployments and "aggregator EFD" for the LDF deployment.

          There's also a new section on how to configure the Replicator source connector.

          Show
          afausti Angelo Fausti added a comment - This PR https://github.com/lsst-sqre/kafka-efd-demo/pull/26 adds the Replicator connector plugin to our Kafka Connect container image and provides an example of a configuration that includes topic replication, continuous schema migration, and schema translation. This PR https://github.com/lsst-sqre/sqr-034/pull/3 updates SQR-34 adding the concept of "source EFD" for the Summit and test stand deployments and "aggregator EFD" for the LDF deployment. There's also a new section on how to configure the Replicator source connector.

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel