Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21713

InfluxDB-based services engineering

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sqre-f19-efd-2
    • Story Points:
      40
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Fall 2019

      Description

      This epic continues engineering of the EFD ecosystem deployment, including investigation on how to replicate the kafka stream from the primary summit EFD out to replicate EFDs.

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          I this epic, we investigated different approaches to deploy Kafka and give access to the cluster using both Nodeport and load balancers. We've added features to the kafka-connect-manager to select for instance the SAL timestamp we want to use as the InfluxDB timestamp and reviewed the error policy configuration so that the connector can restart itself if there's a problem with a specific topic. We have tested a new release (1.2.3) of the InfluxDB Sink Connector and found a regression - this version it is not handling arrays properly. We've written a technote for the EFD operation (SQR-034) and tested two new connectors successfully, the Confluent JDBC connector to write data to the Oracle database at LDF and the Replicator connector to replicate both topics and schemas from our source EFD to the aggregator EFD. That is also documented in SQR-034.

          I've started playing with Argo CD as an alternative to Terraform for managing the EFD deployments. The GitOps way to synchronize the deployment state (configurations and secrets) from GH with the cluster state is great. I've showed how to integrate Vault with ArgoCD to manage secrets. In a future epic we will finalize the Argo CD work.

          Show
          afausti Angelo Fausti added a comment - I this epic, we investigated different approaches to deploy Kafka and give access to the cluster using both Nodeport and load balancers. We've added features to the kafka-connect-manager to select for instance the SAL timestamp we want to use as the InfluxDB timestamp and reviewed the error policy configuration so that the connector can restart itself if there's a problem with a specific topic. We have tested a new release (1.2.3) of the InfluxDB Sink Connector and found a regression - this version it is not handling arrays properly. We've written a technote for the EFD operation (SQR-034) and tested two new connectors successfully, the Confluent JDBC connector to write data to the Oracle database at LDF and the Replicator connector to replicate both topics and schemas from our source EFD to the aggregator EFD. That is also documented in SQR-034. I've started playing with Argo CD as an alternative to Terraform for managing the EFD deployments. The GitOps way to synchronize the deployment state (configurations and secrets) from GH with the cluster state is great. I've showed how to integrate Vault with ArgoCD to manage secrets. In a future epic we will finalize the Argo CD work.
          Hide
          afausti Angelo Fausti added a comment -

          This epic can be reviewed. I need a new epic for the Argo CD work. I'm keeping my current open stories for the EFD in my backlog for now before moving to a new epic.

          Show
          afausti Angelo Fausti added a comment - This epic can be reviewed. I need a new epic for the Argo CD work. I'm keeping my current open stories for the EFD in my backlog for now before moving to a new epic.

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              frossie Frossie Economou
              Reviewers:
              Frossie Economou
              Watchers:
              Angelo Fausti, Frossie Economou
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel