Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21713

InfluxDB-based services engineering

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sqre-f19-efd-2
    • Story Points:
      40
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Fall 2019

      Description

      This epic continues engineering of the EFD ecosystem deployment, including investigation on how to replicate the kafka stream from the primary summit EFD out to replicate EFDs.

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          I this epic, we investigated different approaches to deploy Kafka and give access to the cluster using both Nodeport and load balancers. We've added features to the kafka-connect-manager to select for instance the SAL timestamp we want to use as the InfluxDB timestamp and reviewed the error policy configuration so that the connector can restart itself if there's a problem with a specific topic. We have tested a new release (1.2.3) of the InfluxDB Sink Connector and found a regression - this version it is not handling arrays properly. We've written a technote for the EFD operation (SQR-034) and tested two new connectors successfully, the Confluent JDBC connector to write data to the Oracle database at LDF and the Replicator connector to replicate both topics and schemas from our source EFD to the aggregator EFD. That is also documented in SQR-034.

          I've started playing with Argo CD as an alternative to Terraform for managing the EFD deployments. The GitOps way to synchronize the deployment state (configurations and secrets) from GH with the cluster state is great. I've showed how to integrate Vault with ArgoCD to manage secrets. In a future epic we will finalize the Argo CD work.

          Show
          afausti Angelo Fausti added a comment - I this epic, we investigated different approaches to deploy Kafka and give access to the cluster using both Nodeport and load balancers. We've added features to the kafka-connect-manager to select for instance the SAL timestamp we want to use as the InfluxDB timestamp and reviewed the error policy configuration so that the connector can restart itself if there's a problem with a specific topic. We have tested a new release (1.2.3) of the InfluxDB Sink Connector and found a regression - this version it is not handling arrays properly. We've written a technote for the EFD operation (SQR-034) and tested two new connectors successfully, the Confluent JDBC connector to write data to the Oracle database at LDF and the Replicator connector to replicate both topics and schemas from our source EFD to the aggregator EFD. That is also documented in SQR-034. I've started playing with Argo CD as an alternative to Terraform for managing the EFD deployments. The GitOps way to synchronize the deployment state (configurations and secrets) from GH with the cluster state is great. I've showed how to integrate Vault with ArgoCD to manage secrets. In a future epic we will finalize the Argo CD work.
          Hide
          afausti Angelo Fausti added a comment -

          This epic can be reviewed. I need a new epic for the Argo CD work. I'm keeping my current open stories for the EFD in my backlog for now before moving to a new epic.

          Show
          afausti Angelo Fausti added a comment - This epic can be reviewed. I need a new epic for the Argo CD work. I'm keeping my current open stories for the EFD in my backlog for now before moving to a new epic.

            People

            Assignee:
            afausti Angelo Fausti
            Reporter:
            frossie Frossie Economou
            Reviewers:
            Frossie Economou
            Watchers:
            Angelo Fausti, Frossie Economou
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Summary Panel