Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23917

EFD replication service and support

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sqre-s20-efd-2
    • Story Points:
      72
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Spring 2020

      Attachments

      Stories in Epic (Custom Issue Matrix)

      Key Summary Story Points Assignee Status
       
      DM-23575

      Add support to Kafka Replicator connector in kafka-connect-manager

      4.2 Angelo Fausti Done
       
      DM-23446

      Deploy the EFD on the LSP stable cluster at NCSA

      1.4 Angelo Fausti Done
       
      DM-23419

      Implementation plan for the LDF EFD

      2.8 Angelo Fausti Done
       
      DM-23858

      Add cp-helm-charts to int and stable argocd

      Christine Banek Done
       
      DM-20516

      Add Kafka Replicator Connector to kafka-connect-manager Helm chart

      2.8 Angelo Fausti Done
       
      DM-24856

      Register aggregated topic schemas to a secondary schema registry

      2.8 Angelo Fausti Done
       
      DM-25099

      Scale up Kafka Connect to 3 replicas for the EFD at the ncsa-teststand cluster

      0.7 Angelo Fausti Done
       
      DM-24779

      Investigate alternatives for the Parquet Sink connector

      0.7 Angelo Fausti Done
       
      DM-24774

      Plan to replace the Confluent replicator connector and other enterprise components of Confluent Kafka

      1.4 Angelo Fausti Done
       
      DM-24772

      Refresh versions for the EFD apps

      2.8 Angelo Fausti Done
       
      DM-24600

      Create Faust Records dynamically from Avro schemas

      5.6 Angelo Fausti Done
       
      DM-24688

      Deploy the EFD on the Kueyen cluster at the base

      2.8 Angelo Fausti Done
       
      DM-24485

      Investigate solutions to enabled Avro in a Faust application

      4.2 Angelo Fausti Done
       
      DM-24403

      Investigate Faust windowing feature for data aggregation

      5.6 Angelo Fausti Done
       
      DM-23924

      Latency characterization for the Summit EFD with M1M3 +M2 data streams

      7 Angelo Fausti Done
       
      DM-23961

      Dump Summit EFD data and restore at NCSA before the Observatory Shutdwon

      1.4 Angelo Fausti Done
       
      DM-25190

      Investigate Control Center instability on NCSA teststand cluster

      2.8 Angelo Fausti Done
       
      DM-25172

      Redeploy Kafka on the NCSA test stand and investigations after the cluster intervention

      1.4 Angelo Fausti Done
       
      DM-24000

      Enable private_sndStamp as the InfluxDB timestamp for the Summit EFD

      1.4 Angelo Fausti Done
       
      DM-23990

      Deploy the EFD on the LSP integration cluster at NCSA

      5.6 Angelo Fausti Done
       
      DM-23973

      Deploy the EFD replicator on the LSP integration cluster

      4.2 Angelo Fausti Done
       
      DM-24266

      Investigate Faust for the kafka-aggregator app

      4.2 Angelo Fausti Done
       
      DM-25012

      Implement the compute() method in the Aggregator class

      4.2 Angelo Fausti Done

        Activity

        Hide
        afausti Angelo Fausti added a comment - - edited

        Summary of the activities developed in this epic:

        • Deployment of the EFD at NCSA on Int and Stable clusters
        • Latency characterization of the M1M3 and M2 real data at the Summit EFD
        • Deployment of the Confluent Replicator connector on Int and data replication tests from the Tucson test stand EFD
        • Migration of the Summit EFD data to the NCSA before the observatory shutdown
        • Deployment of the EFD at the Base on the Kueyen cluster
        • Initial implementation of the Kafka Aggregator
        • Maintenance of the different EFD instances
        Show
        afausti Angelo Fausti added a comment - - edited Summary of the activities developed in this epic: Deployment of the EFD at NCSA on Int and Stable clusters Latency characterization of the M1M3 and M2 real data at the Summit EFD Deployment of the Confluent Replicator connector on Int and data replication tests from the Tucson test stand EFD Migration of the Summit EFD data to the NCSA before the observatory shutdown Deployment of the EFD at the Base on the Kueyen cluster Initial implementation of the Kafka Aggregator Maintenance of the different EFD instances
        Hide
        frossie Frossie Economou added a comment -

        This cycle saw intense demands for Angelo's time from Commissioning and Integration activities, including supporting deployments at the summit, base, and test stands. As a result some of the aggregation work has been displaced to F20A. The T/CAM will do a better job accounting for these demands in future cycles.

        Show
        frossie Frossie Economou added a comment - This cycle saw intense demands for Angelo's time from Commissioning and Integration activities, including supporting deployments at the summit, base, and test stands. As a result some of the aggregation work has been displaced to F20A. The T/CAM will do a better job accounting for these demands in future cycles.

          People

          • Assignee:
            afausti Angelo Fausti
            Reporter:
            frossie Frossie Economou
            Reviewers:
            Frossie Economou
            Watchers:
            Angelo Fausti, Frossie Economou
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Summary Panel