Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:5.6
-
Epic Link:
-
Team:SQuaRE
Description
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.
In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment.
Attachments
Issue Links
- blocks
-
DM-18778 Prototype for EFD LFA in Data Backbone
- Won't Fix
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link |
|
| Description |
DMTN-82 makes the initial propose on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Watchers | Angelo Fausti, Frossie Economou [ Angelo Fausti, Frossie Economou ] | Angelo Fausti, Frossie Economou, Kian-Tat Lim, Michelle Butler [ Angelo Fausti, Frossie Economou, Kian-Tat Lim, Michelle Butler ] |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, writing data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Summary | Write technote proposing the EFD end-to-end data workflow | Write a technote on the EFD operations |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, writing data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.
In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment. |
| Story Points | 2.8 | 5.6 |
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Done [ 10002 ] |