Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:5.6
-
Epic Link:
-
Team:SQuaRE
Description
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.
In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment.
Attachments
Issue Links
- blocks
-
DM-18778 Prototype for EFD LFA in Data Backbone
- Won't Fix
Activity
| Field | Original Value | New Value |
|---|---|---|
| Epic Link |
|
| Description |
DMTN-82 makes the initial propose on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components, see SQR-031.
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Watchers | Angelo Fausti, Frossie Economou [ Angelo Fausti, Frossie Economou ] | Angelo Fausti, Frossie Economou, Kian-Tat Lim, Michelle Butler [ Angelo Fausti, Frossie Economou, Kian-Tat Lim, Michelle Butler ] |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, and in particular, Kafka Connect to write data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, writing data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.
This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/
- Add discussion on fault tolerance with Kafka
- Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.
| Summary | Write technote proposing the EFD end-to-end data workflow | Write a technote on the EFD operations |
| Description |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In the meantime, in SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB and deployed the EFD at the lab in Tucson and at the Summit testing AuxTel components (SQR-031)
In this technote, we'll detail the EFD data workflow based on initial requirements, backup needs and on what learned so far using Kafka for multi-datacenter replication, writing data to the Oracle database at NCSA and to other formats that are useful for the Science Platform like parquet. |
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.
In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment. |
Complete version of the document with the following structure:
1 Introduction
2 The SAL Kafka producer
3 The Kafka Connect manager
4 Data replication and fault tolerance
5 Downsampling and data retention
6 The Aggregator
7 Options for long-term storage at the LDF
8 Monitoring
9 References
| Story Points | 2.8 | 5.6 |
| Resolution | Done [ 10000 ] | |
| Status | In Progress [ 3 ] | Done [ 10002 ] |
The technote on EFD Operations SQR-034 is shaping up
https://sqr-034.lsst.io/v/DM-21396/index.html