Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:5.6
-
Epic Link:
-
Team:SQuaRE
Description
DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.
In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment.
Attachments
Issue Links
- blocks
-
DM-18778 Prototype for EFD LFA in Data Backbone
- Won't Fix
Activity
Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.
This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/
- Add discussion on fault tolerance with Kafka
- Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.
Complete version of the document with the following structure:
1 Introduction
2 The SAL Kafka producer
3 The Kafka Connect manager
4 Data replication and fault tolerance
5 Downsampling and data retention
6 The Aggregator
7 Options for long-term storage at the LDF
8 Monitoring
9 References
The technote on EFD Operations SQR-034 is shaping up
https://sqr-034.lsst.io/v/DM-21396/index.html