Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21396

Write a technote on the EFD operations

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.

      In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment.

      Attachments

        Issue Links

          Activity

            The technote on EFD Operations SQR-034 is shaping up

            https://sqr-034.lsst.io/v/DM-21396/index.html

            afausti Angelo Fausti added a comment - The technote on EFD Operations SQR-034 is shaping up https://sqr-034.lsst.io/v/DM-21396/index.html
            afausti Angelo Fausti added a comment - - edited

            Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.

            afausti Angelo Fausti added a comment - - edited Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.

            This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/

            afausti Angelo Fausti added a comment - This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/
            • Add discussion on fault tolerance with Kafka
            • Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.
            afausti Angelo Fausti added a comment - Add discussion on fault tolerance with Kafka Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.

            Complete version of the document with the following structure:

            1 Introduction
            2 The SAL Kafka producer
            3 The Kafka Connect manager
            4 Data replication and fault tolerance
            5 Downsampling and data retention
            6 The Aggregator
            7 Options for long-term storage at the LDF
            8 Monitoring
            9 References

            afausti Angelo Fausti added a comment - Complete version of the document with the following structure: 1 Introduction 2 The SAL Kafka producer 3 The Kafka Connect manager 4 Data replication and fault tolerance 5 Downsampling and data retention 6 The Aggregator 7 Options for long-term storage at the LDF 8 Monitoring 9 References
            afausti Angelo Fausti added a comment - https://github.com/lsst-sqre/sqr-034/pull/2

            People

              afausti Angelo Fausti
              afausti Angelo Fausti
              Angelo Fausti, Frossie Economou, Gregory Dubois-Felsmann, Kian-Tat Lim, Michelle Butler [X] (Inactive), Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.