Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21396

Write a technote on the EFD operations

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      DMTN-82 makes the initial proposal on how the EFD data would be available for the Science Platform. In SQR-029, we prototyped our EFD solution based on Kafka and InfluxDB. In SQR-031 we deployed the EFD at the lab in Tucson and at the Summit supporting AuxTel tests.

      In this technote, we'll detail the EFD operations, which includes the data flow from the Summit to LDF, data replication, fault tolerance, data retention policies and formats to store the raw EFD data for the lifetime of the experiment.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment -

            The technote on EFD Operations SQR-034 is shaping up

            https://sqr-034.lsst.io/v/DM-21396/index.html

            Show
            afausti Angelo Fausti added a comment - The technote on EFD Operations SQR-034 is shaping up https://sqr-034.lsst.io/v/DM-21396/index.html
            Hide
            afausti Angelo Fausti added a comment - - edited

            Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.

            Show
            afausti Angelo Fausti added a comment - - edited Still missing a description of the aggregator (I am looking at the Kafka Stream API) and long-term storage options at the LDF.
            Hide
            afausti Angelo Fausti added a comment -

            This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/

            Show
            afausti Angelo Fausti added a comment - This library for Kafka stream processing in Python looks like a good option for the Aggregator https://faust.readthedocs.io/en/latest/
            Hide
            afausti Angelo Fausti added a comment -
            • Add discussion on fault tolerance with Kafka
            • Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.
            Show
            afausti Angelo Fausti added a comment - Add discussion on fault tolerance with Kafka Disscuss downsampling of the time-series in InfluxDB to increase query performance over extended periods using multiple retention policies.
            Hide
            afausti Angelo Fausti added a comment -

            Complete version of the document with the following structure:

            1 Introduction
            2 The SAL Kafka producer
            3 The Kafka Connect manager
            4 Data replication and fault tolerance
            5 Downsampling and data retention
            6 The Aggregator
            7 Options for long-term storage at the LDF
            8 Monitoring
            9 References

            Show
            afausti Angelo Fausti added a comment - Complete version of the document with the following structure: 1 Introduction 2 The SAL Kafka producer 3 The Kafka Connect manager 4 Data replication and fault tolerance 5 Downsampling and data retention 6 The Aggregator 7 Options for long-term storage at the LDF 8 Monitoring 9 References
            Show
            afausti Angelo Fausti added a comment - https://github.com/lsst-sqre/sqr-034/pull/2

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti, Frossie Economou, Gregory Dubois-Felsmann, Kian-Tat Lim, Michelle Butler, Michelle Gower
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel