Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24266

Investigate Faust for the kafka-aggregator app

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As proposed in DMTN-082 the LSP users are generally interested in telemetry data at a frequency closer to the cadence of the observations.

      The original proposal is that “all telemetry topics sampled with a frequency higher than 1Hz are (1) downsampled at 1Hz and (2) aggregated to 1Hz using general statistics like min, max, mean, median stdev”.

      Commands and event topics should not be aggregated as they are typically low-frequency and can be read directly from the raw EFD data sources.

      Another aspect of the aggregation is to resample the telemetry topics in a regular time grid to make it easier to correlate them.

      The aggregator will produce a new set of aggregated telemetry topics in Kafka that can be consumed by a connector and stored in a specific format. In another ticket we'll use the parquet data format to store the aggregated topics.

      In this ticket, we'll investigate and evaluate Faust, a Python asyncio stream processing library. Faust supports Avro serialization and multiple instances of a Faust worker can be started independently to distribute stream processing across nodes or CPU cores.

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          This ticket accounts for investigation work before adopting Faust. As a deliverable I would point to the aggregation example in the kafka-aggregator repository.

          See also the README in that repo for preliminary documentation about the use of Faust in this project.

          Show
          afausti Angelo Fausti added a comment - This ticket accounts for investigation work before adopting Faust. As a deliverable I would point to the aggregation example in the kafka-aggregator repository . See also the README in that repo for preliminary documentation about the use of Faust in this project.

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel