Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24600

Create Faust Records dynamically from Avro schemas

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In kafka-aggregator, Faust agents aggregate the data stream on tumbling windows of fixed size (see DM-24403). Faust Records describe the data structure for the aggregated topics and with faust-avro we can serialize that to Avro.

      Generally, a field representing a value in the source topic will have the corresponding aggregation fields e.g. min, mean, median, and max in the destination topic. 

      In DM-24485 we demonstrated that we can parse a Faust Record into an Avro schema with faust-avro which allows us to register schemas for the aggregated topics in the Schema Registry.

      However, we still need to create the Faust Record for the aggregated topic in the first place.

      The goal for this ticket is to demonstrate we can create the Faust Record classes dynamically, otherwise, we would have to pre-generate a large amount of static code to create Faust Records for each schema we have in the EFD.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Using the make_record() to create Faust Records classes dynamically seems to be working fine. Surprisingly kafkafkaaggregator models and kafkaaggregator model MODEL commands are still working:

            Show
            afausti Angelo Fausti added a comment - - edited Using the make_record() to create Faust Records classes dynamically seems to be working fine. Surprisingly kafkafkaaggregator models and kafkaaggregator model MODEL commands are still working:
            Hide
            afausti Angelo Fausti added a comment -

            The kafkaaggregator register command used to register the corresponding Avro schema also works with dynamic records:

            Note that thanks to faust-avro, Avro schemas are created from Faust Records for free.

            Show
            afausti Angelo Fausti added a comment - The kafkaaggregator register command used to register the corresponding Avro schema also works with dynamic records: Note that thanks to faust-avro , Avro schemas are created from Faust Records for free.
            Hide
            afausti Angelo Fausti added a comment -

            Initial implementation of the Field, Topic, and Aggregator classes.

            Show
            afausti Angelo Fausti added a comment - Initial implementation of the Field , Topic , and Aggregator classes.
            Show
            afausti Angelo Fausti added a comment - See PR https://github.com/lsst-sqre/kafka-aggregator/pull/4
            Hide
            afausti Angelo Fausti added a comment -

            Jonathan Sick I would appreciate very much your review here.

            Show
            afausti Angelo Fausti added a comment - Jonathan Sick I would appreciate very much your review here.
            Hide
            jsick Jonathan Sick added a comment -

            Looks good here. Some comments via the PR https://github.com/lsst-sqre/kafka-aggregator/pull/5

            Show
            jsick Jonathan Sick added a comment - Looks good here. Some comments via the PR  https://github.com/lsst-sqre/kafka-aggregator/pull/5
            Hide
            afausti Angelo Fausti added a comment -

            Merged through DM-25012.

            Show
            afausti Angelo Fausti added a comment - Merged through DM-25012 .

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Reviewers:
                Jonathan Sick
                Watchers:
                Angelo Fausti, Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel