Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24485

Investigate solutions to enabled Avro in a Faust application

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There are at least two packages of interest in PyPI to enable Avro in Faust applications.

      I first looked at the python-schema-registry-client package which implements a synchronous Confluent Schema Registry client and a codec for Faust.

      See more about Faust codecs and using different serializers in the Faust documentation.

      The other package out there is faust-avro which also implements an Avro codec and Confluent Schema Registry support for Faust.

      Looking at the code faust-avro seems more attractive for its hability to parse a Faust record into an Avro schema I think that's an important feature missing in the python-schema-registry-client which assumes the Avro schema is provided in addition to the Faust record (see here).

      In addition, the faust-avro implementation is asyncio based. A drawback, however, is that it subclasses the Faust base classes like faust.app.App, faust.serializers.codec.Codec, faust.serializers.schemas.Schemas, faust.models.record.Record, faust.types.topic.Topic, etc but that also makes using it much easier as we show in this ticket.

      Perhaps a hybrid solution would be to use only the parsers implemented in faust-avro and our own library kafkit that already implements an async schema registry client.

      I think that implies rewriting faust_avro/serializers.py using kafkit as a codec for Faust similar to what python-schema-registry-client does.  However, I don't know if that's possible without changing the faust-avro source code.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Jonathan Sick this is a very simple PR that shows how to convert the test topic aggregation example to use Avro serialized messages with faust-avro.

            This option is attractive because it worked out of the box, so I think I'll continue using faust-avro and test it with real-life topics from SAL.

            https://github.com/lsst-sqre/kafka-aggregator/pull/2

            Show
            afausti Angelo Fausti added a comment - - edited Jonathan Sick this is a very simple PR that shows how to convert the test topic aggregation example to use Avro serialized messages with faust-avro . This option is attractive because it worked out of the box, so I think I'll continue using faust-avro  and test it with real-life topics from SAL. https://github.com/lsst-sqre/kafka-aggregator/pull/2
            Hide
            afausti Angelo Fausti added a comment -

            Jonathan Sick I've added one more commit to the PR, it is ready to review now. 

            Show
            afausti Angelo Fausti added a comment - Jonathan Sick I've added one more commit to the PR, it is ready to review now. 
            Hide
            jsick Jonathan Sick added a comment - - edited

            Comments on the PR. I'm not totally sold on faust-avro for the reasons you specify and the fact that faust-avro is pretty new and made by just one organization. That said, it works so it seems good to use it to get going.

            Show
            jsick Jonathan Sick added a comment - - edited Comments on the PR. I'm not totally sold on faust-avro for the reasons you specify and the fact that faust-avro is pretty new and made by just one organization. That said, it works so it seems good to use it to get going.
            Hide
            afausti Angelo Fausti added a comment -

            Thanks for the suggestions!

            Show
            afausti Angelo Fausti added a comment - Thanks for the suggestions!

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Reviewers:
                Jonathan Sick
                Watchers:
                Angelo Fausti, Jonathan Sick
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel