Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35920

Rewrite Kafka salobj to use confluent_kafka

    XMLWordPrintable

Details

    Description

      The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.

      Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:

      • salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
      • confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).

      I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.

      Attachments

        Issue Links

          Activity

            People

              rowen Russell Owen
              rowen Russell Owen
              Tiago Ribeiro
              Russell Owen, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.