Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35920

Rewrite Kafka salobj to use confluent_kafka

    XMLWordPrintable

    Details

      Description

      The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.

      Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:

      • salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
      • confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).

      I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              rowen Russell Owen
              Reviewers:
              Tiago Ribeiro
              Watchers:
              Russell Owen, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.