Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35920

Rewrite Kafka salobj to use confluent_kafka

    XMLWordPrintable

    Details

      Description

      The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.

      Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:

      • salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
      • confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).

      I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment - - edited

            I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design.

            I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in DM-36072.

            Pull request: https://github.com/lsst-ts/ts_salobj/pull/251

            Show
            rowen Russell Owen added a comment - - edited I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design. I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in DM-36072 . Pull request: https://github.com/lsst-ts/ts_salobj/pull/251
            Hide
            rowen Russell Owen added a comment -

            Reviewed on github

            Show
            rowen Russell Owen added a comment - Reviewed on github
            Hide
            rowen Russell Owen added a comment -

            Merged to the kafka branch.

            Show
            rowen Russell Owen added a comment - Merged to the kafka branch.

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              rowen Russell Owen
              Reviewers:
              Tiago Ribeiro
              Watchers:
              Russell Owen, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.