Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35920

Rewrite Kafka salobj to use confluent_kafka

    XMLWordPrintable

    Details

      Description

      The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.

      Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:

      • salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
      • confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).

      I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.

        Attachments

          Issue Links

            Activity

            No builds found.
            rowen Russell Owen created issue -
            Hide
            rowen Russell Owen added a comment - - edited

            I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design.

            I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in DM-36072.

            Pull request: https://github.com/lsst-ts/ts_salobj/pull/251

            Show
            rowen Russell Owen added a comment - - edited I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design. I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in DM-36072 . Pull request: https://github.com/lsst-ts/ts_salobj/pull/251
            aclements Andy Clements made changes -
            Field Original Value New Value
            Epic Link DM-27720 [ 442125 ] DM-27721 [ 442128 ]
            wvreeven Wouter van Reeven made changes -
            Sprint TSSW Sprint - Aug 15 - Aug 29 [ 1183 ] TSSW Sprint - Aug 15 - Aug 29, TSSW Sprint - Aug 29 - Sep 12 [ 1183, 1187 ]
            rowen Russell Owen made changes -
            Link This issue relates to DM-36072 [ DM-36072 ]
            rowen Russell Owen made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            rowen Russell Owen made changes -
            Reviewers Tiago Ribeiro [ tribeiro ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            rowen Russell Owen added a comment -

            Reviewed on github

            Show
            rowen Russell Owen added a comment - Reviewed on github
            rowen Russell Owen made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            rowen Russell Owen made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Hide
            rowen Russell Owen added a comment -

            Merged to the kafka branch.

            Show
            rowen Russell Owen added a comment - Merged to the kafka branch.

              People

              Assignee:
              rowen Russell Owen
              Reporter:
              rowen Russell Owen
              Reviewers:
              Tiago Ribeiro
              Watchers:
              Russell Owen, Tiago Ribeiro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.