Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ts_middleware
-
Labels:
-
Story Points:5
-
Sprint:TSSW Sprint - Aug 15 - Aug 29, TSSW Sprint - Aug 29 - Sep 12
-
Team:Telescope and Site
-
Urgent?:No
Description
The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.
Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:
- salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
- confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).
I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.
Attachments
Issue Links
- relates to
-
DM-36072 Fix ReadTopic.aget to not take data from next
- Done
Activity
Sprint | TSSW Sprint - Aug 15 - Aug 29 [ 1183 ] | TSSW Sprint - Aug 15 - Aug 29, TSSW Sprint - Aug 29 - Sep 12 [ 1183, 1187 ] |
Status | To Do [ 10001 ] | In Progress [ 3 ] |
Reviewers | Tiago Ribeiro [ tribeiro ] | |
Status | In Progress [ 3 ] | In Review [ 10004 ] |
Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
Resolution | Done [ 10000 ] | |
Status | Reviewed [ 10101 ] | Done [ 10002 ] |
I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design.
I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in
DM-36072.Pull request: https://github.com/lsst-ts/ts_salobj/pull/251