Fix Version/s: None
Sprint:TSSW Sprint - Aug 15 - Aug 29, TSSW Sprint - Aug 29 - Sep 12
Team:Telescope and Site
The Kafka version of salobj uses aiokafka, which is based on kafka_python. Unfortunately kafka_python appears to be moribund; there have been no new releases for almost 2 years.
Switch to confluent_kafka, since it is maintained and the company claims it will continue to be. The API is not as nice, but it's good enough. Two other advantages are:
- salobj kafka already uses confluent_kafka for some things, because aiokafka is missing some needed APIs.
- confluent_kafka is designed to be high performance, so we might see a small speed increase (and are very unlikely to lose speed).
I did already test confluent_kafka for salobj, so I don't expect this to be a big job. Nonetheless I am leaving time to rerun the performance tests.
- relates to
DM-36072 Fix ReadTopic.aget to not take data from next
I started by updating kafkaprototype's confluent branch and trying some variations (including timing tests, which showed that several alternatives I tried all performed the same) to pick a final design.
I then applied that to ts_salobj, which is what is in review. In the process of doing so I found a number of race conditions that I had to fix, most of them applied to the DDS version of ts_salobj in
Pull request: https://github.com/lsst-ts/ts_salobj/pull/251