Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:4.2
-
Epic Link:
-
Team:SQuaRE
-
Urgent?:No
Description
The definition of which timestamp is used as the InfluxDB time is done in the connector configuration. The default behavior is to use sys_time(). We want to replace that by private_sndStamp obtained from the SAL topics.
In DM-21391, we implemented the --timestamp option in the Kafka connect manager to configure the connector:
and in this PR we changed the connector to support to Unix timestamps as double
https://github.com/lensesio/stream-reactor/pull/626
That was released in the version 1.2.3 of the connector:
https://github.com/lensesio/stream-reactor/releases/tag/1.2.3
but, unfortunately, that version introduced a regression and cannot be used (DM-21895).
In this ticket, we'll build a special version of the connector previous to 1.23 that supports Unix timestamps as double.
We'll also need to update the kafka connect manager Helm chart so that we can configure which timestamp to use.
Then, we'll deploy the new connector in the Tucson teststand. For testing the connector with the private_sndStamp we'll send data to a second database.
Attachments
Issue Links
- relates to
-
DM-24000 Enable private_sndStamp as the InfluxDB timestamp for the Summit EFD
- Done
-
DM-21895 Investigate regression in InfluxDB Sink Connector version 1.2.3
- To Do
-
DM-21391 Add the ability to select which field to use as the InfluxDB timestamp
- Done
-
DM-21419 The definition of which timestamp the EFD uses should be done by the SAL Kafka producer
- Done
Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD.
There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.