Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:4.2
-
Epic Link:
-
Team:SQuaRE
-
Urgent?:No
Description
The definition of which timestamp is used as the InfluxDB time is done in the connector configuration. The default behavior is to use sys_time(). We want to replace that by private_sndStamp obtained from the SAL topics.
In DM-21391, we implemented the --timestamp option in the Kafka connect manager to configure the connector:
and in this PR we changed the connector to support to Unix timestamps as double
https://github.com/lensesio/stream-reactor/pull/626
That was released in the version 1.2.3 of the connector:
https://github.com/lensesio/stream-reactor/releases/tag/1.2.3
but, unfortunately, that version introduced a regression and cannot be used (DM-21895).
In this ticket, we'll build a special version of the connector previous to 1.23 that supports Unix timestamps as double.
We'll also need to update the kafka connect manager Helm chart so that we can configure which timestamp to use.
Then, we'll deploy the new connector in the Tucson teststand. For testing the connector with the private_sndStamp we'll send data to a second database.
Attachments
Issue Links
- relates to
-
- Done
-
- Done
-
- Done
-
- Done
Activity
Waiting on data to flow to the Tucson Teststand EFD again to test this implementation.
Michael Reuter a quick look using a query like this:
SELECT "private_sndStamp" FROM "efd-test-timestamp"."autogen"."lsst.sal.DSM.logevent_heartbeat"
|
shows that the InfluxDB time corresponds now to the private_sndStamp field. Note that the time precision is milliseconds, which is how it was implemented in the connector https://github.com/lensesio/stream-reactor/pull/626
I'm giving tacit approval to this. Preliminary checks look good, but I haven't gone through all the checks I want to for complete signoff.
For the moment, this change was only deployed to the Tucson test stand EFD. In coordination with Michael Reuter we'll deploy the new influxdb connector configuration to the other EFD instances when appropriate.
To verify if a given EFD instance is configured to use the private_sndStamp as the InfluxDB time, one can look at the Argo CD UI. Search for the influxdb-sink app (or influxdb-sink-staging app for this test) and check that the kafka-connect-manager container is started with the option --timestamp private_sndStamp
Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD.
There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.