# Use private_sndStamp as the InfluxDB timestamp

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
4.2
• Team:
SQuaRE
• Urgent?:
No

## Description

The definition of which timestamp is used as the InfluxDB time is done in the connector configuration. The default behavior is to use sys_time(). We want to replace that by private_sndStamp obtained from the SAL topics.

In DM-21391, we implemented the --timestamp option in the Kafka connect manager to configure the connector:

https://kafka-connect-manager.lsst.io/cli-reference.html#cmdoption-connect-manager-create-influxdb-sink-timestamp

and in this PR we changed the connector to support to Unix timestamps as double
https://github.com/lensesio/stream-reactor/pull/626

That was released in the version 1.2.3 of the connector:

https://github.com/lensesio/stream-reactor/releases/tag/1.2.3

but, unfortunately, that version introduced a regression and cannot be used (DM-21895).

In this ticket, we'll build a special version of the connector previous to 1.23 that supports Unix timestamps as double.

We'll also need to update the kafka connect manager Helm chart so that we can configure which timestamp to use.

Then, we'll deploy the new connector in the Tucson teststand. For testing the connector with the private_sndStamp we'll send data to a second database.

## Activity

Angelo Fausti created issue -
Field Original Value New Value
Epic Link DM-22721 [ 427980 ]
 Link This issue relates to DM-21419 [ DM-21419 ]
 Link This issue relates to DM-21391 [ DM-21391 ]
 Link This issue relates to DM-21895 [ DM-21895 ]
 Status To Do [ 10001 ] In Progress [ 3 ]
Hide
Angelo Fausti added a comment -

Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD.

There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.

Show
Angelo Fausti added a comment - Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD. There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.
Hide
Angelo Fausti added a comment - - edited

Waiting on data to flow to the Tucson Teststand EFD again to test this implementation.

Show
Angelo Fausti added a comment - - edited Waiting on data to flow to the Tucson Teststand EFD again to test this implementation.
 Reviewers Michael Reuter [ mareuter ] Status In Progress [ 3 ] In Review [ 10004 ]
 Attachment Screen Shot 2020-02-19 at 2.44.11 PM.png [ 42617 ]
Hide
Angelo Fausti added a comment - - edited

Michael Reuter a quick look using a query like this:

 SELECT "private_sndStamp" FROM "efd-test-timestamp"."autogen"."lsst.sal.DSM.logevent_heartbeat" 

shows that the InfluxDB time corresponds now to the private_sndStamp field. Note that the time precision is milliseconds, which is how it was implemented in the connector https://github.com/lensesio/stream-reactor/pull/626

Show
Angelo Fausti added a comment - - edited Michael Reuter a quick look using a query like this: SELECT "private_sndStamp" FROM "efd-test-timestamp"."autogen"."lsst.sal.DSM.logevent_heartbeat" shows that the InfluxDB time corresponds now to the private_sndStamp field. Note that the time precision is milliseconds, which is how it was implemented in the connector https://github.com/lensesio/stream-reactor/pull/626
Hide
Michael Reuter added a comment -

I'm giving tacit approval to this. Preliminary checks look good, but I haven't gone through all the checks I want to for complete signoff.

Show
Michael Reuter added a comment - I'm giving tacit approval to this. Preliminary checks look good, but I haven't gone through all the checks I want to for complete signoff.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
Hide
Angelo Fausti added a comment -

For the moment, this change was only deployed to the Tucson test stand EFD. In coordination with Michael Reuter we'll deploy the new influxdb connector configuration to the other EFD instances when appropriate.

To verify if a given EFD instance is configured to use the private_sndStamp as the InfluxDB time, one can look at the Argo CD UI. Search for the influxdb-sink app (or influxdb-sink-staging app for this test) and check that the kafka-connect-manager container is started with the option --timestamp private_sndStamp

Show
Angelo Fausti added a comment - For the moment, this change was only deployed to the Tucson test stand EFD. In coordination with Michael Reuter we'll deploy the new influxdb connector configuration to the other EFD instances when appropriate. To verify if a given EFD instance is configured to use the private_sndStamp as the InfluxDB time, one can look at the Argo CD UI. Search for the influxdb-sink app (or influxdb-sink-staging app for this test) and check that the kafka-connect-manager container is started with the option --timestamp private_sndStamp
 Story Points 2.8 4.2
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Link This issue relates to TSS-3463 [ TSS-3463 ]
 Link This issue relates to DM-24000 [ DM-24000 ]

## People

• Assignee:
Angelo Fausti
Reporter:
Angelo Fausti
Reviewers:
Michael Reuter
Watchers:
Angelo Fausti, Michael Reuter, Simon Krughoff