Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23496

Use private_sndStamp as the InfluxDB timestamp

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The definition of which timestamp is used as the InfluxDB time is done in the connector configuration. The default behavior is to use sys_time(). We want to replace that by private_sndStamp obtained from the SAL topics.

      In DM-21391, we implemented the --timestamp option in the Kafka connect manager to configure the connector:

      https://kafka-connect-manager.lsst.io/cli-reference.html#cmdoption-connect-manager-create-influxdb-sink-timestamp

      and in this PR we changed the connector to support to Unix timestamps as double
      https://github.com/lensesio/stream-reactor/pull/626

      That was released in the version 1.2.3 of the connector:

      https://github.com/lensesio/stream-reactor/releases/tag/1.2.3

      but, unfortunately, that version introduced a regression and cannot be used (DM-21895).

      In this ticket, we'll build a special version of the connector previous to 1.23 that supports Unix timestamps as double.

      We'll also need to update the kafka connect manager Helm chart so that we can configure which timestamp to use.

      Then, we'll deploy the new connector in the Tucson teststand. For testing the connector with the private_sndStamp we'll send data to a second database.

        Attachments

          Issue Links

            Activity

            afausti Angelo Fausti created issue -
            afausti Angelo Fausti made changes -
            Field Original Value New Value
            Epic Link DM-22721 [ 427980 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21419 [ DM-21419 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21391 [ DM-21391 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21895 [ DM-21895 ]
            afausti Angelo Fausti made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            afausti Angelo Fausti added a comment -

            Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD.

            There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.

            Show
            afausti Angelo Fausti added a comment - Michael Reuter we have everything in place to test the new timestamps on the Tucson test stand EFD. There are two influxdb-sink connectors running at the moment. The first connector writes to the efd database and is using the ingestion time as the InfluxDB timestamp as before. The second connector writes to the efd-test-timestamp database and is using the private_sndStamp as the InfluxDB timestamp. This way you can compare the results and evaluate the second connector before we switch over.
            Hide
            afausti Angelo Fausti added a comment - - edited

            Waiting on data to flow to the Tucson Teststand EFD again to test this implementation.

            Show
            afausti Angelo Fausti added a comment - - edited Waiting on data to flow to the Tucson Teststand EFD again to test this implementation.
            afausti Angelo Fausti made changes -
            Reviewers Michael Reuter [ mareuter ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            afausti Angelo Fausti made changes -
            Hide
            afausti Angelo Fausti added a comment - - edited

            Michael Reuter a quick look using a query like this:

            SELECT "private_sndStamp" FROM "efd-test-timestamp"."autogen"."lsst.sal.DSM.logevent_heartbeat"
            

            shows that the InfluxDB time corresponds now to the private_sndStamp field. Note that the time precision is milliseconds, which is how it was implemented in the connector https://github.com/lensesio/stream-reactor/pull/626

            Show
            afausti Angelo Fausti added a comment - - edited Michael Reuter a quick look using a query like this: SELECT "private_sndStamp" FROM "efd-test-timestamp"."autogen"."lsst.sal.DSM.logevent_heartbeat" shows that the InfluxDB time corresponds now to the private_sndStamp field. Note that the time precision is milliseconds, which is how it was implemented in the connector https://github.com/lensesio/stream-reactor/pull/626
            Hide
            mareuter Michael Reuter added a comment -

            I'm giving tacit approval to this. Preliminary checks look good, but I haven't gone through all the checks I want to for complete signoff.

            Show
            mareuter Michael Reuter added a comment - I'm giving tacit approval to this. Preliminary checks look good, but I haven't gone through all the checks I want to for complete signoff.
            mareuter Michael Reuter made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            afausti Angelo Fausti added a comment -

            For the moment, this change was only deployed to the Tucson test stand EFD. In coordination with Michael Reuter we'll deploy the new influxdb connector configuration to the other EFD instances when appropriate.

            To verify if a given EFD instance is configured to use the private_sndStamp as the InfluxDB time, one can look at the Argo CD UI. Search for the influxdb-sink app (or influxdb-sink-staging app for this test) and check that the kafka-connect-manager container is started with the option --timestamp private_sndStamp

            Show
            afausti Angelo Fausti added a comment - For the moment, this change was only deployed to the Tucson test stand EFD. In coordination with Michael Reuter we'll deploy the new influxdb connector configuration to the other EFD instances when appropriate. To verify if a given EFD instance is configured to use the private_sndStamp as the InfluxDB time, one can look at the Argo CD UI. Search for the influxdb-sink app (or influxdb-sink-staging app for this test) and check that the kafka-connect-manager container is started with the option --timestamp private_sndStamp
            afausti Angelo Fausti made changes -
            Story Points 2.8 4.2
            afausti Angelo Fausti made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to TSS-3463 [ TSS-3463 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-24000 [ DM-24000 ]

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Reviewers:
                Michael Reuter
                Watchers:
                Angelo Fausti, Michael Reuter, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel