Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23496

Use private_sndStamp as the InfluxDB timestamp

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The definition of which timestamp is used as the InfluxDB time is done in the connector configuration. The default behavior is to use sys_time(). We want to replace that by private_sndStamp obtained from the SAL topics.

      In DM-21391, we implemented the --timestamp option in the Kafka connect manager to configure the connector:

      https://kafka-connect-manager.lsst.io/cli-reference.html#cmdoption-connect-manager-create-influxdb-sink-timestamp

      and in this PR we changed the connector to support to Unix timestamps as double
      https://github.com/lensesio/stream-reactor/pull/626

      That was released in the version 1.2.3 of the connector:

      https://github.com/lensesio/stream-reactor/releases/tag/1.2.3

      but, unfortunately, that version introduced a regression and cannot be used (DM-21895).

      In this ticket, we'll build a special version of the connector previous to 1.23 that supports Unix timestamps as double.

      We'll also need to update the kafka connect manager Helm chart so that we can configure which timestamp to use.

      Then, we'll deploy the new connector in the Tucson teststand. For testing the connector with the private_sndStamp we'll send data to a second database.

        Attachments

          Issue Links

            Activity

            afausti Angelo Fausti created issue -
            afausti Angelo Fausti made changes -
            Field Original Value New Value
            Epic Link DM-22721 [ 427980 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21419 [ DM-21419 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21391 [ DM-21391 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-21895 [ DM-21895 ]
            afausti Angelo Fausti made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            afausti Angelo Fausti made changes -
            Reviewers Michael Reuter [ mareuter ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            afausti Angelo Fausti made changes -
            mareuter Michael Reuter made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            afausti Angelo Fausti made changes -
            Story Points 2.8 4.2
            afausti Angelo Fausti made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to TSS-3463 [ TSS-3463 ]
            afausti Angelo Fausti made changes -
            Link This issue relates to DM-24000 [ DM-24000 ]

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Reviewers:
                Michael Reuter
                Watchers:
                Angelo Fausti, Michael Reuter, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel