Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21391

Add the ability to select which field to use as the InfluxDB timestamp

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Attachments

        Issue Links

          Activity

          afausti Angelo Fausti created issue -
          afausti Angelo Fausti made changes -
          Field Original Value New Value
          Epic Link DM-18190 [ 241166 ]
          Hide
          afausti Angelo Fausti added a comment - - edited

          Add --timestamp option to kafka-connect-manager to select a specific field to be used as the InfluxDB timestamp.

          The default (generic) behavior is still to use the current system time.

          Show
          afausti Angelo Fausti added a comment - - edited Add --timestamp option to kafka-connect-manager to select a specific field to be used as the InfluxDB timestamp. The default (generic) behavior is still to use the current system time.
          afausti Angelo Fausti made changes -
          Story Points 2.8
          Team SQuaRE [ 10302 ]
          afausti Angelo Fausti made changes -
          Epic Link DM-18190 [ 241166 ] DM-20445 [ 342674 ]
          afausti Angelo Fausti made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          afausti Angelo Fausti added a comment -

          This can be done in KSQL, see https://docs.lenses.io/connectors/sink/influx.html#kcql-support

          the --timestamp option was added, and the default behavior preserved.

          Example:

          $ connect_manager create influxdb-sink -d efd -t 10 --dry-run mytopic1 --timestamp private_sndStamp
          {
              "connect.influx.db": "efd",
              "connect.influx.kcql": "INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP private_sndStamp",
              "connect.influx.url": "https://localhost:8086",
              "connect.influx.username": "-",
              "connector.class": "com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector",
              "task.max": 10,
              "topics": "mytopic1"
          }
          

          $ connect_manager create influxdb-sink -d efd -t 10 --dry-run mytopic1
          {
              "connect.influx.db": "efd",
              "connect.influx.kcql": "INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP sys_time()",
              "connect.influx.url": "https://localhost:8086",
              "connect.influx.username": "-",
              "connector.class": "com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector",
              "task.max": 10,
              "topics": "mytopic1"
          }
          

          Show
          afausti Angelo Fausti added a comment - This can be done in KSQL, see https://docs.lenses.io/connectors/sink/influx.html#kcql-support the --timestamp option was added, and the default behavior preserved. Example: $ connect_manager create influxdb-sink -d efd -t 10 --dry-run mytopic1 --timestamp private_sndStamp { "connect.influx.db": "efd", "connect.influx.kcql": "INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP private_sndStamp", "connect.influx.url": "https://localhost:8086", "connect.influx.username": "-", "connector.class": "com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector", "task.max": 10, "topics": "mytopic1" } $ connect_manager create influxdb-sink -d efd -t 10 --dry-run mytopic1 { "connect.influx.db": "efd", "connect.influx.kcql": "INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP sys_time()", "connect.influx.url": "https://localhost:8086", "connect.influx.username": "-", "connector.class": "com.datamountaineer.streamreactor.connect.influx.InfluxSinkConnector", "task.max": 10, "topics": "mytopic1" }
          Hide
          afausti Angelo Fausti added a comment -

          java.lang.IllegalArgumentException: Invalid value for field:private_sndStamp.Value '1.3261721783120623E9' is not a valid field for the timestamp
          	at com.datamountaineer.streamreactor.connect.influx.writers.TimestampValueCoerce$.apply(TimestampValueCoerce.scala:35)
          	at com.datamountaineer.streamreactor.connect.influx.writers.InfluxBatchPointsBuilder$$anonfun$buildPointFromStruct$2$$anonfun$19.apply(InfluxBatchPointsBuilder.scala:237)
          

          Show
          afausti Angelo Fausti added a comment - java.lang.IllegalArgumentException: Invalid value for field:private_sndStamp.Value '1.3261721783120623E9' is not a valid field for the timestamp at com.datamountaineer.streamreactor.connect.influx.writers.TimestampValueCoerce$.apply(TimestampValueCoerce.scala:35) at com.datamountaineer.streamreactor.connect.influx.writers.InfluxBatchPointsBuilder$$anonfun$buildPointFromStruct$2$$anonfun$19.apply(InfluxBatchPointsBuilder.scala:237)
          afausti Angelo Fausti made changes -
          Summary Add the ability to select which topic field to use as InfluxDB timestamp Add the ability to select which field to use as the InfluxDB timestamp
          Hide
          afausti Angelo Fausti added a comment -

          It seems reasonable to add support for double types in the InfluxDB Sink connector.

          Show
          afausti Angelo Fausti added a comment - It seems reasonable to add support for double types in the InfluxDB Sink connector.
          Hide
          afausti Angelo Fausti added a comment - - edited

          See PR https://github.com/Landoop/stream-reactor/pull/626

          With this change in the InfluxDB Sink connector we can understand SAL timestamps that are stored as double, then we coerce to Long with milliseconds precision to use with InfluxDB.

          Show
          afausti Angelo Fausti added a comment - - edited See PR https://github.com/Landoop/stream-reactor/pull/626 With this change in the InfluxDB Sink connector we can understand SAL timestamps that are stored as double , then we coerce to Long with milliseconds precision to use with InfluxDB.
          afausti Angelo Fausti made changes -
          Link This issue relates to DM-21419 [ DM-21419 ]
          Hide
          afausti Angelo Fausti added a comment -

          The change was included in the 1.2.3 release of the connector https://github.com/lensesio/stream-reactor/releases.

          Show
          afausti Angelo Fausti added a comment - The change was included in the 1.2.3 release of the connector https://github.com/lensesio/stream-reactor/releases .
          Hide
          afausti Angelo Fausti added a comment -

          The --timestamp option was successfully tested with version 1.2.3 of the InfluxDB Sink connector during review of DM-21419

          Show
          afausti Angelo Fausti added a comment - The --timestamp option was successfully tested with version 1.2.3 of the InfluxDB Sink connector during review of  DM-21419
          afausti Angelo Fausti made changes -
          Resolution Done [ 10000 ]
          Status In Progress [ 3 ] Done [ 10002 ]
          afausti Angelo Fausti made changes -
          Epic Link DM-20445 [ 342674 ] DM-21713 [ 424613 ]
          afausti Angelo Fausti made changes -
          Link This issue relates to DM-23496 [ DM-23496 ]

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel