Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21334

Handling NaN's in the Landoop InfluxDB Sink connector

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      During DM-21300 we saw a topic lsst.sal.ATDome.logevent_azimuthCommandedState that was producing NaN values. The Java/Scala code in the InfluxDB Sink Connector serialized those to \ufffd and InfluxDB is not able to write them.

      Caused by: org.influxdb.InfluxDBException$UnableToParseException: partial write: unable to parse 'lsst.sal.ATDome.logevent_azimuthCommandedState ATDomeID=1i,azimuth=�,commandedState=1i,priority=0i,private_host=1994757124i,private_kafkaStamp=1567888128.2887785,private_origin=32i,private_rcvStamp=1567888127.9708674,private_revCode="5544b90a",private_seqNum=1i,private_sndStamp=1567887306.2843742 1567888091465571857': invalid boolean dropped=0
      

      There was a discussion on where this should be fixed and how to store missing values in InfluxDB.

      After some investigation, the conclusion was to drop the field with the missing value right before writing to InfluxDB. The rationale for that is explained in this notebook:

      https://github.com/lsst-sqre/influx-demo/blob/u/afausti/missings/On_storing_missing_values_in_InfluxDB.ipynb

      There's a PR on influxdb-java that might fix this

      https://github.com/influxdata/influxdb-java/pull/616

      In this ticket we are going to build the Landoop InfluxDB Sink with the latest influxdb-java and try to reproduce the problem.

        Attachments

          Issue Links

            Activity

            There are no comments yet on this issue.

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti, Bo Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel