Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30642

Increase the number of partitions for high throughput EFD topics

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: efd
    • Labels:
      None

      Description

      As discussed in DM-30357, we have to analyze the size vs. frequency of the EFD topics to decide which topics need an increased number of partitions to increase the data throughput and reduce latency for these topics.

      A few examples of such topics from the recent MTM1M3 tests are:

      lsst.sal.MTM1M3.logevent_appliedStaticForces
      lsst.sal.MTM1M3.pidData
      lsst.sal.MTM1M3.logevent_appliedElevationForces
      lsst.sal.MTM1M3.logevent_appliedForces
      lsst.sal.MTM1M3.logevent_appliedThermalForces
      lsst.sal.MTM1M3.logevent_appliedBalanceForces
      lsst.sal.MTM1M3.logevent_appliedAzimuthForces
      lsst.sal.MTM1M3.logevent_forceActuatorWarning
      

      SAL Kafka creates the topics in Kafka with a single partition, but kafka-connect-manager seems the right place to add the topic partition configuration, perhaps as a separate job.

      All connectors (replicator, influxdb, jdbc-sink and s3-sink) will benefit from this.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            We decided to apply this configuration when the topic is created. kafka-connect-manager could still run a job to enforce that.

            Tiago opened DM-31474 to add support to partition configuration per topic on ts_salkafka. Once we have that implemented, we can configure multiple InfluxDB connectors and increase the number of connector tasks to distribute the load.

            Show
            afausti Angelo Fausti added a comment - - edited We decided to apply this configuration when the topic is created. kafka-connect-manager could still run a job to enforce that. Tiago opened DM-31474 to add support to partition configuration per topic on ts_salkafka . Once we have that implemented, we can configure multiple InfluxDB connectors and increase the number of connector tasks to distribute the load.
            Hide
            afausti Angelo Fausti added a comment -

            The EFD data throughput and storage sizing spreadsheet has the topic throughput information.

            Show
            afausti Angelo Fausti added a comment - The EFD data throughput and storage sizing spreadsheet has the topic throughput information.
            Hide
            afausti Angelo Fausti added a comment -

            Still relevant but must be planned for another epic. Moving to my backlog for now.

            Show
            afausti Angelo Fausti added a comment - Still relevant but must be planned for another epic. Moving to my backlog for now.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.