Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30642

Increase the number of partitions for high throughput EFD topics

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: efd
    • Labels:
      None

      Description

      As discussed in DM-30357, we have to analyze the size vs. frequency of the EFD topics to decide which topics need an increased number of partitions to increase the data throughput and reduce latency for these topics.

      A few examples of such topics from the recent MTM1M3 tests are:

      lsst.sal.MTM1M3.logevent_appliedStaticForces
      		 
      lsst.sal.MTM1M3.pidData
      		 
      lsst.sal.MTM1M3.logevent_appliedElevationForces
      		 
      lsst.sal.MTM1M3.logevent_appliedForces
      		 
      lsst.sal.MTM1M3.logevent_appliedThermalForces
      		 
      lsst.sal.MTM1M3.logevent_appliedBalanceForces
      		 
      lsst.sal.MTM1M3.logevent_appliedAzimuthForces
      		 
      lsst.sal.MTM1M3.logevent_forceActuatorWarning

      SAL Kafka creates the topics in Kafka with a single partition, but kafka-connect-manager seems the right place to add the topic partition configuration, perhaps as a separate job.

      All connectors (replicator, influxdb, jdbc-sink and s3-sink) will benefit from this.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31474 Implement mechanism to customize number of partitions per topic in the salkafka producer when splitting topics

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30357 EFD replication service maintenance

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            afausti Angelo Fausti added a comment - - edited

            We decided to apply this configuration when the topic is created. kafka-connect-manager could still run a job to enforce that.

            Tiago opened DM-31474 to add support to partition configuration per topic on ts_salkafka. Once we have that implemented, we can configure multiple InfluxDB connectors and increase the number of connector tasks to distribute the load.

            Show
            afausti Angelo Fausti added a comment - - edited We decided to apply this configuration when the topic is created. kafka-connect-manager could still run a job to enforce that. Tiago opened DM-31474 to add support to partition configuration per topic on ts_salkafka . Once we have that implemented, we can configure multiple InfluxDB connectors and increase the number of connector tasks to distribute the load.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            The EFD data throughput and storage sizing spreadsheet has the topic throughput information.

            Show
            afausti Angelo Fausti added a comment - The EFD data throughput and storage sizing spreadsheet has the topic throughput information.
            Hide
            Permalink
            afausti Angelo Fausti added a comment -

            Still relevant but must be planned for another epic. Moving to my backlog for now.

            Show
            afausti Angelo Fausti added a comment - Still relevant but must be planned for another epic. Moving to my backlog for now.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.