Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16224

Reduce excessive database traffic from the Replication Controller

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
    • Templates:
    • Story Points:
      0.5
    • Sprint:
      DB_F18_10
    • Team:
      Data Access and Database

      Description

      A development effort associated with this ticket is meant to address one of the side-effects introduced during the code review of DM-14262. The new version of the code (which is now formally correct) puts too much unnecessary load onto the MariaDB service when updating persistent states of the Replication-Qserv synchronization requests. This results in million rows of mostly unneeded data stored in the databases each time such synchronization is happening. Each such row contains names of a databases and chunk numbers which are sent from the Replication system to the Qserv workers. Another problem with the implementation is the rapid growth of the database space used by the Replication system (roughly 200 GB over 48 hours).

      In this ticket the detailed reporting of the database names and chunks numbers will be replaced with a simple counter of replicas sent to Qserv workers by each such request.

      The second improvement is to be made in the default configuration of the master Replication Controller. The frequency of checks made by the Cluster Health Monitoring and Replication threads is going to be significantly reduces.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  gapon Igor Gaponenko
                  Reporter:
                  gapon Igor Gaponenko
                  Reviewers:
                  Igor Gaponenko
                  Watchers:
                  Fritz Mueller, Igor Gaponenko, Vaikunth Thukral
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel