Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16224

Reduce excessive database traffic from the Replication Controller

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
    • Templates:
    • Story Points:
      0.5
    • Sprint:
      DB_F18_10
    • Team:
      Data Access and Database

      Description

      A development effort associated with this ticket is meant to address one of the side-effects introduced during the code review of DM-14262. The new version of the code (which is now formally correct) puts too much unnecessary load onto the MariaDB service when updating persistent states of the Replication-Qserv synchronization requests. This results in million rows of mostly unneeded data stored in the databases each time such synchronization is happening. Each such row contains names of a databases and chunk numbers which are sent from the Replication system to the Qserv workers. Another problem with the implementation is the rapid growth of the database space used by the Replication system (roughly 200 GB over 48 hours).

      In this ticket the detailed reporting of the database names and chunks numbers will be replaced with a simple counter of replicas sent to Qserv workers by each such request.

      The second improvement is to be made in the default configuration of the master Replication Controller. The frequency of checks made by the Cluster Health Monitoring and Replication threads is going to be significantly reduces.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                Igor Gaponenko
                Watchers:
                Fritz Mueller, Igor Gaponenko, Vaikunth Thukral
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel