Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13301

Configurable worker identity and chunk availability map in Qserv

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      This ticket proposes two improvements to be made in a scope of the Qserv worker services:

      1. replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's information schema with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services.
      2. adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module wpublish to support the proposed improvement.

      For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

      Make sure the unit tests were updated accordingly.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  gapon Igor Gaponenko
                  Reporter:
                  gapon Igor Gaponenko
                  Reviewers:
                  Andy Salnikov
                  Watchers:
                  Andy Salnikov, Fritz Mueller, Igor Gaponenko
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel