Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13301

Configurable worker identity and chunk availability map in Qserv

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      This ticket proposes two improvements to be made in a scope of the Qserv worker services:

      1. replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's information schema with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services.
      2. adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module wpublish to support the proposed improvement.

      For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

      Make sure the unit tests were updated accordingly.

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment - - edited

            A summary of modifications and tests

            Database schema: admin/templates/configuration/tmp/configure/sql/qserv-worker.sql

            • reinforced table schema for qservw_worker.Dbs (added the PRIMARY key to the database name column to avoid duplicates)
            • added a new table qservw_worker.Chunks with a list of registered chunks. The table rows have logical dependency onto the database names in table qservw_worker.Dbs. Unfortunately this dependency can't be reinforced in the database schema due to a present protocol for setting up databases on the workers and loading workers with tables. The protocol requires to load chunks (and register them in the chunks table) via wmgr before publish databases. This would violate the FOREIGN key constraint. This may be reconsidered when the next generation catalog loader will be implemented.
            • added a new table qservw_worker.Id to store a persistent identifier of a worker. Setting the default identifier to a UUID generated at a time when the MySQL database is being initialized for the worker. The identifier can be changed later if needed.

            Web service: core/modules/wmgr/python/dbMgr.py

            • extended the Web service for loading chunks to register (if needed) new chunks in the above mentioned MySQL table qservw_worker.Chunks

            C++ class: core/modules/wpublish/ChunkInventory

            • reimplemented to fetch chunks from MySQL table qservw_worker.Chunks instead of parsing table names in databases. This should also speed up the startup time of the workers at a presence of thousands of chunks
            • added a reader of the unique identifiers of workers. Extended class contract with a method returning the identifier
            • minor code clean up to reimplement the ugly code

            Unit test: core/modules/wpublish/testChunkInventory.cc

            • reimplemented to support a new mechanism for loading chunks
            • added a test for worker identifiers

            Integration tests

            • verified that the standard integration tests work as expected after those modifications
            • verified that the chunk list gets properly populated with new chunks by the wmgr service when loading catalos into the worker databases
            • verified that the unique worker identity is set up after initializing the worker databases

            Schema migration support

            • implemented a module and a schema patch to automate migrating existing installations of database qservw_worker to the new schema. The tool is to be run by:

              qserv-smig.py -c mysql://root:CHANGEME@127.0.0.1:13306/qservw_worker wdb

            Show
            gapon Igor Gaponenko added a comment - - edited A summary of modifications and tests Database schema: admin/templates/configuration/tmp/configure/sql/qserv-worker.sql reinforced table schema for qservw_worker.Dbs (added the PRIMARY key to the database name column to avoid duplicates) added a new table qservw_worker.Chunks with a list of registered chunks. The table rows have logical dependency onto the database names in table qservw_worker.Dbs . Unfortunately this dependency can't be reinforced in the database schema due to a present protocol for setting up databases on the workers and loading workers with tables. The protocol requires to load chunks (and register them in the chunks table) via wmgr before publish databases. This would violate the FOREIGN key constraint. This may be reconsidered when the next generation catalog loader will be implemented. added a new table qservw_worker.Id to store a persistent identifier of a worker. Setting the default identifier to a UUID generated at a time when the MySQL database is being initialized for the worker. The identifier can be changed later if needed. Web service: core/modules/wmgr/python/dbMgr.py extended the Web service for loading chunks to register (if needed) new chunks in the above mentioned MySQL table qservw_worker.Chunks C++ class: core/modules/wpublish/ChunkInventory reimplemented to fetch chunks from MySQL table qservw_worker.Chunks instead of parsing table names in databases. This should also speed up the startup time of the workers at a presence of thousands of chunks added a reader of the unique identifiers of workers. Extended class contract with a method returning the identifier minor code clean up to reimplement the ugly code Unit test: core/modules/wpublish/testChunkInventory.cc reimplemented to support a new mechanism for loading chunks added a test for worker identifiers Integration tests verified that the standard integration tests work as expected after those modifications verified that the chunk list gets properly populated with new chunks by the wmgr service when loading catalos into the worker databases verified that the unique worker identity is set up after initializing the worker databases Schema migration support implemented a module and a schema patch to automate migrating existing installations of database qservw_worker to the new schema. The tool is to be run by: qserv-smig.py -c mysql://root:CHANGEME@127.0.0.1:13306/qservw_worker wdb
            Hide
            salnikov Andy Salnikov added a comment - - edited

            Sorry, forgot to mark it as reviewed.

            Show
            salnikov Andy Salnikov added a comment - - edited Sorry, forgot to mark it as reviewed.
            Hide
            gapon Igor Gaponenko added a comment -

            Andy Salnikov Thanks!

            And I've just realized I made a small mistake in this branch when adding support for schema migration. I didn't include a DDL for table qservw_worker.QMetadata into the initialization script:

            admin/templates/configuration/tmp/configure/sql/qserv-worker.sql
            

            This won't prevent the schema migration script from upgrading to the newer version of the database. However, it will confuse the script if run on a database which has already been initialized to the new schema. I don't think this is critical. I've already fixed that in the next ticket in this series DM-13302, which will be finished (hopefully) today.

            Show
            gapon Igor Gaponenko added a comment - Andy Salnikov Thanks! And I've just realized I made a small mistake in this branch when adding support for schema migration. I didn't include a DDL for table qservw_worker.QMetadata into the initialization script: admin/templates/configuration/tmp/configure/sql/qserv-worker.sql This won't prevent the schema migration script from upgrading to the newer version of the database. However, it will confuse the script if run on a database which has already been initialized to the new schema. I don't think this is critical. I've already fixed that in the next ticket in this series DM-13302 , which will be finished (hopefully) today.

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                Andy Salnikov
                Watchers:
                Andy Salnikov, Fritz Mueller, Igor Gaponenko
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: