Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13301

Configurable worker identity and chunk availability map in Qserv

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      This ticket proposes two improvements to be made in a scope of the Qserv worker services:

      1. replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's information schema with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services.
      2. adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module wpublish to support the proposed improvement.

      For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

      Make sure the unit tests were updated accordingly.

        Attachments

          Issue Links

            Activity

            gapon Igor Gaponenko created issue -
            gapon Igor Gaponenko made changes -
            Field Original Value New Value
            Epic Link DM-10681 [ 32633 ]
            gapon Igor Gaponenko made changes -
            Link This issue blocks DM-10424 [ DM-10424 ]
            gapon Igor Gaponenko made changes -
            Description This ticket has two goals in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # add a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            This ticket has two goals in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            gapon Igor Gaponenko made changes -
            Description This ticket has two goals in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            This ticket set up two goals in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            gapon Igor Gaponenko made changes -
            Description This ticket set up two goals in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            gapon Igor Gaponenko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            gapon Igor Gaponenko made changes -
            Assignee Igor Gaponenko [ gapon ]
            gapon Igor Gaponenko made changes -
            Description This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of pulling available chunk numbers via MySQL _information schema_ with a configuration based one. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will allow a consistent view onto a stable collection of chunks. It will also lay a foundation for the replication system and Qserv worker services to effectively cooperate on a list of resources (chunks) which are available to the workers. The implementation of this taks will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing a list of tables witin the MySQL _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            gapon Igor Gaponenko made changes -
            Description This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing a list of tables witin the MySQL _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.
            This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing a list of tables witin the MySQL _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

            Make sure the unit tests were updated accordingly.
            gapon Igor Gaponenko made changes -
            Link This issue blocks DM-13302 [ DM-13302 ]
            gapon Igor Gaponenko made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            gapon Igor Gaponenko made changes -
            Description This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing a list of tables witin the MySQL _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

            Make sure the unit tests were updated accordingly.
            This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

            Make sure the unit tests were updated accordingly.
            gapon Igor Gaponenko made changes -
            Description This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services. The implementation of this task will require making proper modifications to the Qserv's module *wpublish*
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

            Make sure the unit tests were updated accordingly.
            This ticket proposes two improvements to be made in a scope of the Qserv worker services:
            # replacing the current mechanism of deriving a list of available chunk numbers by parsing table names within the MySQL's _information schema_ with a new one based on an explicit persistent configuration stored within a database. The present mechanism has one major limitation preventing from integrating Qserv with the dynamic chunk replication system - it runs only one time when the worker services are being started. In the proposed model a list of chunk numbers (along with the corresponding table and database names) will be stored in a dedicated table seen by the relevant worker services. This will also allow a consistent view onto a stable collection of chunks. It will also lay a foundation for implementing a safe and efficient coordination mechanism between the replication system and Qserv worker services.
            # adding a persistent support for the unique identity of the worker services. The idea here is to store some unique (in a scope of a particular Qserv cluster) name within databases used by the workers. These identifiers will be used as a foundation for setting up worker-specific resources and by the replication system. Extend the implementation and the API of the Qserv's module *wpublish* to support the proposed improvement.

            For both tasks make proper changes to the Qserv management, integration and data loading scripts to populate the newly added table with a valid set of chunks and the worker identity strings, so that these resources would be available to the corresponding worker services.

            Make sure the unit tests were updated accordingly.
            salnikov Andy Salnikov made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            gapon Igor Gaponenko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            gapon Igor Gaponenko made changes -
            Link This issue is triggering DM-13526 [ DM-13526 ]

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                Andy Salnikov
                Watchers:
                Andy Salnikov, Fritz Mueller, Igor Gaponenko
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel