Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27606

Dedicated thread pool for the REST server of the Master Replication Controller

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Objectives

      In the current implementation of the Master Replication Controller, the built-in REST server (based on qhttp) shares the same pool of the BOOST ASIO threads with other Replication/Ingest system's activities. This results in delayed responses for requests made to the REST server.

      Hence the task in the context of this ticket is to set up a dedicated thread pool for the REST server.

      The second task is to investigate the most optional combination of the Controllers configuration parameters given the number of the hardware threads (CPU cores) available on a master machine:

      • Size of the Controller's thread pool
      • Size of the REST server's thread pool (the one to be added with this ticket)
      • Size of a pool of MySQL connectors to the Replication System's MySQL server

      The idea here is to ensure all activities that require accessing the persistent state were not blocking and were not causing extreme utilization of the ost machine's resources.

        Attachments

          Activity

          Hide
          gapon Igor Gaponenko added a comment -

          Optimized configuration of the Master Controller

          The tests were conducted in the "large" Qserv cluster at NCSA.
          The current values of the relevant parameters were:

          SELECT * FROM qservReplica.config;
          +------------+------------------------------------+--------------------+
          | category   | param                              | value              |
          +------------+------------------------------------+--------------------+
          | controller | http_server_threads                | 16                 |
          | controller | num_threads                        | 128                |
          | database   | services_pool_size                 | 128                |
          

          This setup assumed that for each BOOST ASIO thread of the Controller there is one available database connector in the connection pool. This allows up to 128 simultaneous queries to be made at a time. At the same time, parameter {(controller/http_server_threads}} while being present was not used by the implementation. Hence having database/services_pool_size=128 be equal to controller/num_threads=128 made a sense.
          In the new implementation (this ticket) there are controller/http_server_threads=16 added. And each of these extra threads may require connecting to the MySQL service to execute various queries. This problem has been solved by making the following change to the configuraton to ensure database/services_pool_size = controller/num_threads + controller/http_server_threads:

          UPDATE qservReplica.config SET value='144' WHERE category='database' AND param='services_pool_size';
          SELECT * FROM qservReplica.config;
          +------------+------------------------------------+--------------------+
          | category   | param                              | value              |
          +------------+------------------------------------+--------------------+
          | controller | http_server_threads                | 16                 |
          | controller | num_threads                        | 128                |
          | database   | services_pool_size                 | 144                |
          

          Show
          gapon Igor Gaponenko added a comment - Optimized configuration of the Master Controller The tests were conducted in the "large" Qserv cluster at NCSA. The current values of the relevant parameters were: SELECT * FROM qservReplica.config; + ------------+------------------------------------+--------------------+ | category | param | value | + ------------+------------------------------------+--------------------+ | controller | http_server_threads | 16 | | controller | num_threads | 128 | | database | services_pool_size | 128 | This setup assumed that for each BOOST ASIO thread of the Controller there is one available database connector in the connection pool. This allows up to 128 simultaneous queries to be made at a time. At the same time, parameter {(controller/http_server_threads}} while being present was not used by the implementation. Hence having database/services_pool_size=128 be equal to controller/num_threads=128 made a sense. In the new implementation (this ticket) there are controller/http_server_threads=16 added. And each of these extra threads may require connecting to the MySQL service to execute various queries. This problem has been solved by making the following change to the configuraton to ensure database/services_pool_size = controller/num_threads + controller/http_server_threads : UPDATE qservReplica.config SET value= '144' WHERE category= 'database' AND param= 'services_pool_size' ; SELECT * FROM qservReplica.config; + ------------+------------------------------------+--------------------+ | category | param | value | + ------------+------------------------------------+--------------------+ | controller | http_server_threads | 16 | | controller | num_threads | 128 | | database | services_pool_size | 144 |
          Show
          gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/581
          Hide
          npease Nate Pease added a comment -

          On Slack Igor asked me to review this, I assume it should be in review.

          Show
          npease Nate Pease added a comment - On Slack Igor asked me to review this, I assume it should be in review.
          Hide
          npease Nate Pease added a comment -

          All the changes look good.

          Show
          npease Nate Pease added a comment - All the changes look good.

            People

            Assignee:
            gapon Igor Gaponenko
            Reporter:
            gapon Igor Gaponenko
            Reviewers:
            Nate Pease
            Watchers:
            Fabrice Jammes, Fritz Mueller, Igor Gaponenko, Nate Pease
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: