Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17812

Worker database management in the Replication system


    • Type: Improvement
    • Status: In Progress
    • Priority: Undefined
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:


      Extend the Replication system to support operations on the Qserv worker databases via worker's MariaDB service. The operations shall be broadcasted either to all known workers or a subset of those. Build the programmatic API and the command-line tool for that.

      This feature is needed for implementing the new catalog ingest in Qserv.

      Operations to be supported:

      • databases:
        • retrieving a list of known databases
        • creating
        • deleting
        • enable in Qserv
        • disable in Qserv
      • tables:
        • retrieving a list of known tables in a scope of a database
        • retrieving a schema
        • creating
        • deleting
      • privileges:
        • retrieving existing privileges for user qsmaster
        • granting privileges to user qsmaster to issues SELECT statements against all tables of a database
        • revoking privileges

      Investigate pros and cons of the following options:

      • SQL: allowing a client to send arbitrary queries to be executed by MySQL user root.
      • functional: having a restricted mechanism which would support specific operations only.

      The SQL option has a big advantage - it's flexibility. Once implemented and deployed it won't require to make any further modifications to the Qserv worker. On the other hand it seems to be less secure compared with the functional approach.

      Investigate two protocols:

      • XRootD/SSI: extend Qserv worker with the corresponding commands.
      • BOOST ASIO: extend Replication workers to handle the requests

      And for the protocol options, the first one (XRootD/SSI) might seem to be easier, at least at the surface. However, it may have long-term risks in case if it will be phased out in Qserv. Besides, it will also require the Qserv workers to be up in order to implement the operations. In contrast with that a requirement for the BOOST ASIO option is to only have the worker's MariaDB server up, which will become a separate Docker container in the Kubernetes-based Qserv deployments. Another argument in favor of the second protocol option is that the Replication workers will need to be extended to talk to the worker's database servers anyway in order to implement the new catalog ingests. A known complication of the BOOST ASIO protocol is that it will require extending the Configuration (of the Replication system) to store connection parameters of the worker database services.

      Investigate options to secure the protocol:

      • using the same secret file (string) which is already used by the existing wmgr Web service might be a good option here
      • relying on the MariaDB password for the root account of the worker database services

      Preliminary choice:

      1. the functional over SQL
      2. and BOOST ASIO over XRootD/SSI
      3. using the MariaDB root account's password


          Container Issues

            Issue Links



                • Assignee:
                  gapon Igor Gaponenko
                  gapon Igor Gaponenko
                  Andy Salnikov
                  Andy Salnikov, Fritz Mueller, Igor Gaponenko
                • Votes:
                  0 Vote for this issue
                  3 Start watching this issue


                  • Created:

                    Summary Panel