Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14154

Setting up a Qserv and Replication cluster at IN2P3

    Details

      Description

      Set up a testing environment combining Qserv + the Replication system in the upper part of the IN2P3 cluster. This included the following nodes:

      • ccqserv125: Qserv czar and the Replication System's controller
      • ccqserv126 - ccqserv149: Qserv worker servers and the Replication System's worker agents

      Configure Qserv to use a separate set of ports and to use a separate data directories to avoid direct conflicts with the main stream (other) Qserv installation.

      Preload the cluster with 1/6th of the following catalogs which are (as of now) being deployed at the NCSA's PDAC Qserv Cluster:

      • sdss_stripe82_01
      • wise_00

      The data will be copied from the first 5 worker nodes of that (source) cluster:

      • lsst-qserv-db01 - lsst-qserv-db05

      The total amount of data to be transgferred from those 5 nodes is: 3.7 TB

      Extra actions to be taken in a context of this ticket will also include:

      • building and deploying a specially configured version of the Qserv Docker container (to allow separate ports)
      • packaging tools of the Replication systems into a separate container to be deployed at teh new cluster
      • develop simple set of the management tools for starting/stopping the Replication System's services

      Port number modifications in container image qserv/qserv:tickets_DM-10424:

      % cat admin/templates/installation/qserv-meta.conf
       
      # Port number for worker management service
      port = 25012
       
      # Port number for cmsd server (not used in mono setup)
      cmsd_manager_port = 22131
       
      # Port number for xrootd server
      xrootd_port = 21094
       
      # Port number for mysql-proxy, this is the primary interface for qserv clients
      port = 24040
       
      # Port number for mysql server
      port = 23306
      

      Data directories for the Qserv instance:

      /qserv/replication/
      /qserv/replication/data/
      /qserv/replication/log/
      /qserv/replication/tmp/
      

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  gapon Igor Gaponenko
                  Reporter:
                  gapon Igor Gaponenko
                  Watchers:
                  Fritz Mueller, Igor Gaponenko
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel