Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10424

Replication Framework (integration with Qserv)

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Templates:
    • Story Points:
      10
    • Sprint:
      DB_S18_02, DB_S18_03, DB_S18_04, DB_S18_05, DB_F18_06
    • Team:
      Data Access and Database

      Description

      This task has two goals:

      1. establishing a mechanism to notify Qserv on a completion of the replication operations after new replicas are created. This also includes a possible negotiation process when replicas need to be removed from a cluster
      2. serve as an anchor for the first code review of the project

       

      Here is a link to my presentation on the design and the architecture of the Replication system:

      http://www.slac.stanford.edu/~gapon/TALKS/2018_Apr_LSST_GroupMeeting/QservReplicationSystem.pdf

       

      Suggested areas for code review (assuming Qsev module replica or its subfolders sql/ and tools/):

      1.  Nate Pease Messaging network:
        • Messenger, MessengerConnectionWorkerServerWorkerServerConnection
        • tools/qserv-replica-messenger-test
      2. Andy Salnikov Configuration service:
        • Configuration, ConfigurationMySQL, ConfigurationFile
        • sql/replication.sql, sql/replication_config*.sql
      3. Andy Salnikov Database and persistent state support:
        • DatabaseMySQL, DatabaseServicesDatabaseServicesMySQL
        • sql/replication.sql
        • tools/qserv-replica-mysql-test - a simple test for the database API
        • tools/qserv-replica-test-sql - experimentation (And a demo) with the C++11's variadic templates for query construction
      4. Nate Pease Replication framework:
        • controller-side: Controller, Request, RequestMessenger (intermediate base class for specific requests based on the Messenger), FindRequest, FindAllRequest, ReplicationRequest, DeleteRequest, StatusRequest (a family of classes), StopRequest (a family of classes), ServiceManagementRequest (a family of classes), JobJobController
        • worker-side: WorkerProcessor, WorkerRequest, WorkerFindRequestWorkerFindAllRequest,WorkerReplicationRequest, WorkerDeleteRequest
        • shared: ServiceProvider, Configuration, DatabaseServices, QservMgtServices, ChunkLocker
        • tools/qserv-replica-worker - the worker-side server with a built-in file service
        • tools/qserv-replica-controller-cmd - tool for invoking individual requests
        • tools/qserv-replica-controller-admin - tool for inspecting/managing worker services
        • tools/qserv-replica-embedded-test - all-in-one solution encapsulating both Controller and (multiple) worker-side services
      5. Fritz Mueller High-level algorithms:
        • Job - base class for a family of requests
        • JobController - job controller
        • specific jobs: FindJob, FindAllJob, ReplicateJob, PurgeJob, RebalanceJob, CreateReplicaJob, DeleteReplicaJob, MoveReplicaJob, PurgeJob, FixUpJob, VerifyJob
        • tools/qserv-replica-master - a fixed-logic replication master executing a sequence of jobs in an infinite loop
        • tools/qserv-replica-jobctrl-test - a simple test of the JobController
        • tools/qserv-replica-job-* - a collection of single-job controllers
      6. John Gates File transfer services, operations with files:
        • FileServer, FileServerConnection, FileClient, FileUtils
        • tools/qserv-replica-calc-cs - a test for the iterative computation of check-sums (see FileUtils)
        • tools/qserv-replica-file-server - stand alone file service
        • tools/qserv-replica-file-read - an example application using the file service to read a remote file
      7. John Gates Integration with Qserv (over XRootD/SSI)
        • QservMgtServices - provides services for Jobs
        • QservMgtRequest - base class for a family of Controller requests
        • specific requests: AddReplicaQservMgtRequest, RemoveReplicaQservMgtRequestGetReplicasQservMgtRequest, SetReplicasQservMgtRequest
        • relevant Job classes: QservGetReplicasJob
        • tools/qserv-replica-worker-notify - tool for sending commands to Qserv workers
        • tools/qserv-replica-job-sync - single job controller application for getting Qserv collection of chunks in synch with the actual disposition of data

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  gapon Igor Gaponenko
                  Reporter:
                  fritzm Fritz Mueller
                  Reviewers:
                  Andy Salnikov, Fritz Mueller, John Gates, Nate Pease
                  Watchers:
                  Andy Salnikov, Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel