Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25901

Consolidating 32 replication tools into fewer command-line applications

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Goals

      There are 32 command line tools in the current implementation of the replica module of Qserv. The source code of the tools is at:

      core/modules/replica/tools/qserv-replica-calc-cs.cc
      core/modules/replica/tools/qserv-replica-config.cc
      core/modules/replica/tools/qserv-replica-controller-cmd.cc
      core/modules/replica/tools/qserv-replica-database-test.cc
      core/modules/replica/tools/qserv-replica-file-export.cc
      core/modules/replica/tools/qserv-replica-file-ingest.cc
      core/modules/replica/tools/qserv-replica-file-read.cc
      core/modules/replica/tools/qserv-replica-file-server.cc
      core/modules/replica/tools/qserv-replica-httpreader-test.cc
      core/modules/replica/tools/qserv-replica-job-abort.cc
      core/modules/replica/tools/qserv-replica-job-admin.cc
      core/modules/replica/tools/qserv-replica-job-chunks.cc
      core/modules/replica/tools/qserv-replica-job-delete-worker.cc
      core/modules/replica/tools/qserv-replica-job-fixup.cc
      core/modules/replica/tools/qserv-replica-job-health.cc
      core/modules/replica/tools/qserv-replica-job-index.cc
      core/modules/replica/tools/qserv-replica-job-move.cc
      core/modules/replica/tools/qserv-replica-job-purge.cc
      core/modules/replica/tools/qserv-replica-job-rebalance.cc
      core/modules/replica/tools/qserv-replica-job-replicate.cc
      core/modules/replica/tools/qserv-replica-job-sql.cc
      core/modules/replica/tools/qserv-replica-job-sync.cc
      core/modules/replica/tools/qserv-replica-job-verify.cc
      core/modules/replica/tools/qserv-replica-master-http.cc
      core/modules/replica/tools/qserv-replica-messenger-test.cc
      core/modules/replica/tools/qserv-replica-mysql-test.cc
      core/modules/replica/tools/qserv-replica-qhttp-test.cc
      core/modules/replica/tools/qserv-replica-transactions-test.cc
      core/modules/replica/tools/qserv-replica-worker-all.cc
      core/modules/replica/tools/qserv-replica-worker-notify.cc
      core/modules/replica/tools/qserv-replica-worker-ping.cc
      core/modules/replica/tools/qserv-replica-worker.cc
      

      This results in a noticeable delay in building the package in the CI and other environments.

      A goal of this effort is to refactor (where it makes sense) the applications by consolidating them into a fewer number of command-line tools. Note that this will result in changes in the command line syntax of the applications. In particular, a proposed (new) syntax would be:

      qserv-replica-job <APPLICATION> [job-specific parameters, options, and flags of the original tool]
      qserv-replica-file <APPLICATION> [file-operation-specific parameters, options, and flags of the original tool]
      qserv-replica-test <APPLICATION> [test-specific parameters, options, and flags of the original tool]
      

      API version number

      Change the version number from 2 to 3 as reported by the REST service of the Master Replication Controller:

      /metadata/version
      

      Documentation

      The documentation on the Replication and Ingest systems needs to be updated accordingly. These are the links to the documents to be updated:

        Attachments

          Activity

          Hide
          fritzm Fritz Mueller added a comment -

          LGTM – thanks!

          Show
          fritzm Fritz Mueller added a comment - LGTM – thanks!

            People

            Assignee:
            gapon Igor Gaponenko
            Reporter:
            gapon Igor Gaponenko
            Reviewers:
            Fritz Mueller
            Watchers:
            Fritz Mueller, Igor Gaponenko, Nate Pease
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.