Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13302

Adding support for worker-specific resources in Qserv

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Extend the present implementation of Qserv worker plugin (xrootd SSI) to recognize (and route to the corresponding handler) requests to the worker-specific resources. The proposed extension is planned to be used for the following tasks:

      • integrating Qserv with the replication system which requires an interface for interacting with the worker services to notify the ones about changes in chunk availability on the corresponding nodes
      • allowing to pull worker-specific stats and counters into the custom service monitoring applications

        Attachments

          Issue Links

            Activity

            Hide
            jgates John Gates added a comment -

            Looks good. Mostly style comments. I dislike blank lines in code, indentation, brackets, and a blank line  in some places is enough, but that could just be me

            Show
            jgates John Gates added a comment - Looks good. Mostly style comments. I dislike blank lines in code, indentation, brackets, and a blank line  in some places is enough, but that could just be me
            Hide
            gapon Igor Gaponenko added a comment - - edited

            This is a follow up for this ticket. I forgot to mention how to test if the worker-specific resources are known to XRootD/cmsd. This has two simple steps:

            1. connect to a worker's database and get its unique identifier from the database by

              SELECT Id FROM qservw_worker.Id

              This may yield something like:

              4175eabd-0793-11e8-a29e-fa163e50be0b

            2. log onto a master node of a Qserv cluster and use the xrdfs utility (it's in any recent version of the LSST Stack) and check this resource by:

              echo 'locate /worker/4175eabd-0793-11e8-a29e-fa163e50be0b' | xrdfs 127.0.0.1:1094
              [127.0.0.1:1094] / > locate /worker/4175eabd-0793-11e8-a29e-fa163e50be0b
              [::172.16.1.76]:1094 Server ReadWrite 
              [127.0.0.1:1094] / > Goodbye.
              

            This recipe has been tested on both single-node integration test setup, as well as a distributed Docker-based Qserv cluster.

            NOTE: the resource locator command will NOT actually engage with the worker-side SSI plugin. It will just ask the worker-side cmsd if this resource is known to the worker. A dedicated test for a protocol exists, however it's not in this branch. It will be put into branch DM-13303.

            Show
            gapon Igor Gaponenko added a comment - - edited This is a follow up for this ticket. I forgot to mention how to test if the worker-specific resources are known to XRootD/cmsd. This has two simple steps: connect to a worker's database and get its unique identifier from the database by SELECT Id FROM qservw_worker.Id This may yield something like: 4175eabd-0793-11e8-a29e-fa163e50be0b log onto a master node of a Qserv cluster and use the xrdfs utility (it's in any recent version of the LSST Stack) and check this resource by: echo 'locate /worker/4175eabd-0793-11e8-a29e-fa163e50be0b' | xrdfs 127.0.0.1:1094 [127.0.0.1:1094] / > locate /worker/4175eabd-0793-11e8-a29e-fa163e50be0b [::172.16.1.76]:1094 Server ReadWrite [127.0.0.1:1094] / > Goodbye. This recipe has been tested on both single-node integration test setup, as well as a distributed Docker-based Qserv cluster. NOTE : the resource locator command will NOT actually engage with the worker-side SSI plugin. It will just ask the worker-side cmsd if this resource is known to the worker. A dedicated test for a protocol exists, however it's not in this branch. It will be put into branch DM-13303 .

              People

              • Assignee:
                gapon Igor Gaponenko
                Reporter:
                gapon Igor Gaponenko
                Reviewers:
                John Gates
                Watchers:
                Fritz Mueller, Igor Gaponenko, John Gates
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: