For the first part of adapting the integration tests to a multi-node setup, all the requisite individual parts have to be exercised. I am documenting all the steps that were needed for a successful multi-node setup here. I used lsst-dbdev1 as the Czar, and lsst-dbdev3 as the Worker:
1) First step is to create and configure Czar (dbdev1) and Worker (dbdev3) setups correctly. This was done through documentation in DM-595 and automated by Fabrice Jammes for IN2P3 in DM-213 . The script/instructions were adapted to setup for the dbdev machines.
2) After czar and worker are setup, the worker node must be registered in CSS. This is done on the czar as follows:
qserv-admin.py
|
>> create node worker-dbdev3 host=lsst-dbdev3.ncsa.illinois.edu runDir=/usr/local/home/vaikunth/qserv-run mysqlConn=13306;
|
Any further nodes can be added here as well. Note that mysqlConn can be given a value like lo:13306 for an SSH connection (requiring your password), but without the leading lo: it connects directly. Related - Currently deleting or updating a node is not possible (DM-2411)
3) The next test was to load data on the workers from the czar using qserv-data-loader.py. In addition to the data loader, the integration tests need access to the worker databases for inserting, selecting etc. for various purposes. This is not an issue on a mono-node setup but required a change in permissions to be able to use the same functionality on the worker. Through various stages the final set of grants on the worker that are needed are (subject to change):
+----------------------------------------------------------------------------------------+
|
| Grants for qsmaster@lsst-dbdev%.ncsa.illinois.edu |
|
+----------------------------------------------------------------------------------------+
|
| GRANT USAGE ON *.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
| GRANT SELECT ON `mysql`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
| GRANT ALL PRIVILEGES ON `qservScratch`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
| GRANT ALL PRIVILEGES ON `Subchunks\_%`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
| GRANT ALL PRIVILEGES ON `qservTest%`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
| GRANT ALL PRIVILEGES ON `qservw_worker`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu'|
|
| GRANT ALL PRIVILEGES ON `q\_%`.* TO 'qsmaster'@'lsst-dbdev%.ncsa.illinois.edu' |
|
+----------------------------------------------------------------------------------------+
|
Parts of the integration test use the root account for mysql which is not ideal. This should be changed to user qsmaster for all purposes eventually (DM-2412), since granting all privileges to root on workers is not feasbile. In addition, we should think about the final set of grants that should be given on a worker and include those in the worker configuration procedure instead of manually doing it on each of them.
4) Data can be loaded to any registered nodes in CSS via the qserv-data-loader.py command by appending the worker node list:
qserv-data-loader.py --worker worker-dbdev3 --worker worker-dbdev4 <other parameters>
|
This was sucessfully tested to work on the command line but there was an error while trying to do it through the integration tests. This is fixed by Andy Salnikov in DM-2417 and data now loads correctly on the worker.
Based on discussion at hangout Nov 05, use existing single-node partitioner, then scp csv files produced by partitioner, and ssh <load> on each node.