Summary of the results for this round of tests.
Setup:
- 6 server nodes, 32 cores, 64GB RAM each
- 4x375GiB local SSD storage on each node
- replication factor 3, QUORUM consistency for reads and writes
- 6 client machines, 32 cores each
- 75k visits generated, storage errors after that
Select time as a function of visit number:
Store time:
Comparing with 3-node cluster for reading performance, numbers are for per-CCD real time per query.
|
3 nodes |
6 nodes |
obj_select_real at 50k visits, sec |
0.52 |
0.32 |
src_select_real at 50k visits, sec |
2.8 |
2.1 |
fsrc_select_real at 50k visits, sec |
1.4 |
0.7 |
There is some improvement in numbers but it is not an expected 2x times improvement. I think that is because there is a significant overhead on client side for converting query results into a its final format. Here are the CPU time per corresponding query:
|
3 nodes |
6 nodes |
obj_select_cpu at 50k visits, sec |
0.17 |
0.16 |
src_select_cpu at 50k visits, sec |
0.94 |
1.13 |
fsrc_select_cpu at 50k visits, sec |
0.43 |
0.49 |
Better measure for performance may be a difference between real and CPU times:
|
3 nodes |
6 nodes |
obj_select_cpu at 50k visits, sec |
0.35 |
0.16 |
src_select_cpu at 50k visits, sec |
1.9 |
0.97 |
fsrc_select_cpu at 50k visits, sec |
0.97 |
0.21 |
Here the improvement is more dramatic, factor of 2 or better (but the number are not very precise of course).
I think I need to optimize client side to make overhead much smaller. I am now converting results to afw.Table like I did in older SQL code, and I think AP really wants to use pandas. I did not try to optimize afw conversion at all but there maybe ways to make that faster and for pandas I have found one recipe for optimal Cassandra performance: https://groups.google.com/a/lists.datastax.com/g/python-driver-user/c/1v-KHtyA0Zs
One more interesting observation - looking at JMX metrics from Cassandra I noticed that write latency behaves differently on different nodes, e.g. DiaSource latency:
On this plot one node (apdb-server-1) has lower latency that all other nodes, and one node (apdb-server-4) has significantly higher latency. This looks odd as my impression is that all nodes should look more or less similar. Not sure what it means, just a note to possibly investigate later.
OK, here it goes:
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.128.0.116 553.12 KiB 256 ? 9dea29d2-81e1-47f7-b090-790273382b5f rack1
UN 10.128.0.114 437.55 KiB 256 ? dc3c9e53-b336-4cde-9a68-0c1ecc92aa18 rack1
UN 10.128.0.115 80.01 MiB 256 ? 36ddf99b-742a-48d4-a1b7-e82c8140d189 rack1
UN 10.128.0.110 304.19 KiB 256 ? ba56d50c-eb46-407b-9a9f-45452a11550b rack1
UN 10.128.0.34 382.9 KiB 256 ? 5b298a67-c59b-4b3c-8e9a-00263e14d0a6 rack1
UN 10.128.0.30 232.81 KiB 256 ? e0be23ed-2516-45c5-88c5-bbdf9ffe9481 rack1
Started short run with 1k visits, no problems so far after 50 visits.