The input data set:
- uncompressed size: 17 GB
- files: 792
The workflow equally divided 792 files into 132 files per each Qserv worker (6 workers) and 12 processing streams per worker. There were 72 such streams to be run in parallel across all workers. To optimize further operations with the input data, the workflow merged 11 files into a single TSV/CSV file to be processed by each stream. Thus the total number of input files was reduces to 72.
The amount of data to be processed at each worker is reported below:
There were 72 super-transactions started to ingest catalog data (one such transaction per each loading stream).
The file ingest stage per se took less than 7 hours which corresponds to 0.7 GB/s of the aggregate ingest performance.
Database publishing stage was quite lengthy. It took 5 hours for the operation to complete. Most of the time during the operation was spent by consolidating MySQL partitions into monolithic tables for every single chunk and its incarnation for both partitioned tables (chunk table and the corresponding overlap table). While this transformation was happening the CPU and I/O resources of the worker nodes were underutilized. Perhaps, the number of worker threads need to be increased from 16 threads per worker (matching the number of CPU _cores) to some bigger number like 32 or even higher. This may help hiding various latencies in the system.
It took 3 hours and 32 minutes to build the secondary index. The index table was 86 GB. The performance of the operation is known to be limited by the insert time into the corresponding InnoDB table at Qserv czar database. The build time is generally a function of the total number of entries (object identifiers) in the index. The performance could be significantly improved if the index table would be split into many MySQL partitions corresponding to super-transactions. By the way, this is what happens if an ingest workflow chooses to make contributions to the index at a commit time of super-transactions.