# Fix a configuration of the partitioner in the Qserv integration test

XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
2
• Sprint:
DB_S21_12
• Team:
Data Access and Database

# Goals

This effort was triggered by DM-27915 and DM-26754.

The current version of the Qserv integration tests (Git package https://github.com/lsst/qserv_testdata) fails to load partitioned files into Qserv. A root cause of the problem has been found in subtle differences in how the partitioner's configuration system and the Qserv data loader script qserv-data-loader.py treat the JSON-based configuration files of the partitioning tools. Though the later correctly interprets the new configuration format (JSON) it still gives higher priority to parameters specified first in a chain of the configuration files:

 qserv-data-loader.py -c high.json -c medium.json - c low.json 

While the former assumes that any subsequent file would override values of similar parameters (if any) found in the previous files. Moreover, the partitioner's configuration system would also assume that any single-value overrides of the parameters specified in the command line would have the highest priority:

 sph-partition -c low.json -c medium.json - c high.json --part.id="objectId" --csv.out.delimiter="," 

Streamlining the priority was one of the improvements made to the partitioner's configuration during its migration to the new configuration system in DM-26754.

Further analysis of the problem has reviled that the partitioner's configuration files of the Qserv integration test were loosely written allowing the above-described ambiguity to break the tests. Hence, a solution to the problem is to make corrections to the configuration files.

## Other actions

Update documentation of package https://github.com/lsst/partition regarding the behavior of the partitioner's configuration system as mentioned in DM-28010.

#### Activity

Hide
Fritz Mueller added a comment -

lgtm – thanks!

Show
Fritz Mueller added a comment - lgtm – thanks!

#### People

Assignee:
Igor Gaponenko
Reporter:
Igor Gaponenko
Reviewers:
Fritz Mueller
Watchers:
Fritz Mueller, Igor Gaponenko, Nate Pease