Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28008

Fix a configuration of the partitioner in the Qserv integration test

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Goals

      This effort was triggered by DM-27915 and DM-26754.

      The current version of the Qserv integration tests (Git package https://github.com/lsst/qserv_testdata) fails to load partitioned files into Qserv. A root cause of the problem has been found in subtle differences in how the partitioner's configuration system and the Qserv data loader script qserv-data-loader.py treat the JSON-based configuration files of the partitioning tools. Though the later correctly interprets the new configuration format (JSON) it still gives higher priority to parameters specified first in a chain of the configuration files:

      qserv-data-loader.py -c high.json -c medium.json - c low.json 
      

      While the former assumes that any subsequent file would override values of similar parameters (if any) found in the previous files. Moreover, the partitioner's configuration system would also assume that any single-value overrides of the parameters specified in the command line would have the highest priority:

      sph-partition -c low.json -c medium.json - c high.json --part.id="objectId" --csv.out.delimiter=","
      

      Streamlining the priority was one of the improvements made to the partitioner's configuration during its migration to the new configuration system in DM-26754.

      Further analysis of the problem has reviled that the partitioner's configuration files of the Qserv integration test were loosely written allowing the above-described ambiguity to break the tests. Hence, a solution to the problem is to make corrections to the configuration files.

      Other actions

      Update documentation of package https://github.com/lsst/partition regarding the behavior of the partitioner's configuration system as mentioned in DM-28010.

        Attachments

          Issue Links

            Activity

            Hide
            fritzm Fritz Mueller added a comment -

            lgtm – thanks!

            Show
            fritzm Fritz Mueller added a comment - lgtm – thanks!

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Fritz Mueller
              Watchers:
              Fritz Mueller, Igor Gaponenko, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.