Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26102

Enable auto-tuning for the XRootD "spread" in Qserv

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      This ticket is based on DM-25891.

      The new version of XRootD allows to auto-tune the client parameter "spread" by setting its value to 0. Hence, an objective of this effort is to test the effect of auto-tuning when processing multiple shared scan queries at a large-scale Qserv installation. If it works, and if it doesn't have any side effects on the performance of Qserv then make auto-tuning the default parameter of Qserv.

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment -

            The "auto-tuning" has been tested in the dev Qserv cluster at NCSA by launching simultaneously up to 140 of the “shared scan” queries:

            SELECT MAX(ra) FROM wise_00.allwise_p3as_mep WHERE ra < ...
            

            Where each query had a unique threshold for ra.

            All queries successfully finished. The query run time was in a range of 1h30m - 3h50m. That spread (in the query run time) wasn’t a surprise, and it’s not directly caused by xrootd.

            Show
            gapon Igor Gaponenko added a comment - The "auto-tuning" has been tested in the dev Qserv cluster at NCSA by launching simultaneously up to 140 of the “shared scan” queries: SELECT MAX (ra) FROM wise_00.allwise_p3as_mep WHERE ra < ... Where each query had a unique threshold for ra . All queries successfully finished. The query run time was in a range of 1h30m - 3h50m. That spread (in the query run time) wasn’t a surprise, and it’s not directly caused by xrootd.
            Show
            gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/558
            Hide
            jgates John Gates added a comment -

            I don't see any problems.

            Show
            jgates John Gates added a comment - I don't see any problems.

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              John Gates
              Watchers:
              Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: