Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6573

Determine what is occurring during mmap and mlock call that speeds up queries.

    XMLWordPrintable

Details

    • Story
    • Status: To Do
    • Resolution: Unresolved
    • None
    • None
    • None
    • Data Access and Database

    Description

      Calling mmap and mlock causes queries to run significantly faster, including solo queries, which is unexpected. Further, the mlock call must complete before the query is passed to mysql and only one mlock call can be be running at a time or the speed benefit vanishes. This is not the expected behavior and it would be good to know exactly what is happening. At this point, this has only been tested on the in2p3 cluster.

      Attachments

        Issue Links

          Activity

            This issue has been escalated to the DMCCB since it seems no work has been done for a long time.

            Is it still a valid issue?

            Please address with a comment or a change of status.

            gcomoretto Gabriele Comoretto [X] (Inactive) added a comment - This issue has been escalated to the DMCCB since it seems no work has been done for a long time. Is it still a valid issue? Please address with a comment or a change of status.

            Still relevant; low priority.

            fritzm Fritz Mueller added a comment - Still relevant; low priority.
            jgates John Gates added a comment -

            While a speedup was expected, it was unexpected that solo queries finished about 40% faster with the tables locked in memory with both MariadDB and MySQL. This likely comes down to how those programs access storage in full table scans. While mlock is supposed to allow lazy locking (lock the pages of the file into memory as they are read) the Linux Kernel doesn't really support this, so all the pages of the file must be in memory before locking. abh knows more about this. It would be interesting to so see how much affect using mlock has on a system using SSD storage.

            jgates John Gates added a comment - While a speedup was expected, it was unexpected that solo queries finished about 40% faster with the tables locked in memory with both MariadDB and MySQL. This likely comes down to how those programs access storage in full table scans. While mlock is supposed to allow lazy locking (lock the pages of the file into memory as they are read) the Linux Kernel doesn't really support this, so all the pages of the file must be in memory before locking. abh  knows more about this. It would be interesting to so see how much affect using mlock has on a system using SSD storage.

            People

              jgates John Gates
              jgates John Gates
              Fritz Mueller, Gabriele Comoretto [X] (Inactive), John Gates
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.