Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4690

Design worker scheduler for shared scans

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Attachments

        Activity

        Hide
        jgates John Gates added a comment - - edited

        The basic design for the multi-table shared scan scheduler for the workers has been decided after a fair amount of deliberation. It is broken into two main pieces, the scheduler and the resource manager.

        The resource manager tracks which tables are in memory with a reference count for each, and how much memory is available. The individual schedulers ask the memory manager the required tables from the chunk are in memory or if there is enough room in memory to load them before starting a new task. If there's enough room, reference counters are incremented and files are locked in memory as mysql reads them. When a task completes, reference counters for the tables are decremented and tables unlocked.

        The scheduler is broken into 4 sub-schedulers, Interactive, FastScan, NormalScan, and SlowScan. They are listed in order of priority, the first scans get the first crack at resources, with some mechanisms to ensure that each sub-scheduler can run at least one task at any given time. Interactive is used for queries that only need to use a few chunks on a worker, and is essentially a Fifo with some grouping by chunkID. The scans are for queries that will use most or all of the chunks on the worker. Each scan goes through all the chunks on the worker in order, advancing to the next chunk when it has done all the tasks for the current chunk. FastScan handles queries that use only the Object table, and should complete in about an hour. NormalScan would be queries that use the Object table and the Source table or the Object_Extended table, and should complete in about 8 hours. SlowScan includes Object and ForcedSource tables and should complete in 12 hours.

        Show
        jgates John Gates added a comment - - edited The basic design for the multi-table shared scan scheduler for the workers has been decided after a fair amount of deliberation. It is broken into two main pieces, the scheduler and the resource manager. The resource manager tracks which tables are in memory with a reference count for each, and how much memory is available. The individual schedulers ask the memory manager the required tables from the chunk are in memory or if there is enough room in memory to load them before starting a new task. If there's enough room, reference counters are incremented and files are locked in memory as mysql reads them. When a task completes, reference counters for the tables are decremented and tables unlocked. The scheduler is broken into 4 sub-schedulers, Interactive, FastScan, NormalScan, and SlowScan. They are listed in order of priority, the first scans get the first crack at resources, with some mechanisms to ensure that each sub-scheduler can run at least one task at any given time. Interactive is used for queries that only need to use a few chunks on a worker, and is essentially a Fifo with some grouping by chunkID. The scans are for queries that will use most or all of the chunks on the worker. Each scan goes through all the chunks on the worker in order, advancing to the next chunk when it has done all the tasks for the current chunk. FastScan handles queries that use only the Object table, and should complete in about an hour. NormalScan would be queries that use the Object table and the Source table or the Object_Extended table, and should complete in about 8 hours. SlowScan includes Object and ForcedSource tables and should complete in 12 hours.
        Hide
        jbecla Jacek Becla added a comment -

        And I'll that the speed of each scan will be configurable (if not in the first implementation, definitely in the production one)

        Show
        jbecla Jacek Becla added a comment - And I'll that the speed of each scan will be configurable (if not in the first implementation, definitely in the production one)

          People

          • Assignee:
            jgates John Gates
            Reporter:
            fritzm Fritz Mueller
            Watchers:
            Andy Hanushevsky, Jacek Becla, John Gates
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Summary Panel