Understanding the impact of Linux scheduler on your application

Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Understanding the impact of Scheduler
on your application
Atish Patra
Dhaval Giani
Linux Kernel Developer, Oracle

"You don't need to know how to operate an X-ray machine, but you do need
to know that if you swallow a penny, an X-ray is an option."
- Brendan Gregg's friend

Program Agenda
Introduction to Scheduler
Load Balancing
Application Analysis
Scheduler Tunables
Conclusion
1
2
3
4
5
4

Introduction
5

Linux Scheduler
• Who gets to run when (and where)
6

Completely Fair Scheduler
• Two tasks
• T1 runs
• Followed by T2
• Followed by T1
• …
7

How long?
• T1 has nice -1 and T2 has nice 0
• T1 gets to run twice as long as T2!
• nice
• weight
• vruntime
vruntime ⍺ 1/weight

Group Scheduling
• Consider Apache spawns 1000 threads for every Oracle thread.
• Does that mean Apache gets to run 1000 times as often?
• Enter Group Scheduling
• Apache Group has all the Apache threads
• Oracle Group has all the Oracle threads
• Oracle and Apache groups have their own nice values
9

Symmetric Multiprocessing
10
CPU 0
CPU 3CPU 2
CPU 1
Last Level Cache (LLC)

Load Balancing
• Consider
• 4 CPUs (quad core laptops!)
• 4 tasks (let’s consider same nice value)
• Each CPU gets 1
• Now consider
• 4 CPUs
• 5 tasks (1 has nice value 1)
• Now what?
11

Load Balancing with group scheduling
• Consider Apache with 1000 threads
• Consider Oracle with 4 threads
• Apache and Oracle have same weight
• 4 CPUs. Apache gets 2, Oracle gets 2
• Now what if we had a third group with 3 threads?
12

Non Uniform Memory Access
13
CPU 0 CPU 1 CPU 4 CPU 5
CPU 2 CPU 3 CPU 6 CPU 7

Load Balancing
• Problems:
– Who runs where ?
– Who moves where ?
– How much movement ?
• Too much: tasks play ping pong!
• Too less: cores go idle!
14

Scheduling domains/groups
15

Why scheduling domains?
• Issues
– System wide task movement and search cost
– Hardware properties
– Solution
– A hierarchal search to reduce the cost
16

Scheduling Domains - Example
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Last Level Cache
Last Level Cache
Last Level Cache
Last Level Cache

Scheduling Domains - SMT Domain
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Last Level Cache
Last Level Cache
Last Level Cache
Last Level Cache

Scheduling Domains - MC Domain
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Last Level Cache
Last Level Cache
Last Level Cache
Last Level Cache

Scheduling Domains - Die/CPU Domain
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Last Level Cache
Last Level Cache
Last Level Cache
Last Level Cache

Scheduling Domains - NUMA Domain
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Last Level Cache
Last Level Cache
Last Level Cache
Last Level Cache

Load Balancing (Pull)
• Periodic balancing
– Happens at every scheduling tick
– Trace domain to find imbalance
– One CPU performs the search
– Frequency reduces as we traverse up the hierarchy
22

0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15

T1
T2
T3
T4
• New Idle balancing
– average idle time > migration_cost
– average idle time > domain search
cost
– Needs SD_BALANCE_NEWIDLE
enabled

• New Idle balancing
– average idle time > migration_cost
– average idle time > domain search
cost
– Needs SD_BALANCE_NEWIDLE
enabled
T1T2
T3T4

• Wakeup load balancing
– cache affinity
– waker-wakee relationship
– search in the same LLC domain
T1
T2
T3T4

– cache affinity
T2
T3T4

– cache affinity
T2
T3T4
T1

– cache affinity
T2
T3T4
T1
T5

• Fork/Exec load balancing
– Cache affinity might not matter
– Find the least loaded Group
– Find the least loaded CPU
– Needs SD_BALANCE_FORK/EXEC
enabled
T2
T5T4
T1
T3 T6
T7

• Fork/Exec load balancing
– Cache affinity might not matter
– Find the least loaded Group
– Find the least loaded CPU
– Needs SD_BALANCE_FORK/EXEC
enabled
T2
T5T4
T1
T3 T6
T7
T8

Application Analysis

Application Analysis - Tools (Our experience)
• Manual instrumentation
– printk/pr_* : NO!
– trace_printk(): Maybe!
• Issues
– Kernel modification
– Parsing difficulty
36

Application Analysis - perf
• Perf
– call graphs
# sudo perf record -a -g -e cycles perf bench sched
messaging -g 1000 -l 10000
# sudo perf report --no-children -g
-s symbol > perf_callg.out
• Difficult to process in this format
37

• Flamegraph is immensely useful
$ sudo perf script |
./stackcollapse-perf.pl > out.perf-folded
$./flamegraph.pl out.perf-folded > perf-kernel.svg
38

• Use perf stat for migrations and cache-misses
$ sudo perf stat -a -e migrations -e cache-misses
-e cache-references perf bench sched messaging
-g 100 -l 1000
Performance counter stats for 'system wide':
7,847,249 migrations
20,950,769,373 cache-misses # 32.308 % of all cache refs (100.00%)
64,847,047,418 cache-references

Application Analysis - Ftrace
•trace-cmd or manual
•Able to trace each task wakeup/migration events
#trace-cmd record -e sched:sched_wakeup_new -e sched:sched_migrate_task perf bench sched messaging -g 100 -l 100
•trace-cmd spawns a process on each processor
•Might be tricky if task migrations/wakeup analysis
40

Application Analysis - Impact of Scheduling domains/
groups
41
• /proc/schedstat
– scheduling domains/groups follow hardware ?
– load balancing statistics
– Too many new idle balances especially across NUMA ?
• Try to reduce relax_domain_level

Application Analysis - mpstat
• mpstat -P ALL -u 5
42

Application Analysis - eBPF
• New kid on the block
•Find cpu idleness due to scheduler
•https://github.com/iovisor/bcc/blob/master/tools/
cpuunclaimed.py
43

Application Analysis - eBPF
•Find run-queue length distribution
•https://github.com/iovisor/bcc/
blob/master/tools/runqlen.py
44

Scheduler Tunables

Scheduler Tunables
46
•sysctl -A | grep "sched" | grep -v “domain”
kernel.sched_cfs_bandwidth_slice_us = 5000
kernel.sched_child_runs_first = 0
kernel.sched_latency_ns = 24000000 (24ms)
kernel.sched_migration_cost_ns = 500000 (0.5ms)
kernel.sched_min_granularity_ns = 10000000 (10ms)
kernel.sched_nr_migrate = 32
kernel.sched_schedstats = 0
kernel.sched_time_avg_ms = 1000
kernel.sched_wakeup_granularity_ns = 15000000 (15ms)

Scheduler Tunables
47
• sched_migration_cost_ns: Is task cache-hot ?
Task bouncing Cores go idle
Less wakeup preemption Reduces scheduling
delay
• sched_wakeup_granularity_ns: To preempt or not to ?

Scheduler Tunables
48
• sched_latency_ns: Period where each task guaranteed is to run
• sched_min_granularity_ns : Minimum guaranteed runtime for a task
• sched period = max(sched_latency, nr_tasks*sched_min_granularity)
Reduces latency Improves throughput
sched_min_granularity_ns

Conclusions
• Understand your application
• How does it interact with the system?
• Use the tools at your disposal
• Is it a scheduler issue?
• Experiment with different tunables
• Tune one at a time
• Might be a kernel bug!

Thank You!
atish.patra@oracle.com
dhaval.giani@oracle.com

Understanding the impact of Linux scheduler on your application

More Related Content

Understanding the impact of Linux scheduler on your application