Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
Report
Share
Report
Share
1 of 142
Download to read offline
More Related Content
Velocity 2015 linux perf tools
1. Linux
Performance
Tools
Brendan Gregg
Senior Performance Architect
Performance Engineering Team
bgregg@netflix.com
@brendangregg
2. This
Tutorial
• A tour of many Linux performance tools
– To show you what can be done
– With guidance for how to do it
• This includes objectives, discussion, live demos
– See the video of this tutorial
Observability
Benchmarking
Tuning
Sta<c
Tuning
3. • Massive AWS EC2 Linux cloud
– 10s of thousands of cloud instances
• FreeBSD for content delivery
– ~33% of US Internet traffic at night
• Over 50M subscribers
– Recently launched in ANZ
• Use Linux server tools as needed
– After cloud monitoring (Atlas, etc.) and
instance monitoring (Vector) tools
6. Methodologies
• Objectives:
– Recognize the Streetlight Anti-Method
– Perform the Workload Characterization Method
– Perform the USE Method
– Learn how to start with the questions, before using tools
– Be aware of other methodologies
8. Methodologies
• There are dozens of performance tools for Linux
– Packages: sysstat, procps, coreutils, …
– Commercial products
• Methodologies can provide guidance for choosing and
using tools effectively
• A starting point, a process, and an ending point
10. Street
Light
An<-‐Method
1. Pick observability tools that are:
– Familiar
– Found on the Internet
– Found at random
2. Run tools
3. Look for obvious issues
12. Blame
Someone
Else
An<-‐Method
1. Find a system or environment component you are not
responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1
13. Actual
Methodologies
• Problem Statement Method
• Workload Characterization Method
• USE Method
• Off-CPU Analysis
• CPU Profile Method
• RTFM Method
• Active Benchmarking (covered later)
• Static Performance Tuning (covered later)
• …
14. Problem
Statement
Method
1. What makes you think there is a performance problem?
2. Has this system ever performed well?
3. What has changed recently? (Software? Hardware?
Load?)
4. Can the performance degradation be expressed in
terms of latency or run time?
5. Does the problem affect other people or applications (or
is it just you)?
6. What is the environment? Software, hardware,
instance types? Versions? Configuration?
15. Workload
Characteriza<on
Method
1. Who is causing the load? PID, UID, IP addr, ...
2. Why is the load called? code path, stack trace
3. What is the load? IOPS, tput, type, r/w
4. How is the load changing over time?
16. The
USE
Method
• For every resource, check:
1. Utilization
2. Saturation
3. Errors
• Definitions:
– Utilization: busy time
– Saturation: queue length or queued time
– Errors: easy to interpret (objective)
• Helps if you have a functional (block) diagram of your
system / software / environment, showing all resources
Start with the questions, then find the tools
Resource
U<liza<on
(%)
X
17. USE
Method
for
Hardware
• For every resource, check:
1. Utilization
2. Saturation
3. Errors
• Including busses & interconnects
18. Linux
USE
Method
Example
hOp://www.brendangregg.com/USEmethod/use-‐linux.html
20. CPU
Profile
Method
1. Take a CPU profile
2. Understand all software in profile > 1%
• Discovers a wide range of performance issues by their
CPU usage
• Narrows software
to study
Flame
Graph
21. RTFM
Method
• How to understand performance tools or metrics:
1. Man pages
2. Books
3. Web search
4. Co-workers
5. Prior talk slides/video (this one!)
6. Support services
7. Source code
8. Experimentation
9. Social
23. Tools
• Objectives:
– Perform the USE Method for resource utilization
– Perform Workload Characterization for disks, network
– Perform the CPU Profile Method using flame graphs
– Have exposure to various observability tools:
• Basic: vmstat, iostat, mpstat, ps, top, …
• Intermediate: tcpdump, netstat, nicstat, pidstat, sar, …
• Advanced: ss, slaptop, perf_events, …
– Perform Active Benchmarking
– Understand tuning risks
– Perform Static Performance Tuning
24. Command
Line
Tools
• Useful to study even if you never use them: GUIs and
commercial products often use the same interfaces
$ vmstat 1!
procs -----------memory---------- ---swap-- …!
r b swpd free buff cache si so …!
9 0 0 29549320 29252 9299060 0 …!
2 0 0 29547876 29252 9299332 0 …!
4 0 0 29548124 29252 9299460 0 …!
5 0 0 29548840 29252 9299592 0 …!
/proc,
/sys,
…
Kernel
25. Tool
Types
Type Characteristic
Observability Watch activity. Safe, usually, depending on
resource overhead.
Benchmarking Load test. Caution: production tests can
cause issues due to contention.
Tuning Change. Danger: changes could hurt
performance, now or later with load.
Static Check configuration. Should be safe.
29. up<me
• One way to print load averages:
• A measure of resource demand: CPUs + disks
– Other OSes only show CPUs: easier to interpret
• Exponentially-damped moving averages
• Time constants of 1, 5, and 15 minutes
– Historic trend without the line graph
• Load > # of CPUs, may mean CPU saturation
– Don’t spend more than 5 seconds studying these
$ uptime!
07:42:06 up 8:16, 1 user, load average: 2.27, 2.84, 2.91!
30. top
(or
htop)
• System and per-process interval summary:
• %CPU is summed across all CPUs
• Can miss short-lived processes (atop won’t)
• Can consume noticeable CPU to read /proc
$ top - 18:50:26 up 7:43, 1 user, load average: 4.11, 4.91, 5.22!
Tasks: 209 total, 1 running, 206 sleeping, 0 stopped, 2 zombie!
Cpu(s): 47.1%us, 4.0%sy, 0.0%ni, 48.4%id, 0.0%wa, 0.0%hi, 0.3%si, 0.2%st!
Mem: 70197156k total, 44831072k used, 25366084k free, 36360k buffers!
Swap: 0k total, 0k used, 0k free, 11873356k cached!
!
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5738 apiprod 20 0 62.6g 29g 352m S 417 44.2 2144:15 java
1386 apiprod 20 0 17452 1388 964 R 0 0.0 0:00.02 top
1 root 20 0 24340 2272 1340 S 0 0.0 0:01.51 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
[…]!
49. Other
Tools
• You may also use collectl, atop, dstat, or another
measure-all tool
• The tool isn’t important – it’s important to have a way to
measure everything
• In cloud environments, you are probably using a
monitoring product, developed in-house or commercial.
– We develop Atlas for cloud-wide monitoring, and Vector for
instance-level analysis (both NetflixOSS)
– Same method applies…
59. perf_events
• Provides the "perf" command
• In Linux source code: tools/perf
– Usually pkg added by linux-tools-common, etc.
• Multi-tool with many capabilities
– CPU profiling
– PMC profiling
– Static & dynamic tracing
• Covered later in Profiling & Tracing
60. <ptop
• IPC by process, %MISS, %BUS
• Needs some love. perfmon2 library integration?
• Still can’t use it in clouds yet (needs PMCs enabled)
61. rdmsr
• Model Specific Registers (MSRs), unlike PMCs, can be
read by default in Xen guests
– Timestamp clock, temp, power, …
– Use rdmsr(1) from the msr-tools package to read them
– From https://github.com/brendangregg/msr-cloud-tools:
ec2-guest# ./showboost!
[...]!
TIME C0_MCYC C0_ACYC UTIL RATIO MHz!
06:11:35 6428553166 7457384521 51% 116% 2900!
06:11:40 6349881107 7365764152 50% 115% 2899!
06:11:45 6240610655 7239046277 49% 115% 2899!
[...]!
ec2-guest# ./cputemp 1!
CPU1 CPU2 CPU3 CPU4!
61 61 60 59!
60 61 60 60!
[...]!
Real
CPU
MHz
CPU
Temperature
62. More
Advanced
Tools…
• Some others worth mentioning:
Tool
Descrip-on
ltrace
Library
call
tracer
ethtool
Mostly
interface
tuning;
some
stats
snmpget
SNMP
network
host
sta<s<cs
lldptool
Can
get
LLDP
broadcast
stats
blktrace
Block
I/O
event
tracer
/proc
Many
raw
kernel
counters
pmu-‐tools
On-‐
and
off-‐core
CPU
counter
tools
63. Advanced
Tracers
• Many options on Linux:
– perf_events, ftrace, eBPF, SystemTap, ktap, LTTng,
dtrace4linux, sysdig
• Most can do static and dynamic tracing
– Static: pre-defined events (tracepoints)
– Dynamic: instrument any software (kprobes, uprobes).
Custom metrics on-demand. Catch all.
• Many are in-development
68. Benchmarking
• ~100% of benchmarks are wrong
• Results are usually misleading:
you benchmark A, but actually measure B, and conclude
you measured C
• Common mistakes:
– Testing the wrong target: eg, FS cache instead of disk
– Choosing the wrong target: eg, disk instead of FS cache
… doesn’t resemble real world usage
– Invalid results: eg, bugs
• The energy needed to refute benchmarks is multiple
orders of magnitude bigger than to run them
69. Ac<ve
Benchmarking
(Method)
1. Run the benchmark for hours
2. While running, analyze and confirm the performance
limiter using observability tools
– Disk benchmark: run iostat, …
– CPU benchmark: run pidstat, perf, flame graphs, …
– …
• Answer the question: why isn't the result 10x?
We just covered the observability tools – use them!
70. lmbench
• CPU, memory, and kernel micro-benchmarks
• Eg, memory latency by stride size:
$ lat_mem_rd 100m 128 > out.latencies!
some R processing…!
L1
cache
L2
cache
Main
Memory
L3
cache
75. Tuning
Tools
• Generic interfaces:
– sysctl, /sys
• Many areas have custom tuning tools:
– Applications: their own config
– CPU/scheduler: nice, renice, taskset, ulimit, chcpu
– Storage I/O: tune2fs, ionice, hdparm, blockdev, …
– Network: ethtool, tc, ip, route
– Dynamic patching: stap, kpatch
76. Tuning
Methods
• Scientific Method:
1. Question
2. Hypothesis
3. Prediction
4. Test
5. Analysis
• Any observational or benchmarking tests you can try
before tuning?
• Consider risks, and see previous tools
79. Sta<c
Tools
• Static Performance Tuning: check the static state and
configuration of the system
– CPU types & flags
– CPU frequency scaling config
– Storage devices
– File system capacity
– File system and volume configuration
– Route table
– State of hardware
– etc.
• What can be checked on a system without load
• Methodology by Richard Elling (2000)
80. CPU
Types
&
Flags
$ more /proc/cpuinfo !
processor! !: 0!
vendor_id! !: GenuineIntel!
cpu family !: 6!
model ! !: 42!
model name !: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz!
stepping ! !: 7!
microcode! !: 0x1a!
cpu MHz ! !: 1600.000!
cache size !: 6144 KB!
physical id !: 0!
siblings ! !: 4!
core id ! !: 0!
cpu cores! !: 4!
apicid ! !: 0!
initial apicid!: 0!
fpu ! ! !: yes!
fpu_exception !: yes!
cpuid level !: 13!
wp ! ! !: yes!
flags ! !: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc ar!
ch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni
pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2
x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts
dtherm tpr_shadow vnmi flexpriority ept vpid!
[…]!
CPU
speed
s<ll
maOers
81. CPU
Frequency
Scaling
• Kernel may be configured to dynamically modify CPU
frequency:
– See Documentation/cpu-freq/governors.txt,
and scaling_governor == performance
• Not to be confused with Intel Turbo Boost (which is H/W)
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies !
3101000 3100000 2900000 2700000 2500000 2300000 2100000 1900000 1700000 1600000 !
# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor !
ondemand!
87. Profiling
• Objectives:
– Profile CPU usage by stack sampling
– Generate CPU flame graphs
– Understand gotchas with stacks & symbols
88. CPU
Profiling
A!
B!
block
interrupt
on-‐CPU
off-‐CPU
A!
B!
A! A!
B!
A!
syscall
<me
• Record stacks at a timed interval: simple and effective
– Pros: Low (deterministic) overhead
– Cons: Coarse accuracy, but usually sufficient
stack
samples:
A!
89. perf_events
• Introduced earlier: multi-tool, profiler. Provides "perf".
usage: perf [--version] [--help] [OPTIONS] COMMAND [ARGS]!
!
The most commonly used perf commands are:!
annotate Read perf.data (created by perf record) and display annotated code!
archive Create archive with object files with build-ids found in perf.data file!
bench General framework for benchmark suites!
buildid-cache Manage build-id cache.!
buildid-list List the buildids in a perf.data file!
data Data file related processing!
diff Read perf.data files and display the differential profile!
evlist List the event names in a perf.data file!
inject Filter to augment the events stream with additional information!
kmem Tool to trace/measure kernel memory(slab) properties!
kvm Tool to trace/measure kvm guest os!
list List all symbolic event types!
lock Analyze lock events!
mem Profile memory accesses!
record Run a command and record its profile into perf.data!
report Read perf.data (created by perf record) and display the profile!
sched Tool to trace/measure scheduler properties (latencies)!
script Read perf.data (created by perf record) and display trace output!
stat Run a command and gather performance counter statistics!
test Runs sanity tests.!
timechart Tool to visualize total system behavior during a workload!
top System profiling tool.!
trace strace inspired tool!
probe Define new dynamic tracepoints!
!
See 'perf help COMMAND' for more information on a specific command.!
90. perf_events:
CPU
profiling
• Sampling full stack traces at 99 Hertz, for 30 secs:
# perf record -F 99 -ag -- sleep 30!
[ perf record: Woken up 9 times to write data ]!
[ perf record: Captured and wrote 2.745 MB perf.data (~119930 samples) ]!
# perf report -n --stdio!
1.40% 162 java [kernel.kallsyms] [k] _raw_spin_lock
|!
--- _raw_spin_lock!
| !
|--63.21%-- try_to_wake_up!
| | !
| |--63.91%-- default_wake_function!
| | | !
| | |--56.11%-- __wake_up_common!
| | | __wake_up_locked!
| | | ep_poll_callback!
| | | __wake_up_common!
| | | __wake_up_sync_key!
| | | | !
| | | |--59.19%-- sock_def_readable!
[…78,000 lines truncated…]!
93. perf_events:
Flame
Graphs
• Flame Graphs:
– x-axis: alphabetical stack sort, to maximize merging
– y-axis: stack depth
– color: random, or hue can be a dimension (eg, diff)
• Interpretation:
– Top edge is on-CPU, beneath it is ancestry
• Currently made from Perl + JavaScript & SVG
• Easy to get working
– http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
git clone --depth 1 https://github.com/brendangregg/FlameGraph!
cd FlameGraph!
perf record -F 99 -a –g -- sleep 30!
perf script | ./stackcollapse-perf.pl |./flamegraph.pl > perf.svg!
98. perf_events:
Gotchas
• Stack traces and symbols often don't work.
– Can be a significant project to fix them. It is worth it!
• C:
– stacks: --fno-omit-frame-pointer
• Java:
– stacks on x86: -XX:+PreserveFramePointer
(JDK-8068945 for JDK9, JDK-8072465 for JDK8)
– symbols: perf-map-agent (other solutions exist)
• Node.js:
– symbols: run v8 with --perf_basic_prof
http://www.brendangregg.com/blog/2015-02-27/linux-profiling-at-netflix.html
99. perf_events:
Counters
• Performance Monitoring Counters (PMCs):
• Identify CPU cycle breakdowns, esp. stall types
• PMCs not enabled by-default in clouds (yet)
• Can be time-consuming to use (CPU manuals)
– Please develop front-ends. Eg, tiptop.
$ perf list | grep –i hardware!
cpu-cycles OR cycles [Hardware event]!
stalled-cycles-frontend OR idle-cycles-frontend [Hardware event]!
stalled-cycles-backend OR idle-cycles-backend [Hardware event]!
instructions [Hardware event]!
[…]!
branch-misses [Hardware event]!
bus-cycles [Hardware event]!
L1-dcache-loads [Hardware cache event]!
L1-dcache-load-misses [Hardware cache event]!
[…]!
rNNN (see 'perf list --help' on how to encode it) [Raw hardware event … !
mem:<addr>[:access] [Hardware breakpoint]!
104. Linux
Tracing
Tools
• Many choices (too many?), all still in development
irace
perf_events
eBPF
LTTng
dtrace4linux
SystemTap
ktap
OEL
DTrace
sysdig
105. Linux
Tracing
is
Magic!
• (Thanks Deirdré Straughan & General Zoi's Pony Creator)
irace
perf_events
eBPF
LTTng
dtrace4linux
SystemTap
ktap
OEL
DTrace
sysdig
106. Choosing
a
Tracer
• Some companies standardize on one tracer
– eg, SystemTap, LTTng, …
107. Choosing
a
Tracer
• My approach is: Study
what
Linux
already
has
built-‐in
(perf_events,
irace,
eBPF?)
perf_events
irace
eBPF
live
tracing,
coun<ng
in-‐kernel
summaries
PMCs,
stack
profiling,
trace-‐dump-‐
analyze
Purpose?
Y
N
Is
it
sufficient?
…
Try
SystemTap
Try
LTTng
109. irace
• Added by Steven Rostedt and others since 2.6.27
• Already enabled on our servers (3.2+)
– CONFIG_FTRACE, CONFIG_FUNCTION_PROFILER, …
– Use directly via /sys/kernel/debug/tracing:
– See Linux source: Documentation/trace/ftrace.txt
linux-4.0.0+# ls /sys/kernel/debug/tracing/!
available_events max_graph_depth stack_max_size!
available_filter_functions options stack_trace!
available_tracers per_cpu stack_trace_filter!
buffer_size_kb printk_formats trace!
buffer_total_size_kb README trace_clock!
current_tracer saved_cmdlines trace_marker!
dyn_ftrace_total_info saved_cmdlines_size trace_options!
enabled_functions set_event trace_pipe!
events set_ftrace_filter trace_stat!
free_buffer set_ftrace_notrace tracing_cpumask!
function_profile_enabled set_ftrace_pid tracing_max_latency!
instances set_graph_function tracing_on!
kprobe_events set_graph_notrace tracing_thresh!
kprobe_profile snapshot!
110. irace
Front-‐Ends
• Steven wrote a front-end: trace-cmd
– Multi-tool, works well
• I've developed the "perf-tools" front-ends
– https://github.com/brendangregg/perf-tools
– Both single & multi-purpose, Unix-like
– Unsupported hacks: see WARNINGs
• perf-tools:
– single-purpose: iosnoop, iolatency, opensnoop
– multi-tools: funccount, funcgraph, kprobe
111. iosnoop
• Block I/O (disk) events with latency:
# ./iosnoop –ts!
Tracing block I/O. Ctrl-C to end.!
STARTs ENDs COMM PID TYPE DEV BLOCK BYTES LATms!
5982800.302061 5982800.302679 supervise 1809 W 202,1 17039600 4096 0.62!
5982800.302423 5982800.302842 supervise 1809 W 202,1 17039608 4096 0.42!
5982800.304962 5982800.305446 supervise 1801 W 202,1 17039616 4096 0.48!
5982800.305250 5982800.305676 supervise 1801 W 202,1 17039624 4096 0.43!
[…]!
# ./iosnoop –h!
USAGE: iosnoop [-hQst] [-d device] [-i iotype] [-p PID] [-n name] [duration]!
-d device # device string (eg, "202,1)!
-i iotype # match type (eg, '*R*' for all reads)!
-n name # process name to match on I/O issue!
-p PID # PID to match on I/O issue!
-Q # include queueing time in LATms!
-s # include start time of I/O (s)!
-t # include completion time of I/O (s)!
-h # this usage message!
duration # duration seconds, and use buffers!
[…]!
131. SystemTap
• Fully programmable, fully featured
– Including access to user-level tracepoints
• Compiles tracing programs into kernel modules
– Needs a compiler, and takes time
• “Works great on Red Hat”
– I keep trying on other distros and have hit trouble in the past
– Make sure you are on the latest version (>=2.6)
• "Works great with kernel debuginfo"
– Suited for kernel developers who have it handy
– A difficult requirement for our cloud environment
132. "Lightweight"
SystemTap
• SystemTap can be used without kernel debuginfo
– Makes life harder, but some tasks are still possible
– providers: nd_syscall, kprobe.function, kernel.trace, …
– args via: int_arg(), uint_arg(), pointer_arg(), user_string()
• Something I've experimented with. Examples:
– https://github.com/brendangregg/systemtap-lwtools/
# stap -e 'global a; probe nd_syscall.write { a <<< int_arg(3); } probe end
{ print(@hist_log(a)); }'!
^Cvalue |-------------------------------------------------- count!
[…]!
8 | 0!
16 |@@@@@@@@@@@@@@@@@@@@@@ 22!
32 |@@@@@@@@@@@@@@@ 15!
64 |@@@@@@@@@@@@@@@@@ 17!
128 |@@ 2!
256 |@@ 2!
512 | 0!
133. ktap
• Was a very promising new Linux tracer:
– Sampling, static & dynamic tracing
– Lightweight, simple. Uses bytecode.
– Suited for embedded devices
• Development suspended while eBPF integrates
• Will it restart?
134. sysdig
• sysdig: Innovative new tracer. Simple expressions:
• Replacement for strace? (or “perf trace” will)
• Programmable “chisels”. Eg, one of mine:
• Currently syscalls and user-level processing only
– I'm not sure it can be optimized enough for kernel tracing,
unless it adopts eBPF for in-kernel processing & summaries
# sysdig -c fileslower 1!
TIME PROCESS TYPE LAT(ms) FILE!
2014-04-13 20:40:43.973 cksum read 2 /mnt/partial.0.0!
2014-04-13 20:40:44.187 cksum read 1 /mnt/partial.0.0!
2014-04-13 20:40:44.689 cksum read 2 /mnt/partial.0.0!
[…]!
sysdig fd.type=file and evt.failed=true!
sysdig evt.type=open and fd.name contains /etc!
sysdig -p"%proc.name %fd.name" "evt.type=accept and proc.name!=httpd”!
135. Present
&
Future
• Present:
– ftrace & perf_events solving many needs today:
• PMC profiling, stack profiling, tracepoint & dynamic tracing, …
• Expected Future:
– eBPF for kernel summaries & advanced programs
– eBPF perf integration to improve ease of use
• Possible Future:
– eBPF high level language (ktap?)
– ftrace/eBPF integration
– Other tracer eBPF integration (SystemTap, LTTng, sysdig?)
– One of the other tracers going mainline?
136. The
Tracing
Landscape,
May
2015
Scope
&
Capability
Ease
of
use
sysdig
perf
irace
eBPF
ktap
stap
Stage
of
Development
(my
opinion)
dtrace4L.
(brutal)
(less
brutal)
(alpha)
(mature)
138. Methodologies
Summary
• Objectives:
– Recognize the Streetlight Anti-Method
– Perform the Workload Characterization Method
– Perform the USE Method
– Be aware of other methodologies
Try to start with the questions (methodology), to help guide
your use of the tools
139. Tools
Summary
• Objectives:
– Perform the USE Method for resource utilization
– Perform Workload Characterization for disks, network
– Have exposure to various observability tools:
• Basic: vmstat, iostat, mpstat, ps, top, …
• Intermediate: tcpdump, netstat, nicstat, pidstat, sar, …
• Advanced: ss, slaptop, perf_events, …
– Perform Active Benchmarking
– Understand tuning risks
– Perform Static Performance Tuning
Print out the tools diagrams for your office wall
140. Profiling
&
Tracing
Summary
• Objectives:
– Profile CPU usage by stack sampling
– Generate CPU flame graphs
– Understand gotchas with stacks & symbols
– Understand frameworks: tracepoints, kprobes, uprobes
– Understand mainline tracers: ftrace, perf_events, eBPF
– Awareness of other tracers: SystemTap, LTTng, ktap, sysdig
– Awareness of what tracing can accomplish (eg, perf-tools)
I've hopefully turned some unknown unknowns into known
unknowns
141. References
&
Links
– Systems
Performance:
Enterprise
and
the
Cloud,
Pren<ce
Hall,
2013
– hOp://www.brendangregg.com/linuxperf.html
incl.
tools
diagrams
as
PNGs
– hOp://www.brendangregg.com/perf.html#FlameGraphs
– hOp://www.brendangregg.com/blog/2015-‐02-‐27/linux-‐profiling-‐at-‐neqlix.html
– hOp://www.brendangregg.com/blog/2015-‐03-‐17/linux-‐performance-‐analysis-‐perf-‐tools.html
– hOp://www.brendangregg.com/blog/2015-‐05-‐15/ebpf-‐one-‐small-‐step.html
– nicstat:
hOp://sourceforge.net/projects/nicstat/
– <ptop:
hOp://<ptop.gforge.inria.fr/
• Tiptop:
Hardware
Performance
Counters
for
the
Masses,
Erven
Rohou,
Inria
Research
Report
7789,
Nov
2011.
– irace
&
perf-‐tools
• hOps://github.com/brendangregg/perf-‐tools
• hOp://lwn.net/Ar<cles/608497/
Ftrace:
The
hidden
light
switch
– MSR
tools:
hOps://github.com/brendangregg/msr-‐cloud-‐tools
– pcstat:
hOps://github.com/tobert/pcstat
– eBPF:
hOp://lwn.net/Ar<cles/603983/
– ktap:
hOp://www.ktap.org/
– SystemTap:
hOps://sourceware.org/systemtap/
– sysdig:
hOp://www.sysdig.org/
– Tux
by
Larry
Ewing;
Linux®
is
the
registered
trademark
of
Linus
Torvalds
in
the
U.S.
and
other
countries.