Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
SlideShare a Scribd company logo
Linux Kernel Debugging Essentials
Lubomir Rintel
base48 Hackerspace, Brno
http://base48.cz/

February 19th , 2014

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

1 / 12
Plan

Understanding the way kernel can fail
Introduction to infrastructure
Obtaining & unserstanding the traces
Heplful in-kernel mechanisms
Building and trying the replacement code

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

2 / 12
Kernel failures

Warning, possibly non-fatal: OOPS
A fatal error: Panic
Usually accompained by a state dump
Result of a bug or a hardware failure
# echo c >/proc/sysrq-trigger

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

3 / 12
Kernel configuration

Vanilla kernels
Distribution kernels
Configuration options (build-time)
Runtime configuration (sysctls)
Boot time configuration (kernel parameters)
Documentation/kernel-parameters.txt

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

4 / 12
Lab setup

For details & non-Fedora setup, see the hackerspace wiki
# yum -y install kernel-debug{,-devel}
# debuginfo -y install kernel-debug
# yum -y install kexec-tools crash

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

5 / 12
Console output
Message buffer
Screen, UART, USB debugging dongle
Early console for boot problems
Slowed down output possible
Syslog via Netpoll API
# dmesg
rhgb, quiet
console={ttyS*,uart}
earlyprintk={vga,serial,dbgp}
kernel.printk_delay
kernel.printk_ratelimit
netconsole.ko
g_dbgp.ko
lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

6 / 12
Reading OOPS messages

Processor status
Traceback
# objdump -lSd {vmlinux,module.ko}
# addr2line

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

7 / 12
Memory dumps

Crash dumps
OHCI1394 DMA, can be early too
Hypervisor specific dumps
We can extract console output from them!
crashkernel=auto
# systemctl start kdump.service
# firescope

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

8 / 12
Analyzing dumps

crash

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

9 / 12
Helpful kernel tools
Kernel hacking menu
Lockdep detector
Hung task detector
List corruption detector
Stack protector
Memory poison
CONFIG_DEBUG_*
kernel.sysrq
kernel.*panic*
kernel.prove_locking
kernel.hung_task_timeout_secs

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

10 / 12
Loading our code into kernel
Understand kernel coding
Pick a good book
Linux Kernel Development, Robert Love, Addison-Wesley:
Developer’s Library
Understanding the Linux Kernel, Corbet et al., O’Reilly (available
online)
Try out what do the debugging facilities do!
# make -C /lib/modules/$(uname -r)/build 
SUBDIRS=$PWD modules
# insmod module.ko

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

11 / 12
Thank you for attending!

The source to the slides are embedded in the PDF file. Use pdftk or
Acrobat Reader’s ”attachments” to extract them.

lkundrak@v3.sk

Kernel Debugging

February 19th , 2014

12 / 12

More Related Content

What's hot

Kernel Recipes 2015 - Porting Linux to a new processor architecture
Kernel Recipes 2015 - Porting Linux to a new processor architectureKernel Recipes 2015 - Porting Linux to a new processor architecture
Kernel Recipes 2015 - Porting Linux to a new processor architecture
Anne Nicolas
 
Kernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologiesKernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologies
Anne Nicolas
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
Brendan Gregg
 
bcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challengesbcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challenges
IO Visor Project
 
How to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One ExploitHow to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One Exploit
Jiahong Fang
 
Kernel Recipes 2015: Introduction to Kernel Power Management
Kernel Recipes 2015: Introduction to Kernel Power ManagementKernel Recipes 2015: Introduction to Kernel Power Management
Kernel Recipes 2015: Introduction to Kernel Power Management
Anne Nicolas
 
Make Your Containers Faster: Linux Container Performance Tools
Make Your Containers Faster: Linux Container Performance ToolsMake Your Containers Faster: Linux Container Performance Tools
Make Your Containers Faster: Linux Container Performance Tools
Kernel TLV
 
SystemV vs systemd
SystemV vs systemdSystemV vs systemd
SystemV vs systemd
All Things Open
 
Proxy arp
Proxy arpProxy arp
Proxy arp
Marian Marinov
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Brendan Gregg
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
Brendan Gregg
 
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelKernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Anne Nicolas
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
Satpal Parmar
 
Kernel Recipes 2015 - Hardened kernels for everyone
Kernel Recipes 2015 - Hardened kernels for everyoneKernel Recipes 2015 - Hardened kernels for everyone
Kernel Recipes 2015 - Hardened kernels for everyone
Anne Nicolas
 
Building
BuildingBuilding
Building
Satpal Parmar
 
Linux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene PirogovLinux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene Pirogov
Pivorak MeetUp
 
Crash Dump Analysis 101
Crash Dump Analysis 101Crash Dump Analysis 101
Crash Dump Analysis 101
John Howard
 
Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)
yang firo
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Anne Nicolas
 

What's hot (20)

Kernel Recipes 2015 - Porting Linux to a new processor architecture
Kernel Recipes 2015 - Porting Linux to a new processor architectureKernel Recipes 2015 - Porting Linux to a new processor architecture
Kernel Recipes 2015 - Porting Linux to a new processor architecture
 
Kernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologiesKernel Recipes 2015: Kernel packet capture technologies
Kernel Recipes 2015: Kernel packet capture technologies
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPFUSENIX ATC 2017 Performance Superpowers with Enhanced BPF
USENIX ATC 2017 Performance Superpowers with Enhanced BPF
 
bcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challengesbcc/BPF tools - Strategy, current tools, future challenges
bcc/BPF tools - Strategy, current tools, future challenges
 
How to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One ExploitHow to Root 10 Million Phones with One Exploit
How to Root 10 Million Phones with One Exploit
 
Kernel Recipes 2015: Introduction to Kernel Power Management
Kernel Recipes 2015: Introduction to Kernel Power ManagementKernel Recipes 2015: Introduction to Kernel Power Management
Kernel Recipes 2015: Introduction to Kernel Power Management
 
Make Your Containers Faster: Linux Container Performance Tools
Make Your Containers Faster: Linux Container Performance ToolsMake Your Containers Faster: Linux Container Performance Tools
Make Your Containers Faster: Linux Container Performance Tools
 
SystemV vs systemd
SystemV vs systemdSystemV vs systemd
SystemV vs systemd
 
Proxy arp
Proxy arpProxy arp
Proxy arp
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
 
USENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame GraphsUSENIX ATC 2017: Visualizing Performance with Flame Graphs
USENIX ATC 2017: Visualizing Performance with Flame Graphs
 
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernelKernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
Kernel Recipes 2019 - Hunting and fixing bugs all over the Linux kernel
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
 
Kernel Recipes 2015 - Hardened kernels for everyone
Kernel Recipes 2015 - Hardened kernels for everyoneKernel Recipes 2015 - Hardened kernels for everyone
Kernel Recipes 2015 - Hardened kernels for everyone
 
Building
BuildingBuilding
Building
 
Linux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene PirogovLinux Tracing Superpowers by Eugene Pirogov
Linux Tracing Superpowers by Eugene Pirogov
 
Crash Dump Analysis 101
Crash Dump Analysis 101Crash Dump Analysis 101
Crash Dump Analysis 101
 
Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)Linux kernel debugging(ODP format)
Linux kernel debugging(ODP format)
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
 

Similar to Linux Kernel Debugging Essentials workshop

Introduction to Linux Kernel Development
Introduction to Linux Kernel DevelopmentIntroduction to Linux Kernel Development
Introduction to Linux Kernel Development
Levente Kurusa
 
Web Server Free Bsd
Web Server Free BsdWeb Server Free Bsd
Web Server Free Bsd
Kwanchai Charoennet
 
Lec 10-linux-review
Lec 10-linux-reviewLec 10-linux-review
Lec 10-linux-review
abinaya m
 
NodeMCU with Blynk and Firebase
NodeMCU with Blynk and FirebaseNodeMCU with Blynk and Firebase
NodeMCU with Blynk and Firebase
Eueung Mulyana
 
Basic Linux Internals
Basic Linux InternalsBasic Linux Internals
Basic Linux Internals
mukul bhardwaj
 
Lecture 5 Kernel Development
Lecture 5 Kernel DevelopmentLecture 5 Kernel Development
Lecture 5 Kernel Development
Mohammed Farrag
 
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdf
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdfBasics_of_Kernel_Panic_Hang_and_ Kdump.pdf
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdf
stroganovboris
 
Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)
Bud Siddhisena
 
Linux Module Programming
Linux Module ProgrammingLinux Module Programming
Linux Module Programming
Amir Payberah
 
[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020
Akihiro Suda
 
Talk 160920 @ Cat System Workshop
Talk 160920 @ Cat System WorkshopTalk 160920 @ Cat System Workshop
Talk 160920 @ Cat System Workshop
Quey-Liang Kao
 
Development platform virtualization using qemu
Development platform virtualization using qemuDevelopment platform virtualization using qemu
Development platform virtualization using qemu
Premjith Achemveettil
 
Porting Android
Porting AndroidPorting Android
Porting Android
Opersys inc.
 
Porting Android ABS 2011
Porting Android ABS 2011Porting Android ABS 2011
Porting Android ABS 2011
Opersys inc.
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
Akihiro Suda
 
DCSF19 Hardening Docker daemon with Rootless mode
DCSF19 Hardening Docker daemon with Rootless modeDCSF19 Hardening Docker daemon with Rootless mode
DCSF19 Hardening Docker daemon with Rootless mode
Docker, Inc.
 
Compiling linux kernel and submit first patch
Compiling linux kernel and submit first patchCompiling linux kernel and submit first patch
Compiling linux kernel and submit first patch
Mayur Patil
 
Portin g
Portin gPortin g
Portin g
thebalabe
 
Kdump
KdumpKdump
Linux Kernel - Let's Contribute!
Linux Kernel - Let's Contribute!Linux Kernel - Let's Contribute!
Linux Kernel - Let's Contribute!
Levente Kurusa
 

Similar to Linux Kernel Debugging Essentials workshop (20)

Introduction to Linux Kernel Development
Introduction to Linux Kernel DevelopmentIntroduction to Linux Kernel Development
Introduction to Linux Kernel Development
 
Web Server Free Bsd
Web Server Free BsdWeb Server Free Bsd
Web Server Free Bsd
 
Lec 10-linux-review
Lec 10-linux-reviewLec 10-linux-review
Lec 10-linux-review
 
NodeMCU with Blynk and Firebase
NodeMCU with Blynk and FirebaseNodeMCU with Blynk and Firebase
NodeMCU with Blynk and Firebase
 
Basic Linux Internals
Basic Linux InternalsBasic Linux Internals
Basic Linux Internals
 
Lecture 5 Kernel Development
Lecture 5 Kernel DevelopmentLecture 5 Kernel Development
Lecture 5 Kernel Development
 
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdf
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdfBasics_of_Kernel_Panic_Hang_and_ Kdump.pdf
Basics_of_Kernel_Panic_Hang_and_ Kdump.pdf
 
Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)Recipe of a linux Live CD (archived)
Recipe of a linux Live CD (archived)
 
Linux Module Programming
Linux Module ProgrammingLinux Module Programming
Linux Module Programming
 
[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020
 
Talk 160920 @ Cat System Workshop
Talk 160920 @ Cat System WorkshopTalk 160920 @ Cat System Workshop
Talk 160920 @ Cat System Workshop
 
Development platform virtualization using qemu
Development platform virtualization using qemuDevelopment platform virtualization using qemu
Development platform virtualization using qemu
 
Porting Android
Porting AndroidPorting Android
Porting Android
 
Porting Android ABS 2011
Porting Android ABS 2011Porting Android ABS 2011
Porting Android ABS 2011
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
 
DCSF19 Hardening Docker daemon with Rootless mode
DCSF19 Hardening Docker daemon with Rootless modeDCSF19 Hardening Docker daemon with Rootless mode
DCSF19 Hardening Docker daemon with Rootless mode
 
Compiling linux kernel and submit first patch
Compiling linux kernel and submit first patchCompiling linux kernel and submit first patch
Compiling linux kernel and submit first patch
 
Portin g
Portin gPortin g
Portin g
 
Kdump
KdumpKdump
Kdump
 
Linux Kernel - Let's Contribute!
Linux Kernel - Let's Contribute!Linux Kernel - Let's Contribute!
Linux Kernel - Let's Contribute!
 

More from Lubomir Rintel

Namespaces for Kazimir
Namespaces for KazimirNamespaces for Kazimir
Namespaces for Kazimir
Lubomir Rintel
 
Namespaces in Linux
Namespaces in LinuxNamespaces in Linux
Namespaces in Linux
Lubomir Rintel
 
LinuxAlt 2013: Writing a driver for unknown USB device
LinuxAlt 2013: Writing a driver for unknown USB deviceLinuxAlt 2013: Writing a driver for unknown USB device
LinuxAlt 2013: Writing a driver for unknown USB device
Lubomir Rintel
 
A journey through the years of UNIX and Linux service management
A journey through the years of UNIX and Linux service managementA journey through the years of UNIX and Linux service management
A journey through the years of UNIX and Linux service management
Lubomir Rintel
 
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
Lubomir Rintel
 
Reverse Engineering: Writing a Linux driver for an unknown device
Reverse Engineering: Writing a Linux driver for an unknown deviceReverse Engineering: Writing a Linux driver for an unknown device
Reverse Engineering: Writing a Linux driver for an unknown device
Lubomir Rintel
 
SELinux basics
SELinux basicsSELinux basics
SELinux basics
Lubomir Rintel
 
Brno meetr: Packaging Ruby Gems into RPM
Brno meetr: Packaging Ruby Gems into RPMBrno meetr: Packaging Ruby Gems into RPM
Brno meetr: Packaging Ruby Gems into RPM
Lubomir Rintel
 

More from Lubomir Rintel (8)

Namespaces for Kazimir
Namespaces for KazimirNamespaces for Kazimir
Namespaces for Kazimir
 
Namespaces in Linux
Namespaces in LinuxNamespaces in Linux
Namespaces in Linux
 
LinuxAlt 2013: Writing a driver for unknown USB device
LinuxAlt 2013: Writing a driver for unknown USB deviceLinuxAlt 2013: Writing a driver for unknown USB device
LinuxAlt 2013: Writing a driver for unknown USB device
 
A journey through the years of UNIX and Linux service management
A journey through the years of UNIX and Linux service managementA journey through the years of UNIX and Linux service management
A journey through the years of UNIX and Linux service management
 
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
 
Reverse Engineering: Writing a Linux driver for an unknown device
Reverse Engineering: Writing a Linux driver for an unknown deviceReverse Engineering: Writing a Linux driver for an unknown device
Reverse Engineering: Writing a Linux driver for an unknown device
 
SELinux basics
SELinux basicsSELinux basics
SELinux basics
 
Brno meetr: Packaging Ruby Gems into RPM
Brno meetr: Packaging Ruby Gems into RPMBrno meetr: Packaging Ruby Gems into RPM
Brno meetr: Packaging Ruby Gems into RPM
 

Recently uploaded

How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
SynapseIndia
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...
BookNet Canada
 
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
uuuot
 
Dev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous DiscoveryDev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous Discovery
UiPathCommunity
 
Supercomputing from the Desktop Workstation
Supercomputingfrom the Desktop WorkstationSupercomputingfrom the Desktop Workstation
Supercomputing from the Desktop Workstation
Larry Smarr
 
Artificial Intelligence (AI), Robotics and Computational fluid dynamics
Artificial Intelligence (AI), Robotics and Computational fluid dynamicsArtificial Intelligence (AI), Robotics and Computational fluid dynamics
Artificial Intelligence (AI), Robotics and Computational fluid dynamics
Chintan Kalsariya
 
Running a Go App in Kubernetes: CPU Impacts
Running a Go App in Kubernetes: CPU ImpactsRunning a Go App in Kubernetes: CPU Impacts
Running a Go App in Kubernetes: CPU Impacts
ScyllaDB
 
K2G - Insurtech Innovation EMEA Award 2024
K2G - Insurtech Innovation EMEA Award 2024K2G - Insurtech Innovation EMEA Award 2024
K2G - Insurtech Innovation EMEA Award 2024
The Digital Insurer
 
AI_dev Europe 2024 - From OpenAI to Opensource AI
AI_dev Europe 2024 - From OpenAI to Opensource AIAI_dev Europe 2024 - From OpenAI to Opensource AI
AI_dev Europe 2024 - From OpenAI to Opensource AI
Raphaël Semeteys
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
Aurora Consulting
 
6 Different Types of Printed Circuit Boards.pdf
6 Different Types of Printed Circuit Boards.pdf6 Different Types of Printed Circuit Boards.pdf
6 Different Types of Printed Circuit Boards.pdf
shammikudrat
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
John Sterrett
 
Insurwave - Insurtech Innovation Award 2024
Insurwave - Insurtech Innovation Award 2024Insurwave - Insurtech Innovation Award 2024
Insurwave - Insurtech Innovation Award 2024
The Digital Insurer
 
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
amitchopra0215
 
MYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
MYIR Product Brochure - A Global Provider of Embedded SOMs & SolutionsMYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
MYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
Linda Zhang
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
Lessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien RiouxLessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien Rioux
crioux1
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
shanthidl1
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
Matthew Sinclair
 

Recently uploaded (20)

How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...Transcript: Details of description part II: Describing images in practice - T...
Transcript: Details of description part II: Describing images in practice - T...
 
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
一比一原版(msvu毕业证书)圣文森山大学毕业证如何办理
 
Dev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous DiscoveryDev Dives: Mining your data with AI-powered Continuous Discovery
Dev Dives: Mining your data with AI-powered Continuous Discovery
 
Supercomputing from the Desktop Workstation
Supercomputingfrom the Desktop WorkstationSupercomputingfrom the Desktop Workstation
Supercomputing from the Desktop Workstation
 
Artificial Intelligence (AI), Robotics and Computational fluid dynamics
Artificial Intelligence (AI), Robotics and Computational fluid dynamicsArtificial Intelligence (AI), Robotics and Computational fluid dynamics
Artificial Intelligence (AI), Robotics and Computational fluid dynamics
 
Running a Go App in Kubernetes: CPU Impacts
Running a Go App in Kubernetes: CPU ImpactsRunning a Go App in Kubernetes: CPU Impacts
Running a Go App in Kubernetes: CPU Impacts
 
K2G - Insurtech Innovation EMEA Award 2024
K2G - Insurtech Innovation EMEA Award 2024K2G - Insurtech Innovation EMEA Award 2024
K2G - Insurtech Innovation EMEA Award 2024
 
AI_dev Europe 2024 - From OpenAI to Opensource AI
AI_dev Europe 2024 - From OpenAI to Opensource AIAI_dev Europe 2024 - From OpenAI to Opensource AI
AI_dev Europe 2024 - From OpenAI to Opensource AI
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
 
6 Different Types of Printed Circuit Boards.pdf
6 Different Types of Printed Circuit Boards.pdf6 Different Types of Printed Circuit Boards.pdf
6 Different Types of Printed Circuit Boards.pdf
 
Database Management Myths for Developers
Database Management Myths for DevelopersDatabase Management Myths for Developers
Database Management Myths for Developers
 
Insurwave - Insurtech Innovation Award 2024
Insurwave - Insurtech Innovation Award 2024Insurwave - Insurtech Innovation Award 2024
Insurwave - Insurtech Innovation Award 2024
 
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
@Call @Girls Pune 0000000000 Riya Khan Beautiful Girl any Time
 
MYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
MYIR Product Brochure - A Global Provider of Embedded SOMs & SolutionsMYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
MYIR Product Brochure - A Global Provider of Embedded SOMs & Solutions
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
 
Lessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien RiouxLessons Of Binary Analysis - Christien Rioux
Lessons Of Binary Analysis - Christien Rioux
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
 

Linux Kernel Debugging Essentials workshop

  • 1. Linux Kernel Debugging Essentials Lubomir Rintel base48 Hackerspace, Brno http://base48.cz/ February 19th , 2014 lkundrak@v3.sk Kernel Debugging February 19th , 2014 1 / 12
  • 2. Plan Understanding the way kernel can fail Introduction to infrastructure Obtaining & unserstanding the traces Heplful in-kernel mechanisms Building and trying the replacement code lkundrak@v3.sk Kernel Debugging February 19th , 2014 2 / 12
  • 3. Kernel failures Warning, possibly non-fatal: OOPS A fatal error: Panic Usually accompained by a state dump Result of a bug or a hardware failure # echo c >/proc/sysrq-trigger lkundrak@v3.sk Kernel Debugging February 19th , 2014 3 / 12
  • 4. Kernel configuration Vanilla kernels Distribution kernels Configuration options (build-time) Runtime configuration (sysctls) Boot time configuration (kernel parameters) Documentation/kernel-parameters.txt lkundrak@v3.sk Kernel Debugging February 19th , 2014 4 / 12
  • 5. Lab setup For details & non-Fedora setup, see the hackerspace wiki # yum -y install kernel-debug{,-devel} # debuginfo -y install kernel-debug # yum -y install kexec-tools crash lkundrak@v3.sk Kernel Debugging February 19th , 2014 5 / 12
  • 6. Console output Message buffer Screen, UART, USB debugging dongle Early console for boot problems Slowed down output possible Syslog via Netpoll API # dmesg rhgb, quiet console={ttyS*,uart} earlyprintk={vga,serial,dbgp} kernel.printk_delay kernel.printk_ratelimit netconsole.ko g_dbgp.ko lkundrak@v3.sk Kernel Debugging February 19th , 2014 6 / 12
  • 7. Reading OOPS messages Processor status Traceback # objdump -lSd {vmlinux,module.ko} # addr2line lkundrak@v3.sk Kernel Debugging February 19th , 2014 7 / 12
  • 8. Memory dumps Crash dumps OHCI1394 DMA, can be early too Hypervisor specific dumps We can extract console output from them! crashkernel=auto # systemctl start kdump.service # firescope lkundrak@v3.sk Kernel Debugging February 19th , 2014 8 / 12
  • 10. Helpful kernel tools Kernel hacking menu Lockdep detector Hung task detector List corruption detector Stack protector Memory poison CONFIG_DEBUG_* kernel.sysrq kernel.*panic* kernel.prove_locking kernel.hung_task_timeout_secs lkundrak@v3.sk Kernel Debugging February 19th , 2014 10 / 12
  • 11. Loading our code into kernel Understand kernel coding Pick a good book Linux Kernel Development, Robert Love, Addison-Wesley: Developer’s Library Understanding the Linux Kernel, Corbet et al., O’Reilly (available online) Try out what do the debugging facilities do! # make -C /lib/modules/$(uname -r)/build SUBDIRS=$PWD modules # insmod module.ko lkundrak@v3.sk Kernel Debugging February 19th , 2014 11 / 12
  • 12. Thank you for attending! The source to the slides are embedded in the PDF file. Use pdftk or Acrobat Reader’s ”attachments” to extract them. lkundrak@v3.sk Kernel Debugging February 19th , 2014 12 / 12