Linux Kernel Tutorials

Linux Kernel and Driver Development Training
Linux Kernel and Driver

Development Training
Embedded Linux
Experts
free electrons
Copyright 2004-2017, Free Electrons.
Creative Commons BY-SA 3.0 license.
Latest update: January 1, 2017.
Document updates and sources:
http://free- electrons.com/doc/training/linuxkernel
Corrections, suggestions, contributions and translations are welcome!
Send them to feedback@free-electrons.com
free electrons - Embedded Linux, kernel, drivers - Development, consulting, training and support. http://free-electrons.com
1/475
Rights to copy
Copyright 2004-2017, Free Electrons
License: Creative Commons Attribution - Share Alike 3.0
http://creativecommons.org/licenses/by-sa/3.0/legalcode
You are free:
to copy, distribute, display, and perform the work
to make derivative works
to make commercial use of the work
Under the following conditions:
Attribution. You must give the original author credit.
Share Alike. If you alter, transform, or build upon this work, you may distribute
the resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of
this work.
Any of these conditions can be waived if you get permission from the copyright
holder.
Your fair use and other rights are in no way aected by the above.
2/475
Hyperlinks in the document

There are many hyperlinks in the document
Regular hyperlinks:
http://kernel.org/
Kernel documentation links:

Documentation/kmemcheck.txt
Links to kernel source les and directories:

drivers/input
include/linux/fb.h
Links to the declarations, denitions and instances of kernel

symbols (functions, types, data, structures):
platform_get_irq()
GFP_KERNEL
struct file_operations
3/475
Free Electrons at a glance
Engineering company created in 2004

(not a training company!)
Locations: Orange, Toulouse, Lyon (France)
Serving customers all around the world
Head count: 12
Only Free Software enthusiasts!
Focus: Embedded Linux, Linux kernel Free Software / Open

Source for embedded and real-time systems.
Activities: development, training, consulting, technical

support.
Added value: get the best of the user and development

community and the resources it oers.
4/475
Free Electrons on-line resources
All our training materials:

http://free-electrons.com/docs/
Technical blog:
http://free-electrons.com/blog/
Quarterly newsletter:
http://lists.freeelectrons.com/mailman/listinfo/newsletter
News and discussions (Google +):
https://plus.google.com/+FreeElectronsDevelopers
News and discussions (LinkedIn):
http://linkedin.com/groups/Free-Electrons-4501089
Quick news (Twitter):
http://twitter.com/free_electrons
Linux Cross Reference - browse Linux kernel sources on-line:
http://lxr.free-electrons.com
5/475
Generic course information
Generic course
information
Embedded Linux
Experts
free electrons
6/475
Hardware used in this training session

BeagleBone Black, from CircuitCo
Texas Instruments AM335x (ARM Cortex-A8
CPU)
SoC with 3D acceleration, additional
processors (PRUs) and lots of peripherals.

512 MB of RAM
4 GB of on-board eMMC storage
Ethernet, USB host and USB device,
microSD, micro HDMI

2 x 46 pins headers, with access to many
expansion buses (I2C, SPI, UART and more)

A huge number of expansion boards, called
capes. See http://beagleboardtoys.com/.
7/475
Do not damage your BeagleBone Black!
Do not remove power abruptly:
Boards components have been damaged by removing the

power or USB cable in an abrupt way, not leaving the PMIC
the time to switch o the components in a clean way. See
http://bit.ly/1FWHNZi
Reboot (reboot) or shutdown (halt) the board in software
when Linux is running.
You can also press the RESET button to reset and reboot.
When there is no software way, you can also switch o the
board by pressing the POWER button for 8 seconds.
Do not leave your board powered on a metallic surface (like a

laptop with a metal nish).
8/475
Shopping list: hardware for this course
BeagleBone Black - Multiple distributors:
See http://beagleboard.org/Products/
Nintendo Nunchuck with UEXT connector:
Olimex: http://j.mp/1dTYLfs
Breadboard jumper wires - Male ends:
Olimex: http://j.mp/IUaBsr
USB Serial Cable - Male ends:
Olimex: http://j.mp/1eUuY2K
USB Serial Cable - Female ends:
Olimex: http://j.mp/18Hk8yF
Note that both USB serial cables are the same.
Only the gender of their connector changes.
9/475
Participate!
During the lectures...
Don't hesitate to ask questions. Other people in the audience

may have similar questions too.
This helps the trainer to detect any explanation that wasn't

clear or detailed enough.
Don't hesitate to share your experience, for example to

compare Linux / Android with other operating systems used
in your company.
Your point of view is most valuable, because it can be similar

to your colleagues' and dierent from the trainer's.
Your participation can make our session more interactive and

make the topics easier to learn.
10/475
Practical lab guidelines

During practical labs...
We cannot support more than 8 workstations at once (each

with its board and equipment). Having more would make the
whole class progress slower, compromising the coverage of the
whole training agenda (exception for public sessions: up to 10
people).
So, if you are more than 8 participants, please form up to 8

working groups.
Open the electronic copy of your lecture materials, and use it

throughout the practical labs to nd the slides you need again.
Don't hesitate to copy and paste commands from the PDF

slides and labs.
11/475
Advise: write down your commands!

During practical labs, write down all your commands in a text le.
You can save a lot of time re-using

commands in later labs.
This helps to replay your work if

you make signicant mistakes.
You build a reference to remember

commands in the long run.
That's particular useful to keep
kernel command line settings that
you used earlier.
Also useful to get help from the
instructor, showing the commands
that you run.
Lab commands
Cross-compiling kernel:
export ARCH=arm
export CROSS_COMPILE=arm-linuxmake sama5_defcong
Booting kernel through tftp:
setenv bootargs console=ttyS0 root=/dev/nfs
setenv bootcmd tftp 0x21000000 zImage; tftp
0x22000000 dtb; bootz 0x21000000 - 0x2200...
Making ubifs images:
mkfs.ubifs -d rootfs -o root.ubifs -e 124KiB
-m 2048 -c 1024
Encountered issues:
Restart NFS server after editing /etc/exports!
gedit ~/lab-history.txt
12/475
Cooperate!
As in the Free Software and Open Source community, cooperation

during practical labs is valuable in this training session:
If you complete your labs before other people, don't hesitate

to help other people and investigate the issues they face. The
faster we progress as a group, the more time we have to
explore extra topics.
Explain what you understood to other participants when

needed. It also helps to consolidate your knowledge.
Don't hesitate to report potential bugs to your instructor.
Don't hesitate to look for solutions on the Internet as well.
13/475
Command memento sheet
This memento sheet gives

command examples for the most
typical needs (looking for les,
extracting a tar archive...)
It saves us 1 day of UNIX / Linux

command line training.
Our best tip: in the command line

shell, always hit the Tab key to
complete command names and le
paths. This avoids 95% of typing
mistakes.
Get an electronic copy on

http://free-electrons.com/
doc/training/embeddedlinux/command_memento.pdf
14/475
vi basic commands
The vi editor is very useful to

make quick changes to les in an
embedded target.
Though not very user friendly at

rst, vi is very powerful and its
main 15 commands are easy to
learn and are sucient for 99% of
everyone's needs!
Get an electronic copy on

doc/training/embeddedlinux/vi_memento.pdf
You can also take the quick tutorial

by running vimtutor. This is a
worthy investment!
15/475
Practical lab - Training Setup
Prepare your lab environment
Download and extract the lab

archive
16/475
Linux Kernel Introduction
Linux Kernel
Introduction
Embedded Linux
Experts
free electrons
17/475
Linux Kernel Introduction
Linux features
18/475
History
The Linux kernel is one component of a system, which also

requires libraries and applications to provide features to end
users.
The Linux kernel was created as a hobby in 1991 by a Finnish
student, Linus Torvalds.
Linux quickly started to be used as the kernel for free software

operating systems
Linus Torvalds has been able to create a large and dynamic

developer and user community around Linux.
Nowadays, more than one thousand people contribute to each

kernel release, individuals or companies big and small.
19/475
Linux kernel key features
Portability and hardware

support. Runs on most
architectures.
Security. It can't hide its

aws. Its code is reviewed
by many experts.
Scalability. Can run on

super computers as well as
on tiny devices (4 MB of
RAM is enough).
Stability and reliability.
Modularity. Can include

only what a system needs
even at run time.
Compliance to standards
and interoperability.
Exhaustive networking
support.
Easy to program. You can

learn from existing code.
Many useful resources on
the net.
20/475
Linux kernel in the system
21/475
Linux kernel main roles
Manage all the hardware resources: CPU, memory, I/O.
Provide a set of portable, architecture and hardware

independent APIs to allow user space applications and
libraries to use the hardware resources.
Handle concurrent accesses and usage of hardware
resources from dierent applications.
Example: a single network interface is used by multiple user

space applications through various network connections. The
kernel is responsible to ``multiplex'' the hardware resource.
22/475
System calls
The main interface between the kernel and user space is the
set of system calls
About 300 system calls that provide the main kernel services
File and device operations, networking operations,

inter-process communication, process management, memory
mapping, timers, threads, synchronization primitives, etc.
This interface is stable over time: only new system calls can
be added by the kernel developers
This system call interface is wrapped by the C library, and

user space applications usually never make a system call
directly but rather use the corresponding C library function
23/475
Pseudo lesystems
Linux makes system and kernel information available in user

space through pseudo lesystems, sometimes also called
virtual lesystems
Pseudo lesystems allow applications to see directories and

les that do not exist on any real storage: they are created
and updated on the y by the kernel
The two most important pseudo lesystems are
proc, usually mounted on /proc:

Operating system related information (processes, memory
management parameters...)
sysfs, usually mounted on /sys:
Representation of the system as a set of devices and buses.
Information about these devices.
24/475
Inside the Linux kernel
25/475
Supported hardware architectures
See the arch/ directory in the kernel sources
Minimum: 32 bit processors, with or without MMU, and gcc

support
32 bit architectures (arch/ subdirectories)

Examples: arm, avr32, blackfin, c6x, m68k, microblaze,
score, um
64 bit architectures:
Examples: alpha, arm64, ia64, tile
32/64 bit architectures

Examples: mips, powerpc, sh, sparc, x86
Find details in kernel sources: arch/<arch>/Kconfig,

arch/<arch>/README, or Documentation/<arch>/
26/475
Embedded Linux Kernel Usage
Embedded Linux
Kernel Usage
Embedded Linux
Experts
free electrons
27/475
Embedded Linux Kernel Usage
Linux kernel sources
28/475
Location of kernel sources
The ocial versions of the Linux kernel, as released by Linus

Torvalds, are available at http://www.kernel.org
Many chip vendors supply their own kernel sources
These versions follow the development model of the kernel

However, they may not contain the latest development from a
specic area yet. Some features in development might not be
ready for mainline inclusion yet.
Focusing on hardware support rst
Can have a very important delta with mainline Linux
Useful only when mainline hasn't caught up yet.
Many kernel sub-communities maintain their own kernel, with

usually newer but less stable features
Architecture communities (ARM, MIPS, PowerPC, etc.),

device drivers communities (I2C, SPI, USB, PCI, network,
etc.), other communities (real-time, etc.)
No ocial releases, only development trees are available.
29/475
Getting Linux sources
The kernel sources are available from

http://kernel.org/pub/linux/kernel as full tarballs
(complete kernel sources) and patches (dierences between
two kernel versions).
However, more and more people use the git version control
system. Absolutely needed for kernel development!
Fetch the entire kernel sources and history

git clone git://git.kernel.org/pub/scm/linux/kernel/
git/torvalds/linux.git
Create a branch that starts at a specic stable version
git checkout -b <name-of-branch> v3.11
Web interface available at http://git.kernel.org/cgit/
linux/kernel/git/torvalds/linux.git/tree/.
Read more about Git at http://git-scm.com/
30/475
Linux kernel size (1)
Linux 4.6 sources:

Raw size: 730 MB (53,600 les, approx 21,400,000 lines)
gzip compressed tar archive: 130 MB
xz compressed tar archive: 85 MB
Minimum Linux 3.17 compiled kernel size, booting on the

ARM Versatile board (hard drive on PCI, ext2 lesystem, ELF
executable support, framebuer console and input devices):
876 KB (compressed), 2.3 MB (raw)
Why are these sources so big?

Because they include thousands of device drivers, many
network protocols, support many architectures and
lesystems...
The Linux core (scheduler, memory management...) is pretty

small!
31/475
Linux kernel size (2)
As of kernel version 4.6 (in lines).
drivers/: 57.0%
arch/: 16.3%
fs/: 5.5%
sound/: 4.4%
net/: 4.3%
include/: 3.5%
Documentation/: 2.8%
tools/: 1.3%
kernel/: 1.2%
firmware/: 0.6%
lib/: 0.5%
mm/: 0.5%
scripts/: 0.4%
crypto/: 0.4%
security/: 0.3%
block/: 0.1%
...
32/475
Practical lab - Downloading kernel source code
Clone the mainline Linux source

tree with git
33/475
Kernel Source Code
Kernel Source
Code
Embedded Linux
Experts
free electrons
34/475
Kernel Source Code
Linux Code and Device Drivers
35/475
Programming language
Implemented in C like all Unix systems. (C was created to

implement the rst Unix systems)
A little Assembly is used too:
No C++ used, see http://www.tux.org/lkml/#s15-3

All the code compiled with gcc
CPU and machine initialization, exceptions

Critical library routines.
Many gcc specic extensions used in the kernel code, any

ANSI C compiler will not compile the kernel
See https://gcc.gnu.org/onlinedocs/gcc-6.2.0/gcc/CExtensions.html
Ongoing work to compile the kernel with the LLVM compiler.
36/475
No C library
The kernel has to be standalone and can't use user space

code.
Architectural reason: user space is implemented on top of

kernel services, not the opposite.
Technical reason: the kernel is on its own during the boot up

phase, before it has accessed a root lesystem.
Hence, kernel code has to supply its own library

implementations (string utilities, cryptography,
uncompression...)
So, you can't use standard C library functions in kernel code.

(printf(), memset(), malloc(),...).
Fortunately, the kernel provides similar C functions for your

convenience, like printk(), memset(), kmalloc(), ...
37/475
Portability
The Linux kernel code is designed to be portable
All code outside arch/ should be portable

To this aim, the kernel provides macros and functions to
abstract the architecture specic details
Endianness
cpu_to_be32()
cpu_to_le32()
be32_to_cpu()
le32_to_cpu()
I/O memory access

Memory barriers to provide ordering guarantees if needed
DMA API to ush and invalidate caches if needed
38/475
No oating point computation
Never use oating point numbers in kernel code. Your code

may be run on a processor without a oating point unit (like
on certain ARM CPUs).
Don't be confused with oating point related conguration
options
They are related to the emulation of oating point operation

performed by the user space applications, triggering an
exception into the kernel.
Using soft-oat, i.e. emulation in user space, is however
recommended for performance reasons.
39/475
No stable Linux internal API
The internal kernel API to implement kernel code can undergo

changes between two releases.
In-tree drivers are updated by the developer proposing the API

change: works great for mainline code.
An out-of-tree driver compiled for a given version may no

longer compile or work on a more recent one.
See Documentation/stable_api_nonsense.txt in kernel

sources for reasons why.
Of course, the kernel to user space API does not change

(system calls, /proc, /sys), as it would break existing
programs.
40/475
Kernel memory constraints
No memory protection
Accessing illegal memory locations result in (often fatal)

kernel oopses.
Fixed size stack (8 or 4 KB). Unlike in user space, there's no

way to make it grow.
Kernel memory can't be swapped out (for the same reasons).
41/475
Linux kernel licensing constraints
The Linux kernel is licensed under the GNU General Public

License version 2
However, when the software is redistributed, either modied

or unmodied, the GPL requires that you redistribute the
software under the same license, with the source code
This license gives you the right to use, study, modify and share
the software freely
If modications are made to the Linux kernel (for example to

adapt it to your hardware), it is a derivative work of the kernel,
and therefore must be released under GPLv2
The validity of the GPL on this point has already been veried
in courts
However, you're only required to do so
At the time the device starts to be distributed

To your customers, not to the entire world
42/475
Proprietary code and the kernel
It is illegal to distribute a binary kernel that includes statically

compiled proprietary drivers
The kernel modules are a gray area: are they derived works of
the kernel or not?
The general opinion of the kernel community is that

proprietary drivers are bad: http://j.mp/fbyuuH
From a legal point of view, each driver is probably a dierent
case
Is it really useful to keep your drivers secret?
There are some examples of proprietary drivers, like the Nvidia

graphics drivers
They use a wrapper between the driver and the kernel

Unclear whether it makes it legal or not
43/475
Advantages of GPL drivers
You don't have to write your driver from scratch. You can
reuse code from similar free software drivers.
You could get free community contributions, support, code

review and testing, though this generally only happens with
code submitted for the mainline kernel.
Your drivers can be freely and easily shipped by others (for

example by Linux distributions or embedded Linux build
systems).
Pre-compiled drivers work with only one kernel version and

one specic conguration, making life dicult for users who
want to change the kernel version.
Legal certainty, you are sure that a GPL driver is ne from a

legal point of view.
44/475
Advantages of in-tree kernel drivers
Once your sources are accepted in the mainline tree...
There are many more people reviewing your code, allowing to

get cost-free security xes and improvements.
You can also get changes from people modifying internal

kernel APIs.
Accessing your code is easier for users.
This will for sure reduce your maintenance and support work
45/475
User space device drivers 1/3
In some cases, it is possible to implement device drivers in

user space!
Can be used when
The kernel provides a mechanism that allows user space

applications to directly access the hardware.
There is no need to leverage an existing kernel subsystem such
as the networking stack or lesystems.
There is no need for the kernel to act as a ``multiplexer'' for
the device: only one application accesses the device.
46/475
Possibilities for user space device drivers:
USB with libusb, http://www.libusb.org/

SPI with spidev, Documentation/spi/spidev
I2C with i2cdev, Documentation/i2c/dev-interface
Memory-mapped devices with UIO, including interrupt
handling, DocBook/uio-howto/
Certain classes of devices (printers, scanners, 2D/3D graphics

acceleration) are typically handled partly in kernel space,
partly in user space.
47/475
Advantages
No need for kernel coding skills. Easier to reuse code between

devices.
Drivers can be written in any language, even Perl!
Drivers can be kept proprietary.
Driver code can be killed and debugged. Cannot crash the
kernel.
Can be swapped out (kernel code cannot be).
Can use oating-point computation.
Less in-kernel complexity.
Potentially higher performance, especially for memory-mapped
devices, thanks to the avoidance of system calls.
Drawbacks
Less straightforward to handle interrupts.

Increased interrupt latency vs. kernel code.
48/475
Kernel Source Code
Linux sources
49/475
Linux sources structure 1/5
arch/<ARCH>
block/
Linux copying conditions (GNU GPL)
CREDITS
Block layer core
COPYING
Architecture specic code

arch/<ARCH>/mach-<machine>, machine/board specic code
arch/<ARCH>/include/asm, architecture-specic headers
arch/<ARCH>/boot/dts, Device Tree source les, for some
architectures
Linux main contributors
crypto/
Cryptographic libraries
50/475
Documentation/
drivers/
Filesystems (fs/ext4/, etc.)
include/
Legacy: rmware images extracted from old drivers
fs/
All device drivers except sound ones (usb, pci...)
firmware/
Kernel documentation sources

Also available on https://www.kernel.org/doc/
(includes functions prototypes and comments extracted from
source code).
Kernel headers
include/linux/
Linux kernel core headers
51/475
include/uapi/
init/
Top level description le for conguration parameters
kernel/
Part of the kernel build system
Kconfig
Code used for process communication
Kbuild
Linux initialization (including init/main.c)
ipc/
User space API headers
Linux kernel core (very small!)
lib/
Misc library routines (zlib, crc32...)
52/475
MAINTAINERS
Makefile
Overview and building instructions
REPORTING-BUGS
Network support code (not drivers)
README
Memory management code (small too!)
net/
Top Linux Makele (sets arch and version)
mm/
Maintainers of each kernel part. Very useful!
Bug report instructions
samples/
Sample code (markers, kprobes, kobjects...)
53/475
scripts/
security/
Code for various user space tools (mostly C)
usr/
Sound support code and drivers
tools/
Security model implementations (SELinux...)
sound/
Executables for internal or external use
Code to generate an initramfs cpio archive
virt/
Virtualization support (KVM)
54/475
Kernel Source Code
Kernel source management tools
55/475
Cscope
Tool to browse source code (mainly C, but also C++ or Java)
Supports huge projects like the Linux kernel. Typically takes

less than 1 min. to index the whole Linux sources.
In Linux kernel sources, two ways of running it:
cscope -Rk
All les for all architectures at once
make cscope
cscope -d cscope.out
Only les for your current architecture
Allows searching for a symbol, a denition, functions, strings,

les, etc.
Integration with editors like vim and emacs.
Dedicated graphical front-end: KScope
http://cscope.sourceforge.net/
56/475
Cscope screenshot
[Tab]: move the cursor between search results and commands

[Ctrl] [D]: exit cscope
57/475
LXR: Linux Cross Reference
Generic source indexing tool and code browser
Web server based, very easy and fast to use
Very easy to nd the declaration, implementation or usage of

symbols
Supports C and C++
Supports huge code projects such as the Linux kernel (431

MB of source code in version 3.0).
Takes a little time and patience to setup (conguration,

indexing, web server conguration)
You don't need to set up LXR by yourself. Use our

http://lxr.free-electrons.com server!
http://sourceforge.net/projects/lxr
58/475
LXR screenshot
59/475
Practical lab - Kernel Source Code - Exploring
Explore kernel sources manually
Use automated tools to explore the

source code
60/475
Kernel Source Code
Kernel conguration
61/475
Kernel conguration and build system
The kernel conguration and build system is based on

multiple Makeles
One only interacts with the main Makefile, present at the

top directory of the kernel source tree
Interaction takes place
using the make tool, which parses the Makele

through various targets, dening which action should be done
(conguration, compilation, installation, etc.). Run make help
to see all available targets.
Example
cd linux-3.6.x/
make <target>
62/475
Kernel conguration (1)
The kernel contains thousands of device drivers, lesystem

drivers, network protocols and other congurable items
Thousands of options are available, that are used to

selectively compile parts of the kernel source code
The kernel conguration is the process of dening the set of

options with which you want your kernel to be compiled
The set of options depends
On your hardware (for device drivers, etc.)

On the capabilities you would like to give to your kernel
(network capabilities, lesystems, real-time, etc.)
63/475
Kernel conguration (2)
The conguration is stored in the .config le at the root of

kernel sources
As options have dependencies, typically never edited by hand,

but through graphical or text interfaces:
Simple text le, key=value style
make xconfig, make gconfig (graphical)

make menuconfig, make nconfig (text)
You can switch from one to another, they all load/save the
same .config le, and show the same set of options
To modify a kernel in a GNU/Linux distribution: the

conguration les are usually released in /boot/, together
with kernel images: /boot/config-3.2.0-31-generic
64/475
Kernel or module?
The kernel image is a single le, resulting from the linking

of all object les that correspond to features enabled in the
conguration
This is the le that gets loaded in memory by the bootloader

All included features are therefore available as soon as the
kernel starts, at a time where no lesystem exists
Some features (device drivers, lesystems, etc.) can however

be compiled as modules
These are plugins that can be loaded/unloaded dynamically to

add/remove features to the kernel
Each module is stored as a separate le in the lesystem,
and therefore access to a lesystem is mandatory to use
modules
This is not possible in the early boot procedure of the kernel,
because no lesystem is available
65/475
Kernel option types
There are dierent types of options

bool options, they are either
true (to include the feature in the kernel) or

false (to exclude the feature from the kernel)
tristate options, they are either
true (to include the feature in the kernel image) or

module (to include the feature as a kernel module) or
false (to exclude the feature)
int options, to specify integer values
hex options, to specify hexadecimal values
string options, to specify string values
66/475
Kernel option dependencies
There are dependencies between kernel options
For example, enabling a network driver requires the network

stack to be enabled
Two types of dependencies
depends on dependencies. In this case, option A that depends

on option B is not visible until option B is enabled
select dependencies. In this case, with option A depending
on option B, when option A is enabled, option B is
automatically enabled
make xconfig allows to see all options, even the ones that
cannot be selected because of missing dependencies. In this
case, they are displayed in gray.
67/475
make xcong
make xconfig
The most common graphical interface to congure the kernel.
Make sure you read

help -> introduction: useful options!
File browser: easier to load conguration les
Search interface to look for parameters
Required Debian / Ubuntu packages: qt5-default g++

pkg-config
68/475
make xcong screenshot
69/475
make xcong search interface

Looks for a keyword in the parameter name. Allows to select or
unselect found parameters.
70/475
Kernel conguration options
71/475
Corresponding .cong le excerpt

Options are grouped by sections and are prexed with CONFIG_.
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
CONFIG_NTFS_RW=y
72/475
make gcong
make gconfig
GTK based graphical

conguration interface.
Functionality similar to that
of make xconfig.
Just lacking a search

functionality.
Required Debian packages:

libglade2-dev
73/475
make menucong
make menuconfig
Useful when no graphics are

available. Pretty convenient
too!
Same interface found in

other tools: BusyBox,
Buildroot...

libncurses-dev
74/475
make ncong
make nconfig
A newer, similar text

interface
More user friendly (for

example, easier to access
help information).

libncurses-dev
75/475
make oldcong
make oldconfig
Needed very often!
Useful to upgrade a .config le from an earlier kernel release
Issues warnings for conguration parameters that no longer

exist in the new kernel.
Asks for values for new parameters (while xconfig and

menuconfig silently set default values for new parameters).
If you edit a .config le by hand, it's strongly recommended to

run make oldconfig afterwards!
76/475
Undoing conguration changes
A frequent problem:
After changing several kernel conguration settings, your

kernel no longer works.
If you don't remember all the changes you made, you can get
back to your previous conguration:
$ cp .config.old .config
All the conguration interfaces of the kernel (xconfig,

menuconfig, oldconfig...) keep this .config.old backup
copy.
77/475
Conguration per architecture
The set of conguration options is architecture dependent
Some conguration options are very architecture-specic

Most of the conguration options (global kernel options,
network subsystem, lesystems, most of the device drivers) are
visible in all architectures.
By default, the kernel build system assumes that the kernel is

being built for the host architecture, i.e. native compilation
The architecture is not dened inside the conguration, but at

a higher level
We will see later how to override this behaviour, to allow the

conguration of kernels for a dierent architecture
78/475
Kernel Source Code
Compiling and installing the kernel

for the host system
79/475
Kernel compilation
make
in the main kernel source directory

Remember to run multiple jobs in parallel if you have multiple
CPU cores. Example: make -j 4
No need to run as root!
Generates
vmlinux, the raw uncompressed kernel image, in the ELF

format, useful for debugging purposes, but cannot be booted
arch/<arch>/boot/*Image, the nal, usually compressed,
kernel image that can be booted
bzImage for x86, zImage for ARM, vmImage.gz for Blackn,

etc.
arch/<arch>/boot/dts/*.dtb, compiled Device Tree les (on

some architectures)
All kernel modules, spread over the kernel source tree, as .ko
(Kernel Object) les.
80/475
Kernel installation
make install
Installs
Does the installation for the host system by default, so needs

to be run as root. Generally not used when compiling for an
embedded system, as it installs les on the development
workstation.
/boot/vmlinuz-<version>
Compressed kernel image. Same as the one in
arch/<arch>/boot
/boot/System.map-<version>
Stores kernel symbol addresses
/boot/config-<version>
Kernel conguration for this version
Typically re-runs the bootloader conguration utility to take

the new kernel into account.
81/475
Module installation
make modules_install
Does the installation for the host system by default, so needs

to be run as root
Installs all modules in /lib/modules/<version>/
kernel/
Module .ko (Kernel Object) les, in the same directory
structure as in the sources.
modules.alias
Module aliases for module loading utilities. Example line:
alias sound-service-?-0 snd_mixer_oss
modules.dep, modules.dep.bin (binary hashed)
Module dependencies
modules.symbols, modules.symbols.bin (binary hashed)
Tells which module a given symbol belongs to.
82/475
Kernel cleanup targets
Clean-up generated les (to force

re-compilation):
make clean
Remove all generated les. Needed when

switching from one architecture to another.
Caution: it also removes your .config le!
make mrproper
Also remove editor backup and patch reject les

(mainly to generate patches):
make distclean
If you are in a git tree, remove all les not

tracked (and ignored) by git:
git clean -fdx
83/475
Kernel Source Code
Cross-compiling the kernel
84/475
Cross-compiling the kernel
When you compile a Linux kernel for another CPU architecture
Much faster than compiling natively, when the target system

is much slower than your GNU/Linux workstation.
Much easier as development tools for your GNU/Linux

workstation are much easier to nd.
To make the dierence with a native compiler, cross-compiler

executables are prexed by the name of the target system,
architecture and sometimes library. Examples:
mips-linux-gcc, the prex is mips-linuxarm-linux-gnueabi-gcc, the prex is arm-linux-gnueabi-
85/475
Specifying cross-compilation (1)
The CPU architecture and cross-compiler prex are dened through

the ARCH and CROSS_COMPILE variables in the toplevel Makele.
ARCH is the name of the architecture. It is dened by the

name of the subdirectory in arch/ in the kernel sources
Example: arm if you want to compile a kernel for the arm

architecture.
CROSS_COMPILE is the prex of the cross compilation tools
Example: arm-linux- if your compiler is arm-linux-gcc
86/475
Specifying cross-compilation (2)

Two solutions to dene ARCH and CROSS_COMPILE:
Pass ARCH and CROSS_COMPILE on the make command line:

make ARCH=arm CROSS_COMPILE=arm-linux- ...
Drawback: it is easy to forget to pass these variables when
you run any make command, causing your build and
conguration to be screwed up.
Dene ARCH and CROSS_COMPILE as environment variables:

export ARCH=arm
export CROSS_COMPILE=arm-linuxDrawback: it only works inside the current shell or terminal.
You could put these settings in a le that you source every
time you start working on the project. If you only work on a
single architecture with always the same toolchain, you could
even put these settings in your ~/.bashrc le to make them
permanent and visible from any terminal.
87/475
Predened conguration les
Default conguration les available, per board or per-CPU

family
They are stored in arch/<arch>/configs/, and are just

minimal .config les
This is the most common way of conguring a kernel for
embedded platforms
Run make help to nd if one is available for your platform
To load a default conguration le, just run

make acme_defconfig
This will overwrite your existing .config le!
To create your own default conguration le
make savedefconfig, to create a minimal conguration le

mv defconfig arch/<arch>/configs/myown_defconfig
88/475
Conguring the kernel
After loading a default conguration le, you can adjust the

conguration to your needs with the normal xconfig,
gconfig or menuconfig interfaces
As the architecture is dierent from your host architecture
Some options will be dierent from the native conguration

(processor and architecture specic options, specic drivers,
etc.)
Many options will be identical (lesystems, network protocols,
architecture-independent drivers, etc.)
89/475
Device Tree
Many embedded architectures have a lot of non-discoverable

hardware.
Depending on the architecture, such hardware is either

described using C code directly within the kernel, or using a
special hardware description language in a Device Tree.
ARM, PowerPC, OpenRISC, ARC, Microblaze are examples of

architectures using the Device Tree.
A Device Tree Source, written by kernel developers, is
compiled into a binary Device Tree Blob, passed at boot time
to the kernel.
There is one dierent Device Tree for each board/platform

supported by the kernel, available in
arch/arm/boot/dts/<board>.dtb.
The bootloader must load both the kernel image and the
Device Tree Blob in memory before starting the kernel.
90/475
Customize your board device tree!

Often needed for embedded board users:
To describe external devices attached

to non-discoverable busses (such as
I2C) and congure them.
To congure pin muxing: choosing

what SoC signals are made available
on the board external connectors.
To congure some system parameters:

ash partitions, kernel command line
(other ways exist)
Useful reference: Device Tree for

Dummies, Thomas Petazzoni (Apr.
2014): http://j.mp/1jQU6NR
91/475
Building and installing the kernel
Run make
Copy the nal kernel image to the target storage
make install is rarely used in embedded development, as the

kernel image is a single le, easy to handle
can be zImage, vmlinux, bzImage in arch/<arch>/boot

copying the Device Tree Blob might be necessary as well, they
are available in arch/<arch>/boot/dts
It is however possible to customize the make install

behaviour in arch/<arch>/boot/install.sh
make modules_install is used even in embedded

development, as it installs many modules and description les
make INSTALL_MOD_PATH=<dir>/ modules_install

The INSTALL_MOD_PATH variable is needed to install the
modules in the target root lesystem instead of your host root
lesystem.
92/475
Booting with U-Boot
Recent versions of U-Boot can boot the zImage binary.

Older versions require a special kernel image format: uImage
uImage is generated from zImage using the mkimage tool. It is

done automatically by the kernel make uImage target.
On some ARM platforms, make uImage requires passing a
LOADADDR environment variable, which indicates at which
physical memory address the kernel will be executed.
In addition to the kernel image, U-Boot can also pass a

Device Tree Blob to the kernel.
The typical boot process is therefore:
1. Load zImage or uImage at address X in memory
2. Load <board>.dtb at address Y in memory
3. Start the kernel with bootz X - Y (zImage case), or
bootm X - Y (uImage case)
The - in the middle indicates no initramfs
93/475
Kernel command line
In addition to the compile time conguration, the kernel

behaviour can be adjusted with no recompilation using the
kernel command line
The kernel command line is a string that denes various
arguments to the kernel
It is very important for system conguration

root= for the root lesystem (covered later)
console= for the destination of kernel messages
Many more exist. The most important ones are documented in
Documentation/kernel-parameters.txt in kernel sources.
This kernel command line is either
Passed by the bootloader. In U-Boot, the contents of the

bootargs environment variable is automatically passed to the
kernel
Built into the kernel, using the CONFIG_CMDLINE option.
94/475
Practical lab - Kernel compiling and booting

1st lab: board and bootloader setup:
Prepare the board and access its

serial port
Congure its bootloader to use

TFTP
2nd lab: kernel compiling and booting:
Set up a cross-compiling
environment
Cross-compile a kernel for an ARM

target platform
Boot this kernel from a directory

on your workstation, accessed by
the board through NFS
95/475
Kernel Source Code
Using kernel modules
96/475
Advantages of modules
Modules make it easy to develop drivers without rebooting:

load, test, unload, rebuild, load...
Useful to keep the kernel image size to the minimum

(essential in GNU/Linux distributions for PCs).
Also useful to reduce boot time: you don't spend time

initializing devices and kernel features that you only need later.
Caution: once loaded, have full control and privileges in the

system. No particular protection. That's why only the root
user can load and unload modules.
97/475
Module dependencies
Some kernel modules can depend on other modules, which

need to be loaded rst.
Example: the usb-storage module depends on the scsi_mod,

libusual and usbcore modules.
Dependencies are described both in

/lib/modules/<kernel-version>/modules.dep and in
/lib/modules/<kernel-version>/modules.dep.bin
These les are generated when you run
make modules_install.
98/475
Kernel log
When a new module is loaded, related information is available in
the kernel log.
The kernel keeps its messages in a circular buer (so that it

doesn't consume more memory with many messages)
Kernel log messages are available through the dmesg

command (diagnostic message)
Kernel log messages are also displayed in the system console

(console messages can be ltered by level using the loglevel
kernel parameter, or completely disabled with the quiet
parameter).
Note that you can write to the kernel log from user space too:
echo "<n>Debug info" > /dev/kmsg
99/475
Module utilities (1)
<module_name>: name of the module le without the trailing .ko
modinfo <module_name> (for modules in /lib/modules)

modinfo <module_path>.ko
Gets information about a module without loading it:
parameters, license, description and dependencies.
sudo insmod <module_path>.ko

Tries to load the given module. The full path to the module
object le must be given.
100/475
Understanding module loading issues
When loading a module fails, insmod often doesn't give you

enough details!
Details are often available in the kernel log.
Example:
$ sudo insmod ./intr_monitor.ko
insmod: error inserting './intr_monitor.ko': -1 Device or resource busy
$ dmesg
[17549774.552000] Failed to register handler for irq channel 2
101/475
sudo modprobe <module_name>

Most common usage of modprobe: tries to load all the
modules the given module depends on, and then this module.
Lots of other options are available. modprobe automatically
looks in /lib/modules/<version>/ for the object le
corresponding to the given module name.
lsmod
Displays the list of loaded modules
Compare its output with the contents of /proc/modules!
102/475
sudo rmmod <module_name>

Tries to remove the given module.
Will only be allowed if the module is no longer in use (for
example, no more processes opening a device le)
sudo modprobe -r <module_name>

Tries to remove the given module and all dependent modules
(which are no longer needed after removing the module)
103/475
Passing parameters to modules
Find available parameters:

modinfo usb-storage
Through insmod:
sudo insmod ./usb-storage.ko delay_use=0
Through modprobe:
Set parameters in /etc/modprobe.conf or in any le in
/etc/modprobe.d/:
options usb-storage delay_use=0
Through the kernel command line, when the driver is built
statically into the kernel:
usb-storage.delay_use=0
usb-storage is the driver name

delay_use is the driver parameter name. It species a delay
before accessing a USB storage device (useful for rotating
devices).
0 is the driver parameter value
104/475
Check module parameter values
How to nd the current values for the parameters of a loaded

module?
Check /sys/module/<name>/parameters.
There is one le per parameter, containing the parameter

value.
105/475
Useful reading
Linux Kernel in a Nutshell, Dec 2006
By Greg Kroah-Hartman, O'Reilly

http://www.kroah.com/lkn/
A good reference book and guide on

conguring, compiling and managing the
Linux kernel sources.
Freely available on-line!

Great companion to the printed book for
easy electronic searches!
Available as single PDF le on
http://freeelectrons.com/community/kernel/lkn/
Our rating: 2 stars
106/475
Developing Kernel Modules
Developing Kernel
Modules
Embedded Linux
Experts
free electrons
107/475
Hello Module 1/2

/* hello.c */
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
static int __init hello_init(void)
{
pr_alert("Good morrow to this fair assembly.\n");
return 0;
}
static void __exit hello_exit(void)
{
pr_alert("Alas, poor world, what treasure hast thou lost!\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Greeting module");
MODULE_AUTHOR("William Shakespeare");
108/475
Hello Module 2/2
__init
__exit
removed after initialization (static kernel or module.)

discarded when module compiled statically into the kernel, or
when module unloading support is not enabled.
Example available on
http://git.free-electrons.com/trainingmaterials/plain/code/hello/hello.c
109/475
Hello Module Explanations
Headers specic to the Linux kernel: linux/xxx.h
An initialization function
Called when the module is loaded, returns an error code (0 on

success, negative value on failure)
Declared by the module_init() macro: the name of the
function doesn't matter, even though <modulename>_init()
is a convention.
A cleanup function
No access to the usual C library, we're doing kernel

programming
Called when the module is unloaded

Declared by the module_exit() macro.
Metadata information declared using MODULE_LICENSE(),

MODULE_DESCRIPTION() and MODULE_AUTHOR()
110/475
Symbols Exported to Modules 1/2
From a kernel module, only a limited number of kernel

functions can be called
Functions and variables have to be explicitly exported by the

kernel to be visible to a kernel module
Two macros are used in the kernel to export functions and
variables:
EXPORT_SYMBOL(symbolname), which exports a function or

variable to all modules
EXPORT_SYMBOL_GPL(symbolname), which exports a function
or variable only to GPL modules
A normal driver should not need any non-exported function.
111/475
Symbols exported to modules 2/2
112/475
Module License
Several usages
Used to restrict the kernel functions that the module can use if
it isn't a GPL licensed module
Dierence between EXPORT_SYMBOL() and

EXPORT_SYMBOL_GPL()
Used by kernel developers to identify issues coming from

proprietary drivers, which they can't do anything about
(Tainted kernel notice in kernel crashes and oopses).
Useful for users to check that their system is 100% free (check
/proc/sys/kernel/tainted)
Values
GPL compatible (see include/linux/license.h: GPL,

GPL v2, GPL and additional rights, Dual MIT/GPL,
Dual BSD/GPL, Dual MPL/GPL)
Proprietary
113/475
Compiling a Module
Two solutions
Out of tree
When the code is outside of the kernel source tree, in a

dierent directory
Advantage: Might be easier to handle than modications to
the kernel itself
Drawbacks: Not integrated to the kernel
conguration/compilation process, needs to be built separately,
the driver cannot be built statically
Inside the kernel tree
Well integrated into the kernel conguration/compilation

process
Driver can be built statically if needed
114/475
Compiling an out-of-tree Module 1/2
The below Makefile should be reusable for any single-le

out-of-tree Linux module
The source le is hello.c
Just run make to build the hello.ko le
.
ifneq ($(KERNELRELEASE),)
obj-m := hello.o
else
KDIR := /path/to/kernel/sources
all:
<tab>$(MAKE) -C $(KDIR) M=$$PWD
endif
.
KDIR: kernel source or headers directory (see next slides)
115/475
Compiling an out-of-tree Module 2/2
The module Makefile is interpreted with KERNELRELEASE

undened, so it calls the kernel Makefile, passing the module
directory in the M variable
The kernel Makefile knows how to compile a module, and
thanks to the M variable, knows where the Makefile for our
module is. This module Makefile is then interpreted with
KERNELRELEASE dened, so the kernel sees the obj-m
denition.
116/475
Modules and Kernel Version
To be compiled, a kernel module needs access to the kernel

headers, containing the denitions of functions, types and
constants.
Two solutions
The sources or headers must be congured
Full kernel sources (congured + make modules_prepare)

Only kernel headers (linux-headers-* packages in
Debian/Ubuntu distributions, or directory created by
make headers_install)
Many macros or functions depend on the conguration
A kernel module compiled against version X of kernel headers

will not load in kernel version Y
modprobe / insmod will say Invalid module format
117/475
New Driver in Kernel Sources 1/2
To add a new driver to the kernel sources:
Add your new source le to the appropriate source directory.

Example: drivers/usb/serial/navman.c
Single le drivers in the common case, even if the le is several
thousand lines of code big. Only really big drivers are split in
several les or have their own directory.
Describe the conguration interface for your new driver by
adding the following lines to the Kconfig le in this directory:
.
config USB_SERIAL_NAVMAN
tristate "USB Navman GPS device"
depends on USB_SERIAL
help
To compile this driver as a module, choose M
here: the module will be called navman.
.
118/475
New Driver in Kernel Sources 2/2
Add a line in the Makefile le based on the Kconfig setting:

obj-$(CONFIG_USB_SERIAL_NAVMAN) += navman.o
It tells the kernel build system to build navman.c when the
USB_SERIAL_NAVMAN option is enabled. It works both if
compiled statically or as a module.
Run make xconfig and see your new options!

Run make and your new les are compiled!
See Documentation/kbuild/ for details and more elaborate
examples like drivers with several source les, or drivers in their
own subdirectory, etc.
119/475
Hello Module with Parameters 1/2
/* hello_param.c */
#include <linux/init.h>
#include <linux/module.h>
MODULE_LICENSE("GPL");
/* A couple of parameters that can be passed in: how many
times we say hello, and to whom */
static char *whom = "world";
module_param(whom, charp, 0);
static int howmany = 1;
module_param(howmany, int, 0);
120/475
Hello Module with Parameters 2/2

static int __init hello_init(void)
{
int i;
for (i = 0; i < howmany; i++)
pr_alert("(%d) Hello, %s\n", i, whom);
return 0;
}
static void __exit hello_exit(void)
{
pr_alert("Goodbye, cruel %s\n", whom);
}
module_init(hello_init);
module_exit(hello_exit);
Thanks to Jonathan Corbet for the example!

Source code available on:
http://git.free-electrons.com/trainingmaterials/plain/code/hello-param/hello_param.c
121/475
Declaring a module parameter
module_param(
name, /* name of an already defined variable */
type, /* either byte, short, ushort, int, uint, long, ulong,
charp, bool or invbool. (checked at run time!) */
perm /* for /sys/module/<module_name>/parameters/<param>,
0: no such module parameter value file */
);
/* Example */
static int irq=5;
module_param(irq, int, S_IRUGO);
Modules parameter arrays are also possible with

module_param_array().
122/475
Practical lab - Writing Modules
Create, compile and load your rst

module
Add module parameters
Access kernel internals from your

module
123/475
Useful general-purpose kernel APIs
Useful
general-purpose
kernel APIs
Embedded Linux
Experts
free electrons
124/475
Memory/string utilities
In include/linux/string.h
Memory-related: memset(), memcpy(), memmove(), memscan(),

memcmp(), memchr()
String-related: strcpy(), strcat(), strcmp(), strchr(),
strrchr(), strlen() and variants
Allocate and copy a string: kstrdup(), kstrndup()
Allocate and copy a memory area: kmemdup()
In include/linux/kernel.h
String to int conversion: simple_strtoul(),

simple_strtol(), simple_strtoull(), simple_strtoll()
Other string functions: sprintf(), sscanf()
125/475
Linked lists
Convenient linked-list facility in include/linux/list.h
Used in thousands of places in the kernel
Add a struct list_head member to the structure whose

instances will be part of the linked list. It is usually named
node when each instance needs to only be part of a single list.
Dene the list with the LIST_HEAD() macro for a global list,
or dene a struct list_head element and initialize it with
INIT_LIST_HEAD() for lists embedded in a structure.
Then use the list_*() API to manipulate the list
Add elements: list_add(), list_add_tail()

Remove, move or replace elements: list_del(),
list_move(), list_move_tail(), list_replace()
Test the list: list_empty()
Iterate over the list: list_for_each_*() family of macros
126/475
Linked Lists Examples (1)
From include/linux/atmel_tc.h
/*
* Definition of a list element, with a
* struct list_head member
*/
struct atmel_tc
{
/* some members */
struct list_head node;
};
127/475
Linked Lists Examples (2)
From drivers/misc/atmel_tclib.c
/* Define the global list */

static LIST_HEAD(tc_list);
static int __init tc_probe(struct platform_device *pdev) {
struct atmel_tc *tc;
tc = kzalloc(sizeof(struct atmel_tc), GFP_KERNEL);
/* Add an element to the list */
list_add_tail(&tc->node, &tc_list);
}
struct atmel_tc *atmel_tc_alloc(unsigned block, const char *name)
{
struct atmel_tc *tc;
/* Iterate over the list elements */
list_for_each_entry(tc, &tc_list, node) {
/* Do something with tc */
}
[...]
}
128/475
Linux device and driver model
Linux device and

driver model
Embedded Linux
Experts
free electrons
129/475
Introduction
130/475
The need for a device model?
The Linux kernel runs on a wide range of architectures and

hardware platforms, and therefore needs to maximize the
reusability of code between platforms.
For example, we want the same USB device driver to be

usable on a x86 PC, or an ARM platform, even though the
USB controllers used on these platforms are dierent.
This requires a clean organization of the code, with the device

drivers separated from the controller drivers, the hardware
description separated from the drivers themselves, etc.
This is what the Linux kernel Device Model allows, in

addition to other advantages covered in this section.
131/475
Kernel and Device Drivers

In Linux, a driver is always
interfacing with:
a framework that allows the

driver to expose the
hardware features in a
generic way.
a bus infrastructure, part

of the device model, to
detect/communicate with
the hardware.
This section focuses on the

device model, while kernel
frameworks are covered later in
this training.
132/475
Device Model data structures
The device model is organized around three main data

structures:
The struct bus_type structure, which represent one type of

bus (USB, PCI, I2C, etc.)
The struct device_driver structure, which represents one
driver capable of handling certain devices on a certain bus.
The struct device structure, which represents one device
connected to a bus
The kernel uses inheritance to create more specialized versions

of struct device_driver and struct device for each bus
subsystem.
In order to explore the device model, we will
First look at a popular bus that oers dynamic enumeration,

the USB bus
Continue by studying how buses that do not oer dynamic
enumerations are handled.
133/475
Bus Drivers
The rst component of the device model is the bus driver
One bus driver for each type of bus: USB, PCI, SPI, MMC,
I2C, etc.
It is responsible for
Registering the bus type (struct bus_type)

Allowing the registration of adapter drivers (USB controllers,
I2C adapters, etc.), able to detect the connected devices, and
providing a communication mechanism with the devices
Allowing the registration of device drivers (USB devices, I2C
devices, PCI devices, etc.), managing the devices
Matching the device drivers against the devices detected by
the adapter drivers.
Provides an API to both adapter drivers and device drivers
Dening driver and device specic structures, mainly
struct usb_driver and struct usb_interface
134/475
Example of the USB bus
135/475
Example: USB Bus 1/2
136/475
Example: USB Bus 2/2
Core infrastructure (bus driver)
Adapter drivers
drivers/usb/core
struct bus_type is dened in drivers/usb/core/driver.c
and registered in drivers/usb/core/usb.c
drivers/usb/host
For EHCI, UHCI, OHCI, XHCI, and their implementations on
various systems (Atmel, IXP, Xilinx, OMAP, Samsung, PXA,
etc.)
Device drivers
Everywhere in the kernel tree, classied by their type
137/475
Example of Device Driver
To illustrate how drivers are implemented to work with the

device model, we will study the source code of a driver for a
USB network card
It is USB device, so it has to be a USB device driver

It is a network device, so it has to be a network driver
Most drivers rely on a bus infrastructure (here, USB) and
register themselves in a framework (here, network)
We will only look at the device driver side, and not the
adapter driver side
The driver we will look at is drivers/net/usb/rtl8150.c
138/475
Device Identiers
Denes the set of devices that this driver can manage, so that
the USB core knows for which devices this driver should be
used
The MODULE_DEVICE_TABLE() macro allows depmod to extract

at compile time the relation between device identiers and
drivers, so that drivers can be loaded automatically by udev.
See /lib/modules/$(uname -r)/modules.{alias,usbmap}
static struct usb_device_id rtl8150_table[] = {

{ USB_DEVICE(VENDOR_ID_REALTEK, PRODUCT_ID_RTL8150) },
{ USB_DEVICE(VENDOR_ID_MELCO, PRODUCT_ID_LUAKTX) },
{ USB_DEVICE(VENDOR_ID_MICRONET, PRODUCT_ID_SP128AR) },
{ USB_DEVICE(VENDOR_ID_LONGSHINE, PRODUCT_ID_LCS8138TX) },
[...]
{}
};
.MODULE_DEVICE_TABLE(usb, rtl8150_table);
139/475
Instantiation of usb_driver
struct usb_driver is a structure dened by the USB core.

Each USB device driver must instantiate it, and register itself
to the USB core using this structure
This structure inherits from struct device_driver, which is

dened by the device model.
.
static struct usb_driver rtl8150_driver = {
.name = "rtl8150",
.probe = rtl8150_probe,
.disconnect = rtl8150_disconnect,
.id_table = rtl8150_table,
.suspend = rtl8150_suspend,
.resume = rtl8150_resume
};
.
140/475
Driver (Un)Registration
When the driver is loaded or unloaded, it must register or

unregister itself from the USB core
Done using usb_register() and usb_deregister(),

provided by the USB core.
static int __init usb_rtl8150_init(void)

{
return usb_register(&rtl8150_driver);
}
static void __exit usb_rtl8150_exit(void)
{
usb_deregister(&rtl8150_driver);
}
module_init(usb_rtl8150_init);
module_exit(usb_rtl8150_exit);
.
Note: this code has now been replaced by a shorter

module_usb_driver() macro call.
141/475
At Initialization
The USB adapter driver that corresponds to the USB

controller of the system registers itself to the USB core
The rtl8150 USB device driver registers itself to the USB

core
The USB core now knows the association between the

vendor/product IDs of rtl8150 and the struct usb_driver
structure of this driver
142/475
When a Device is Detected
143/475
Probe Method
The probe() method receives as argument a structure

describing the device, usually specialized by the bus
infrastructure (struct pci_dev, struct usb_interface,
etc.)
This function is responsible for
Initializing the device, mapping I/O memory, registering the

interrupt handlers. The bus infrastructure provides methods to
get the addresses, interrupt numbers and other device-specic
information.
Registering the device to the proper kernel framework, for
example the network infrastructure.
144/475
Probe Method Example

.
static int rtl8150_probe(struct usb_interface *intf,

const struct usb_device_id *id)
{
rtl8150_t *dev;
struct net_device *netdev;
netdev = alloc_etherdev(sizeof(rtl8150_t));
[...]
dev = netdev_priv(netdev);
tasklet_init(&dev->tl, rx_fixup, (unsigned long)dev);
spin_lock_init(&dev->rx_pool_lock);
[...]
netdev->netdev_ops = &rtl8150_netdev_ops;
alloc_all_urbs(dev);
[...]
usb_set_intfdata(intf, dev);
SET_NETDEV_DEV(netdev, &intf->dev);
register_netdev(netdev);
return 0;
}.
145/475
The Model is Recursive
146/475
Platform drivers
147/475
Non-discoverable buses
On embedded systems, devices are often not connected

through a bus allowing enumeration, hotplugging, and
providing unique identiers for devices.
For example, the devices on I2C buses or SPI buses, or the

devices directly part of the system-on-chip.
However, we still want all of these devices to be part of the

device model.
Such devices, instead of being dynamically detected, must be
statically described in either:
The kernel source code

The Device Tree, a hardware description le used on some
architectures.
148/475
Platform devices
Amongst the non-discoverable devices, a huge family are the

devices that are directly part of a system-on-chip: UART
controllers, Ethernet controllers, SPI or I2C controllers,
graphic or audio devices, etc.
In the Linux kernel, a special bus, called the platform bus has
been created to handle such devices.
It supports platform drivers that handle platform devices.
It works like any other bus (USB, PCI), except that devices are
enumerated statically instead of being discovered dynamically.
149/475
Implementation of a Platform Driver
The driver implements a struct platform_driver structure

(example taken from drivers/tty/serial/imx.c, simplied)
static struct platform_driver serial_imx_driver = {

.probe
= serial_imx_probe,
.remove
= serial_imx_remove,
.id_table
= imx_uart_devtype,
.driver
= {
.name = "imx-uart",
.of_match_table = imx_uart_dt_ids,
.pm
= &imx_serial_port_pm_ops,
},
};
.
And registers its driver to the platform driver infrastructure
static int __init imx_serial_init(void) {

ret = platform_driver_register(&serial_imx_driver);
}
static void __exit imx_serial_cleanup(void) {
platform_driver_unregister(&serial_imx_driver);
}.
150/475
Platform Device Instantiation: old style (1/2)
As platform devices cannot be detected dynamically, they are

dened statically
By direct instantiation of struct platform_device

structures, as done on a few old ARM platforms. Denition
done in the board-specic or SoC specic code.
By using a device tree, as done on Power PC (and on most
ARM platforms) from which struct platform_device
structures are created
Example on ARM, where the instantiation was done in

arch/arm/mach-imx/mx1ads.c
static struct platform_device imx_uart1_device = {

.name = "imx-uart",
.id = 0,
.num_resources = ARRAY_SIZE(imx_uart1_resources),
.resource = imx_uart1_resources,
.dev = {
.platform_data = &uart_pdata,
}
};
.
151/475
Platform device instantiation: old style (2/2)
The device was part of a list
static struct platform_device *devices[] __initdata = {

&cs89x0_device,
&imx_uart1_device,
&imx_uart2_device,
.};
And the list of devices was added to the system during board
initialization
static void __init mx1ads_init(void)

{
[...]
platform_add_devices(devices, ARRAY_SIZE(devices));
}
MACHINE_START(MX1ADS, "Freescale MX1ADS")
[...]
.init_machine = mx1ads_init,
MACHINE_END
.
152/475
The Resource Mechanism
Each device managed by a particular driver typically uses

dierent hardware resources: addresses for the I/O registers,
DMA channels, IRQ lines, etc.
Such information can be represented using struct resource,

and an array of struct resource is associated to a
struct platform_device
Allows a driver to be instantiated for multiple devices

functioning similarly, but with dierent addresses, IRQs, etc.
153/475
Declaring resources (old style)
static struct resource imx_uart1_resources[] = {

[0] = {
.start = 0x00206000,
.end = 0x002060FF,
.flags = IORESOURCE_MEM,
},
[1] = {
.start = (UART1_MINT_RX),
.end = (UART1_MINT_RX),
.flags = IORESOURCE_IRQ,
},
};
.
154/475
Using Resources (old style)
When a struct platform_device was added to the system

using platform_add_device(), the probe() method of the
platform driver was called
This method is responsible for initializing the hardware,

registering the device to the proper framework (in our case,
the serial driver framework)
The platform driver has access to the I/O resources:
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);

base = ioremap(res->start, PAGE_SIZE);
.sport->rxirq = platform_get_irq(pdev, 0);
155/475
platform_data Mechanism (old style)
In addition to the well-dened resources, many drivers require

driver-specic information for each platform device
Such information could be passed using the platform_data

eld of struct device (from which
struct platform_device inherits)
As it is a void * pointer, it could be used to pass any type of
information.
Typically, each driver denes a structure to pass information

through struct platform_data
156/475
platform_data example 1/2
The i.MX serial port driver denes the following structure to

be passed through struct platform_data
struct imxuart_platform_data {
int (*init)(struct platform_device *pdev);
void (*exit)(struct platform_device *pdev);
unsigned int flags;
void (*irda_enable)(int enable);
unsigned int irda_inv_rx:1;
unsigned int irda_inv_tx:1;
unsigned short transceiver_delay;
};
.
The MX1ADS board code instantiated such a structure
static struct imxuart_platform_data uart1_pdata = {

.flags = IMXUART_HAVE_RTSCTS,
.};
157/475
platform_data Example 2/2
The uart_pdata structure was associated to the

struct platform_device structure in the MX1ADS board
le (the real code was slightly more complicated)
struct platform_device mx1ads_uart1 = {

.name = "imx-uart",
.dev {
.platform_data = &uart1_pdata,
},
.resource = imx_uart1_resources,
[...]
};
.
The driver can access the platform data:
static int serial_imx_probe(struct platform_device *pdev)

{
struct imxuart_platform_data *pdata;
pdata = pdev->dev.platform_data;
if (pdata && (pdata->flags & IMXUART_HAVE_RTSCTS))
sport->have_rtscts = 1;
. [...]
158/475
Device Tree
On many embedded architectures, manual instantiation of

platform devices was considered to be too verbose and not
easily maintainable.
Such architectures are moving, or have moved, to use the

Device Tree.
It is a tree of nodes that models the hierarchy of devices in

the system, from the devices inside the processor to the
devices on the board.
Each node can have a number of properties describing various

properties of the devices: addresses, interrupts, clocks, etc.
At boot time, the kernel is given a compiled version, the

Device Tree Blob, which is parsed to instantiate all the
devices described in the DT.
On ARM, they are located in arch/arm/boot/dts.
159/475
Device Tree example

.
uart0: serial@44e09000 {
compatible = "ti,omap3-uart";
ti,hwmods = "uart1";
clock-frequency = <48000000>;
reg = <0x44e09000 0x2000>;
interrupts = <72>;
status = "disabled";
};
.
serial@44e09000 is the node name
uart0 is a label, that can be referred to in other parts of the

DT as &uart0
other lines are properties. Their values are usually strings, list
of integers, or references to other nodes.
160/475
Device Tree inheritance (1/2)
Each particular hardware platform has its own device tree.
However, several hardware platforms use the same processor,

and often various processors in the same family share a
number of similarities.
To allow this, a device tree le can include another one. The
trees described by the including le overlays the tree described
by the included le. This can be done:
Either by using the /include/ statement provided by the

Device Tree language.
Either by using the #include statement, which requires calling
the C preprocessor before parsing the Device Tree.
Linux currently uses either one technique or the other,

(dierent from one ARM subarchitecture to another, for
example).
161/475
Device Tree inheritance (2/2)
162/475
Device Tree: compatible string
With the device tree, a device is bound to the corresponding

driver using the compatible string.
The of_match_table eld of struct device_driver lists the

compatible strings supported by the driver.
#if defined(CONFIG_OF)
static const struct of_device_id omap_serial_of_match[] = {
{ .compatible = "ti,omap2-uart" },
{},
};
MODULE_DEVICE_TABLE(of, omap_serial_of_match);
#endif
static struct platform_driver serial_omap_driver = {
.probe
= serial_omap_probe,
.remove
= serial_omap_remove,
.driver
= {
.name = DRIVER_NAME,
.pm
= &serial_omap_dev_pm_ops,
.of_match_table = of_match_ptr(omap_serial_of_match),
},
};
163/475
Device Tree Resources
The drivers will use the same mechanism that we saw

previously to retrieve basic information: interrupts numbers,
physical addresses, etc.
The available resources list will be built up by the kernel at

boot time from the device tree, so that you don't need to
make any unnecessary lookups to the DT when loading your
driver.
Any additional information will be specic to a driver or the

class it belongs to, dening the bindings
164/475
Device Tree bindings
The compatible string and the associated properties dene

what is called a device tree binding.
Device tree bindings are all documented in

Documentation/devicetree/bindings.
Since the Device Tree is normally part of the kernel ABI, the
bindings must remain compatible over-time.
A new kernel must be capable of using an old Device Tree.

This requires a very careful design of the bindings. They are all
reviewed on the devicetree@vger.kernel.org mailing list.
A Device Tree binding should contain only a description of the

hardware and not conguration.
An interrupt number can be part of the Device Tree as it

describes the hardware.
But not whether DMA should be used for a device or not, as it
is a conguration choice.
165/475
sysfs
The bus, device, drivers, etc. structures are internal to the

kernel
The sysfs virtual lesystem oers a mechanism to export

such information to user space
Used for example by udev to provide automatic module

loading, rmware loading, device le creation, etc.
sysfs is usually mounted in /sys
/sys/bus/ contains the list of buses

/sys/devices/ contains the list of devices
/sys/class enumerates devices by class (net, input,
block...), whatever the bus they are connected to. Very useful!
Take your time to explore /sys on your workstation.
166/475
References
Device Tree for Dummies,

Thomas Petazzoni (Apr.
2014):
http://j.mp/1jQU6NR
Kernel documentation
Documentation/drivermodel/
Documentation/
devicetree/
Documentation/
filesystems/sysfs.txt
http://devicetree.org
The kernel source code
Full of examples of other

drivers!
167/475
Introduction to the I2C subsystem
Introduction to
the I2C subsystem
Embedded Linux
Experts
free electrons
168/475
What is I2C?
A very commonly used low-speed bus to connect on-board

and external devices to the processor.
Uses only two wires: SDA for the data, SCL for the clock.
It is a master/slave bus: only the master can initiate

transactions, and slaves can only reply to transactions
initiated by masters.
In a Linux system, the I2C controller embedded in the

processor is typically the master, controlling the bus.
Each slave device is identied by a unique I2C address. Each

transaction initiated by the master contains this address,
which allows the relevant slave to recognize that it should
reply to this particular transaction.
169/475
An I2C bus example
170/475
The I2C subsystem
Like all bus subsystems, the I2C subsystem is responsible for:
Providing an API to implement I2C controller drivers

Providing an API to implement I2C device drivers, in kernel
space
Providing an API to implement I2C device drivers, in user
space
The core of the I2C subsystem is located in drivers/i2c.
The I2C controller drivers are located in drivers/i2c/busses.
The I2C device drivers are located throughout drivers/,

depending on the type of device (ex: drivers/input for input
devices).
171/475
Registering an I2C device driver
Like all bus subsystems, the I2C subsystem denes a

struct i2c_driver that inherits from
struct device_driver, and which must be instantiated and
registered by each I2C device driver.
As usual, this structure points to the ->probe() and

->remove() functions.
It also contains an id_table eld that must point to a list of
device IDs (which is a list of tuples containing a string and
some private driver data). It is used for non-DT based probing
of I2C devices.
The i2c_add_driver() and i2c_del_driver() functions are

used to register/unregister the driver.
If the driver doesn't do anything else in its init()/exit()

functions, it is advised to use the module_i2c_driver()
macro instead.
172/475
Registering an I2C device driver: example

.
static const struct i2c_device_id <driver>_id[] = {

{ "<device-name>", 0 },
{ }
};
MODULE_DEVICE_TABLE(i2c, <driver>_id);
#ifdef CONFIG_OF
static const struct of_device_id <driver>_dt_ids[] = {
{ .compatible = "<vendor>,<device-name>", },
{ }
};
MODULE_DEVICE_TABLE(of, <driver>_dt_ids);
#endif
static struct i2c_driver <driver>_driver = {
.probe
= <driver>_probe,
.remove
= <driver>_remove,
.id_table
= <driver>_id,
.driver = {
.name = "<driver-name>",
.owner = THIS_MODULE,
.of_match_table = of_match_ptr(<driver>_dt_ids),
},
};
module_i2c_driver(<driver>_driver);
173/475
Registering an I2C device: non-DT
On non-DT platforms, the struct i2c_board_info structure

allows to describe how an I2C device is connected to a board.
Such structures are normally dened with the
I2C_BOARD_INFO() helper macro.
Takes as argument the device name and the slave address of

the device on the bus.
An array of such structures is registed on a per-bus basis using

i2c_register_board_info(), when the platform is
initialized.
174/475
Registering an I2C device, non-DT example
static struct i2c_board_info <board>_i2c_devices[] __initdata = {

{
I2C_BOARD_INFO("cs42l51", 0x4a),
},
};
void board_init(void)
{
/*
* Here should be the registration of all devices, including
* the I2C controller device.
*/
i2c_register_board_info(0, <board>_i2c_devices,
ARRAY_SIZE(<board>_i2c_devices));
/* More devices registered here */
175/475
Registering an I2C device, in the DT
In the Device Tree, the I2C controller device is typically

dened in the .dtsi le that describes the processor.
Normally dened with status = "disabled".
At the board/platform level:
the I2C controller device is enabled (status = "okay")

the I2C bus frequency is dened, using the clock-frequency
property.
the I2C devices on the bus are described as children of the I2C
controller node, where the reg property gives the I2C slave
address on the bus.
176/475
Registering an I2C device, DT example (1/2)
.
Denition of the I2C controller, sun7i-a20.dtsi le
.
i2c0: i2c@01c2ac00 {
compatible = "allwinner,sun7i-a20-i2c",
"allwinner,sun4i-a10-i2c";
reg = <0x01c2ac00 0x400>;
interrupts = <GIC_SPI 7 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&apb1_gates 0>;
status = "disabled";
#address-cells = <1>;
#size-cells = <0>;
};
.
177/475
Registering an I2C device, DT example (2/2)

.
Denition of the I2C device,

sun7i-a20-olinuxino-micro.dts le
.
&i2c0 {
pinctrl-names = "default";
pinctrl-0 = <&i2c0_pins_a>;
status = "okay";
axp209: pmic@34 {
compatible = "x-powers,axp209";
reg = <0x34>;
interrupt-parent = <&nmi_intc>;
interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
interrupt-controller;
#interrupt-cells = <1>;
};
};
.
178/475
probe() and remove()
The ->probe() function is responsible for initializing the

device and registering it in the appropriate kernel framework.
It receives as argument:
A struct i2c_client pointer, which represents the I2C

device itself. This structure inherits from struct device.
A struct i2c_device_id pointer, which points to the I2C
device ID entry that matched the device that is being probed.
The ->remove() function is responsible for unregistering the

device from the kernel framework and shut it down. It receives
as argument:
The same struct i2c_client pointer that was passed as

argument to ->probe()
179/475
Probe/remove example
.
static int <driver>_probe(struct i2c_client *client,

const struct i2c_device_id *id)
{
/* initialize device */
/* register to a kernel framework */
i2c_set_clientdata(client, <private data>);
return 0;
}
static int <driver>_remove(struct i2c_client *client)
{
<private data> = i2c_get_clientdata(client);
/* unregister device from kernel framework */
/* shut down the device */
return 0;
}.
180/475
Practical lab - Linux device model for an I2C driver
Modify the Device Tree to

instantiate an I2C device.
Implement a driver that registers as

an I2C driver.
Make sure that the probe/remove

functions are called when there is a
device/driver match.
Explore the sysfs entries related to

your driver and device.
181/475
Communicating with the I2C device: raw API
The most basic API to communicate with the I2C device provides
functions to either send or receive data:
int i2c_master_send(struct i2c_client *client, const

char *buf, int count);
Sends the contents of buf to the client.
int i2c_master_recv(struct i2c_client *client, char

*buf, int count);
Receives count bytes from the client, and store them into buf.
182/475
Communicating with the I2C device: message transfer
The message transfer API allows to describe transfers that

consists of several messages, with each message being a
transaction in one direction:
int i2c_transfer(struct i2c_adapter *adap, struct

i2c_msg *msg, int num);
The struct i2c_adapter pointer can be found by using

client->adapter
The struct i2c_msg structure denes the length, location,

and direction of the message.
183/475
I2C: message transfer example

.
struct i2c_msg msg[2];

int error;
u8 start_reg;
u8 buf[10];
msg[0].addr = client->addr;
msg[0].flags = 0;
msg[0].len = 1;
msg[0].buf = &start_reg;
start_reg = 0x10;
msg[1].addr = client->addr;
msg[1].flags = I2C_M_RD;
msg[1].len = sizeof(buf);
msg[1].buf = buf;
error
= i2c_transfer(client->adapter, msg, 2);
.
184/475
SMBus calls
SMBus is a subset of the I2C protocol.
It denes a standard set of transactions, for example to read

or write a register into a device.
Linux provides SMBus functions that should be used instead

of the raw API, if the I2C device supports this standard type of
transactions. The driver can then be used on both SMBus and
I2C adapters (can't use I2C commands on SMBus adapters).
Example: the i2c_smbus_read_byte_data() function allows
to read one byte of data from a device register.
It does the following operations:

S Addr Wr [A] Comm [A] S Addr Rd [A] [Data] NA P
Which means it rst writes a one byte data command
(Comm), and then reads back one byte of data ([Data]).
See Documentation/i2c/smbus-protocol for details.
185/475
List of SMBus functions
Read/write one byte

s32 i2c_smbus_read_byte(const struct i2c_client *client);
s32 i2c_smbus_write_byte(const struct i2c_client *client, u8 value);
Write a command byte, and read or write one byte

s32 i2c_smbus_read_byte_data(const struct i2c_client *client, u8 command);
s32 i2c_smbus_write_byte_data(const struct i2c_client *client, u8 command, u8 value);
Write a command byte, and read or write one word

s32 i2c_smbus_read_word_data(const struct i2c_client *client, u8 command);
s32 i2c_smbus_write_word_data(const struct i2c_client *client, u8 command, u16 value);
Write a command byte, and read or write a block of data

(max 32 bytes)
s32 i2c_smbus_read_block_data(const struct i2c_client *client, u8 command, u8 *values);
s32 i2c_smbus_write_block_data(const struct i2c_client *client, u8 command, u8 length,
const u8 *values);
Write a command byte, and read or write a block of data (no

limit)
s32 i2c_smbus_read_i2c_block_data(const struct i2c_client *client, u8 command, u8
length, u8 *values);
s32 i2c_smbus_write_i2c_block_data(const struct i2c_client *client, u8 command, u8
length, const u8 *values);
186/475
I2C functionality
Not all I2C controllers support all functionalities.
The I2C controller drivers therefore tell the I2C core which
functionalities they support.
An I2C device driver must check that the functionalities they

need are provided by the I2C controller in use on the system.
The i2c_check_functionality() function allows to make

such a check.
Examples of functionalities: I2C_FUNC_I2C to be able to use

the raw I2C functions, I2C_FUNC_SMBUS_BYTE_DATA to be able
to use SMBus commands to write a command and read/write
one byte of data.
See include/uapi/linux/i2c.h for the full list of existing

functionalities.
187/475
References
http://en.wikipedia.org/wiki/I2C, general presentation

of the I2C protocol
Documentation/i2c/, details about the Linux support for I2C
writing-clients, how to write I2C device drivers

instantiating-devices, how to instantiate devices
smbus-protocol, details on the SMBus functions
functionality, how the functionality mechanism works
and many more documentation les
http://free-electrons.com/pub/video/2012/elce/elce2012-anders-board-bringup-i2c.webm, excellent talk: You,

me and I2C from David Anders at ELCE 2012.
188/475
Introduction to pin muxing
Introduction to pin
muxing
Embedded Linux
Experts
free electrons
189/475
What is pin muxing?
Modern SoCs (System on Chip) include more and more

hardware blocks, many of which need to interface with the
outside world using pins.
However, the physical size of the chips remains small, and

therefore the number of available pins is limited.
For this reason, not all of the internal hardware block features
can be exposed on the pins simultaneously.
The pins are multiplexed: they expose either the functionality

of hardware block A or the functionality of hardware block B.
This multiplexing is usually software congurable.
190/475
Pin muxing diagram
191/475
Pin muxing in the Linux kernel
Since Linux 3.2, a pinctrl subsystem has been added.

This subsystem, located in drivers/pinctrl provides a
generic subsystem to handle pin muxing. It oers:
A pin muxing driver interface, to implement the

system-on-chip specic drivers that congure the muxing.
A pin muxing consumer interface, for device drivers.
Most pinctrl drivers provide a Device Tree binding, and the

pin muxing must be described in the Device Tree.
The exact Device Tree binding depends on each driver. Each

binding is documented in
Documentation/devicetree/bindings/pinctrl.
192/475
pinctrl subsystem diagram
193/475
Device Tree binding for consumer devices
The devices that require certains pins to be muxed will use

the pinctrl-<x> and pinctrl-names Device Tree properties.
The pinctrl-0, pinctrl-1, pinctrl-<x> properties link to a
pin conguration for a given state of the device.
The pinctrl-names property associates a name to each state.
The name default is special, and is automatically selected by
a device driver, without having to make an explicit pinctrl
function call.
In most cases, the following is sucient:
i2c@11000 {
pinctrl-0 = <&pmx_twsi0>;
...
};
See
Documentation/devicetree/bindings/pinctrl/pinctrlbindings.txt for details.
194/475
Dening pinctrl congurations
The dierent pinctrl congurations must be dened as child

nodes of the main pinctrl device (which controls the muxing
of pins).
The congurations may be dened at:
the SoC level (.dtsi le), for pin congurations that are often
shared between multiple boards
at the board level (.dts le) for congurations that are board
specic.
The pinctrl-<x> property of the consumer device points to

the pin conguration it needs through a DT phandle.
The description of the congurations is specic to each pinctrl

driver. See Documentation/devicetree/bindings/pinctrl
for the DT bindings documentation.
195/475
Example on OMAP/AM33xx
On OMAP/AM33xx, the
pinctrl-single driver is used.
It is common between multiple
SoCs and simply allows to
congure pins by writing a
value to a register.
In each pin conguration,
a pinctrl-single,pins
value gives a list of
(register, value) pairs
needed to congure the
pins.
To know the correct values,
one must use the SoC and
board datasheets.
am33xx_pinmux: pinmux@44e10800 {
i2c0_pins: pinmux_i2c0_pins {
pinctrl-single,pins = <
/* i2c0_sda.i2c0_sda */
0x188 (PIN_INPUT_PULLUP | MUX_MODE0)
/* i2c0_scl.i2c0_scl */
0x18c (PIN_INPUT_PULLUP | MUX_MODE0)
>;
};
};
i2c0: i2c@44e0b000 {
pinctrl-0 = <&i2c0_pins>;
status = "okay";
tps: tps@2d {
reg = <0x2d>;
};
};
196/475
Example on Allwinner SoC
197/475
Practical lab - Communicate with the Nunchuk
Congure the pinmuxing for the

I2C bus used to communicate with
the Nunchuk
Validate that the I2C

communication works with user
space tools.
Extend the I2C driver started in the

previous lab to communicate with
the Nunchuk.
198/475
Kernel frameworks for device drivers
Kernel frameworks
for device drivers
Embedded Linux
Experts
free electrons
199/475
Kernel and Device Drivers

In Linux, a driver is always
interfacing with:
a framework that allows the

driver to expose the
hardware features to user
space applications.
a bus infrastructure, part

of the device model, to
detect/communicate with
the hardware.
This section focuses on the

kernel frameworks, while the
device model was covered earlier
in this training.
200/475
User space vision of devices
201/475
Types of devices
Under Linux, there are essentially three types of devices:
Network devices. They are represented as network

interfaces, visible in user space using ifconfig.
Block devices. They are used to provide user space

applications access to raw storage devices (hard disks, USB
keys). They are visible to the applications as device les in
/dev.
Character devices. They are used to provide user space

applications access to all other types of devices (input, sound,
graphics, serial, etc.). They are also visible to the applications
as device les in /dev.
Most devices are character devices, so we will study these in

more details.
202/475
Major and minor numbers
Within the kernel, all block and character devices are

identied using a major and a minor number.
The major number typically indicates the family of the device.
The minor number typically indicates the number of the

device (when there are for example several serial ports)
Most major and minor numbers are statically allocated, and

identical across all Linux systems.
They are dened in Documentation/devices.txt.
203/475
Devices: everything is a le
A very important Unix design decision was to represent most

system objects as les
It allows applications to manipulate all system objects with

the normal le API (open, read, write, close, etc.)
So, devices had to be represented as les to the applications
This is done through a special artifact called a device le
It is a special type of le, that associates a le name visible to

user space applications to the triplet (type, major, minor) that
the kernel understands
All device les are by convention stored in the /dev directory
204/475
Device les examples

Example of device les in a Linux system
$ ls -l /dev/ttyS0 /dev/tty1 /dev/sda1 /dev/sda2
brw-rw---- 1 root disk
8, 1 2011-05-27 08:56
brw-rw---- 1 root disk
8, 2 2011-05-27 08:56
crw------- 1 root root
4, 1 2011-05-27 08:57
crw-rw---- 1 root dialout 4, 64 2011-05-27 08:56
crw-rw-rw- 1 root root
1, 5 2011-05-27 08:56
/dev/zero
/dev/sda1
/dev/sda2
/dev/tty1
/dev/ttyS0
/dev/zero
Example C code that uses the usual le API to write data to a

serial port
int fd;
fd = open("/dev/ttyS0", O_RDWR);
write(fd, "Hello", 5);
close(fd);
205/475
Creating device les
Before Linux 2.6.32, on basic Linux systems, the device les

had to be created manually using the mknod command
mknod /dev/<device> [c|b] major minor

Needed root privileges
Coherency between device les and devices handled by the
kernel was left to the system developer
The devtmpfs virtual lesystem can be mounted on /dev and

contains all the devices known to the kernel. The
CONFIG_DEVTMPFS_MOUNT kernel conguration option makes
the kernel mount it automatically at boot time, except when
booting on an initramfs.
206/475
Character drivers
207/475
A character driver in the kernel
From the point of view of an application, a character device is

essentially a le.
The driver of a character device must therefore implement

operations that let applications think the device is a le:
open, close, read, write, etc.
In order to achieve this, a character driver must implement

the operations described in the struct file_operations
structure and register them.
The Linux lesystem layer will ensure that the driver's

operations are called when a user space application makes the
corresponding system call.
208/475
From user space to the kernel: character devices
209/475
File operations
Here are the most important operations for a character driver.

All of them are optional.
#include <linux/fs.h>
struct file_operations {
ssize_t (*read) (struct file *, char __user *,
size_t, loff_t *);
ssize_t (*write) (struct file *, const char __user *,
size_t, loff_t *);
long (*unlocked_ioctl) (struct file *, unsigned int,
unsigned long);
int (*mmap) (struct file *, struct vm_area_struct *);
int (*open) (struct inode *, struct file *);
int (*release) (struct inode *, struct file *);
};
210/475
open() and release()
int foo_open(struct inode *i, struct file *f)
Called when user space opens the device le.

struct inode is a structure that uniquely represents a le in
the system (be it a regular le, a directory, a symbolic link, a
character or block device)
struct file is a structure created every time a le is opened.
Several le structures can point to the same inode structure.
Contains information like the current position, the opening

mode, etc.
Has a void *private_data pointer that one can freely use.
A pointer to the file structure is passed to all other
operations
int foo_release(struct inode *i, struct file *f)
Called when user space closes the le.
211/475
read()
ssize_t foo_read(struct file *f, char __user *buf,

size_t sz, loff_t *off)
Called when user space uses the read() system call on the
device.
Must read data from the device, write at most sz bytes to the
user space buer buf, and update the current position in the
le off. f is a pointer to the same le structure that was
passed in the open() operation
Must return the number of bytes read.
0 is usually interpreted by userspace as the end of the le.
On UNIX, read() operations typically block when there isn't
enough data to read from the device
212/475
write()
ssize_t foo_write(struct file *f,

const char __user *buf, size_t sz, loff_t *off)
Called when user space uses the write() system call on the
device
The opposite of read, must read at most sz bytes from buf,
write it to the device, update off and return the number of
bytes written.
213/475
Exchanging data with user space 1/3
Kernel code isn't allowed to directly access user space

memory, using memcpy() or direct pointer dereferencing
Doing so does not work on some architectures

If the address passed by the application was invalid, the
application would segfault.
Never trust user space. A malicious application could pass a
kernel address which you could overwrite with device data
(read case), or which you could dump to the device (write
case).
To keep the kernel code portable, secure, and have proper

error handling, your driver must use special kernel functions to
exchange data with user space.
214/475
A single value
get_user(v, p);
put_user(v, p);
The kernel variable v gets the value pointed by the user space
pointer p
The value pointed by the user space pointer p is set to the
contents of the kernel variable v.
A buer
unsigned long copy_to_user(void __user *to,

const void *from, unsigned long n);
unsigned long copy_from_user(void *to,

const void __user *from, unsigned long n);
The return value must be checked. Zero on success, non-zero

on failure. If non-zero, the convention is to return -EFAULT.
215/475
216/475
Zero copy access to user memory
Having to copy data to or from an intermediate kernel buer

can become expensive when the amount of data to transfer is
large (video).
Zero copy options are possible:
mmap() system call to allow user space to directly access

memory mapped I/O space. See our mmap() chapter.
get_user_pages_fast() to get a mapping to user pages
without having to copy them. See http://j.mp/1sML7lP
(Kernel API doc). This API is more complex to use though.
217/475
unlocked_ioctl()
long unlocked_ioctl(struct file *f,

unsigned int cmd, unsigned long arg)
Associated to the ioctl() system call.

Called unlocked because it didn't hold the Big Kernel Lock
(gone now).
Allows to extend the driver capabilities beyond the limited
read/write API.
For example: changing the speed of a serial port, setting video
output format, querying a device serial number...
cmd is a number identifying the operation to perform
arg is the optional argument passed as third argument of the
ioctl() system call. Can be an integer, an address, etc.
The semantic of cmd and arg is driver-specic.
218/475
ioctl() example: kernel side
static long phantom_ioctl(struct file *file, unsigned int cmd,

unsigned long arg)
{
struct phm_reg r;
void __user *argp = (void __user *)arg;
switch (cmd) {
case PHN_SET_REG:
if (copy_from_user(&r, argp, sizeof(r)))
return -EFAULT;
/* Do something */
break;
case PHN_GET_REG:
if (copy_to_user(argp, &r, sizeof(r)))
return -EFAULT;
/* Do something */
break;
default:
return -ENOTTY;
}
return 0; }
Selected excerpt from drivers/misc/phantom.c
219/475
Ioctl() Example: Application Side

int main(void)
{
int fd, ret;
struct phm_reg reg;
fd = open("/dev/phantom");
assert(fd > 0);
reg.field1 = 42;
reg.field2 = 67;
ret = ioctl(fd, PHN_SET_REG, & reg);
assert(ret == 0);
return 0;
}
220/475
The concept of kernel frameworks
221/475
Beyond character drivers: kernel frameworks
Many device drivers are not implemented directly as character

drivers
They are implemented under a framework, specic to a given
device type (framebuer, V4L, serial, etc.)
The framework allows to factorize the common parts of drivers

for the same type of devices
From user space, they are still seen as character devices by the
applications
The framework allows to provide a coherent user space
interface (ioctl, etc.) for every type of device, regardless of
the driver
222/475
Kernel Frameworks
223/475
Example: Framebuer Framework
Kernel option CONFIG_FB
menuconfig FB
Implemented in C les in drivers/video/fbdev/core

Implements a single character driver and denes the
user/kernel API
tristate "Support for frame buffer devices"
First part of include/linux/fb.h
Denes the set of operations a framebuer driver must

implement and helper functions for the drivers
struct fb_ops
Second part of include/linux/fb.h
224/475
Framebuer driver operations
Here are the operations a framebuer driver can or must

implement, and dene them in a struct fb_ops structure
static struct fb_ops xxxfb_ops = {
.owner = THIS_MODULE,
.fb_open = xxxfb_open,
.fb_read = xxxfb_read,
.fb_write = xxxfb_write,
.fb_release = xxxfb_release,
.fb_check_var = xxxfb_check_var,
.fb_set_par = xxxfb_set_par,
.fb_setcolreg = xxxfb_setcolreg,
.fb_blank = xxxfb_blank,
.fb_pan_display = xxxfb_pan_display,
/* Needed !!! */
.fb_fillrect = xxxfb_fillrect,
.fb_copyarea = xxxfb_copyarea,
/* Needed !!! */
.fb_imageblit = xxxfb_imageblit, /* Needed !!! */
.fb_cursor = xxxfb_cursor,
/* Optional !!! */
.fb_rotate = xxxfb_rotate,
.fb_sync = xxxfb_sync,
.fb_ioctl = xxxfb_ioctl,
.fb_mmap = xxxfb_mmap,
};
225/475
Framebuer driver code
In the probe() function, registration of the framebuer device

and operations
static int xxxfb_probe (struct pci_dev *dev,
const struct pci_device_id *ent)
{
struct fb_info *info;
[...]
info = framebuffer_alloc(sizeof(struct xxx_par), device);
[...]
info->fbops = &xxxfb_ops;
[...]
if (register_framebuffer(info) > 0)
return -EINVAL;
[...]
}
register_framebuffer() will create the character device

that can be used by user space applications with the generic
framebuer API.
226/475
Driver-specic Data Structure
Each framework denes a structure that a device driver must

register to be recognized as a device in this framework
struct uart_port for serial ports, struct netdev for network

devices, struct fb_info for framebuers, etc.
In addition to this structure, the driver usually needs to store

additional information about its device
This is typically done
By subclassing the appropriate framework structure

By storing a reference to the appropriate framework structure
Or by including your information in the framework structure
227/475
Driver-specic Data Structure Examples 1/2
i.MX serial driver: struct imx_port is a subclass of

struct uart_port
struct imx_port {
struct uart_port port;
struct timer_list timer;
unsigned int old_status;
int txirq, rxirq, rtsirq;
unsigned int have_rtscts:1;
[...]
};
ds1305 RTC driver: struct ds1305 has a reference to

struct rtc_device
struct ds1305 {
struct spi_device
struct rtc_device
[...]
};
*spi;
*rtc;
228/475
Driver-specic Data Structure Examples 2/2
rtl8150 network driver: struct rtl8150 has a reference to

struct net_device and is allocated within that framework
structure.
struct rtl8150 {
unsigned long flags;
struct usb_device *udev;
struct tasklet_struct tl;
[...]
};
229/475
Links between structures 1/4
The framework typically contains a struct device * pointer

that the driver must point to the corresponding
struct device
It's the relation between the logical device (for example a

network interface) and the physical device (for example the
USB network adapter)
The device structure also contains a void * pointer that the

driver can freely use.
It's often used to link back the device to the higher-level

structure from the framework.
It allows, for example, from the struct platform_device
structure, to nd the structure describing the logical device
230/475
static int serial_imx_probe(struct platform_device *pdev)

{
struct imx_port *sport;
[...]
sport = devm_kzalloc(&pdev->dev, sizeof(*sport), GFP_KERNEL);
[...]
/* setup the link between uart_port and the struct
* device inside the platform_device */
sport->port.dev = &pdev->dev;
[...]
/* setup the link between the struct device inside
* the platform device to the imx_port structure */
platform_set_drvdata(pdev, sport);
[...]
uart_add_one_port(&imx_reg, &sport->port);
}
static int serial_imx_remove(struct platform_device *pdev)
{
/* retrieve the imx_port from the platform_device */
struct imx_port *sport = platform_get_drvdata(pdev);
[...]
uart_remove_one_port(&imx_reg, &sport->port);
[...]
}
231/475
Device Model: per device allocations (1)
With the Device Model, structures need to be allocated for

each new device.
Such structures then need to be released:
When the module is removed,

When devices are removed (on a bus which can detect this),
In the probe() routine, before returning after a failure.
232/475
Device Model: per device allocations (2)

devm_() functions have been introduced to help managing such
allocations.
They now exist in many parts of the kernel
Allocations are now attached to a struct device structure.
Example: call to devm_kzalloc() in the previous example.
Before each device is destroyed, the resources attached to it

are automatically freed. This is not limited to memory
allocation. This can also apply to registration of interrupts,
for example.
This greatly simplies remove() functions and error

management code.
We will see many devm_ functions in the subsystems that we

will study.
See Documentation/driver-model/devres.txt for

implementation details and available functions.
233/475

static int ds1305_probe(struct spi_device *spi)
{
struct ds1305
*ds1305;
[...]
/* set up driver data */
ds1305 = devm_kzalloc(&spi->dev, sizeof(*ds1305), GFP_KERNEL);
if (!ds1305)
return -ENOMEM;
ds1305->spi = spi;
spi_set_drvdata(spi, ds1305);
[...]
/* register RTC ... from here on, ds1305->ctrl needs locking */
ds1305->rtc = devm_rtc_device_register(&spi->dev, "ds1305",
&ds1305_ops, THIS_MODULE);
[...]
}
static int ds1305_remove(struct spi_device *spi)
{
struct ds1305 *ds1305 = spi_get_drvdata(spi);
[...]
}
234/475

static int rtl8150_probe(struct usb_interface *intf,
const struct usb_device_id *id)
{
struct usb_device *udev = interface_to_usbdev(intf);
rtl8150_t *dev;
netdev = alloc_etherdev(sizeof(rtl8150_t));
dev = netdev_priv(netdev);
[...]
dev->udev = udev;
dev->netdev = netdev;
[...]
usb_set_intfdata(intf, dev);
SET_NETDEV_DEV(netdev, &intf->dev);
[...]
}
static void rtl8150_disconnect(struct usb_interface *intf)
{
rtl8150_t *dev = usb_get_intfdata(intf);
[...]
}
235/475
The input subsystem
The input
subsystem
Embedded Linux
Experts
free electrons
236/475
What is the input subsystem?
The input subsystem takes care of all the input events coming
from the human user.
Initially written to support the USB HID (Human Interface

Device) devices, it quickly grew up to handle all kind of inputs
(using USB or not): keyboards, mice, joysticks, touchscreens,
etc.
The input subsystem is split in two parts:
Device drivers: they talk to the hardware (for example via

USB), and provide events (keystrokes, mouse movements,
touchscreen coordinates) to the input core
Event handlers: they get events from drivers and pass them
where needed via various interfaces (most of the time through
evdev)
In user space it is usually used by the graphic stack such as

X.Org, Wayland or Android's InputManager.
237/475
Input subsystem diagram
238/475
Input subsystem overview
Kernel option CONFIG_INPUT
menuconfig INPUT
Implemented in drivers/input/
input.c, input-polldev.c, evbug.c
Implements a single character driver and denes the

user/kernel API
tristate "Generic input layer (needed for keyboard,

mouse, ...)"
include/uapi/linux/input.h
Denes the set of operations a input driver must implement

and helper functions for the drivers
struct input_dev for the device driver part

struct input_handler for the event handler part
include/linux/input.h
239/475
Input subsystem API 1/3

An input device is described by a very long struct input_dev
structure, an excerpt is:
.
struct input_dev {
const char *name;
[...]
unsigned long evbit[BITS_TO_LONGS(EV_CNT)];
unsigned long keybit[BITS_TO_LONGS(KEY_CNT)];
[...]
int (*getkeycode)(struct input_dev *dev,
struct input_keymap_entry *ke);
[...]
int (*open)(struct input_dev *dev);
[...]
int (*event)(struct input_dev *dev, unsigned int type,
unsigned int code, int value);
[...]
};
Before being used it, this structure must be allocated and

initialized, typically with: struct input_dev *devm_input_
allocate_device(struct device *dev);
240/475
Depending on the type of events that will be generated, the

input bit elds evbit and keybit must be congured: For
example, for a button we only generate EV_KEY type events,
and from these only BTN_0 events code:
set_bit(EV_KEY, myinput_dev.evbit);
set_bit(BTN_0, myinput_dev.keybit);
set_bit() is an atomic operation allowing to set a particular

bit to 1 (explained later).
Once the input device is allocated and lled, the function to

register it is:
int input_register_device(struct input_dev *);
When the driver is unloaded, the input device will be

unregistered using:
void input_unregister_device(struct input_dev *);
241/475
The events are sent by the driver to the event handler using
input_event(struct input_dev *dev, unsigned int
type, unsigned int code, int value);
The event types are documented in

Documentation/input/event-codes.txt
An event is composed by one or several input data changes
(packet of input data changes) such as the button state, the
relative or absolute position along an axis, etc..
After submitting potentially multiple events, the input core
must be notied by calling:
void input_sync(struct input_dev *dev):
The input subsystem provides other wrappers such as
input_report_key(), input_report_abs(), ...
242/475
Polled input subclass
The input subsystem provides a subclass supporting simple

input devices that do not raise interrupts but have to be
periodically scanned or polled to detect changes in their state.
A polled input device is described by a

struct input_polled_dev structure:
struct input_polled_dev {
void *private;
void (*open)(struct input_polled_dev *dev);
void (*close)(struct input_polled_dev *dev);
void (*poll)(struct input_polled_dev *dev);
unsigned int poll_interval; /* msec */
unsigned int poll_interval_max; /* msec */
unsigned int poll_interval_min; /* msec */
struct input_dev *input;
/* private: */
struct delayed_work work;
}.
243/475
Polled input subsystem API
Allocating the struct input_polled_dev structure is done

using devm_input_allocate_polled_device()
Among the handlers of the struct input_polled_dev only

the poll() method is mandatory, this function polls the
device and posts input events.
The elds id, name, evkey and keybit of the input eld
must be initialized too.
If none of the poll_interval elds are lled then the default

poll interval is 500ms.
The device registration/unregistration is done with:
input_register_polled_device(struct input_polled_dev
*dev).
Unregistration is automatic after using
devm_input_allocate_polled_device()!
244/475
evdev user space interface
The main user space interface to input devices is the event

interface
Each input device is represented as a /dev/input/event<X>
character device
A user space application can use blocking and non-blocking
reads, but also select() (to get notied of events) after
opening this device.
Each read will return struct input_event structures of the
following format:
struct input_event {
struct timeval time;
unsigned short type;
unsigned short code;
unsigned int value;
};
A very useful application for input device testing is evtest,

from http://cgit.freedesktop.org/evtest/
245/475
Practical lab - Expose the Nunchuk to user space
Extend the Nunchuk driver to

expose the Nunchuk features to
user space applications, as an input
device.
Test the operation of the Nunchuk

using sample user space
applications.
246/475
Memory Management
Memory
Management
Embedded Linux
Experts
free electrons
247/475
Physical and Virtual Memory
248/475
Virtual Memory Organization
1GB reserved for kernel-space
Contains kernel code and core data

structures, identical in all address spaces
Most memory can be a direct mapping
of physical memory at a xed oset
Complete 3GB exclusive mapping

available for each user space process
Process code and data (program, stack,

...)
Memory-mapped les
Not necessarily mapped to physical
memory (demand fault paging used for
dynamic mapping to physical memory
pages)
Diers from one address space to
another
249/475
Physical / virtual memory mapping
250/475
Accessing more physical memory
Only less than 1GB memory addressable directly through

kernel virtual address space
If more physical memory is present on the platform, part of
the memory will not be accessible by kernel space, but can be
used by user space
To allow the kernel to access more physical memory:
Change 1GB/3GB memory split (2GB/2GB)

(CONFIG_VMSPLIT_3G) reduces total memory available for
each process
Change for a 64 bit architecture ;-) See
Documentation/x86/x86_64/mm.txt for an example.
Activate highmem support if available for your architecture:
Allows kernel to map parts of its non-directly accessible

memory
Mapping must be requested explicitly
Limited addresses ranges reserved for this usage
See http://lwn.net/Articles/75174/ for useful

explanations
251/475
Notes on user space memory
New user space memory is allocated either from the already

allocated process memory, or using the mmap system call
Note that memory allocated may not be physically allocated:
Kernel uses demand fault paging to allocate the physical page

(the physical page is allocated when access to the virtual
address generates a page fault)
... or may have been swapped out, which also induces a page
fault
User space memory allocation is allowed to over-commit

memory (more than available physical memory) can lead to
out of memory
OOM killer kicks in and selects a process to kill to retrieve

some memory. That's better than letting the system freeze.
252/475
Back to kernel memory
Kernel memory allocators (see following slides) allocate

physical pages, and kernel allocated memory cannot be
swapped out, so no fault handling required for kernel memory.
Most kernel memory allocation functions also return a kernel

virtual address to be used within the kernel space.
Kernel memory low-level allocator manages pages. This is the

nest granularity (usually 4 KB, architecture dependent).
However, the kernel memory management handles smaller

memory allocations through its allocator (see SLAB allocators
used by kmalloc()).
253/475
Allocators in the Kernel
254/475
Page Allocator
Appropriate for medium-size allocations
A page is usually 4K, but can be made greater in some

architectures (sh, mips: 4, 8, 16 or 64 KB, but not
congurable in x86 or arm).
Buddy allocator strategy, so only allocations of power of two

number of pages are possible: 1 page, 2 pages, 4 pages, 8
pages, 16 pages, etc.
Typical maximum size is 8192 KB, but it might depend on the

kernel conguration.
The allocated area is virtually contiguous (of course), but also
physically contiguous. It is allocated in the identity-mapped
part of the kernel memory space.
This means that large areas may not be available or hard to

retrieve due to physical memory fragmentation.
255/475
Page Allocator API: Get free pages
unsigned long get_zeroed_page(int flags)
unsigned long __get_free_page(int flags)
Returns the virtual address of a free page, initialized to zero

flags: see the next pages for details.
Same, but doesn't initialize the contents
unsigned long __get_free_pages(int flags,

unsigned int order)
Returns the starting virtual address of an area of several

contiguous pages in physical RAM, with order being
log2(number_of_pages).Can be computed from the size with
the get_order() function.
256/475
Page Allocator API: Free Pages
void free_page(unsigned long addr)
Frees one page.
void free_pages(unsigned long addr,

unsigned int order)
Frees multiple pages. Need to use the same order as in

allocation.
257/475
Page Allocator Flags

The most common ones are:
GFP_KERNEL
GFP_ATOMIC
RAM allocated from code which is not allowed to block

(interrupt handlers or critical sections). Never blocks, allows to
access emergency pools, but can fail if no free memory is
readily available.
GFP_DMA
Standard kernel memory allocation. The allocation may block

in order to nd enough available memory. Fine for most needs,
except in interrupt handler context.
Allocates memory in an area of the physical memory usable for

DMA transfers. See our DMA chapter.
Others are dened in include/linux/gfp.h
258/475
SLAB Allocator 1/2
The SLAB allocator allows to create caches, which contains a

set of objects of the same size
The object size can be smaller or greater than the page size
The SLAB allocator takes care of growing or reducing the size

of the cache as needed, depending on the number of allocated
objects. It uses the page allocator to allocate and free pages.
SLAB caches are used for data structures that are present in
many many instances in the kernel: directory entries, le
objects, network packet descriptors, process descriptors, etc.
See /proc/slabinfo
They are rarely used for individual drivers.
See include/linux/slab.h for the API
259/475
SLAB Allocator 2/2
260/475
Dierent SLAB Allocators
There are three dierent, but API compatible,

implementations of a SLAB allocator in the Linux kernel. A
particular implementation is chosen at conguration time.
SLAB: legacy, well proven allocator.

Still the default in most ARM defconfig les.
SLOB: much simpler. More space ecient but doesn't scale
well. Saves a few hundreds of KB in small systems (depends
on CONFIG_EXPERT)
Linux 4.4 on ARM: used in 5 defconfig les
SLUB: more recent and simpler than SLAB, scaling much
better (in particular for huge systems) and creating less
fragmentation.
Linux 4.4 on ARM: used in 0 defconfig les
261/475
kmalloc Allocator
The kmalloc allocator is the general purpose memory allocator

in the Linux kernel
For small sizes, it relies on generic SLAB caches, named
kmalloc-XXX in /proc/slabinfo
For larger sizes, it relies on the page allocator
The allocated area is guaranteed to be physically contiguous
The allocated area size is rounded up to the size of the
smallest SLAB cache in which it can t (while using the SLAB
allocator directly allows to have more exibility)
It uses the same ags as the page allocator (GFP_KERNEL,
GFP_ATOMIC, GFP_DMA, etc.) with the same semantics.
Maximum sizes, on x86 and arm (see http://j.mp/YIGq6W):
- Per allocation: 4 MB
- Total allocations: 128 MB
Should be used as the primary allocator unless there is a
strong reason to use another one.
262/475
kmalloc API 1/2
#include <linux/slab.h>
void *kmalloc(size_t size, int flags);
void kfree(const void *objp);
Allocate size bytes, and return a pointer to the area (virtual

address)
size: number of bytes to allocate
flags: same ags as the page allocator
Free an allocated area
Example: (drivers/infiniband/core/cache.c)
struct ib_update_work *work;
work = kmalloc(sizeof *work, GFP_ATOMIC);
...
kfree(work);
263/475
kmalloc API 2/2
void *kzalloc(size_t size, gfp_t flags);
Allocates a zero-initialized buer
void *kcalloc(size_t n, size_t size, gfp_t flags);
Allocates memory for an array of n elements of size size, and

zeroes its contents.
void *krealloc(const void *p, size_t new_size, gfp_t flags);
Changes the size of the buer pointed by p to new_size, by

reallocating a new buer and copying the data, unless
new_size ts within the alignment of the existing buer.
264/475
devm_ kmalloc functions
Allocations with automatic freeing when the corresponding device

or module is unprobed.
void *devm_kmalloc(struct device *dev, size_t size, int flags);
void *devm_kzalloc(struct device *dev, size_t size, int flags);
void *devm_kcalloc(struct device *dev, size_t n, size_t size, gfp_t flags);
void *devm_kfree(struct device *dev, void *p);
Useful to immediately free an allocated buer
For use in probe() functions.
265/475
vmalloc Allocator
The vmalloc() allocator can be used to obtain virtually

contiguous memory zones, but not physically contiguous. The
requested memory size is rounded up to the next page.
The allocated area is in the kernel space part of the address
space, but outside of the identically-mapped area
Allocations of fairly large areas is possible (almost as big as
total available memory, see http://j.mp/YIGq6W again),
since physical memory fragmentation is not an issue, but areas
cannot be used for DMA, as DMA usually requires physically
contiguous buers.
Example use: to allocate kernel buers to load module code.
API in include/linux/vmalloc.h
void *vmalloc(unsigned long size);
Returns a virtual address
void vfree(void *addr);
266/475
Kernel memory debugging
Kmemcheck
Dynamic checker for access to uninitialized memory.

Only available on x86 so far (Linux 4.9 status), but will help to
improve architecture independent code anyway.
See Documentation/kmemcheck.txt for details.
Kmemleak
Dynamic checker for memory leaks

This feature is available for all architectures.
See Documentation/kmemleak.txt for details.
Both have a signicant overhead. Only use them in development!
267/475
I/O Memory and Ports
I/O Memory and

Ports
Embedded Linux
Experts
free electrons
268/475
Port I/O vs. Memory-Mapped I/O
MMIO
Same address bus to address memory and I/O devices

Access to the I/O devices using regular instructions
Most widely used I/O method across the dierent architectures
supported by Linux
PIO
Dierent address spaces for memory and I/O devices

Uses a special class of CPU instructions to access I/O devices
Example on x86: IN and OUT instructions
269/475
MMIO vs PIO
270/475
Requesting I/O ports
Tells the kernel which driver is using which I/O ports
Allows to prevent other drivers from using the same I/O ports,
but is purely voluntary.
struct resource *request_region(

unsigned long start,
unsigned long len,
char *name);
Tries to reserve the given region and returns NULL if

unsuccessful.
request_region(0x0170, 8, "ide1");
void release_region(
unsigned long len);
271/475
/proc/ioports example (x86)

0000-001f
0020-0021
0040-0043
0050-0053
0070-0077
0080-008f
00a0-00a1
00c0-00df
00f0-00ff
0170-0177
01f0-01f7
0376-0376
03f6-03f6
03f8-03ff
0800-087f
...
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
dma1
pic1
timer0
timer1
rtc
dma page reg
pic2
dma2
fpu
ide1
ide0
ide1
ide0
serial
0000:00:1f.0
272/475
Accessing I/O ports
Functions to read/write bytes (b), word (w) and longs (l) to

I/O ports:
unsigned in[bwl](unsigned long port)
void out[bwl](value, unsigned long port)
And the strings variants: often more ecient than the

corresponding C loop, if the processor supports such
operations!
void ins[bwl](unsigned port, void *addr,

unsigned long count)
void outs[bwl](unsigned port, void *addr,

unsigned long count)
Examples
read 8 bits
oldlcr = inb(baseio + UART_LCR)
write 8 bits
outb(MOXA_MUST_ENTER_ENCHANCE, baseio + UART_LCR)
273/475
Requesting I/O memory
Functions equivalent to request_region() and

release_region(), but for I/O memory.
struct resource *request_mem_region(

unsigned long len,
char *name);
void release_mem_region(
unsigned long len);
274/475
/proc/iomem example
00000000-0009efff
0009f000-0009ffff
000a0000-000bffff
000c0000-000cffff
000f0000-000fffff
00100000-3ffadfff
00100000-0030afff
0030b000-003b4bff
3ffae000-3fffffff
40000000-400003ff
40001000-40001fff
40400000-407fffff
40800000-40bfffff
a0000000-a0000fff
e8000000-efffffff
...
:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
System RAM
reserved
Video RAM area
Video ROM
System ROM
System RAM
Kernel code
Kernel data
reserved
0000:00:1f.1
0000:02:01.0
PCI CardBus #03
PCI CardBus #03
pcmcia_socket0
PCI Bus #01
275/475
Mapping I/O memory in virtual memory
Load/store instructions work with virtual addresses
To access I/O memory, drivers need to have a virtual address

that the processor can handle, because I/O memory is not
mapped by default in virtual memory.
The ioremap function satises this need:

#include <asm/io.h>
void __iomem *ioremap(phys_addr_t phys_addr,
unsigned long size);
void iounmap(void __iomem *addr);
Caution: check that ioremap() doesn't return a NULL address!
276/475
ioremap()
ioremap(0xFFEBC00, 4096) = 0xCDEFA000

277/475
Managed API
Using request_mem_region() and ioremap() in device drivers is

now deprecated. You should use the below "managed" functions
instead, which simplify driver coding and error handling:
devm_ioremap()
devm_iounmap()
devm_ioremap_resource()
Takes care of both the request and remapping operations!
278/475
Accessing MMIO devices
Directly reading from or writing to addresses returned by

ioremap() (pointer dereferencing) may not work on some
architectures.
To do PCI-style, little-endian accesses, conversion being done

automatically
unsigned read[bwl](void *addr);
void write[bwl](unsigned val, void *addr);
To do raw access, without endianness conversion

unsigned __raw_read[bwl](void *addr);
void __raw_write[bwl](unsigned val, void *addr);
Example
32 bits write
__raw_writel(1 << KS8695_IRQ_UART_TX,
membase + KS8695_INTST);
279/475
Avoiding I/O access issues
Caching on I/O ports or memory already disabled
Use the macros, they do the right thing for your architecture
The compiler and/or CPU can reorder memory accesses,
which might cause troubles for your devices is they expect one
register to be read/written before another one.
Memory barriers are available to prevent this reordering

rmb() is a read memory barrier, prevents reads to cross the
barrier
wmb() is a write memory barrier
mb() is a read-write memory barrier
Starts to be a problem with CPUs that reorder instructions

and SMP.
See Documentation/memory-barriers.txt for details
280/475
/dev/mem
Used to provide user space applications with direct access to

physical addresses.
Usage: open /dev/mem and read or write at given oset.

What you read or write is the value at the corresponding
physical address.
Used by applications such as the X server to write directly to

device memory.
On x86, arm, arm64, tile, powerpc, unicore32, s390:

CONFIG_STRICT_DEVMEM option to restrict /dev/mem non-RAM
addresses, for security reasons (Linux 3.10 status).
281/475
Practical lab - I/O Memory and Ports
Add UART devices to the board

device tree
Access I/O registers to control the

device and send rst characters to
it.
282/475
The misc subsystem
The misc
subsystem
Embedded Linux
Experts
free electrons
283/475
Why a misc subsystem?
The kernel oers a large number of frameworks covering a

wide range of device types: input, network, video, audio, etc.
However, there are some devices that really do not t in any

of the existing frameworks.
These frameworks allow to factorize common functionality

between drivers and oer a consistent API to user space
applications.
Highly customized devices implemented in a FPGA, or other

weird devices for which implementing a complete framework is
not useful.
The drivers for such devices could be implemented directly as

raw character drivers (with cdev_init() and cdev_add()).
But there is a subsystem that makes this work a little bit
easier: the misc subsystem.
It is really only a thin layer above the character driver API.

Another advantage is that devices are integrated in the Device
Model (device les appearing in devtmpfs, which you don't
have with raw character devices).
284/475
Misc subsystem diagram
285/475
Misc subsystem API (1/2)
The misc subsystem API mainly provides two functions, to

register and unregister a single misc device:
int misc_register(struct miscdevice * misc);

void misc_deregister(struct miscdevice *misc);
A misc device is described by a struct miscdevice structure:

struct miscdevice {
int minor;
const char *name;
const struct file_operations *fops;
struct list_head list;
struct device *parent;
struct device *this_device;
const char *nodename;
umode_t mode;
};
286/475
Misc subsystem API (2/2)
The main elds to be lled in struct miscdevice are:
minor, the minor number for the device, or

MISC_DYNAMIC_MINOR to get a minor number automatically
assigned.
name, name of the device, which will be used to create the

device node if devtmpfs is used.
fops, pointer to a struct file_operations structure, that

describes which functions implement the read, write, ioctl,
etc. operations.
287/475
User space API for misc devices
misc devices are regular character devices

The operations they support in user space depends on the
operations the kernel driver implements:
The open() and close() system calls to open/close the

device.
The read() and write() system calls to read/write to/from
the device.
The ioctl() system call to call some driver-specic
operations.
288/475
Practical lab - Output-only serial port driver
Extend the driver started in the

previous lab by registering it into
the misc subsystem.
Implement serial output

functionality through the misc
subsystem.
Test serial output using user space

applications.
289/475
Processes, scheduling and interrupts
Processes,
scheduling and
interrupts
Embedded Linux
Experts
free electrons
290/475
Processes and scheduling
291/475
Process, thread?
Confusion about the terms process, thread and task

In Unix, a process is created using fork() and is composed of
An address space, which contains the program code, data,

stack, shared libraries, etc.
One thread, entity known by the scheduler.
Upon creation, a process contains one thread
Additional threads can be created inside an existing process,

using pthread_create()
They run in the same address space as the initial thread of the
process
They start executing a function passed as argument to
pthread_create()
292/475
Process, thread: kernel point of view
The kernel represents each thread running in the system by a

structure of type struct task_struct
From a scheduling point of view, it makes no dierence

between the initial thread of a process and all additional
threads created dynamically using pthread_create()
293/475
A thread life
294/475
Execution of system calls
The execution of system calls takes place in the context of the

thread requesting them.
295/475
Sleeping
296/475
Sleeping
Sleeping is needed when a process (user space or kernel space) is

waiting for data.
297/475
How to sleep 1/3
Must declare a wait queue, which will be used to store the list
of threads waiting for an event
Dynamic queue declaration:
Typically one queue per device managed by the driver

It's convenient to embed the wait queue inside a per-device
data structure.
Example from drivers/net/ethernet/marvell/mvmdio.c:
struct orion_mdio_dev {
...
wait_queue_head_t smi_busy_wait;
};
struct orion_mdio_dev *dev;
...
init_waitqueue_head(&dev->smi_busy_wait);
Static queue declaration:
Using a global variable when a global resource is sucient

DECLARE_WAIT_QUEUE_HEAD(module_queue);
298/475
How to sleep 2/3

Several ways to make a kernel process sleep
void wait_event(queue, condition);
int wait_event_killable(queue, condition);
Sleeps until the task is woken up and the given C expression is

true. Caution: can't be interrupted (can't kill the user space
process!)
Can be interrupted, but only by a fatal signal (SIGKILL).

Returns -ERESTARTSYS if interrupted.
int wait_event_interruptible(queue, condition);
Can be interrupted by any signal. Returns -ERESTARTSYS if

interrupted.
299/475
How to sleep 3/3
int wait_event_timeout(queue, condition, timeout);
Also stops sleeping when the task is woken up and the timeout
expired. Returns 0 if the timeout elapsed, non-zero if the
condition was met.
int wait_event_interruptible_timeout(queue,
condition, timeout);
Same as above, interruptible. Returns 0 if the timeout elapsed,

-ERESTARTSYS if interrupted, positive value if the condition
was met.
300/475
How to Sleep - Example
ret = wait_event_interruptible
(sonypi_device.fifo_proc_list,
kfifo_len(sonypi_device.fifo) != 0);
if (ret)
return ret;
301/475
Waking up!
Typically done by interrupt handlers when data sleeping processes

are waiting for become available.
wake_up(&queue);
Wakes up all processes in the wait queue
wake_up_interruptible(&queue);
Wakes up all processes waiting in an interruptible sleep on the

given queue
302/475
Exclusive vs. non-exclusive
wait_event_interruptible() puts a task in a non-exclusive

wait.
All non-exclusive tasks are woken up by wake_up() /

wake_up_interruptible()
wait_event_interruptible_exclusive() puts a task in an

exclusive wait.
wake_up() / wake_up_interruptible() wakes up all

non-exclusive tasks and only one exclusive task
wake_up_all() / wake_up_interruptible_all() wakes up
all non-exclusive and all exclusive tasks
Exclusive sleeps are useful to avoid waking up multiple tasks

when only one will be able to ``consume'' the event.
Non-exclusive sleeps are useful when the event can ``benet''

to multiple tasks.
303/475
Sleeping and waking up - Implementation

The scheduler doesn't keep
evaluating the sleeping condition!
wait_event(queue, cond);
The process is put in the
TASK_UNINTERRUPTIBLE
state.
wake_up(&queue);
All processes waiting in
queue are woken up, so they
get scheduled later and have
the opportunity to evaluate
the condition again and go
back to sleep if it is not met.
See include/linux/wait.h for

implementation details.
304/475
Interrupt Management
305/475
Registering an interrupt handler 1/2

The managed API is recommended:
int devm_request_irq(struct device *dev,
unsigned int irq,
irq_handler_t handler,
unsigned long irq_flags,
const char *devname,
void *dev_id);
device for automatic freeing at device or module release time.

irq is the requested IRQ channel. For platform devices, use
platform_get_irq() to retrieve the interrupt number.
handler is a pointer to the IRQ handler
irq_flags are option masks (see next slide)
devname is the registered name (for /proc/interrupts)
dev_id is an opaque pointer. It can typically be used to pass
a pointer to a per-device data structure. It cannot be NULL as
it is used as an identier for freeing interrupts on a shared line.
306/475
Releasing an interrupt handler
void devm_free_irq(struct device *dev,

unsigned int irq, void *dev_id);
Explicitly release an interrupt handler. Done automatically in

normal situations.
Dened in include/linux/interrupt.h
307/475
Registering an interrupt handler 2/2
Main irq_flags bit values

(can be combined, 0 when no ags are needed):
IRQF_SHARED
The interrupt channel can be shared by several devices.

When an interrupt is received, all the interrupt handlers
registered on the same interrupt line are called.
This requires a hardware status register telling whether an IRQ
was raised or not.
308/475
Interrupt handler constraints
No guarantee in which address space the system will be in

when the interrupt occurs: can't transfer data to and from
user space.
Interrupt handler execution is managed by the CPU, not by

the scheduler. Handlers can't run actions that may sleep,
because there is nothing to resume their execution. In
particular, need to allocate memory with GFP_ATOMIC.
Interrupt handlers are run with all interrupts disabled on the

local CPU (see http://lwn.net/Articles/380931).
Therefore, they have to complete their job quickly enough, to
avoiding blocking interrupts for too long.
309/475
/proc/interrupts on Raspberry Pi 2 (ARM, Linux 4.1)

.
CPU0
CPU1
CPU2
16:
0
0
0
32: 1660065960
0
0
49:
0
0
0
50:
0
0
0
65:
77339
0
0
66:
2
0
0
75:
1
0
0
77:
825261
0
0
82:
819926
0
0
83:
6
0
0
96:
0
0
0
97: 45040705 26523650 16191929
FIQ:
usb_fiq
IPI0:
0
0
0
IPI1:
0
0
0
IPI2: 34944338 35870609 37410637
IPI3:
9
10
9
IPI4:
3
3
1
IPI5:
0
0
0
IPI6:
0
0
0
IPI7:
0
0
0
Err:
0
CPU3
0 ARMCTRL
0 ARMCTRL 32
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
0 ARMCTRL
47339273
ARMCTRL
0
0
12127900
11
1
0
0
0
16 Edge bcm2708_fb dma

Edge dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1
49 Edge 3f200000.gpio:bank0
50 Edge 3f200000.gpio:bank1
65 Edge 3f00b880.mailbox
66 Edge VCHIQ doorbell
75 Edge
77 Edge DMA IRQ
82 Edge mmc0
83 Edge uart-pl011
96 Edge arch_timer
97 Edge arch_timer
CPU wakeup interrupts

Timer broadcast interrupts
Rescheduling interrupts
Function call interrupts
Single function call interrupts
CPU stop interrupts
IRQ work interrupts
completion interrupts
Note: interrupt numbers shown on the left-most column are virtual numbers
when the Device Tree is used. The real physical interrupt numbers can be seen
in /sys/kernel/debug/irq_domain_mapping.
310/475
Interrupt handler prototype
irqreturn_t foo_interrupt(int irq, void *dev_id)
irq, the IRQ number

dev_id, the per-device pointer that was passed to
devm_request_irq()
Return value
IRQ_HANDLED: recognized and handled interrupt

IRQ_NONE: used by the kernel to detect spurious interrupts,
and disable the interrupt line if none of the interrupt handlers
has handled the interrupt.
311/475
Typical interrupt handler's job
Acknowledge the interrupt to the device (otherwise no more

interrupts will be generated, or the interrupt will keep ring
over and over again)
Read/write data from/to the device
Wake up any process waiting for such data, typically on a

per-device wait queue:
wake_up_interruptible(&device_queue);
312/475
Threaded interrupts
In 2.6.30, support for threaded interrupts has been added to the
Linux kernel
The interrupt handler is executed inside a thread.
Allows to block during the interrupt handler, which is often
needed for I2C/SPI devices as the interrupt handler needs to
communicate with them.
Allows to set a priority for the interrupt handler execution,
which is useful for real-time usage of Linux
int devm_request_threaded_irq(
struct device *dev,
unsigned int irq,
irq_handler_t handler, irq_handler_t thread_fn
unsigned long flags, const char *name, void *dev);
handler, ``hard IRQ'' handler

thread_fn, executed in a thread
313/475
Top half and bottom half processing
Splitting the execution of interrupt handlers in 2 parts

Top half
This is the real interrupt handler, which should complete as

quickly as possible since all interrupts are disabled. It takes the
data out of the device and if substantial post-processing is
needed, schedule a bottom half to handle it.
Bottom half
Is the general Linux name for various mechanisms which allow

to postpone the handling of interrupt-related work.
Implemented in Linux as softirqs, tasklets or workqueues.
314/475
Top half and bottom half diagram
315/475
Softirqs
Softirqs are a form of bottom half processing
The softirqs handlers are executed with all interrupts enabled,

and a given softirq handler can run simultaneously on multiple
CPUs
They are executed once all interrupt handlers have completed,

before the kernel resumes scheduling processes, so sleeping is
not allowed.
The number of softirqs is xed in the system, so softirqs are

not directly used by drivers, but by complete kernel
subsystems (network, etc.)
The list of softirqs is dened in include/linux/interrupt.h:

HI, TIMER, NET_TX, NET_RX, BLOCK, BLOCK_IOPOLL, TASKLET,
SCHED, HRTIMER, RCU
The HI and TASKLET softirqs are used to execute tasklets
316/475
Tasklets
Tasklets are executed within the HI and TASKLET softirqs.

They are executed with all interrupts enabled, but a given
tasklet is guaranteed to execute on a single CPU at a time.
Tasklets are typically created with the tasklet_init()

function, when your driver manages multiple devices,
otherwise statically with DECLARE_TASKLET(). A tasklet is
simply implemented as a function. Tasklets can easily be used
by individual device drivers, as opposed to softirqs.
The interrupt handler can schedule tasklet execution with:
tasklet_schedule() to get it executed in the TASKLET softirq

tasklet_hi_schedule() to get it executed in the HI softirq
(higher priority)
317/475
Tasklet Example: simplied atmel_serial.c 1/2
/* The tasklet function */

static void atmel_tasklet_func(unsigned long data) {
struct uart_port *port = (struct uart_port *)data;
[...]
}
/* Registering the tasklet */
init function(...) {
[...]
tasklet_init(&atmel_port->tasklet,
atmel_tasklet_func, (unsigned long)port);
[...]
}
318/475
Tasklet Example: simplied atmel_serial.c 2/2
/* Removing the tasklet */

cleanup function(...) {
[...]
tasklet_kill(&atmel_port->tasklet);
[...]
}
/* Triggering execution of the tasklet */
somewhere function(...) {
tasklet_schedule(&atmel_port->tasklet);
}
319/475
Workqueues
Workqueues are a general mechanism for deferring work. It is

not limited in usage to handling interrupts.
The function registered as workqueue is executed in a thread,
which means:
All interrupts are enabled

Sleeping is allowed
A workqueue is registered with INIT_WORK() and typically

triggered with queue_work()
The complete API, in include/linux/workqueue.h provides

many other possibilities (creating its own workqueue threads,
etc.)
320/475
Interrupt management summary
Device driver
Interrupt handler
In the probe() function, for each device, use

devm_request_irq() to register an interrupt handler for the
device's interrupt channel.
Called when an interrupt is raised.
Acknowledge the interrupt
If needed, schedule a per-device tasklet taking care of handling
data.
Wake up processes waiting for the data on a per-device queue
Device driver
In the remove() function, for each device, the interrupt

handler is automatically unregistered.
321/475
Practical lab - Interrupts
Adding read capability to the

character driver developed earlier.
Register an interrupt handler for

each device.
Waiting for data to be available in

the read le operation.
Waking up the code when data are

available from the devices.
322/475
Concurrent Access to Resources: Locking
Concurrent Access
to Resources:
Locking
Embedded Linux
Experts
free electrons
323/475
Sources of concurrency issues
In terms of concurrency, the kernel has the same constraint as

a multi-threaded program: its state is global and visible in all
executions contexts
Concurrency arises because of
Interrupts, which interrupts the current thread to execute an

interrupt handler. They may be using shared resources
(memory addresses, hardware registers...)
Kernel preemption, if enabled, causes the kernel to switch from
the execution of one system call to another. They may be
using shared resources.
Multiprocessing, in which case code is really executed in
parallel on dierent processors, and they may be using shared
resources as well.
The solution is to keep as much local state as possible and for

the shared resources that can't be made local (such as
hardware ones), use locking.
324/475
Concurrency protection with locks
325/475
Linux mutexes
The kernel's main locking primitive
The process requesting the lock blocks when the lock is

already held. Mutexes can therefore only be used in contexts
where sleeping is allowed.
Mutex denition:
Initializing a mutex statically:
#include <linux/mutex.h>
DEFINE_MUTEX(name);
Or initializing a mutex dynamically:
void mutex_init(struct mutex *lock);
326/475
Locking and Unlocking Mutexes 1/2
void mutex_lock(struct mutex *lock);
int mutex_lock_killable(struct mutex *lock);
Tries to lock the mutex, sleeps otherwise.

Caution: can't be interrupted, resulting in processes you
cannot kill!
Same, but can be interrupted by a fatal (SIGKILL) signal. If

interrupted, returns a non zero value and doesn't hold the
lock. Test the return value!!!
int mutex_lock_interruptible(struct mutex *lock);
Same, but can be interrupted by any signal.
327/475
Locking and Unlocking Mutexes 2/2
int mutex_trylock(struct mutex *lock);
int mutex_is_locked(struct mutex *lock);
Never waits. Returns a non zero value if the mutex is not

available.
Just tells whether the mutex is locked or not.
void mutex_unlock(struct mutex *lock);
Releases the lock. Do it as soon as you leave the critical

section.
328/475
Spinlocks
Locks to be used for code that is not allowed to sleep

(interrupt handlers), or that doesn't want to sleep (critical
sections). Be very careful not to call functions which can
sleep!
Originally intended for multiprocessor systems
Spinlocks never sleep and keep spinning in a loop until the
lock is available.
Spinlocks cause kernel preemption to be disabled on the CPU
executing them.
The critical section protected by a spinlock is not allowed to
sleep.
329/475
Initializing Spinlocks
Statically
DEFINE_SPINLOCK(my_lock);
Dynamically
void spin_lock_init(spinlock_t *lock);
330/475
Using Spinlocks 1/2
Several variants, depending on where the spinlock is called:
void spin_lock(spinlock_t *lock);
void spin_unlock(spinlock_t *lock);
Doesn't disable interrupts. Used for locking in process context

(critical sections in which you do not want to sleep).
void spin_lock_irqsave(spinlock_t *lock,

unsigned long flags);
void spin_unlock_irqrestore(spinlock_t *lock,

unsigned long flags);
Disables / restores IRQs on the local CPU.

Typically used when the lock can be accessed in both process
and interrupt context, to prevent preemption by interrupts.
331/475
Using Spinlocks 2/2
void spin_lock_bh(spinlock_t *lock);
void spin_unlock_bh(spinlock_t *lock);
Disables software interrupts, but not hardware ones.

Useful to protect shared data accessed in process context and
in a soft interrupt (bottom half ).
No need to disable hardware interrupts in this case.
Note that reader / writer spinlocks also exist.
332/475
Spinlock example
Spinlock structure embedded into struct uart_port

struct uart_port {
spinlock_t lock;
/* Other fields */
};
Spinlock taken/released with protection against interrupts

static unsigned int ulite_tx_empty
(struct uart_port *port) {
unsigned long flags;
spin_lock_irqsave(&port->lock, flags);
/* Do something */
spin_unlock_irqrestore(&port->lock, flags);
}
333/475
Deadlock Situations
They can lock up your system. Make sure they never happen!
Don't call a function that can try to get access to the same
lock
Holding multiple locks is risky!
334/475
Kernel lock validator
From Ingo Molnar and Arjan van de Ven
Adds instrumentation to kernel locking code

Detect violations of locking rules during system life, such as:
Locks acquired in dierent order (keeps track of locking

sequences and compares them).
Spinlocks acquired in interrupt handlers and also in process
context when interrupts are enabled.
Not suitable for production systems but acceptable overhead in

development.
See Documentation/locking/lockdep-design.txt for

details
335/475
Alternatives to Locking
As we have just seen, locking can have a strong negative

impact on system performance. In some situations, you could
do without it.
By using lock-free algorithms like Read Copy Update (RCU).

RCU API available in the kernel (See
http://en.wikipedia.org/wiki/RCU).
When available, use atomic operations.
336/475
Atomic Variables 1/2
Useful when the shared resource is an integer value

Even an instruction like n++ is not guaranteed to be atomic on
all processors!
Atomic operations denitions
atomic_t
#include <asm/atomic.h>
Contains a signed integer (at least 24 bits)
Atomic operations (main ones)
Set or read the counter:
void atomic_set(atomic_t *v, int i);
int atomic_read(atomic_t *v);
Operations without return value:
void atomic_inc(atomic_t *v);
void atomic_dec(atomic_t *v);
void atomic_add(int i, atomic_t *v);
void atomic_sub(int i, atomic_t *v);
337/475
Atomic Variables 2/2
Similar functions testing the result:
int atomic_inc_and_test(...);
int atomic_dec_and_test(...);
int atomic_sub_and_test(...);
Functions returning the new value:
int atomic_inc_return(...);
int atomic_dec_return(...);
int atomic_add_return(...);
int atomic_sub_return(...);
338/475
Atomic Bit Operations
Supply very fast, atomic operations
On most platforms, apply to an unsigned long * type.
Apply to a void * type on a few others.

Set, clear, toggle a given bit:
void set_bit(int nr, unsigned long * addr);
void clear_bit(int nr, unsigned long * addr);
void change_bit(int nr, unsigned long * addr);
Test bit value:
int test_bit(int nr, unsigned long *addr);
Test and modify (return the previous value):
int test_and_set_bit(...);
int test_and_clear_bit(...);
int test_and_change_bit(...);
339/475
Practical lab - Locking
Add locking to the driver to

prevent concurrent accesses to
shared resources
340/475
Kernel Debugging
Kernel Debugging
Embedded Linux
Experts
free electrons
341/475
Debugging Using Messages (1)

Three APIs are available
The old printk(), no longer recommended for new debugging

messages
The pr_*() family of functions: pr_emerg(), pr_alert(),

pr_crit(), pr_err(), pr_warning(), pr_notice(),
pr_info(), pr_cont()
and the special pr_debug() (see next pages)
Dened in include/linux/printk.h
They take a classic format string with arguments
Example:
pr_info("Booting CPU %d\n", cpu);
[
202.350064] Booting CPU 1
342/475
Debugging Using Messages (2)
The dev_*() family of functions: dev_emerg(),

dev_alert(), dev_crit(), dev_err(), dev_warning(),
dev_notice(), dev_info()
and the special dev_dbg() (see next page)
They take a pointer to struct device as rst argument, and

then a format string with arguments
Dened in include/linux/device.h
To be used in drivers integrated with the Linux device model
Example:
dev_info(&pdev->dev, "in probe\n");
[
[
25.878382] feserial 48024000.serial: in probe

25.884873] feserial 481a8000.serial: in probe
343/475
pr_debug() and dev_dbg()
When the driver is compiled with DEBUG dened, all these

messages are compiled and printed at the debug level. DEBUG
can be dened by #define DEBUG at the beginning of the
driver, or using ccflags-$(CONFIG_DRIVER) += -DDEBUG in
the Makefile
When the kernel is compiled with CONFIG_DYNAMIC_DEBUG,
then these messages can dynamically be enabled on a per-le,
per-module or per-message basis
See Documentation/dynamic-debug-howto.txt for details

Very powerful feature to only get the debug messages you're
interested in.
When DEBUG is not dened and CONFIG_DYNAMIC_DEBUG is not

enabled, these messages are not compiled in.
344/475
Conguring The Priority
Each message is associated to a priority, ranging from 0 for

emergency to 7 for debug.
All the messages, regardless of their priority, are stored in the
kernel log ring buer
Typically accessed using the dmesg command
Some of the messages may appear on the console, depending

on their priority and the conguration of
The loglevel kernel parameter, which denes the priority

above which messages are displayed on the console. See
Documentation/kernel-parameters.txt for details.
The value of /proc/sys/kernel/printk, which allows to
change at runtime the priority above which messages are
displayed on the console. See
Documentation/sysctl/kernel.txt for details.
345/475
DebugFS
A virtual lesystem to export debugging information to user space.

Kernel conguration: DEBUG_FS
The debugging interface disappears when Debugfs is

congured out.
You can mount it as follows:
Kernel hacking -> Debug Filesystem
sudo mount -t debugfs none /sys/kernel/debug
First described on http://lwn.net/Articles/115405/

API documented in the Linux Kernel Filesystem API:
DocBook/filesystems/
346/475
DebugFS API
Create a sub-directory for your driver:
struct dentry *debugfs_create_dir(const char *name,

struct dentry *parent);
Expose an integer as a le in DebugFS:
struct dentry *debugfs_create_{u,x}{8,16,32}

(const char *name, mode_t mode, struct dentry *parent,
u8 *value);
Expose a binary blob as a le in DebugFS:
u for decimal representation

x for hexadecimal representation
struct dentry *debugfs_create_blob(const char *name,

mode_t mode, struct dentry *parent,
struct debugfs_blob_wrapper *blob);
Also possible to support writable DebugFS les or customize

the output using the more generic debugfs_create_file()
function.
347/475
Deprecated Debugging Mechanisms
Some additional debugging mechanisms, whose usage is now

considered deprecated
Adding special ioctl() commands for debugging purposes.

DebugFS is preferred.
Adding special entries in the proc lesystem. DebugFS is

preferred.
Adding special entries in the sysfs lesystem. DebugFS is

preferred.
Using printk(). The pr_*() and dev_*() functions are

preferred.
348/475
Using Magic SysRq
Allows to run multiple debug / rescue commands even when

the kernel seems to be in deep trouble
Example commands:
On PC: press [Alt] + [SysRq] + <character>

simultaneously
On embedded: in the console, send a break character
(Picocom: press [Ctrl] + a followed by [Ctrl] + \ ), then
press <character>
h: show available commands
s: sync all mounted lesystems
b: reboot the system
n: makes RT processes nice-able.
w: shows the kernel stack of all sleeping processes
t: shows the kernel stack of all running processes
You can even register your own!
Detailed in Documentation/sysrq.txt
349/475
kgdb - A Kernel Debugger
The execution of the kernel is fully controlled by gdb from

another machine, connected through a serial line.
Can do almost everything, including inserting breakpoints in

interrupt handlers.
Feature supported for the most popular CPU architectures
350/475
Using kgdb 1/2
Details available in the kernel documentation: DocBook/kgdb/
Recommended to turn on CONFIG_FRAME_POINTER to aid in

producing more reliable stack backtraces in gdb.
You must include a kgdb I/O driver. One of them is kgdb

over serial console (kgdboc: kgdb over console, enabled by
CONFIG_KGDB_SERIAL_CONSOLE)
Congure kgdboc at boot time by passing to the kernel:
kgdboc=<tty-device>,<bauds>.
For example: kgdboc=ttyS0,115200
351/475
Using kgdb 2/2
Then also pass kgdbwait to the kernel: it makes kgdb wait for
a debugger connection.
Boot your kernel, and when the console is initialized, interrupt

the kernel with Alt + SysRq + g.
On your workstation, start gdb as follows:
gdb ./vmlinux
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0
Once connected, you can debug a kernel the way you would
debug an application program.
352/475
Debugging with a JTAG Interface

Two types of JTAG dongles
The ones oering a gdb compatible interface, over a serial port
or an Ethernet connection. gdb can directly connect to them.
The ones not oering a gdb compatible interface are generally
supported by OpenOCD (Open On Chip Debugger):
http://openocd.sourceforge.net/
OpenOCD is the bridge between the gdb debugging language

and the JTAG interface of the target CPU.
See the very complete documentation:
http://openocd.org/documentation/
For each board, you'll need an OpenOCD conguration le
(ask your supplier)
353/475
More Kernel Debugging Tips
Make sure CONFIG_KALLSYMS_ALL is enabled
Is turned on by default
To get oops messages with symbol names instead of raw
addresses
On ARM, if your kernel doesn't boot or hangs without any

message, you can activate early debugging options
(CONFIG_DEBUG_LL and CONFIG_EARLYPRINTK), and add
earlyprintk to the kernel command line.
354/475
Practical lab - Kernel debugging
Use the dynamic printk feature.
Add debugfs entries
Load a broken driver and see it

crash
Analyze the error information

dumped by the kernel.
Disassemble the code and locate

the exact C instruction which
caused the failure.
355/475
Porting the Linux Kernel to an ARM Board
Porting the Linux

Kernel to an ARM
Board
Embedded Linux
Experts
free electrons
356/475
Porting the Linux kernel
The Linux kernel supports a lot of dierent CPU architectures

Each of them is maintained by a dierent group of
contributors
The organization of the source code and the methods to port

the Linux kernel to a new board are therefore very
architecture-dependent
See the MAINTAINERS le for details
For example, some architectures use the Device Tree, some do

not.
This presentation is focused on the ARM architecture only
357/475
Architecture, CPU and Machine
In the source tree, each architecture has its own directory
This directory contains generic ARM code
arch/arm for the ARM architecture

boot, common, configs, kernel, lib, mm, nwfpe, vfp,
oprofile, tools
And many directories for dierent SoC families
mach-* directories: mach-pxa for PXA CPUs, mach-imx for

Freescale iMX CPUs, etc.
Before the ARM cleanup, these directories contained support

for the SoC family (GPIO, clocks, pinmux, power
management, interrupt controller, etc.) and for the various
boards.
Nowadays, they contain a lot less code, essentially a small
SoC description le, power management and SMP code.
Some CPU types share some code, in directories named

plat-*
Device Tree source les in arch/arm/boot/dts.
358/475
Before the Device Tree and ARM cleanup
Until 2011, the ARM architecture wasn't using the Device

Tree, and a large portion of the SoC support was located in
arch/arm/mach-<foo>.
Each board supported by the kernel was associated to an

unique machine ID.
The entire list of machine ID can be downloaded at http://

www.arm.linux.org.uk/developer/machines/download.php
and one could freely register an additional one.
The Linux kernel was dening a machine structure for each

board, which associates the machine ID with a set of
information and callbacks.
The bootloader had to pass the machine ID to the kernel in a

specic ARM register.
359/475
The Device Tree and the ARM cleanup
As the ARM architecture gained signicantly in popularity,

some major refactoring was needed.
First, the Device Tree was introduced on ARM: instead of

using C code to describe SoCs and boards, a specialized
language is used.
Second, many driver infrastructures were created to replace
custom code in arch/arm/mach-<foo>:
The
The
The
The
common clock framework in drivers/clk

pinctrl subsystem in drivers/pinctrl
irqchip subsystem in drivers/irqchip
clocksource subsystem in drivers/clocksource
The amount of code in mach-<foo> has now signicantly

reduced.
360/475
Adding the support for a new ARM board
Provided the SoC used on your board is supported by the Linux

kernel:
1. Create a Device Tree le in arch/arm/boot/dts, generally
named <soc-name>-<board-name>.dts, and make it include
the relevant SoC .dtsi le.
Your Device Tree will describe all the SoC peripherals that are
enabled, the pin muxing, as well as all the devices on the
board.
2. Modify arch/arm/boot/dts/Makefile to make sure your

Device Tree gets built as a DTB during the kernel build.
3. If needed, develop the missing device drivers for the devices
that are on your board outside the SoC.
361/475
Example of the Freescale iMX28 SoCs
The hardware platform used in this training is based on the

AM335x processor from Texas Instruments.
This platform inherits from the OMAP family of TI, for which
kernel support has been around for a long time.
Due to this, and the complexity of the platform, the AM335x

and OMAP support in the kernel hasn't fully migrated yet to
all the infrastructures created during the ARM cleanup.
Therefore, to illustrate this section, we will take the example

of the Freescale iMX28 platform, on which Free Electrons has
worked specically.
362/475
Studying the Crystalfontz CFA-10036 platform
Crystalfontz CFA-10036
Uses the Freescale iMX28 SoC, from the

MXS family.
128MB of RAM
1 serial port, 1 LED
1 I2C bus, equipped with an OLED display
1 SD-Card slot
363/475
Crystalfontz CFA-10036 Device Tree, header
Mandatory Device Tree language denition

.
./dts-v1/
Include the .dtsi le describing the SoC
.
.#include "imx28.dtsi"
Start the root of the tree
A human-readable string to describe the machine
.
/
.{
.
m
.odel = "Crystalfontz CFA-10036 Board";
A list of compatible strings, from the most specic one to the

most general one. Can be used by kernel code to do a SoC or
board-specic check.
.
.compatible = "crystalfontz,cfa10036", "fsl,imx28";
364/475
Crystalfontz CFA-10036 Device Tree, chosen/memory
Denition of the default kernel command line. Some

additional operating-system specic entries can be added in
chosen:
chosen {
bootargs = "console=ttyS0,115200 earlyprintk";
.};
Denition of the size and location of the RAM:
memory {
device_type = "memory";
reg = <0x40000000 0x8000000>; /* 128 MB */
};
.
365/475
Crystalfontz CFA-10036, bus/UART
Start of the internal SoC peripherals.
apb@80000000 {
apbh@80000000 {
apbx@80040000 {
The CFA-10036 has one debug UART, so the corresponding

controller is enabled:
duart: serial@80074000 {
pinctrl-0 = <&duart_pins_b>;
status = "okay";
};
366/475
Crystalfontz CFA-10036 Device Tree, Muxing
Denition of a few pins that will be muxed as GPIO, for LEDs

and reset.
pinctrl@80018000 {
ssd1306_cfa10036: ssd1306-10036@0 {
reg = <0>;
fsl,pinmux-ids = <
0x2073 /* MX28_PAD_SSP0_D7__GPIO_2_7 */
>;
fsl,drive-strength = <0>;
fsl,voltage = <1>;
fsl,pull-up = <0>;
};
led_pins_cfa10036: leds-10036@0 {
reg = <0>;
fsl,pinmux-ids = <
0x3043 /* MX28_PAD_AUART1_RX__GPIO_3_4 */
>;
fsl,drive-strength = <0>;
fsl,voltage = <1>;
fsl,pull-up = <0>;
};
};
367/475
Crystalfontz CFA-10036 Device Tree, LED
One LED is connected to this platform. Note the reference to

the led_pins_cfa10036 muxing conguration.
leds {
compatible = "gpio-leds";
pinctrl-0 = <&led_pins_cfa10036>;
power {
gpios = <&gpio3 4 1>;
default-state = "on";
};
};
368/475
Crystalfontz CFA-10036 Device Tree, SD Card/USB
The platform also has a USB port
usb0: usb@80080000 {
pinctrl-0 = <&usb0_otg_cfa10036>;
status = "okay";
};
and an SD Card slot:
ssp0: ssp@80010000 {
compatible = "fsl,imx28-mmc";
pinctrl-0 = <&mmc0_4bit_pins_a
&mmc0_cd_cfg &mmc0_sck_cfg>;
bus-width = <4>;
status = "okay";
};
369/475
Crystalfontz CFA-10036 Device Tree, I2C bus
An I2C bus, with a Solomon SSD1306 OLED display

connected on it:
i2c0: i2c@80058000 {
pinctrl-0 = <&i2c0_pins_b>;
status = "okay";
ssd1306: oled@3c {
compatible = "solomon,ssd1306fb-i2c";
pinctrl-0 = <&ssd1306_cfa10036>;
reg = <0x3c>;
reset-gpios = <&gpio2 7 0>;
solomon,height = <32>;
solomon,width = <128>;
solomon,page-offset = <0>;
};
};
370/475
Crystalfontz CFA-10036 Device Tree, Breakout Boards
The CFA-10036 can be plugged in other breakout boards, and

the device tree also allows us to describe this, using includes.
For example, the CFA-10057:
.
.#include "imx28-cfa10036.dts"
This allows to have a layered description. This can also be

done for boards that have a lot in common, like the
BeagleBone and the BeagleBone Black, or the AT91
SAMA5D3-based boards.
371/475
Crystalfontz CFA-10036: build the DTB
To ensure that the Device Tree Blob gets built for this board
Device Tree Source, one need to ensure it is listed in
arch/arm/boot/dts/Makefile:
.
dtb-$(CONFIG_ARCH_MXS) += imx28-cfa10036.dtb \
imx28-cfa10037.dtb \
imx28-evk.dtb
.
372/475
Understanding the SoC support
Let's consider another ARM platform here, the Marvell

Armada 370/XP.
For this platform, the core of the SoC support is located in

arch/arm/mach-mvebu
The board-v7.c le (see code on the next slide) contains the
"entry point" of the SoC denition, the DT_MACHINE_START ..
MACHINE_END denition:
Denes the list of platform compatible strings that will match

this platform, in this case marvell,armada-370-xp. This
allows the kernel to know which DT_MACHINE structure to use
depending on the DTB that is passed at boot time.
Denes various callbacks for the platform initialization, the
most important one being the .init_machine callback, which
calls of_platform_populate(). This function travels through
the Device Tree and instantiates all the devices.
373/475
arch/arm/mach-mvebu/board-v7.c
.
static void __init mvebu_dt_init(void)

{
if (of_machine_is_compatible("marvell,armadaxp"))
i2c_quirk();
of_platform_populate(NULL, of_default_bus_match_table, NULL, NULL);
}
static const char * const armada_370_xp_dt_compat[] __initconst = {
"marvell,armada-370-xp",
NULL,
};
DT_MACHINE_START(ARMADA_370_XP_DT, "Marvell Armada 370/XP (Device Tree)")
.l2c_aux_val
= 0,
.l2c_aux_mask = ~0,
.smp
= smp_ops(armada_xp_smp_ops),
.init_machine = mvebu_dt_init,
.init_irq
= mvebu_init_irq,
.restart
= mvebu_restart,
.reserve
= mvebu_memblock_reserve,
.dt_compat
= armada_370_xp_dt_compat,
MACHINE_END
374/475
Components of the minimal SoC support
The minimal SoC support consists in

An SoC entry point le, arch/arm/mach-mvebu/board-v7.c
At least one SoC .dtsi DT and one board .dts DT, in
arch/arm/boot/dts
A interrupt controller driver, drivers/irqchip/irq-armada-370-xp.c
A timer driver, drivers/clocksource/time-armada-370-xp.c
An earlyprintk implementation to get early messages from the console,
arch/arm/Kconfig.debug and arch/arm/include/debug

A serial port driver in drivers/tty/serial. For Armada 370/XP, the
8250 driver drivers/tty/serial/8250 is used.
This allows to boot a minimal system up to user space, using a

root lesystem in initramfs.
375/475
Extending the minimal SoC support

Once the minimal SoC support is in place, the following core
components should be added:
Support for the clocks. Usually requires some clock drivers, as

well as DT representations of the clocks. See
drivers/clk/mvebu for Armada 370/XP clock drivers.
Support for pin muxing, through the pinctrl subsystem. See

drivers/pinctrl/mvebu for the Armada 370/XP drivers.
Support for GPIOs, through the GPIO subsystem. See

drivers/gpio/gpio-mvebu.c for the Armada 370/XP GPIO
driver.
Support for SMP, through struct smp_operations. See

arch/arm/mach-mvebu/platsmp.c.
376/475
Adding device drivers
Once the core pieces of the SoC support have been implemented,
the remaining part is to add drivers for the dierent hardware
blocks:
Ethernet driver, in
drivers/net/ethernet/marvell/mvneta.c
SATA driver, in drivers/ata/sata_mv.c
I2C driver, in drivers/i2c/busses/i2c-mv64xxx.c
SPI driver, in drivers/spi/spi-orion.c
PCIe driver, in drivers/pci/host/pci-mvebu.c
USB driver, in drivers/usb/host/ehci-orion.c
etc.
377/475
Power Management
Power
Management
Embedded Linux
Experts
free electrons
378/475
PM Building Blocks
Several power management building blocks
Suspend and resume

CPUidle
Runtime power management
Frequency and voltage scaling
Applications
Independent building blocks that can be improved gradually

during development
379/475
Clock Framework (1)
Generic framework to manage clocks used by devices in the

system
Allows to reference count clock users and to shutdown the

unused clocks to save power
Simple API described in DocBook/kernel-api/clk.html.
clk_get() to get a reference to a clock

clk_enable() to start the clock
clk_disable() to stop the clock
clk_put() to free the clock source
clk_get_rate() to get the current rate
380/475
Clock Framework (2)
The common clock framework
Allows to declare the available clocks and their association to

devices in the Device Tree (preferred) or statically in the
source code (old method)
Provides a debugfs representation of the clock tree
Is implemented in drivers/clk
381/475
Diagram overview of the common clock framework
382/475
Clock Framework (3)
The interface of the CCF divided into two halves:

Common Clock Framework core
Common denition of struct clk

Common implementation of the clk.h API (dened in
drivers/clk/clk.c)
struct clk_ops: operations invoked by the clk API
implementation
Not supposed to be modied when adding a new driver
Hardware-specic
Callbacks registered with struct clk_ops and the

corresponding hardware-specic structures
Has to be written for each new hardware clock
383/475
Clock Framework (4)
Hardware clock operations: device tree

The device tree is the mandatory way to declare a clock
and to get its resources, as for any other driver using DT we
have to:
Parse the device tree to setup the clock: the resources but
also the properties are retrieved.
Declare the compatible clocks and associate it with an
initialization function using CLK_OF_DECLARE
384/475
Suspend and Resume
Infrastructure in the kernel to support suspend and resume

Platform hooks
prepare(), enter(), finish(), valid() in a

struct platform_suspend_ops structure
Registered using the suspend_set_ops() function
See arch/arm/mach-at91/pm.c
Device drivers
suspend() and resume() hooks in the *_driver structures

(struct platform_driver, struct usb_driver, etc.)
See drivers/net/ethernet/cadence/macb.c
385/475
Triggering Suspend
struct suspend_ops functions are called by the

enter_state() function.
enter_state() also takes care of executing the suspend and

resume functions for your devices.
The execution of this function can be triggered from user
space. To suspend to RAM:
echo mem > /sys/power/state
Can also use the s2ram program from

http://suspend.sourceforge.net/
Read kernel/power/suspend.c
386/475
Runtime Power Management
According to the kernel conguration interface: Enable

functionality allowing I/O devices to be put into energy-saving
(low power) states at run time (or autosuspended) after a
specied period of inactivity and woken up in response to a
hardware-generated wake-up event or a driver's request.
New hooks must be added to the drivers:

runtime_suspend(), runtime_resume(), runtime_idle()
API and details on Documentation/power/runtime_pm.txt
See also Kevin Hilman's presentation at ELC Europe 2010:

http://elinux.org/images/c/cd/ELC-2010-khilmanRuntime-PM.odp
387/475
Saving Power in the Idle Loop
The idle loop is what you run when there's nothing left to run
in the system.
Implemented in all architectures in

arch/<arch>/kernel/process.c
Example to read: look for cpu_idle in

arch/arm/kernel/process.c
Each ARM cpu denes its own arch_idle function.
The CPU can run power saving HLT instructions, enter NAP
mode, and even disable the timers (tickless systems).
See also http://en.wikipedia.org/wiki/Idle_loop
388/475
Managing Idle
Adding support for multiple idle levels
Modern CPUs have several sleep states oering dierent

power savings with associated wake up latencies
The dynamic tick feature allows to remove the periodic tick to

save power, and to know when the next event is scheduled, for
smarter sleeps.
CPUidle infrastructure to change sleep states
Platform-specic driver dening sleep states and transition

operations
Platform-independent governors (ladder and menu)
Available for x86/ACPI, not supported yet by all ARM cpus.
(look for cpuidle* les under arch/arm/)
See Documentation/cpuidle/ in kernel sources.
389/475
PowerTOP
https://01.org/powertop/
With dynamic ticks, allows to x parts of kernel code and

applications that wake up the system too often.
PowerTOP allows to track the worst oenders
Now available on ARM cpus implementing CPUidle
Also gives you useful hints for reducing power.
390/475
Frequency and Voltage Scaling (1)

Frequency and voltage scaling possible through the cpufreq kernel
infrastructure.
Generic infrastructure: drivers/cpufreq/cpufreq.c and

include/linux/cpufreq.h
Generic governors, responsible for deciding frequency and
voltage transitions
performance: maximum frequency

powersave: minimum frequency
ondemand: measures CPU consumption to adjust frequency
conservative: often better than ondemand. Only increases
frequency gradually when the CPU gets loaded.
userspace: leaves the decision to a user space daemon.
This infrastructure can be controlled from

/sys/devices/system/cpu/cpu<n>/cpufreq/
391/475
Frequency and Voltage Scaling (2)
CPU drivers in drivers/cpufreq. Example:

drivers/cpufreq/omap-cpufreq.c
Must implement the operations of the cpufreq_driver
structure and register them using
cpufreq_register_driver()
init() for initialization

exit() for cleanup
verify() to verify the user-chosen policy
setpolicy() or target() to actually perform the frequency
change
See Documentation/cpu-freq/ for useful explanations
392/475
PM Quality Of Service interface
Kernel and user mode interface for registering performance

expectations by drivers, subsystems and user space
applications.
Two dierent PM QoS frameworks are available:
PM QoS classes for CPU DMA latency, network latency and

and network throughput.
The per-device PM QoS framework API to manage per-device
latency.
According to these requirements, PM QoS allows kernel

drivers to adjust their power management
See Documentation/power/pm_qos_interface.txt
Still needs deploying in most drivers
393/475
Regulator Framework
Modern embedded hardware have hardware responsible for

voltage and current regulation
The regulator framework allows to take advantage of this
hardware to save power when parts of the system are unused
A consumer interface for device drivers (i.e users)

Regulator driver interface for regulator drivers
Machine interface for board conguration
sysfs interface for user space
See Documentation/power/regulator/ in kernel sources.
394/475
BSP Work for a New Board

In case you just need to create a BSP for your board, and your
CPU already has full PM support, you should just need to:
Create clock denitions and bind your devices to them.
Implement PM handlers (suspend, resume) in the drivers for

your board specic devices.
Implement runtime PM handlers in your drivers.
Implement board specic power management if needed

(mainly battery management)
Implement regulator framework hooks for your board if

needed.
All other parts of the PM infrastructure should be already

there: suspend / resume, cpuidle, cpu frequency and voltage
scaling.
395/475
Useful Resources
Documentation/power/ in the Linux kernel sources.
Will give you many useful details.
http://wiki.linaro.org/WorkingGroups/PowerManagement/
Ongoing developments on the ARM platform.
Introduction to kernel power management, Kevin Hilman,

Linaro
http://elinux.org/images/d/dd/Intro_Kernel_PM.svg
https://www.youtube.com/watch?v=Um0oRanCtzY
Tips and ideas for prolonging battery life (Amit Kucheria)
http://j.mp/fVdxKh
396/475
The kernel development and contribution process
The kernel
development and
contribution
process
Embedded Linux
Experts
free electrons
397/475
Linux versioning scheme and

development process
398/475
Until 2.6 (1)
One stable major branch every 2 or 3 years
One development branch to integrate new functionalities and

major changes
Identied by an even middle number

Examples: 1.0.x, 2.0.x, 2.2.x, 2.4.x
Identied by an odd middle number

Examples: 2.1.x, 2.3.x, 2.5.x
After some time, a development version becomes the new base
version for the stable branch
Minor releases once in while: 2.2.23, 2.5.12, etc.
399/475
Until 2.6 (2)
400/475
Changes since Linux 2.6
Since 2.6.0, kernel developers have been able to introduce

lots of new features one by one on a steady pace, without
having to make disruptive changes to existing subsystems.
Since then, there has been no need to create a new

development branch massively breaking compatibility with the
stable branch.
Thanks to this, more features are released to users at a

faster pace.
401/475
Versions since 2.6.0
From 2003 to 2011, the ocial kernel versions were named

2.6.x.
Linux 3.0 was released in July 2011
Linux 4.0 was released in April 2015

This is only a change to the numbering scheme
Ocial kernel versions are now named x.y

(3.0, 3.1, 3.2, ..., 3.19, 4.0, 4.1, etc.)
Stabilized versions are named x.y.z (3.0.2, 4.2.7, etc.)
It eectively only removes a digit compared to the previous
numbering scheme
402/475
New development model
Using merge and bug xing windows
403/475
More stability for the kernel source tree
Issue: bug and security xes only released for

most recent stable kernel versions.
Some people need to have a recent kernel,

but with long term support for security
updates.
You could get long term support from a

commercial embedded Linux provider.
You could reuse sources for the kernel used in

Ubuntu Long Term Support releases (5 years
of free security updates).
The http://kernel.org front page shows

which versions will be supported for some
time (up to 2 or 3 years), and which ones
won't be supported any more ("EOL: End Of
Life")
404/475
What's new in each Linux release? (1)

The ocial list of changes for each Linux release is just a huge list
of individual patches!
commit aa6e52a35d388e730f4df0ec2ec48294590cc459
Author: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Date: Wed Jul 13 11:29:17 2011 +0200
at91: at91-ohci: support overcurrent notification
Several USB power switches (AIC1526 or MIC2026) have a digital output
that is used to notify that an overcurrent situation is taking
place. This digital outputs are typically connected to GPIO inputs of
the processor and can be used to be notified of these overcurrent
situations.
Therefore, we add a new overcurrent_pin[] array in the at91_usbh_data
structure so that boards can tell the AT91 OHCI driver which pins are
used for the overcurrent notification, and an overcurrent_supported
boolean to tell the driver whether overcurrent is supported or not.
The code has been largely borrowed from ohci-da8xx.c and
ohci-s3c2410.c.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Nicolas Ferre <nicolas.ferre@atmel.com>
Very dicult to nd out the key changes and to get the global
picture out of individual changes.
405/475
What's new in each Linux release? (2)
Fortunately, there are some useful resources available
http://wiki.kernelnewbies.org/LinuxChanges
(some versions are missing)
http://lwn.net
http://www.linux-arm.info
News about Linux on ARM, including kernel changes.
http://linuxfr.org, for French readers
406/475
Contributing to the Linux kernel
407/475
Solving Issues
If you face an issue, and it doesn't look specic to your work

but rather to the tools you are using, it is very likely that
someone else already faced it.
Search the Internet for similar error reports.
You have great chances of nding a solution or workaround,

or at least an explanation for your issue.
Otherwise, reporting the issue is up to you!
408/475
Getting Help
If you have a support contract, ask your vendor.

Otherwise, don't hesitate to share your questions and issues
Either contact the Linux mailing list for your architecture (like
linux-arm-kernel or linuxsh-dev...).
Or contact the mailing list for the subsystem you're dealing
with (linux-usb-devel, linux-mtd...). Don't ask the maintainer
directly!
Most mailing lists come with a FAQ page. Make sure you read
it before contacting the mailing list.
Useful IRC resources are available too (for example on
http://kernelnewbies.org).
Refrain from contacting the Linux Kernel mailing list, unless
you're an experienced developer and need advice.
409/475
Reporting Linux Bugs
First make sure you're using the latest version
Make sure you investigate the issue as much as you can: see
Documentation/BUG-HUNTING
Check for previous bugs reports. Use web search engines,

accessing public mailing list archives.
If the subsystem you report a bug on has a mailing list, use it.
Otherwise, contact the ocial maintainer (see the
MAINTAINERS le). Always give as many useful details as
possible.
410/475
How to Become a Kernel Developer?
Recommended resources
See Documentation/SubmittingPatches for guidelines and

http://kernelnewbies.org/UpstreamMerge for very helpful
advice to have your changes merged upstream (by Rik van
Riel).
Watch the Write and Submit your rst Linux kernel Patch
talk by Greg. K.H:
http://www.youtube.com/watch?v=LLBrBBImJt4
How to Participate in the Linux Community (by Jonathan

Corbet). A guide to the kernel development process
http://j.mp/tX2Ld6
411/475
Contribute to the Linux Kernel (1)
Clone Linus Torvalds' tree:
Keep your tree up to date

git pull
Look at the master branch and check whether your issue /

change hasn't been solved / implemented yet. Also check the
maintainer's git tree and mailing list (see the MAINTAINERS
le).You may miss submissions that are not in mainline yet.
If the maintainer has its own git tree, create a remote branch
tracking this tree. This is much better than creating another
clone (doesn't duplicate common stu):
git remote add linux-omap git://git.kernel.org/pub/

scm/linux/kernel/git/tmlind/linux-omap.git
git fetch linux-omap
412/475
Either create a new branch starting from the current commit

in the master branch:
git checkout -b feature
Or, if more appropriate, create a new branch starting from the

maintainer's master branch:
git checkout -b feature linux-omap/master (remote tree

/ remote branch)
In your new branch, implement your changes.
Test your changes (must at least compile them).
Run git add to add any new les to the index.
413/475
Congure git send-email
Make sure you already have congured your name and e-mail
address (should be done before the rst commit).
git config --global user.name 'My Name'

git config --global user.email me@mydomain.net
Congure your SMTP settings. Example for a Google Mail

account:
git config -global sendemail.smtpserver smtp.googlemail.com

git config --global sendemail.smtpserverport 587
git config --global sendemail.smtpencryption tls
git config --global sendemail.smtpuser jdoe@gmail.com
git config --global sendemail.smtppass xxx
414/475
Group your changes by sets of logical changes, corresponding

to the set of patches that you wish to submit.
Commit and sign these groups of changes (signing required by
Linux developers).
The easiest way is to look at previous commit summaries on

the main le you modify
git commit -s
Make sure your rst description line is a useful summary and
starts with the name of the modied subsystem. This rst
description line will appear in your e-mails
git log --pretty=oneline <path-to-file>
Examples subject lines ([PATCH] omitted):

Documentation: prctl/seccomp_filter
PCI: release busn when removing bus
ARM: add support for xz kernel decompression
415/475
Remove previously generated patches
rm 00*.patch
Have git generate patches corresponding to your branch
If your branch is based on mainline
If your branch is based on a remote branch
scripts/checkpatch.pl --strict 00*.patch
Now, send your patches to yourself
git format-patch <remote>/<branch>..<your branch>
You can run a last check on all your patches (easy)
git format-patch master..<your branch>
git send-email --compose -to me@mydomain.com 00*.patch
If you have just one patch, or a trivial patch, you can remove
the empty line after In-Reply-To:. This way, you won't add
a summary e-mail introducing your changes (recommended
otherwise).
416/475
Check that you received your e-mail properly, and that it looks
good.
Now, nd the maintainers for your patches
scripts/get_maintainer.pl ~/patches/00*.patch
Russell King <linux@arm.linux.org.uk> (maintainer:ARM PORT)
Nicolas Pitre <nicolas.pitre@linaro.org>
(commit_signer:1/1=100%)
linux-arm-kernel@lists.infradead.org (open list:ARM PORT)
linux-kernel@vger.kernel.org (open list)
Now, send your patches to each of these people and lists
git send-email --compose --to linux@arm.linux.org.uk

--to nicolas.pitre@linaro.org --to linux-armkernel@lists.infradead.org --to linuxkernel@vger.kernel.org 00*.patch
Wait for replies about your changes, take the comments into
account, and resubmit if needed, until your changes are
eventually accepted.
417/475
If you use git format-patch to produce your patches, you

will need to update your branch and may need to group your
changes in a dierent way (one patch per commit).
Here's what we recommend
Update your master branch
Back to your branch, implement the changes taking

community feedback into account. Commit these changes.
Still in your branch: reorganize your commits and commit
messages
git checkout master; git pull
git rebase --interactive origin/master

git rebase allows to rebase (replay) your changes starting
from the latest commits in master. In interactive mode, it also
allows you to merge, edit and even reorder commits, in an
interactive way.
Third, generate the new patches with git format-patch.
418/475
Kernel Resources
Kernel Resources
Embedded Linux
Experts
free electrons
419/475
Kernel Development News
Linux Weekly News
http://lwn.net/
The weekly digest o all Linux and free software information

sources
In depth technical discussions about the kernel
Subscribe to nance the editors ($7 / month)
Articles available for non subscribers after 1 week.
420/475
Useful Reading (1)

Essential Linux Device Drivers, April 2008
http://elinuxdd.com/
By Sreekrishnan Venkateswaran, an
embedded IBM engineer with more than
10 years of experience
Covers a wide range of topics not covered

by LDD: serial drivers, input drivers, I2C,
PCMCIA and Compact Flash, PCI, USB,
video drivers, audio drivers, block drivers,
network drivers, Bluetooth, IrDA, MTD,
drivers in user space, kernel debugging,
etc.
Probably the most wide ranging and

complete Linux device driver book I've
read -- Alan Cox
421/475
Useful Reading (2)

Linux Device Drivers, 4th edition, November
2017 (estimated, keeps slipping!)
http://shop.oreilly.com/product/
0636920030867.do
By Jonathan Corbet, Alessandro Rubini,

Greg Kroah-Hartman, Jessica McKellar,
O'Reilly
Expected to be a great book, if as good

as the previous edition (Free PDF:
community/kernel/ldd3/), which is now
out of date.
422/475
Useful Reading (3)
Linux Kernel Development, 3rd Edition,

Jun 2010
Robert Love, Novell Press

http://freeelectrons.com/redir/lkd3-book.html
A very synthetic and pleasant way to
learn about kernel subsystems (beyond
the needs of device driver writers)
The Linux Programming Interface, Oct

2010
Michael Kerrisk, No Starch Press

http://man7.org/tlpi/
A gold mine about the kernel interface
and how to use it
423/475
Useful Online Resources
Kernel documentation
Linux kernel mailing list FAQ
http://www.tux.org/lkml/
Complete Linux kernel FAQ
Read this before asking a question to the mailing list
Kernel Newbies
https://kernel.org/doc/
http://kernelnewbies.org/
Glossary, articles, presentations, HOWTOs, recommended
reading, useful tools for people getting familiar with Linux
kernel or driver development.
Kernel glossary
http://kernelnewbies.org/KernelGlossary
424/475
International Conferences
Embedded Linux Conference:

http://embeddedlinuxconference.com/
Linux Plumbers: http://linuxplumbersconf.org
Organized by the Linux Foundation in the USA

(February-April) and in Europe (October-November)
Very interesting kernel and user space topics for embedded
systems developers.
Presentation slides and videos freely available
Conference on the low-level plumbing of Linux: kernel, audio,
power management, device management, multimedia, etc.
linux.conf.au: http://linux.org.au/conf/
In Australia / New Zealand

Features a few presentations by key kernel hackers.
425/475
Continue to learn after the course

Here are a few suggestions:
Do the Eudyptula Challenge:
http://eudyptula-challenge.org/. A series of Linux kernel
programming exercises of increasing complexity, up to getting
patches accepted into the mainline sources.
Run your labs again on your own hardware. The Nunchuk lab
should be rather straightforward, but the serial lab will be
quite dierent if you use a dierent processor.
Help with tasks keeping the kernel code clean and up-to-date:
http://kernelnewbies.org/KernelJanitors
Learn by reading the kernel code by yourself, ask questions
and propose improvements.
Implement and share drivers for your own hardware, of course!
426/475
Last slides
Last slides
Embedded Linux
Experts
free electrons
427/475
Last slide
Thank you!
And may the Source be with you
428/475
Backup slides
Backup slides
Embedded Linux
Experts
free electrons
429/475
Backup slides
DMA
430/475
DMA Integration
431/475
Peripheral DMA
432/475
DMA Controllers
433/475
DMA descriptors
434/475
Backup slides
DMA Usage
435/475
Constraints with a DMA
A DMA deals with physical addresses, so:
Programming a DMA requires retrieving a physical address at

some point (virtual addresses are usually used)
The memory accessed by the DMA shall be physically
contiguous
The CPU can access memory through a data cache
Using the cache can be more ecient (faster accesses to the

cache than the bus)
But the DMA does not access to the CPU cache, so one need
to take care of cache coherency (cache content vs memory
content)
Either ush or invalidate the cache lines corresponding to the
buer accessed by DMA and processor at strategic times
436/475
DMA Memory Constraints
Need to use contiguous memory in physical space.
Can use any memory allocated by kmalloc() (up to 128 KB)

or __get_free_pages() (up to 8MB).
Can use block I/O and networking buers, designed to

support DMA.
Can not use vmalloc() memory (would have to setup DMA

on each individual physical page).
437/475
Memory Synchronization Issues
Memory caching could interfere with DMA

Before DMA to device
After DMA from device
Need to make sure that all writes to DMA buer are

committed.
Before drivers read from DMA buer, need to make sure that
memory caches are ushed.
Bidirectional DMA
Need to ush caches before and after the DMA transfer.
438/475
Linux DMA API
The kernel DMA utilities can take care of:
Either allocating a buer in a cache coherent area,
Or making sure caches are ushed when required,
Managing the DMA mappings and IOMMU (if any).
See Documentation/DMA-API.txt for details about the Linux

DMA generic API.
Most subsystems (such as PCI or USB) supply their own

DMA API, derived from the generic one. May be sucient for
most needs.
439/475
Coherent or Streaming DMA Mappings
Coherent mappings
The kernel allocates a suitable buer and sets the mapping for
the driver.
Can simultaneously be accessed by the CPU and device.
So, has to be in a cache coherent memory area.
Usually allocated for the whole time the module is loaded.
Can be expensive to setup and use on some platforms.
Streaming mappings
The kernel just sets the mapping for a buer provided by the
driver.
Use a buer already allocated by the driver.
Mapping set up for each transfer. Keeps DMA registers free on
the hardware.
The recommended solution.
440/475
Allocating Coherent Mappings
The kernel takes care of both buer allocation and mapping

#include <asm/dma-mapping.h>
void *
dma_alloc_coherent(
struct device *dev,
size_t size,
dma_addr_t *handle,
gfp_t gfp
);
/* Output: buffer address */

/*
/*
/*
/*
device structure */
Needed buffer size in bytes */
Output: DMA bus address */
Standard GFP flags */
void dma_free_coherent(struct device *dev,

size_t size, void *cpu_addr, dma_addr_t handle);
441/475
Setting up streaming mappings

Works on buers already allocated by the driver
#include <linux/dmapool.h>
dma_addr_t dma_map_single(
struct device *,
void *,
size_t,
enum dma_data_direction
/*
/*
/*
/*
*
*
device structure */
input: buffer to use */
buffer size */
Either DMA_BIDIRECTIONAL,
DMA_TO_DEVICE or
DMA_FROM_DEVICE */
);
void dma_unmap_single(struct device *dev, dma_addr_t handdle,
size_t size, enum dma_data_direction dir);
442/475
DMA Streaming Mapping Notes
When the mapping is active: only the device should access the
buer (potential cache issues otherwise).
The CPU can access the buer only after unmapping!
Another reason: if required, this API can create an

intermediate bounce buer (used if the given buer is not
usable for DMA).
The Linux API also supports scatter / gather DMA streaming

mappings.
443/475
Backup slides
DMA transfers
444/475
Starting DMA transfers
If the device you're writing a driver for is doing peripheral

DMA, no external API is involved.
If it relies on an external DMA controller, you'll need to
Ask the hardware to use DMA, so that it will drive its request
line
Use Linux DMAEngine framework, especially its slave API
445/475
DMAEngine Slave API
In order to start a DMA transfer, you need to call the

following functions from your driver
1. Request a channel for exclusive use with
dma_request_channel(), or one of its variants
2. Congure it for our use case, by lling a
struct dma_slave_config structure with various parameters
(source and destination adresses, accesses width, etc.) and
passing it as an argument to dmaengine_slave_config()
3. Start a new transaction with
dmaengine_prep_slave_single() or
dmaengine_prep_slave_sg()
4. Put the transaction in the driver pending queue using
dmaengine_submit()
5. And nally ask the driver to process all pending transactions
using dma_async_issue_pending()
Of course, this needs to be done in addition to the DMA

mapping seen previously
Some frameworks abstract it away from you, such as SPI and
ASoC
446/475
Backup slides
mmap
447/475
mmap
Possibility to have parts of the virtual address space of a

program mapped to the contents of a le
Particularly useful when the le is a device le
Allows to access device I/O memory and ports without having

to go through (expensive) read, write or ioctl calls
One can access to current mapped les by two means:
/proc/<pid>/maps
pmap <pid>
448/475
/proc/<pid>/maps
start-end
...
7f4516d04000-7f4516d06000
7f4516d07000-7f4516d0b000
...
7f4518728000-7f451874f000
7f451874f000-7f451894f000
7f451894f000-7f4518951000
7f4518951000-7f4518952000
...
7f451da4f000-7f451dc3f000
7f451de3e000-7f451de41000
7f451de41000-7f451de4c000
...
perm offset major:minor inode
mapped file name
rw-s 1152a2000 00:05 8406

rw-s 120f9e000 00:05 8406
/dev/dri/card0
/dev/dri/card0
r-xp
---p
r--p
rw-p
/lib/x86_64-linux-gnu/libexpat.so.1.5.2
00000000
00027000
00027000
00029000
08:01
08:01
08:01
08:01
268909
268909
268909
268909
r-xp 00000000 08:01 1549

r--p 001ef000 08:01 1549
rw-p 001f2000 08:01 1549
/usr/bin/Xorg
/usr/bin/Xorg
/usr/bin/Xorg
449/475
mmap Overview
450/475
How to Implement mmap - User Space
Open the device le
Call the mmap system call (see man mmap for details):
void * mmap(
void *start,
size_t length,
int prot,
int flags,
int fd,
off_t offset
);
/*
/*
/*
/*
/*
/*
Often 0, preferred starting address */

Length of the mapped area */
Permissions: read, write, execute */
Options: shared mapping, private copy... */
Open file descriptor */
Offset in the file */
You get a virtual address you can write to or read from.
451/475
How to Implement mmap - Kernel Space
Character driver: implement an mmap le operation and add it

to the driver le operations:
int (*mmap) (
struct file *,
struct vm_area_struct *
);
/* Open file structure */

/* Kernel VMA structure */
Initialize the mapping.
Can be done in most cases with the remap_pfn_range()

function, which takes care of most of the job.
452/475
remap_pfn_range()
pfn: page frame number
The most signicant bits of the page address (without the bits
corresponding to the page size).
#include <linux/mm.h>
int remap_pfn_range(
struct vm_area_struct *, /* VMA struct */
unsigned long virt_addr, /* Starting user
* virtual address */
unsigned long pfn,
/* pfn of the starting
* physical address */
unsigned long size,
/* Mapping size */
pgprot_t prot
/* Page permissions */
);
453/475
Simple mmap implementation

static int acme_mmap
(struct file * file, struct vm_area_struct *vma)
{
size = vma->vm_end - vma->vm_start;
if (size > ACME_SIZE)
return -EINVAL;
if (remap_pfn_range(vma,
vma->vm_start,
ACME_PHYS >> PAGE_SHIFT,
size,
vma->vm_page_prot))
return -EAGAIN;
return 0;
}
454/475
devmem2
http://free-electrons.com/pub/mirror/devmem2.c, by
Jan-Derk Bakker
Very useful tool to directly peek (read) or poke (write) I/O
addresses mapped in physical address space from a shell
command line!
Very useful for early interaction experiments with a device,

without having to code and compile a driver.
Uses mmap to /dev/mem.
Examples (b: byte, h: half, w: word)
devmem2 0x000c0004 h (reading)

devmem2 0x000c0008 w 0xffffffff (writing)
devmem is now available in BusyBox, making it even easier to

use.
455/475
mmap Summary
The device driver is loaded. It denes an mmap le operation.
A user space process calls the mmap system call.
The mmap le operation is called.
It initializes the mapping using the device physical address.
The process gets a starting address to read from and write to

(depending on permissions).
The MMU automatically takes care of converting the process

virtual addresses into physical ones.
Direct access to the hardware without any expensive read or

write system calls
456/475
Backup slides
Introduction to Git
457/475
What is Git?
A version control system, like CVS, SVN, Perforce or

ClearCase
Originally developed for the Linux kernel development, now

used by a large number of projects, including U-Boot,
GNOME, Buildroot, uClibc and many more
Contrary to CVS or SVN, Git is a distributed version control
system
No central repository
Everybody has a local repository
Local branches are possible, and very important
Easy exchange of code between developers
Well-suited to the collaborative development model used in
open-source projects
458/475
Install and Setup
Git is available as a package in your distribution
Everything is available through the git command
sudo apt-get install git

git has many commands, called using git <command>, where
<command> can be clone, checkout, branch, etc.
Help can be found for a given command using
git help <command>
Set up your name and e-mail address
They will be referenced in each of your commits

git config --global user.name 'My Name'
git config --global user.email me@mydomain.net
459/475
Clone a Repository
To start working on a project, you use Git's clone operation.
With CVS or SVN, you would have used the checkout

operation, to get a working copy of the project (latest version)
With Git, you get a full copy of the repository, including the
history, which allows to perform most of the operations oine.
Cloning Linus Torvalds' Linux kernel repository

git:// is a special Git protocol. Most repositories can also be

accessed using http://, but this is slower.
After cloning, in linux/, you have the repository and a

working copy of the master branch.
460/475
Explore the History
git log will list all the commits. The latest commit is the
rst.
commit 4371ee353c3fc41aad9458b8e8e627eb508bc9a3
Author: Florian Fainelli <florian@openwrt.org>
Date: Mon Jun 1 02:43:17 2009 -0700
MAINTAINERS: take maintainership of the cpmac Ethernet driver
This patch adds me as the maintainer of the CPMAC (AR7)
Ethernet driver.
Signed-off-by: Florian Fainelli <florian@openwrt.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
git log -p will list the commits with the corresponding di

The history in Git is not linear like in CVS or SVN, but it is a
graph of commits
Makes it a little bit more complicated to understand at the

beginning
But this is what allows the powerful features of Git
(distributed, branching, merging)
461/475
Visualize the History: gitk
gitk is a graphical tool that represents the history of the

current Git repository
Can be installed from the gitk package
462/475
Visualize the History: cgit
Another great tool is cgit, a web interface to Git. For the

kernel sources, it is used on http://git.kernel.org/
463/475
Update your Repository
The repository that has been cloned at the beginning will

change over time
Updating your local repository to reect the changes of the

remote repository will be necessary from time to time
git pull
Internally, does two things
Fetch the new changes from the remote repository

(git fetch)
Merge them in the current branch (git merge)
464/475
Tags
The list of existing tags can be found using
To check out a working copy of the repository at a given tag
git log v2.6.30..master
List of changes with di on a given le between two tags
git checkout <tagname>
To get the list of changes between a given tag and the latest
available version
git tag -l
git log -p v2.6.29..v2.6.30 MAINTAINERS
With gitk
gitk v2.6.30..master
465/475
Branches
To start working on something, the best is to make a branch
It is local-only, nobody except you sees the branch

It is fast
It allows to split your work on dierent topics, try something
and throw it away
It is cheap, so even if you think you're doing something small
and quick, do a branch
Unlike other version control systems, Git encourages the use

of branches. Don't hesitate to use them.
466/475
Branches
Create a branch
Move to this branch
git checkout -b <branchname>
List of local branches
git checkout <branchname>
Both at once (create and switch to branch)
git branch <branchname>
git branch
List of all branches, including remote branches
git branch -a
467/475
Making Changes
Edit a le with your favorite text editor

Get the status of your working copy
Git has a feature called the index, which allows you to stage
your commits before committing them. It allows to commit
only part of your modications, by le or even by chunk.
On each modied le
git add <filename>
Then commit. No need to be on-line or connected to commit
git status
Linux requires the -s option to sign your changes

git commit -s
If all modied les should be part of the commit
git commit -as
468/475
Sharing Changes: E-mail
The simplest way of sharing a few changes is to send patches

by e-mail
The rst step is to generate the patches
git format-patch master..<yourbranch>

Will generate one patch for each of the commits done on
<yourbranch>
The patch les will be 0001-...., 0002-...., etc.
The second step is to send these patches by e-mail
git send-email --compose -to email@domain.com 00*.patch
Required Ubuntu package: git-email

In a later slide, we will see how to use git cong to set the
SMTP server, port, user and password.
469/475
Sharing Changes: Your Own Repository
If you do a lot of changes and want to ease collaboration with

others, the best is to have your own public repository
Use a git hosting service on the Internet:
GitLab (http://gitlab.com/)
GitHub (https://github.com/)
For public repositories. Need to pay for private repositories.
Publish on your own web server
Open Source server. Proprietary and commercial extensions

available.
Easy to implement.
Just needs git software on the server and ssh access.
Drawback: only supports http cloning (less ecient)
Set up your own git server
Most exible solution.

Today's best solutions are gitolite
(https://github.com/sitaramc/gitolite) for the server
and cgit for the web interface
(http://git.zx2c4.com/cgit/about/).
470/475
Sharing changes: HTTP Hosting
Create a bare version of your repository
cd /tmp
git clone --bare ~/project project.git
touch project.git/git-daemon-export-ok
Transfer the contents of project.git to a publicly-visible

place (reachable read-only by HTTP for everybody, and
read-write by you through SSH)
Tell people to clone

http://yourhost.com/path/to/project.git
Push your changes using
git push ssh://yourhost.com/path/toproject.git

srcbranch:destbranch
471/475
Tracking Remote Trees
In addition to the ocial Linus Torvalds tree, you might want

to use other development or experimental trees
The git remote command allows to manage remote trees
git remote add rt git://git.kernel.org/pub/scm/

linux/kernel/git/rt/linux-stable-rt.git
Get the contents of the tree
The OMAP tree at git://git.kernel.org/pub/scm/linux/

kernel/git/tmlind/linux-omap.git
The stable realtime tree at git://git.kernel.org/pub/scm/
linux/kernel/git/rt/linux-stable-rt.git
git fetch rt
Switch to one of the branches
git checkout rt/master
472/475
git-gui
http://www.git-scm.com/docs/git-gui
A graphical interface to create and manipulate commits,
replacing multiple git command-line commands.
Not meant for history browsing (opens gitk when needed).
Example usage on Ubuntu/Debian:

sudo apt-get install git-gui
git gui blame Makefile
473/475
About Git
We have just seen the very basic features of Git.
Many more interesting features are available (rebasing, bisection,
merging and more). For more details:
Git Manual
http://schacon.github.com/
git/user-manual.html
Git Book (freely available on-line,

or in print form)
http://git-scm.com/book
Git ocial website

http://git-scm.com/
Video: James Bottomley's tutorial

on using Git
http://bit.ly/2fZJxLZ
474/475
Practical lab - Going further: git
Get familiar with git by

contributing to a real project: the
Linux kernel
Send your patches to the

maintainers and mailing lists.
475/475

Linux Kernel Tutorials

Uploaded by

Copyright:

Available Formats

Linux Kernel Tutorials

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Linux Kernel Tutorials

Uploaded by

Copyright:

Available Formats

Linux Kernel and Driver Development Training

Linux Kernel and Driver

Hyperlinks in the document

Kernel documentation links:

Links to kernel source les and directories:

Links to the declarations, denitions and instances of kernel

Free Electrons at a glance

Engineering company created in 2004

Locations: Orange, Toulouse, Lyon (France)

Serving customers all around the world

Focus: Embedded Linux, Linux kernel Free Software / Open

Activities: development, training, consulting, technical

Added value: get the best of the user and development

Free Electrons on-line resources

All our training materials:

Generic course information

Hardware used in this training session

processors (PRUs) and lots of peripherals.

microSD, micro HDMI

expansion buses (I2C, SPI, UART and more)

capes. See http://beagleboardtoys.com/.

Do not damage your BeagleBone Black!

Do not remove power abruptly:

Boards components have been damaged by removing the

Do not leave your board powered on a metallic surface (like a

Shopping list: hardware for this course

BeagleBone Black - Multiple distributors:

Only the gender of their connector changes.

Don't hesitate to ask questions. Other people in the audience

This helps the trainer to detect any explanation that wasn't

Don't hesitate to share your experience, for example to

Your point of view is most valuable, because it can be similar

Your participation can make our session more interactive and

Practical lab guidelines

We cannot support more than 8 workstations at once (each

So, if you are more than 8 participants, please form up to 8

Open the electronic copy of your lecture materials, and use it

Don't hesitate to copy and paste commands from the PDF

Advise: write down your commands!

You can save a lot of time re-using

This helps to replay your work if

You build a reference to remember

As in the Free Software and Open Source community, cooperation

If you complete your labs before other people, don't hesitate

Explain what you understood to other participants when

Don't hesitate to report potential bugs to your instructor.

Don't hesitate to look for solutions on the Internet as well.

Command memento sheet

This memento sheet gives

It saves us 1 day of UNIX / Linux

Our best tip: in the command line

Get an electronic copy on

The vi editor is very useful to

Though not very user friendly at

Get an electronic copy on

You can also take the quick tutorial

Practical lab - Training Setup

Prepare your lab environment

Download and extract the lab

Linux Kernel Introduction