Linux_10_Embedded Linux Kernel and Driver

Development _ VII

1
Driver development
DMA

2
 DMA memory constraints
Need to use contiguous memory in physical space.
Can use any memory allocated by kmalloc (up to 128 KB) or
__get_free_pages (up to 8MB).
Can use block I/O and networking buffers,
designed to support DMA.
Can not use vmalloc memory
(would have to setup DMA on each individual page).

3
 Reserving memory for DMA
To make sure you've got enough RAM for big DMA transfers...
Example assuming you have 32 MB of RAM, and need 2 MB for DMA:
Boot your kernel with mem=30
The kernel will just use the first 30 MB of RAM.
Driver code can now reclaim the 2 MB left:
dmabuf = ioremap (
0x1e00000, /* Start: 30 MB */
0x200000 /* Size: 2 MB */
);

4
 Memory synchronization issues
Memory caching could interfere with DMA
Before DMA to device:
Need to make sure that all writes to DMA buffer are committed.
After DMA from device:
Before drivers read from DMA buffer, need to make sure that memory caches are flushed.
Bidirectional DMA
Need to flush caches before and after the DMA transfer.

5
 Linux DMA API
The kernel DMA utilities can take care of:
Either allocating a buffer in a cache coherent area,
Or make sure caches are flushed when required,
Managing the DMA mappings and IOMMU (if any).
See Documentation/DMA-API.txt
for details about the Linux DMA generic API.
Most subsystems (such as PCI or USB) supply their own DMA API, derived from the generic
one. May be sufficient for most needs.

6
 Limited DMA address range?
By default, the kernel assumes that your device
can DMA to any 32 bit address. Not true for all devices!
To tell the kernel that it can only handle 24 bit addresses:
if (dma_set_mask (dev, /* device structure */
0x00ffffff /* 24 bits
*/
))
use_dma = 1; /* Able to use DMA */
else
use_dma = 0; /* Will have to do without DMA */

7
Coherent or streaming DMA mappings
Coherent mappings Streaming mappings
The kernel allocates a suitable buffer The kernel just sets the mapping for a buffer provided by
and sets the mapping for the driver. the driver.
Can simultaneously be accessed by the CPU Use a buffer already allocated
and device. by the driver.
So, has to be in a cache coherent memory Mapping set up for each transfer. Keeps DMA registers
area. free on the hardware.
Usually allocated for the whole time the Some optimizations also available.
module is loaded.
The recommended solution.
Can be expensive to setup and use on some
platforms.

8
 Allocating coherent mappings
The kernel takes care of both the buffer allocation and mapping:
include <asm/dma-mapping.h>
void * /* Output: buffer
address */
dma_alloc_coherent(
struct device *dev, /* device structure */
size_t size, /* Needed buffer size in bytes */
dma_addr_t *handle, /* Output: DMA bus address */
gfp_t gfp /* Standard GFP flags */
);
void dma_free_coherent(struct device *dev,
size_t size, void *cpu_addr, dma_addr_t handle);

9
 DMA pools (1)
dma_alloc_coherent usually allocates buffers with __get_free_pages (minimum: 1 page).
You can use DMA pools to allocate smaller coherent mappings:
<include linux/dmapool.h>
Create a DMA pool:
struct dma_pool *
dma_pool_create (
const char *name, /* Name string */
struct device *dev, /* device structure */
size_t size, /* Size of pool buffers */
size_t align, /* Hardware alignment (bytes) */
size_t allocation /* Address boundaries not to be crossed */
);

10
 DMA pools (2)
Allocate from pool
void * dma_pool_alloc ( Note
struct dma_pool *pool, DMA pools only
gfp_t mem_flags, used by USB core
dma_addr_t *handle and 2 SCSI
); drivers

Free buffer from pool

void dma_pool_free (
struct dma_pool *pool,
void *vaddr,
dma_addr_t dma);
Destroy the pool (free all buffers first!)
void dma_pool_destroy (struct dma_pool *pool);

11
 Setting up streaming mappings
Works on buffers already allocated by the driver
<include linux/dmapool.h>
dma_addr_t dma_map_single(
struct device *, /* device structure */
void *, /*
input: buffer to use */
size_t, /*
buffer size */
enum dma_data_direction /* Either DMA_BIDIRECTIONAL,
DMA_TO_DEVICE or DMA_FROM_DEVICE */
);
void dma_unmap_single(struct device *dev, dma_addr_t handle, size_t
size, enum dma_data_direction dir);

12
 DMA streaming mapping notes
When the mapping is active: only the device should access the buffer (potential cache issues
otherwise).
The CPU can access the buffer only after unmapping!
Use locking to prevent CPU access to the buffer.
Another reason: if required, this API can create an intermediate bounce buffer (used if the given
buffer is not usable for DMA).
The Linux API also supports scatter / gather DMA streaming mappings.

13
 DMA summary
Most drivers can use the specific API provided by their
subsystem: USB, PCI, SCSI... Otherwise they can use the Linux generic API:
Coherent mappings Streaming mappings
DMA buffer allocated by the kernel DMA buffer allocated by the driver
Set up for the whole module life Set up for each transfer
Can be expensive. Not recommended. Cheaper. Saves DMA registers.
Let both the CPU and device Only the device can access the buffer
access the buffer at the same time. when the mapping is active.
Main functions: Main functions:
dma_alloc_coherent dma_map_single
dma_free_coherent dma_unmap_single

14
Driver development
New Device Model

15
 Device Model features (1)
Originally created to make power management simpler
Now goes much beyond.
Used to represent the architecture and state of the system
Has a representation in userspace: sysfs
Now the preferred interface with userspace (instead of /proc)
Easy to implement thanks to the device interface:
include/linux/device.h

16
 Device model features (2)
Allows to view the system for several points of view:
From devices existing in the system: their power state, the bus they are
attached to, and the driver responsible for them.
From the system bus structure: which bus is connected to which bus
(e.g. USB bus controller on the PCI bus), existing devices and devices
potentially accepted (with their drivers)
From the various kinds ("classes") of devices: input, net, sound...
Existing devices for each class. Convenient to find all the input devices
without actually knowing how they are physically connected.

17
sysfs
Userspace representation of the Device Model.
Configure it with
CONFIG_SYSFS=y (Filesystems -> Pseudo
filesystems)
Mount it with
sudo mount -t sysfs none /sys
Spend time exploring /sys on your workstation!

18
 sysfs tools
http://linux-diag.sourceforge.net/Sysfsutils.html
libsysfs - The library's purpose is to provide a consistent and stable
interface for querying system device information exposed through sysfs.
Used by udev (see later).
systool - A utility built upon libsysfs that lists devices by bus,
class, and topology.

19
 Device Model references
Very useful and clear documentation in the kernel
sources!
Documentation/driver-model/
Documentation/filesystems/sysfs.txt

20
Ethernet over USB

21
 Ethernet over USB (1)
If your platform doesn't have an Ethernet controller, but has an USB
controller capable of acting as a USB device, you can use Ethernet over
USB
On the target, compile the USB Gadget support (CONFIG_USB_GADGET)
and the USB Ethernet gadget driver (CONFIG_USB_ETH).
USB Gadget support requires support for the target USB controller,
many of them are already supported by the Linux kernel
If compiled as a module, the USB Ethernet gadget driver is g_ether.

22
 Ethernet over USB (2)
Once the module is loaded, an usb0 network interface will appear,
that should be configured as usual (IP address, etc.)
On the host, compile the usbnet driver (CONFIG_USB_USBNET), and
load this module. In most distributions kernels, it is already available.
An usb0 network interface will also appear on the host, it should be
configured properly to be able to dialog with the target (IP addresses
in the same network)

23
 Ethernet over USB (3)
root@om-gta02:~# ifconfig usb0
usb0 Link encap:Ethernet HWaddr 3E:04:A8:D5:9E:7E
inet addr:192.168.0.202 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[...] Target
USB
thomas@surf:~$ sudo ifconfig usb0 Host
usb0 Link encap:Ethernet HWaddr be:26:39:84:59:80
inet addr:192.168.0.200 Bcast:192.168.0.223 Mask:255.255.255.224
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[...]
thomas@surf:~$ ping -c 1 192.168.0.202
PING 192.168.0.202 (192.168.0.202) 56(84) bytes of data.
64 bytes from 192.168.0.202: icmp_seq=1 ttl=64 time=1.80 ms

--- 192.168.0.202 ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.804/1.804/1.804/0.000 ms

24
 Advice and resources
Getting help and contributions

25
 Solving issues
If you face an issue, and it doesn't look specific to your work but rather to the tools
you are using, it is very likely that someone else already faced it.
Search the Internet for similar error reports.
You have great chances of finding a solution or workaround, or at least an
explanation for your issue.
Otherwise, reporting the issue is up to you!

26
 Getting help
If you have a support contract, ask your vendor.
Otherwise, don't hesitate to share your questions and issues
Either contact the Linux mailing list for your architecture (like linux-arm-kernel or linuxsh-dev...).
Or contact the mailing list for the subsystem you're dealing with (linux-usb-devel, linux-mtd...). Don't ask
the maintainer directly!
Most mailing lists come with a FAQ page. Make sure you read it before contacting the mailing list.
Useful IRC resources are available too
(for example on http://kernelnewbies.org).
Refrain from contacting the Linux Kernel mailing list, unless you're an experienced developer and need
advice.

27
 Getting contributions
Applies if your project can interest other people:
developing a driver or filesystem, porting Linux on a new processor, board
or device available on the market...
External contributors can help you a lot by
Testing
Writing documentation
Making suggestions
Even writing code

28
Encouraging contributions
Open your development process: mailing list, Wiki, public CVS
read access
Let everyone contribute according to their skills and interests.
Release early, release often
Take feedback and suggestions into account
Recognize contributions
Make sure status and documentation are up to date
Publicize your work and progress to broader audiences

29
 Advice and resources
Bug report and patch submission

30
 Reporting Linux bugs
First make sure you're using the latest version
Make sure you investigate the issue as much as you can:
see Documentation/BUG-HUNTING
Make sure the bug has not been reported yet. A bug tracking system
(http://bugzilla.kernel.org/) exists but very few kernel developers use it. Best to use web search
engines (accessing public mailing list archives)
If the subsystem you report a bug on has a mailing list, use it. Otherwise, contact the official
maintainer (see the MAINTAINERS file). Always give as many useful details as possible.

31
 How to submit patches or drivers
Don't merge patches addressing different issues
You should identify and contact the official maintainer
for the files to patch.
See Documentation/SubmittingPatches for details. For trivial patches, you can copy
the Trivial Patch Monkey.
See also http://kernelnewbies.org/UpstreamMerge for very helpful advice to have your code
merged upstream (by Rik van Riel).
Special subsystems:
ARM platform: it's best to submit your ARM patches to Russell King's patch system:
http://www.arm.linux.org.uk/developer/patches/

32
 How to become a kernel developer?
Greg Kroah-Hartman wrote useful guidelines in the kernel documentation:
Documentation/HOWTO
How to Participate in the Linux Community (by Jonathan Corbet)
A Guide To The Kernel Development Process
http://ldn.linuxfoundation.org/documentation/how-participate-linux-community
Read this last document first.
It is probably sufficient!

33
Advice and resources
References

34
 Specific training materials
Free Electrons is working on dedicated training materials for specific device
/ driver types:
Linux USB drivers
http://free-electrons.com/articles/linux-usb
More will be available in the next months: block, network, input, audio,
graphics...
Don't hesitate to ask us to create the ones you need for a training session!

35
 Information sites (1)
Linux Weekly News
http://lwn.net/
The weekly digest off all Linux and free software information
sources
In depth technical discussions about the kernel
Subscribe to finance the editors ($5 / month)
Articles available for non subscribers
after 1 week.

36
Information sites (2)
KernelTrap
http://kerneltrap.org/
Forum website for kernel developers
News, articles, whitepapers, discussions, polls,
interviews
Perfect if a digest is not enough!

37
 Useful reading (1)
Linux Device Drivers, 3rd edition, Feb 2005
By Jonathan Corbet, Alessandro Rubini,
Greg Kroah-Hartman, O'Reilly
http://www.oreilly.com/catalog/linuxdrive3/
Freely available on-line!
Great companion to the printed book
for easy electronic searches!
http://lwn.net/Kernel/LDD3/ (1 PDF file per chapter)
http://free-electrons.com/community/kernel/ldd3/ (single PDF file)
A must-have book for Linux device driver writers!

38
 Useful reading (2)
Essential Linux Device Drivers, April 2008
By Sreekrishnan Venkateswaran, an indian
embedded IBM engineer with > 10 years experience
Covers a wide range of topics not covered by LDD :
serial drivers, input drivers, I2C, PCMCIA and
Compact Flash, PCI, USB, video drivers, audio
drivers, block drivers, network drivers, Bluetooth,
IrDA, MTD, drivers in userspace, kernel debugging,
etc.
« Probably the most wide ranging and complete
Linux device driver book I've read » -- Alan Cox

39
 Useful reading (3)
Linux Kernel in a Nutshell, Dec 2006
By Greg Kroah-Hartman, O'Reilly
http://www.kroah.com/lkn/
A good reference book and guide on configuring,
compiling and managing the Linux kernel sources.
Freely available on-line!
Great companion to the printed book
for easy electronic searches!
Available as single PDF file on
http://free-electrons.com/community/kernel/lkn/

40
Useful reading (4)
Linux Kernel Development, 2nd Edition, Jan 2005
Robert Love, Novell Press
http://free-electrons.com/redirect/lkd2-book.html
A very synthetic and pleasant way to learn about kernel
subsystems (beyond the needs of device driver writers)
Understanding the Linux Kernel, 3rd edition, Nov 2005
Daniel P. Bovet, Marco Cesati, O'Reilly
http://oreilly.com/catalog/understandlk/
An extensive review of Linux kernel internals,
covering Linux 2.6 at last.
Unfortunately, only covers the PC architecture.

41
 Useful on-line resources
Linux kernel mailing list FAQ
http://www.tux.org/lkml/
Complete Linux kernel FAQ
Read this before asking a question to the mailing list
Kernel Newbies
http://kernelnewbies.org/
Glossary, articles, presentations, HOWTOs,
recommended reading, useful tools for people
getting familiar with Linux kernel or driver development.
Kernel glossary:
http://kernelnewbies.org/KernelGlossary

42
 Embedded Linux Wiki
The embedded Linux Wiki contains loads of useful resources
for embedded systems developers:
Many HOWTO documents of all kinds, covering topics like system size,
boot time, multimedia, power management, toolchains...
Kernel patches not available in mainstream yet (e.g. Linux Tiny)
Community resource: hacker interviews, book reviews,
event coverage...
Is open to everyone. Contributions are welcome!
http://elinux.org

43
 ARM resources
ARM Linux project: http://www.arm.linux.org.uk/
Developer documentation: http://www.arm.linux.org.uk/developer/
arm-linux-kernel mailing list:
http://lists.arm.linux.org.uk/mailman/listinfo/linux-arm-kernel
FAQ: http://www.arm.linux.org.uk/armlinux/mlfaq.php
How to post kernel fixes:
http://www.arm.uk.linux.org/developer/patches/
ARMLinux @ Simtec: http://armlinux.simtec.co.uk/
A few useful resources: FAQ, documentation and Who's who!
ARM Limited: http://www.linux-arm.com/
Wiki with links to useful developer resources

44
 International conferences (1)
Useful conferences featuring Linux kernel presentations
Ottawa Linux Symposium (July): http://linuxsymposium.org/
Lots of kernel topics by major kernel hackers.
Freely available proceedings.
Fosdem: http://fosdem.org (Brussels, February)
For developers. Kernel presentations from well-known kernel hackers.
Embedded Linux Conference: http://embeddedlinuxconference.com/
Organized by the CE Linux Forum: California
(San Jose, April), in Europe (November). Frequent technical sessions in Japan. Very
interesting kernel topics for embedded systems developers. Presentation slides freely
available.

45
International conferences (2)
linux.conf.au: http://linux.org.au/conf/ (Australia / New Zealand)
Features a few presentations by key kernel hackers.
Don't miss our free conference videos on
http://free-electrons.com/community/videos/conferences/!

46
Advice and resources
Last advice

47
Use the Source, Luke!
Many resources and tricks on the Internet find you
will, but solutions to all technical issues only in the
Source lie.

Thanks to LucasArts

48
 Annexes
Quiz answers

49
 Quiz answers
Interrupt handling
Q: Why did the kernel segfault at module unload (forgetting to unregister a handler in a shared
interrupt line)?
A: Kernel memory is allocated at module load time, to host module code. This memory is freed at
module unload time. If you forget to unregister a handler and an interrupt comes, the cpu will try to
jump to the address of the handler, which is in a freed memory area. Crash!

50
 Annexes
Kernel sources

51
Checking the integrity of sources
Kernel source integrity can be checked through OpenPGP digital signatures.
Full details on http://www.kernel.org/signature.html
Details about GnuPG: http://www.gnupg.org/gph/en/manual.html
Import the public GnuPG key of kernel developers:
gpg --keyserver pgp.mit.edu --recv-keys 0x517D0F0E
If blocked by your firewall, look for 0x517D0F0E on http://pgp.mit.edu/,
copy and paste the key to a linuxkey.txt file:
gpg --import linuxkey.txt
Download the signature file corresponding to your kernel version
(at the same location), and run the signature check:
gpg --verify linux-2.6.11.12.tar.bz2.sign

52
 Annexes
Slab caches and memory pools

53
Slab caches
Also called lookaside caches
Slab caches: Objects that can hold any number
of memory areas of the same size.
Optimum use of available RAM
and reduced fragmentation.
Mainly used in Linux core subsystems: filesystems (open files, inode
and file caches...), networking... Live stats on /proc/slabinfo.
May be useful in device drivers too, though not used so often.
Linux 2.6: used by USB and SCSI drivers.

54
Slab cache API (1)
#include <linux/slab.h>
Creating a cache:
cache = kmem_cache_create (
name, /* Name for /proc/slabinfo
*/
size, /* Cache object size */
align, /* Cache alignment */
flags, /* Options: DMA, debugging,
tracing... */
constructor); /* Optional, called after each allocation */
Example: drivers/usb/host/uhci-hcd.c
uhci_up_cachep = kmem_cache_create(
"uhci_urb_priv", sizeof(struct urb_priv),
0, 0, NULL);

55
 Slab cache API (2)
Since Linux 2.6.22, a macro can simplify cache creation in most cases:
#define KMEM_CACHE(__struct, __flags)\
kmem_cache_create(#__struct,\
sizeof(struct __struct),\
__alignof__(struct __struct),\
(__flags), NULL)
Example: kernel/pid.c
pid_cachep = KMEM_CACHE(pid, SLAB_PANIC);

56
 Slab cache API (3)
Allocating from the cache:
object = kmem_cache_alloc (cache, flags);
or object = kmem_cache_zalloc (cache, flags);
Freeing an object:
kmem_cache_free (cache, object);
Destroying the whole cache:
kmem_cache_destroy (cache);
More details and an example in the Linux Device Drivers book:
http://lwn.net/images/pdf/LDD3/ch08.pdf

57
 Memory pools
Useful for memory allocations that cannot fail
Kind of lookaside cache trying to keep a minimum number of pre-
allocated objects ahead of time.
Use with care: otherwise can result in a lot of unused memory that
cannot be reclaimed! Use other solutions whenever possible.

58
Memory pool API (1)
#include <linux/mempool.h>
Mempool creation:
mempool = mempool_create (
min_nr,
alloc_function,
free_function,
pool_data);

59
 Memory pool API (2)
Allocating objects:
object = mempool_alloc (pool, flags);
Freeing objects:
mempool_free (object, pool);
Resizing the pool:
status = mempool_resize (
pool, new_min_nr, flags);
Destroying the pool (caution: free all objects first!):
mempool_destroy (pool);

60
Memory pool implementation
Call alloc
mempool_create function
min_nr
times

No Take an
Call alloc
mempool_alloc Success? object from
function
the pool

Yes

pool Yes
count Add freed
mempool_free New object
< object to pool
min_nr?
No

Call free
function
on object

61
 Memory pools using slab caches
Idea: use slab cache functions to allocate and free objects.
The mempool_alloc_slab and mempool_free_slab functions supply a link with slab
cache routines.
So, you will find many code examples looking like:
cache = kmem_cache_create (...);
pool = mempool_create (
min_nr,
mempool_alloc_slab,
mempool_free_slab,
cache);
There's a shorthand pool creation function for this case:
pool = mempool_create_slab_pool(min_nr, cache);

62
 Annexes
Init runlevels

63
System V init runlevels (1)
Introduced by System V Unix
Much more flexible than in BSD
/etc/initab excerpt:
Make it possible to start or stop
id:5:initdefault:
different services for each
runlevel # System initialization.
si::sysinit:/etc/rc.d/rc.sysinit
Correspond to the argument l0:0:wait:/etc/rc.d/rc 0
given to /sbin/init. l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
Runlevels defined in l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
/etc/inittab. l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

64
System V init runlevels (2)
Standard levels Customizable levels: 2, 3, 4, 5
init 0 init 3
Halt the system Often multi-user mode, with
init 1 only command-line login
Single user mode for maintenance init 5
init 6 Often multi-user mode, with
Reboot the system graphical login

init S
Single user mode for maintenance.
Mounting only /. Often identical to 1

65
init scripts
According to /etc/inittab settings, init <n> runs:
First /etc/rc.d/rc.sysinit for all runlevels
Then scripts in /etc/rc<n>.d/
Starting services (1, 3, 5, S):
runs S* scripts with the start option
Killing services (0, 6):
runs K* scripts with the stop option
Scripts are run in file name lexical order
Just use ls -l to find out the order!

66
/etc/init.d
Repository for all available init scripts
/etc/rc<n>.d/ only contains links to the /etc/init.d/
scripts needed for runlevel n
/etc/rc1.d/ example (from Fedora Core 3)

67
Handling init scripts by hand
Simply call the /etc/init.d scripts!
sudo /etc/init.d/sshd start
Starting sshd: [ OK ]
sudo /etc/init.d/nfs stop
Shutting down NFS mountd: [FAILED]
Shutting down NFS daemon:
[FAILED]Shutting down NFS quotas:
[FAILED]
Shutting down NFS services: [ OK ]
sudo /etc/init.d/pcmcia status
cardmgr (pid 3721) is running...
sudo /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

68
 Related documents
All the technical presentations and training materials created and used by Free Electrons,
available under a free documentation license (more than 1500 pages!).

http://free-electrons.com/training Linux USB drivers

Introduction to Unix and GNU/Linux Real-time in embedded Linux systems
Embedded Linux kernel and driver development Introduction to uClinux
Free Software tools for embedded Linux systems Linux on TI OMAP processors
Audio in embedded Linux systems Free Software development tools
Multimedia in embedded Linux systems Java in embedded Linux systems
Introduction to GNU/Linux and Free
http://free-electrons.com/articles Software

Advantages of Free Software in embedded Linux and ecology

systems
What's new in Linux 2.6?
Embedded Linux optimizations
How to port Linux on a new PDA
Embedded Linux from Scratch... in 40 min!

69
Thank You

70