Kernel configuration and building in Linux 2.

5
Kai Germaschewski Sam Ravnborg
University of Iowa Ericsson Diax A/S
kai@germaschewski.name sam@ravnborg.org

Abstract cial solution?

As the Linux kernel evolved from a student’s

The development phase of Linux 2.5 brought terminal emulation program towards a full-
substantial changes to the kernel configuration featured UNIX-like kernel, changes to the way
process, the actual kernel build and, in partic- it was built became necessary and were inte-
ular, implementation and building of loadable grated, so the kernel build system basically fol-
modules. lowed the evolutionary development of the ker-
nel itself.
The first part of this paper will give an
overview of the user-visible changes which oc- In the science world, in particular people run-
cured in Linux 2.5, on the one hand for users ning numerical simulations, many people con-
which build kernels themselves, on the other sider a build system completely unnecessary,
hand for developers which maintain drivers or they just run
other parts of the kernel, in order to help port-
ing to Linux 2.5/2.6. f77 code.f
./a.out
The second part of the paper deals with the ac-
tual design and implementation of the current
However, this approach obviously doesn’t
kbuild, showing how GNU make is actually
scale to large projects. To keep projects main-
flexible enough to allow for nice condensed
tainable, some kind of modularization occurs
Makefile fragments which per subdirectory de-
the code is divided into a number of source files
scribe which objects to build into the kernel or
and as the project is growing further, a direc-
as loadable modules. The paper ends with an
tory hierarchy is introduced which helps orga-
outlook showing possible approaches for im-
nizing the code even further.
plementing additional features.
During development, normally only one or a
The paper also explains the improvements in
few files are edited and then the developer
handling loadable kernel modules, including
wants to rebuild the program, in this case the
symbol versioning, and the necessary build
kernel vmlinux, be it just for compile-time
system changes.
checks or testing.

First of all, one does not want to enter all the

1 Introduction and history commands manually for each build, so some
type of script is necessary to record those com-
Why is a kernel build system necessary at all, mands. Next, it is actually a waste to recom-
and why does the Linux kernel use its own spe- pile every file if only few have changed. Smart
Linux Symposium 186

programmers recognized this a long time ago support in configuration and building of
and invented a tool called make which is still those objects.
the most popular build tool used today. We as-
sume in this paper that the audience is familiar
with the basics of make. In particular the high configurability and sup-
port for loadable modules distinguish the
So even Linux 0.01 came already with a Make- Linux kernel from most other projects, and it
file which took care of building the kernel. thus comes as no surprise that its build system
also evolved away from a standard Makefile.
As time passed and Linux matured, new fea-
However, make is still the underlying tool used
tures were incorporated into the build system,
for building the kernel. In fact, the extensibility
such as
of the GNU version of make [1] in conjunction
with some support scripts / C code renders it
• Automatic generation of dependency possible to meet the goals listed above.
information. make only handles simple
dependencies like the dependency of an 2 A dummy’s guide to kbuild
object file on the corresponding source au-
tomatically, other prerequisites as for ex-
ample included header files need to be This section is addressed to users and will ex-
added to the Makefile explicitly, a task plain how to use the kernel build system in
which can be (and was) automated. Linux-2.5/2.6. “Users” (as opposed to “devel-
opers”) here mean people who download the
• Configurability. As the code base for linux kernel tree source, possibly apply patches
the Linux kernel expanded, a need for a and then build and install their own kernels. Of
user selectable configuration became ap- course, since kernel developers need to build
parent and was introduced before release and run kernels, too, this section is of relevance
of Linux 1.0. This system allows the user for them as well.
to answer questions with respect to which
The build system is based on GNU make, i.e.
components are desired, and then only all commands are given to make by invoking it
builds those components into the kernel. as

• Different architectures and cross–

make <target>
compilation. Linux introduced support
for different architectures, which means
the kernel is build from a large arch- Contrary to many userspace packages which
independent code base as well as some are using autoconf/automake, there is no pre-
machine-specific low-level code. It is ceding ./configure necessary, the neces-
also often necessary to cross–compile sary configuration process is embedded into
the kernel, i.e. do the compilation on a the build process.
different platform than it is actually run
on. The actual targets are in part platform-specific,
for example on i386 one typical wants to build
• Loadable modules. Within Linux 1.3, the boot image bzImage and modules. A list
support for loadable kernel modules was of supported targets for the platform can be ob-
introduced, which again needed special tained from make help.
Linux Symposium 187

Arch maintainers should setup their arch- tween those two steps if a change of configura-
specific Makefile in a way that invoking make tion options is desired.
without parameters will build the commonly
used boot target for the architecture, for exam- The last remaining step is the installation of
ple on i386 just typing the newly built kernel. The procedure to in-
stall the boot image depends of course on the
make bootloader used.

For lilo, the kernel boot image bzImage should

will build bzImage and modules (the latter be copied to a certain location (typically /boot),
only when CONFIG_MODULES is selected, of then /etc/lilo.conf may need an appropriate en-
course) which is what is typically needed. try and finally /sbin/lilo must be run.
If one just runs make after unpacking the ker- For grub, copying the kernel image to /boot
nel source tarball, make will actually just error and possibly editing /etc/grub.conf should suf-
out, asking you to configure your kernel first fice.
by running make *config. (In Linux-2.4
and before, it would invoke make config An important change is that on i386 bzIm-
for you, but this is the wrong choice in 99% age/zImage can not be directly booted from
of the cases, since nobody likes answering a a floppy disk anymore. Instead the targets
straight sequence of a couple of hundred ques- zdisk and fdimage create a boot floppy
tions. . . ) disk and a boot disk image, respectively. Those
targets now require mtools and syslinux to be
To generate a new kernel configuration, it is installed.
recommended to use make menuconfig,
make xconfig (which uses Qt now) or Since the actual installation of the boot
make gconfig (uses gtk). image varies as described above, one can
give the install target to make, which
However in most cases, it is easier to adapt an will invoke a user– or distro–provided
existing kernel configuration to the current ker- script, ˜/bin/installkernel or
nel than to create a new one from scratch. This /sbin/installkernel which can be
is done by copying the .config file into the top- customized for the local setup.
level directory of the source tree. kbuild will
recognize that the .config file may need adap- Installing modules is simpler, just invok-
tion for the current kernel source and automat- ing make modules_install will do the
ically run make oldconfig for you, which necessary work. By default this will in-
makes sure that .config is consistent with the stall into /lib/modules/‘uname -r‘/,
current rules and asks the user about the value though this can be customized by setting
of previously not existing options. INSTALL_MOD_PATH, e.g. if one wants to col-
lect the modules for transfer onto a different
So the normal sequence for building a kernel is machine.
just
This is basically all knowledge which is needed
cp /my/old/.config .config to build a Linux kernel—everything else is
make handled automatically by the build system.
Applying patches, editing files, changing con-
where one could insert a make *config be- figuration options or adding compiler flags—
Linux Symposium 188

the build system will notice the change and re- to override the SUBDIRS variable on the com-
build whatever is needed. The one exception mand line, which will force make to only de-
to this rule is changing the architecture (by set- scend into the given subtree. This can be very
ting the ARCH variable), which needs an ex- useful for faster build times, but it bypasses
plicit make distclean to work correctly. some dependencies and thus does not guaran-
tee to result in a consistent state.

3 kbuild for kernel developers So while e.g. working on the hisax ISDN
driver, it’s useful to call make as
3.1 kbuild in the daily work
make SUBDIRS=drivers/isdn/hisax \
modules
Since developers tend to build kernels and
modules a lot, the previous section of course
also applies to them, in particular the fact that for compile checks etc. However, before
just running make will recognize all changes installing a new kernel and modules, the
and rebuild whatever is necessary to generate a authors advise to always run a full make
consistent vmlinux and modules. bzImage/vmlinux/modules (or other-
wise, do not complain ;).
Some additional features exist to support the
development / debugging process:
3.2 Integrating a driver

• make some/path/file.o will re- Basically each subdirectory in the Linux ker-
build the single file given, using compiler nel tree contains a file called Makefile, which
flags (e.g. -DMODULE) according to the is included by make during the kernel build
current .config. process. However, these Makefiles are differ-
ent from regular Makefiles in that they nor-
• make some/path/file.i will mally don’t have any targets or rules, but only
generate a preprocessed version of set variables which tell the build process what
/some/path/file.c, again using should be built and the latter takes control of
compiler flags for the current configura- the actual compiling and linking.
tion.
In conjunction with the Makefile there nor-
• make some/path/file.s will gen- mally exists a Kconfig file, these files were
erate a file containing the raw assembler introduced with the configurator rewrite by
code for some/path/file.[cS]. Roman Zippel and replace the old Con-
fig.in/Config.help files used during the config-
• make some/path/file.lst (little uration phase of the kernel build.
known but very useful) gives interspersed
assembler code with the C source, relo- This paper does not intend to elaborate on the
cated to the correct virtual address when a new kernel configuration system, however the
current System.map exists. following examples will provide some basic
usage guidance.

Another useful feature for the daily work, The most common case is adding a new driver
which has existed for a long time, is the ability which is built from a single source file.
Linux Symposium 189

config TIGON3
tristate "Broadcom Tigon3 support"
depends on PCI
help
This driver supports Broadcom Tigon3 based gigabit Ethernet cards.

If you want to compile this driver as a module ( = code which can be

inserted in and removed from the running kernel whenever you want),
say M here and read <file:Documentation/modules.txt>. This is
recommended. The module will be called tg3.

Figure 1: Kconfig fragment for the Tigon3 driver.

Figure 1 shows the Kconfig fragment for the be compiled as built-in objects, and will
Tigon3 driver, which defines a config option finally be linked into vmlinux.
TIGON3 (the corresponding variable will be
given the name CONFIG_TIGON3), which is • obj-m. All objects listed in obj-m and
a tristate, i.e. can have the values y, m, or not not listed in obj-y will be compiled
n with the obvious meanings (a config option as modules (so they actually end up being
which has been turned off, actually has the called e.g. tg3.ko in 2.5/2.6).
value "", to be correct here). The fragment • obj-. All objects listed in obj- and not
depends on PCI states that this option is in obj-y or obj-m will be ignored by
only selectable when the option PCI is also set kbuild.
(that is, if the kernel supports the PCI bus).
The Makefile fragment for the Tigon3 driver Since the build system does not have any fur-
ther information on tg3.o, it will try to build
obj-$(CONFIG_TIGON3) += tg3.o
it from a source file called tg3.c (or an as-
sembler source tg3.S, which only happens in
the architecture dependent part of the kernel,
is very short and though a little awkward at though).
first, a very elegant way to quickly express
what files are supposed to be built. The This is all what is needed to integrate a simple
idea dates back to Micheal Elizabeth Castain’s driver into the kernel build, other than of course
dancing Makefiles [2] and was globally intro- writing the driver (tg3.c) itself.
duced into the kernel by Linus shortly before It is also possible to list more than one object to
the release of kernel 2.4. be built in the Makefile statement. The Make-
file line dealing with the eepro100 driver looks
What happens is that depending on the con- like the following:
fig option CONFIG_TIGON3, the value tg3.o
is appended to either of the variables obj-y, obj-$(CONFIG_EEPRO100) += \
obj-m or obj-. eepro100.o mii.o

The meaning of those special variables is as

If this driver is selected, the mii.o support
follows:
module also needs to be compiled, which is
achieved by simply appending it to the state-
• obj-y. All objects listen in obj-y will ment.
Linux Symposium 190

Other network drivers will, if selected, also 4 What is new in Linux-2.5/2.6’s

add mii.o to the list of objects to be built— kbuild?
this is fine, the build system handles this case.
It is even possible that a support module like
In this section, we describe some of the steps
mii.o got added to the list of built-in objects
in the evolution of the kernel build system
obj-y and obj-m—again, the build system
during the development phase of Linux 2.5.
recognizes this fact and just compiles the built-
One purpose is to show how this evolution
in version, which will also be usable for the
could actually be divided into small, Linus-
drivers compiled modular.
compatible “piece-meal” patches without the
The new e100 driver examplifies two more fea- famous “flag-day” patches and with only little
tures. drivers/net/Makefile only contains the breakage along the way.
line
We will also show how using the extensions
provided by GNU make were actually ex-
ploited to provide a better build system while
still using a standard tool instead of creating a
obj-$(CONFIG_E100) += e100/ specialized build solution for the kernel from
scratch.

We start by comparing drivers/isdn/Makefile

in 2.4 and 2.5 (Figure 4), where many of the
which tells kbuild that it should descend into improvements are easily seen. (a) shows the
the e100/ subdirectory if the option CONFIG_ Makefile as it is present in Linux 2.4.20, and
E100 is set. What to do there will then be (b) shows the simpler variant present in 2.5.
determined by drivers/net/e100/Makefile (Fig- kbuild has been adapted incrementally to allow
ure 2): the more concise syntax. The following sec-
tions will describe the internals that eventually
The first line after the comment looks famil- allowed for the layout seen in (b).
iar, it advises the build system to build e100.o
built-in/modular depending on the value of 4.1 O_TARGET / linking objects in subdirecto-
CONFIG_E100. When CONFIG_E100 equals ries
“m” the e100 driver is built as a module and
will be named em e100.ko. First of all, we start with a short descrip-
tion of what the kbuild interal implementa-
The next line then states that e100.o is a com- tion, which is hidden in the top-level Make-
posite object which should be linked from the file and scripts/Makefile.* typically does in a
subdirectory: From the kbuild Makefile lo-
listed individual object files—these object files cated in the subdirectory we obtain a list
will automatically compiled with the appropri- of what to build from the variables obj-y
ate flags. (built-in) and obj-m (modular) as explained
in the previous section. The default target
As a last point, instead of using the vari- in scripts/Makefile.build is __build and the
able <modname>-objs to declare the compo- corresponding rule, shown in Figure 5, defines
nents of the module <modname>.o, the variant what work needs to be done. Important here
is that we build O_TARGET or L_TARGET, re-
<modname>-y can be used, which allows for spectively, when building vmlinux and obj-m
easy definition of optional parts to a composite when compiling modules. As opposed to 2.4,
modules, as seen in the example in Figure 3. in 2.5 O_TARGET is a kbuild internal variable
Linux Symposium 191

#
# Makefile for the Intels E100 ethernet driver

obj-$(CONFIG_E100) += e100.o

e100-objs := e100_main.o e100_config.o e100_phy.o \

e100_eeprom.o e100_test.o

Figure 2: /drivers/net/e100/Makefile

#
# Makefile for the Linux X.25 Packet layer.
#

obj-$(CONFIG_X25) += x25.o

x25-y := af_x25.o x25_dev.o x25_facilities.o x25_in.o \

x25_link.o x25_out.o x25_route.o x25_subr.o \
x25_timer.o x25_proc.o
x25-$(CONFIG_SYSCTL) += sysctl_net_x25.o

Figure 3: net/x25/Makefile

and needs no longer be defined in the kbuild

makefiles. Except for the rare case of build-
ing an actual library, O_TARGET is used in the
built-in case and we find the rule how to make vmlinux
it as |-- drivers/built-in.o
| ‘-- isdn/built-in.o
| |-- isdn.o
$(O_TARGET): $(obj-y) FORCE | | |-- isdn_common.o
$(call if_changed,link_o_target) | | ‘-- isdn_net.o
| |
| |-- hisax/built-in.o
| | |-- hisax.o
So O_TARGET is linked from the objects listed | | | |-- config.o
in obj-y, which contains files locally com- | | | ‘-- isdnl*.o
piled in the current directory as well as objects | | ‘-- hisax_fcpcipnp.o
which are built in subdirectories by descend- | |
| ‘-- icn/built-in.o
ing. In Figure 6, we see how going from the | ‘-- icn.o
leaves to the root, the O_TARGET in each subdi- |
rectory (here always called built-in.o) accumu- ‘-- fs/built-in.o
lates the objects built below that directory until
we finally end up with vmlinux at the root of the Figure 6: The hierarchy for linking vmlinux
hierarchy containing all built-in objects gen-
erated throughout the tree (this example only
shows a small fraction of the objects linked in
a normal build).
Linux Symposium 192

(a)

O_TARGET := vmlinux-obj.o
export-objs := isdn_common.o

list-multi := isdn.o
isdn-objs := isdn_net.o isdn_tty.o isdn_v110.o isdn_common.o
isdn-objs-$(CONFIG_ISDN_PPP) += isdn_ppp.o
isdn-objs += $(isdn-objs-y)

obj-$(CONFIG_ISDN) += isdn.o
obj-$(CONFIG_ISDN_PPP_BSDCOMP) += isdn_bsdcomp.o

mod-subdirs := hisax
subdir-$(CONFIG_ISDN_HISAX) += hisax
subdir-$(CONFIG_ISDN_DRV_ICN) += icn

obj-y += $(addsuffix /vmlinux-obj.o, $(subdir-y))

include $(TOPDIR)/Rules.make

isdn.o: $(isdn-objs)
$(LD) -r -o $@ $(isdn-objs)

(b)

obj-$(CONFIG_ISDN) += isdn.o
obj-$(CONFIG_ISDN_PPP_BSDCOMP) += isdn_bsdcomp.o

isdn-y := isdn_net_lib.o isdn_fsm.o isdn_tty.o \

isdn_v110.o isdn_common.o
isdn-$(CONFIG_ISDN_PPP) += isdn_ppp.o

obj-$(CONFIG_ISDN_DRV_HISAX) += hisax/
obj-$(CONFIG_ISDN_DRV_ICN) += icn/

Figure 4: drivers/isdn/Makefile in (a) 2.4.20 and (b) adapted for the build system in 2.5

In Linux 2.4, the name for the object which In kbuild-2.4, we need to explicitly add the
accumulates all built-in objects in and below subdirectories to descend into to the variables
the current subdirectory was chosen by setting subdir-y/m, and then also add the subdir-
generated built-in objects to obj-y so that they
the variable O_TARGET in the local Makefile. get linked. This is redundant and error-prone,
In Linux-2.5, it is instead just set to built-in.o in 2.5 it is sufficient to just add the objects gen-
by the build system. This allows to get rid of erated in the subdirectories to the list of ob-
the assignment of O_TARGET in every subdir jects to be linked, and the build system will de-
Makefile and, more importantly, allows for fur- duce from there that it needs to descend into
the named subdirectories. To simplify things
ther clean-up: further, the name of the O_TARGET (now al-
ways being built-in.o) itself is left out and only
the trailing slash is kept:
Linux Symposium 193

__build: $(if $(KBUILD_BUILTIN),$(O_TARGET) $(L_TARGET) $(extra-y)) \

$(if $(KBUILD_MODULES),$(obj-m)) \
$(subdir-ym) $(always)
@:

Figure 5: The default __build rule from scripts/Makefile.build

obj-$(CONFIG_...HISAX) += hisax/ each multi-part module in the directory depend

obj-$(CONFIG_...ICN) += icn/ on the set of all components for all multi-part
modules in that directory, which is actually
4.2 Multi-part modules too large, as it of course only is dependend
its own components; however the latter is not
As can be seen from Figure 4, a number of implementable within the restrictions of GNU
statements were necessary for generating a make. When doing the link, the variable link_
multi-part module, isdn.o in that example. First multi_deps recovers the right list of compo-
of all, the parts constituting the module need to nents from the target $@, so that linker is in-
be declared by assigning them to the variable voked correctly.
isdn-objs. This step is of course essential
and was kept in 2.5. Another interesting detail is that we here as
well as in other places need to uniquify the
However, it was also necessary to declare that prerequisites, so that listing a component mul-
isdn.o is a multi-part module by listing it in tiple times doesn’t lead to a link error. GNU
the variable list-multi. This information make offers the sort function, which throws
is redundant as it can be deduced by checking away duplicates, however it is unfortunately
for the existence of <module>-objs, which is not usable for this purpose since it sorts, i.e.
now done in 2.5. reorders its arguments and thus changes the
link/init order. The workaround here is to use
Furthermore, in 2.4 a link rule has to be ex-
plicitly given for each multi-part object, which the variable $^ which actually uniquifies the
was annoying and error-prone. In the new build list of prerequisites exactly as needed. Finally,
system, this link rule is generated by make, hit- since $^ lists all prerequisites which as men-
ting just about the limits of what GNU make tioned above exceeds the list of components for
is capable of. We use a feature called “static
pattern rules,” and the code looks like the fol- the current module, we filter the uniquified list
lowing: with that list of components to get the informa-
tion we need.
cmd_link_multi-m = $(LD) ... \
-o $@ $(link_multi_deps) 4.3 Including Rules.make

$(multi-used-m) : \
%.o: $(multi-objs-m) FORCE Each subdirectory Makefile in kbuild-2.4
$(call if_changed,link_multi-m) needed to include $(TOPDIR)/Rules.make ex-
plicitly. In 2.5, when descending into subdirec-
tories, the build system now always calls make
multi-used-m contains all multi-part mod- with the same Makefile, scripts/Makefile.build,
ules we want to be built in the current directory which then again includes the local subdi-
and multi-objs-m contains all of the indi- rectory Makefile, so the statement to include
vidual objects those are built of. This makes Rules.make could be dropped.
Linux Symposium 194

Furthermore, in 2.5. the build is still organized 4.5 Compiling built-in objects and modules in
in a recursive way, i.e. make is only invoked a single pass, recognizing changed com-
to build the objects in the local subdirectory mand line arguments
and other instances of make are spawned for
the underlying directories. However, it does The major performance issue for the kernel
not actually descend into the subdirectories, build are the invocations of make (most of the
it always does its work from the top-level time is of course normally spent compiling /
directory and prepends the path as necessary. linking, but this cost is independent of the build
One of the advantages is that the output system used). make has to read the local Make-
includes the correct paths, so a compiler file, the general rules and all the dependencies
warning will not show “inode.c: Warning and figure out the work to be done from there.
...”, but “fs/ext2/inode.c: ...”, which makes it An obvious way to optimize the performance
easier to recognize where the problem occurs. of the build system is thus to avoid unneces-
More importantly, it allows to use relative sary invocations. In 2.4, make needs to do sep-
paths throughout the build, so that paths arate passes for modules and built-in objects
like “BUG in /home/kai/src/kernel/ and within each directory, it will even call it-
v2.5/linux-2.5.isdn/include/linux/ self again, so an about four-times performance
fs.h” are history. Renaming/moving a kernel increase is possible by just combining those in-
tree will not cause spurious rebuilds due to vocations into a single pass.
changing paths as seen above anymore, and
The primary reason why kbuild-2.4 needs two
tools like “ccache” can work more effectively.
passes for built-in and modules lies in its flags
handling. This means that it tries to check not
4.4 Objects exporting symbols only whether prerequisites have changed (e.g.
the C source for an object), but also if the com-
piler flags have changed.
The old module symbol versioning scheme This objective was achieved by generating a
used with Linux 2.4 needed the Makefiles to .<target>.flags file like the following (simpli-
declare which objects export symbols to mod- fied) for each target built:
ules, which was done by listing them in the
variable export-objs. In 2.5, module ver- ifeq (-D__KERNEL__ -DMODULE
sioning was completely redesigned, removing -DEXPORT_SYMTAB,
the need for this explicit declaration. The $(CFLAGS) -DEXPORT_SYMTAB)))
FILES_FLAGS_UP_TO_DATE += config.o
changes are so complex that they are rewarded
endif
their own section in this paper.

Here we conclude the comparison between a On a rebuild, the Makefile would read all those
2.4 and 2.5 subdirectory Makefile, where we .*.flags fragments and forces all files which are
have shown that all the redundant and de- not listed in FILES_FLAGS_UP_TO_DATE to
ducible information has been removed and the be rebuild.
necessary information is revealed to the build
system in a very compact form. The flaw of this method is that it cannot handle
differing flags for different groups of files, so
Two additional important internal changes, make needs to invoked twice, once for the tar-
which did not affect the subdirectory Makefile gets to be built-in with the normal CFLAGS, and
layout will be described in the following: again for the modular targets with -DMODULE
Linux Symposium 195

added to CFLAGS. In the example above it is easier use in a rule as shown above, is that all
also visible that the handling for -DEXPORT_ the checking is done within the context of the
SYMTAB is broken, this method can not de- actual rule and not in a unrelated place later in
the Makefile. This allows for the use and cor-
tect when a file was added / removed from rect checking of GNU make’s per target vari-
the list of files exporting symbols, since the ables, e.g.
-DEXPORT_SYMTAB was hardcoded on both
sides of the comparison and thus useless—the
modkern_cflags := $(CFLAGS_KERNEL)
only way to fix this within in the old frame- $(real-objs-m) : \
work would have been to invoke make four modkern_cflags := $(CFLAGS_MODULE)
times, for all combinations of built-in/module
and export/no-export.
which sets modkern_cflags to
A more flexible scheme to handle changing $(CFLAGS_KERNEL) by default, but to
command lines within GNU make was created: $(CFLAGS_MODULE) for objects listed in
$(real-objs-m), i.e. for objects compiled
As an example, we present the rule which is as modules. The compilation rule can then
responsible for linking built-in objects into a just use $(modkern_cflags) to get the
library in Figure 7. The actual use is pretty right flags for the current object, where the
simple, instead of writing the command di- mechanism described above will take care of
rectly into the command part of the rule, it is recognizing changes and acting accordingly.
instead assigned to the variable cmd_link_
l_target and the build system takes care of 4.6 Dependencies
executing the command as necessary, keeping
track of changes to the command line itself.
Between configuration and building of a ker-
The implementation works as follows: Af- nel, the old kernel build needed the user to
ter executing the command, the macro if_ run “make dep”, which served to purposes:
changed, records the command line into the It generated dependency information for the C
file .<target>.cmd. As make is invoked again source files on headers and other included files,
during a rebuild, it will include those .*.cmd and it generated the version checksums for ex-
files. As it tries to decide whether to rebuild ported symbols.
L_TARGET, it will find FORCE in the prerequi-
sites, which actually forces it to always rerun Both of these task have become unnecessary in
the command part of the rule. 2.5, so the reliance on the user to rerun “make
dep” as needed is gone (additionally, the sys-
However, the command part of the rule now tem in 2.4 is broken that in some modversions
does the actual work: It checks whether any cases it’s not even sufficient to rerun “make
of the prerequisites changed, i.e. $? is non- dep”, the only solution then is to do “make
empty or if the command line changed, which distclean” and start over).
is achieved by the two filter-out state-
2.4 used a small tool called mkdep to generate
ments. Only if either of those two conditions dependencies for C sources. This tools basi-
is met, if_changed expands to a command cally extracted the names of the included files
rebuilding the target, otherwise it is empty and out of the source, but did not actually recur-
the target will not be rebuilt. sively scan those includes then. So, if foo.c in-
cludes foo.h, which itself includes bar.h, mkdep
The advantage of this method, apart from the would only pick up the dependency of foo.c on
foo.h, but foo.c also needs recompiling when
Linux Symposium 196

cmd_link_l_target = rm -f $@; $(AR) $(EXTRA_ARFLAGS) rcs $@ $(obj-y)

$(L_TARGET): $(obj-y) FORCE

$(call if_changed,link_l_target)

targets += $(L_TARGET)

[...]

if_changed = $(if $(strip $? \

$(filter-out $(cmd_$(1)),$(cmd_$@))\
$(filter-out $(cmd_$@),$(cmd_$(1)))),\
@set -e; \
$(cmd_$(1)); \
echo ’cmd_$@ := $(cmd_$(1))’ > $(@D)/.$(@F).cmd)

Figure 7: Checking for a changed command line

foo.h changes. This problem was solved in to be created (for all files, even for en-
2.4 by assuming that foo.h would reside in in- tire subsystem which may not be selected
clude/* (which is mostly, but not always, true). in the configuration at all), even though
For those files it would generate another set of
dependencies, basically: this information is totally useless for a
first build—it’s only useful for deciding
whether a file needs to be rebuilt.
foo.h: bar.h
@touch $@

The build system in Linux 2.5 instead uses

So as bar.h changes, this rule will update the gcc’s -MD flag to generate the dependency in-
timestamp on foo.h, which will then be seen by formation during the build. This flag generates
the rule for foo.c and cause foo.c to be rebuild. the full list of all files included during the com-
pile, so in the example above it would generate
This method has several disadvantages: “foo.o: foo.c foo.h bar.h” (and “baz.h” as that
gets added). This procedure is much simpler,
• Changing the timestamp on files which and it gets around all the disadvantages listed
have not actually been modified confuses above.
a number of source management systems.
The only quirk which is applied similarly in 2.4
• It only works for header files in the in- and 2.5 is related to the high configurability of
clude/* subdirectories. the linux kernel.

• As foo.h is changed to also include baz.h, Using the gcc generated list of dependencies
the dependency information does not get as-is has the drawback that virtually every file
updated, so a subsequent change to baz.h in the kernel includes <linux/config.h> which
will erroneously not cause foo.c to be re- then again includes <linux/autoconf.h>
compiled.
If a user reruns make *config to change a
• Starting from a clean tree, the user has configuration option, linux/autoconf.h will be
to wait for the dependency information regenerated. make will notice this and rebuild
Linux Symposium 197

every file which includes autconf.h, i.e. basi- in-kernel loader in 2.5. A common problem
cally all files. This is correct, but extremely is that Linux does not guarantee a stable
annoying if the user just changed some option binary interface to modules, in fact the binary
CONFIG_THIS_DRIVER from n to m. interface often changes between releases in
a stable kernel series and even depending on
So we use the same trick that “mkdep” ap- the configuration of the kernel. One simple
plied before. We replace the dependency on example is the struct net_device,
linux/autoconf.h by a dependency on every which embeds a spinlock_t. If the kernel
config option which is mentioned in any of the is configured for uni-processor operation, this
listed prerexquisites. lock expands to nothing, so the layout of the
The effect is that if a user changes the struct net_device changes. When
CONFIG_THIS_DRIVER option, only the ob-
calling register_netdev(struct
jects which (themselves, or in any of net_device *) where the in-kernel func-
the included files) reference CONFIG_THIS_ tion register_netdev() assumes the
DRIVER will be rebuilt, which most likely is
SMP layout, though the module set up the
only this one driver. argument in the UP layout, we have an obvious
mismatch which often leads to hard to explain
kernel crashes.
5 Modules and the kernel build
Other operating systems solve this problem by
process prescribing a stable ABI between kernel and
modules, however in Linux it is preferred to not
The implementation of loadable kernel mod- carry around binary compatibility layers and
ules has been substantially rewritten by Rusty cope with unflexible interfaces, instead since
Russell in the development cycle 2.5. These the source is openly accessible, one just needs
changes are so complex that this paper will not to recompile the modules so that they match
attempt to describe them in detail. Instead, we the kernel.
concentrate on the changes which were done in
Now, it is easily possible for users to get this
the build system to accomodate the new con-
wrong and we thus want a way to detect ver-
cepts.
sion mismatches and refuse to load the mod-
ules or at least warn. This is what “module
5.1 Module symbol versions
symbol versioning” accomplishes. The basic
idea is to analyze the exported symbols, includ-
Loadable modules need to interface with the ing the types of the arguments for function calls
kernel. They do this by accessing certain and generate a checksum for a specific layout.
data structures and functions which have been If anything changes in the ABI, the versioning
marked as exported symbols in the source. process will generate a different checksum and
That means not all global symbols in the kernel thus detect the mismatch. The main work in
are accessible to modules, but only an explic- this scheme is done by the program genksyms,
itly exported API. which is basically a C parser that reads a pre-
processed source file and finds the definitions
These symbols remain unresolved in the for the exported symbols from there.
loadable module objects at build time and
are then resolved at load time, either by an This procedure has caused trouble in the
external program, modutils, in 2.4, or by an
Linux Symposium 198

build system for a long time. In Linux 2.4, but it has no way to make sure that it is up-
the “make dep” stage, apart from build- to-date since it is located in a different subdi-
ing dependency information, preprocesses rectory.
all source files which export symbols (that
is why they need to be specifically de- Module versioning is instead implemented as
clared in the Makefiles) and then gener- a two stage process, the first stage is the nor-
ates include/linux/modversions.h which man- mal build, which also generates all the check-
gles the exported symbols with the gen- sums. After this stage is completed, we can
erated checksum, using the C preproces- be sure that all checksums are up-to-date now,
sor. The kernel will then not export the and then just record this up-to-date information
symbol register_netdev, but instead into the modules. This is one of the reasons
register_netdev_R43d2381. A mod- why modules have been renamed with a “.ko”
ule referencing register_netdev will end extension: The first stage just builds the nor-
up with an unresolved symbol register_ mal “.o” objects, and afterwards a postprocess-
netdev_R43d2381, so loading it into the ing step follows, which builds “.ko” modules
kernel will work fine. Has the module how- adding version checksums for unresolved sym-
ever built against a different kernel or a differ- bols and other information.
ent configuration, the checksum has changed In more detail, the following steps are exe-
and any attempt to load it will result in an error cuted:
about unresolved symbols.

This implementation was rather fragile, as • Compiling

it relies on the user to rerun “make dep” Knowledge of which source files export
whenever the version information has possibly symbols is not required up front. As
changed, and even if only one symbol changed, an EXPORT_SYMBOL(foo) is encoun-
that basically forces a recompilation of every tered, the definition of EXPORT_SYMBOL
file. In addition, some of the optimizations from include/linux/module.h will generate
made in 2.4’s build system were actually bro- special sections with tables containg the
ken, leading to the well-known fact that it can name of the symbol, its address and its
get into a state where not even running “make checksum. Actually, since the checksum
dep” will recover from generating inconsis- is not known at this time, the value of
tent version information, and starting over the checksum is set to a symbol called
from “make mrproper/distclean” is __crc_foo. This is a trick which allows
needed. to use the linker to record the checksum
even after the object file is already com-
Module versioning is still a challenge to the piled.
build system in 2.5, the underlying reason
for that is that it introduces cross-directory As the object file has been generated, we
dependencies, which a recursive build sys- check it for the existance of the special
tem cannot easily handle. For example, section mentioned above. If it exists,
the ISDN module drivers/isdn/hisax/hisax.ko the source file did export symbols and
uses register_isdn(), which is exported genksyms is run to obtain the checksums
by drivers/isdn/isdn_common.o. So building for those symbols. Finally, these check-
hisax.ko needs knowledge of the checksum sums are entered into the object using the
generated from drivers/isdn/isdn_common.o, linker in conjunction with a small linker
script.
Linux Symposium 199

$ nm drivers/isdn/i4l/isdn.ko | grep __crc

86849dd0 A __crc_isdn_ppp_register_compressor
843d2381 A __crc_isdn_ppp_unregister_compressor
66d136e2 A __crc_register_isdn

Figure 8: Examining the checksums for exported symbols

The checksums can easily examined at view, is that the actual symbols are not man-
running the command shown in Figure 8. gled, so it became possible to force a module
load even if the checksums do not match—
• Postprocessing though the kernel will set the taint flag in these
After stage one, we have the check- case.
sums for the exported symbols embedded
The module postprocessing step, introduced
within vmlinux and the modules. What
mainly for the module symbol versioning, al-
is yet to be done is recording the check-
lowed for a number of additional features, i.e.
sums into the consumers, that is adding
module aliases / device table handling, addi-
the checksums for unresolved symbols
tional version checks as well as recognition of
into the modules.
unresolved symbols during the build stage.
This step was initially handled by a small
shell script but is now done by a C pro-
gram for performance reasons, which also
handles other postprocessing needs like
generating aliases. 6 Conclusion and Outlook
This program basically reads all the ex-
ported symbols and their checksums from
all modules, and then scans the modules
for unresolved symbols. For each unre- This paper presented an introduction to using
solved symbol, an entry in a table associ- the kernel build system for the Linux kernel
ating the symbol string with the checksum 2.5 and 2.6 for users who want to compile their
is made, this table is output as C source own kernels and developers working on ker-
module.mod.c and compiled and linked nel code. We also showed how in the transi-
into the final .ko module object. tion from kbuild-2.4 to 2.5, features of GNU
make could be applied to remove redundant in-
Figure 9 shows an excerpt from formation and allow for simpler Makefile frag-
drivers/isdn/hisax/hisax.mod.c which ments as well as a more consistent and fool-
calls register_isdn(). The proof build system.
checksum obviously matches the
checksum for the exported symbol Additionally, parts of the internal implementa-
in drivers/isdn/i4l/isdn.ko, so that the tion have been described and an overview over
module will load without complaint. changes related to the new module loader and
new module versioning system has been given.

An additional advantage of the new way of The kernel build system in 2.5 has been im-
handling module version symbols, apart from proved significantly, but some features remain
being cleaner from a build system point of to be implemented.
Linux Symposium 200

static const struct modversion_info ____versions[]

__attribute__((section("__versions"))) = {
{ 0xfa7bbba7, "struct_module" },
{ 0x66d136e2, "register_isdn" },
{ 0x1a1a4f09, "__request_region" },
[...]

Figure 9: Excerpt from drivers/isdn/hisax/hisax.mod.c, generated by the postprocessing stage

Separate source and object directories module post-processing. On the other hand, a
global Makefile which contains also needs to
As opposed to kernel 2.4, source files are not incorporate dependencies for all files will use
altered or touched during the build in 2.5 any- a significant amount of memory and may turn
more, enhancing interoperability with source out to be problematic on low–end systems.
management systems. The next step is to al-
There are two ways to implement a global
low for completely separate source and ob-
Makefile: One possibility is using GNU make
ject directory trees, so that the source can be
itself, replacing the rules to actually compile /
completely read-only and multiple builds at the
link objects by dummy routines recording the
same time from the same source are possible.
necessary actions into a global Makefile. The
The current code in 2.5 has taken preparatory
second possibility is, as the subdir Makefiles
steps for this feature but work is not completed
have a very consistent form by now, to write a
yet.
specialized parser for those files and have that
generate a global Makefile.
Non-recursive build
Whether switching to a non-recursive build
It is an open question whether it is actually ad- system is worth the tradeoffs will be investi-
visable to switch to a non-recursive build sys- gated in the Linux 2.7 development cycle.
tem. Obviously, distributing build information
with the source files is desirable, this trend is References
visible in e.g. the split of the global Con-
figure.help file into per-directory fragments
[1] GNU make http://www.gnu.org/
which eventually were unified with the new
software/make/make.html
Kconfig configuration info. Of course it is es-
sential to keep the build information in the per- [2] Michael Elizabeth Castain:
subdirectory Makefiles distributed as it is cur- dancing-makefiles
rently, it would be a step back to collapse it into http://www.kernel.org/pub/
one big file. linux/kernel/projects/
kbuild/dancing-makefiles-2.
However this does not preclude collecting the
4.0-test10.gz
distributed information when starting a build
and generating a global Makefile, which is then
used as a main stage. The advantage of this
method is that it can handle cross-directory de-
pendencies more easily, whereas the current
system has to resort to a two-stage process for
Proceedings of the
Linux Symposium

July 23th–26th, 2003

Ottawa, Ontario
Canada
Conference Organizers
Andrew J. Hutton, Steamballoon, Inc.
Stephanie Donovan, Linux Symposium
C. Craig Ross, Linux Symposium

Review Committee
Alan Cox, Red Hat, Inc.
Andi Kleen, SuSE, GmbH
Matthew Wilcox, Hewlett-Packard
Gerrit Huizenga, IBM
Andrew J. Hutton, Steamballoon, Inc.
C. Craig Ross, Linux Symposium
Martin K. Petersen, Wild Open Source, Inc.

Proceedings Formatting Team

John W. Lockhart, Red Hat, Inc.

Authors retain copyright to all submitted papers, but have granted unlimited redistribution rights
to all as a condition of submission.