End-System Optimizations for High-Speed TCP

Jeffrey S. Chase, Andrew J. Gallatin, and Kenneth G. Yocum

Department of Computer Science
Duke University
Durham, NC 27708-0129
fchase, grantg@cs.duke.edu, gallatin@freebsd.org

Abstract a step function, but the fastest networks tend to stay

close to the limits of the hosts. Given a sufficiently fast
Delivered TCP performance on high-speed networks is
network, achievable TCP performance depends on opti-
often limited by the sending and receiving hosts, rather
mizations to minimize networking overheads. With Gi-
than by the network hardware or the TCP protocol im-
gabit Ethernet widely deployed and 10 Gb/s Ethernet on
plementation itself. In this case, systems can achieve
the horizon, these optimizations are highly relevant to-
higher bandwidth by reducing host overheads through a
day.
variety of optimizations above and below the TCP proto-
col stack, given support from the network interface. This
This paper focuses on approaches to low-overhead
paper surveys the most important of these optimizations
networking that are now emerging into common prac-
and illustrates their effects quantitatively with empirical
tice. The key techniques discussed are interrupt coalesc-
results from an experimental network delivering up to
ing, checksum offloading, and zero-copy data movement
two gigabits per second of end-to-end TCP bandwidth.
by page remapping. We also explore the effect of larger
packet sizes, e.g., the Jumbo Frames standard for Ether-
1 Introduction net. The network interface plays a key role for each of
these features, and an increasing number of commercial
Modern TCP/IP implementations can transfer data at a network adapters support them. We quantify their per-
high percentage of available network link bandwidth, re- formance potential and briefly outline alternative struc-
flecting the success of many years of refinements. On the tures for high-speed TCP/IP networking that could of-
fastest networks, end-to-end (application-to-application) fer similar benefits in a more general way for the next
throughput is often limited by the capability of the end generation of adapters and systems: user-level network
systems to generate, transmit, receive, and process the interfaces, scatter-gather I/O frameworks, TCP/IP pro-
data at network speeds. Delivered performance is deter- tocol offloading, and Remote Direct Memory Access
mined by a combination of factors relating to the host (RDMA).
hardware, interactions between the host and the network
adapter, and host system software. This paper is organized as follows. Section 2 gives an
This paper explores the end-system factors that can overview of approaches to low-overhead TCP/IP com-
limit bandwidth for TCP on high-speed networks and munication, and their implications for the network inter-
the techniques to overcome those limitations. It is tempt- face and operating system software. Section 3 describes
ing to suppose that the advances in CPU power given by a complete system for high-speed TCP, with more detail
Moore’s Law will render these limitations increasingly on the interactions between the network interface and
irrelevant, but this is not the case. The limiting factor is the operating system for zero-copy data movement and
not CPU processing power but the ability to move data checksum offloading. Section 4 presents empirical re-
through the host I/O system and memory. Wider data- sults from two platforms on an experimental 2.5 Gb/s
paths can improve raw hardware bandwidth, but more network, illustrating the impact of these techniques.
bandwidth is invariably more expensive for a given level The results provide a quantitative snapshot of the cur-
of technology. Advances in network bandwidth follow rent state of the art for end-to-end TCP communication,
 This work is supported by the National Science Foundation (EIA- yielding insights into networking behavior on the next
9870724 and EIA-9972879), Intel Corporation, and Myricom. generation of networks and hosts. Section 5 concludes.
 sender receiver spends most of its time waiting for memory. Given suf-
ficient memory system bandwidth, multiprocessors or
move data from move data from multithreading may improve performance to the extent
application to system buffer to that the software is able to extract parallelism from the
system buffer application network processing; in particular, multiprocessors may
yield higher aggregate throughputs for servers handling
multiple concurrent streams.
TCP/IP protocol TCP/IP protocol To deliver the maximum performance of the hard-
ware, the network interface and OS software must co-
compute checksum compare checksum operate to minimize both classes of overheads at the end
systems. The following subsections give an overview
of the approaches to reducing per-packet and per-byte
network driver network driver overheads respectively, using optimizations on the net-
work interface and in host system software above and
below the TCP/IP protocol stack. This can improve the
transmit packet to deposit packet in peak bandwidth of individual TCP streams as well as the
network interface host memory aggregate throughput across multiple streams.

2.1 Reducing Per-Packet Overheads

Figure 1: Sources of end-system overhead for TCP/IP. Extended frames. One way to limit per-packet over-
heads is to use larger packets, reducing the number of
packets needed to carry a given amount of data. For ex-
2 End-System Optimizations ample, standard IEEE 802.3 Ethernet frame sizes impose
a Maximum Transmission Unit (MTU) of 1500 bytes,
This section outlines the factors that affect end-system but many Gigabit Ethernet vendors have followed a pro-
performance for TCP, and gives an overview of the key posal by Alteon Networks to support “Jumbo” frame
optimizations proposed and studied in the literature. sizes up to 9000 bytes. Up-to-date TCP/IP implemen-
Data transmission using TCP involves the host op- tations automatically discover and use the largest MTU
erating system (OS) facilities for memory and process accepted by the complete network path for a given con-
management, as well as the TCP/IP protocol stack and nection. For bulk data transfer over TCP, the MTU is
the network device and its driver. Figure 1 depicts the the typical unit of data exchange between the host and
key sources of overhead in transmitting data over a TCP the adapter. It is this MTU that affects end-system over-
connection in a typical system. End systems incur CPU head, independent of the transfer sizes actually carried
overhead for processing each network packet or frame. on the links; for example, ATM networks may transpar-
These per-packet costs include the overhead to execute ently segment and reassemble large IP packets for trans-
the TCP/IP protocol code, allocate and release memory port as 53-byte cells, but the small cell size does not
buffers, and field device interrupts for packet arrival and impact TCP/IP performance. Section 4 quantifies end-
transmit completion. The systems incur additional costs system performance for MTU sizes up to 32KB.
for each byte of data sent or received. These per-byte Interrupt Coalescing. A second way to limit per-
costs — incurred at the stages shown in grey in Figure 1 packet overhead is to amortize packet handling costs
— include overheads to move data within the end sys- across multiple packets. For example, interrupts to no-
tem and to compute checksums to detect data corruption tify the host of transmit-complete events and packet ar-
in the network. rival can impose a significant overhead. Most high-
At a given packet size, faster networks carry more speed network interfaces support interrupt coalescing or
packets as well as more bytes per unit of time, increas- interrupt suppression, selectively delaying interrupts if
ing both per-packet and per-byte overheads on the host. more packets are pending transmission or delivery to the
At high speeds these costs may consume a significant host [6]. This reduces the total number of interrupts de-
share of host CPU cycles and memory system band- livered; the host interrupt handler may process multiple
width, siphoning off resources needed for application event notifications from each interrupt. While some in-
processing of the data. The end system ultimately satu- terrupt coalescing schemes increase end-to-end latency
rates under the combined load of application processing slightly, they reduce interrupt costs during periods of
and networking overhead, limiting delivered bandwidth. high bandwidth demand, and may deliver better cache
Faster CPUs alone do not help appreciably if the CPU performance for processing the notifications.
2.2 Reducing Per-Byte Overheads floading, as discussed in Section 3.2.
Integrated copy/checksum. Another approach to re-
Per-byte costs dominate per-packet costs for bulk data ducing per-byte costs is to coalesce multiple operations
transfer because they are fundamentally limited by the that touch the data, completing multiple steps in a single
host memory system rather than the CPU. The only un- traversal of the packet [4]. For example, some TCP/IP
avoidable per-byte costs occur in the application and in implementations integrate checksumming with the data
moving data between host memory and the network in- copy to or from application memory. A similar approach
terface. An ideal system would entirely eliminate the could apply to other processing steps such as compres-
remaining per-byte overheads. sion or encryption.
Zero-copy networking. In principle, it is possible
to eliminate all data copying within the host. Oper-
ating systems need not copy data to make it available 2.3 The Role of the Network Interface
to the network interface because advanced network de- Most of the optimizations outlined above require some
vices support scatter/gather DMA, in which the data for support from the network adapter [6]. Advanced net-
a frame may span multiple regions of host memory. The work interfaces can help reduce end-system overheads
OS can avoid other internal data copying with flexible by supporting larger packets at the host interface, seg-
mechanisms for buffer management. menting and reassembling packets as needed to match
More problematic is the copying of data between ap- the link frame sizes, depositing packets in host mem-
plication memory and OS buffers. On the sender, copy- ory intelligently, coalescing interrupts, or incorporat-
ing the data protects it from modification if the process ing checksum hardware. There is a rebirth of interest
reuses the memory before the transmit of the old data in supporting TCP/IP protocol functions directly on the
completes. On the receiver, copying allows the OS to adapter (TCP offloading), with several products on the
place the incoming data at arbitrary virtual addresses market and under development.
specified by the application. Avoiding this copying in- All techniques to avoid copying of incoming data rely
volves complex relationships among the network hard- on the network interface to process the incoming packet
ware and software, OS buffering schemes, and the ap- stream and deposit the data in suitable host memory
plication programming interface (API). buffers. For page remapping with the Unix socket API,
A basic technique for copy avoidance is to use vir- the adapter must (at minimum) separate incoming packet
tual memory to change buffer access or address bindings headers from their payloads and deposit the payloads
without copying the data to a new buffer. For exam- into buffers aligned to the system virtual memory page
ple, virtual memory allows the system to protect against size. The OS may then deliver the data to an applica-
modifications to buffers with pending transmits, even if tion buffer by updating virtual mappings to reference the
they are accessible in the process address space. Sec- payload’s physical location, if the virtual address of the
tion 3.1 discusses a key technique — page remapping application buffer is also suitably aligned. If the MTU
or page flipping — in detail, and outlines an implemen- is smaller than a page, the adapter would need to recog-
tation that is compatible with a standard socket API for nize the TCP connection for each incoming packet and
networking. Research systems have used page remap- “pack” payloads into page-size buffers for each connec-
ping for several years [1, 7], and it is now emerging into tion.
more common use. IO-Lite [9] generalizes this tech- Several recent proposals promise more general sup-
nique by replacing sockets with a new scatter/gather I/O port for low-overhead networking by a comprehensive
API in which the OS rather than the application selects restructuring of the end systems. IO-Lite’s redesigned
the addresses for incoming data. API allows more flexible data placement and reduces
Checksum offloading. To detect data corruption in copying for inter-process communication and storage
the network, TCP/IP generates an end-to-end checksum access as well [9]. Another approach termed Remote
for each packet covering all protocol headers and data. Direct Memory Access (RDMA) inserts directives into
In most implementations, software running on the host the packet stream to name the buffer destinations for in-
CPU computes these checksums on both sending and re- coming data; the receiving adapter recognizes these di-
ceiving sides. This requires the CPU to load all of the rectives and “steers” the data directly into its buffers.
data through the memory hierarchy for a sequence of RDMA is similar to sender-based memory manage-
add operations. One way to eliminate this cost is to com- ment [2]: it handles arbitrary MTUs and buffer align-
pute the checksum in hardware on the network adapter’s ments, and supports protocols layered above TCP, e.g.,
DMA interface as the packet passes through on its way for network storage access. RDMA features are already
to or from host memory [8]. The TCP/IP stack and the present in the Virtual Interface Architecture (VI); VI
network device driver must cooperate in checksum of- supports user-level networking [10] in which applica-
tions interact directly with the network adapter, bypass- a physical page frame. Packet data is stored in linked
ing the kernel buffering system entirely. Adapters sup- chains of mbufs passed between levels of the system;
porting the VI standard over TCP/IP networks have been the TCP/IP protocol stack adds and removes headers and
announced. checksums by manipulating the buffers in the chain. On
a normal transmission, the socket layer copies data from
user memory into a chain, which is passed through the
3 A System for High-Speed TCP
TCP/IP stack to the network driver. On the receiving
This section outlines a complete experimental system for side, the driver constructs a chain containing each in-
high-speed TCP, in order to illustrate the OS issues and coming packet header and payload, and passes the chain
host/interface interactions for zero-copy networking and through the TCP/IP stack to the socket layer. When the
checksum offloading. receiving process accepts the data, e.g., with a read sys-
The experimental system is based on FreeBSD 4.0, tem call, a socket-layer routine copies the payload into
a free Unix system descending from the Berkeley 4.4 user memory and frees the chain.
BSD code base. We chose FreeBSD in part because it Page remapping is implemented in a variant of the
has a high-quality network subsystem incorporating all kernel routine uiomove, which directs the movement of
relevant TCP/IP refinements. We modified FreeBSD to data to and from the process virtual memory for all vari-
support zero-copy sockets and checksum offloading, and ants of the I/O read and write system calls. The new
added code for integrated copy/checksum from a recent code activates when a process requests the kernel to
Linux release. transfer a page or more of data between a network socket
The network interface used in our experiments is and a page-aligned user buffer. Instead of copying data
Trapeze/Myrinet [3]. Myrinet (http://www.myri.com) is to or from the mbuf chain, uiomove passes the data by
a reliable short-haul interconnect with very low latency, reference.
configurable maximum frame sizes, and link speeds up On the sender, uiomove creates a new external mbuf
to 2.5 Gb/s in each direction. Trapeze is a firmware pro- that references the page frame backing the application’s
gram for Myrinet adapters. Trapeze and its associated virtual buffer. The mbuf chain and its pages are then
network driver incorporate support for the optimizations passed through the TCP/IP stack to the network driver,
discussed in Section 2.1 and 2.2. While Trapeze/Myrinet which attaches the pages to outgoing messages as pay-
is not representative of networks encountered by TCP loads. In Unix systems, the page remapping scheme
systems “in the wild,” it is ideal for probing the limits of must preserve the copy semantics of the existing socket
end-system hardware and software for high-speed net- interface [1, 7]. The application may overwrite its send
working. buffer after it requests the send but before the TCP ac-
The FreeBSD extensions described in this section are knowledgement is received, completing the transmit. To
compatible with network interfaces other than Trapeze. handle this case, the kernel marks pages with pending
In particular, we have tested the system with an Alteon transmit as copy-on-write, disabling write access to the
Gigabit Ethernet network, yielding TCP bandwidths just page. The copy-on-write mapping is released when the
under the maximum link speed. receiver acknowledges the data.
On the receiver, uiomove installs a new virtual trans-
lation for the page frame containing the data, as refer-
3.1 Zero-Copy Sockets enced by the external mbuf passed up through the pro-
The primary support for zero-copy TCP/IP (page remap- tocol stack from the network driver. Copy-on-write is
ping) resides at the socket layer in the OS kernel. The unnecessary because there is no need to retain the kernel
optimizations trigger only if the data transfer sizes re- buffer after the read; uiomove simply releases the mbuf
quested by the application are larger than the page headers, leaving the buffer page mapped into the appli-
size. On the receiver, page remapping requires an cation address space. It also releases any pages left un-
MTU matched to the page size, page-aligned application mapped by the remapping operations, preserving equi-
buffers, and a network interface that deposits packet data librium between the process and the kernel buffer pool.
on a page boundary. In general, this last requirement
means that the network interface must split the packet 3.2 Checksum Offloading
header from its data for common protocols such as TCP.
Page remapping extends the conventional FreeBSD The network interface and its driver act in concert to im-
send/receive path, which is based on variable-sized ker- plement checksum offloading. The Myrinet and Alteon
nel network buffers (mbufs). Standard mbufs contain devices support checksum offloading in the host-PCI
their own buffer space, while an external mbuf holds a DMA engine, which computes the raw 16-bit one’s com-
reference to another kernel buffer, e.g., a file buffer or plement checksum of each DMA transfer as it moves
data to and from host memory. It is not necessary to adapter connected to a 66 MHz 64-bit PCI slot.
change the TCP/IP stack to use this checksum; simply The Myrinet adapter is a pre-production prototype
setting a flag in the mbuf chain bypasses the software based on the LANai-9 chip; it has a 133 MHz pro-
checksum. However, three factors complicate hardware cessor supporting the Myrinet-2000 link speed of
checksumming for IP: 250 MB/s (2 Gb/s). The prototype adapters run in
a test mode with a link speed of 320 MB/s.
 A packet’s data may span multiple host buffers;
the device accesses each buffer with a separate  Compaq Monet. The Compaq XP1000 Profes-
DMA. On the sender, Trapeze and Alteon adapters sional Workstations have a 500 MHz Alpha 21264
combine these partial checksums in firmware using CPU, a 4MB L2 cache, and the Digital 21272
one’s complement addition. “Tsunami” chipset. Each machine has an M2M-
PCI64A-2 Myrinet adapter connected to a 33 MHz
 TCP and UDP actually use two checksums: one 64-bit PCI slot and an M2M-SW16 Myrinet switch.
for the IP header (including fields overlapping with The Myrinet adapter is based on the LANai-7 chip;
the TCP or UDP header) and a second end-to-end it has a 66 MHz processor supporting a link speed
checksum covering the TCP or UDP header and of 160 MB/s (1.28 Gb/s).
packet data. In a conventional system, TCP or
UDP computes its end-to-end checksum before IP All machines run extended FreeBSD 4.0 kernels with
fills in its overlapping IP header fields (e.g., op- Trapeze firmware on the Myrinet adapters, as described
tions) on the sender, and after the IP layer restores in Section 3. The machines are configured with suffi-
these fields on the receiver. Checksum offloading cient buffer memory to maintain the link speed (512 KB
involves computing these checksums below the IP socket buffers). All results are from a unidirectional data
stack; thus the driver or NIC firmware must par- transfer over a single TCP connection. A single connec-
tially dismantle the IP header in order to compute a tion delivers enough bandwidth to saturate the receiver,
correct checksum. so multiple streams cannot deliver better aggregate per-
formance. The application is a modified version of net-
 Since the checksums are stored in the headers at the perf version 2.1pl3, a standard tool for benchmarking
front of each IP packet, a sender must complete the TCP/IP performance. The machines are isolated and
checksum before it can transmit the packet head- otherwise idle.
ers on the link. If the checksums are computed by
the host-NIC DMA engine, then the last byte of the
packet must arrive on the NIC before the firmware
4.1 TCP Bandwidth
can determine the complete checksum. Figure 2 shows TCP bandwidth on the PowerEdge plat-
form as a function of the MTU size. The MTUs al-
The last issue may require a compromise between lowed payloads of 1500 bytes (standard Ethernet), 4KB
host overhead and packet latency, since the adapter must (FDDI), 8KB (Ethernet with Jumbo Frames), and multi-
handle large packets in a store-and-forward fashion. ples of 8KB up to 32 KB. The points are averages from
50 netperf runs that each send data as fast as the system
4 Empirical Results allows for 60 seconds, then compute the average band-
width over the interval. The left-hand graph is from a
This section presents quantitative results showing the standard netperf that does not access its data; the right-
impact of zero-copy networking, checksum offloading, hand graph is from a modified version that accesses all
integrated copy/checksum, and MTU size (extended data using longword stores on the sender and longword
frames) on TCP performance. We focus on point-to- loads on the receiver.
point bandwidth in a small Myrinet LAN, where the For the baseline bandwidth results (labeled no opti-
stresses on the end systems are greatest; of course the mizations) the end systems copy the data in the con-
results apply to WAN networks if they deliver sufficient ventional fashion and compute all checksums in soft-
bandwidth to stress the hosts. We tested two hardware ware. The copy and checksum overhead saturates the
configurations: host CPUs well below the link speed; bandwidth peaks
at 824 Mb/s, and stays below 630 Mb/s when netperf
 Dell PowerEdge. The Dell PowerEdge 4400 accesses its data.
servers use a 733 MHz Intel Xeon CPU (32KB L1 The lines labeled zero-copy show the effect of en-
cache, 256KB L2 cache), and a ServerWorks Ser- abling page remapping. The size and alignment of net-
verSet III LE chipset. Each machine has 2-way perf’s buffers and transfer requests allow the systems to
interleaved RAM and an M2M-PCI64B Myrinet avoid almost all copy operations for MTUs equal to or
 2000
2000
1750

1500
1500

Bandwidth HMbsL
Bandwidth HMbsL

1250

1000 1000

750
zero-copy & checksum offloading
500 zero-copy 500

no optimizations
250
integrated copychecksum
1.5 4 8 16 24 32
MTU HKBL 1.5 4 8 16 24 32
MTU HKBL

Figure 2: TCP Bandwidth on the PowerEdge 4400 platform. The left-hand graph shows bandwidth when the applica-
tion process does not access the data. The right-hand graph shows the impact of this access on delivered bandwidth.

larger than the 4KB system page size. Moving from sum the data rather than touching it) because the sender
a 1500-byte to a 4KB MTU enables page remapping, stores to the data; on this Xeon P6-based platform, the
yielding a sudden jump in bandwidth. This illustrates the CPU can store to memory at only 215 MB/s, but it can
maximum benefit from page remapping or other zero- read at 504 MB/s. Of course these bandwidths are lower
copy networking enhancements on this platform. Band- when the CPU is competing with the I/O system for
width is lower for MTUs that are not a multiple of the memory bandwidth, as it is in this case.
page size because the socket API must copy the “odd” The lines labeled integrated copy/checksum show the
data; Figure 2 omits these points for clarity. effect of combining the conventional copy and checksum
Since checksums are computed in software for the into a single software loop, as implemented in Linux
zero-copy line, host CPU saturation again limits band- 2.3.99-pre8 kernels for this platform. Microbenchmark
width, this time at a higher peak of 1.5 Gb/s. The right- results confirm a bandwidth limitation of roughly 100
hand graph shows that the percentage improvement from MB/s for this loop, about half the 4400’s memory copy
copy avoidance is lower when netperf accesses its data, bandwidth of 207 MB/s. The integrated loop may ac-
yielding a peak of 820 Mb/s. This results from two fac- tually reduce bandwidth with large MTUs on this plat-
tors. First, the additional load drives the CPU to satu- form, but we have not investigated why. What is impor-
ration at a lower bandwidth, thus the CPU is spending tant is that the expected benefits of integrating the copy
less of its time copying data, diminishing the potential and checksum can be elusive unless the loop is carefully
benefit from avoiding the copy. Second, memory caches tuned for specific platforms. In the left-hand graph, the
create a synergy between copying and application pro- integrated copy/checksum appears to offer a benefit at
cessing; each may benefit from data left in cache by the 8KB MTUs, but this effect disappears if the application
other. Despite these factors, avoiding the copy yields touches the data.
a 26% bandwidth improvement at the 8KB MTU. We
omitted the lines for checksum offloading with copy be-
cause the results are similar to zero-copy, although the
4.2 Effect of Larger MTUs
benefit is slightly lower. Moving beyond 4KB along either x-axis in Figure 2
The lines labeled zero-copy & checksum offloading shows the effect of reducing total per-packet overhead
show the combined effect of page remapping and use by increasing the MTU. Larger MTUs mean fewer pack-
of checksum hardware. In the left-hand graph the host ets, and hence fewer interrupts and less protocol process-
CPUs never touch the data. Bandwidth jumps to 2 Gb/s ing overhead. Varying MTU size also reveals the likely
at the 4KB MTU as page remapping replaces copying. benefits of other approaches to reducing per-packet over-
At this point the host CPUs are near saturation from per- heads, e.g., interrupt coalescing or TCP offloading.
packet overheads including buffer management, inter- The data clearly shows a “sweet spot” at the 8KB
rupt handling, and TCP/IP protocol costs. In the right- MTU, which approximates the Jumbo Frames proposal
hand graph netperf touches the data, yielding a much for Ethernet. Standard Ethernet 1500-byte MTUs never
lower peak bandwidth of 1.18 Gb/s for zero-copy & yield a bandwidth above 740 Mb/s in these experiments.
checksum offloading at a 32KB MTU. This is somewhat In addition to reducing per-packet overheads, larger
slower than the left-hand zero-copy points (which check- MTUs enable page remapping optimizations for sockets
on the receiver, potentially reducing per-byte overheads. remapping), network buffer management, the TCP/IP
On this platform, bandwidth improvements from in- protocol stack, and the Trapeze network driver. The
creasing MTU size beyond 8KB are modest. This is due y -axis in Figure 3 gives the CPU utilization attributed
to two factors. For the bottom two lines on both graphs, to each overhead category. The x-axis gives six dif-
performance is dominated by per-byte overheads, mem- ferent configurations to show the effect of varying the
ory system bandwidth, and caching behaviors. In the MTU size and enabling page remapping and checksum
left-hand zero-copy & checksum offloading line, larger offloading. For these experiments TCP bandwidth was
MTUs do not improve bandwidth beyond 2 Gb/s be- held constant at 370 Mb/s by a slow sender, so it is possi-
cause the bottleneck shifts to the network interface CPU; ble to directly compare receiver CPU overheads for dif-
however, host CPU utilization at this speed drops to 45% ferent configurations.
with the 32KB MTU. Larger MTUs do yield improve- With a standard Ethernet MTU the Monet is near 95%
ments for the three remaining zero-copy cases, but this saturation. The CPU spends about 65% of its time pro-
effect diminishes with larger packet sizes as data move- cessing 30,000 packets per second through the network
ment overheads increasingly dominate. The effect of driver and TCP/IP stack. Total packet handling costs
larger MTUs is more apparent from a breakdown of how drop to 22% with an 8KB MTU, and to 10% with a
the CPU spends its time. 32KB MTU. Iprobe attributes approximately 20% of
CPU time to interrupt dispatching at a 1500-byte MTU,
100%
and 5% at an 8KB MTU; in each case it is 20% to 25%
Idle
Copy & Checksum of total overhead on the Alpha, showing that interrupt
Interrupt
VM
coalescing is an important optimization at higher band-
80% TPZ Driver
Buffering
widths, even with Jumbo Frames. It is interesting to
TCP/IP
note that actual TCP/IP protocol overhead accounts for
at most 12% of CPU time even with a 1500-byte MTU.
60% Protocol costs are significant but are not the limiting fac-
CPU Utilization

tor for a well-implemented TCP/IP stack [5].

The last three configurations in Figure 3 show the ef-
40%
fects of checksum offloading and page remapping on
per-byte overheads, with per-packet overheads held low
by the 32KB MTU. Avoiding the copy or offloading the
checksum drops the total CPU time spent on data ac-
20%
cess from 24% to roughly 15%. For page remapping
(zero-copy), this effect is partially offset by per-page
costs to manipulate translations in the virtual memory
0%
Ethernet MTU 8KB MTU 32KB MTU 32KB MTU, 32KB MTU, 32KB MTU, system; this overhead totals about 3% of CPU time and
Checksum Zero-Copy Checksum
Offloading Offloading, is independent of MTU size. Applying both optimiza-
Zero-Copy
tions effectively eliminates all data access costs, and the
Figure 3: TCP Receiver CPU Utilization Breakdown reduced memory contention causes other overheads to
drop slightly; the Monet handles the 370 Mb/s of band-
width with a comfortable 10% CPU utilization.

4.3 CPU Utilization

5 Conclusion
Figure 3 shows a breakdown of receiver CPU utiliza-
tion on the Monet platform in order to better illustrate Technology trends suggest that the fastest networks will
the overheads for high-speed TCP. The data is from continue to carry link bandwidths close to the memory
iprobe suite-4.3 (Instruction Probe), an on-line profiling and I/O subsystem limits on most hosts. Thus the end
tool from Compaq that uses the Alpha on-chip perfor- systems remain the bottlenecks to high-speed TCP/IP
mance counters to report detailed execution profiles with performance in current and future hardware.
low overhead (3%-5%). We do not report bandwidth re- To achieve end-to-end TCP/IP networking perfor-
sults from Monet because its I/O bus is slower than the mance on the fastest networks, it is necessary to min-
Myrinet-2000 link speed, but the effects of networking imize the host overheads to handle packets and their
overhead are qualitatively similar on the two platforms. data. This paper reviews the techniques used by oper-
Figure 3 classifies overheads into five categories: data ating systems and high-speed network interfaces to re-
access for copying and checksums, interrupt handling, duce end-system overheads. The approaches to reduc-
virtual memory costs (buffer page allocation and/or page ing overheads include larger frame sizes (MTUs), inter-
rupt coalescing, copy avoidance by page remapping, in- Kenneth G. Yocum is a doctoral student in Computer
tegrated copy/checksum, and hardware checksum com- Science at Duke University. His research focuses on
putation. These optimizations increase delivered band- high-speed network storage and Internet service virtu-
width by delaying saturation of the host CPUs, while alization. He holds a Bachelor’s degree in Computer
leaving more CPU power for the application. Science from Stanford University. He is the primary de-
The experiments reported in this paper give a quan- veloper of the Trapeze networking system.
titative snapshot of a high-quality TCP/IP stack supple-
mented with these optimizations, running at bandwidths References
up to 2 Gb/s on current hardware. At these speeds,
8KB frame sizes (e.g., Jumbo Frames for Ethernet) yield [1] J. Brustoloni. Interoperation of copy avoidance in
bandwidth improvements up to 70%, relative to standard network and file I/O. In Proceedings of IEEE Info-
1500-byte Ethernet frame sizes. However, there is little com, pages 534–542, 1999.
benefit to increasing frame sizes beyond 16KB. In our
system, per-packet overheads are dominated by operat- [2] Greg Buzzard, David Jacobson, Milon Mackey,
ing system buffer management, which is more expensive Scott Marovich, and John Wilkes. An implemen-
than packet processing in the network driver and TCP/IP tation of the Hamlyn sender-managed interface ar-
protocol stack. Interrupt handling constitutes up to 20% chitecture. In Proceedings of the Second Sympo-
of the per-packet overhead on Alpha systems with an sium on Operating Systems Design and Implemen-
8KB frame size, showing the value of interrupt coalesc- tation. USENIX Association, October 1996.
ing. With 8KB and larger MTUs, per-byte costs domi-
nate per-packet costs on the platforms we tested, show- [3] Jeffrey S. Chase, Darrell C. Anderson, Andrew J.
ing that copy avoidance and hardware checksum offload- Gallatin, Alvin R. Lebeck, and Kenneth G. Yocum.
ing are the most important optimizations. Taken to- Network I/O with Trapeze. In 1999 Hot Intercon-
gether, these yield bandwidth improvements up to 70%, nects Symposium, August 1999.
and can double bandwidth if application processing is [4] David Clark and David Tennenhouse. Architec-
ignored. tural considerations for a new generation of pro-
tocols. In Proceedings of the SIGCOMM Confer-
ence, September 1990.
Acknowledgments
[5] David D. Clark, Van Jacobson, John Romkey, and
The equipment used in this work was provided by the Howard Salwen. An analysis of TCP processing
National Science Foundation, Intel Corporation, and overhead. IEEE Communications, 27(6):23–29,
Myricom Inc. We thank Bob Felderman of Myricom June 1989.
for his frequent technical assistance, and Myricom for
making Myrinet-2000 prototypes available. Darrell An- [6] Peter Druschel, Larry L. Peterson, and Bruce S.
derson and the anonymous IEEE reviewers offered valu- Davie. Experiences with a high-speed network
able comments on the paper. adaptor: A software perspective. In Proceedings
of the ACM SIGCOMM Conference, pages 2–13,
August 1994.
Biographical Summaries
[7] Hsiao-Keng and Jerry Chu. Zero-copy TCP in So-
Jeffrey S. Chase joined the Computer Science faculty at laris. In Proceedings of the USENIX 1996 Annual
Duke University in 1995. He holds a Bachelor’s degree Technical Conference, January 1996.
in Mathematics and Computer Science from Dartmouth
College, and a Ph.D. in Computer Science from the Uni- [8] K. Kleinpaste, P. Steenkiste, and B. Zill. Software
versity of Washington. His research interests are operat- support for outboard buffering and checksumming.
ing systems, distributed systems, networking, and stor- In Proceedings of the ACM SIGCOMM Confer-
age systems. ence, August 1995.
Andrew J. Gallatin is a research staff member in the [9] V. S. Pai, P. Druschel, and W. Zwaenepoel. IO-
Network Storage Lab at Duke University. He holds an Lite: A unified I/O buffering and caching system.
M.S. degree in Computer Science from Rensselaer Poly- ACM Transactions on Computer Systems (TOCS),
technic Institute and a Bachelor’s degree from SUNY 18(1):37–66, February 2000.
Buffalo. He is a frequent contributor to the FreeBSD
Project’s Alpha port, and is working to incorporate the [10] Thorsten von Eicken, Anindya Basu, Vineet Buch,
features described in this article into FreeBSD-5.0 and Werner Vogels. U-Net: A user-level network
 interface for parallel and distributed computing. In
Proceedings of the Fifteenth ACM Symposium on
Operating Systems Principles, 1995.

Keywords: TCP, high-speed network, network

adapter, zero-copy, copy avoidance, checksum, operat-
ing system, memory management, socket.