Article

mTCP: a highly scalable user-level TCP stack for multicore systems

Authors:

Eun Young Jeong,

Muhammad Jamshed,

KyoungSoo ParkAuthors Info & Claims

NSDI'14: Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

Pages 489 - 502

Published: 02 April 2014 Publication History

Abstract

Scaling the performance of short TCP connections on multicore systems is fundamentally challenging. Although many proposals have attempted to address various shortcomings, inefficiency of the kernel implementation still persists. For example, even state-of-the-art designs spend 70% to 80% of CPU cycles in handling TCP connections in the kernel, leaving only small room for innovation in the user-level program.

This work presents mTCP, a high-performance user-level TCP stack for multicore systems. mTCP addresses the inefficiencies from the ground up--from packet I/O and TCP connection management to the application interface. In addition to adopting well-known techniques, our design (1) translates multiple expensive system calls into a single shared memory reference, (2) allows efficient flow-level event aggregation, and (3) performs batched packet I/O for high I/O efficiency. Our evaluations on an 8-core machine showed that mTCP improves the performance of small message transactions by a factor of 25 compared to the latest Linux TCP stack and a factor of 3 compared to the best-performing research system known so far. It also improves the performance of various popular applications by 33% to 320% compared to those on the Linux stack.

References

[1]

Facebook. https://www.facebook.com/.

[2]

Google. https://www.google.com/.

[3]

How long does it take to make a context switch? http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html.

[4]

Intel DPDK: Data Plane Development Kit. http://dpdk.org/.

[5]

Intel VMDq Technology. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/vmdq-technology-paper.pdf.

[6]

Libevent. http://libevent.org/.

[7]

Libzero for DNA: Zero-copy flexible packet processing on top of DNA. http://www.ntop.org/products/pf_ring/libzero-for-dna/.

[8]

Lighttpd. http://www.lighttpd.net/.

[9]

memcached - a distributed memory object caching system. http://memcached.org.

[10]

The NewReno modification to TCP's fast recovery algorithm. http://www.ietf.org/rfc/rfc2582.txt.

[11]

The open group base specifications issue 6, IEEE Std 1003.1. http://pubs.opengroup.org/onlinepubs/007904975/basedefs/pthread.h. html.

[12]

Packet I/O Engine. http://shader.kaist.edu/packetshader/io_engine/.

[13]

PCI-SIG SR-IOV Primer: An Introduction to SR-IOV Technology. http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf.

[14]

The SO_REUSEPORT socket option. https://lwn. net/Articles/542629/.

[15]

The Apache HTTP Server Project. http://httpd. apache.org/.

[16]

The Apache Portable Runtime Project. http://apr. apache.org/.

[17]

Transmission control protocol. http://www.ietf. org/rfc/rfc793.txt.

[18]

Twitter. https://twitter.com/.

[19]

A. Baumann, P. Barham, P.-E. Dagand, T. Harris, R. Isaacs, S. Peter, T. Roscoe, A. Schüpbach, and A. Singhania. The multikernel: a new OS architecture for scalable multicore systems. In Proceedings of the ACM SIGOPS symposium on Operating systems principles (SOSP), 2009.

Digital Library

[20]

S. Boyd-Wickizer, H. Chen, R. Chen, Y. Mao, F. Kaashoek, R. Morris, A. Pesterev, L. Stein, M. Wu, Y. Dai, Y. Zhang, and Z. Zhang. Corey: an operating system for many cores. In Proceedings of the USENIX conference on Operating systems design and implementation (OSDI), 2008.

Digital Library

[21]

M. Dobrescu, N. Egi, K. Argyraki, B.-G. Chun, K. Fall, G. Iannaccone, A. Knies, M. Manesh, and S. Ratnasamy. RouteBricks: exploiting parallelism to scale software routers. In Proceedings of the ACM SIGOPS Symposium on Operating Systems Principles (SOSP), 2009.

Digital Library

[22]

D. Ely, S. Savage, and D. Wetherall. Alpine: a user-level infrastructure for network protocol development. In Proceedings of the conference on USENIX Symposium on Internet Technologies and Systems (USIT), 2001.

Digital Library

[23]

D. R. Engler, M. F. Kaashoek, and J. O'Toole, Jr. Exokernel: an operating system architecture for application-level resource management. In Proceedings of the ACM symposium on Operating systems principles (SOSP), 1995.

Digital Library

[24]

G. R. Ganger, D. R. Engler, M. F. Kaashoek, H. M. Briceño, R. Hunt, and T. Pinckney. Fast and flexible application-level networking on exokernel systems. ACM Transactions on Computer Systems (TOCS), 20(1):49-83, Feb. 2002.

Digital Library

[25]

H. S. Gunawi, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Deploying safe user-level network services with ictcp. In Proceedings of the conference on Symposium on Opearting Systems Design & Implementation (OSDI), 2004.

Digital Library

[26]

S. Ha, I. Rhee, and L. Xu. CUBIC: a new TCP-friendly high-speed TCP variant. SIGOPS Oper. Syst. Rev., 42(5):64-74, July 2008.

Digital Library

[27]

S. Han, K. Jang, K. Park, and S. B. Moon. PacketShader: a GPU-accelerated software router. In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM), 2010.

Digital Library

[28]

S. Han, S. Marshall, B.-G. Chun, and S. Ratnasamy. MegaPipe: a new programming interface for scalable network I/O. In Proceedings of the USENIX conference on Operating Systems Design and Implementation (OSDI), 2012.

Digital Library

[29]

H. Härtig, M. Hohmuth, J. Liedtke, S. Schönberg, and J. Wolter. The performance of µ-kernel-based systems. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), 1997.

Digital Library

[30]

H. Härtig, M. Hohmuth, J. Liedtke, J. Wolter, and S. Schönberg. The performance of µ-kernel-based systems. SIGOPS Oper. Syst. Rev., 31(5):66-77, Oct. 1997.

Digital Library

[31]

S. Ihm and V. S. Pai. Towards understanding modern web traffic. In Proceedings of the ACM SIGCOMM conference on Internet measurement conference (IMC), 2011.

Digital Library

[32]

K. Jang, S. Han, S. Han, S. Moon, and K. Park. SSLShader: cheap SSL acceleration with commodity processors. In Proceedings of the USENIX conference on Networked systems design and implementation (NSDI), 2011.

Digital Library

[33]

R. Kapoor, G. Porter, M. Tewari, G. M. Voelker, and A. Vahdat. Chronos: Predictable Low Latency for Data Center Applications. In Proceedings of the Third ACM Symposium on Cloud Computing (SOCC), 2012.

Digital Library

[34]

H. Lim, D. Han, D. G. Andersen, and M. Kaminsky. MICA: A Holistic Approach to Fast In-Memory Key-Value Storage. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2014.

Digital Library

[35]

C. Mitchell, Y. Geng, and J. Li. Using one-sided RDMA reads to build a fast, CPU-efficient key-value store. In Proceedings of the USENIX Annual Technical Conference (ATC), 2013.

Digital Library

[36]

R. Nishtala, H. Fugal, S. Grimm, M. Kwiatkowski, H. Lee, H. C. Li, R. McElroy, M. Paleczny, D. Peek, P. Saab, et al. Scaling memcache at Facebook. In Proceedings of the USENIX conference on Networked Systems Design and Implementation (NSDI), 2013.

Digital Library

[37]

A. Pesterev, J. Strauss, N. Zeldovich, and R. T. Morris. Improving network connection locality on multicore systems. In Proceedings of the ACM european conference on Computer Systems (EuroSys), 2012.

Digital Library

[38]

R. Rashid, D. Julin, D. Orr, R. Sanzi, R. Baron, A. Forin, D. Golub, and M. Jones. Mach: A system software kernel. In Proceedings of the Computer Society International Conference (COMPCON), 1989.

[39]

L. Rizzo. netmap: a novel framework for fast packet I/O. In Proceedings of the USENIX conference on Annual Technical Conference (ATC), 2012.

Digital Library

[40]

L. Soares and M. Stumm. FlexSC: flexible system call scheduling with exception-less system calls. In Proceedings of the USENIX conference on Operating systems design and implementation (OSDI), 2010.

Digital Library

[41]

L. Soares and M. Stumm. Exception-less system calls for event-driven servers. In Proceedings of the USENIX conference on USENIX annual technical conference (ATC), 2011.

Digital Library

[42]

C. A. Thekkath, T. D. Nguyen, E. Moy, and E. D. Lazowska. Implementing network protocols at user level. IEEE/ACM Transactions on Networking (TON), 1(5):554-565, Oct. 1993.

Digital Library

[43]

V. Vasudevan, D. G. Andersen, and M. Kaminsky. The case for VOS: the vector operating system. In Proceedings of the USENIX conference on Hot topics in operating systems (HotOS), 2011.

Digital Library

[44]

D. Wentzlaff and A. Agarwal. Factored operating systems (fos): the case for a scalable operating system for multicores. ACM SIGOPS Operating Systems Review, 43(2):76-85, Apr. 2009.

Digital Library

[45]

S. Woo, E. Jeong, S. Park, J. Lee, S. Ihm, and K. Park. Comparison of caching strategies in modern cellular backhaul networks. In Proceeding of the annual international conference on Mobile systems, applications, and services (MobiSys), 2013.

Digital Library

Cited By

Skiadopoulos AXie ZZhao MCai QAgarwal SAdelmann JAhern DContavalli CGoldflam MMayatskikh VRaja RWalton DAgarwal RMukherjee SKozyrakis CGavrilovska ATerry D(2024)High-throughput and flexible host networking for accelerated computingProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691960(405-423)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691960
Parola FQi SNarappa ARamakrishnan KRisso F(2024)SURE: Secure Unikernels Make Serverless Computing Rapid and EfficientProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698558(668-688)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698558
Jia YTian KYou YChen YChen KWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Skyloft: A General High-Efficient Scheduling Framework in User SpaceProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695973(265-279)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695973
Show More Cited By

mTCP: a highly scalable user-level TCP stack for multicore systems
1. General and reference
  1. Cross-computing tools and techniques
2. Social and professional topics
  1. Professional topics
    1. Computing profession

Recommendations

MTCP: scalable TCP-like congestion control for reliable multicast

We present MTCP, a congestion control scheme for large-scale reliable multicast. Congestion control for reliable multicast is important, because of its wide applications in multimedia and collaborative computing, yet non-trivial, because of the ...
MTCP: Scalable TCP-like Congestion Control for Reliable Multicast
Evaluating TCP-friendliness in light of Concurrent Multipath Transfer

In prior work, a CMT protocol using SCTP multihoming (termed SCTP-based CMT) was proposed and investigated for improving application throughput. SCTP-based CMT was studied in (bottleneck-independent) wired networking scenarios with ns-2 simulations. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

NSDI'14: Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation

April 2014

546 pages

ISBN:9781931971096

Program Chairs:
Ratul Mahajan
Microsoft Research
,
Ion Stoica
University of California at Berkeley

Sponsors

USENIX Assoc: USENIX Assoc

In-Cooperation

Publisher

USENIX Association

United States

Publication History

Published: 02 April 2014

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

122
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Skiadopoulos AXie ZZhao MCai QAgarwal SAdelmann JAhern DContavalli CGoldflam MMayatskikh VRaja RWalton DAgarwal RMukherjee SKozyrakis CGavrilovska ATerry D(2024)High-throughput and flexible host networking for accelerated computingProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691960(405-423)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691960
Parola FQi SNarappa ARamakrishnan KRisso F(2024)SURE: Secure Unikernels Make Serverless Computing Rapid and EfficientProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698558(668-688)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698558
Jia YTian KYou YChen YChen KWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Skyloft: A General High-Efficient Scheduling Framework in User SpaceProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695973(265-279)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695973
Dong HArora SKrieger OAppavoo J(2024)Can OS Specialization give new life to old carbon in the cloud?Proceedings of the 17th ACM International Systems and Storage Conference10.1145/3688351.3689158(83-90)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3688351.3689158
Matsumoto NSuda A(2024)bypass4netns: Accelerating TCP/IP Communications in Rootless ContainersProceedings of the Asian Internet Engineering Conference 202410.1145/3674213.3674221(67-75)Online publication date: 9-Aug-2024
https://dl.acm.org/doi/10.1145/3674213.3674221
Brahmakshatriya ARinard CGhobadi MAmarasinghe S(2024)NetBlocks: Staging Layouts for High-Performance Custom Host Network StacksProceedings of the ACM on Programming Languages10.1145/36563968:PLDI(467-491)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656396
Schuh HKrishnamurthy ACuller DLevy HRizzo LKhan SStephens BTsafrir DMUSUVATHI MGupta RAbu-Ghazaleh N(2024)CC-NIC: a Cache-Coherent Interface to the NICProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3617232.3624868(52-68)Online publication date: 27-Apr-2024
https://dl.acm.org/doi/10.1145/3617232.3624868
Cai PKarsten M(2023)Kernel vs. User-Level Networking: Don't Throw Out the Stack with the InterruptsProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36267807:3(1-25)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3626780
Parola FProcopio RQuerio RRisso F(2023)Comparing User Space and In-Kernel Packet Processing for Edge Data CentersACM SIGCOMM Computer Communication Review10.1145/3594255.359425753:1(14-29)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.1145/3594255.3594257
Rosa LGarbugli ACorradi ABellavista P(2023)INSANEProceedings of the 24th International Middleware Conference10.1145/3590140.3629105(57-70)Online publication date: 27-Nov-2023
https://dl.acm.org/doi/10.1145/3590140.3629105
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten