Article

Free access

Characterization of network processing overheads in Xen

Authors:

Srihari Makineni,

Don NewellAuthors Info & Claims

VTDC '06: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing

Page 2

https://doi.org/10.1109/VTDC.2006.3

Published: 17 November 2006 Publication History

Abstract

I/O virtualization techniques developed recently have led to significant changes in network processing. These techniques require network packets go through additional layers of processing. These additional layers have introduced significant overheads. So it is important to understand performance implications of this additional processing on network processing (TCP/IP). Our goals in this paper are to measure network I/O performance in a Xen virtualized environment and to provide a detailed architectural characterization of network processing highlighting major sources of overheads and their impact. In this paper, we study two modes of I/O virtualizations: 1) running I/O service VM along with the guest on the same CPU, and 2) running I/O service VM on a separate CPU. We measure TCP/IP processing performance in these two modes and compare it to that of on the native Linux machine. Our measurements show that both Rx and Tx performance suffer by more than 50% in virtualized environment. We have noticed that pathlength has increased by 3 to 4 times than that of the native processing. Most of this overhead comes from the Xen VMM layer and Dom0 VM processing. Our data also shows that running the Dom0 VM on a separate CPU is more expensive than running both Dom0 and guest VM on the same CPU. We provide a detailed characterization of this additional processing which we hope will help the Xen community focus on right areas for optimization.

References

[1]

Advanced Micro Devices. AMD64 Virtualization Codenamed "Pacifica" Technology, Secure Virtual Machine Architecture Reference Manual, May 2005.

[2]

A. Foong, J. Fung, and D. Newell, "An In-Depth Analysis of the Impact of Processor Affinity on Network Performance. In Proceeding of IEEE Int'l Conf. Networks, IEEE Press, 2004.

[3]

A. Foong et al., "TCP Performance Analysis Re-visited", IEEE International Symposium on Performance Analysis of Software and Systems, March 2003.

[4]

A. Menon, J. R. Santos: http://xenoprof.sourceforge.net/xenoprof_2.0.txt (Accessed June 2006).

[5]

A. Menon, A. Cox, W. Zwaenepoel, Optimizing Network Virtualization in Xen, 2006 USENIX Annual Technical Conference.

[6]

A. Menon et al. Diagnosing Performance: Overheads in the Xen Virtual Machine Environment. In First ACM/USENIX Conference on Virtual Execution Environments (VEE'05), June 2005.

[7]

C. A. Waldspurger. Memory resource management in VMware ESX server. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI 2002), ACM Operating Systems Review, Winter 2002.

[8]

C. P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M. Rosenblum. Optimizing the Migration of Virtual Computers. In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI 2002), ACM Operating Systems Review, Winter 2002.

[9]

D. Gupta et al. Enforcing Performance Isolation Across Virtual Machines in Xen. HP Labs, Technical Report HPL- 2006-77.

[10]

Intel Virtualization Technology Specification for the IA-32 Intel Architecture, April 2005.

[11]

Iperf benchmark, http://dast.nlanr.net/Projects/Iperf/ (Accessed June 2006).

[12]

J.E. Smith, and R. Nair. Virtual Machines: versatile platforms for systems and processes. Morgan Kaufmann publishers. May 2005.

[13]

L. Cherkasova and R. Gardner. Measuring CPU Overhead for I/O Processing in the Xen Virtual Machine Monitor. In Proceedings of the USENIX Annual Technical Conference, April 2005.

[14]

M. Rosemblum and T. Garfinkel. Virtual Machine Monitors: Current Technology and Future trends. IEEE Computer, 38(5): 39-47, 2005.

[15]

Oprofile. http://oprofile.sourceforge.net (Accessed June 2006).

[16]

P. Barham, B. Dragovic, K. Fraser, et al. "Xen and the Art of Virtualization." SOSP 2003.

[17]

P. Barham et al. Xen and the Art of Virtualization. In proceedings of the ACM symposium on operating systems principles, Oct 2003.

[18]

SWsoft Virtuozzo, "Top Ten Considerations for Choosing a Server Virtualization Technology".

[19]

SWsoft Virtuozzo "An Introduction to OS Server Virtualization and a New Approach to Server Consolidation."

[20]

Virtual Machine Optimized Technologies, AzulSystems. http://www.azulsystems.com/products/cpools_tech.html

[21]

Xen: The Xen Virtual Machine Monitor. http://www.cl.cam.ac.uk/Research/SRG/netos/xen/archit ecture.html

[22]

L. Zhao, S. Makineni, Ramesh Illikkal, D. Newell and L. Bhuyan, "TCP/IP Cache Characterization in Commercial Server Workloads, Seventh Workshop on Computer Architecture Evaluation using Commercial Workloads (CAECW-7), along with HPCA-10, February 2004.

[23]

S. Makineni and R. Iyer, "Architectural Characterization of TCP/IP Packet Processing on the Pentium M microprocessor," Int'l Conf. on High Performance Computer Architecture (HPCA-10), Feb 2004.

[24]

G. Regnier, S. Makineni, R. Illikkal, R. Iyer, et al., "TCP onloading for data center servers," IEEE Computer, vol. 37, no. 11, pp. 48-58, Nov. 2004.

[25]

"Alacritech SLIC: A Data Path TCP Offload methodology", http://www.alacritech.com/html/techreview.html

[26]

R. Huggahalli, R. Iyer and S. Tetrick, Direct Cache Access for High Bandwidth Network I/O," 32nd Annual International Symposium on Computer Architecture (ISCA 2005), June 2005.

[27]

S. Makineni et al., "Receive Side Coalescing for Accelerating TCP/IP Processing", International Conference on High Performance Computing (HiPC), Dec 2006.

Cited By

Zhang WShi YZheng YLiu LCui L(2017)Resource and performance prediction at high utilization for N-Tier cloud-based service systemsProceedings of the Australasian Computer Science Week Multiconference10.1145/3014812.3014857(1-9)Online publication date: 30-Jan-2017
https://dl.acm.org/doi/10.1145/3014812.3014857
Gugnani SLu XPanda DAnjum AZhao X(2016)Performance characterization of hadoop workloads on SR-IOV-enabled virtualized InfiniBand clustersProceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies10.1145/3006299.3006313(36-45)Online publication date: 6-Dec-2016
https://dl.acm.org/doi/10.1145/3006299.3006313
Reddy PShyamala K(2016)New scoring formula to rank hypervisors' performance complementing with statistical analysis using DOEFuture Generation Computer Systems10.1016/j.future.2016.02.01261:C(54-65)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1016/j.future.2016.02.012
Show More Cited By

Characterization of network processing overheads in Xen

Recommendations

Diagnosing performance overheads in the xen virtual machine environment
VEE '05: Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments

Virtual Machine (VM) environments (e.g., VMware and Xen) are experiencing a resurgence of interest for diverse uses including server consolidation and shared hosting. An application's performance in a virtual machine environment can differ markedly from ...
Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well ...
Xen and co.: communication-aware CPU scheduling for consolidated xen-based hosting platforms
VEE '07: Proceedings of the 3rd international conference on Virtual execution environments

Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

VTDC '06: Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing

November 2006

106 pages

ISBN:0769528731

Publisher

IEEE Computer Society

United States

Publication History

Published: 17 November 2006

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 5 of 10 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
851
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang WShi YZheng YLiu LCui L(2017)Resource and performance prediction at high utilization for N-Tier cloud-based service systemsProceedings of the Australasian Computer Science Week Multiconference10.1145/3014812.3014857(1-9)Online publication date: 30-Jan-2017
https://dl.acm.org/doi/10.1145/3014812.3014857
Gugnani SLu XPanda DAnjum AZhao X(2016)Performance characterization of hadoop workloads on SR-IOV-enabled virtualized InfiniBand clustersProceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies10.1145/3006299.3006313(36-45)Online publication date: 6-Dec-2016
https://dl.acm.org/doi/10.1145/3006299.3006313
Reddy PShyamala K(2016)New scoring formula to rank hypervisors' performance complementing with statistical analysis using DOEFuture Generation Computer Systems10.1016/j.future.2016.02.01261:C(54-65)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1016/j.future.2016.02.012
Zhang JLu XArnold MPanda DBalaji PXu C(2015)MVAPICH2 over openstack with SR-IOVProceedings of the 15th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2015.166(71-80)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1109/CCGrid.2015.166
Huang QLee P(2013)An experimental study of cascading performance interference in a virtualized environmentACM SIGMETRICS Performance Evaluation Review10.1145/2479942.247994840:4(43-52)Online publication date: 29-Apr-2013
https://dl.acm.org/doi/10.1145/2479942.2479948
Venzano DMichiardi P(2013)A Measurement Study of Data-Intensive Network Traffic Patterns in a Private CloudProceedings of the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing10.1109/UCC.2013.93(476-481)Online publication date: 9-Dec-2013
https://dl.acm.org/doi/10.1109/UCC.2013.93
Jose JLi MLu XKandalla KArnold MPanda DEpema D(2013)SR-IOV support for virtualization on infiniband clustersProceedings of the 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2013.76(385-392)Online publication date: 13-May-2013
https://dl.acm.org/doi/10.1109/CCGrid.2013.76
Bourguiba MHaddadou KPujolle G(2011)Evaluating Xen-based virtual routers performanceInternational Journal of Communication Networks and Distributed Systems10.1504/IJCNDS.2011.0395356:3(268-282)Online publication date: 1-Apr-2011
https://dl.acm.org/doi/10.1504/IJCNDS.2011.039535
Lee MKrishnakumar AKrishnan PSingh NYajnik S(2010)Supporting soft real-time tasks in the xen hypervisorACM SIGPLAN Notices10.1145/1837854.173601245:7(97-108)Online publication date: 17-Mar-2010
https://dl.acm.org/doi/10.1145/1837854.1736012
Lee MKrishnakumar AKrishnan PSingh NYajnik SFiuczynski MBerger EWarfield A(2010)Supporting soft real-time tasks in the xen hypervisorProceedings of the 6th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments10.1145/1735997.1736012(97-108)Online publication date: 17-Mar-2010
https://dl.acm.org/doi/10.1145/1735997.1736012
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents