research-article

HeteroVisor: Exploiting Resource Heterogeneity to Enhance the Elasticity of Cloud Platforms

Authors:

Karsten SchwanAuthors Info & Claims

VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 79 - 92

https://doi.org/10.1145/2731186.2731191

Published: 14 March 2015 Publication History

Abstract

This paper presents HeteroVisor, a heterogeneity-aware hypervisor, that exploits resource heterogeneity to enhance the elasticity of cloud systems. Introducing the notion of 'elasticity' (E) states, HeteroVisor permits applications to manage their changes in resource requirements as state transitions that implicitly move their execution among heterogeneous platform components. Masking the details of platform heterogeneity from virtual machines, the E-state abstraction allows applications to adapt their resource usage in a fine-grained manner via VM-specific 'elasticity drivers' encoding VM-desired policies. The approach is explored for the heterogeneous processor and memory subsystems evolving for modern server platforms, leading to mechanisms that can manage these heterogeneous resources dynamically and as required by the different VMs being run. HeteroVisor is implemented for the Xen hypervisor, with mechanisms that go beyond core scaling to also deal with memory resources, via the online detection of hot memory pages and transparent page migration. Evaluation on an emulated heterogeneous platform uses workload traces from real-world data, demonstrating the ability to provide high on-demand performance while also reducing resource usage for these workloads.

References

[1]

O. Agmon Ben-Yehuda, M. Ben-Yehuda, A. Schuster, and D. Tsafrir. The resource-as-a-service (RaaS) cloud. In Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing, HotCloud'12, pages 12--12, Berkeley, CA, USA, 2012. USENIX Association.

Digital Library

[2]

O. Agmon Ben-Yehuda, E. Posener, M. Ben-Yehuda, A. Schuster, and A. Mu'alem. Ginseng: Market-driven memory allocation. In Proceedings of the 10th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE '14, pages 41--52, New York, NY, USA, 2014. ACM. 10.1145/2576195.2576197.

Digital Library

[3]

D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan. FAWN: a fast array of wimpy nodes. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, SOSP '09, pages 1--14. ACM, 2009. 10.1145/1629575.1629577.

Digital Library

[4]

P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP '03, pages 164--177, New York, NY, USA, 2003. ACM. 10.1145/945445.945462.

Digital Library

[5]

L. A. Barroso. Brawny cores still beat wimpy cores, most of the time. Micro, IEEE, 30 (4): 20-24, july-aug. 2010. ISSN 0272-1732. 10.1109/MM.2010.61.

[6]

R. Bryant, A. Tumanov, O. Irzak, A. Scannell, K. Joshi, M. Hiltunen, A. Lagar-Cavilla, and E. de Lara. Kaleidoscope: cloud micro-elasticity via VM state coloring. In Proceedings of the sixth conference on Computer systems, EuroSys '11, pages 273--286, New York, NY, USA, 2011. ACM. 10.1145/1966445.1966471.

Digital Library

[7]

T. Cao, S. M. Blackburn, T. Gao, and K. S. McKinley. The yin and yang of power and performance for asymmetric hardware and managed software. In Proceedings of the 39th Annual International Symposium on Computer Architecture, ISCA '12, pages 225--236, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[8]

C. Delimitrou and C. Kozyrakis. Paragon: QoS-aware scheduling for heterogeneous datacenters. In Proceedings of the 18th international conference on Architectural support for programming languages and operating systems, ASPLOS '13, pages 77--88, New York, NY, USA, 2013. ACM. 10.1145/2451116.2451125.

Digital Library

[9]

Q. Deng, D. Meisner, L. Ramos, T. F. Wenisch, and R. Bianchini. MemScale: active low-power modes for main memory. In Proceedings of the sixteenth international conference on Architectural support for programming languages and operating systems, ASPLOS XVI, pages 225--238. ACM, 2011. 10.1145/1950365.1950392.

Digital Library

[10]

Q. Deng, D. Meisner, A. Bhattacharjee, T. F. Wenisch, and R. Bianchini. CoScale: Coordinating CPU and memory system DVFS in server systems. In Proceedings of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '12, pages 143--154. IEEE, 2012. 10.1109/MICRO.2012.22.

Digital Library

[11]

X. Dong, Y. Xie, N. Muralimanohar, and N. P. Jouppi. Simple but effective heterogeneous main memory with on-chip memory controller support. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC '10. IEEE, 2010. 10.1109/SC.2010.50.

Digital Library

[12]

A. Fedorova, J. C. Saez, D. Shelepov, and M. Prieto. Maximizing power efficiency with asymmetric multicore systems. Commun. ACM, 52 (12): 48--57, Dec. 2009. http://doi.acm.org/10.1145/1610252.1610270.

Digital Library

[13]

G. Galante and L. C. E. d. Bona. A survey on cloud computing elasticity. In Proceedings of the 2012 IEEE/ACM Fifth International Conference on Utility and Cloud Computing, UCC '12, pages 263--270. IEEE Computer Society, 2012. 10.1109/UCC.2012.30.

Digital Library

[14]

A. Gandhi, T. Zhu, M. Harchol-Balter, and M. A. Kozuch. SOFTScale: stealing opportunistically for transient scaling. In Proceedings of the 13th International Middleware Conference, Middleware '12, pages 142--163, New York, NY, USA, 2012. Springer-Verlag New York, Inc.

Digital Library

[15]

A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant resource fairness: fair allocation of multiple resource types. In Proceedings of the 8th USENIX conference on Networked systems design and implementation, NSDI'11. USENIX Association, 2011.

Digital Library

[16]

P. Greenhalgh. Big.LITTLE Processing with ARM CortexTM-A15 & Cortex-A7. White paper, ARM, Sept 2011.

[17]

M. Guevara, B. Lubin, and B. C. Lee. Navigating heterogeneous processors with market mechanisms. In High Performance Computer Architecture (HPCA2013), 2013 IEEE 19th International Symposium on, pages 95--106, 2013. 10.1109/HPCA.2013.6522310.

Digital Library

[18]

J. L. Hellerstein. Google cluster data. Google research blog, Jan. 2010. Posted at http://googleresearch.blogspot.com/2010/01/google-cluster-data.html.

[19]

Y.-J. Hong, J. Xue, and M. Thottethodi. Dynamic server provisioning to minimize cost in an IaaS cloud. In Proceedings of the international conference on Measurement and modeling of computer systems, SIGMETRICS '11, pages 147--148, New York, NY, USA, 2011. ACM. 10.1145/1993744.1993799.

Digital Library

[20]

V. Janapa Reddi, B. C. Lee, T. Chilimbi, and K. Vaid. Web search using mobile cores: quantifying and mitigating the price of efficiency. In Proceedings of the 37th annual international symposium on Computer architecture, ISCA '10, pages 314--325, New York, NY, USA, 2010. ACM. 10.1145/1815961.1816002.

Digital Library

[21]

S. T. Jones, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Geiger: monitoring the buffer cache in a virtual machine environment. In Proceedings of the 12th international conference on Architectural support for programming languages and operating systems, ASPLOS XII, pages 14--24. ACM, 2006. 10.1145/1168857.1168861.

Digital Library

[22]

V. Kazempour, A. Kamali, and A. Fedorova. AASH: an asymmetry-aware scheduler for hypervisors. In Proceedings of the 6th ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '10, pages 85--96, New York, NY, USA, 2010. ACM. 10.1145/1735997.1736011.

Digital Library

[23]

D. Koufaty, D. Reddy, and S. Hahn. Bias scheduling in heterogeneous multi-core architectures. In Proceedings of the 5th European conference on Computer systems, EuroSys '10, pages 125--138, New York, NY, USA, 2010. ACM.

Digital Library

[24]

R. Kumar, K. I. Farkas, N. P. Jouppi, P. Ranganathan, and D. M. Tullsen. Single-ISA heterogeneous multi-core architectures: The potential for processor power reduction. In Proceedings of the 36th annual IEEE/ACM International Symposium on Microarchitecture, MICRO 36. IEEE, 2003.

Digital Library

[25]

Y. Kwon, C. Kim, S. Maeng, and J. Huh. Virtualizing performance asymmetric multi-core systems. In Proceedings of the 38th annual international symposium on Computer architecture, ISCA '11, pages 45--56, New York, NY, USA, 2011. ACM. 10.1145/2000064.2000071.

Digital Library

[26]

W. Lang, J. M. Patel, and S. Shankar. Wimpy node clusters: what about non-wimpy workloads? In Proceedings of the Sixth International Workshop on Data Management on New Hardware, DaMoN' 10, pages 47--55. ACM, 2010. 10.1145/1869389.1869396.

Digital Library

[27]

M. Lee and K. Schwan. Region scheduling: efficiently using the cache architectures via page-level affinity. In Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '12, pages 451--462. ACM, 2012. 10.1145/2150976.2151023.

Digital Library

[28]

K. Lim, Y. Turner, J. R. Santos, A. AuYoung, J. Chang, P. Ranganathan, and T. F. Wenisch. System-level implications of disaggregated memory. In Proceedings of the 2012 IEEE 18th International Symposium on High-Performance Computer Architecture, HPCA '12, pages 1--12. IEEE, 2012. 10.1109/HPCA.2012.6168955.

Digital Library

[29]

G. H. Loh. 3D-stacked memory architectures for multi-core processors. In Proceedings of the 35th Annual International Symposium on Computer Architecture, ISCA '08, pages 453--464. IEEE Computer Society, 2008. 10.1109/ISCA.2008.15.

Digital Library

[30]

G. H. Loh, N. Jayasena, K. McGrath, M. O'Connor, S. Reinhardt, and J. Chung. Challenges in heterogeneous die-stacked and off-chip memory systems. In In Proc. of 3rd Workshop on SoCs, Heterogeneity, and Workloads (SHAW), Feb 2012.

[31]

P. Lu and K. Shen. Virtual machine memory access tracing with hypervisor exclusive cache. In 2007 USENIX Annual Technical Conference on Proceedings of the USENIX Annual Technical Conference, ATC'07, pages 3:1--3:15, Berkeley, CA, USA, 2007. USENIX Association.

Digital Library

[32]

R. Nathuji and K. Schwan. VirtualPower: coordinated power management in virtualized enterprise systems. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, SOSP '07, pages 265--278. ACM, 2007. 10.1145/1294261.1294287.

Digital Library

[33]

R. Nathuji, C. Isci, and E. Gorbatov. Exploiting platform heterogeneity for power efficient data centers. In Proceedings of the Fourth International Conference on Autonomic Computing, ICAC '07, pages 5--. IEEE Computer Society, 2007. 10.1109/ICAC.2007.16.

Digital Library

[34]

R. Nathuji, A. Kansal, and A. Ghaffarkhah. Q-clouds: managing performance interference effects for QoS-aware clouds. In Proceedings of the 5th European conference on Computer systems, EuroSys '10, pages 237--250. ACM, 2010. 10.1145/1755913.1755938.

Digital Library

[35]

Nvidia. Variable SMP: A multi-core CPU architecture for low power and high performance. White paper, 2011.

[36]

ää, and Hui}Ou2012Z. Ou, H. Zhuang, J. K. Nurminen, A. Ylä Jää, and P. Hui. Exploiting hardware heterogeneity within the same instance type of Amazon EC2. In Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing, HotCloud'12, pages 4--4, Berkeley, CA, USA, 2012. USENIX Association.

Digital Library

[37]

P. Padala, K.-Y. Hou, K. G. Shin, X. Zhu, M. Uysal, Z. Wang, S. Singhal, and A. Merchant. Automated control of multiple virtualized resources. In Proceedings of the 4th ACM European conference on Computer systems, EuroSys '09, pages 13--26, New York, NY, USA, 2009. ACM. 10.1145/1519065.1519068.

Digital Library

[38]

V. Pallipadi and A. Starikovskiy. The ondemand governor: Past, present and future. Linux Symposium, 2: 223--238, 2006.

[39]

S. Panneerselvam and M. M. Swift. Chameleon: operating system support for dynamic processors. In Proceedings of the 17th international conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVII, pages 99--110, New York, NY, USA, 2012. ACM. 10.1145/2150976.2150988.

Digital Library

[40]

L. E. Ramos, E. Gorbatov, and R. Bianchini. Page placement in hybrid memory systems. In Proceedings of the international conference on Supercomputing, ICS '11, pages 85--95, New York, NY, USA, 2011. ACM. 10.1145/1995896.1995911.

Digital Library

[41]

J. C. Saez, M. Prieto, A. Fedorova, and S. Blagodurov. A comprehensive scheduler for asymmetric multicore systems. In 5th EuroSys, pages 139--152, New York, NY, USA, 2010. http://doi.acm.org/10.1145/1755913.1755929.

Digital Library

[42]

Z. Shen, S. Subbiah, X. Gu, and J. Wilkes. CloudScale: elastic resource scaling for multi-tenant cloud systems. In Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC '11, pages 5:1--5:14, New York, NY, USA, 2011. ACM. 10.1145/2038916.2038921.

Digital Library

[43]

T. Somu Muthukaruppan, A. Pathania, and T. Mitra. Price theory based power management for heterogeneous multi-cores. In 19th Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS, pages 161--176. ACM, 2014. 10.1145/2541940.2541974.

Digital Library

[44]

K. Sudan, K. Rajamani, W. Huang, and J. Carter. Tiered memory: An iso-power memory architecture to address the memory power wall. Computers, IEEE Transactions on, 61 (12): 1697--1710, Dec 2012. ISSN 0018-9340. 10.1109/TC.2012.119.

Digital Library

[45]

K. Van Craeynest, A. Jaleel, L. Eeckhout, P. Narvaez, and J. Emer. Scheduling heterogeneous multi-cores through performance impact estimation (PIE). In Proceedings of the 39th Annual International Symposium on Computer Architecture, ISCA '12, pages 213--224, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[46]

C. A. Waldspurger. Memory resource management in VMware ESX server. In Proceedings of the 5th USENIX conference on Operating systems design and implementation, OSDI'02, Berkeley, CA, USA, 2002. USENIX Association. 10.1145/844128.844146.

Digital Library

[47]

W. Wang, B. Liang, and B. Li. Revenue maximization with dynamic auctions in IaaS cloud markets. In Quality of Service (IWQoS), 2013 IEEE/ACM 21st International Symposium on, pages 1--6, 2013. 10.1109/IWQoS.2013.6550265.

[48]

D. Wong and M. Annavaram. KnightShift: Scaling the energy proportionality wall through server-level heterogeneity. In Proceedings of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO '12, pages 119--130. IEEE Computer Society, 2012. 10.1109/MICRO.2012.20.

Digital Library

[49]

S. M. Zahedi and B. C. Lee. REF: Resource elasticity fairness with sharing incentives for multiprocessors. In International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '14, pages 145--160. ACM, 2014. 10.1145/2541940.2541962.

Digital Library

[50]

W. Zhao and Z. Wang. Dynamic memory balancing for virtual machines. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, VEE '09, pages 21--30. ACM, 2009. 10.1145/1508293.1508297.

Digital Library

Cited By

Lee TMonga SMin CEom YDruschel PKaufmann AMace JFlinn JSeltzer M(2023)MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613167(17-34)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3600006.3613167
Heo TWang YCui WHuh JZhang L(2022)Adaptive Page Migration Policy With Huge Pages in Tiered Memory SystemsIEEE Transactions on Computers10.1109/TC.2020.303668671:1(53-68)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TC.2020.3036686
Tian CLiu HLiao XJin H(2022)UCat: heterogeneous memory management for unikernelsFrontiers of Computer Science10.1007/s11704-022-1201-y17:1Online publication date: 8-Aug-2022
https://doi.org/10.1007/s11704-022-1201-y
Show More Cited By

Index Terms

HeteroVisor: Exploiting Resource Heterogeneity to Enhance the Elasticity of Cloud Platforms
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems

Recommendations

HeteroVisor: Exploiting Resource Heterogeneity to Enhance the Elasticity of Cloud Platforms
VEE '15

This paper presents HeteroVisor, a heterogeneity-aware hypervisor, that exploits resource heterogeneity to enhance the elasticity of cloud systems. Introducing the notion of 'elasticity' (E) states, HeteroVisor permits applications to manage their ...
Full system simulation of many-core heterogeneous SoCs using GPU and QEMU semihosting
GPGPU-5: Proceedings of the 5th Annual Workshop on General Purpose Processing with Graphics Processing Units

Modern system-on-chips are evolving towards complex and heterogeneous platforms with general purpose processors coupled with massively parallel manycore accelerator fabrics (e.g. embedded GPUs). Platform developers are looking for efficient full-system ...
IC cloud: Enabling compositional cloud

Cloud computing has attracted great interest from both academic and industrial communities. Different paradigms, architectures and applications based on the concept of cloud have emerged. Although many of them have been quite successful, efforts are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE '15: Proceedings of the 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 2015

238 pages

ISBN:9781450334501

DOI:10.1145/2731186

General Chair:
Ada Gavrilovska
Georgia Tech
,
Program Chairs:
Angela Demke Brown
University of Toronto
,
Bjarne Steensgaard
Microsoft

ACM SIGPLAN Notices Volume 50, Issue 7
VEE '15
July 2015
221 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2817817
Editor:
Andy Gill
University of Kansas, Lawrence, KS
Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VEE '15

Sponsor:

VEE '15: 11th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 14 - 15, 2015

Istanbul, Turkey

Acceptance Rates

VEE '15 Paper Acceptance Rate 16 of 50 submissions, 32%;

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
720
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)4

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lee TMonga SMin CEom YDruschel PKaufmann AMace JFlinn JSeltzer M(2023)MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613167(17-34)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3600006.3613167
Heo TWang YCui WHuh JZhang L(2022)Adaptive Page Migration Policy With Huge Pages in Tiered Memory SystemsIEEE Transactions on Computers10.1109/TC.2020.303668671:1(53-68)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TC.2020.3036686
Tian CLiu HLiao XJin H(2022)UCat: heterogeneous memory management for unikernelsFrontiers of Computer Science10.1007/s11704-022-1201-y17:1Online publication date: 8-Aug-2022
https://doi.org/10.1007/s11704-022-1201-y
Doudali TZahka DGavrilovska A(2021)Cori: Dancing to the Right Beat of Periodic Data Movements over Hybrid Memory Systems2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS49936.2021.00043(350-359)Online publication date: May-2021
https://doi.org/10.1109/IPDPS49936.2021.00043
Choi JBlagodurov STseng H(2021)Dancing in the Dark: Profiling for Tiered Memory2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS49936.2021.00011(13-22)Online publication date: May-2021
https://doi.org/10.1109/IPDPS49936.2021.00011
Liu WLiu HLiao XJin HZhang Y(2021)Straggler-Aware Parallel Graph Processing in Hybrid Memory Systems2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid51090.2021.00031(217-226)Online publication date: May-2021
https://doi.org/10.1109/CCGrid51090.2021.00031
Cai MHuang H(2021)A survey of operating system support for persistent memoryFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-020-9395-315:4Online publication date: 1-Aug-2021
https://dl.acm.org/doi/10.1007/s11704-020-9395-3
Doudali TZahka DGavrilovska A(2020)The Case for Optimizing the Frequency of Periodic Data Movements over Hybrid Memory SystemsProceedings of the International Symposium on Memory Systems10.1145/3422575.3422788(137-143)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3422575.3422788
Sharma PAli-Eldin AShenoy P(2019)Resource DeflationProceedings of the Fourteenth EuroSys Conference 201910.1145/3302424.3303945(1-17)Online publication date: 25-Mar-2019
https://dl.acm.org/doi/10.1145/3302424.3303945
Yan ZLustig DNellans DBhattacharjee ABahar IHerlihy MWitchel ELebeck A(2019)Nimble Page Management for Tiered Memory SystemsProceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/3297858.3304024(331-345)Online publication date: 4-Apr-2019
https://dl.acm.org/doi/10.1145/3297858.3304024
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents