research-article

Venice: An Effective Resource Sharing Architecture for Data Center Servers

Authors:

Sally A. Mckee,

Qianlong Zhang,

Lixin Zhang, and

Dan MengAuthors Info & Claims

ACM Transactions on Computer Systems (TOCS), Volume 36, Issue 1

Article No.: 2, Pages 1 - 26

https://doi.org/10.1145/3310360

Published: 14 March 2019 Publication History

Abstract

Consolidated server racks are quickly becoming the standard infrastructure for engineering, business, medicine, and science. Such servers are still designed much in the way when they were organized as individual, distributed systems. Given that many fields rely on big-data analytics substantially, its cost-effectiveness and performance should be improved, which can be achieved by flexibly allowing resources to be shared across nodes. Here we describe Venice, a family of data-center server architectures that includes a strong communication substrate as a first-class resource. Venice supports a diverse set of resource-joining mechanisms that enables applications to leverage non-local resources efficiently.

We have constructed a hardware prototype to better understand the implications of design decisions about system support for resource sharing. We use it to measure the performance of at-scale applications and to explore performance, power, and resource-sharing transparency tradeoffs (i.e., how many programming changes are needed). We analyze these tradeoffs for sharing memory, accelerators, and NICs. We find that reducing/hiding latency is particularly important, the chosen communication channels should match the sharing access patterns of the applications, and of which we can improve performance by exploiting inter-channel collaboration.

References

[1]

2014. HUAWEI DC3.0. Retrieved on December 20, 2018 from http://www.huawei.com/ilink/en/download/HW_349607.

[2]

2014. Zynq®-7000 All Programmable SoC. Retrieved on December 20, 2018 from www.xilinx.com/products/silicon-devices/soc/zynq-7000.html.

[3]

2017. OpenCAPI Consortium. Retrieved on December 20, 2018 from https://opencapi.org/.

[4]

2018. CCIX Consortium. Retrieved on December 20, 2018 from https://www.ccixconsortium.com/.

[5]

2018. Gen-Z Consortium. Retrieved on December 20, 2018 from http://genzconsortium.org/.

[6]

2018. Infiniband Performance Benchmarks. Retrieved on December 20, 2018 from http://www.mellanox.com/page/performance_infiniband.

[7]

Y. Ajima, Y. Takagi, T. Inoue, S. Hiramoto, and T. Shimizu. 2011. The tofu interconnect. In Proc. IEEE Annual Symposium on High Performance Interconnects. 87--94.

Digital Library

[8]

C. Amza, A. L. Cox, S. Dwarkadas, P. Keleher, Honghui Lu, R. Rajamony, W. Yu, and W. Zwaenepoel. 1996. TreadMarks: Shared memory computing on networks of workstations. Computer 29, 2 (Feb. 1996), 18--28.

Digital Library

[9]

E. Anderson, J. Brooks, C. Grassl, and S. Scott. 1997. Performance of the CRAY T3E multiprocessor. In Proc. ACM/IEEE International Conference on Supercomputing. 39--39.

Digital Library

[10]

B. Arimilli, R. Arimilli, V. Chung, S. Clark, W. Denzel, B. Drerup, T. Hoefler, J. Joyner, J. Lewis, J. Li, N. Ni, and R. Rajamony. 2010. The PERCS high-performance interconnect. In Proc. IEEE Annual Symposium on High Performance Interconnects. 75--82.

Digital Library

[11]

T. Benson, A. Akella, and D. A. Maltz. 2010. Network traffic characteristics of data centers in the wild. In Proc. ACM SIGCOMM Conference on Internet Measurement. 267--280.

Digital Library

[12]

N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood. 2011. The gem5 simulator. SIGARCH Computer Architecture News 39, 2 (May 2011), 1--7.

Digital Library

[13]

Marco Ceriani, Simone Secchi, Oreste Villa, Antonino Tumeo, and Gianluca Palermo. 2017. Exploring Efficient Hardware Support for Applications with Irregular Memory Patterns on Multinode Manycore Architectures. IEEE Transactions on Parallel and Distributed Systems 28, 6 (2017), 1635–1648.

Digital Library

[14]

Yisong Chang, Ke Zhang, Sally A. McKee, Lixin Zhang, Mingyu Chen, Liqiang Ren, and Zhiwei Xu. 2016. Extending on-chip interconnects for rack-level remote resource access. In Proc. 2016 IEEE 34th International Conference on Computer Design (ICCD’16). IEEE, 56--63.

[15]

Michael D. Dahlin, Randolph Y. Wang, Thomas E. Anderson, and David A. Patterson. 1994. Cooperative caching: Using remote client memory to improve file system performance. In Proc. USENIX Conference on Operating Systems Design and Implementation. 19.

Digital Library

[16]

Oracle Berkeley DB. 2017. Retrieved on December 20, 2018 from http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html.

[17]

J. Dean and S. Ghemawat. 2008. MapReduce: Simplified data processing on large clusters. Commun ications of the ACM 51, 1 (Jan. 2008), 107--113.

Digital Library

[18]

M. J. Feeley, W. E. Morgan, E. P. Pighin, A. R. Karlin, H. M. Levy, and C. A. Thekkath. 1995. Implementing global memory management in a workstation cluster. In Proc. ACM Symposium on Operating Systems Principles. 201--212.

Digital Library

[19]

Andrew V. Goldberg. 1997. An efficient implementation of a scaling minimum-cost flow algorithm. Journal of Algorithms 22, 1 (Jan. 1997), 1--29.

Digital Library

[20]

Graph500. 2016. Retrieved on December 20, 2018 from http://www.graph500.org/.

[21]

Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. 2017. Efficient memory disaggregation with infiniswap. In Proc. NSDI. 649--667.

Digital Library

[22]

M. R. Hines, M. Lewandowski, and K. Gopalan. 2005. Anemone: Adaptive network memory engine. In Proc. ACM Symposium on Operating Systems Principles. 1.

Digital Library

[23]

Rui Hou, Tao Jiang, Liuhang Zhang, Pengfei Qi, Jianbo Dong, Haibin Wang, Xiongli Gu, and Shujie Zhang. 2013. Cost effective data center servers. In Proc. IEEE International Symposium on High Performance Computer Architecture. 179--187.

Digital Library

[24]

Iperf. 2014. Retrieved on December 20, 2018 from http://iperf.fr/.

[25]

H. Jin, X.-H. Sun, Y. Chen, and T. Ke. 2010. REMEM: Remote memory as checkpointing storage. In Proc. IEEE International Conference on Cloud Computing Technology and Science. 319--326.

Digital Library

[26]

M. J. Kumar. 2013. Rack scale architecture for cloud. In Intel Developer Forum.

[27]

J. Laudon and D. Lenoski. 1997. The SGI origin: A ccNUMA highly scalable server. In Proc. ACM International Symposium on Computer Architecture. 241--251.

Digital Library

[28]

D. Lenoski, J. Laudon, K. Gharachorloo, W.-D. Weber, A. Gupta, J. Hennessy, M. Horowitz, and M. S. Lam. 1992. The Stanford Dash multiprocessor. IEEE Computer 25, 3 (March 1992), 63--79.

Digital Library

[29]

K. Li and P. Hudak. 1989. Memory coherence in shared virtual memory systems. ACM Transactions on Computer Systems 7, 4 (Nov. 1989), 321--359.

Digital Library

[30]

K. Lim, J. Chang, T. Mudge, P. Ranganathan, S. K. Reinhardt, and T. F. Wenisch. 2009. Disaggregated memory for expansion and sharing in blade servers. In Proc. ACM International Symposium on Computer Architecture. 267--278.

Digital Library

[31]

K. Lim, P. Ranganathan, Jichuan Chang, C. Patel, T. Mudge, and S. Reinhardt. 2008. Understanding and designing new server architectures for emerging warehouse-computing environments. In Proc. ACM International Symposium on Computer Architecture. 315--326.

Digital Library

[32]

Kevin Lim, Yoshio Turner, Jose Renato Santos, Alvin AuYoung, Jichuan Chang, Parthasarathy Ranganathan, and Thomas F. Wenisch. 2012. System-level implications of disaggregated memory. In IEEE International Symposium on High-Performance Comp Architecture. IEEE, 1--12.

Digital Library

[33]

David Mayhew and Venkata Krishnan. 2003. PCI express and advanced switching: Evolutionary path to building next generation interconnects. In Proc. Symposium on High Performance Interconnects. 21--29.

[34]

Timothy Prickett Morgan. 2014. On-Chip Networking May Survive Calxeda Shutdown. Retrieved January 2014 from http://www.enterprisetech.com/2014/01/02/chip-networking-may-survive-calxeda-shutdown.

[35]

Michael Nelson, Beng-Hong Lim, and Greg Hutchins. 2005. Fast transparent migration for virtual machines. In Proc. USENIX Annual Technical Conference. 391--394.

Digital Library

[36]

S. Novakovic, A. Daglis, E. Bugnion, B. Falsafi, and B. Grot. 2014. Scale-out NUMA. In Proc. ACM International Conference on Architectural Support for Programming Languages and Operating Systems. 3--18.

Digital Library

[37]

J. Oleszkiewicz, L. Xiao, and Y. Liu. 2004. Parallel network RAM: Effectively utilizing global cluster memory for large data-intensive parallel programs. In Proc. International Conference on Parallel Processing, Vol. 1. 353--360.

Digital Library

[38]

Oracle Corp. 2014. MySQL: The World’s Most Popular Open-Source Database. Retrieved from http://www.mysql.com.

[39]

A. Putnam, A. M. Caulfield, E. S. Chung, D. Chiou, K. Constantinides, J. Demme, H. Esmaeilzadeh, J. Fowers, G. Gopal, J. Gray, M. Haselman, S Hauck, S. Heil, A. Hormati, J.-Y. Kim, S. Lanka, J. Larus, E. Peterson, S. Pope, A. Smith, J. Thong, P. Xiao, and D. Burger. 2014. A reconfigurable fabric for accelerating large-scale datacenter services. In Proc. ACM International Symposium on Computer Architecuture. 13--24.

Digital Library

[40]

Anil Rao. 2012. AMD | SeaMicro Technology Overview. Retrieved October 10, 2018 from http://www.seamicro.com/sites/default/files/SM_TO01_64_v2.7.pdf.

[41]

J. Regula. 2013. Integrating rack level connectivity into a PCI express switch. In Proc. Hot Chips: A Symposium on High Performance Chips. 259--266.

[42]

ScaleMP. 2011. Versatile SMP (vSMP) Architecture. Retrieved October 10, 2018 from http://www.scalemp.com/technology/versatile-smp-vsmp-architecture/.

[43]

T. Sherwood, E. Perelman, G. Hamerly, and B. Calder. 2002. Automatically characterizing large scale program behavior. In Proc. International Conference on Architectural Support for Programming Languages and Operating Systems. 319--326.

Digital Library

[44]

L. Wang, J. Zhan, C. Luo, Y. Zhu, Q. Yang, Y. He, W. Gao, Z. Jia, Y. Shi, S. Zhang, C. Zheng, G. Lu, K. Zhan, X. Li, and B. Qiu. 2014. BigDataBench: A big data benchmark suite from internet services. In Proc. IEEE International Symposium On High Performance Computer Architecture. 488--499.

[45]

Wiki. 2017. Intel Xeon Microprocessors. Retrieved October 10, 2018 from http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#Haswell-based_Xeons.

[46]

Steven Cameron Woo, Moriyoshi Ohara, Evan Torrie, Jaswinder Pal Singh, and Anoop Gupta. 1995. The SPLASH-2 programs: Characterization and methodological considerations. In Proc. ACM International Symposium on Computer Architecture. 24--36.

Digital Library

[47]

M. Xie, Y. Lu, K. Wang, L. Liu, H. Cao, and X. Yang. 2012. Tianhe-1A interconnect and message-passing services. IEEE Micro 32, 1 (Jan. 2012), 8--20.

Digital Library

[48]

Di Xu, Chenggang Wu, and Pen-Chung Yew. 2010. On mitigating memory bandwidth contention through bandwidth-aware scheduling. In Proc. IEEE/ACM/IFIP International Conference on Parallel Architectures and Compilation Techniques. 237--248.

Digital Library

[49]

M. Zaharia, M. F. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. 2012. Spark: Cluster computing with working sets. In Proc. USENIX Conference on Hot Topics in Cloud Computing. 10.

Digital Library

[50]

E. W. Felten and J. Zahorjan. 1991. Issues in the Implementation of a Remote Memory Paging System. Technical Report 91-03-09, University of Washington, Department of Computer Science and Engineering.

[51]

J. Zawodny. 2009. Redis: Lightweight key/value store that goes the extra mile. Linux Magazine 79 (Aug. 2009).

Cited By

Puri ABellamkonda KNarreddy KJose JTamarapalli VNarayanan V(2024)DRackSim: Simulating CXL-enabled Large-Scale Disaggregated Memory SystemsProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656059(3-14)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3615979.3656059
Xia ZDong NWu JMa C(2024)Multivariate Knowledge Tracking Based on Graph Neural Network in ASSISTmentsIEEE Transactions on Learning Technologies10.1109/TLT.2023.330101117(32-43)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TLT.2023.3301011
Liu KHan YZhang JChen ZSarro FHarman MHuang GMa YJust RFraser G(2023)Who Judges the Judge: An Empirical Study on Online Judge TestsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598060(334-346)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598060
Show More Cited By

Index Terms

Venice: An Effective Resource Sharing Architecture for Data Center Servers
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing

Recommendations

Flexible Device Sharing in PCIe Clusters using Device Lending
ICPP Workshops '18: Workshop Proceedings of the 47th International Conference on Parallel Processing

Processing workloads may have very high IO demands, exceeding the capabilities provided by resource virtualization and requiring direct access to the physical hardware. For computers that are interconnected in PCI Express (PCIe) networks, we have ...
Read More
Transparently bridging semantic gap in CPU management for virtualized environments

Consolidated environments are progressively accommodating diverse and unpredictable workloads in conjunction with virtual desktop infrastructure and cloud computing. Unpredictable workloads, however, aggravate the semantic gap between the virtual ...
Read More
Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending
Abstract
Modern workloads often exceed the processing and I/O capabilities provided by resource virtualization, requiring direct access to the physical hardware in order to reduce latency and computing overhead. For computers interconnected in a cluser, ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems

ACM Transactions on Computer Systems Volume 36, Issue 1

February 2018

222 pages

ISSN:0734-2071

EISSN:1557-7333

DOI:10.1145/3319851

Editor:
Michael Swift
University of Wisconsin, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2019

Accepted: 01 November 2018

Revised: 01 August 2018

Received: 01 May 2017

Published in TOCS Volume 36, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Chinese Academy of Science
National Science Fund for Outstanding Young Scholars, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
776
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)3

Other Metrics

View Author Metrics

Citations

Cited By

Puri ABellamkonda KNarreddy KJose JTamarapalli VNarayanan V(2024)DRackSim: Simulating CXL-enabled Large-Scale Disaggregated Memory SystemsProceedings of the 38th ACM SIGSIM Conference on Principles of Advanced Discrete Simulation10.1145/3615979.3656059(3-14)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3615979.3656059
Xia ZDong NWu JMa C(2024)Multivariate Knowledge Tracking Based on Graph Neural Network in ASSISTmentsIEEE Transactions on Learning Technologies10.1109/TLT.2023.330101117(32-43)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TLT.2023.3301011
Liu KHan YZhang JChen ZSarro FHarman MHuang GMa YJust RFraser G(2023)Who Judges the Judge: An Empirical Study on Online Judge TestsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598060(334-346)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598060
Zhao GHuang ZZhuang YLiu JLiu QLiu ZWu JChen EFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Simulating Student Interactions with Two-stage Imitation Learning for Intelligent Educational SystemsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615060(3423-3432)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615060
Gao WWang HLiu QWang FLin XYue LZhang ZLv RWang SChen HDuh WHuang HKato MMothe JPoblete B(2023)Leveraging Transferable Knowledge Concept Graph Embedding for Cold-Start Cognitive DiagnosisProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591774(983-992)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591774
Qiu ZXiao PNguyen O(2022)Construction of Data Resource Sharing Platform in College Students’ Ideological and Political Education Based on Deep LearningWireless Communications & Mobile Computing10.1155/2022/29058872022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/2905887
Hou Z(2022)Remote English Teaching Resource Sharing Based on Internet O2O ModelScientific Programming10.1155/2022/12178072022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/1217807
Rodrigues LPereira FToda APalomino POliveira WPessoa MCarvalho LOliveira DOliveira ECristea AIsotani S(2022)Are They Learning or Playing? Moderator Conditions of Gamification’s Success in Programming ClassroomsACM Transactions on Computing Education10.1145/348573222:3(1-27)Online publication date: 9-Jun-2022
https://dl.acm.org/doi/10.1145/3485732
Shen DQin CZhu HXu TChen EXiong H(2021)Joint Representation Learning with Relation-Enhanced Topic Models for Intelligent Job Interview AssessmentACM Transactions on Information Systems10.1145/346965440:1(1-36)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3469654
Wang HLian DTong HLiu QHuang ZChen E(2021)HyperSoRec: Exploiting Hyperbolic User and Item Representations with Multiple Aspects for Social-aware RecommendationACM Transactions on Information Systems10.1145/346391340:2(1-28)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3463913
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents