research-article

Memory Efficient Loss Recovery for Hardware-based Transport in Datacenter

Authors:

Jiansong Zhang,

Yongqiang Xiong,

Enhong ChenAuthors Info & Claims

APNet '17: Proceedings of the First Asia-Pacific Workshop on Networking

Pages 22 - 28

https://doi.org/10.1145/3106989.3106993

Published: 03 August 2017 Publication History

Abstract

Limited by the small on-chip memory, hardware-based transport typically implements go-back-N loss recovery mechanism, which costs very few memory but is well-known to perform inferior even under small packet loss ratio. We present MELO, an efficient selective retransmission mechanism for hardware-based transport, which consumes only a constant small memory regardless of the number of concurrent connections. Specifically, MELO employs an architectural separation between data and meta data storage and uses a shared bits pool allocation mechanism to reduce meta data on-chip memory footprint. By only adding in average 23B extra on-chip states for each connection, MELO achieves up to 14.02x throughput while reduces 99% tail FCT by 3.11x compared with go-back-N under certain loss ratio.

References

[1]

2008. InfiniBand architecture volume 1, general specifications, release 1.2.1. InfiniBand Trade Association.

[2]

2010. Supplement to InfiniBand architecture specification volume 1 release 1.2.2 annex A16: RDMA over converged ethernet (RoCE). InfiniBand Trade Association.

[3]

2012. Supplement to InfiniBand architecture specification volume 1 release 1.2.2 annex A17: RoCEv2 (IP routable RoCE). InfiniBand Trade Association.

[4]

Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari Sridharan. 2010. Data Center TCP (DCTCP). In Proceedings of the ACM SIGCOMM 2010 Conference (SIGCOMM '10). ACM, New York, NY, USA, 63--74.

Digital Library

[5]

Remzi H Arpaci-Dusseau and Andrea C Arpaci-Dusseau. 2014. Operating systems: Three easy pieces. Vol. 151. Arpaci-Dusseau Books Wisconsin.

Digital Library

[6]

Adrian M Caulfield, Eric S Chung, Andrew Putnam, Hari Angepat, Jeremy Fowers, Michael Haselman, Stephen Heil, Matt Humphrey, Puneet Kaur, Joo-Young Kim, and others. 2016. A cloud-scale acceleration architecture. In Microarchitecture (MICRO), 2016 49th Annual IEEE/ACM International Symposium on. IEEE, 1--13.

Digital Library

[7]

Cisco. 2015. Priority Flow Control: Build Reliable Layer 2 Infrastructure. (2015). http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-542809_ns783_Networking_Solutions_White_Paper.html.

[8]

Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over Commodity Ethernet at Scale. In Proceedings of the 2016 conference on ACM SIGCOMM 2016 Conference. ACM, 202--215.

Digital Library

[9]

Chuanxiong Guo, Lihua Yuan, Dong Xiang, Yingnong Dang, Ray Huang, Dave Maltz, Zhaoyi Liu, Vin Wang, Bin Pang, Hua Chen, and others. 2015. Pingmesh: A large-scale system for data center network latency measurement and analysis. ACM SIGCOMM Computer Communication Review 45, 4 (2015), 139--152.

Digital Library

[10]

Shuihai Hu, Yibo Zhu, Peng Cheng, Chuanxiong Guo, Kun Tan, Jitendra Padhye, and Kai Chen. 2016. Deadlocks in Datacenter Networks: Why Do They Form, and How to Avoid Them. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks. ACM, 92--98.

Digital Library

[11]

ieee. 2010. 802.1Qbb - Priority-based Flow Control. (2010). http://www.ieee802.org/1/pages/802.1bb.html.

[12]

Anuj Kalia, Michael Kaminsky, and David G Andersen. 2016. Design Guidelines for High Performance RDMA Systems. In 2016 USENIX Annual Technical Conference (USENIX ATC 16).

Digital Library

[13]

Matt Mathis, Jamshid Mahdavi, Sally Floyd, and Allyn Romanow. 1996. TCP selective acknowledgment options. Technical Report.

[14]

Mellanox. 2012. Mellanox EN Driver for Linux. (2012). http://www.mellanox.com/page/products_dyn?product_family=27&mtag=linux_driver.

[15]

Andrew Putnam, Adrian M Caulfield, Eric S Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, Jeremy Fowers, Gopi Prashanth, Gopal Jan, and others. 2014. A reconfigurable fabric for accelerating large-scale datacenter services. international symposium on computer architecture 42, 3 (2014), 13--24.

Digital Library

[16]

Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion control for large-scale RDMA deployments. In ACM SIGCOMM Computer Communication Review, Vol. 45. ACM, 523--536.

Digital Library

Cited By

Skiadopoulos AXie ZZhao MCai QAgarwal SAdelmann JAhern DContavalli CGoldflam MMayatskikh VRaja RWalton DAgarwal RMukherjee SKozyrakis CGavrilovska ATerry D(2024)High-throughput and flexible host networking for accelerated computingProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691960(405-423)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691960
Ma JGuo ZSong M(2024)SSPRD: A Shared-Storage-Based Hardware Packet Reordering and Deduplication System for Multipath Transmission in Wide Area NetworksMicromachines10.3390/mi1511132315:11(1323)Online publication date: 30-Oct-2024
https://doi.org/10.3390/mi15111323
Ma JGuo ZPan YZhang MZhao ZSun ZChang Y(2024)ORNIC: A High-Performance RDMA NIC with Out-of-Order Packet Direct Write Method for Multipath TransmissionElectronics10.3390/electronics1401008814:1(88)Online publication date: 28-Dec-2024
https://doi.org/10.3390/electronics14010088
Show More Cited By

Index Terms

Memory Efficient Loss Recovery for Hardware-based Transport in Datacenter
1. Networks
  1. Network types
    1. Data center networks

Recommendations

On using forward error correction for loss recovery in optical burst switched networks

An important issue in optical burst switched (OBS) networks is the loss of bursts at intermediate nodes due to contention. Such contention losses, usually do not mean a situation of congestion. In this paper, we propose for the first time, a loss ...
Energy efficient Phase Change Memory based main memory for future high performance systems
IGCC '11: Proceedings of the 2011 International Green Computing Conference and Workshops

Phase Change Memory (PCM) has recently attracted a lot of attention as a scalable alternative to DRAM for main memory systems. As the need for high-density memory increases, DRAM has proven to be less attractive from the point of view of scaling and ...
A durable and energy efficient main memory using phase change memory technology
ISCA '09: Proceedings of the 36th annual international symposium on Computer architecture

Using nonvolatile memories in memory hierarchy has been investigated to reduce its energy consumption because nonvolatile memories consume zero leakage power in memory cells. One of the difficulties is, however, that the endurance of most nonvolatile ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

APNet '17: Proceedings of the First Asia-Pacific Workshop on Networking

August 2017

127 pages

ISBN:9781450352444

DOI:10.1145/3106989

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

APNet'17

APNet'17: First Asia-Pacific Workshop on Networking

August 3 - 4, 2017

Hong Kong, China

Acceptance Rates

Overall Acceptance Rate 50 of 118 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
232
Total Downloads

Downloads (Last 12 months)69
Downloads (Last 6 weeks)19

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Skiadopoulos AXie ZZhao MCai QAgarwal SAdelmann JAhern DContavalli CGoldflam MMayatskikh VRaja RWalton DAgarwal RMukherjee SKozyrakis CGavrilovska ATerry D(2024)High-throughput and flexible host networking for accelerated computingProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691960(405-423)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691960
Ma JGuo ZSong M(2024)SSPRD: A Shared-Storage-Based Hardware Packet Reordering and Deduplication System for Multipath Transmission in Wide Area NetworksMicromachines10.3390/mi1511132315:11(1323)Online publication date: 30-Oct-2024
https://doi.org/10.3390/mi15111323
Ma JGuo ZPan YZhang MZhao ZSun ZChang Y(2024)ORNIC: A High-Performance RDMA NIC with Out-of-Order Packet Direct Write Method for Multipath TransmissionElectronics10.3390/electronics1401008814:1(88)Online publication date: 28-Dec-2024
https://doi.org/10.3390/electronics14010088
Pan YGuo ZZhang M(2024)Design of a Fast and Scalable FPGA-Based Bitmap for RDMA NetworksElectronics10.3390/electronics1324490013:24(4900)Online publication date: 12-Dec-2024
https://doi.org/10.3390/electronics13244900
Huang PZhang XChen ZLiu CChen G(2024)LEFT: LightwEight and FasT packet Reordering for RDMAProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663418(67-73)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663418
Zhang JWang YZhong XYu MPan HZhang YGuan ZChe BWan ZPan THuang T(2024)PACC: A Proactive CNP Generation Scheme for Datacenter NetworksIEEE/ACM Transactions on Networking10.1109/TNET.2024.336177132:3(2586-2599)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3361771
Li ZHuang JWang SWang J(2024)Achieving Low Latency for Multipath Transmission in RDMA Based Data Center NetworkIEEE Transactions on Cloud Computing10.1109/TCC.2024.336507512:1(337-346)Online publication date: Jan-2024
https://doi.org/10.1109/TCC.2024.3365075
Oyama TShiomoto K(2024)Implementation of NVMe Over TCP Using SPDK and Its Performance Measurement2024 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR)10.1109/CQR62340.2024.10705884(13-18)Online publication date: 9-Sep-2024
https://doi.org/10.1109/CQR62340.2024.10705884
Yu ZSu BBai WRaindel SBraverman VJin XSchulzrinne HKohler EMaltz DMisra V(2023)Understanding the Micro-Behaviors of Hardware Offloaded Network Stacks with LuminaProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604837(1074-1087)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604837
He ZChen YHua B(2023)RoUD: Scalable RDMA over UD in Lossy Data Center Networks2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid57682.2023.00014(36-46)Online publication date: May-2023
https://doi.org/10.1109/CCGrid57682.2023.00014
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents