DxHash: A Memory-saving Consistent Hashing Algorithm
Authors: 
Chao Dong, Fang Wang, Dan FengAuthors Info & Claims
Chao Dong, Fang Wang, Dan FengAuthors Info & ClaimsArticle No.: 3, Pages 1 - 22
Abstract
Consistent Hashing (CH) algorithms are widely adopted in networks and distributed systems for their ability to achieve load balancing and minimize disruptions. However, the rise of the Internet of Things (IoT) has introduced new challenges for existing CH algorithms, characterized by high memory usage and update overhead. This article presents DxHash, a novel CH algorithm based on repeated pseudo-random number generation. DxHash offers significant benefits, including a remarkably low memory footprint, high lookup throughput, and minimal update overhead. Additionally, we introduce a weighted variant of DxHash, enabling adaptable weight adjustments to handle heterogeneous load distribution. Through extensive evaluation, we demonstrate that DxHash outperforms AnchorHash, a state-of-the-art CH algorithm, in terms of the reduction of up to 98.4% in memory footprint and comparable performance in lookup and update.
References
[1]
Shahbaz Akhtar Abid, Mazliza Othman, and Nadir Shah. 2014. A survey on DHT-based routing for large-scale mobile ad hoc networks. ACM Comput. Surv. 47, 2, Article 20 (Aug.2014), 46 pages. DOI:
[2]
Stephanos Androutsellis-Theotokis and Diomidis Spinellis. 2004. A survey of peer-to-peer content distribution technologies. ACM Computing Surveys (CSUR) 36, 4 (2004), 335–371.
[3]
Ben Appleton and Michael O’Reilly. 2015. Multi-probe consistent hashing. arXiv preprint arXiv:1505.00062 (2015).
[4]
Elaine B. Barker, William C. Barker, William E. Burr, W. Timothy Polk, and Miles E. Smid. 2007. SP 800-57. Recommendation for Key Management, Part 1: General (Revised), Technical Report, National Institute of Standards & Technology, Gaithersburg, MD.
[5]
Chanwoo Chung, Jinhyung Koo, Junsu Im, Arvind, and Sungjin Lee. 2019. LightStore: Software-defined network-attached key-value drives. In Proceedings of the 24th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’19). Association for Computing Machinery, New York, NY, 939–953. DOI:
[6]
Cisco. 2020. Cisco Annual Internet Report (2018–2023) White Paper. https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.pdf
[7]
Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. 2007. Dynamo: Amazon’s highly available key-value store. ACM SIGOPS Operating Systems Review 41, 6 (2007), 205–220.
[8]
Daniel E. Eisenbud, Cheng Yi, Carlo Contavalli, Cody Smith, Roman Kononov, Eric Mann-Hielscher, Ardas Cilingiroglu, Bin Cheyney, Wentao Shang, and Jinnah Dylan Hosein. 2016. Maglev: A fast and reliable software network load balancer. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI’16). USENIX Association, Santa Clara, CA, 523–535. https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud
[9]
Xiang Fu, Can Peng, and Weihong Han. 2015. A consistent hashing based data redistribution algorithm. In Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques, Xiaofei He, Xinbo Gao, Yanning Zhang, Zhi-Hua Zhou, Zhi-Yong Liu, Baochuan Fu, Fuyuan Hu, and Zhancheng Zhang (Eds.). Springer International Publishing, Cham, 559–566.
[10]
Pulkit Goel, Kumar Rishabh, and Vasudeva Varma. 2017. An alternate load distribution scheme in DHTs. In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom’17). 218–222. DOI:
[11]
Harshit Gupta, Zhuangdi Xu, and Umakishore Ramachandran. 2018. DataFog: Towards a holistic data management platform for the IoT age at the network edge. In USENIX Workshop on Hot Topics in Edge Computing (HotEdge’18). USENIX Association, Boston, MA. https://www.usenix.org/conference/hotedge18/presentation/gupta
[12]
David Karger, Eric Lehman, Tom Leighton, Rina Panigrahy, Matthew Levine, and Daniel Lewin. 1997. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the world wide web. In Proceedings of the 29th Annual ACM Symposium on Theory of Computing (STOC’97). Association for Computing Machinery, New York, NY, 654–663. DOI:
[13]
David Karger, Alex Sherman, Andy Berkheimer, Bill Bogstad, Rizwan Dhanidina, Ken Iwamoto, Brian Kim, Luke Matkins, and Yoav Yerushalmi. 1999. Web caching with consistent hashing. Computer Networks 31, 11–16 (1999), 1203–1213.
[14]
Shaishav Kumar and Raghavendra Udupa. 2011. Learning hash functions for cross-view similarity search. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Volume 2 (IJCAI’11). AAAI Press, 1360–1365.
[15]
Avinash Lakshman and Prashant Malik. 2010. Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review 44, 2 (2010), 35–40.
[16]
John Lamping and Eric Veach. 2014. A Fast, Minimal Memory, Consistent Hash Algorithm. (2014). arxiv:cs.DS/1406.2294
[17]
Zaoxing Liu, Zhihao Bai, Zhenming Liu, Xiaozhou Li, Changhoon Kim, Vladimir Braverman, Xin Jin, and Ion Stoica. 2019. DistCache: Provable load balancing for large-scale storage systems with distributed caching. In 17th USENIX Conference on File and Storage Technologies (FAST’19). USENIX Association, Boston, MA, 143–157. https://www.usenix.org/conference/fast19/presentation/liu
[18]
Petar Maymounkov and David Mazieres. 2002. Kademlia: A peer-to-peer information system based on the xor metric. In International Workshop on Peer-to-peer Systems. Springer, 53–65.
[19]
Gal Mendelson, Shay Vargaftik, Katherine Barabash, Dean H. Lorenz, Isaac Keslassy, and Ariel Orda. 2021. AnchorHash: A scalable consistent hash. IEEE/ACM Transactions on Networking 29, 2 (2021), 517–528. DOI:
[20]
Satoshi Nakamoto. 2008. Bitcoin: A peer-to-peer electronic cash system. https://bitcoin.org/bitcoin.pdf
[21]
Yuichi Nakatani. 2021. Structured allocation-based consistent hashing with improved balancing for cloud infrastructure. IEEE Transactions on Parallel and Distributed Systems 32, 9 (2021), 2248–2261. DOI:
[22]
Andreas N. Philippou, Costas Georghiou, and George N. Philippou. 1983. A generalized geometric distribution and some of its properties. Statistics & Probability Letters 1, 4 (1983), 171–175.
[23]
Jiwu Shu, Youmin Chen, Qing Wang, Bohong Zhu, Junru Li, and Youyou Lu. 2020. TH-DPMS: Design and implementation of an RDMA-enabled distributed persistent memory storage system. ACM Transactions on Storage 16, 4, Article 24 (Oct.2020), 31 pages. DOI:
[24]
Ronak Singhal. 2008. Inside Intel® core microarchitecture (nehalem). In 2008 IEEE Hot Chips 20 Symposium (HCS’08). IEEE, 1–25.
[25]
I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan. 2003. Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Transactions on Networking 11, 1 (2003), 17–32. DOI:
[26]
D. G. Thaler and C. V. Ravishankar. 1998. Using name-based mappings to increase hit rates. IEEE/ACM Transactions on Networking 6, 1 (1998), 1–14. DOI:
[27]
Xiaoming Wang and Dmitri Loguinov. 2007. Load-balancing performance of consistent hashing: Asymptotic analysis of random node join. IEEE/ACM Transactions on Networking 15, 4 (2007), 892–905. DOI:
[28]
Yair Weiss, Antonio Torralba, and Rob Fergus. 2009. Spectral hashing. In Proceedings of the 2009 Conference on Computer Vision and Pattern Recognition, 293–300.
[29]
Chenggang Wu, Vikram Sreekanti, and Joseph M. Hellerstein. 2019. Autoscaling tiered cloud storage in anna. Proceedings of the VLDB Endowment 12, 6 (Feb.2019), 624–638. DOI:
[30]
Di Wu, Dmitri I. Arkhipov, Eskindir Asmare, Zhijing Qin, and Julie A. McCann. 2015. UbiFlow: Mobility management in urban-scale software defined IoT. In 2015 IEEE Conference on Computer Communications (INFOCOM’15). 208–216. DOI:
Index Terms
- DxHash: A Memory-saving Consistent Hashing Algorithm
Recommendations
A Write-efficient and Consistent Hashing Scheme for Non-Volatile Memory
ICPP '18: Proceedings of the 47th International Conference on Parallel ProcessingThe development of non-volatile memory technologies (NVMs) has attracted interest in designing data structures that are efficiently adapted to NVMs. In this context, several NVM-friendly hashing schemes have been proposed to reduce extra writes to NVMs, ...
A Consistent Hashing Based Data Redistribution Algorithm
IScIDE 2015: Revised Selected Papers, Part II, of the 5th International Conference on Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques - Volume 9243Data partitioning is the basis of massive data distributed storage management and consistent hashing algorithm is one of the most popular data partitioning methods. We analyzes consistent hashing algorithm and find it can not guarantee the dynamic load ...
Load-balancing performance of consistent hashing: asymptotic analysis of random node join
Balancing of structured peer-to-peer graphs, including their zone sizes, has recently become an important topic of distributed hash table (DHT) research. To bring analytical understanding into the various peer-join mechanisms based on consistent hashing,...
Comments
Information & Contributors
Information
Published In
February 2024
178 pages
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 February 2024
Online AM: 03 November 2023
Accepted: 27 October 2023
Revised: 21 September 2023
Received: 09 March 2023
Published in TOIT Volume 24, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China (NSFC)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 302Total Downloads
- Downloads (Last 12 months)302
- Downloads (Last 6 weeks)38
Reflects downloads up to 26 Jul 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in