research-article

An In-depth Comparative Analysis of Cloud Block Storage Workloads: Findings and Implications

Authors:

Patrick P. C. Lee,

Chao ShiAuthors Info & Claims

ACM Transactions on Storage, Volume 19, Issue 2

Article No.: 16, Pages 1 - 32

https://doi.org/10.1145/3572779

Published: 06 March 2023 Publication History

Abstract

Cloud block storage systems support diverse types of applications in modern cloud services. Characterizing their input/output (I/O) activities is critical for guiding better system designs and optimizations. In this article, we present an in-depth comparative analysis of production cloud block storage workloads through the block-level I/O traces of billions of I/O requests collected from two production systems, Alibaba Cloud and Tencent Cloud Block Storage. We study their characteristics of load intensities, spatial patterns, and temporal patterns. We also compare the cloud block storage workloads with the notable public block-level I/O workloads from the enterprise data centers at Microsoft Research Cambridge, and we identify the commonalities and differences of the three sources of traces. To this end, we provide 6 findings through the high-level analysis and 16 findings through the detailed analysis on load intensity, spatial patterns, and temporal patterns. We discuss the implications of our findings on load balancing, cache efficiency, and storage cluster management in cloud block storage systems.

References

[1]

Irfan Ahmad. 2007. Easy and efficient disk I/O workload characterization in VMware ESX server. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’07). 149–158.

Digital Library

[2]

Alibaba. 2022. Alibaba Block Traces. Retrieved from https://github.com/alibaba/block-traces.

[3]

Alibaba. 2022. Alibaba Cloud Block Storage. Retrieved from https://www.alibabacloud.com/help/doc-detail/63136.htm.

[4]

Amazon. 2022. Amazon EBS. Retrieved from https://aws.amazon.com/ebs/.

[5]

Dulcardo Arteaga, Jorge Cabrera, Jing Xu, Swaminathan Sundararaman, and Ming Zhao. 2016. CloudCache: On-demand flash cache management for cloud computing. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 355–369.

[6]

Dulcardo Arteaga and Ming Zhao. 2014. Client-side flash caching for cloud systems. In Proceedings of the 7th ACM International Systems and Storage Conference (SYSTOR’14). 1–11.

Digital Library

[7]

Berk Atikoglu, Yuehai Xu, Eitan Frachtenberg, Song Jiang, and Mike Paleczny. 2012. Workload analysis of a large-scale key-value store. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’12). 53–64.

Digital Library

[8]

Doug Beaver, Sanjeev Kumar, Harry C. Li, Jason Sobel, and Peter Vajgel. 2010. Finding a needle in Haystack: Facebook’s photo storage. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI’10). 47–60.

[9]

Medha Bhadkamkar, Jorge Guerra, Luis Useche, Sam Burnett, Jason Liptak, Raju Rangaswami, and Vagelis Hristidis. 2009. BORG: Block-reORGanization for self-optimizing storage systems. In Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST’09). 183–196.

[10]

Yu Cai, Yixin Luo, Erich F. Haratsch, Ken Mai, and Onur Mutlu. 2015. Data retention in MLC NAND flash memory: Characterization, optimization, and recovery. In Proceedings of the 21st IEEE International Symposium on High Performance Computer Architecture (HPCA’15). IEEE, 551–563.

[11]

Zhichao Cao, Siying Dong, Sagar Vemuri, and David H. C. Du. 2020. Characterizing, modeling, and benchmarking RocksDB key-value workloads at Facebook. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 209–223.

Digital Library

[12]

Jeremy C. W. Chan, Qian Ding, Patrick P. C. Lee, and Helen H. W. Chan. 2014. Parity logging with reserved space: Towards efficient updates and recovery in erasure-coded clustered storage. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST’14). 163–176.

[13]

Tzi-cker Chiueh, Weafon Tsao, Hou-Chiang Sun, Ting-Fang Chien, An-Nan Chang, and Cheng-Ding Chen. 2014. Software orchestrated flash array. In Proceedings of the 7th ACM International Systems and Storage Conference (SYSTOR’14). 1–11.

Digital Library

[14]

Peter Desnoyers. 2012. Analytic modeling of SSD write performance. In Proceedings of the 5th ACM International Systems and Storage Conference (SYSTOR’12). 1–10.

Digital Library

[15]

Shujie Han, Patrick P. C. Lee, Fan Xu, Yi Liu, Cheng He, and Jiongzhou Liu. 2021. An in-depth study of correlated failures in production SSD-based data centers. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST’21). 417–429.

[16]

Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2016. Slacker: Fast distribution with lazy docker containers. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 181–195.

[17]

Jun He, Sudarsun Kannan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2017. The unwritten contract of solid state drives. In Proceedings of the 12th ACM European Conference on Computer Systems (EuroSys’17). 127–144.

Digital Library

[18]

W. W. Hsu and A. J. Smith. 2003. Characteristics of I/O traffic in personal computer and server workloads. IBM Syst. J. 42, 2 (2003), 347–372.

Digital Library

[19]

Swaroop Kavalanekar, Bruce Worthington, Qi Zhang, and Vishal Sharda. 2008. Characterization of storage workload traces from production windows servers. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’08). 119–128.

[20]

Chunghan Lee, Tatsuo Kumano, Tatsuma Matsuki, Hiroshi Endo, Naoto Fukumoto, and Mariko Sugawara. 2017. Understanding storage traffic characteristics on enterprise virtual desktop infrastructure. In Proceedings of the 10th ACM International Systems and Storage Conference (SYSTOR’17). 1–11.

Digital Library

[21]

Huiba Li, Yiming Zhang, Dongsheng Li, Zhiming Zhang, Shengyun Liu, Peng Huang, Zheng Qin, Kai Chen, and Yongqiang Xiong. 2019. URSA: Hybrid block storage for cloud-scale virtual disks. In Proceedings of the 14th ACM European Conference on Computer Systems (EuroSys’19). 1–17.

Digital Library

[22]

Jinhong Li, Qiuping Wang, Patrick P. C. Lee, and Chao Shi. 2020. An in-depth analysis of cloud block storage workloads in large scale production. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’20). 37–47.

[23]

Qiao Li, Liang Shi, Chun Jason Xue, Kaijie Wu, Cheng Ji, Qingfeng Zhuge, and Edwin H.-M. Sha. 2016. Access characteristic guided read and write cost regulation for performance improvement on flash memory. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST’16). 125–132.

Digital Library

[24]

Ren Shuo Liu, Chia Lin Yang, and Wei Wu. 2012. Optimizing NAND flash-based SSDs via retention relaxation. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). 1–11.

[25]

Shuyang Liu, Shucheng Wang, Qiang Cao, Ziyi Lu, Hong Jiang, Jie Yao, Yuanyuan Dong, and Puyuan Yang. 2019. Analysis of and optimization for write-dominated hybrid storage nodes in cloud. In Proceedings of ACM Symposium on Cloud Computing (SoCC’19). 403–415.

Digital Library

[26]

Zaoxing Liu, Zhihao Bai, Zhenming Liu, Xiaozhou Li, Changhoon Kim, Vladimir Braverman, Xin Jin, and Ion Stoica. 2019. DistCache: Provable load balancing for large-scale storage systems with distributed caching. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST’19). 143–157.

[27]

Stathis Maneas, Kaveh Mahdaviani, Tim Emami, and Bianca Schroeder. 2020. A study of SSD reliability in large scale enterprise storage deployments. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 137–149.

Digital Library

[28]

Dutch T. Meyer, Gitika Aggarwal, Brendan Cully, Geoffrey Lefebvre, Michael J. Feeley, Norman C. Hutchinson, and Andrew Warfield. 2008. Parallax: Virtual disks for virtual machines. In Proceedings of the 3rd ACM European Conference on Computer Systems (EuroSys’08). 41–54.

Digital Library

[29]

James Mickens, Edmund B. Nightingale, Jeremy Elson, Krishna Nareddy, Darren Gehring, Bin Fan, Asim Kadav, Vijay Chidambaram, and Osama Khan. 2014. Blizzard: Fast, cloud-scale block storage for cloud-oblivious applications. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI’14). 257–273.

[30]

Microsoft. 2022. MSR Cambridge Traces. Retrieved from http://iotta.snia.org/traces/388.

[31]

Changwoo Min, Kangnyeon Kim, Hyunjin Cho, Sang-Won Lee, and Young Ik Eom. 2012. SFS: Random write considered harmful in solid state drives. In Proceedings of the 10th USENIX Conference on File and Storage Technologies (FAST’12). 1–16.

[32]

Asit K. Mishra, Joseph L. Hellerstein, Walfredo Cirne, and Chita R. Das. 2010. Towards characterizing cloud backend workloads: Insights from Google compute clusters. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’10). 34–41.

Digital Library

[33]

Dushyanth Narayanan, Austin Donnelly, and Antony Rowstron. 2008. Write off-loading: Practical Power Management for Enterprise Storage. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST’08). 253–267.

Digital Library

[34]

Alma Riska and Erik Riedel. 2006. Disk drive level workload characterization. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’06). 97–102.

[35]

Mendel Rosenblum and John K. Ousterhout. 1992. The design and implementation of a log-structured file system. ACM Trans. Comput. Syst. 10, 1 (1992), 26–52.

Digital Library

[36]

Mohit Saxena, Michael M. Swift, and Yiying Zhang. 2012. FlashTier: A lightweight, consistent and durable storage cache. In Proceedings of the 7th ACM European Conference on Computer Systems (EuroSys’12). 267–280.

Digital Library

[37]

Gokul Soundararajan, Vijayan Prabhakaran, Mahesh Balakrishnan, and Ted Wobber. 2010. Extending SSD lifetimes with disk-based write caches. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). 101–114.

[38]

C. Spearman. 1987. The proof and measurement of association between two things. Amer. J. Psychol. 100, 3/4 (1987), 441–471.

[39]

Mojtaba Tarihi, Hossein Asadi, and Hamid Sarbazi-Azad. 2015. DiskAccel: Accelerating disk-based experiments by representative sampling. In Proceedings of the ACM Special Interest Group for the Computer Performance Evaluation Community (SIGMETRICS’15). 297–308.

Digital Library

[40]

Tencent. 2022. Tencent Block Storage. Retrieved from http://iotta.snia.org/traces/27917.

[41]

Akshat Verma, Ricardo Koller, Luis Useche, and Raju Rangaswami. 2010. SRCMap: Energy proportional storage using dynamic consolidation. In Proceedings of the 8th USENIX Conference on File and Storage Technologies (FAST’10). 267–280.

[42]

Muhammad Wajahat, Aditya Yele, Tyler Estro, Anshul Gandhi, and Erez Zadok. 2019. Distribution fitting and performance modeling for storage traces. In Proceedings of the 27th IEEE International Symposium on the Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS’19). IEEE, 138–151.

[43]

Carl A. Waldspurger, Nohhyun Park, Alexander Garthwaite, and Irfan Ahmad. 2015. Efficient MRC construction with SHARDS. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST’15). 95–110.

Digital Library

[44]

Hua Wang, Xinbo Yi, Ping Huang, Bin Cheng, and Ke Zhou. 2018. Efficient SSD caching by avoiding unnecessary writes using machine learning. In Proceedings of the 47th ACM International Conference on Parallel Processing (ICPP’18). 1–10.

Digital Library

[45]

Qiuping Wang, Jinhong Li, Patrick P. C. Lee, Tao Ouyang, Chao Shi, and Lilong Huang. 2022. Separating data via block invalidation time inference for write amplification reduction in log-structured storage. In Proceedings of the 20th USENIX Conference on File and Storage Technologies (FAST’22). 429–444.

[46]

Shucheng Wang, Ziyi Lu, Qiang Cao, Hong Jiang, Jie Yao, Yuanyuan Dong, and Puyuan Yang. 2020. BCW: Buffer-controlled writes to HDDs for SSD-HDD hybrid storage server. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST’20). 253–266.

[47]

Jake Wires, Stephen Ingram, Zachary Drudi, Nicholas J. A. Harvey, and Andrew Warfield. 2014. Characterizing storage workloads with counter stacks. In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI’14). 335–349.

[48]

Erci Xu, Mai Zheng, Feng Qin, Yikang Xu, and Jiesheng Wu. 2019. Lessons and actions: What we learned from 10K SSD-related storage system failures. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC’19). 961–976.

[49]

Gala Yadgar, Moshe Gabel, Shehbaz Jaffer, and Bianca Schroeder. 2021. SSD-based workload characteristics and their performance implications. ACM Trans. Storage 17, 1 (2021), 1–26.

Digital Library

[50]

Jing Yang, Shuyi Pei, and Qing Yang. 2019. WARCIP: Write amplification reduction by clustering I/O pages. In Proceedings of the 12th ACM International Systems and Storage Conference (SYSTOR’19). 155–166.

Digital Library

[51]

Juncheng Yang, Yao Yue, and K. V. Rashmi. 2020. A large scale analysis of hundreds of in-memory cache clusters at Twitter. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI’20). 191–208.

[52]

Yu Zhang, Ping Huang, Ke Zhou, Hua Wang, Jianying Hu, Yongguang Ji, and Bin Cheng. 2020. OSCA: An online-model based cache allocation scheme in cloud block storage systems. In Proceedings of USENIX Annual Technical Conference (USENIX ATC’20). 785–798.

[53]

Yiming Zhang, Huiba Li, Shengyun Liu, Jiawei Xu, and Guangtao Xue. 2020. PBS: An efficient erasure-coded block storage system based on speculative partial writes. ACM Trans. Storage 16, 1 (2020), 1–25.

Digital Library

[54]

Deng Zhou, Wen Pan, Wei Wang, and Tao Xie. 2015. I/O characteristics of smartphone applications and their implications for eMMC design. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC’15). 12–21.

Digital Library

Cited By

Yao XLi QLin KGan XZhang JGao CShen ZXu QYang CXue J(2024)Extremely-Compressed SSDs with I/O Behavior PredictionACM Transactions on Storage10.1145/367704420:4(1-38)Online publication date: 16-Jul-2024
https://dl.acm.org/doi/10.1145/3677044
Liu YYuan XLi HTan ZHuang JXiao JLi WMo T(2024)SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3664816Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3664816
Zhang PLiu MSong XCao DGao ZNie L(2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3656045
Show More Cited By

Index Terms

An In-depth Comparative Analysis of Cloud Block Storage Workloads: Findings and Implications
1. Information systems
  1. Information storage systems
    1. Storage architectures
      1. Cloud based storage
    2. Storage management

Recommendations

A machine learning based write policy for SSD cache in cloud block storage
DATE '20: Proceedings of the 23rd Conference on Design, Automation and Test in Europe

Nowadays, SSD cache plays an important role in cloud storage systems. The associated write policy, which enforces an admission control policy regarding filling data into the cache, has a significant impact on the performance of the cache system and the ...
A Contract-aware and Cost-effective LSM Store for Cloud Storage with Low Latency Spikes
Cloud storage is gaining popularity because features such as pay-as-you-go significantly reduce storage costs. However, the community has not sufficiently explored its contract model and latency characteristics. As LSM-Tree-based key-value stores (LSM ...
Block storage scheduling based on SLA in cloud storage systems
EDB '16: Proceedings of the Sixth International Conference on Emerging Databases: Technologies, Applications, and Theory

According to the growth of the utilization of cloud computing, the required storage capacity is also increasing. The cloud computing services can allocate immediately storage volume according to user requirements, and cloud storage should be scalable. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Storage

ACM Transactions on Storage Volume 19, Issue 2

May 2023

269 pages

ISSN:1553-3077

EISSN:1553-3093

DOI:10.1145/3585541

Editor:
Erez Zadok
Stony Brook University, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 March 2023

Online AM: 22 November 2022

Accepted: 15 November 2022

Revised: 01 September 2022

Received: 10 February 2022

Published in TOS Volume 19, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Alibaba Innovation Research (AIR) program and the Research Matching Grant Scheme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
832
Total Downloads

Downloads (Last 12 months)452
Downloads (Last 6 weeks)54

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yao XLi QLin KGan XZhang JGao CShen ZXu QYang CXue J(2024)Extremely-Compressed SSDs with I/O Behavior PredictionACM Transactions on Storage10.1145/367704420:4(1-38)Online publication date: 16-Jul-2024
https://dl.acm.org/doi/10.1145/3677044
Liu YYuan XLi HTan ZHuang JXiao JLi WMo T(2024)SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3664816Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3664816
Zhang PLiu MSong XCao DGao ZNie L(2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3656045
Hsu WLin H(2024)Context-detail-aware United Network for Single Image DerainingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363940720:5(1-18)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3639407
Zhou YWang FShi ZFeng D(2024)The Static Allocation is Not a Static: Optimizing SSD Address Allocation Through Boosting Static PolicyIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.340736735:8(1373-1386)Online publication date: Aug-2024
https://doi.org/10.1109/TPDS.2024.3407367
Song PGuo DYang XTang SWang M(2024)Emotional Video Captioning With Vision-Based Emotion Interpretation NetworkIEEE Transactions on Image Processing10.1109/TIP.2024.335904533(1122-1135)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3359045
Zhou HFeng DHu YWang WHuang H(2024)CoRD: Combining Raid and Delta for Fast Partial Updates in Erasure-Coded Storage ClustersProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.1109/SC41406.2024.00113(1-14)Online publication date: 17-Nov-2024
https://dl.acm.org/doi/10.1109/SC41406.2024.00113
Bhattacharjee SSharma HChoudhury TAbdelmoniem A(2024)Leveraging chaos for enhancing encryption and compression in large cloud data transfersThe Journal of Supercomputing10.1007/s11227-024-05906-380:9(11923-11957)Online publication date: 4-Feb-2024
https://dl.acm.org/doi/10.1007/s11227-024-05906-3
Azadiabad SKhendek F(2024)Dependability of Network Services in the Context of NFV: A Taxonomy and State of the Art ClassificationJournal of Network and Systems Management10.1007/s10922-024-09810-232:2Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1007/s10922-024-09810-2
Cai ZTang CLi MTrahay FLi JSha ZWu JYang FLiao J(2023)Re-aligning Across-page Requests for Flash-based Solid-state DrivesProceedings of the 52nd International Conference on Parallel Processing10.1145/3605573.3605652(736-745)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3605573.3605652
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents