research-article

Container-aware I/O stack: bridging the gap between container storage drivers and solid state devices

Authors:

Hai JinAuthors Info & Claims

VEE 2022: Proceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

Pages 18 - 30

https://doi.org/10.1145/3516807.3516818

Published: 25 February 2022 Publication History

Abstract

Solid State Devices (SSDs) have been widely adopted in containerized cloud platforms as they provide parallel and high-speed data accesses for critical data-intensive applications. Unfortunately, the I/O stack of the physical host overlooks the layered and independent nature of containers, thus I/O operations require expensive file redirect (between the storage driver, Overlay2/EXT4, and the virtual file system, VFS) and are scheduled sequentially. Moreover, containers suffer from significant I/O contention as resources at the native file system are shared between them. This paper presents a Container-aware I/O stack (CAST). CAST is made up of Layer-aware VFS (LaVFS) and Container-aware Native File System (CaFS). LaVFS locates files based on layer information and enables simultaneous Copy-on-Write (CoW) operations and thus avoids the overhead of searching and modifying files. CaFS, on the other hand, provides contention-free access by designing fine-grain resource allocation at the native file system. Experimental results using a NVMe SSD with micro-benchmarks and real-world applications show that CAST achieves 216%-219% (38%-98%, respectively) improvement over the original I/O stack.

References

[1]

Sungyong Ahn, Kwanghyun La, and Jihong Kim. 2016. Improving I/O Resource Sharing of Linux Cgroup for NVMe SSDs on Multi-core Systems. In Proceedings of the USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage’16). 1–5.

[2]

Srivatsa S. Bhat, Rasha Eqbal, Austin T. Clements, M. Frans Kaashoek, and Nickolai Zeldovich. 2017. Scaling a File System to Many Cores Using an Operation Log. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP’17). 69–86. https://doi.org/10.1145/3132747.3132779

Digital Library

[3]

Min Changwoo. 2021. FxMark. https://github.com/sslab-gatech/fxmark

[4]

Min Changwoo, Kashyap Sanidhya, Maass Steffen, Kang Woonhak, and Kim Taesoo. 2016. Understanding Manycore Scalability of File Systems. In Proceedings of the USENIX Annual Technical Conference (ATC’16). 71–85.

[5]

Google Cloud. 2021. Kubernetes. https://kubernetes.io/

[6]

Docker Docs. 2021. Overlay2. https://docs.docker.com/storage/storagedriver/overlayfs-driver/

[7]

Hao Fan, Shengwei Bian, Song Wu, Song Jiang, Shadi Ibrahim, and Hai Jin. 2021. Gear: Enable Efficient Container Storage and Deployment with a New Image Format. In Proceedings of the IEEE International Conference on Distributed Computing Systems (ICDCS’21). 115–125. https://doi.org/10.1109/ICDCS51616.2021.00020

[8]

Hao Fan, Song Wu, Shadi Ibrahim, Ximing Chen, Hai Jin, Jiang Xiao, and Haibing Guan. 2019. NCQ-Aware I/O Scheduling for Conventional Solid State Drives. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS’19). 523–532. https://doi.org/10.1109/IPDPS.2019.00062

[9]

Fan Guo, Yongkun Li, Min Lv, Yinlong Xu, and John C. S. Lui. 2019. HP-Mapper: A High Performance Storage Driver for Docker Containers. In Proceedings of the ACM Symposium on Cloud Computing (SOCC’19). 325–336. https://doi.org/10.1145/3357223.3362718

Digital Library

[10]

Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2016. Slacker: Fast Distribution with Lazy Docker Containers. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’16). 181–195.

[11]

Docker Inc. 2021. Docker. https://www.docker.com/

[12]

Junbin Kang, Benlong Zhang, Tianyu Wo, Chunming Hu, and Jinpeng Huai. 2014. MultiLanes: Providing Virtualized Storage for OS-Level Virtualization on Many Cores. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’14). 317–329.

[13]

Junbin Kang, Benlong Zhang, Tianyu Wo, Weiren Yu, Lian Du, Shuai Ma, and Jinpeng Huai. 2015. SpanFS: A Scalable File System on Fast Storage Devices. In Proceedings of the USENIX Annual Technical Conference (ATC’15). 249–261.

Digital Library

[14]

Jongseok Kim, Cassiano Campes, Joo-Young Hwang, Jinkyu Jeong, and Euiseong Seo. 2021. Z-Journal: Scalable Per-Core Journaling. In Proceedings of the USENIX Annual Technical Conference (ATC’21). 893–906.

[15]

Chu Li, Dan Feng, Yu Hua, and Fang Wang. 2016. Improving RAID Performance Using an Endurable SSD Cache. In Proceedings of the International Conference on Parallel Processing (ICPP’16). 396–405. https://doi.org/10.1109/ICPP.2016.52

[16]

Feifei Li. 2019. Cloud-Native Database Systems at Alibaba: Opportunities and Challenges. Proceedings of the VLDB Endowment, 12, 12 (2019), 2263–2272. https://doi.org/10.14778/3352063.3352141

Digital Library

[17]

Huiba Li, Yifan Yuan, Rui Du, Kai Ma, Lanzheng Liu, and Windsor Hsu. 2020. DADI: Block-Level Image Service for Agile and Elastic Application Deployment. In Proceedings of the USENIX Annual Technical Conference (ATC’20). 727–740.

[18]

Xiaojian Liao, Youyou Lu, Erci Xu, and Jiwu Shu. 2021. Max: A Multicore-Accelerated File System for Flash Storage. In Proceedings of the USENIX Annual Technical Conference (ATC’21). 877–891.

[19]

Sean McDaniel, Stephen Herbein, and Michela Taufer. 2015. A Two-Tiered Approach to I/O Quality of Service in Docker Containers. In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER’15). 490–491. https://doi.org/10.1109/CLUSTER.2015.77

Digital Library

[20]

Daejun Park and Dongkun Shin. 2017. iJournaling: Fine-Grained Journaling for Improving the Latency of Fsync System Call. In Proceedings of the USENIX Annual Technical Conference (ATC’17). 787–798.

[21]

Vijayan Prabhakaran, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2005. Analysis and Evolution of Journaling File Systems. In Proceedings of the USENIX Annual Technical Conference (ATC’05). 196–215.

[22]

Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-tree Filesystem. ACM Transactions on Storage (TOS), 9, 3 (2013), 1–32. https://doi.org/10.1145/2501620.2501623

Digital Library

[23]

Kai Shen and Stan Park. 2013. FlashFQ: A Fair Queueing I/O Scheduler for Flash-Based SSDs. In Proceedings of the USENIX Annual Technical Conference (ATC’13). 67–78.

[24]

Yongseok Son, Sunggon Kim, Heon Y. Yeom, and Hyuck Han. 2018. High-Performance Transaction Processing in Journaling File Systems. In Proceedings of the USENIX Conference on File and Storage Technologies (FAST’18). 227–240.

[25]

Yu Sun, Jiaxin Lei, Seunghee Shin, and Hui Lu. 2020. Baoverlay: A Block-Accessible Overlay File System for Fast and Efficient Container Storage. In Proceedings of the ACM Symposium on Cloud Computing (SOCC’20). 90–104. https://doi.org/10.1145/3419111.3421291

Digital Library

[26]

Sysdig. 2021. 2021 Container Security and Usage Report. https://sysdig.com/wp-content/uploads/2021-container-security-and-usage-report.pdf

[27]

Vasily Tarasov, Lukas Rupprecht, Dimitris Skourtis, Amit Warke, Dean Hildebrand, Mohamed Mohamed, Nagapramod Mandagere, Wenji Li, Raju Rangaswami, and Ming Zhao. 2017. In Search of the Ideal Storage Configuration for Docker Containers. In Proceedings of the IEEE International Workshops on Foundations and Applications of Self* Systems (FAS*W’17). 199–206.

[28]

Vasily Tarasov, Erez Zadok, and Spencer Shepler. 2016. Filebench: A Flexible Framework for File System Benchmarking. USENIX login, 41, 1 (2016), 6–12.

[29]

Arash Tavakkol, Mohammad Sadrosadati, Saugata Ghose, Jeremie Kim, Yixin Luo, Yaohua Wang, Nika Mansouri Ghiasi, Lois Orosa, Juan Gómez-Luna, and Onur Mutlu. 2018. FLIN: Enabling Fairness and Enhancing Performance in Modern NVMe Solid State Drives. In Proceedings of the ACM/IEEE Annual International Symposium on Computer Architecture (ISCA’18). 397–410. https://doi.org/10.1109/ISCA.2018.00041

Digital Library

[30]

Charles P. Wright, Jay Dave, Puja Gupta, Harikesavan Krishnan, David P. Quigley, Erez Zadok, and Mohammad Nayyer Zubair. 2006. Versatility and Unix Semantics in Namespace Unification. ACM Transactions on Storage (TOS), 2, 1 (2006), 74–105. https://doi.org/10.1145/1138041.1138045

Digital Library

[31]

Xingbo Wu, Wenguang Wang, and Song Jiang. 2015. Totalcow: Unleash the Power of Copy-on-Write for Thin-Provisioned Containers. In Proceedings of the Asia-Pacific Workshop on Systems (APSys’15). 1–7. https://doi.org/10.1145/2797022.2797024

Digital Library

[32]

Qiumin Xu, Manu Awasthi, Krishna T. Malladi, Janki Bhimani, Jingpei Yang, and Murali Annavaram. 2017. Performance Analysis of Containerized Applications on Local and Remote Storage. In Proceedings of the International Conference on Massive Storage Systems and Technology (MSST’17). 1–12.

Cited By

Zhao NLin MAlbahar HPaul AHuan ZAbraham SChen KTarasov VSkourtis DAnwar AButt A(2024)An End-to-end High-performance Deduplication Scheme for Docker Registries and Docker Container Storage SystemsACM Transactions on Storage10.1145/364381920:3(1-35)Online publication date: 6-Jun-2024
https://dl.acm.org/doi/10.1145/3643819
Huang HWang HRao JWu SFan HYu CJin HSuo KPan L(2024)vKernel: Enhancing Container Isolation via Private Code and DataIEEE Transactions on Computers10.1109/TC.2024.338398873:7(1711-1723)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3383988
Huang ZZhang QFan HWu SYu CJin HDeng JGu JTang Z(2024)Multi-Grained Trace Collection, Analysis, and Management of Diverse Container ImagesIEEE Transactions on Computers10.1109/TC.2024.338396673:7(1698-1710)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1109/TC.2024.3383966
Show More Cited By

Index Terms

Container-aware I/O stack: bridging the gap between container storage drivers and solid state devices
1. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        File systems management

Recommendations

An empirical study of redundant array of independent solid-state drives (RAIS)

Solid-state drives (SSD) are popular storage media devices alongside magnetic hard disk drives (HDD). SSD flash chips are packaged in HDD form factors and SSDs are compatible with regular HDD device drivers and I/O buses. This compatibility allows easy ...
RIO: Order-Preserving and CPU-Efficient Remote Storage Access
EuroSys '23: Proceedings of the Eighteenth European Conference on Computer Systems

Modern NVMe SSDs and RDMA networks provide dramatically higher bandwidth and concurrency. Existing networked storage systems (e.g., NVMe over Fabrics) fail to fully exploit these new devices due to inefficient storage ordering guarantees. Severe ...
Improve Effective Capacity and Lifetime of Solid State Drives
NAS '13: Proceedings of the 2013 IEEE Eighth International Conference on Networking, Architecture and Storage

Flash-based SSDs are becoming increasingly popular in modern storage systems, especially in high-performance computing infrastructures. However, several inherent technical limitations still remain to prevent their widespread deployment. One of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VEE 2022: Proceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

February 2022

87 pages

ISBN:9781450392518

DOI:10.1145/3516807

General Chair:
John Criswell
Rochester Institute of Technology, USA
,
Program Chairs:
Dan Williams
Virginia Tech, USA
,
Yubin Xia
Shanghai Jiao Tong University, China

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation of China

Conference

VEE '22

Sponsor:

VEE '22: 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments

March 1, 2022

Virtual, Switzerland

Acceptance Rates

Overall Acceptance Rate 80 of 235 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
464
Total Downloads

Downloads (Last 12 months)93
Downloads (Last 6 weeks)13

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhao NLin MAlbahar HPaul AHuan ZAbraham SChen KTarasov VSkourtis DAnwar AButt A(2024)An End-to-end High-performance Deduplication Scheme for Docker Registries and Docker Container Storage SystemsACM Transactions on Storage10.1145/364381920:3(1-35)Online publication date: 6-Jun-2024
https://dl.acm.org/doi/10.1145/3643819
Huang HWang HRao JWu SFan HYu CJin HSuo KPan L(2024)vKernel: Enhancing Container Isolation via Private Code and DataIEEE Transactions on Computers10.1109/TC.2024.338398873:7(1711-1723)Online publication date: Jul-2024
https://doi.org/10.1109/TC.2024.3383988
Huang ZZhang QFan HWu SYu CJin HDeng JGu JTang Z(2024)Multi-Grained Trace Collection, Analysis, and Management of Diverse Container ImagesIEEE Transactions on Computers10.1109/TC.2024.338396673:7(1698-1710)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1109/TC.2024.3383966
Han H(2023)High-Performance OverlayFS for ContainersJournal of Digital Contents Society10.9728/dcs.2023.24.11.284124:11(2841-2847)Online publication date: 30-Nov-2023
https://doi.org/10.9728/dcs.2023.24.11.2841
Choi KSeo HHan HRyu MKang S(2023)CredsCacheFuture Generation Computer Systems10.1016/j.future.2023.04.027147:C(44-58)Online publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.future.2023.04.027
Wang KWu SLi SHuang ZFan HYu CJin H(2023)Precise control of page cache for containersFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-022-2455-018:2Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1007/s11704-022-2455-0

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents