research-article

Wharf: Sharing Docker Images in a Distributed File System

Authors:

Lukas Rupprecht,

Vasily Tarasov,

Mohamed Mohamed,

Dimitrios Skourtis,

Dean HildebrandAuthors Info & Claims

SoCC '18: Proceedings of the ACM Symposium on Cloud Computing

Pages 174 - 185

https://doi.org/10.1145/3267809.3267836

Published: 11 October 2018 Publication History

Abstract

Container management frameworks, such as Docker, package diverse applications and their complex dependencies in self-contained images, which facilitates application deployment, distribution, and sharing. Currently, Docker employs a shared-nothing storage architecture, i.e. every Docker-enabled host requires its own copy of an image on local storage to create and run containers. This greatly inflates storage utilization, network load, and job completion times in the cluster. In this paper, we investigate the option of storing container images in and serving them from a distributed file system. By sharing images in a distributed storage layer, storage utilization can be reduced and redundant image retrievals from a Docker registry become unnecessary. We introduce Wharf, a middleware to transparently add distributed storage support to Docker. Wharf partitions Docker's runtime state into local and global parts and efficiently synchronizes accesses to the global state. By exploiting the layered structure of Docker images, Wharf minimizes the synchronization overhead. Our experiments show that compared to Docker on local storage, Wharf can speed up image retrievals by up to 12x, has more stable performance, and introduces only a minor overhead when accessing data on distributed storage.

References

[1]

2018. Amazon Elastic Container Service. (2018). https://aws.amazon.com/ecs/.

[2]

2018. AUFS - Another Union Filesystem. (2018). http://aufs.sourceforge.net. Accessed March 2018.

[3]

2018. Azure Container Service. (2018). https://azure.microsoft.com/en-us/services/container-service/.

[4]

2018. CoreOS. (2018). https://coreos.com/.

[5]

2018. Device-mapper snapshot support. (2018). https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt. Accessed March 2018.

[6]

2018. Docker. (2018). https://www.docker.com/.

[7]

2018. Docker Hub. (2018). https://hub.docker.com/.

[8]

2018. Etcd: A distributed, reliable key-value store for the most critical data of a distributed system. (2018). https://coreos.com/etcd/.

[9]

2018. Google Kubernetes Engine. (2018). https://cloud.google.com/kubernetes-engine/.

[10]

2018. Harbor: An enterprise-class container registry server based on Docker Distribution. (2018). http://vmware.github.io/harbor/.

[11]

2018. IBM Cloud Container Service. (2018). https://www.ibm.com/cloud/container-service.

[12]

2018. IBM Spectrum Scale. (2018). https://www.ibm.com/us-en/marketplace/scale-out-file-and-object-storage.

[13]

2018. OpenVZ. (2018). https://openvz.org. Accessed March 2018.

[14]

2018. Overlay Filesystem. (2018). https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt. Accessed March 2018.

[15]

Michael Albrecht, Patrick Donnelly, Peter Bui, and Douglas Thain. 2012. Makeflow: A Portable Abstraction for Data Intensive Computing on Clusters, Clouds, and Grids. In Workshop on Scalable Workflow Enactment Engines and Technologies (SWEET).

Digital Library

[16]

Ali Anwar, Mohamed Mohamed, Vasily Tarasov, Michael Littley, Lukas Rupprecht, Yue Cheng, Nannan Zhao, Dimitrios Skourtis, Amit S Warke, Heiko Ludwig, and others. 2018. Improving Docker Registry Design Based on Production Workload Analysis. In 16th USENIX Conference on File and Storage Technologies (FAST).

Digital Library

[17]

Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, and Douglas Thain. 2011. Work Queue + Python: A Framework for Scalable Scientific Ensemble Applications. In Workshop on Python for High Performance and Scientific Computing (PyHPC).

[18]

Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes. 2016. Borg, Omega, and Kubernetes. ACM Queue 14, 1 (2016).

Digital Library

[19]

Jack Clark. 2014. Google: 'EVERYTHING at Google runs in a container'. (2014). https://www.theregister.co.uk/2014/05/23/google_containerization_two_billion/.

[20]

Lisa Gerhardt, Wahid Bhimji, Shane Canon, Markus Fasel, Doug Jacobsen, Mustafa Mustafa, Jeff Porter, and Vakho Tsulaia. 2017. Shifter: Containers for HPC. Journal of Physics: Conference Series 898, 8 (2017).

[21]

Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2016. Slacker: Fast Distribution with Lazy Docker Containers. In 14th USENIX Conference on File and Storage Technologies (FAST).

Digital Library

[22]

Nick Hazekamp and Douglas Thain. 2017. Makeflow Examples Repository. (2017). http://github.com/cooperative-computing-lab/makeflow-examples.

[23]

Patrick Hunt, Mahadev Konar, Flavio Paiva Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free Coordination for Internet-scale Systems. In USENIX Annual Technical Conference (ATC).

Digital Library

[24]

Wang Kangjin, Yang Yong, Li Ying, Luo Hanmei, and Ma Lin. 2017. FID: A Faster Image Distribution System for Docker Platform. In 2nd IEEE International Workshop on Foundations and Applications of Self* Systems (FAS* W).

[25]

Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. 2017. Singularity: Scientific Containers for Mobility of Compute. PloS One 12, 5 (2017).

[26]

Horacio Andrés Lagar-Cavilla, Joseph Andrew Whitney, Adin Matthew Scannell, Philip Patchin, Stephen M Rumble, Eyal De Lara, Michael Brudno, and Mahadev Satyanarayanan. 2009. SnowFlock: Rapid Virtual Machine Cloning for Cloud Computing. In Proceedings of the 4th ACM European Conference on Computer Systems (EuroSys).

Digital Library

[27]

Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. 2017. My VM is Lighter (and Safer) than your Container. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP).

Digital Library

[28]

Paul Menage. 2007. Adding Generic Process Containers to the Linux Kernel. In Linux Symposium.

[29]

Senthil Nathan, Rahul Ghosh, Tridib Mukherjee, and Krishnaprasad Narayanan. 2017. CoMICon: A Co-operative Management System for Docker Container Images. In Proceedings of the IEEE International Conference on Cloud Engineering (IC2E).

[30]

Bogdan Nicolae, Gabriel Antoniu, Luc Bougé, Diana Moise, and Alexandra Carpen-Amarie. 2011. BlobSeer: Next-generation Data Management for Large Scale Infrastructures. J. Parallel and Distrib. Comput. 71, 2 (2011).

Digital Library

[31]

Bill Nowicki. 1989. Nfs: Network file system protocol specification. Technical Report.

[32]

Chunyi Peng, Minkyong Kim, Zhe Zhang, and Hui Lei. 2012. VDN: Virtual machine image distribution network for cloud data centers. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM).

[33]

Reid Priedhorsky and Tim Randles. 2017. Charliecloud: Unprivileged Containers for User-defined Software Stacks in HPC. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

Digital Library

[34]

Kaveh Razavi, Ana Ion, and Thilo Kielmann. 2014. Squirrel: Scatter Hoarding VM Image Contents on IaaS Compute Nodes. In Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing (HPDC).

Digital Library

[35]

Kaveh Razavi and Thilo Kielmann. 2013. Scalable Virtual Machine Deployment using VM Image Caches. In Proceedings of the ACM/IEEE International Conference on High Performance Computing, Networking, Storage and Analysis (SC).

Digital Library

[36]

Joshua Reich, Oren Laadan, Eli Brosh, Alex Sherman, Vishal Misra, Jason Nieh, and Dan Rubenstein. 2012. VMTorrent: Scalable P2P Virtual Machine Streaming. In Proceedings of the 8th International Conference on emerging Networking EXperiments and Technologies (CoNEXT).

Digital Library

[37]

Ohad Rodeh, Josef Bacik, and Chris Mason. 2013. BTRFS: The Linux B-tree filesystem. ACM Transactions on Storage (TOS) 9, 3 (2013).

Digital Library

[38]

Neil Savage. 2018. Going Serverless. Commun. ACM 61, 2 (2018).

Digital Library

[39]

Stephen Soltesz, Herbert Pötzl, Marc E Fiuczynski, Andy Bavier, and Larry Peterson. 2007. Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors. ACM SIGOPS Operating Systems Review 41, 3 (2007).

Digital Library

[40]

Chunqiang Tang. 2011. FVD: A High-Performance Virtual Machine Image Format for Cloud. In USENIX Annual Technical Conference (ATC).

Digital Library

[41]

Vasily Tarasov, Lukas Rupprecht, Dimitris Skourtis, Amit Warke, Dean Hildebrand, Mohamed Mohamed, Nagapramod Mandagere, Wenji Li, Raju Rangaswami, and Ming Zhao. 2017. In Search of the Ideal Storage Configuration for Docker Containers. In 2nd IEEE International Workshop on Foundations and Applications of Self* Systems (FAS* W).

[42]

Teng Wang, Kathryn Mohror, Adam Moody, Kento Sato, and Weikuan Yu. 2016. An Ephemeral Burst-buffer File System for Scientific Applications. In Proceedings of the ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

Digital Library

Cited By

Song JKim BKwak MLee BSeo EJeong JBagchi SZhang Y(2024)A secure, fast, and resource-efficient serverless platform with function REWINDProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692029(597-613)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692029
Li HZhang ZYuan YDu RMa KLiu LZhang YHsu W(2024)Block-level Image Service for the CloudACM Transactions on Storage10.1145/362067220:1(1-28)Online publication date: 30-Jan-2024
https://dl.acm.org/doi/10.1145/3620672
Huang ZZhang QFan HWu SYu CJin HDeng JGu JTang Z(2024)Multi-Grained Trace Collection, Analysis, and Management of Diverse Container ImagesIEEE Transactions on Computers10.1109/TC.2024.338396673:7(1698-1710)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1109/TC.2024.3383966
Show More Cited By

Index Terms

Wharf: Sharing Docker Images in a Distributed File System
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing
2. Information systems
  1. Information storage systems
    1. Storage architectures
      1. Cloud based storage

Recommendations

CFS: A Distributed File System for Large Scale Container Platforms
SIGMOD '19: Proceedings of the 2019 International Conference on Management of Data

We propose CFS, a distributed file system for large scale container platforms. CFS supports both sequential and random file accesses with optimized storage for both large files and small files, and adopts different replication protocols for different ...
State machine replication in containers managed by Kubernetes

Computer virtualization brought fast resource provisioning to data centers and the deployment of pay-per-use cost models. The system virtualization provided by containers like Docker has improved this flexibility of resource provisioning. Applications ...
AWS EC2 vs. Joyent's Triton: A Comparison of Docker Container-hosting Platforms
ScienceCloud '17: Proceedings of the 8th Workshop on Scientific Cloud Computing

Containers, and in particular Docker, have emerged as promising addition and in some cases, alternative to virtual machines (VMs) for application deployment in cloud environments. Docker containers enable massive scalability and rapid deployment of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '18: Proceedings of the ACM Symposium on Cloud Computing

October 2018

546 pages

ISBN:9781450360111

DOI:10.1145/3267809

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SoCC '18

Sponsor:

SoCC '18: ACM Symposium on Cloud Computing

October 11 - 13, 2018

CA, Carlsbad, USA

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
1,028
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)6

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song JKim BKwak MLee BSeo EJeong JBagchi SZhang Y(2024)A secure, fast, and resource-efficient serverless platform with function REWINDProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692029(597-613)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692029
Li HZhang ZYuan YDu RMa KLiu LZhang YHsu W(2024)Block-level Image Service for the CloudACM Transactions on Storage10.1145/362067220:1(1-28)Online publication date: 30-Jan-2024
https://dl.acm.org/doi/10.1145/3620672
Huang ZZhang QFan HWu SYu CJin HDeng JGu JTang Z(2024)Multi-Grained Trace Collection, Analysis, and Management of Diverse Container ImagesIEEE Transactions on Computers10.1109/TC.2024.338396673:7(1698-1710)Online publication date: 8-Apr-2024
https://dl.acm.org/doi/10.1109/TC.2024.3383966
Makris AKorontanis IPsomakelis ETserpes K(2024)An Efficient Storage Solution for Cloud/Edge Computing Infrastructures : (Invited Paper)2024 IEEE International Conference on Service-Oriented System Engineering (SOSE)10.1109/SOSE62363.2024.00017(92-101)Online publication date: 15-Jul-2024
https://doi.org/10.1109/SOSE62363.2024.00017
Shi YYang YYi CChen BCai J(2024)Toward Online Reliability-Enhanced Microservice Deployment With Layer Sharing in Edge ComputingIEEE Internet of Things Journal10.1109/JIOT.2024.338581611:13(23370-23383)Online publication date: 1-Jul-2024
https://doi.org/10.1109/JIOT.2024.3385816
Feng YShen SWang XXiang QXu HXu CWang W(2024)BREAK: A Holistic Approach for Efficient Container Deployment among Edge CloudsIEEE INFOCOM 2024 - IEEE Conference on Computer Communications10.1109/INFOCOM52122.2024.10621084(1491-1500)Online publication date: 20-May-2024
https://doi.org/10.1109/INFOCOM52122.2024.10621084
Makris APsomakelis EKorontanis ITheodoropoulos TKontopoulos IPateraki MDiou CTserpes K(2024)Edge-driven Docker registry: facilitating XR application deploymentComputing10.1007/s00607-024-01310-0Online publication date: 24-Jun-2024
https://doi.org/10.1007/s00607-024-01310-0
Shi YYang YYi CWang J(2024)Reliability-Enhanced Microservice DeploymentWireless Artificial Intelligent Computing Systems and Applications10.1007/978-3-031-71467-2_40(502-515)Online publication date: 14-Nov-2024
https://doi.org/10.1007/978-3-031-71467-2_40
Liu ZLi JChen CZhang D(2023)Hound: A Parallel Image Distribution System for Cluster Based on DockerJournal of Systems Engineering and Electronics10.23919/JSEE.2023.00010534:4(955-965)Online publication date: Aug-2023
https://doi.org/10.23919/JSEE.2023.000105
Kappes GAnastasiadis S(2023)Diciclo: Flexible User-level Services for Efficient Multitenant IsolationACM Transactions on Computer Systems10.1145/363940442:1-2(1-47)Online publication date: 30-Dec-2023
https://dl.acm.org/doi/10.1145/3639404
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents