research-article

Open access

King louie: reproducible availability benchmarking of cloud-hosted DBMS

Authors:

Daniel Seybold,

Jörg DomaschkaAuthors Info & Claims

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

Pages 144 - 153

https://doi.org/10.1145/3341105.3373968

Published: 30 March 2020 Publication History

Abstract

Cloud resources have become a preferred operational model distributed Database Management Systems (DBMS) by offering the elasticity and virtually unlimited scalability, but increase the risk of failures with increasing cluster sizes. While distributed DBMS provide high-availability mechanisms, it is currently an open research question to what extent they are able to provide availability and performance guarantees in case of cloud resource failures. Especially as existing DBMS benchmarks do not consider availability. We present a comprehensive methodology for evaluating the availability of distributed DBMS in case of cloud resource failures. Based on this methodology, we introduce a novel framework that automates the full evaluation process, including the failure injection, and emphasizes reproducibility. The framework is validated by 16 diverse availability evaluations. The results show that distributed DBMS are not necessary available even if sufficient replicas are available and clients can experience significant downtimes.

References

[1]

Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A View of Cloud Computing. Commun. ACM 53, 4 (April 2010), 50--58.

Digital Library

[2]

Algirdas Avizienis, J-C Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing 1, 1 (2004), 11--33.

Digital Library

[3]

D. Baur, D. Seybold, F. Griesinger, H. Masata, and J. Domaschka. 2018. A Provider-Agnostic Approach to Multi-cloud Orchestration Using a Constraint Language. In CCGRID. 173--182.

[4]

Daniel Baur, Daniel Seybold, Frank Griesinger, Athanasios Tsitsipas, Christopher B Hauser, and Jörg Domaschka. 2015. Cloud Orchestration Features: Are Tools Fit for Purpose?. In IEEE/ACM UCC.

[5]

David Bermbach. 2014. Benchmarking eventually consistent distributed storage systems. KIT Scientific Publishing Karlsruhe.

[6]

David Bermbach, Erik Wittern, and Stefan Tai. 2017. Cloud service benchmarking. Springer.

[7]

Lexi Brent and Alan Fekete. 2019. A Versatile Framework for Painless Benchmarking of Database Management Systems. In Australasian Database Conference. Springer, 45--56.

Digital Library

[8]

Eric Brewer. 2012. CAP twelve years later: How the "rules" have changed. Computer (2012).

[9]

Eric A Brewer. 2000. Towards robust distributed systems. In PODC.

[10]

Brian F Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. In ACM SoCC.

[11]

Ali Davoudian, Liu Chen, and Mengchi Liu. 2018. A Survey on NoSQL Stores. ACM Computing Surveys (CSUR) 51, 2 (2018), 40.

Digital Library

[12]

Jörg Domaschka, Christopher B Hauser, and Benjamin Erb. 2014. Reliability and availability properties of distributed database systems. In Enterprise Distributed Object Computing Conference (EDOC), 2014 IEEE 18th International. IEEE, 226--233.

Digital Library

[13]

Yu Gao, Wensheng Dou, Feng Qin, Chushu Gao, Dong Wang, Jun Wei, Ruirui Huang, Li Zhou, and Yongming Wu. 2018. An empirical study on crash recovery bugs in large-scale distributed systems. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 539--550.

Digital Library

[14]

Anne Geraci, Freny Katki, Louise McMonegal, Bennett Meyer, John Lane, Paul Wilson, Jane Radatz, Mary Yee, Hugh Porteous, and Fredrick Springsteel. 1991. IEEE standard computer dictionary: Compilation of IEEE standard computer glossaries. IEEE Press.

Digital Library

[15]

Jim Gray. 1992. Benchmark handbook: for database and transaction processing systems.

[16]

Haryadi S Gunawi, Mingzhe Hao, Riza O Suminto, Agung Laksono, Anang D Satria, Jeffry Adityatama, and Kurnia J Eliazar. 2016. Why does the cloud stop computing?: Lessons from hundreds of service outages. In Proceedings of the Seventh ACM Symposium on Cloud Computing. ACM, 1--16.

Digital Library

[17]

Abdeltawab Hendawi, Jayant Gupta, Liu Jiayi, Ankur Teredesai, Ramakrishnan Naveen, Shah Mohak, and Mohamed Ali. 2018. Distributed NoSQL Data Stores: Performance Analysis and a Case Study. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 1937--1944.

[18]

Victor Heorhiadi, Shriram Rajagopalan, Hani Jamjoom, Michael K Reiter, and Vyas Sekar. 2016. Gremlin: Systematic resilience testing of microservices. In 2016 IEEE 36th International Conference on Distributed Computing Systems. IEEE.

[19]

Murtadha AI Hubail, Ali Alsuliman, Michael Blow, Michael Carey, Dmitry Lychagin, Ian Maxon, and Till Westmann. 2019. Couchbase Analytics: NoETL for Scalable NoSQL Data Analysis. Proc. VLDB Endow. 12, 12 (Aug. 2019), 2275--2286.

Digital Library

[20]

Ioannis Konstantinou, Evangelos Angelou, Christina Boumpouka, Dimitrios Tsoumakos, and Nectarios Koziris. 2011. On the elasticity of NoSQL databases over cloud management platforms. In Proceedings of the 20th ACM international conference on Information and knowledge management. ACM, 2385--2388.

Digital Library

[21]

Jörn Kuhlenkamp, Markus Klems, and Oliver Röss. 2014. Benchmarking scalability and elasticity of distributed database systems. Proceedings of the VLDB Endowment 7, 12 (2014), 1219--1230.

Digital Library

[22]

Somnath Mazumdar, Daniel Seybold, Kyriakos Kritikos, and Yiannis Verginadis. 2019. A survey on data storage and placement methodologies for Cloud-Big Data ecosystem. Journal of Big Data 6, 1 (2019), 15.

[23]

A. V. Papadopoulos, L. Versluis, A. Bauer, N. Herbst, J. Von Kistowski, A. Ali-eldin, C. Abad, J. N. Amaral, P. Tůma, and A. Iosup. 2019. Methodological Principles for Reproducible Performance Evaluation in Cloud Computing. IEEE Transactions on Software Engineering (2019), 1--1.

[24]

Kia Rahmani, Kartik Nagar, Benjamin Delaware, and Suresh Jagannathan. 2019. CLOTHO: Directed Test Generation for Weakly Consistent Database Systems. Proc. ACM Program. Lang. 3, OOPSLA, Article 117 (Oct. 2019), 28 pages.

Digital Library

[25]

Vincent Reniers, Dimitri Van Landuyt, Ansar Rafique, and Wouter Joosen. 2017. On the state of nosql benchmarks. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion. ACM, 107--112.

Digital Library

[26]

Sherif Sakr. 2014. Cloud-hosted databases: technologies, challenges and opportunities. Cluster Computing 17, 2 (2014), 487--502.

[27]

Daniel Seybold and Jörg Domaschka. 2017. Is Distributed Database Evaluation Cloud-Ready?. In ADBIS. Springer, 100--108.

[28]

Daniel Seybold, Christopher B Hauser, Simon Volpert, and Jörg Domaschka. 2017. Gibbon: An Availability Evaluation Framework for Distributed Databases. In OTM Confederated International Conferences "On the Move to Meaningful Internet Systems". Springer, 31--49.

[29]

Daniel Seybold, Moritz Keppler, Daniel Gründler, and Jörg Domaschka. 2019. Mowgli: Finding your way in the DBMS jungle. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (ICPE '19). ACM, 321--332.

Digital Library

[30]

Daniel Seybold, Nicolas Wagner, Benjamin Erb, and Jörg Domaschka. 2016. Is elasticity of scalable databases a Myth?. In IEEE Big Data.

[31]

Daniel Seybold, Stefan Wesner, and Jörg Domaschka. 2019. King Louie: DBMS Availability Evaluation Data Sets.

[32]

Ariel Tseitlin. 2013. The Antifragile Organization. Commun. ACM 56, 8 (2013).

[33]

Werner Vogels. 2009. Eventually Consistent. Commun. ACM 52, 1 (Jan. 2009), 5.

Digital Library

[34]

Ming Zhong, Kai Shen, and Joel Seiferas. 2008. Replication Degree Customization for High Availability. In EuroSys.

Cited By

Erdelt P(2023)A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and ToolsPerformance Evaluation and Benchmarking10.1007/978-3-031-68031-1_9(124-142)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-68031-1_9
Leznik MGrohmann JKliche NBauer ASeybold DEismann SKounev SDomaschka JFeng DBecker SHerbst NLeitner PPapadopoulos A(2022)Same, Same, but DissimilarProceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering10.1145/3489525.3511699(89-96)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3489525.3511699
Pakrijauskas KMažeika D(2022)Investigation of Stateful Microservice Availability During Failover2022 8th International Conference on Control, Decision and Information Technologies (CoDIT)10.1109/CoDIT55151.2022.9804162(286-290)Online publication date: 17-May-2022
https://doi.org/10.1109/CoDIT55151.2022.9804162
Show More Cited By

Index Terms

King louie: reproducible availability benchmarking of cloud-hosted DBMS
1. Computer systems organization
  1. Architectures
    1. Distributed architectures
      1. Cloud computing
2. Information systems
  1. Data management systems
    1. Database administration
      1. Database performance evaluation
      2. Database utilities and tools

Recommendations

Mowgli: Finding Your Way in the DBMS Jungle
ICPE '19: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

Big Data and IoT applications require highly-scalable database management system (DBMS), preferably operated in the cloud to ensure scalability also on the resource level. As the number of existing distributed DBMS is extensive, the selection and ...
Gibbon: An Availability Evaluation Framework for Distributed Databases
On the Move to Meaningful Internet Systems. OTM 2017 Conferences
Abstract
Driven by new application domains, the database management systems (DBMSs) landscape has significantly evolved from single node DBMS to distributed database management systems (DDBMSs). In parallel, cloud computing became the preferred solution to ...
Adaptive checkpointing with reliable storage in cloud environment

Cloud computing has recently gained popularity as a resource platform for on-demand, high availability and high scalability access to resources, while offering dynamic flexible infrastructures and QoS (Quality of Services) guaranteed services. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

March 2020

2348 pages

ISBN:9781450368667

DOI:10.1145/3341105

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Tomas Cerny
Baylor University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Alessio Bechini
University of Pisa, Italy

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SAC '20

Sponsor:

SIGAPP

SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing

March 30 - April 3, 2020

Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
413
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)14

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Erdelt P(2023)A Cloud-Native Adoption of Classical DBMS Performance Benchmarks and ToolsPerformance Evaluation and Benchmarking10.1007/978-3-031-68031-1_9(124-142)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1007/978-3-031-68031-1_9
Leznik MGrohmann JKliche NBauer ASeybold DEismann SKounev SDomaschka JFeng DBecker SHerbst NLeitner PPapadopoulos A(2022)Same, Same, but DissimilarProceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering10.1145/3489525.3511699(89-96)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3489525.3511699
Pakrijauskas KMažeika D(2022)Investigation of Stateful Microservice Availability During Failover2022 8th International Conference on Control, Decision and Information Technologies (CoDIT)10.1109/CoDIT55151.2022.9804162(286-290)Online publication date: 17-May-2022
https://doi.org/10.1109/CoDIT55151.2022.9804162
Seybold DDomaschka JZhang KGherbi AVenkatasubramanian NVeiga L(2021)Benchmarking-as-a-service for cloud-hosted DBMSProceedings of the 22nd International Middleware Conference: Demos and Posters10.1145/3491086.3492473(12-13)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.1145/3491086.3492473
Erdelt P(2021)Orchestrating DBMS Benchmarking in the Cloud with KubernetesPerformance Evaluation and Benchmarking10.1007/978-3-030-94437-7_6(81-97)Online publication date: 16-Aug-2021
https://dl.acm.org/doi/10.1007/978-3-030-94437-7_6
Chereja IHahn SMatei O(2021)Benchmarking NewSQL Cloud-Native or Cloud-Hosted DatabasesSoftware Engineering Application in Informatics10.1007/978-3-030-90318-3_26(296-310)Online publication date: 17-Nov-2021
https://doi.org/10.1007/978-3-030-90318-3_26
Domaschka JVolpert SSeybold D(2020)Hathi: An MCDM-based Approach to Capacity Planning for Cloud-hosted DBMS2020 IEEE/ACM 13th International Conference on Utility and Cloud Computing (UCC)10.1109/UCC48980.2020.00033(143-154)Online publication date: Dec-2020
https://doi.org/10.1109/UCC48980.2020.00033

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents