research-article

OLTP-Bench: an extensible testbed for benchmarking relational databases

Authors:

Djellel Eddine Difallah,

Carlo Curino, and

Philippe Cudre-MaurouxAuthors Info & Claims

Proceedings of the VLDB Endowment, Volume 7, Issue 4

Pages 277 - 288

https://doi.org/10.14778/2732240.2732246

Published: 01 December 2013 Publication History

Abstract

Benchmarking is an essential aspect of any database management system (DBMS) effort. Despite several recent advancements, such as pre-configured cloud database images and database-as-a-service (DBaaS) offerings, the deployment of a comprehensive testing platform with a diverse set of datasets and workloads is still far from being trivial. In many cases, researchers and developers are limited to a small number of workloads to evaluate the performance characteristics of their work. This is due to the lack of a universal benchmarking infrastructure, and to the difficulty of gaining access to real data and workloads. This results in lots of unnecessary engineering efforts and makes the performance evaluation results difficult to compare. To remedy these problems, we present OLTP-Bench, an extensible "batteries included" DBMS benchmarking testbed. The key contributions of OLTP-Bench are its ease of use and extensibility, support for tight control of transaction mixtures, request rates, and access distributions over time, as well as the ability to support all major DBMSs and DBaaS platforms. Moreover, it is bundled with fifteen workloads that all differ in complexity and system demands, including four synthetic workloads, eight workloads from popular benchmarks, and three workloads that are derived from real-world applications. We demonstrate through a comprehensive set of experiments conducted on popular DBMS and DBaaS offerings the different features provided by OLTP-Bench and the effectiveness of our testbed in characterizing the performance of database services.

References

[1]

JPA Performance Benchmark. http://www.jpab.org.

[2]

OLTPBenchmark.com. http://oltpbenchmark.com.

[3]

pgbench. http://postgresql.org/docs/9.2/static/pgbench.html.

[4]

PolePosition: The Open Source Database Benchmark. http://polepos.org.

[5]

SysBench: A System Performance Benchmark. http://sysbench.sourceforge.net.

[6]

V. Angkanawaraphan and A. Pavlo. AuctionMark: A Benchmark for High-Performance OLTP Systems. http://hstore.cs.brown.edu/projects/auctionmark.

[7]

A. Arasu, M. Cherniack, E. F. Galvez, D. Maier, A. Maskey, E. Ryvkina, M. Stonebraker, and R. Tibbetts. Linear road: A stream data management benchmark. In VLDB, 2004.

Digital Library

[8]

T. G. Armstrong, V. Ponnekanti, D. Borthakur, and M. Callaghan. Linkbench: a database benchmark based on the facebook social graph. In SIGMOD Conference, pages 1185--1196, 2013.

Digital Library

[9]

S. Babu, N. Borisov, S. Duan, H. Herodotou, and V. Thummala. Automated experiment-driven management of (database) systems. In HotOS, 2009.

Digital Library

[10]

D. Bitton, D. J. DeWitt, and C. Turbyfill. Benchmarking database systems a systematic approach. In VLDB, 1983.

Digital Library

[11]

M. J. Cahill, U. Röhm, and A. D. Fekete. Serializable isolation for snapshot databases. SIGMOD, pages 729--738, 2008.

Digital Library

[12]

M. J. Cahill, U. Röhm, and A. D. Fekete. Serializable isolation for snapshot databases. ACM Transactions on Database Systems (TODS), 34(4): 20, 2009.

Digital Library

[13]

R. Cattell. Scalable SQL and NoSQL data stores. SIGMOD Rec., 39: 12--27, 2011.

Digital Library

[14]

M. Cha, H. Haddadi, F. Benevenuto, and K. P. Gummadi. Measuring user influence in Twitter: The million follower fallacy. In ICWSM, May 2010.

[15]

R. Cole, F. Funke, L. Giakoumakis, W. Guy, A. Kemper, S. Krompass, H. Kuno, R. Nambiar, T. Neumann, M. Poess, et al. The mixed workload ch-benchmark. In Proceedings of the Fourth International Workshop on Testing Database Systems, page 8. ACM, 2011.

Digital Library

[16]

B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Benchmarking cloud serving systems with ycsb. In SoCC, pages 143--154, 2010.

Digital Library

[17]

C. Curino, E. Jones, R. A. Popa, N. Malviya, E. Wu, S. Madden, H. Balakrishnan, and N. Zeldovich. Relational Cloud: A Database Service for the Cloud. In CIDR, pages 235--240, 2011.

[18]

F. Funke, A. Kemper, and T. Neumann. Benchmarking hybrid OLTP & OLAP database systems. In BTW, pages 390--409, 2011.

[19]

J. Gray. Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., 1992.

Digital Library

[20]

W. H. Highleyman. Performance Analysis of Transaction Processing Systems. Prentice Hall, 1989.

Digital Library

[21]

D. Kossmann, T. Kraska, and S. Loesing. An evaluation of alternative architectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of data, pages 579--590. ACM, 2010.

Digital Library

[22]

M. Lehn, T. Triebel, C. Gross, D. Stingl, K. Saller, W. Effelsberg, A. Kovacevic, and R. Steinmetz. Designing benchmarks for p2p systems. In From Active Data Management to Event-Based Systems and More. 2010.

Digital Library

[23]

P. Massa and P. Avesani. Controversial users demand local trust metrics: an experimental study on epinions.com community. In AAAI-05, pages 121--126, 2005.

Digital Library

[24]

S. Patil, M. Polte, K. Ren, W. Tantisiriroj, L. Xiao, J. López, G. Gibson, A. Fuchs, and B. Rinaldi. YCSB++: benchmarking and performance debugging advanced features in scalable table stores. SOCC, pages 9:1--9:14, 2011.

Digital Library

[25]

A. Pavlo, E. P. Jones, and S. Zdonik. On predictive modeling for optimizing transaction execution in parallel OLTP systems. Proc. VLDB Endow., 5: 85--96, October 2011.

Digital Library

[26]

D. R. Ports and K. Grittner. Serializable snapshot isolation in postgresql. Proceedings of the VLDB Endowment, 5(12): 1850--1861, 2012.

Digital Library

[27]

S. Ray, B. Simion, and A. Brown. Jackpine: A benchmark to evaluate spatial database performance. In ICDE, 2011.

Digital Library

[28]

E. Sarhan, A. Ghalwash, and M. Khafagy. Specification and implementation of dynamic web site benchmark in telecommunication area. In WEAS, pages 863--867, 2008.

Digital Library

[29]

Scalyr. Even Stranger than Expected: a Systematic Look at EC2 I/O. http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-io/.

[30]

J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. PVLDB, 3(1), 2010.

Digital Library

[31]

A. Schmidt, F. Waas, M. Kersten, M. J. Carey, I. Manolescu, and R. Busse. Xmark: a benchmark for xml data management. In VLDB, 2002.

Digital Library

[32]

B. Schroeder, A. Wierman, and M. Harchol-Balter. Open versus closed: a cautionary tale. NSDI, pages 18--18, 2006.

Digital Library

[33]

M. Seltzer, D. Krinsky, K. Smith, and X. Zhang. The case for application-specific benchmarking. In HotOS, 1999.

Digital Library

[34]

P. Shivam, V. Marupadi, J. Chase, T. Subramaniam, and S. Babu. Cutting corners: workbench automation for server benchmarking. In USENIX, 2008.

Digital Library

[35]

M. Stonebraker and A. Pavlo. The SEATS Airline Ticketing Systems Benchmark. http://hstore.cs.brown.edu/projects/seats.

[36]

The Transaction Processing Council. TPC-C Benchmark (Revision 5.9.0). http://www.tpc.org/tpcc/spec/tpcc_current.pdf, June 2007.

[37]

P. Tözün, I. Pandis, C. Kaynak, D. Jevdjic, and A. Ailamaki. From A to E: analyzing TPC's OLTP benchmarks: the obsolete, the ubiquitous, the unexplored. EDBT, pages 17--28, 2013.

Digital Library

[38]

G. Urdaneta, G. Pierre, and M. van Steen. Wikipedia workload analysis for decentralized hosting. Comput. Netw., 53: 1830--1845, July 2009.

Digital Library

[39]

G. Weikum. Where is the Data in the Big Data Wave? http://wp.sigmod.org/?p=786.

[40]

A. Wolski. TATP Benchmark Description (Version 1.0). http://tatpbenchmark.sourceforge.net, March 2009.

[41]

W. Zheng, R. Bianchini, G. J. Janakiraman, J. R. Santos, and Y. Turner. Justrunit: experiment-based management of virtualized data centers. In USENIX, 2009.

Digital Library

Cited By

Lao JWang YLi YWang JZhang YCheng ZChen WTang MWang J(2024)GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian OptimizationProceedings of the VLDB Endowment10.14778/3659437.365944917:8(1939-1952)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.14778/3659437.3659449
Benson LBinnig CBodensohn JLorenzi FLuo JPorobic DRabl TSanghi ASears RTözün PZiegler T(2024)Surprise Benchmarking: The Why, What, and HowProceedings of the Tenth International Workshop on Testing Database Systems10.1145/3662165.3662763(1-8)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3662165.3662763
Enzler EGoerens OAlonso GGruenheid A(2024)Performance Truthfulness of Differential Privacy for DB TestingProceedings of the Tenth International Workshop on Testing Database Systems10.1145/3662165.3662762(30-35)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3662165.3662762
Show More Cited By

Recommendations

Benchmarking OLTP/web databases in the cloud: the OLTP-bench framework
CloudDB '12: Proceedings of the fourth international workshop on Cloud data management

Benchmarking is a key activity in building and tuning data management systems, but the lack of reference workloads and a common platform makes it a time consuming and painful task. The need for such a tool is heightened with the advent of cloud ...
Read More
Stream Bench: Towards Benchmarking Modern Distributed Stream Computing Frameworks
UCC '14: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing

While big data is becoming ubiquitous, interest in handling data stream at scale is also gaining popularity, which leads to the sprout of many distributed stream computing systems. However, complexity of stream computing and diversity of workloads ...
Read More
Expert SQL Server in-Memory OLTP
Read More

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the VLDB Endowment

Proceedings of the VLDB Endowment Volume 7, Issue 4

December 2013

112 pages

ISSN:2150-8097

Issue’s Table of Contents

Publisher

VLDB Endowment

Publication History

Published: 01 December 2013

Published in PVLDB Volume 7, Issue 4

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

162
Total Citations
View Citations
922
Total Downloads

Downloads (Last 12 months)130
Downloads (Last 6 weeks)10

Other Metrics

View Author Metrics

Citations

Cited By

Lao JWang YLi YWang JZhang YCheng ZChen WTang MWang J(2024)GPTuner: A Manual-Reading Database Tuning System via GPT-Guided Bayesian OptimizationProceedings of the VLDB Endowment10.14778/3659437.365944917:8(1939-1952)Online publication date: 1-Apr-2024
https://dl.acm.org/doi/10.14778/3659437.3659449
Benson LBinnig CBodensohn JLorenzi FLuo JPorobic DRabl TSanghi ASears RTözün PZiegler T(2024)Surprise Benchmarking: The Why, What, and HowProceedings of the Tenth International Workshop on Testing Database Systems10.1145/3662165.3662763(1-8)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3662165.3662763
Enzler EGoerens OAlonso GGruenheid A(2024)Performance Truthfulness of Differential Privacy for DB TestingProceedings of the Tenth International Workshop on Testing Database Systems10.1145/3662165.3662762(30-35)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3662165.3662762
Zhan YXi RLiao JFan SHou M(2024)KnobTune: A Dynamic Database Configuration Tuning Strategy Leveraging Historical Workload SimilaritiesProceedings of the International Conference on Computing, Machine Learning and Data Science10.1145/3661725.3661734(1-8)Online publication date: 12-Apr-2024
https://dl.acm.org/doi/10.1145/3661725.3661734
Geng CBlanas SBond MWang Y(2024)IsoPredict: Dynamic Predictive Analysis for Detecting Unserializable Behaviors in Weakly Isolated Data Store ApplicationsProceedings of the ACM on Programming Languages10.1145/36563918:PLDI(343-367)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656391
Kanellis KFreischuetz JVenkataraman S(2024)Nautilus: A Benchmarking Platform for DBMS Knob TuningProceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning10.1145/3650203.3663336(72-76)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3650203.3663336
Camacho-Rodríguez JAgrawal AGruenheid AGosalia APetculescu CAguilar-Saborit JFloratou ACurino CRamakrishnan R(2024)LST-Bench: Benchmarking Log-Structured Tables in the CloudProceedings of the ACM on Management of Data10.1145/36393142:1(1-26)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639314
Lao JWang YLi YWang JZhang YCheng ZChen WZhou YTang MWang JBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)A Demonstration of GPTuner: A GPT-Based Manual-Reading Database Tuning SystemCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3654739(504-507)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3654739
de la Rúa Martínez JBuso FKouzoupis AOrmenisan ANiazi SBzhalava DMak KJouffrey VRonström MCunningham RZangis RMukhedkar DKhazanchi AVlassov VDowling JBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)The Hopsworks Feature Store for Machine LearningCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3653389(135-147)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3653389
Pavlenko ACahoon JZhu YKroth BNelson MCarter ALiao DWright TCamacho-Rodríguez JSaur KBarcelo PSanchez-Pi NMeliou ASudarshan S(2024)Vertically Autoscaling Monolithic Applications with CaaSPER: Scalable Container-as-a-Service Performance Enhanced Resizing Algorithm for the CloudCompanion of the 2024 International Conference on Management of Data10.1145/3626246.3653378(241-254)Online publication date: 9-Jun-2024
https://dl.acm.org/doi/10.1145/3626246.3653378
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents