research-article

GenBase: a complex analytics genomics benchmark

Authors:

Nadathur Rajagopalan Satish,

Narayanan Sundaram,

Michael StonebrakerAuthors Info & Claims

SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

Pages 177 - 188

https://doi.org/10.1145/2588555.2595633

Published: 18 June 2014 Publication History

Abstract

This paper introduces a new benchmark designed to test database management system (DBMS) performance on a mix of data management tasks (joins, filters, etc.) and complex analytics (regression, singular value decomposition, etc.) Such mixed workloads are prevalent in a number of application areas including most science workloads and web analytics. As a specific use case, we have chosen genomics data for our benchmark and have constructed a collection of typical tasks in this domain. In addition to being representative of a mixed data management and analytics workload, this benchmark is also meant to scale to large dataset sizes and multiple nodes across a cluster. Besides presenting this benchmark, we have run it on a variety of storage systems including traditional row stores, newer column stores, Hadoop, and an array DBMS. We present performance numbers on all systems on single and multiple nodes, and show that performance differs by orders of magnitude between the various solutions. In addition, we demonstrate that most platforms have scalability issues. We also test offloading the analytics onto a coprocessor. The intent of this benchmark is to focus research interest in this area; to this end, all of our data, data generators, and scripts are available on our web site.

References

[1]

Affymetrix. http://www.affymetrix.com.

[2]

Apache hive(tm). http://hive.apache.org.

[3]

Apache mahout: Scalable machine learning and data mining. http://mahout.apache.org.

[4]

Apache (tm) hadoop (r). http://hadoop.apache.org.

[5]

Blas (basic linear algebra subprograms). http://www.netlib.org/blas/.

[6]

Dna sequencing costs. http://www.genome.gov/sequencingcosts.

[7]

Genbase. http://web.mit.edu/~mvartak/www/genmark.html.

[8]

Madlib. http://madlib.net.

[9]

Postgresql. http://www.postgresql.org.

[10]

The r project for statistical computing. http://www.r-project.org.

[11]

Scalapack - scalable linear algebra package. http://www.netlib.org/scalapack/.

[12]

Scidb. http://www.scidb.org.

[13]

Singular value decomposition for genome-wide expression data processing and modeling. www.pnas.org/content/97/18/10101.abstract.

[14]

Tpc transaction processing performance council. www.tpc.org.

[15]

Intel math kernel library (intel mkl) 11.1, 2013.

[16]

A. Arasu, M. Cherniack, E. Galvez, D. Maier, A. S. Maskey, E. Ryvkina, M. Stonebraker, and R. Tibbetts. Linear road: a stream data management benchmark. In Proceedings of the Thirtieth international conference on Very large data bases - Volume 30, VLDB '04, pages 480--491. VLDB Endowment, 2004.

Digital Library

[17]

D. Bitton, C. Turbyfill, D. J. Dewitt, and D. J. Dewitt. Benchmarking database systems: A systematic approach. pages 8--19, 1983.

Digital Library

[18]

M. J. Carey, D. J. DeWitt, J. F. Naughton, M. Asgarian, P. Brown, J. E. Gehrke, and D. N. Shah. The bucky object-relational benchmark. In Proceedings of the 1997 ACM SIGMOD international conference on Management of data, SIGMOD '97, pages 135--146, New York, NY, USA, 1997. ACM.

Digital Library

[19]

M. Clamp, B. Fry, M. Kamal, X. Xie, J. Cuff, M. F. Lin, and E. S. Lander. Distinguishing protein-coding and noncoding genes in the human genome. Proc. National Academy of Sciences of the United States of America, 104(49):19428--33, 2007.

[20]

R. Hazra. Driving industrial innovation on the path to exascale: From vision to reality, 2013. http://newsroom.intel.com/servlet/JiveServlet/download/6314- 25051/Intel_ISC13_keynote_by_Raj_Hazra.pdf.

[21]

J. M. Hellerstein, C. Ré, F. Schoppmann, D. Z. Wang, E. Fratkin, A. Gorajek, K. S. Ng, C. Welton, X. Feng, K. Li, and A. Kumar. The madlib analytics library: or mad skills, the sql. Proc. VLDB Endow., 5(12):1700--1711, Aug. 2012.

Digital Library

[22]

C. Kim, J. Chhugani, N. Satish, E. Sedlar, A. D. Nguyen, T. Kaldewey, V. W. Lee, S. A. Brandt, and P. Dubey. Designing fast architecture-sensitive tree search on modern multicore/many-core processors. ACM Trans. Database Syst., 36(4):22, 2011.

Digital Library

[23]

G. Ostrouchov, W.-C. Chen, D. Schmidt, and P. Patel. Programming with big data in r, 2012. http://r-pbd.org/.

[24]

M. Schatz and B. Langmead. The dna data deluge. IEEE Spectrum, 50(7):28--33, 2013.

[25]

A. Subramanian, P. Tamayo, V. K. Mootha, S. Mukherjee, B. L. Ebert, M. A. Gillette, A. Paulovich, S. L. Pomeroy, T. R. Golub, E. S. Lander, and J. P. Mesirov. Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles. Proc. National Academy of Sciences of the United States of America, 102(43):15545--50, 2005.

Cited By

Mišev DRodionychev MBaumann P(2023)Performance of Null Handling in Array Databases2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386100(247-254)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386100
Kim BKoo KEnkhbat UKim SKim JMoon B(2022)M2BenchProceedings of the VLDB Endowment10.14778/3574245.357425916:4(747-759)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.14778/3574245.3574259
Fang CSong SMei Y(2022)On repairing timestamps for regular interval time seriesProceedings of the VLDB Endowment10.14778/3538598.353860715:9(1848-1860)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.14778/3538598.3538607
Show More Cited By

Index Terms

GenBase: a complex analytics genomics benchmark
1. Human-centered computing
  1. Visualization
    1. Visualization application domains
      1. Scientific visualization
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
  2. Information systems applications

Recommendations

BigBench: towards an industry standard benchmark for big data analytics
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

There is a tremendous interest in big data by academia, industry and a large user base. Several commercial and open source providers unleashed a variety of products to support big data storage and processing. As these products mature, there is a need to ...
SPEC HPG benchmarks for high-performance systems

In this paper, we discuss the results and characteristics of the benchmark suites maintained by the Standard Performance Evaluation Corporation's (SPEC) High-Performance Group (HPG). Currently, SPECHPGhas two lines of benchmark suites for measuring ...
A Benchmark Characterization of the EEMBC Benchmark Suite

Benchmark consumers expect benchmark suites to be complete, accurate, and consistent, and benchmark scores serve as relative measures of performance. However, it is important to understand how benchmarks stress the processors that they aim to test. This ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

June 2014

1645 pages

ISBN:9781450323765

DOI:10.1145/2588555

General Chairs:
Curtis Dyreson
Utah State University, USA
,
Feifei Li
University of Utah, USA
,
Program Chair:
M. Tamer Özsu
University of Waterloo, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'14

Sponsor:

SIGMOD

SIGMOD/PODS'14: International Conference on Management of Data

June 22 - 27, 2014

Utah, Snowbird, USA

Acceptance Rates

SIGMOD '14 Paper Acceptance Rate 107 of 421 submissions, 25%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
601
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mišev DRodionychev MBaumann P(2023)Performance of Null Handling in Array Databases2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386100(247-254)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386100
Kim BKoo KEnkhbat UKim SKim JMoon B(2022)M2BenchProceedings of the VLDB Endowment10.14778/3574245.357425916:4(747-759)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.14778/3574245.3574259
Fang CSong SMei Y(2022)On repairing timestamps for regular interval time seriesProceedings of the VLDB Endowment10.14778/3538598.353860715:9(1848-1860)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.14778/3538598.3538607
Wang ZDing XJin HZhou P(2022)Efficient secure and verifiable location-based skyline queries over encrypted dataProceedings of the VLDB Endowment10.14778/3538598.353860515:9(1822-1834)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.14778/3538598.3538605
Minartz Kd'Hondt JPapapetrou O(2022)Multivariate correlations discovery in static and streaming dataProceedings of the VLDB Endowment10.14778/3514061.351407215:6(1266-1278)Online publication date: 22-Jun-2022
https://dl.acm.org/doi/10.14778/3514061.3514072
Alotaibi RCautis BDeutsch ALatrache MManolescu IYang Y(2020)ESTOCADAProceedings of the VLDB Endowment10.14778/3415478.341551613:12(2949-2952)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3415478.3415516
Peng YZhang YLin XZhang WQin LZhou J(2020)Towards bridging theory and practiceProceedings of the VLDB Endowment10.14778/3372716.337272013:4(463-476)Online publication date: 6-Jan-2020
https://dl.acm.org/doi/10.14778/3372716.3372720
Lu YCheng PChen Z(2020)Design and Implementation of the Tianhe-2 Data Storage and Management SystemJournal of Computer Science and Technology10.1007/s11390-020-9799-435:1(27-46)Online publication date: 17-Jan-2020
https://doi.org/10.1007/s11390-020-9799-4
Boehm MKumar AYang J(2019)Data Management in Machine Learning SystemsSynthesis Lectures on Data Management10.2200/S00895ED1V01Y201901DTM05714:1(1-173)Online publication date: 25-Feb-2019
https://doi.org/10.2200/S00895ED1V01Y201901DTM057
Barthels CMüller ITaranov KAlonso GHoefler T(2019)Strong consistency is not hard to getProceedings of the VLDB Endowment10.14778/3358701.335870212:13(2325-2338)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.14778/3358701.3358702
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten