The Linked Data Benchmark Council (LDBC) is an EU project that aims to develop industry-strength benchmarks for graph and RDF data management systems. It includes the creation of a non-profit LDBC organization, where industry players and academia come together for managing the development of benchmarks as well as auditing and publishing official results. We present an overview of the project including its goals and organization, and describe its process and design methodology for benchmark development. We introduce so-called "choke-point" based benchmark development through which experts identify key technical challenges, and introduce them in the benchmark workload. Finally, we present the status of two benchmarks currently in development, one targeting graph data management systems using a social network data case, and the other targeting RDF systems using a data publishing case.

References

[1]

P. Boncz, T. Neumann, and O. Erling. TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark. In TPCTC, 2013.

Google Scholar

[2]

D. Dominguez-Sal, N. Martinez-Bazan, V. Muntes-Mulero, P. Baleta, and J. L. Larriba-Pey. A Discussion on the Design of Graph Database Benchmarks. In TPCTC, 2010.

Digital Library

Google Scholar

[3]

S. Duan, A. Kementsietsidis, K. Srinivas, and O. Udrea. Apples and Oranges: A Comparison of RDF Benchmarks and Real RDF Datasets. In SIGMOD. ACM Press, 2011.

Digital Library

Google Scholar

[4]

Y. Guo, A. Qasem, Z. Pan, and J. Heflin. A Requirements Driven Framework for Benchmarking Semantic Web Knowledge Base Systems. TKDE, 19(2):297--309, 2007.

Digital Library

Google Scholar

[5]

K. Huppler. The Art of Building a Good Benchmark. In TPCTC, August 2009.

Digital Library

Google Scholar

[6]

M.-D. Pham, P. A. Boncz, and O. Erling. S3G2: A Scalable Structure-Correlated Social Graph Generator. In TPCTC, 2012.

Google Scholar

[7]

T. Weithöner, T. Liebig, M. Luther, and S. Böhm. What's Wrong with OWL Benchmarks. In SSWS, 2006.

Google Scholar

Cited By

View all

Szárnyas GBebee BBirler ADeutsch AFletcher GGabb HGosnell DGreen AGuo ZHare KHidders JIosup AKiryakov AKovatchev TLi XLibkin LLin HLuo XPrat-Pérez APüroja DQi Svan Rest OSteer BSzakállas DTong BWaudby JWu MYang BYu WZhang CZhang JZhou YBoncz P(2024)The Linked Data Benchmark Council (LDBC): Driving Competition and Collaboration in the Graph Data Management SpacePerformance Evaluation and Benchmarking10.1007/978-3-031-68031-1_7(90-106)Online publication date: 22-Sep-2024
https://doi.org/10.1007/978-3-031-68031-1_7
Hubert NMonnin Pd’Aquin MMonticolo DBrun A(2024)PyGraft: Configurable Generation of Synthetic Schemas and Knowledge Graphs at Your FingertipsThe Semantic Web10.1007/978-3-031-60635-9_1(3-20)Online publication date: 26-May-2024
https://dl.acm.org/doi/10.1007/978-3-031-60635-9_1
Schönberger MTrummer IMauerer W(2023)Quantum-Inspired Digital Annealing for Join OrderingProceedings of the VLDB Endowment10.14778/3632093.363211217:3(511-524)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632112
Show More Cited By

Recommendations

Subsetting the SPEC CPU2006 benchmark suite

On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for ...
A Benchmark Characterization of the EEMBC Benchmark Suite

Benchmark consumers expect benchmark suites to be complete, accurate, and consistent, and benchmark scores serve as relative measures of performance. However, it is important to understand how benchmarks stress the processors that they aim to test. This ...
Linked Data Benchmark Council: 12 years of fostering competition in the graph processing space
ICPE '24 Companion: Companion of the 15th ACM/SPEC International Conference on Performance Engineering

The Linked Data Benchmark Council (LDBC) was originally created in 2012 as part of a European Union-funded project of the same name. Its original goal was to design standard benchmarks for graph processing and to facilitate competition among vendors to ...

Comments

Information & Contributors

Information

Published In

ACM SIGMOD Record Volume 43, Issue 1

March 2014

71 pages

ISSN:0163-5808

DOI:10.1145/2627692

Editors:
Ioana Manolescu
INRIA Saclay
,
Denilson Barbosa
University of Alberta
,
Pablo Barceló
Universidad de Chile
,
Vanessa Braganholo
Universidade Federal Fluminense
,
Marco Brambilla
Politecnico di Milano
,
Chee Yong Chan
National University of Singapore
,
Rada Chirkova
North Carolina State University
,
Anish Das Sarma
Google Research
,
Glenn Paulley
Conestoga College
,
Alkis Simitsis
HP Labs
,
Nesime Tatbul
ETH Zurich
,
Marianne Winslett
University of Illinois

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Published in SIGMOD Volume 43, Issue 1

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
463
Total Downloads

Downloads (Last 12 months)26
Downloads (Last 6 weeks)1

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Szárnyas GBebee BBirler ADeutsch AFletcher GGabb HGosnell DGreen AGuo ZHare KHidders JIosup AKiryakov AKovatchev TLi XLibkin LLin HLuo XPrat-Pérez APüroja DQi Svan Rest OSteer BSzakállas DTong BWaudby JWu MYang BYu WZhang CZhang JZhou YBoncz P(2024)The Linked Data Benchmark Council (LDBC): Driving Competition and Collaboration in the Graph Data Management SpacePerformance Evaluation and Benchmarking10.1007/978-3-031-68031-1_7(90-106)Online publication date: 22-Sep-2024
https://doi.org/10.1007/978-3-031-68031-1_7
Hubert NMonnin Pd’Aquin MMonticolo DBrun A(2024)PyGraft: Configurable Generation of Synthetic Schemas and Knowledge Graphs at Your FingertipsThe Semantic Web10.1007/978-3-031-60635-9_1(3-20)Online publication date: 26-May-2024
https://dl.acm.org/doi/10.1007/978-3-031-60635-9_1
Schönberger MTrummer IMauerer W(2023)Quantum-Inspired Digital Annealing for Join OrderingProceedings of the VLDB Endowment10.14778/3632093.363211217:3(511-524)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.14778/3632093.3632112
Besta MGerstenberger RPeter EFischer MPodstawski MBarthels CAlonso GHoefler T(2023)Demystifying Graph Databases: Analysis and Taxonomy of Data Organization, System Designs, and Graph QueriesACM Computing Surveys10.1145/360493256:2(1-40)Online publication date: 15-Sep-2023
https://dl.acm.org/doi/10.1145/3604932
Besta MGerstenberger RFischer MPodstawski MBlach NEgeli BMitenkov GChlapek WMichalewicz MNiewiadomski HMueller JHoefler TMohror KArnold DBadia R(2023)The Graph Database Interface: Scaling Online Transactional and Analytical Graph Workloads to Hundreds of Thousands of CoresProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3607068(1-18)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3607068
Maekawa SSasaki YFletcher GOnizuka M(2023)GenCATInformation Systems10.1016/j.is.2023.102195115:COnline publication date: 24-May-2023
https://dl.acm.org/doi/10.1016/j.is.2023.102195
Zheng YDou WWang YQin ZTang LGao YWang DWang WWei JRyu SSmaragdakis Y(2022)Finding bugs in Gremlin-based graph database systems via Randomized differential testingProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3533767.3534409(302-313)Online publication date: 18-Jul-2022
https://dl.acm.org/doi/10.1145/3533767.3534409
Sharma CSinha R(2022)FLASc: a formal algebra for labeled property graph schemaAutomated Software Engineering10.1007/s10515-022-00336-y29:1Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1007/s10515-022-00336-y
Cheng SHartig O(2022)LinGBM: A Performance Benchmark for Approaches to Build GraphQL ServersWeb Information Systems Engineering – WISE 202210.1007/978-3-031-20891-1_16(209-224)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20891-1_16
Daga EMeroño-Peñuela AMotta ESaleem MVerborgh RAli MHartig O(2021)Sequential linked dataSemantic Web10.3233/SW-21043612:6(927-958)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.3233/SW-210436
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

Subsetting the SPEC CPU2006 benchmark suite

A Benchmark Characterization of the EEMBC Benchmark Suite

Linked Data Benchmark Council: 12 years of fostering competition in the graph processing space