article

LUBM: A benchmark for OWL knowledge base systems

Authors:

Zhengxiang Pan,

Jeff HeflinAuthors Info & Claims

Web Semantics: Science, Services and Agents on the World Wide Web, Volume 3, Issue 2-3

Pages 158 - 182

https://doi.org/10.1016/j.websem.2005.06.005

Published: 01 October 2005 Publication History

Abstract

We describe our method for benchmarking Semantic Web knowledge base systems with respect to use in large OWL applications. We present the Lehigh University Benchmark (LUBM) as an example of how to design such benchmarks. The LUBM features an ontology for the university domain, synthetic OWL data scalable to an arbitrary size, 14 extensional queries representing a variety of properties, and several performance metrics. The LUBM can be used to evaluate systems with different reasoning capabilities and storage mechanisms. We demonstrate this with an evaluation of two memory-based systems and two systems with persistent storage.

References

[1]

Alexaki, S., The RDFSuite: managing voluminous RDF description bases. In: Proceedings of the Second International Workshop on the Semantic Web (SemWeb'01), in conjunction with the Tenth International World Wide Web Conference (WWW10),

[2]

Alexaki, S., On storing voluminous RDF description: The case of Web Portal Catalogs. In: Proceedings of the Fourth International Workshop on the Web and Databases,

[3]

Bitton, D., DeWitt, D. and Turbyfill, C., Benchmarking database systems, a systematic approach. In: Proceedings of the Ninth International Conference on Very Large Data Bases,

[4]

Bitton, D. and Turbyfill, C., A retrospective on the Wisconsin benchmark. 1994. second ed. Morgan Kaufman.

[5]

Broekstra, J. and Kampman, A., Sesame: a generic architecture for storing and querying RDF and RDF schema. In: Proceedings of the First International Semantic Web Conference (ISWC2002),

[6]

Broekstra, J. and Kampman, A., Inferencing and truth maintenance in RDF schema: exploring a naive practical approach. In: Workshop on Practical and Scalable Semantic Systems (PSSS),

[7]

J.J. Carroll, J.D. Roo (Eds.), OWL Web Ontology Test Cases, W3C Recommendation 10 February 2004, http://www.w3.org/TR/2004/REC-owl-test-20040210/.

[8]

Cattell, R.G.G., An engineering database benchmark. 1991. Morgan Kaufman.

[9]

D. Connolly, et al., DAML+OIL (March 2001) Reference Description, http://www.w3.org/TR/daml+oil-reference.

[10]

M. Dean, G. Schreiber (Eds.), OWL Web Ontology Language Reference, W3C Recommendation 10 February 2004, http://www.w3.org/TR/2004/REC-owl-ref-20040210/.

[11]

Elhaik, Q., Rousset, M.-C. and Ycart, B., Generating random benchmarks for description logics. In: Proceedings of the 1998 Description Logic Workshop (DL'98),

[12]

Garcí***a-Castro, R. and Gómez-Pérez, A., A Benchmark suite for evaluating the performance of the WebODE ontology engineering platform. In: Proceedings of the Third International Workshop on Evaluation of Ontology-Based Tools,

[13]

M. Gensereth, R. Fikes, Knowledge Interchange Format, Stanford Logic Report Logic-92-1, Stanford University, http://logic.standford.edu/kif/kif.html.

[14]

Guo, Y., Heflin, J. and Pan, Z., Benchmarking DAML+OIL repositories. In: Proceedings of the Second International Semantic Web Conference (ISWC2003),

[15]

Guo, Y., Pan, Z. and Heflin, J., Choosing the best knowledge base system for large Semantic Web applications. In: Proceedings of the 13th International World Wide Web Conference (WWW2004)-Alternate Track Papers & Posters,

[16]

Guo, Y., Pan, Z. and Heflin, J., An evaluation of knowledge base systems for large OWL datasets. In: Proceedings of the Third International Semantic Web Conference (ISWC2004),

[17]

Haarslev, V. and Möller, R., Racer: a core inference engine for the Semantic Web. In: Workshop on Evaluation on Ontology-Based Tools, the Second International Semantic Web (ISWC2003),

[18]

Haarslev, V., Möller, R. and Wessel, M., Querying the Semantic Web with Racer+nRQL. In: Proceedings of the Workshop on Description Logics 2004 (ADL2004),

[19]

Horrocks, I., The FaCT system. In: Automated Reasoning with Analytic Tableaux and Related Methods International Conference (Tableaux'98),

[20]

Horrocks, I., The instance store: DL reasoning with large numbers of individuals. In: Proceedings of the 2004 Description Logic Workshop (DL2004),

[21]

Horrocks, I. and Patel-Schneider, P., DL systems comparison. In: Proceedings of the 1998 Description Logic Workshop (DL'98),

[22]

Karvounarakis, G., Alexaki, S., Christophides, V., Plexousakis, D. and Scholl, M., RQL: a declarative query language for RDF. In: Proceedings of the Eleventh International World Wide Web Conference (WWW'02),

[23]

Kopena, J.B. and Regli, W.C., DAMLJessKB: a tool for reasoning with the Semantic Web. In: Proceedings of the Second International Semantic Web Conference (ISWC2003),

[24]

Lewis, D.D., A sequential algorithm for training text classifiers: corrigendum and additional data. SIGIR Forum. v29 i2. 13-19.

[25]

Magkanaraki, A., Benchmarking RDF schemas for the Semantic Web. In: Proceedings of the First International Semantic Web Conference (ISWC2002),

[26]

D. Maynard, et al., D2.1.4: Definition of a Methodology, General Criteria, and Benchmark Suites for Benchmarking Ontology Tools, EU-IST Network of Excellence (NoE) IST-2004-507482 KWEB, 2005.

[27]

Pan, Z. and Heflin, J., DLDB: extending relational databases to support Semantic Web queries. In: Workshop on Practical and Scalable Semantic Systems, the Second International Semantic Web Conference (ISWC2003),

[28]

E. Prud'hommeaux, A. Seaborne (Eds.), SPARQL Query Language for RDF, W3C Working Draft 19 April 2005, http://www.w3.org/TR/2005/WD-rdf-sparql-query-20050419/.

[29]

Riazanov, A. and Voronkov, A., The design and implementation of Vampire. AI Commun. v15 i2-3. 91-110.

[30]

van Rijsbergen, C.J., Information Retrieval. 1979. Butterworths, London.

[31]

A. Seaborne, RDQL-A Query Language for RDF, W3C Member Submission 9 January 2004, http://www.w3.org/Submission/2004/SUBM-RDQL-20040109/.

[32]

M.K. Smith, C. Welty, D.L. McGuinness (Eds.), OWL Web Ontology Language Guide, W3C Recommendation 10 February 2004, http://www.w3.org/TR/2004/REC-owl-guide-20040210/.

[33]

Stonebraker, M., The SEQUIOA 2000 storage benchmark. In: Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data,

[34]

Tempich, C. and Volz, R., Towards a benchmark for Semantic Web reasoners-an analysis of the DAML ontology library. In: Workshop on Evaluation on Ontology-based Tools, the Second International Semantic Web Conference (ISWC2003),

[35]

FaCT++, http://owl.man.ac.uk/factplusplus/.

[36]

Hoolet, http://owl.man.ac.uk/hoolet/.

[37]

Jena-A Semantic Web Framework for Java, http://jena.sourceforge.net/.

[38]

Jess: The Rule Engine for the Java Platform, http://herzberg.ca.sandia.gov/jess.

[39]

KAON: The KArlsruhe ONtology and Semantic Web Tool Suite, http://kaon.semanticweb.org/.

[40]

Kowari, http://www.kowari.org/.

[41]

OWLJessKB: A Semantic Web Reasoning Tool, http://edge.cs.drexel.edu/assemblies/software/owljesskb/.

[42]

Pellet OWL Reasoner, http://www.mindswap.org/2003/pellet/.

[43]

PostgreSQL, http://www.postgresql.org.

[44]

Resource Description Framework (RDF), http://www.w3.org/RDF/.

[45]

The SeRQL Query Language, http://www.openrdf.org/doc/users/ch06.html.

Cited By

Schwabe TAcosta M(2024)Cardinality Estimation over Knowledge Graphs with Embeddings and Graph Neural NetworksProceedings of the ACM on Management of Data10.1145/36392992:1(1-26)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639299
Troullinou GStefanidis KPlexousakis DKondylakis HChua TNgo CKumar RLauw HKa-Wei Lee R(2024)DIAERESIS: Knowledge Graph Partitioning for Efficient Query AnsweringCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651231(987-990)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651231
El Husseini WEl Vaigh CGoasdoué FJaudoin HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Query Optimization for Ontology-Mediated Query AnsweringProceedings of the ACM on Web Conference 202410.1145/3589334.3645567(2138-2148)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645567
Show More Cited By

Recommendations

Choosing the best knowledge base system for large semantic web applications
WWW Alt. '04: Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters

We present an evaluation of four knowledge base systems with respect to use in large Semantic Web applications. We discuss the performance of each system. In particular, we show that existing systems need to place a greater emphasis on scalability.
A Requirements Driven Framework for Benchmarking Semantic Web Knowledge Base Systems

A key challenge for the Semantic Web is to acquire the capability to effectively query large knowledge bases. As there will be several competing systems, we need benchmarks that will objectively evaluate these systems. Development of effective ...
The bio-zen plus ontology
Towards a Metaontology for the Biomedical Domain

Bio-zen plus is an OWL DL ontology for the domain of biomedical research. It incorporates several existing Semantic Web ontologies: the DOLCE foundational ontology, the Simple Knowledge Organisation System (SKOS), the Semantically Interlinked Online ...

Comments

Information & Contributors

Information

Published In

cover image Web Semantics: Science, Services and Agents on the World Wide Web

Web Semantics: Science, Services and Agents on the World Wide Web Volume 3, Issue 2-3

October, 2005

164 pages

ISSN:1570-8268

Issue’s Table of Contents

Copyright © Elsevier B.V. © 2005.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 October 2005

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

455
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schwabe TAcosta M(2024)Cardinality Estimation over Knowledge Graphs with Embeddings and Graph Neural NetworksProceedings of the ACM on Management of Data10.1145/36392992:1(1-26)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3639299
Troullinou GStefanidis KPlexousakis DKondylakis HChua TNgo CKumar RLauw HKa-Wei Lee R(2024)DIAERESIS: Knowledge Graph Partitioning for Efficient Query AnsweringCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651231(987-990)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651231
El Husseini WEl Vaigh CGoasdoué FJaudoin HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Query Optimization for Ontology-Mediated Query AnsweringProceedings of the ACM on Web Conference 202410.1145/3589334.3645567(2138-2148)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645567
Willerval ADiefenbach DBonifati A(2024)Generate and Update Large HDT RDF Knowledge Graphs on Commodity HardwareThe Semantic Web10.1007/978-3-031-60635-9_8(128-144)Online publication date: 26-May-2024
https://dl.acm.org/doi/10.1007/978-3-031-60635-9_8
Hubert NMonnin Pd’Aquin MMonticolo DBrun A(2024)PyGraft: Configurable Generation of Synthetic Schemas and Knowledge Graphs at Your FingertipsThe Semantic Web10.1007/978-3-031-60635-9_1(3-20)Online publication date: 26-May-2024
https://dl.acm.org/doi/10.1007/978-3-031-60635-9_1
Yan LMa ZLi DLu JMa R(2023)RDF(S) Store in Object-Relational DatabasesJournal of Database Management10.4018/JDM.33471035:1(1-32)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.4018/JDM.334710
Liu SGrau BHorrocks IKostylev EMarquis PSon TKern-Isberner G(2023)Revisiting inferential benchmarks for knowledge graph completionProceedings of the 20th International Conference on Principles of Knowledge Representation and Reasoning10.24963/kr.2023/45(461-471)Online publication date: 2-Sep-2023
https://dl.acm.org/doi/10.24963/kr.2023/45
Zhang XHu PNenov YHorrocks IElkind E(2023)Enhancing datalog reasoning with hypertree decompositionsProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/377(3383-3393)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/377
Calautti MMilani MPieris A(2023)Semi-Oblivious Chase Termination for Linear Existential Rules: An Experimental StudyProceedings of the VLDB Endowment10.14778/3611479.361149316:11(2858-2870)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611493
Rabbani KLissandrini MHose K(2023)Extraction of Validating Shapes from Very Large Knowledge GraphsProceedings of the VLDB Endowment10.14778/3579075.357907816:5(1023-1032)Online publication date: 6-Mar-2023
https://dl.acm.org/doi/10.14778/3579075.3579078
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents