Abstract
Analysing the performance of OWL reasoners on expressive OWL ontologies is an ongoing challenge. In this paper, we present a new approach to performance analysis based on justifications for entailments of OWL ontologies. Justifications are minimal subsets of an ontology that are sufficient for an entailment to hold, and are commonly used to debug OWL ontologies. In JustBench, justifications form the key unit of test, which means that individual justifications are tested for correctness and reasoner performance instead of entire ontologies or random subsets. Justifications are generally small and relatively easy to analyse, which makes them very suitable for transparent analytic micro-benchmarks. Furthermore, the JustBench approach also allows us to isolate reasoner errors and inconsistent behaviour. We present the results of initial experiments using JustBench with FaCT++, HermiT, and Pellet. Finally, we show how JustBench can be used by reasoner developers and ontology engineers seeking to understand and improve the performance characteristics of reasoners and ontologies.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Bail, S., Parsia, B., Sattler, U.: The justificatory structure of OWL ontologies. In: OWLED (2010)
Boyer, B.: Robust Java benchmarking (2008), http://www.ibm.com/developerworks/java/library/j-benchmark1.html
Brandt, S., Küsters, R., Turhan, A.-Y.: Approximation and difference in description logics. In: Proc. of KR 2002. Morgan Kaufmann Publishers, San Francisco (2002)
Cuenca Grau, B., Horrocks, I., Kazakov, Y., Sattler, U.: Modular reuse of ontologies: Theory and practice. J. of Artificial Intelligence Research 31, 273–318 (2008)
Cuenca Grau, B., Horrocks, I., Motik, B., Parsia, B., Patel-Schneider, P., Sattler, U.: OWL 2: The next step for OWL. J. of Web Semantics 6(4), 309–322 (2008)
Dewitt, D.J.: The Wisconsin benchmark: Past, present, and future. In: Gray, J. (ed.) The Benchmark Handbook for Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)
Franco, J.V.: On the probabilistic performance of algorithms for the satisfiability problem. Inf. Process. Lett. 23(2), 103–106 (1986)
Gardiner, T., Tsarkov, D., Horrocks, I.: Framework for an automated comparison of description logic reasoners. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 654–667. Springer, Heidelberg (2006)
Guo, Y., Pan, Z., Heflin, J.: LUBM: A benchmark for OWL knowledge base systems. Web Semantics: Science, Services and Agents on the World Wide Web 3(2-3), 158–182 (2005)
Horrocks, I., Patel-Schneider, P.F.: Evaluating optimized decision procedures for propositional modal K(m) satisfiability. J. Autom. Reasoning 28(2), 173–204 (2002)
Hustadt, U., Schmidt, R.A.: Scientific benchmarking with temporal logic decision procedures. In: Fensel, D., Giunchiglia, F., McGuinness, D.L., Williams, M.-A. (eds.) KR, pp. 533–546. Morgan Kaufmann, San Francisco (2002)
Lin, H., Sirin, E.: Pellint - a performance lint tool for pellet. In: Dolbear, C., Ruttenberg, A., Sattler, U. (eds.) OWLED. CEUR Workshop Proceedings, vol. 432. CEUR-WS.org. (2008)
Ma, L., Yang, Y., Qiu, Z., Xie, G.T., Pan, Y., Liu, S.: Towards a complete OWL ontology benchmark. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 125–139. Springer, Heidelberg (2006)
Pan, Z.: Benchmarking DL reasoners using realistic ontologies. In: OWLED (2005)
Parsia, B., Schneider, T.: The modular structure of an ontology: An empirical study. In: Lin, F., Sattler, U., Truszczynski, M. (eds.) KR. AAAI Press, Menlo Park (2010)
Rudolph, S., Tserendorj, T., Hitzler, P.: What is approximate reasoning? In: Calvanese, D., Lausen, G. (eds.) RR 2008. LNCS, vol. 5341, pp. 150–164. Springer, Heidelberg (2008)
Schaerf, M., Cadoli, M.: Tractable reasoning via approximation. Artificial Intelligence 74, 249–310 (1995)
Selman, B., Mitchell, D.G., Levesque, H.J.: Generating hard satisfiability problems. Artif. Intell. 81(1-2), 17–29 (1996)
Wang, T., Parsia, B.: Ontology performance profiling and model examination: first steps. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudré-Mauroux, P. (eds.) ASWC 2007 and ISWC 2007. LNCS, vol. 4825, p. 595. Springer, Heidelberg (2007)
Weithöner, T., Liebig, T., Luther, M., Böhm, S.: What’s wrong with OWL benchmarks? In: SSWS (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bail, S., Parsia, B., Sattler, U. (2010). JustBench: A Framework for OWL Benchmarking. In: Patel-Schneider, P.F., et al. The Semantic Web – ISWC 2010. ISWC 2010. Lecture Notes in Computer Science, vol 6496. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17746-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-642-17746-0_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17745-3
Online ISBN: 978-3-642-17746-0
eBook Packages: Computer ScienceComputer Science (R0)