article

Application of aboutness to functional benchmarking in information retrieval

Authors:

Chun-Hung ChengAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 19, Issue 4

Pages 337 - 370

https://doi.org/10.1145/502795.502796

Published: 01 October 2001 Publication History

Abstract

Experimental approaches are widely employed to benchmark the performance of an information retrieval (IR) system. Measurements in terms of recall and precision are computed as performance indicators. Although they are good at assessing the retrieval effectiveness of an IR system, they fail to explore deeper aspects such as its underlying functionality and explain why the system shows such performance. Recently, inductive (i.e., theoretical) evaluation of IR systems has been proposed to circumvent the controversies of the experimental methods. Several studies have adopted the inductive approach, but they mostly focus on theoretical modeling of IR properties by using some metalogic. In this article, we propose to use inductive evaluation for functional benchmarking of IR models as a complement of the traditional experiment-based performance benchmarking. We define a functional benchmark suite in two stages: the evaluation criteria based on the notion of "aboutness," and the formal evaluation methodology using the criteria. The proposed benchmark has been successfully applied to evaluate various well-known classical and logic-based IR models. The functional benchmarking results allow us to compare and analyze the functionality of the different IR models.

References

[1]

AGOSTI, M. AND SMEATON, A. F., EDS. 1996. Information Retrieval and Hypertext. Kluwer, Hingham, Mass.]]

Digital Library

[2]

AMATI, G. AND GEORGATOS, K. 1996. Relevance as deduction: A logical view of information retrieval. In Proceedings of the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]

[3]

BAEZA-YATES, R. AND RIBEIRO-NETO, B. 1999. Modern Information Retrieval. ACM Press and Addison-Wesley, New York.]]

Digital Library

[4]

BARWISE, J. 1989. The Situation in Logic.InCLSI Lecture Notes 17, Stanford, Calif.]]

[5]

BARWISE, J. AND ETCHEMENDY, J. 1990. Information, Infons and Inference. In Situation Theory and its Applications, R. Cooper, et al. Eds., CLSI Lecture Notes 1, 33-78.]]

[6]

BARWISE,J.AND SELIGMAN, J. 1997. Information Flow-The Logic of Distributed Systems. Cambridge University Press, Cambridge, MA.]]

Digital Library

[7]

BRUZA, P. D. AND HUIBERS, T. W. C. 1996. A study of aboutness in information retrieval. Artif. Intell. Rev. 10, 1-27.]]

Digital Library

[8]

BRUZA,P.D.AND HUIBERS, T. W. C. 1994. Investigating aboutness axioms using information fields. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval (Dublin), 112-121.]]

Digital Library

[9]

BRUZA,P.D.AND LALMAS, M. 1996. Logic based information retrieval: Is it really worth it? In Proceedings of WIRUL 96, the Second Workshop on Information Retrieval, Uncertainty and Logic (Glasgow).]]

[10]

BRUZA,P.D.AND VAN LINDER, B. 1998. Preferential models of query by navigation. In Information Retrieval, Logic and Uncertainty, F. Crestani, M. Lalmas, and C. J. van Rijsbergen, Eds. Springer- Verlag, New York.]]

[11]

BRUZA,P.D.AND SONG, D. 2001. Informational inference via information flow. In Proceedings of the Twelfth International Workshop on Database and Expert Systems Applications (Munich, September 3-7), 327-341.]]

Digital Library

[12]

BRUZA,P.D.,SONG,D.,AND WONG, K. F. 2000a. Aboutness from commonsense perspective. J. Am. Soc. Inf. Sci. (JASIS) 51, 12, 1090-1105.]]

Digital Library

[13]

BRUZA,P.D.,SONG, D., WONG,K.F.,AND CHENG, C. H. 2000b. Commonsense aboutness for information retrieval. In Proceedings of the International Conference on Advances in Intelligent Systems: Theory and Applications (AISTA 2000) (Canberra, Australia, February 2-4), 317- 324.]]

[14]

BRUZA,P.D.,SONG,D.,AND WONG, K. F. 1999. Fundamental properties of aboutness. In Proceedings of the Twenty-Second Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval (SIGIR'99) (Berkeley, Calif., August 15-19).]]

Digital Library

[15]

CRESTANI, F., RUTHVEN, I., SANDERSON, M., AND VAN RIJSBERGEN, C. J. 1995. The troubles with using a logical model of IR on a large collection of documents. In Proceedings of the Fourth Text Retrieval Conference (TREC-4), 509-526.]]

[16]

CRESTANI,F.AND VAN RIJSBERGEN, C. J. 1995a. Information retrieval by logic imaging. J. Doc. 51, 1, 3-17.]]

[17]

CRESTANI,F.AND VAN RIJSBERGEN, C. J. 1995b. Probability kinematics in information retrieval. In Proceedings of the ACMSIGIR Conference on Research and Development in Information Retrieval (Seattle, Wash.), 291-299.]]

Digital Library

[18]

CRESTANI, F., LALMAS, M., AND VAN RIJSBERGEN, C. J. 1998. A study of probability kinematics in information retrieval. ACM Trans. Inf. Syst. 16,3.]]

Digital Library

[19]

CRESTANI,F.,AND VAN RIJSBERGEN, C. J. 1998. A study of Probability kinematics in information retrieval. ACM Trans. Inf. Sys., 16, 3.]]

Digital Library

[20]

DUBOIS, D., FARINAS DEL CERRO, L., HERZIG, A., AND PRADE, H. 1997. Qualitative relevance and independence: A roadmap. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI-97), 62-67.]]

[21]

FRAKES,W.B.AND BAEZA-YATES, R., EDS. 1992. Information Retrieval, Data Structures & Algorithms. Prentice-Hall, Englewood, Cliffs, N.J.]]

Digital Library

[22]

HUIBERS, T. W. C. 1996. An axiomatic theory for information retrieval. PhD thesis, Utrecht University, The Netherlands.]]

[23]

HUIBERS,T.W.C.AND BRUZA, P. D. 1994. Situations, a general framework for studying information retrieval. In Information Retrieval: New Systems and Current Research, vol. 2, Taylor Graham, Ed.]]

[24]

HUIBERS,T.W.C.,LALMAS, M., AND VAN RIJSBERGEN, C. J. 1996. Information retrieval and situation theory. SIGIR Forum 30, 1, 11-25.]]

Digital Library

[25]

HUNTER, A. 1995. Using default logic in information retrieval. In Symbolic and Quantitative Approaches to Uncertainty, Lecture Notes in Computer Science, vol. 946, Springer-Verlag, New York, 235-242.]]

Digital Library

[26]

HUNTER, A. 1996. Intelligent text handling using default logic, In Proceedings of the Eighth IEEE International Conference on Tools with Artificial Intelligence (TAI'96), IEEE Computer Society Press, Los Alamitos, Calif., 34-40.]]

Digital Library

[27]

HUTCHINS, W. J. 1977. On the problem of 'aboutness' in document analysis. J. Inf. 1, 1, 17-35.]]

[28]

KRAUS, S., LEHMANN,D.,AND MAGIDOR, M. 1990. Nonmonotonic reasoning, preferential models and cumulative logics. Artif. Intell. 44, 167-207.]]

Digital Library

[29]

LALMAS, M. 1998. Logical models in information retrieval: Introduction and overview. Inf. Proc. Manage. 34, 1, 19-33.]]

Digital Library

[30]

LALMAS, M. 1997. Information retrieval and Dempster-Shafer's theory of evidence. In Applications of Uncertainty Formalisms, A. Hunter and S. Parson, Eds., Chapter 8, Lecture Notes in Computer Science, Springer-Verlag, New York.]]

[31]

LALMAS, M. 1996. Theories of information and uncertainty for the modeling of information retrieval: An application of situation theory and Dempster-Shafer's theory of evidence. PhD thesis, University of Glasgow.]]

[32]

LALMAS,M.AND BRUZA, P. D. 1998. The use of logic in information retrieval modeling. Knowl. Eng. Rev. In press.]]

[33]

LANDMAN, F. W. 1986. Towards a Theory of Information. The Status of Partial Objects in Semantics. Foris, Dordrecht.]]

[34]

LOSEE R. M. 1998. Text Retrieval and Filtering: Analytic Models of Performance. Kluwer, Hinghamn, Mass.]]

Digital Library

[35]

LOSEE R. M. 1997. Comparing Boolean and probabilistic information retrieval systems across queries and disciplines. J. Am. Soc. Inf. Sci. 48, 2, 143-156.]]

Digital Library

[36]

MARON, M. E. 1977. On indexing, retrieval and the meaning of about. J. Am. Soc. Inf. Sci. 28,1, 38-43.]]

[37]

NIE, J. 1992. Towards a probabilistic modal logic for semantic-based information retrieval. In Proceedings of the ACM-SIGIR Conference on Research and Development in Information Retrieval (Copenhagen), 140-151.]]

Digital Library

[38]

NIE, J. 1989. An information retrieval model based on modal logic. Inf. Proc. Manage. 25,5, 477-491.]]

[39]

NIE, J., BRISEBOIS, M., AND LEPAGE, F. 1995. Information retrieval as counterfactual. Comput. J. 38, 8, 643-657.]]

[40]

PROPER,H.A.AND BRUZA, P. D. 1999. What is information discovery about? J. Am. Soc. Inf. Sci. 50, 9, 737-750.]]

Digital Library

[41]

VAN RIJSBERGEN, C. J. 1993. The state of information retrieval: Logic and information. Comput. Bull., February.]]

[42]

VAN RIJSBERGEN, C. J. 1986a. A non-classical logic for information retrieval. Comput. J. 29,6, 481-485.]]

[43]

VAN RIJSBERGEN, C. J. 1986b. A new theoretical framework for information retrieval. In Proceedings of the Ninth International SIGIR Conference in Research and Development in Information Retrieval, 194-200.]]

Digital Library

[44]

VAN RIJSBERGEN, C. J. 1989. Towards an information logic. In Proceedings of the 12th International SIGIR Conference in Research and Development in Information Retrieval, pp. 77-86.]]

Digital Library

[45]

VAN RIJSBERGEN, C. J. 1979. Information Retrieval, Second edition. Butterworths, London.]]

Digital Library

[46]

VAN RIJSBERGEN,C.J.AND LALMAS, M. 1996. An information calculus for information retrieval. J. Am. Soc. Inf. Sci. 47, 5, 385-398.]]

Digital Library

[47]

ROELLEKE,T.AND FUHR, N. 1996. Retrieval of complex objects using a four-valued logic. In Proceedings of the ACM-SIGIR Conference on Research and Development in Information Retrieval (Zurich), 206-214.]]

Digital Library

[48]

SALTON, G. 1988. Automatic Text Processing. Addison-Wesley, Reading, Mass.]]

Digital Library

[49]

SEBASTIANI, F. 1998. On the role of logic in information retrieval. Inf. Proc. Manage. 34, 1, 1-18.]]

Digital Library

[50]

SINGHAL, A., BUCKLEY,C.,AND MITRA, M. 1996. Pivoted document length normalization. In Proceedings of the Nineteenth ACM-SIGIR Conference on Research and Development in Information Retrieval (Zurich), 21-29.]]

Digital Library

[51]

SONG, D. 2000. A commonsense aboutness theory for information retrieval modeling. PhD thesis. The Chinese University of Hong Kong.]]

Digital Library

[52]

SONG,D.AND BRUZA, P. D. 2001. Discovering information flow using a high dimensional conceptual space. In Proceedings of the 24th Annual International Conference on Research and Development in Information Retrieval (SIGIR'01) (New Orleans, La., Sept. 9-13), 327-333.]]

Digital Library

[53]

SONG, D., WONG,K.F.,BRUZA,P.D.,AND CHENG, C. H. 2000a. Fundamental properties of the core matching functions for information retrieval. In Proceedings of the Thirteenth International Florida Artificial Intelligence Society Conference (FLAIRS 2000) (Orlando, Fl., May 22-24), 118-122.]]

Digital Library

[54]

SONG, D., WONG,K.F.,BRUZA,P.D.,AND CHENG, C. H. 2000b. Towards a commonsense aboutness theory for information retrieval modeling. In Proceedings of the Fourth World Multiconference on Systemics, Cybernetics and Informatics (SCI 2000) (Orlando, Fl., July) 23-26.]]

[55]

SONG, D., WONG,K.F.,BRUZA,P.D.,AND CHENG, C. H. 1999. Towards functional benchmarking of information retrieval models. In Proceedings of the Twelfth International Florida Artificial Intelligence Society Conference (FLAIRS'99) (Orlando, Fl., May 3-5), 389-393.]]

Digital Library

[56]

TURTLE,H.R.AND CROFT, W. B. 1992. A comparison of text retrieval models. Comput. J. 35,3, 279-290.]]

Digital Library

[57]

XU, J. AND CROFT, B. 1996. Query Expansion Using Local and Global Document Analysis. In Proceedings of the 19th International SIGIR Conference in Research and Development in Information Retrieval, pp. 4-11.]]

Digital Library

Cited By

Luk R(2020)Why is Information Retrieval a Scientific Discipline?Foundations of Science10.1007/s10699-020-09685-x27:2(427-453)Online publication date: 28-Jun-2020
https://doi.org/10.1007/s10699-020-09685-x
Ermakova LMothe J(2016)Document re-ranking based on topic-comment structure2016 IEEE Tenth International Conference on Research Challenges in Information Science (RCIS)10.1109/RCIS.2016.7549352(1-10)Online publication date: Jun-2016
https://doi.org/10.1109/RCIS.2016.7549352
Julien CTirilly PLeide JGuastavino C(2012)Constructing a true LCSH tree of a science and engineering collectionJournal of the American Society for Information Science and Technology10.1002/asi.2274963:12(2405-2418)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1002/asi.22749
Show More Cited By

Index Terms

Application of aboutness to functional benchmarking in information retrieval

Recommendations

A commonsense aboutness theory for information retrieval modeling
Application-specific benchmarking
Negations and document length in logical retrieval

An unsolved problem in logic-based information retrieval is how to obtain automatically logical representations for documents and queries. This problem limits the impact of logical models for information retrieval because their full expressive power ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 19, Issue 4

October 2001

120 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/502795

Issue’s Table of Contents

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2001

Published in TOIS Volume 19, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
1,156
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Luk R(2020)Why is Information Retrieval a Scientific Discipline?Foundations of Science10.1007/s10699-020-09685-x27:2(427-453)Online publication date: 28-Jun-2020
https://doi.org/10.1007/s10699-020-09685-x
Ermakova LMothe J(2016)Document re-ranking based on topic-comment structure2016 IEEE Tenth International Conference on Research Challenges in Information Science (RCIS)10.1109/RCIS.2016.7549352(1-10)Online publication date: Jun-2016
https://doi.org/10.1109/RCIS.2016.7549352
Julien CTirilly PLeide JGuastavino C(2012)Constructing a true LCSH tree of a science and engineering collectionJournal of the American Society for Information Science and Technology10.1002/asi.2274963:12(2405-2418)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1002/asi.22749
Smith GBrien CAshman H(2012)Evaluating implicit judgments from image search clickthrough dataJournal of the American Society for Information Science and Technology10.1002/asi.2274263:12(2451-2462)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1002/asi.22742
Ye ZHuang JHe BLin H(2012)Mining a multilingual association dictionary from Wikipedia for cross-language information retrievalJournal of the American Society for Information Science and Technology10.1002/asi.2269663:12(2474-2487)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1002/asi.22696
Blanke TLalmas MHuibers T(2012)A framework for the theoretical evaluation of XML retrievalJournal of the American Society for Information Science and Technology10.1002/asi.2267463:12(2463-2473)Online publication date: 1-Dec-2012
https://dl.acm.org/doi/10.1002/asi.22674
Fang HTao TZhai C(2011)Diagnostic Evaluation of Information Retrieval ModelsACM Transactions on Information Systems10.1145/1961209.196121029:2(1-42)Online publication date: 1-Apr-2011
https://dl.acm.org/doi/10.1145/1961209.1961210
Blanke TLalmas M(2011)Specificity aboutness in XML retrievalInformation Retrieval10.1007/s10791-010-9144-614:1(68-88)Online publication date: 1-Feb-2011
https://dl.acm.org/doi/10.1007/s10791-010-9144-6
Blanke TLalmas M(2009)Specificity Aboutness in XML RetrievalProceedings of the 2nd International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory10.1007/978-3-642-04417-5_16(176-187)Online publication date: 3-Sep-2009
https://dl.acm.org/doi/10.1007/978-3-642-04417-5_16
Lau RBruza PSong D(2008)Towards a belief-revision-based adaptive and context-sensitive information retrieval systemACM Transactions on Information Systems10.1145/1344411.134441426:2(1-38)Online publication date: 8-Apr-2008
https://dl.acm.org/doi/10.1145/1344411.1344414
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents