research-article

TF-IDF uncovered: a study of theories and probabilities

Authors:

Thomas Roelleke,

Jun WangAuthors Info & Claims

SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

Pages 435 - 442

https://doi.org/10.1145/1390334.1390409

Published: 20 July 2008 Publication History

Abstract

Interpretations of TF-IDF are based on binary independence retrieval, Poisson, information theory, and language modelling. This paper contributes a review of existing interpretations, and then, TF-IDF is systematically related to the probabilities P(q|d) and P(d|q). Two approaches are explored: a space of independent, and a space of disjoint terms. For independent terms, an "extreme" query/non-query term assumption uncovers TF-IDF, and an analogy of P(d|q) and the probabilistic odds O(r|d, q) mirrors relevance feedback. For disjoint terms, a relationship between probability theory and TF-IDF is established through the integral + 1/x dx = log x. This study uncovers components such as divergence from randomness and pivoted document length to be inherent parts of a document-query independence (DQI) measure, and interestingly, an integral of the DQI over the term occurrence probability leads to TF-IDF.

References

[1]

Akiko Aizawa. An information-theoretic perspective of tf-idf measures. Information Processing and Management, 39:45--65, January 2003.

Digital Library

[2]

Gianni Amati and C. J. van Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM TOIS, 20(4):357--389, October 2002.

Digital Library

[3]

K. Church and W Gale. Inverse document frequency (idf): A measure of deviation from poisson. In Third Workshop on Very Large Corpora, pages 121--130, 1995.

[4]

W.B. Croft and D.J. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35:285--295, 1979.

[5]

Arjen de Vries and Thomas Roelleke. Relevance information: A loss of entropy but a gain for idf? In ACM SIGIR, Salvador, Brazil, 2005.

Digital Library

[6]

David A. Grossman and Ophir Frieder. Information Retrieval. Algorithms and Heuristics, 2nd ed., volume 15 of The Information Retrieval Series. Springer, 2004.

Digital Library

[7]

Djoerd Hiemstra. A probabilistic justification for using tf.idf term weighting in information retrieval. International Journal on Digital Libraries, 3(2):131--139, 2000.

[8]

John Lafferty and ChengXiang Zhai. Probabilistic Relevance Models Based on Document and Query Generation, chapter 1. Kluwer, 2003.

[9]

Qiaozhu Mei, Hui Fang, and ChengXiang Zhai. A study of Poisson query generation model for information retrieval. In ACM SIGIR, pages 319--326, New York, 2007.

Digital Library

[10]

J.M. Ponte and W.B. Croft. A language modeling approach to information retrieval. ACM SIGIR, pages 275--281, 1998.

Digital Library

[11]

S. E. Robertson and S. Walker. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. ACM SIGIR, pages 232--241, 1994.

Digital Library

[12]

S.E. Robertson. Understanding inverse document frequency: On theoretical arguments for idf. Journal of Documentation, 60:503--520, 2004.

[13]

S.E. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129--146, 1976.

[14]

Thomas Roelleke. A frequency-based and a Poisson-based probability of being informative. In ACM SIGIR, pages 227--234, Toronto, Canada, 2003.

Digital Library

[15]

Thomas Roelleke and Jun Wang. A parallel derivation of probabilistic information retrieval models. In ACM SIGIR, pages 107--114, Seattle, USA, 2006.

Digital Library

[16]

S.K.M. Wong and Y.Y. Yao. On modeling information retrieval with probabilistic inference. ACM TOIS, 13(1):38--68, 1995.

Digital Library

[17]

Hugo Zaragoza, Djoerd Hiemstra, and Michael E. Tipping. Bayesian extension to the language model for ad hoc information retrieval. In ACM SIGIR, pages 4--9, Toronto, Canada, 2003.

Digital Library

Cited By

Er RÇoğalmış K(2024)Determining Whether a Turkish Text is Produced by Artificial Intelligence or Human2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI)10.1109/ICMI60790.2024.10585770(1-5)Online publication date: 13-Apr-2024
https://doi.org/10.1109/ICMI60790.2024.10585770
Rahardi MArsya MAminuddin AAbdulloh FMiah MPratikno H(2024)Sentiment Analysis of Israeli-Palestinian Conflict on Indonesian Tweets Using Machine Learning2024 International Conference on Electrical Engineering and Computer Science (ICECOS)10.1109/ICECOS63900.2024.10791149(59-64)Online publication date: 25-Sep-2024
https://doi.org/10.1109/ICECOS63900.2024.10791149
Abdelaal MYayak AKlede KSchöning H(2024)ReClean: Reinforcement Learning for Automated Data Cleaning in ML Pipelines*2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW)10.1109/ICDEW61823.2024.00048(324-330)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDEW61823.2024.00048
Show More Cited By

Index Terms

TF-IDF uncovered: a study of theories and probabilities
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Interpreting TF-IDF term weights as making relevance decisions

A novel probabilistic retrieval model is presented. It forms a basis to interpret the TF-IDF term weights as making relevance decisions. It simulates the local relevance decision-making for every location of a document, and combines all of these “local” ...
R-tfidf, a Variety of tf-idf Term Weighting Strategy in Document Categorization
SKG '11: Proceedings of the 2011 Seventh International Conference on Semantics, Knowledge and Grids

Term weighting strategy plays an essential role in the areas related to text processing such as text categorization and information retrieval. In such systems, term frequency, inverse document frequency, and document length normalization are important ...
An information-theoretic perspective of tf—idf measures

This paper presents a mathematical definition of the "probability-weighted amount of information" (PWI), a measure of specificity of terms in documents that is based on an information-theoretic view of retrieval events. The proposed PWI is expressed as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

July 2008

934 pages

ISBN:9781605581644

DOI:10.1145/1390334

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Mun-Kew Leong
National Library Board, Singapore
,
Program Chairs:
Syung Hyon Myaeng
Information and Communications University, Korea
,
Douglas W. Oard
University of Maryland, College Park, USA
,
Fabrizio Sebastiani
Consiglio Nazionale delle Ricerche, Italy

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '08

Sponsor:

SIGIR '08: The 31st Annual International ACM SIGIR Conference

July 20 - 24, 2008

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
2,823
Total Downloads

Downloads (Last 12 months)206
Downloads (Last 6 weeks)33

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Er RÇoğalmış K(2024)Determining Whether a Turkish Text is Produced by Artificial Intelligence or Human2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI)10.1109/ICMI60790.2024.10585770(1-5)Online publication date: 13-Apr-2024
https://doi.org/10.1109/ICMI60790.2024.10585770
Rahardi MArsya MAminuddin AAbdulloh FMiah MPratikno H(2024)Sentiment Analysis of Israeli-Palestinian Conflict on Indonesian Tweets Using Machine Learning2024 International Conference on Electrical Engineering and Computer Science (ICECOS)10.1109/ICECOS63900.2024.10791149(59-64)Online publication date: 25-Sep-2024
https://doi.org/10.1109/ICECOS63900.2024.10791149
Abdelaal MYayak AKlede KSchöning H(2024)ReClean: Reinforcement Learning for Automated Data Cleaning in ML Pipelines*2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW)10.1109/ICDEW61823.2024.00048(324-330)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDEW61823.2024.00048
Yellapu JReddy KLakshmi GVaibhavi KVivek CKishore T(2024)Spam Comment Detection Using the Ensemble Technique2023 4th International Conference on Intelligent Technologies (CONIT)10.1109/CONIT61985.2024.10626863(1-7)Online publication date: 21-Jun-2024
https://doi.org/10.1109/CONIT61985.2024.10626863
Fronzetti Colladon AMatarazzo MPetruzzellis LVisentin M(2024)Exploring user reactions to luxury brand videos on YouTube: a comparative study of influencers and brand-official channelsInternational Journal of Advertising10.1080/02650487.2024.2367316(1-23)Online publication date: 19-Jun-2024
https://doi.org/10.1080/02650487.2024.2367316
Asdrubali FFronzetti Colladon ASegneri LGandola D(2024)LCA and energy efficiency in buildings: Mapping more than twenty years of researchEnergy and Buildings10.1016/j.enbuild.2024.114684(114684)Online publication date: Aug-2024
https://doi.org/10.1016/j.enbuild.2024.114684
Singh Samra RBarcellos M(2023)DDoS2Vec: Flow-Level Characterisation of Volumetric DDoS Attacks at ScaleProceedings of the ACM on Networking10.1145/36291351:CoNEXT3(1-25)Online publication date: 28-Nov-2023
https://dl.acm.org/doi/10.1145/3629135
Li TLi YZhang MTarkoma SHui P(2023)You Are How You Use Apps: User Profiling Based on Spatiotemporal App Usage BehaviorACM Transactions on Intelligent Systems and Technology10.1145/359721214:4(1-21)Online publication date: 21-Jul-2023
https://dl.acm.org/doi/10.1145/3597212
Kang YDu HForkan AJayaraman PAryani ASellis T(2023)ExpFinder: A hybrid model for expert finding from text-based expertise dataExpert Systems with Applications10.1016/j.eswa.2022.118691211(118691)Online publication date: Jan-2023
https://doi.org/10.1016/j.eswa.2022.118691
Sheridan POnsjö M(2023)The hypergeometric test performs comparably to TF-IDF on standard text analysis tasksMultimedia Tools and Applications10.1007/s11042-023-16615-zOnline publication date: 8-Sep-2023
https://doi.org/10.1007/s11042-023-16615-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents