Article

Indexing shared content in information retrieval systems

Authors:

Andrei Z. Broder,

Marcus Fontoura,

Michael Herscovici,

John McPherson,

Eugene ShekitaAuthors Info & Claims

EDBT'06: Proceedings of the 10th international conference on Advances in Database Technology

Pages 313 - 330

https://doi.org/10.1007/11687238_21

Published: 26 March 2006 Publication History

Abstract

Modern document collections often contain groups of documents with overlapping or shared content. However, most information retrieval systems process each document separately, causing shared content to be indexed multiple times. In this paper, we describe a new document representation model where related documents are organized as a tree, allowing shared content to be indexed just once. We show how this representation model can be encoded in an inverted index and we describe algorithms for evaluating free-text queries based on this encoding. We also show how our representation model applies to web, email, and newsgroup search. Finally, we present experimental results showing that our methods can provide a significant reduction in the size of an inverted index as well as in the time to build and query it.

References

[1]

R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, 1999.

Digital Library

[2]

M. Bilenko and R. J. Mooney. Adaptive duplicate detection using learnable string similarity measures. In KDD '03, pages 39-48, 2003.

Digital Library

[3]

S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. In WWW '98, pages 107-117, 1998.

Digital Library

[4]

A. Z. Broder. Identifying and filtering near-duplicate documents. In CPM '00, pages 1-10, 2000.

Digital Library

[5]

A. Z. Broder, D. Carmel, M. Herscovici, A. Soffer, and J. Zien. Efficient query evaluation using a two-level retrieval process. In CIKM '03, pages 426-434, 2003.

Digital Library

[6]

A. Z. Broder, S. C. Glassman, M. S. Manasse, and G. Zweig. Syntactic clustering of the web. In WWW '97, pages 1157-1166, 1997.

Digital Library

[7]

D. Carmel, D. Cohen, R. Fagin, E. Farchi, M. Herscovici, Y. S. Maarek, and A. Soffer. Static index pruning for information retrieval systems. In SIGIR '01, pages 43-50, 2001.

Digital Library

[8]

J. Cho, N. Shivakumar, and H. Garcia-Molina. Finding replicated web collections. In SIGMOD '00, pages 355-366, 2000.

Digital Library

[9]

E. S. de Moura, C. F. dos Santos, D. R. Fernandes, A. S. Silva, P. Calado, and M. A. Nascimento. Improving web search efficiency via a locality based static pruning method. In WWW '05, pages 235-244, 2005.

Digital Library

[10]

M. Fontoura, E. J. Shekita, J. Y. Zien, S. Rajagopalan, and A. Neumann. High performance index build algorithms for intranet search engines. In VLDB '04, pages 1158-1169, 2004.

Digital Library

[11]

H. Garcia-Molina, J. Ullman, and J. Widom. Database System Implementation. Prentice Hall, 2000.

Digital Library

[12]

Gmail. http://gmail.google.com/gmail/help/about.html.

[13]

S. Heinz and J. Zobel. Efficient single-pass index construction for text databases. JASIST, 54(8), 2003.

Digital Library

[14]

B. Klimt and Y. Yang. The Enron corpus: A new dataset for email classification research. In European Conference on Machine Learning, 2004.

Digital Library

[15]

X. Long and T. Suel. Optimized query execution in large search engines with global page ordering. In VLDB '03, pages 129-140, 2003.

Digital Library

[16]

S. Melnik, S. Raghavan, B. Yang, and H. Garcia-Molina. Building a distributed full-text index for the web. In WWW '01, pages 396-406, 2001.

Digital Library

[17]

A. Moffat and J. Zobel. Compression and fast indexing for multi-gigabyte text databases. Australian Computer Journal, 26(1), 1994.

Digital Library

[18]

F. Scholer, H. E. Williams, J. Yiannis, and J. Zobel. Compression of inverted indexes for fast query evaluation. In SIGIR '02, pages 222-229, 2002.

Digital Library

[19]

R. Stata, P. Hunt, and M. G. Thiruvalluvan. The Bloomba personal content database. In VLDB '04, pages 1214-1223, 2004.

Digital Library

[20]

H. Turtle and J. Flood. Query evaluation: strategies and optimizations. Inf. Proc. Management, 31(6):831-850, 1995.

Digital Library

[21]

I. Witten, A. Moffat, and T. Bell. Managing Gigabytes. Morgan Kaufmann, 1999.

[22]

Z. Zhang. The behavior of duplicate pages on the world wide web. Submitted to CIKM, 2005.

[23]

J. Zobel and A. Moffat. Adding compression to a full-text retrieval system. Software - Practice & Experience, 25(8), 1995.

Digital Library

Cited By

Bendersky MWang XNajork MMetzler DSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)Search and Discovery in Personal Email CollectionsProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3501393(1617-1619)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3501393
Jin XAgun DYang TWu QShen YZhao SMukhopadhyay SZhai CBertino ECrestani FMostafa JTang JSi LZhou XChang YLi YSondhi P(2016)Hybrid Indexing for Versioned Document Search with Cluster-based RetrievalProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983733(377-386)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2983323.2983733
Kanhabua NAnand APerego RSebastiani FAslam JRuthven IZobel J(2016)Temporal Information RetrievalProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2914805(1235-1238)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2914805
Show More Cited By

Indexing shared content in information retrieval systems
1. Information systems
  1. Information retrieval
    1. Search engine architectures and scalability

Recommendations

An efficient indexing method for content-based image retrieval

In this paper, we propose an efficient indexing method for content-based image retrieval. The proposed method introduces the ordered quantization to increase the distinction among the quantized feature descriptors. Thus, the feature point ...
Improving high-dimensional indexing for content-based image retrieval
Inverted file-based indexing for efficient multimedia information retrieval in metric spaces
SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

Content-based similarity search is an important task in multimedia information retrieval (IR). Here, metric space access methods (MAMs) can be applied. They are purely based on the use of a metric distance. No assumption is made about the representation ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

EDBT'06: Proceedings of the 10th international conference on Advances in Database Technology

March 2006

1204 pages

ISBN:3540329609

Editors:
Yannis Ioannidis
University of Athens, Greece
,
Marc H. Scholl
University of Konstanz, Konstanz, Germany
,
Joachim W. Schmidt
Sustainable Content Logistics Centre, Hamburg, Germany
,
Florian Matthes
Chair of Software Engineering for Business Information Systems, Technische Universität München, Boltzmannstraße 3, Garching b. München
,
Mike Hatzopoulos
Department of Informatics, University of Athens Panepistimiopolis, Boltzmannstraße 3, Athens, Greece

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 26 March 2006

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bendersky MWang XNajork MMetzler DSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)Search and Discovery in Personal Email CollectionsProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3501393(1617-1619)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3501393
Jin XAgun DYang TWu QShen YZhao SMukhopadhyay SZhai CBertino ECrestani FMostafa JTang JSi LZhou XChang YLi YSondhi P(2016)Hybrid Indexing for Versioned Document Search with Cluster-based RetrievalProceedings of the 25th ACM International on Conference on Information and Knowledge Management10.1145/2983323.2983733(377-386)Online publication date: 24-Oct-2016
https://dl.acm.org/doi/10.1145/2983323.2983733
Kanhabua NAnand APerego RSebastiani FAslam JRuthven IZobel J(2016)Temporal Information RetrievalProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval10.1145/2911451.2914805(1235-1238)Online publication date: 7-Jul-2016
https://dl.acm.org/doi/10.1145/2911451.2914805
He JSuel THersh WCallan JMaarek YSanderson M(2012)Optimizing positional index structures for versioned document collectionsProceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval10.1145/2348283.2348319(245-254)Online publication date: 12-Aug-2012
https://dl.acm.org/doi/10.1145/2348283.2348319
Claude FFariña AMartínez-Prieto MNavarro G(2011)Indexes for highly repetitive document collectionsProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063646(463-468)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063646
He JSuel TMa WNie JBaeza-Yates RChua TCroft W(2011)Faster temporal range queries over versioned textProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval10.1145/2009916.2009993(565-574)Online publication date: 24-Jul-2011
https://dl.acm.org/doi/10.1145/2009916.2009993
He JZeng JSuel THuang JKoudas NJones GWu XCollins-Thompson KAn A(2010)Improved index compression techniques for versioned document collectionsProceedings of the 19th ACM international conference on Information and knowledge management10.1145/1871437.1871594(1239-1248)Online publication date: 26-Oct-2010
https://dl.acm.org/doi/10.1145/1871437.1871594
U LMamoulis NBerberich KBedathur SElmagarmid AAgrawal D(2010)Durable top-k search in document archivesProceedings of the 2010 ACM SIGMOD International Conference on Management of data10.1145/1807167.1807228(555-566)Online publication date: 6-Jun-2010
https://dl.acm.org/doi/10.1145/1807167.1807228
He JYan HSuel TCheung DSong IChu WHu XLin J(2009)Compact full-text indexing of versioned document collectionsProceedings of the 18th ACM conference on Information and knowledge management10.1145/1645953.1646008(415-424)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1645953.1646008
Yan HDing SSuel TQuemada JLeón GMaarek YNejdl W(2009)Inverted index compression and query processing with optimized document orderingProceedings of the 18th international conference on World wide web10.1145/1526709.1526764(401-410)Online publication date: 20-Apr-2009
https://dl.acm.org/doi/10.1145/1526709.1526764
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents