article

Free access

Self-indexing inverted files for fast text retrieval

Authors:

Alistair Moffat and

Justin ZobelAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 14, Issue 4

Pages 349 - 379

https://doi.org/10.1145/237496.237497

Published: 01 October 1996 Publication History

Abstract

Query-processing costs on large text databases are dominated by the need to retrieve and scan the inverted list of each query term. Retrieval time for inverted lists can be greatly reduced by the use of compression, but this adds to the CPU time required. Here we show that the CPU component of query response time for conjunctive Boolean queries and for informal ranked queries can be similarly reduced, at little cost in terms of storage, by the inclusion of an internal index in each compressed inverted list. This method has been applied in a retrieval system for a collection of nearly two million short documents. Our experimental results show that the self-indexing strategy adds less than 20% to the size of the compressed inverted file, which itself occupies less than 10% of the indexed text, yet can reduce processing time for Boolean queries of 5-10 terms to under one fifth of the previous cost. Similarly, ranked queries of 40-50 terms can be evaluated in as little as 25% of the previous time, with little or no loss of retrieval effectiveness.

References

[1]

BELL,T.C.,MOFFAT, A., NEVILL-MANNING,C.G.,WITTEN,I.H.,AND ZOBEL, J. 1993. Data compression in full-text retrieval systems. J. Am. Soc. Inf. Sci. 44, 9 (Oct.), 508-531.

[2]

BOOKSTEIN, A., KLEIN,S.T.,AND RAITA, T. 1992. Model based concordance compression. In Proceedings of the IEEE Data Compression Conference, J. A. Storer and M. Cohn, Eds. IEEE Computer Society Press, Los Alamitos, Calif., 82-91.

[3]

BUCKLEY,C.AND LEWIT, A. F. 1985. Optimization of inverted vector searches. In Proceed-ings of the ACM-SIGIR International Conference on Research and Development in Informa-tion Retrieval (Montreal, Canada, June). ACM Press, New York, 97-110.

[4]

CHOUEKA, Y., FRAENKEL,A.S.,KLEIN,S.T.,AND SEGAL, E. 1987. Improved techniques for processing queries in full-text systems. In Proceedings of the ACM-SIGIR International Conference on Research and Development in Information Retrieval (New Orleans, La.). ACM Press, New York, 306-315.

[5]

CHOUEKA, Y., FRAENKEL,A.S.,AND KLEIN, S. T. 1988. Compression of concordances in full-text retrieval systems. In Proceedings of the ACM-SIGIR International Conference on Research and Development in Information Retrieval (Grenoble, France, June). ACM Press, New York, 597-612.

[6]

CORMEN,T.H.,LEISERSON,C.E.,AND RIVEST, R. L. 1990. Introduction to Algorithms. The MIT Press, Cambridge, Mass.

[7]

CROFT,W.B.AND SAVINO, P. 1988. Implementing ranking strategies using text signatures. ACM Trans. Off. Inf. Syst. 6, 1 (Jan.), 42-62.

[8]

ELIAS, P. 1975. Universal codeword sets and representations of the integers. IEEE Trans. Inf. Theory IT-21, 2 (Mar.), 194-203.

[9]

ELMASRE,R.AND NAVATHE, S. B. 1989. Fundamentals of Database Systems. Benjamin/ Cummins, Menlo Park, Calif.

[10]

FALOUTSOS, C. 1985a. Access methods for text. ACM Comput. Surv. 17, 1, 49-74.

[11]

FALOUTSOS, C. 1985b. Signature files: Design and performance comparison of some signa-ture extraction methods. In Proceedings of the ACM-SIGIR International Conference on Research and Development in Information Retrieval (Montreal, Canada, June). ACM Press, New York, 63-82.

[12]

FALOUTSOS,C.AND CHRISTODOULAKIS, S. 1984. Signature files: An access method for docu-ments and its analytical performance evaluation. ACM Trans. Off. Inf. Syst. 2, 4 (Oct.), 267-288.

[13]

FOX,E.A.,HARMAN,D.K.,BAEZA-YATES, R., AND LEE, W. C. 1992. Inverted files. In Information Retrieval: Data Structures and Algorithms, W. B. Frakes and R. Baeza-Yates, Eds. Prentice-Hall, Englewood Cliffs, N.J., 28-43.

[14]

FRAENKEL,A.S.AND KLEIN, S. T. 1985. Novel compression of sparse bit-strings:Prelimi-nary report. In Combinatorial Algorithms on Words, A. Apostolico and Z. Galil, Eds. NATO ASI Series F, vol. 12. Springer-Verlag, Berlin, 169-183.

[15]

GALLAGER,R.G.AND VAN VOORHIS, D. C. 1975. Optimal source codes for geometrically distributed integer alphabets. IEEE Trans. Inf. Theory IT-21, 2 (Mar.), 228-230.

[16]

GOLOMB, S. W. 1966. Run-length encodings. IEEE Trans. Inf. Theory IT-12, 3 (July), 399-401.

[17]

HARMAN, D. K., Ed. 1992a. Proceedings of TREC Text Retrieval Conference (Washington, D.C., Nov.). National Institute of Standards Special Publication 500-207. NIST, Washing-ton, D.C.

[18]

HARMAN, D. K. 1992b. Ranking algorithms. In Information Retrieval: Data Structures and Algorithms, W. B. Frakes and R. Baeza-Yates, Eds. Prentice-Hall, Englewood Cliffs, N.J., 363-392.

[19]

HARMAN,D.K.AND CANDELA, G. 1990. Retrieving records from a gigabyte of text on a mini-computer using statistical ranking. J. Am. Soc. Inf. Sci. 41, 8, 581-589.

[20]

HWANG,F.K.AND LIN, S. 1972. A simple algorithm for merging two disjoint linearly ordered sets. SIAM J. Comput. 1, 1, 31-39.

[21]

KENT,A.J.,SACKS-DAVIS, R., AND RAMAMOHANARAO, K. 1990. A signature file scheme based on multiple organizations for indexing very large text databases. J. Am. Soc. Inf. Sci. 41, 7, 508-534.

[22]

KLEIN,S.T.,BOOKSTEIN, A., AND DEERWESTER, S. 1989. Storing text retrieval systems on CD-ROM: Compression and encryption considerations. ACM Trans. Inf. Syst. 7, 3 (July), 230-245.

[23]

LINOFF,G.AND STANFILL, C. 1993. Compression of indexes with full positional information in very large text databases. In Proceedings of the ACM-SIGIR International Conference on Research and Development in Information Retrieval, R. Korfhage, E. Rasmussen, and P. Willett, Eds. ACM Press, New York, 88-97.

[24]

LOVINS, J. B. 1968. Development of a stemming algorithm. Mech. Transl. Comput. 11, 1-2, 22-31.

[25]

LUCARELLA, D. 1988. A document retrieval system based upon nearest neighbour searching. J. Inf. Sci. 14, 25-33.

[26]

MCDONELL, K. J. 1977. An inverted index implementation. Comput. J. 20, 1, 116-123.

[27]

MOFFAT,A.AND ZOBEL, J. 1992. Parameterised compression for sparse bitmaps. In Proceed-ings of the ACM-SIGIR International Conference on Research and Development in Informa-tion Retrieval, N. Belkin, P. Ingwersen, and A. M. Pejtersen, Eds. ACM Press, New York, 274-285.

[28]

MOFFAT, A., ZOBEL, J., AND KLEIN, S. T. 1995. Improved inverted file processing for large text databases. In Proceedings of the 6th Australasian Database Conference, R. Sacks-Davis and J. Zobel, Eds. 162-171.

[29]

MOFFAT, A., ZOBEL, J., AND SACKS-DAVIS, R. 1994. Memory efficient ranking. Inf. Process. Manag. 30, 6, 733-744.

[30]

SACKS-DAVIS, R., KENT,A.J.,AND RAMAMOHANARAO, K. 1987. Multi-key access methods based on superimposed coding techniques. ACM Trans. Database Syst. 12, 4 (Dec.), 655- 696.

[31]

SALTON, G. 1989. Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley, Reading, Mass.

[32]

SALTON,G.AND MCGILL, M. J. 1983. Introduction to Modern Information Retrieval. McGraw-Hill, New York.

[33]

WITTEN,I.H.,MOFFAT, A., AND BELL, T. C. 1994. Managing Gigabytes: Compressing and Indexing Documents and Images. Van Nostrand Reinhold, New York.

[34]

ZOBEL, J., MOFFAT, A., AND SACKS-DAVIS, R. 1992. An efficient indexing technique for full-text database systems. In Proceedings of the International Conference on Very Large Databases, L.-Y. Yuan, Ed. VLDB Endowment Press, Saratoga, Calif., 352-362.

[35]

ZOBEL, J., MOFFAT, A., WILKINSON, R., AND SACKS-DAVIS, R. 1995. Efficient retrieval of partial documents. Inf. Process. Manag. 31, 3, 361-377.

Cited By

Panja S(2024)Information Retrieval Systems in HealthcareTransformative Approaches to Patient Literacy and Healthcare Innovation10.4018/979-8-3693-3661-8.ch009(180-200)Online publication date: 9-Feb-2024
https://doi.org/10.4018/979-8-3693-3661-8.ch009
Gan YGe YZhou CSu SXu ZXu XHui QChen XWang YShan YSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Binary Embedding-based Retrieval at TencentProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599782(4056-4067)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599782
Moffat AMackenzie J(2023)Efficient immediate-access dynamic indexingInformation Processing & Management10.1016/j.ipm.2022.10324860:3(103248)Online publication date: May-2023
https://doi.org/10.1016/j.ipm.2022.103248
Show More Cited By

Index Terms

Self-indexing inverted files for fast text retrieval
1. Information systems

Recommendations

Inverted files versus signature files for text indexing

Two well-known indexing methods are inverted files and signature files. We have undertaken a detailed comparison of these two approaches in the context of text indexing, paying particular attention to query evaluation speed and space requirements. We ...
Read More
Incremental cluster-based retrieval using compressed cluster-skipping inverted files

We propose a unique cluster-based retrieval (CBR) strategy using a new cluster-skipping inverted file for improving query processing efficiency. The new inverted file incorporates cluster membership and centroid information along with the usual document ...
Read More
Inverted file-based indexing for efficient multimedia information retrieval in metric spaces
SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

Content-based similarity search is an important task in multimedia information retrieval (IR). Here, metric space access methods (MAMs) can be applied. They are purely based on the use of a metric distance. No assumption is made about the representation ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 14, Issue 4

Oct. 1996

124 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/237496

Issue’s Table of Contents

Copyright © 1996 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 1996

Published in TOIS Volume 14, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

275
Total Citations
View Citations
3,031
Total Downloads

Downloads (Last 12 months)157
Downloads (Last 6 weeks)17

Other Metrics

View Author Metrics

Citations

Cited By

Panja S(2024)Information Retrieval Systems in HealthcareTransformative Approaches to Patient Literacy and Healthcare Innovation10.4018/979-8-3693-3661-8.ch009(180-200)Online publication date: 9-Feb-2024
https://doi.org/10.4018/979-8-3693-3661-8.ch009
Gan YGe YZhou CSu SXu ZXu XHui QChen XWang YShan YSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Binary Embedding-based Retrieval at TencentProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599782(4056-4067)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599782
Moffat AMackenzie J(2023)Efficient immediate-access dynamic indexingInformation Processing & Management10.1016/j.ipm.2022.10324860:3(103248)Online publication date: May-2023
https://doi.org/10.1016/j.ipm.2022.103248
Aggarwal CAggarwal C(2022)Information Retrieval and Search EnginesMachine Learning for Text10.1007/978-3-030-96623-2_9(257-302)Online publication date: 10-Feb-2022
https://doi.org/10.1007/978-3-030-96623-2_9
McClain SMutschler-Aldine MMonaghan CChiu DSawin JJarvis P(2021)Caching Support for Range Query Processing on Bitmap IndicesProceedings of the 33rd International Conference on Scientific and Statistical Database Management10.1145/3468791.3468800(49-60)Online publication date: 6-Jul-2021
https://dl.acm.org/doi/10.1145/3468791.3468800
Mackenzie JPetri MMoffat A(2021)Anytime Ranking on Document-Ordered IndexesACM Transactions on Information Systems10.1145/346789040:1(1-32)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3467890
Wang JLin CPapakonstantinou YSwanson S(2021)Evaluating List Intersection on SSDs for Parallel I/O Skipping2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00161(1823-1828)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00161
Satvika Thada VSingh J(2021)A Primer on Word EmbeddingData Intelligence and Cognitive Informatics10.1007/978-981-15-8530-2_42(525-541)Online publication date: 9-Jan-2021
https://doi.org/10.1007/978-981-15-8530-2_42
Fontoura MJosifovski VLiu JVenkatesan SZhu XZien J(2020)Evaluation strategies for top-k queries over memory-resident inverted indexesProceedings of the VLDB Endowment10.14778/3402755.34027564:12(1213-1224)Online publication date: 3-Jun-2020
https://dl.acm.org/doi/10.14778/3402755.3402756
Wang GJi HKong DZhang N(2020)Modality-Dependent Cross-Modal Retrieval Based on Graph RegularizationMobile Information Systems10.1155/2020/41646922020(1-17)Online publication date: 31-Jan-2020
https://doi.org/10.1155/2020/4164692
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents