research-article

Compression, SIMD, and Postings Lists

Author:

Andrew TrotmanAuthors Info & Claims

ADCS '14: Proceedings of the 19th Australasian Document Computing Symposium

Pages 50 - 57

https://doi.org/10.1145/2682862.2682870

Published: 26 November 2014 Publication History

Abstract

The three generations of postings list compression strategies (Variable Byte Encoding, Word Aligned Codes, and SIMD Codecs) are examined in order to test whether or not each truly represented a generational change -- they do. Some weaknesses of the current SIMD-based schemes are identified and a new scheme, QMX, is introduced to address both space and decoding inefficiencies. Improvements are examined on multiple architectures and it is shown that different SSE implementations (Intel and AMD) perform differently.

References

[1]

Anh, V.N., A. Moffat, Inverted Index Compression using Word-Aligned Binary Codes. Inf. Ret., 2005. 8(1):151--166.

Digital Library

[2]

Anh, V.N., A. Moffat, Index compression using 64-bit words. Softw. Pract. Exper., 2010. 40(2):131--147.

Digital Library

[3]

Catena, M., C. Macdonald, I. Ounis, On Inverted Index Compression for Search Engine Efficiency, in ECIR 2014, pp. 359--371.

[4]

Dean, J., Challenges in Building Large-scale Information Retrieval Systems: Invited Talk, in WSDM 2009.

Digital Library

[5]

Elias, P., Universal Codeword Sets and the Representation of the Integers. IEEE Trans. Inf. Theory, 1975. 21(2):194--203.

Digital Library

[6]

Golomb, S.W., Run-length Encodings. IEEE Trans. Inf. Theory, 1966. 12(3):399--401.

Digital Library

[7]

Lemire, D., L. Boytsov, Decoding Billions of Integers per Second through Vectorization. Software: Prac. Exper.

[8]

Moffat, A., L. Stuiver, Binary Interpolative Coding for Effective Index Compression. Inf. Ret., 2000. 3(1):25--47.

Digital Library

[9]

Scholer, F., H.E. Williams, J. Yiannis, J. Zobel. Compression of Inverted Indexes for Fast Query Evaluation. in SIGIR 2002, pp. 222--229

Digital Library

[10]

Silvestri, F., R. Venturini, VSEncoding: Efficient Coding and Fast Decoding of Integer Lists via Dynamic Programming, in CIKM 2010, pp. 1219--1228.

Digital Library

[11]

Stepanov, A.A., A.R. Gangolli, D.E. Rose, R.J. Ernst, P.S. Oberoi, SIMD-based Decoding of Posting Lists, in CIKM 2011, pp. 317--326.

Digital Library

[12]

Trotman, A., Compressing Inverted Files. Inf Ret., 2003. 6(1):5--19.

Digital Library

[13]

Trotman, A., X.-F. Jia, M. Crane, Towards an Efficient and Effective Search Engine, in SIGIR 2012 Workshop on Open Source Information Retrieval. 2012. pp. 40--47.

[14]

Williams, H.E., J. Zobel, Compressing Integers for Fast File Access. Computer Journal, 1999. 42(3):193--201.

[15]

Zhang, J., X. Long, T. Suel, Performance of Compressed Inverted List Caching in Search Engines, in WWW 2008, pp. 387--396.

Digital Library

[16]

Zukowski, M., S. Heman, N. Nes, P. Boncz, Super-Scalar RAM-CPU Cache Compression, in ICDE 2006.

Digital Library

Cited By

Moffat AMackenzie J(2023)Efficient immediate-access dynamic indexingInformation Processing & Management10.1016/j.ipm.2022.10324860:3(103248)Online publication date: May-2023
https://doi.org/10.1016/j.ipm.2022.103248
Dong BPopescu ATribaldos VByna SAjo-Franklin JWu K(2022)Real-time and post-hoc compression for data from Distributed Acoustic SensingComputers & Geosciences10.1016/j.cageo.2022.105181166:COnline publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.cageo.2022.105181
Pibiri GVenturini R(2020)Techniques for Inverted Index CompressionACM Computing Surveys10.1145/341514853:6(1-36)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.1145/3415148
Show More Cited By

Index Terms

Compression, SIMD, and Postings Lists
1. Information systems
  1. Information retrieval
    1. Document representation
    2. Search engine architectures and scalability
      1. Search engine indexing

Recommendations

In Vacuo and In Situ Evaluation of SIMD Codecs
ADCS '16: Proceedings of the 21st Australasian Document Computing Symposium

The size of a search engine index and the time to search are inextricably related through the compression codec. This investigation examines this tradeoff using several relatively unexplored SIMD-based codecs including QMX, TurboPackV, and TurboPFor. It ...
SIMD-based decoding of posting lists
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge management

Powerful SIMD instructions in modern processors offer an opportunity for greater search performance. In this paper, we apply these instructions to decoding search engine posting lists. We start by exploring variable-length integer encoding formats used ...
Building a large instruction window through ROB compression
MEDEA '07: Proceedings of the 2007 workshop on MEmory performance: DEaling with Applications, systems and architecture

Current processors require a large number of in-flight instructions in order to look for further parallelism and hide the increasing gap between memory latency and processor cycle time. These in-flight instructions are typically stored in centralized ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ADCS '14: Proceedings of the 19th Australasian Document Computing Symposium

November 2014

132 pages

ISBN:9781450330008

DOI:10.1145/2682862

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

RMIT University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 November 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ADCS '14

ADCS '14: Australasian Document Computing Symposium

November 27 - 28, 2014

VIC, Melbourne, Australia

Acceptance Rates

Overall Acceptance Rate 30 of 57 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
234
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Moffat AMackenzie J(2023)Efficient immediate-access dynamic indexingInformation Processing & Management10.1016/j.ipm.2022.10324860:3(103248)Online publication date: May-2023
https://doi.org/10.1016/j.ipm.2022.103248
Dong BPopescu ATribaldos VByna SAjo-Franklin JWu K(2022)Real-time and post-hoc compression for data from Distributed Acoustic SensingComputers & Geosciences10.1016/j.cageo.2022.105181166:COnline publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.cageo.2022.105181
Pibiri GVenturini R(2020)Techniques for Inverted Index CompressionACM Computing Surveys10.1145/341514853:6(1-36)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.1145/3415148
Trotman ALilly KHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)JASSjr: The Minimalistic BM25 Search Engine for Teaching and Learning Information RetrievalProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401413(2185-2188)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401413
Pibiri GVenturini R(2020)On Optimally Partitioning Variable-Byte CodesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2019.291128832:9(1812-1823)Online publication date: 5-Aug-2020
https://dl.acm.org/doi/10.1109/TKDE.2019.2911288
Quijada Fuentes CPenabad MLadra SGutierrez Retamal G(2020)Compressed Data Structures for Binary Relations in PracticeIEEE Access10.1109/ACCESS.2020.29709838(25949-25963)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2970983
Pibiri GPetri MMoffat ACulpepper JMoffat ABennett PLerman K(2019)Fast Dictionary-Based Compression for Inverted IndexesProceedings of the Twelfth ACM International Conference on Web Search and Data Mining10.1145/3289600.3290962(6-14)Online publication date: 30-Jan-2019
https://dl.acm.org/doi/10.1145/3289600.3290962
Song XYang YJiang YJiang K(2019)Optimizing partitioning strategies for faster inverted index compressionFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-016-6252-513:2(343-356)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1007/s11704-016-6252-5
Mallia ASiedlaczek MSuel T(2019)An Experimental Study of Index Compression and DAAT Query Processing MethodsAdvances in Information Retrieval10.1007/978-3-030-15712-8_23(353-368)Online publication date: 7-Apr-2019
https://doi.org/10.1007/978-3-030-15712-8_23
Trotman ACrane M(2019) Micro‐ and macro‐optimizations of S aa T search Software: Practice and Experience10.1002/spe.268349:5(942-950)Online publication date: 20-Feb-2019
https://doi.org/10.1002/spe.2683
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents