Filtered document retrieval with frequency-sorted indexes

Reviewer: William T. O'Connell

The authors describe a filtering technique used in conjunction with frequency-sorted inverted lists in document databases. By filtering out irrelevant documents quickly, they reduce disk, CPU, and memory costs on similarity queries without degrading retrieval effectiveness. Since each inverted list is sorted on term frequency within a database, their technique limits the search to the first part of the highly relevant lists. A list is considered relevant only if the combination of the list's term frequency and its term importance for a target document is large enough to be likely to affect the final ordering of documents. During simulations, the technique used only 2 percent of the memory required by other standard implementations, without degradation of retrieval effectiveness. However, it is not clear what implementations are being referred to. While the authors give an overview of inversion techniques, they do not relate this work to signature files or to vector space and clustering models [1]. Recent work in weighted-partitioned signature files has also shown CPU, memory, and disk reduction without sacrificing recall and precision effectiveness [2]. Recent developments in vector models, such as latent semantic i ndexing, effectively apply singular value decomposition on the document term-matrices with good results [3]. Additionally, this approach allows general-purpose multidimensional indexes to be used. It would also have been appropriate to have additional discussion of the method's disadvantages, such as additional storage overhead [4] due to the difficulties of compression on frequency-sorted lists and the query cost on dynamic transaction workloads. The authors' idea about frequency-sorted indexes is a good extension, but I would have liked to see more analysis of the implications for storage overhead and dynamic transaction workloads. Overall, the ideas are well organized and presented. I recommend the paper to anyone interested in document retrieval techniques.