Exploit Every Bit: Effective Caching for High-Dimensional Nearest Neighbor Search

High-dimensional \(k\)Image (tang-ieq1-2515603.gif) is missing or otherwise invalid. nearest neighbor (kNN) search has a wide range of applications in multimedia information retrieval. Existing disk-based \(k\)Image (tang-ieq2-2515603.gif) is missing or otherwise invalid. NN search methods incur significant I/O costs in the candidate refinement phase. In this paper, we propose to cache compact approximate representations of data points in main memory in order to reduce the candidate refinement time during \(k\)Image (tang-ieq3-2515603.gif) is missing or otherwise invalid.NN search. This problem raises two challenging issues: (i) which is the most effective encoding scheme for data points to support \(k\)Image (tang-ieq4-2515603.gif) is missing or otherwise invalid. NN search? and (ii) what is the optimal number of bits for encoding a data point? For (i), we formulate and solve a novel histogram optimization problem that decides the most effective encoding scheme. For (ii), we develop a cost model for automatically tuning the optimal number of bits for encoding points. In addition, our approach is generic and applicable to exact / approximate \(k\) Image (tang-ieq5-2515603.gif) is missing or otherwise invalid.NN search methods. Extensive experimental results on real datasets demonstrate that our proposal can accelerate the candidate refinement time of \(k\)Image (tang-ieq6-2515603.gif) is missing or otherwise invalid.NN search by at least an order of magnitude.