research-article

Query Specific Rank Fusion for Image Retrieval

Authors:

Shaoting Zhang,

Dimitris N. MetaxasAuthors Info & Claims

IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 37, Issue 4

Pages 803 - 815

https://doi.org/10.1109/TPAMI.2014.2346201

Published: 01 April 2015 Publication History

Abstract

Recently two lines of image retrieval algorithms demonstrate excellent scalability: 1) local features indexed by a vocabulary tree, and 2) holistic features indexed by compact hashing codes. Although both of them are able to search visually similar images effectively, their retrieval precision may vary dramatically among queries. Therefore, combining these two types of methods is expected to further enhance the retrieval precision. However, the feature characteristics and the algorithmic procedures of these methods are dramatically different, which is very challenging for the feature-level fusion. This motivates us to investigate how to fuse the ordered retrieval sets, i.e., the ranks of images, given by multiple retrieval methods, to boost the retrieval precision without sacrificing their scalability. In this paper, we model retrieval ranks as graphs of candidate images and propose a graph-based query specific fusion approach, where multiple graphs are merged and reranked by conducting a link analysis on a fused graph. The retrieval quality of an individual method is measured on-the-fly by assessing the consistency of the top candidates' nearest neighborhoods. Hence, it is capable of adaptively integrating the strengths of the retrieval methods using local or holistic features for different query images. This proposed method does not need any supervision, has few parameters, and is easy to implement. Extensive and thorough experiments have been conducted on four public datasets, i.e., the UKbench, Corel-5K, Holidays and the large-scale San Francisco Landmarks datasets. Our proposed method has achieved very competitive performance, including state-of-the-art results on several data sets, e.g., the N-S score 3.83 for UKbench.

References

[1]

A. Andoni and P. Indyk, “Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions, ” in Proc. IEEE 47th Annu. Symp. Found. Comput. Sci., Berkeley, CA, USA, Oct. 21-24, 2006, pp. 459–468.

[2]

D. Cai, X. He, and J. Han, “Spectral regression: A unified subspace learning framework for content-based image retrieval,” in Proc. ACM 15th Int. Conf. Multimedia, Augsburg, Germany, Sep. 24-29, 2007, pp. 403–412.

[3]

D. M. Chen, G. Baatz, K. Köser, S. S. Tsai, R. Vedantham, T. Pylvänäinen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, B. Girod, and R. Grzeszczuk, “City-scale landmark indentification on mobile devices,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Colorado Springs, CO, USA, Jun. 20-26, 2011, pp. 737– 744.

[4]

O. Chum, J. Philbin, J. Sivic, M. Isard, and A. Zisserman, “Total recall: Automatic query expansion with a generative feature model for object retrieval,” in Proc. Int. Conf. Comput. Vis., Rio de Janeiro, Brazil, Oct. 14-17, 2007, pp. 1–8.

[5]

M. Datar, N. Immorlica, P. Indyk, and V. S. Mirrokni, “Locality-sensitive hashing scheme based on p-stable distributions,” in Proc. 20th Annu. Symp. Comput. Geometry, 2004, pp. 253–262.

[6]

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Miami, FL, USA, Jun. 20-26, 2009, pp. 248–255.

[7]

P. Duygulu., K. Barnard, J. de Freitas, and D. Forsyth, “ Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary, ” in Proc. 7th Eur. Conf. Comput. Vis., May 27-Jun. 2, 2002, vol. 4, pp. 97–112.

[8]

R. Fagin, R. Kumar, and D. Sivakumar, “ Efficient similarity search and classification via rank aggregation,” in Proc. ACM SIGMOD Int. Conf. Manage. Data, San Diego, CA, USA, Jun. 9-12, 2003, pp. 301–312.

[9]

T. Ge, K. He, Q. Ke, and J. Sun, “Optimized product quantization for approximate nearest neighbor search,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 2946–2953.

[10]

P. Gehler, Sebastian, and Nowozin, “On feature combination for multiclass object classification, ” in Proc. IEEE 12th Int. Conf. Comput. Vis., Kyoto, Japan, Oct. 14-21, 2009, pp. 221–228.

[11]

Y. Gong and S. Lazebnik, “Iterative quantization: A procrustean approach to learning binary codes,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Colorado Springs, CO, USA, Jun. 20-26, 2011, pp. 817–824.

[12]

P. Jaccard, “The distribution of the flora in the alpine zone, ” New Phytologist, vol. 11, no. 2, pp. 37– 50, 1912.

[13]

H. Jégou, M. Douze, and C. Schmid, “Hamming embedding and weak geometric consistency for large scale image search,” in Proc. Eur. Conf. Comput. Vis., Marseille, France, Oct. 12-18, 2008, vol. 1, pp. 304–317.

[14]

H. Jégou, M. Douze, and C. Schmid, “On the burstiness of visual elements,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. , Miami, FL, USA, Jun. 20-25, 2009, pp. 1169–1176.

[15]

H. Jégou, M. Douze, and C. Schmid, “Product quantization for nearest neighbor search,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 33, no. 1, pp. 117–128, Jan. 2011.

[16]

H. Jégou, C. Schmid, H. Harzallah, and J. Verbeek, “Accurate image search using the contextual dissimilarity measure,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 2–11, Jan. 2010.

[17]

Y. Jing and Balujia, “VisualRank: Applying PageRank to large-scale image search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 11, pp. 1877 –1890, Nov. 2008.

[18]

A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” in Adv. Neural Inf. Process. Syst. 26, Lake Tahoe, CA, USA, Dec. 3-6, 2012, pp. 1106– 1114.

[19]

B. Kulis, P. Jain, and K. Grauman, “Fast similarity search for learned metrics,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 12, pp. 2143–2157, Dec. 2009.

[20]

W. Liu, J. Wang, and S.-F. Chang, “Robust and scalable graph-based semisupervised learning,” Proc. IEEE, vol. 100, no. 9, pp. 2624–2638, Sep. 2012.

[21]

W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang, “Supervised hashing with kernels,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2074–2081.

[22]

W. Liu, J. Wang, S. Kumar, and S.-F. Chang, “Hashing with graphs,” in Proc. Int. Conf. Mach. Learn., 2011, pp. 1–8.

[23]

X. Liu, J. He, and B. Lang, “Multiple feature kernel hashing for large-scale visual search,” Pattern Recognit., vol. 47, no. 2, pp. 748–757, 2014.

Digital Library

[24]

D. G. Lowe, “Distinctive image features from scale invariant keypoints, ” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, 2004.

Digital Library

[25]

D. Nistér and H. Stewénius, “Scalable recognition with a vocabulary tree,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., New York City, NY, USA, Jun. 17-22, 2006, pp. 2161–2168.

[26]

M. Norouzi and D. J. Fleet, “Cartesian k-means,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 3017–3024.

[27]

A. Oliva and A. Torralba, “Modeling the shape of the scene: A holistic representation of the spatial envelope, ” Int. J. Comput. Vis., vol. 42, no. 3, pp. 145–175, 2001.

Digital Library

[28]

L. Page, S. Brin, R. Motwani, and T. Winograd, “The PageRank citation ranking: Bringing order to the web,” in Proc. 7th Int. World Wide Web Conf., 1999, pp. 161–172.

[29]

J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman, “Object retrieval with large vocabularies and fast spatial matching,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Minneapolis, MN, USA, Jun. 17-22, 2007, pp. 1–8.

[30]

D. Qin, S. Gammeter, L. Bossard, T. Quack, and L. van Cool, “Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors, ” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Colorado Springs, CO, USA, Jun. 20-26, 2011, pp. 777–784.

[31]

M. Richardson and P. Domingos, “The intelligent surfer: Probabilistic combination of link and content information in PageRank, ” in Proc. Adv. Neural Inf. Process. Syst., 2002, vol. 14, pp. 1441–1448.

[32]

R. Salakhutdinov and G. E. Hinton, “Learning a nonlinear embedding by preserving class neighbourhood structure, ” in Proc. Int. Conf. Artif. Intell. Statist., 2007, pp. 412–419.

[33]

J. Sivic and A. Zisserman, “Video google: A text retrieval approach to object matching in videos,” in Proc. 9th Int. Conf. Comput. Vis., Nice, France, Oct. 13-16, 2003, pp. 1470–1477.

[34]

J. Song, Y. Yang, Z. Huang, H. T. Shen, and R. Hong, “Multiple feature hashing for real-time large scale near-duplicate video retrieval,” in Proc. ACM Int. Conf. Multimedia, 2011, pp. 423 –432.

[35]

A. Torralba, R. Fergus, and Y. Weiss, “Small codes and large image databases for recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Anchorage, AK, USA, Jun. 23-28, 2008, pp. 1 –8.

[36]

A. Vedaldi and B. Fulkerson, “VLFeat: An open and portable library of computer vision algorithms,” in IEEE Conf. Comput. Vis. Pattern Recognit., Singapore, Jul. 19-23, 2010, pp. 1469–1472.

[37]

J. Wang, S. Kumar, and S.-F. Chang, “ Semi-supervised hashing for large-scale search,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 34, no. 12, pp. 2393–2406, Dec. 2012.

[38]

X. Wang, M. Yang, T. Cour, S. Zhu, K. Yu, and T. X. Han, “ Contextual weighting for vocabulary tree based image retrieval,” in Proc. IEEE Int. Conf. Comput. Vis., Barcelona, Spain, Nov. 6-13, 2011, pp. 209–216.

[39]

Y. Weiss, A. Torralba, and R. Fergus, “Spectral hashing,” in Proc. Adv. Neural Inf. Process. Syste. 21, Vancouver, Canada, Dec. 8-13, 2008, pp. 1753–1760.

[40]

Z. Wu, Q. Ke, M. Isard, and J. Sun, “Bundling feature for large scale partial-duplicated web image search,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Miami, FL, USA, Jun. 20-26, 2009, pp. 25–32.

[41]

B. Xu, J. Bu, C. Chen, D. Cai, X. He, W. Liu, and J. Luo, “Efficient manifold ranking for image retrieval,” in Proc. Int. ACM SIGIR conf. Res. dev. Inf. Retrieval, 2011, pp. 525–534.

[42]

X.-T. Yuan and S. Yan, “Visual classification with multi-task joint sparse representation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 3493–3500.

[43]

D. Zhang, F. Wang, and L. Si, “Composite hashing with multiple information sources,” in Proc. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2011, pp. 225–234.

[44]

S. Zhang, J. Huang, H. Li, and D. N. Metaxas, “Automatic image annotation and retrieval using group sparsity,” IEEE Trans. Syst., Man, Cybern., Part B: Cybern., vol. 42, no. 3, pp. 838–849, Jun. 2012.

[45]

S. Zhang, M. Yang, T. Cour, K. Yu, and D. N. Metaxas, “Query specific fusion for image retrieval,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 660–673.

[46]

Y. Zhang, Z. Jia, and T. Chen, “Image retrieval with geometry-preserving visual phrases,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Colorado Springs, CO, USA, Jun. 20-26, 2011, pp. 809–816.

[47]

W. Zhou, Y. Lu, H. Li, Y. Song, and Q. Tian, “Spatial coding for large scale partial-duplicate web image search, ” in Proc. ACM Int. Conf. Multimedia, Florence, Italy, Oct. 25-29, 2010, pp. 511–520.

Cited By

Han DLiu BShao SLiu WZhou Y(2025)Feature aggregation and connectivity for object re-identificationPattern Recognition10.1016/j.patcog.2024.110869157:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.patcog.2024.110869
Luo JYao HXu CSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Cluster-aware similarity diffusion for instance retrievalProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693431(33511-33532)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693431
Salemi AKallumadi SZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Optimization Methods for Personalizing Large Language Models through Retrieval AugmentationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657783(752-762)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657783
Show More Cited By

Index Terms

Query Specific Rank Fusion for Image Retrieval
1. Computing methodologies
2. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
    2. Search engine architectures and scalability

Index terms have been assigned to the content through auto-classification.

Recommendations

Semantic-Aware Co-Indexing for Image Retrieval
In content-based image retrieval, inverted indexes allow fast access to database images and summarize all knowledge about the database. Indexing multiple clues of image contents allows retrieval algorithms search for relevant images from different ...
Image retrieval on large-scale image databases
CIVR '07: Proceedings of the 6th ACM international conference on Image and video retrieval

Online image repositories such as Flickr contain hundreds of millions of images and are growing quickly. Along with that the needs for supporting indexing, searching and browsing is becoming more and more pressing. In this work we will employ the image ...
Query Specific Fusion for Image Retrieval
Proceedings, Part II, of the 12th European Conference on Computer Vision --- ECCV 2012 - Volume 7573

Recent image retrieval algorithms based on local features indexed by a vocabulary tree and holistic features indexed by compact hashing codes both demonstrate excellent scalability. However, their retrieval precision may vary dramatically among queries. ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE Transactions on Pattern Analysis and Machine Intelligence Volume 37, Issue 4

April 2015

208 pages

ISSN:0162-8828

Issue’s Table of Contents

Copyright © 2014.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 April 2015

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Han DLiu BShao SLiu WZhou Y(2025)Feature aggregation and connectivity for object re-identificationPattern Recognition10.1016/j.patcog.2024.110869157:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.patcog.2024.110869
Luo JYao HXu CSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Cluster-aware similarity diffusion for instance retrievalProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693431(33511-33532)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693431
Salemi AKallumadi SZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Optimization Methods for Personalizing Large Language Models through Retrieval AugmentationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657783(752-762)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657783
Kato TKomamizu TIde I(2024)R-DiP: Re-ranking Based Diffusion Pre-computation for Image RetrievalDatabase and Expert Systems Applications10.1007/978-3-031-68312-1_18(233-247)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1007/978-3-031-68312-1_18
Pascotti Valem LPedronette DLatecki L(2023)Rank Flow Embedding for Unsupervised and Semi-Supervised Manifold LearningIEEE Transactions on Image Processing10.1109/TIP.2023.326886832(2811-2826)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TIP.2023.3268868
Gao XMu TGoulermas JSong JWang M(2022)Improving Image Similarity Learning by Adding External MemoryIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2020.304710434:10(4874-4887)Online publication date: 1-Oct-2022
https://dl.acm.org/doi/10.1109/TKDE.2020.3047104
Wang JYi XGuo RJin HXu PLi SWang XGuo XLi CXu XYu KYuan YZou YLong JCai YLi ZZhang ZMo YGu JJiang RWei YXie CLi GLi ZIdreos SSrivastava D(2021)MilvusProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457550(2614-2627)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457550
Valem LPedronette DEl Saddik ADel Bimbo AZhang ZHauptmann ACandan KBertini MXie LWei X(2019)An Unsupervised Genetic Algorithm Framework for Rank Selection and Fusion on Image RetrievalProceedings of the 2019 on International Conference on Multimedia Retrieval10.1145/3323873.3325022(58-62)Online publication date: 5-Jun-2019
https://dl.acm.org/doi/10.1145/3323873.3325022
Zemene ETesfaye YIdrees HPrati APelillo MShah M(2019)Large-Scale Image Geo-Localization Using Dominant SetsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2017.278713241:1(148-161)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.1109/TPAMI.2017.2787132
Dourado IPedronette DTorres R(2019)Unsupervised graph-based rank aggregation for improved retrievalInformation Processing and Management: an International Journal10.1016/j.ipm.2019.03.00856:4(1260-1279)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1016/j.ipm.2019.03.008
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents