research-article

Joint Hypergraph Learning for Tag-Based Image Retrieval

Authors:

Junwei HanAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 27, Issue 9

Pages 4437 - 4451

https://doi.org/10.1109/TIP.2018.2837219

Published: 01 September 2018 Publication History

Abstract

As the image sharing websites like Flickr become more and more popular, extensive scholars concentrate on tag-based image retrieval. It is one of the important ways to find images contributed by social users. In this research field, tag information and diverse visual features have been investigated. However, most existing methods use these visual features separately or sequentially. In this paper, we propose a global and local visual features fusion approach to learn the relevance of images by hypergraph approach. A hypergraph is constructed first by utilizing global, local visual features, and tag information. Then, we propose a pseudo-relevance feedback mechanism to obtain the pseudo-positive images. Finally, with the hypergraph and pseudo relevance feedback, we adopt the hypergraph learning algorithm to calculate the relevance score of each image to the query. Experimental results demonstrate the effectiveness of the proposed approach.

References

[1]

C. Xi, H. Guang, and X. Shunli, “An image registration method based on similarity of edge information,” in Proc. IEEE Int. Symp. Ind. Electron. Piscataway, NJ, USA: IEEE Press, May 2012, pp. 217–224.

[2]

X. Yang, Y. Zhang, T. Yao, C.-W. Ngo, and T. Mei, “Click-boosting multi-modality graph-based reranking for image search,” Multimedia Syst., vol. 21, no. 2, pp. 217–227, Mar. 2015.

Digital Library

[3]

D. Zhou, J. Huang, and B. Schölkopf, “Learning with hypergraphs: Clustering, classification, and embedding,” in Proc. NIPS, vol. 19, 2006, pp. 1–8.

[4]

Y. Zhang, X. Yang, and T. Mei, “Image search reranking with query-dependent click-based relevance feedback,” IEEE Trans. Image Process., vol. 23, no. 10, pp. 4448–4459, Oct. 2014.

[5]

X. Yang, T. Mei, Y. Zhang, J. Liu, and S. Satoh, “Web image search re-ranking with click-based similarity and typicality,” IEEE Trans. Image Process., vol. 25, no. 10, pp. 4617–4630, Oct. 2016.

Digital Library

[6]

Y. Huang, Q. Liu, S. Zhang, and D. N. Metaxas, “Image retrieval via probabilistic hypergraph ranking,” in Proc. IEEE CVPR, Jun. 2010, pp. 3376–3383.

[7]

Q. Liu, Y. Huang, and D. N. Metaxas, “Hypergraph with sampling for image retrieval,” Pattern Recognit., vol. 44, nos. 10–11, pp. 2255–2262, 2011.

Digital Library

[8]

L. Wang, Z. Zhao, and F. Su, “Tag-based social image search with hyperedges correlation,” in Proc. IEEE Vis. Commun. Image Process. Conf., Dec. 2014, pp. 330–333.

[9]

J. Cai, Z.-J. Zha, M. Wang, S. Zhang, and Q. Tian, “An attribute-assisted reranking model for Web image search,” IEEE Trans. Image Process., vol. 24, no. 1, pp. 261–272, Jan. 2015.

[10]

P. Jing, Y. Su, C. Xu, and L. Zhang, “HyperSSR: A hypergraph based semi-supervised ranking method for visual search reranking,” Neurocomputing, vol. 274, pp. 50–57, Jan. 2016.

[11]

Y. Xiang, X. Zhou, T. Chua, and C.-W. Ngo, “A revisit of generative model for automatic image annotation using Markov random fields,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1153–1160.

[12]

S. Agarwal, J. Lim, L. Zelnik-Manor, P. Perona, D. Kriegman, and S. Belongie, “Beyond pairwise clustering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2005, pp. 838–845.

[13]

Y. Huang, Q. Liu, and D. Metaxas, “Video object segmentation by hypergraph cut,” in Proc. IEEE Comput. Vis. Pattern Recognit., Jun. 2009, pp. 1738–1745.

[14]

L. Sun, S. Ji, and J. Ye, “Hypergraph spectral learning for multi-label classification,” in Proc. SIG KDD, 2008, pp. 668–676.

[15]

Z. Tian, T. Hwang, and R. Kuang, “A hypergraph-based learning algorithm for classifying gene expression and arrayCGH data with prior knowledge,” Bioinformatics, vol. 25, no. 21, pp. 2831–2838, Nov. 2009.

Digital Library

[16]

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.

Digital Library

[17]

R. Ji, H. Yao, X. Sun, B. Zhong, and W. Gao, “Towards semantic embedding in visual vocabulary,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 918–925.

[18]

J. Yang, Y.-G. Jiang, A. G. Hauptmann, and C.-W. Ngo, “Evaluating bag-of-visual-words representations in scene classification,” in Proc. ACM SIGMM Workshop Multimedia Inf. Retrieval, Sep. 2007, pp. 197–206.

[19]

R. Jiet al., “Location discriminative vocabulary coding for mobile landmark search,” Int. J. Comput. Vis., vol. 96, no. 3, pp. 290–314, Feb. 2012.

Digital Library

[20]

Y. Jiang, C.-W. Ngo, and J. Yang, “Towards optimal bag-of-features for object categorization and semantic video retrieval,” in Proc. ACM Int. Conf. Image Video Retrieval, 2007, pp. 494–501.

[21]

D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, “Learning with local and global consistency,” in Proc. Adv. Neural Inf. Process. Syst., vol. 16. 2004, pp. 321–328.

[22]

D. Liu, X.-S. Hua, M. Wang, and H. Zhang, “Boost search relevance for tag-based social image retrieval,” in Proc. IEEE Int. Conf. Multimedia Expo, Jun./Jul. 2009, pp. 1636–1639.

[23]

K. Song, Y. Tian, W. Gao, and T. Huang, “Diversifying the image retrieval results,” in Proc. 14th Annu. ACM Int. Conf. Multimedia, 2006, pp. 707–710.

[24]

M. Wang, X.-S. Hua, R. Hong, J. Tang, G.-J. Qi, and Y. Song, “Unified video annotation via multigraph learning,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 5, pp. 733–746, May 2009.

[25]

Y. Yang, F. Nie, D. Xu, J. Luo, Y. Zhuang, and Y. Pan, “A multimedia retrieval framework based on semi-supervised ranking and relevance feedback,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 4, pp. 723–742, Apr. 2012.

Digital Library

[26]

Y. Gao, M. Wang, D. Tao, R. Ji, and Q. Dai, “3-D object retrieval and recognition with hypergraph analysis,” IEEE Trans. Image Process., vol. 21, no. 9, pp. 4290–4303, Sep. 2012.

Digital Library

[27]

B. J. Frey and D. Dueck, “Clustering by passing messages between data points,” Science, vol. 315, no. 5814, pp. 972–976, Feb. 2007.

[28]

English Wikipedia Dataset. Accessed: May 2015. [Online]. Available: https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2

[29]

H. Yu, M. Li, H.-J. Zhang, and J. Feng, “Color texture moments for content-based image retrieval,” in Proc. IEEE Int. Conf. Image Process., Sep.2002, pp. 929–932.

[30]

T. Ojala, M. Pietikäinen, and D. Harwood, “Performance evaluation of texture measures with classification based on Kullback discrimination of distributions,” in Proc. IAPR Int. Conf. Pattern Recognit. (ICPR), vol. 1. 1994, pp. 582–585.

[31]

Y. Cheng, “Mean shift, mode seeking, and clustering,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 17, no. 8, pp. 779–790, Aug. 1995.

[32]

X. Qian, D. Lu, Y. Wang, L. Zhu, Y. Y. Tang, and M. Wang, “Image re-ranking based on topic diversity,” IEEE Trans. Image Process., vol. 26, no. 8, pp. 3734–3747, Aug. 2017.

Digital Library

[33]

J. Yu, D. Tao, M. Wang, and Y. Rui, “Learning to rank using user clicks and visual features for image retrieval,” IEEE Trans. Cybern., vol. 45, no. 4, pp. 767–779, Apr. 2015.

[34]

Y. Gao, M. Wang, Z. Zha, J. Shen, X. Li, and X. Wu, “Visual-textual joint relevance learning for tag-based social image search,” IEEE Trans. Image Process., vol. 22, no. 1, pp. 363–376, Jan. 2013.

Digital Library

[35]

J. Yu, D. Tao, and M. Wang, “Adaptive hypergraph learning and its application in image classification,” IEEE Trans. Image Process., vol. 21, no. 7, pp. 3262–3272, Jul. 2012.

Digital Library

[36]

S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object recognition using shape contexts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 4, pp. 509–522, Apr. 2002.

Digital Library

[37]

N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. Piscataway, NJ, USA: IEEE Press, Jun. 2005, pp. 886–893.

[38]

L. Duan, W. Li, I. W.-H. Tsang, and D. Xu, “Improving Web image search by bag-based reranking,” IEEE Trans. Image Process., vol. 20, no. 11, pp. 3280–3290, Nov. 2011.

Digital Library

[39]

X. Tang, K. Liu, J. Cui, F. Wen, and X. Wang, “IntentSearch: Capturing user intention for one-click Internet image search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1342–1353, Jul. 2012.

Digital Library

[40]

X. Tian, D. Tao, X.-S. Hua, and X. Wu, “Active reranking for Web image search,” IEEE Trans. Image Process., vol. 19, no. 3, pp. 805–820, Mar. 2010.

Digital Library

[41]

J. Yu, X. Yang, F. Gao, and D. Tao, “Deep multimodal distance metric learning using click constraints for image ranking,” IEEE Trans. Cybern., vol. 47, no. 12, pp. 4014–4024, Dec. 2017.

[42]

J. Yu, Y. Rui, and D. Tao, “Click prediction for Web image reranking using multimodal sparse coding,” IEEE Trans. Image Process., vol. 23, no. 5, pp. 2019–2032, May 2014.

[43]

E. Spyromitros-Xioufis, S. Papadopoulos, A. Ginsca, A. Popescu, Y. Kompatsiaris, and I. Vlahavas, “Improving diversity in image search via supervised relevance scoring,” in Proc. ICMR, 2015, pp. 323–330.

[44]

R. Yan and A. G. Hauptmann, “Query expansion using probabilistic local feedback with application to multimedia retrieval,” in Proc. ACM CIKM, 2007, pp. 361–370.

[45]

F. Jing and S. Baluja, “VisualRank: Applying pagerank to large-scale image search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 11, pp. 1877–1890, Nov. 2008.

Digital Library

[46]

M. Wang, H. Li, D. Tao, K. Lu, and X. Wu, “Multimodal graph-based reranking for Web image search,” IEEE Trans. Image Process., vol. 21, no. 11, pp. 4649–4661, Nov. 2012.

Digital Library

[47]

H.-M. Hou, X.-S. Xu, G. Wang, and X.-L. Wang, “Joint-rerank: A novel method for image search reranking,” Multimedia Tools Appl., vol. 74, no. 4, pp. 1423–1442, Feb. 2015.

Digital Library

[48]

S. Liu, P. Cui, H. Luan, W. Zhu, S. Yang, and Q. Tian, “Social visual image ranking for Web image search,” in Proc. MMM, 2013, pp. 239–249.

[49]

K. Arai and R. Ali, “Hierarchical K-means: An algorithm for centroids initialization for K-means,” Rep. Faculty Sci. Eng. Saga Univ., vol. 36, no. 1, pp. 25–31, 2007.

[50]

D. Lu, X. Liu, and X. Qian, “Tag-based image search by social re-ranking,” IEEE Trans. Multimedia, vol. 18, no. 8, pp. 1628–1639, Aug. 2016.

Digital Library

[51]

S. Jouili and S. Tabbone, “Hypergraph-based image retrieval for graph-based representation,” Pattern Recognit., vol. 45, no. 11, pp. 4054–4068, Nov. 2012.

Digital Library

[52]

A. K. C. Wong, S. W. Lu, and M. Rioux, “Recognition and shape synthesis of 3-D objects based on attributed hypergraphs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 11, no. 3, pp. 279–290, Mar. 1989.

Digital Library

[53]

J. Buet al., “Music recommendation by unified hypergraph: Combining social media information and music content,” in Proc. ACM Int. Conf. Multimedia, 2010, pp. 391–400.

[54]

B. Lin, A. Wei, and X. Tian, “Visual re-ranking through greedy selection and rank fusion,” in Proc. Int. Conf. Multimedia Modeling, 2016, pp. 289–300.

[55]

V. L. Lekshmi and A. John, “Bridging the semantic gap in image search via visual semantic descriptors by integrating text and visual features,” in Advances in Intelligent Systems and Computing, vol. 42. Singapore: Springer, Dec. 2015.

[56]

A. Gordo, J. Almazan, J. Revaud, and D. Larlus, “Deep image retrieval: Learning global representations for image search,” in Proc. ECCV, 2016, pp. 241–257.

[57]

T. Mikolov, L. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proc. NIPS, 2013, pp. 3111–3119.

[58]

Y. Liu, J. Shao, J. Xiao, F. Wu, and Y. Zhuang, “Hypergraph spectral hashing for image retrieval with heterogeneous social contexts,” Neurocomputing, vol. 119, pp. 49–58, Nov. 2013.

Digital Library

[59]

L. Wang, Z. Zhao, and F. Su, “Efficient multi-modal hypergraph learning for social image classification with complex label correlations,” Neurocomputing, vol. 171, pp. 242–251, Jan. 2016.

Digital Library

[60]

Q. Fang, J. Sang, C. Xu, and Y. Rui, “Topic-sensitive influencer mining in interest-based social media networks via hypergraph learning,” IEEE Trans. Multimedia, vol. 16, no. 3, pp. 796–812, Apr. 2014.

Digital Library

[61]

J. Zhong, Y. Pang, and X. Li, “Relevance preserving projection and ranking for Web image search reranking,” IEEE Trans. Image Process., vol. 24, no. 11, pp. 4137–4147, Nov. 2015.

[62]

C. Hong, J. Yu, J. Wan, D. Tao, and M. Wang, “Multimodal deep autoencoder for human pose recovery,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 5659–5670, Dec. 2015.

Digital Library

[63]

C. Hong, J. Yu, D. Tao, and M. Wang, “Image-based three-dimensional human pose recovery by multiview locality-sensitive sparse retrieval,” IEEE Trans. Ind. Electron., vol. 62, no. 6, pp. 3742–3751, Jun. 2015.

[64]

C. Hong, J. Yu, J. You, X. Chen, and D. Tao, “Multi-view ensemble manifold regularization for 3D object recognition,” Inf. Sci., vol. 320, pp. 395–405, Nov. 2015.

Digital Library

[65]

L. Zhu, J. Shen, L. Xie, and Z. Cheng, “Unsupervised topic hypergraph hashing for efficient mobile image retrieval,” IEEE Trans. Cybern., vol. 47, no. 11, pp. 3941–3954, Nov. 2017.

[66]

L. Zhu, J. Shen, L. Xie, and Z. Cheng, “Unsupervised visual hashing with semantic assistant for content-based image retrieval,” IEEE Trans. Knowl. Data Eng., vol. 29, no. 2, pp. 472–486, Feb. 2017.

Digital Library

[67]

L. Xie, J. Shen, J. Han, L. Zhu, and L. Shao, “Dynamic multi-view hashing for online image retrieval,” in Proc. IJCAI, 2017, pp. 3133–3139.

[68]

S. Wang, Q. Huang, S. Jiang, and Q. Tian, “S³MKL: Scalable semi-supervised multiple kernel learning for real-world image applications,” IEEE Trans. Multimedia, vol. 14, no. 4, pp. 1259–1274, Aug. 2012.

Digital Library

[69]

J. Han, D. Zhang, G. Cheng, N. Liu, and D. Xu, “Advanced deep-learning techniques for salient and category-specific object detection: A survey,” IEEE Signal Process. Mag., vol. 35, no. 1, pp. 84–100, Jan. 2018.

[70]

J. Han, R. Quan, D. Zhang, and F. Nie, “Robust object co-segmentation using background prior,” IEEE Trans. Image Process., vol. 27, no. 4, pp. 1639–1651, Apr. 2018.

Digital Library

[71]

X. Yao, J. Han, D. Zhang, and F. Nie, “Revisiting co-saliency detection: A novel approach based on two-stage multi-view spectral rotation co-clustering,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3196–3209, Jul. 2017.

Digital Library

[72]

X. Qian, X. Lu, J. Han, B. Du, and X. Li, “On combining social media and spatial technology for poi cognition and image localization,” Proc. IEEE, vol. 105, no. 10, pp. 1937–1952, Oct. 2017.

[73]

X. Qian, C. Li, K. Lan, X. Hou, Z. Li, and J. Han, “POI summarization by aesthetics evaluation from crowd source social media,” IEEE Trans. Image Process., vol. 27, no. 3, pp. 1178–1189, Mar. 2018.

Cited By

Wang SShen JEfthymiou ARudinac SKackovic MWijnberg NWorring M(2024)Prototype-Enhanced Hypergraph Learning for Heterogeneous Information NetworksMultiMedia Modeling10.1007/978-3-031-53311-2_34(462-476)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-53311-2_34
Yang TZhou S(2023)Iteratively Reweighted Hypergraph Subspace ClusteringProceedings of the 2023 7th International Conference on Computer Science and Artificial Intelligence10.1145/3638584.3638587(183-189)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3638584.3638587
Xu QLin JJiang BLiu JLuo B(2023)Hypergraph convolutional network for hyperspectral image classificationNeural Computing and Applications10.1007/s00521-023-08935-w35:29(21863-21882)Online publication date: 22-Aug-2023
https://dl.acm.org/doi/10.1007/s00521-023-08935-w
Show More Cited By

Index Terms

Joint Hypergraph Learning for Tag-Based Image Retrieval
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
  2. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Tag-based social image search with visual-text joint hypergraph learning
MM '11: Proceedings of the 19th ACM international conference on Multimedia

Tag-based social image search has attracted great interest and how to order the search results based on relevance level is a research problem. Visual content of images and tags have both been investigated. However, existing methods usually employ tags ...
Learning tag relevance by neighbor voting for social image retrieval
MIR '08: Proceedings of the 1st ACM international conference on Multimedia information retrieval

Social image retrieval is important for exploiting the increasing amounts of amateur-tagged multimedia such as Flickr images. Since amateur tagging is known to be uncontrolled, ambiguous, and personalized, a fundamental problem is how to reliably ...
Improving social tag-based image retrieval with CBIR technique
ICADL'10: Proceedings of the role of digital libraries in a time of global change, and 12th international conference on Asia-Pacific digital libraries

With the popularity of social image-sharing websites, the amount of images uploaded and shared among the users has increased explosively. To allow keyword search, the system constructs an index from image tags assigned by the users. The tag-based image ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 27, Issue 9

Sept. 2018

92 pages

ISSN:1057-7149

Issue’s Table of Contents

1057-7149 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 September 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang SShen JEfthymiou ARudinac SKackovic MWijnberg NWorring M(2024)Prototype-Enhanced Hypergraph Learning for Heterogeneous Information NetworksMultiMedia Modeling10.1007/978-3-031-53311-2_34(462-476)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1007/978-3-031-53311-2_34
Yang TZhou S(2023)Iteratively Reweighted Hypergraph Subspace ClusteringProceedings of the 2023 7th International Conference on Computer Science and Artificial Intelligence10.1145/3638584.3638587(183-189)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1145/3638584.3638587
Xu QLin JJiang BLiu JLuo B(2023)Hypergraph convolutional network for hyperspectral image classificationNeural Computing and Applications10.1007/s00521-023-08935-w35:29(21863-21882)Online publication date: 22-Aug-2023
https://dl.acm.org/doi/10.1007/s00521-023-08935-w
Zhao SWang LQian XChen J(2022)Enhancing performance-based generative architectural design with sketch-based image retrieval: a pilot study on designing building facade fenestrationsThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-021-02170-x38:8(2981-2997)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00371-021-02170-x
Wang CMa NWu ZZhang JYao Y(2022)Survey of Hypergraph Neural Networks and Its Application to Action RecognitionArtificial Intelligence10.1007/978-3-031-20500-2_32(387-398)Online publication date: 27-Aug-2022
https://dl.acm.org/doi/10.1007/978-3-031-20500-2_32
Li MDu LXu JGuo C(2021)A Hypergraph-based Method for Pharmaceutical Data Similarity RetrievalProceedings of the 4th International Conference on Big Data Technologies10.1145/3490322.3490344(134-140)Online publication date: 24-Sep-2021
https://dl.acm.org/doi/10.1145/3490322.3490344
Cui TChen LXu JXu L(2020)Robust Sparse Low-rank Hypergraph Learning under Complex Noise2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)10.1109/SMC42975.2020.9283388(4088-4094)Online publication date: 11-Oct-2020
https://dl.acm.org/doi/10.1109/SMC42975.2020.9283388
Kang CZhu LQian XHan JWang MTang Y(2019)Geometry and Topology Preserving Hashing for SIFT FeatureIEEE Transactions on Multimedia10.1109/TMM.2018.288386821:6(1563-1576)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1109/TMM.2018.2883868

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents