Article

Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching

Authors:

Xiangdong Zhou,

Baile ShiAuthors Info & Claims

CIVR '07: Proceedings of the 6th ACM international conference on Image and video retrieval

Pages 25 - 32

https://doi.org/10.1145/1282280.1282284

Published: 09 July 2007 Publication History

Abstract

Automatic image annotation automatically labels image content with semantic keywords. For instance, the Relevance Model estimates the joint probability of the keyword and the image [3]. Most of the previous annotation methods assign keywords separately. Recently the correlation between annotated keywords has been used to improve image annotation. However, directly estimating the joint probability of a set of keywords and the unlabeled image is computationally prohibitive. To avoid the computation difficulty we propose a heuristic greedy iterative algorithm to estimate the probability of a keyword subset being the caption of an image. In our approach, the correlations between keywords are analyzed by "Automatic Local Analysis" of text information retrieval. In addition, a new image generation probability estimation method is proposed based on region matching. We demonstrate that our iterative annotation algorithm can incorporate the keyword correlations and the region matching approaches handily to improve the image annotation significantly. The experiments on the ECCV2002 [2] benchmark show that our method outperforms the state-of-the-art continuous feature model MBRM with recall and precision improving 21% and 11% respectively.

References

[1]

Lew, Sebe, Djeraba, Jain. Content-based Multimedia Information Retrieval: State of the Art and Challenges, ACM Transactions on Multimedia Computing, Communications, and Applications, 2006

Digital Library

[2]

P. Duygulu, K. Barnard, J. F. G. de Freitas, and D. A. Forsyth. Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary. ECCV 2002, pages 97--112.

Digital Library

[3]

J. Jeon, V. Lavrenko, and R. Manmatha. Automatic Image Annotation and Retrieval using Cross-Media Relevance Models. In Proc. of SIGIR'03, 2003

Digital Library

[4]

S. L. Feng, R. Manmatha, V. Lavrenko. Multiple Bernoulli Relevance Models for Image and Video Annotation. In Proc. of IEEE CVPR'04, 2004.

Digital Library

[5]

V. Lavrenko, R. Manmatha, and J. Jeon. A Model for Learning the Semantics of Pictures. In Proc. of NIPS'03, 2003.

[6]

J. Wang, J. Li, and G. Wiederhold. SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries. IEEE transactions on Pattern Analysis and Machine Intelligence, Vol. 23(9), 2001.

Digital Library

[7]

Ruofei Zhang, Zhongfei (Mark) Zhang. A Clustering Based Approach to Efficient Image Retrieval. In Proc. of 14th IEEE Int'l Conf. on Tools with Artificial Intelligence (ICTAI 2002) 4--6 Nov. 2002 Page(s):339--346.

Digital Library

[8]

H. W. Kuhn, The Hungarian Method for the Assignment Problem, Naval Research Logistics Quarterly, vol.2, p. 83--97, 1955.

[9]

R. Jin, J. Y. Chai, and L. Si. Effective Automatic Image Annotation via A Coherent Language Model and Active Learning. In Proc. of ACM Int'l Conf. on Multimedia, 2004.

Digital Library

[10]

Y. Jin, L. Khan, L. Wang, M. Awad. Image Annotations By Combining Multiple Evidence & WordNet. In proc. of ACM Inte'l Conf. on Multimedia, pages 706--715, 2005.

Digital Library

[11]

Ricardo Baeza-Yates, Berthier Ribeiro-Neto. Modern Information Retrieval, ACM press, 123--129, 1999.

Digital Library

[12]

Q. Zhang, S. A. Goldman, W. Yu, and J. Fritts, Content-Based Image Retrieval Using Multiple-Instance Learning, In proc. 19th Int'l Conf. Machine Learning, pp.682--689, 2002.

Digital Library

[13]

M. Srikanth, J. Varner, M. Bowden, D. Moldovan. Exploiting Ontologies for Automatic Image Annotation. In proc. of ACM SIGIR'05, 2005.

Digital Library

[14]

Monay, F. and D. Gatica-Perez. On Image Auto-Annotation with Latent Space Models. in Proc. of ACM Int'l Conf. on Multimedia, 2003.

Digital Library

[15]

Monay, F. and D. GaticaPerez. PLSA-based Image AutoAnnotation: Constraining the Latent Space. In proc. of ACM Int'l Conf. on Multimedia, 2004.

Digital Library

[16]

Y. Gao, J. Fan, X. Xue, R. Jain. Automatic Image Annotation by Incorporating Feature Hierarchy and Boosting to Scale up SVM Classifiers. in Proc. of ACM Int'l Conf. on Multimedia, 2006.

Digital Library

[17]

Jia Li, J. Z. Wang. Real-Time Computerized Annotation of Picture. In porc. of ACM Int'l Conf. on Multimedia, 2006.

Digital Library

[18]

T. Hastie, R. Tibshirani, J. Friedman. The Element of Statistical Learining: data mining, inference, and prediction. Springer-Verlag, 2001.

[19]

Fei Song, W. Bruce Croft. A General Language Model for Information Retrieval. In proc. of ACM Int'l Conf. SIGIR 1999.

Digital Library

[20]

J. Shi and J. Malik. Normalized Cuts and Image Segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence. Vol.22:888--905, 2000.

Digital Library

[21]

R. Shi, T. S. Chua, C. H. lee, S. Gao. Bayesian Learning of Hierarchical Multinomial Mixture Models of Concepts for Automatic Image Annotation. In proc. of Int'l Conf. on Image and Video Retrieval, 2006.

Digital Library

[22]

T. Wang, Y. Rui, J. G. Sun, Constraint Based Region matching for Image Retrieval, IJCV, 2004.

Digital Library

[23]

A. Yavlinsky, E. Schofield and S. Ruger. Annotation Using Global Features and Robust Nonparametric Density Estimation. In proc. of Int'l Conf. on Image and Video Retrieval, 2005.

Digital Library

[24]

K. Barnard, P. Duygululu, D. Forsyth, D. Blei M. Jordan. Matching words and pictures, The Journal of Machine Learning Research V.3 2003

Digital Library

[25]

G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross and K. J. Miller. Introduction to WordNet: an on-line lexical database. Intl. Jour. of Lexicography, vol. 3, pp. 235--244, 1990.

[26]

J. Liu, M. Li, W.-Y. Ma, Q. Liu, H. Lu, An Adaptive Graph Model for Automatic Image Annotation. ACM Int'l Conf. on Multimedia Information Retrieval, 2006.

Digital Library

[27]

X. Zhou, L. Chen, J. Ye, Q. Zhang, B. Shi, Automatic Image Semantic Annotation Based on Keyword-Image Document Model. In proc. of Int'l Conf. on Image and Video Retrieval, 2005.

Digital Library

Cited By

Tang JQin TKong DZhou ZLi XWu YGu J(2023)Anomaly Detection in Social-Aware IoT NetworksIEEE Transactions on Network and Service Management10.1109/TNSM.2023.324232020:3(3162-3176)Online publication date: Sep-2023
https://doi.org/10.1109/TNSM.2023.3242320
Jin CLiu J(2017)An Improvement Approach Based on the Label Correlation for Automatic Image AnnotationInternational Journal of Electrical Energy10.18178/ijoee.5.1.96-1005:1(96-100)Online publication date: 2017
https://doi.org/10.18178/ijoee.5.1.96-100
Pobar MIvasic-Kos M(2016)Automatic image annotation refinement2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)10.1109/MIPRO.2016.7522345(1324-1329)Online publication date: May-2016
https://doi.org/10.1109/MIPRO.2016.7522345
Show More Cited By

Index Terms

Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
2. Information systems
  1. Information retrieval

Recommendations

A graph-based image annotation framework

Automatic image annotation is crucial for keyword-based image retrieval because it can be used to improve the textual description of images. In this paper, we propose a unified framework for image annotation, which contains two kinds of learning ...
Effective image annotation for searches using multilevel semantics

There is an increasing need for automatic image annotation tools to enable effective image searching in digital libraries. In this paper, we present a novel probabilistic model for image annotation based on content-based image retrieval techniques and ...
Image automatic annotation via multi-view deep representation

A deep neural network is used as a directed supervised classification for mapping image features to keywords.Stacked auto-encoder is modified by sigmoid predictor and iteration algorithm.Tag frequencies and properties should be paid attention to during ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIVR '07: Proceedings of the 6th ACM international conference on Image and video retrieval

July 2007

655 pages

ISBN:9781595937339

DOI:10.1145/1282280

General Chairs:
Nicu Sebe
Univ. of Amsterdam, The Netherlands
,
Marcel Worring
Univ. of Amsterdam, The Netherlands

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CIVR07

Sponsor:

SIGMM

CIVR07: International Conference on Image and Video Retrieval 2007

July 9 - 11, 2007

Amsterdam, The Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
958
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tang JQin TKong DZhou ZLi XWu YGu J(2023)Anomaly Detection in Social-Aware IoT NetworksIEEE Transactions on Network and Service Management10.1109/TNSM.2023.324232020:3(3162-3176)Online publication date: Sep-2023
https://doi.org/10.1109/TNSM.2023.3242320
Jin CLiu J(2017)An Improvement Approach Based on the Label Correlation for Automatic Image AnnotationInternational Journal of Electrical Energy10.18178/ijoee.5.1.96-1005:1(96-100)Online publication date: 2017
https://doi.org/10.18178/ijoee.5.1.96-100
Pobar MIvasic-Kos M(2016)Automatic image annotation refinement2016 39th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)10.1109/MIPRO.2016.7522345(1324-1329)Online publication date: May-2016
https://doi.org/10.1109/MIPRO.2016.7522345
Gu YQian XLi QWang MHong RTian Q(2015)Image Annotation by Latent Community Detection and Multikernel LearningIEEE Transactions on Image Processing10.1109/TIP.2015.244350124:11(3450-3463)Online publication date: Nov-2015
https://doi.org/10.1109/TIP.2015.2443501
Wang LZhou TLee YCheoi KRyu K(2015)An efficient refinement algorithm for multi-label image annotation with correlation modelTelecommunications Systems10.1007/s11235-015-0030-960:2(285-301)Online publication date: 1-Oct-2015
https://dl.acm.org/doi/10.1007/s11235-015-0030-9
Wang MXia XLe JZhou X(2014)Effective automatic image annotation via integrated discriminative and generative modelsInformation Sciences: an International Journal10.1016/j.ins.2013.11.005262(159-171)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.1016/j.ins.2013.11.005
Zhang XZhao XLi ZXia JJain RChao W(2013)Social image tagging using graph-based reinforcement on multi-type interrelated objectsSignal Processing10.1016/j.sigpro.2012.05.02193:8(2178-2189)Online publication date: 1-Aug-2013
https://dl.acm.org/doi/10.1016/j.sigpro.2012.05.021
Zhang XLi ZChao W(2013)Improving image tags by exploiting web search resultsMultimedia Tools and Applications10.1007/s11042-011-0863-562:3(601-631)Online publication date: 1-Feb-2013
https://dl.acm.org/doi/10.1007/s11042-011-0863-5
Wang ZFeng D(2012)Discovering Semantics from Visual InformationMachine Learning10.4018/978-1-60960-818-7.ch808(1981-2009)Online publication date: 2012
https://doi.org/10.4018/978-1-60960-818-7.ch808
Zeqing Z(2012)A Semantic Annotation Method for Network ImageProceedings of the 2012 International Conference on Industrial Control and Electronics Engineering10.1109/ICICEE.2012.478(1807-1810)Online publication date: 23-Aug-2012
https://dl.acm.org/doi/10.1109/ICICEE.2012.478
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents