article

Data-driven approaches for social image and video tagging

Authors:

Lamberto Ballan,

Tiberio Uricchio,

Alberto Del BimboAuthors Info & Claims

Multimedia Tools and Applications, Volume 74, Issue 4

Pages 1443 - 1468

https://doi.org/10.1007/s11042-014-1976-4

Published: 01 February 2015 Publication History

Abstract

The large success of online social platforms for creation, sharing and tagging of user-generated media has lead to a strong interest by the multimedia and computer vision communities in research on methods and techniques for annotating and searching social media. Visual content similarity, geo-tags and tag co-occurrence, together with social connections and comments, can be exploited to perform tag suggestion as well as to per-form content classification and c lustering and enable more effective semantic indexing and retrieval of visual data. However there is need to overcome the relatively low quality of these metadata: user produced tags and annotations are known to be ambiguous, imprecise and/or incomplete, excessively personalized and limited - and at the same time take into account the `web-scale' quantity of media and the fact that social network users continuously add new images and create new terms. We will review the state of the art approaches to automatic annotation and tag refinement for social images, considering also the temporal patterns of their usage, and discuss extensions to tag suggestion and localization in web video sequences.

References

[1]

Alonso O, Gertz M, Baeza-Yates R (2007) On the value of temporal information in information retrieval. SIGIR Forum 41(2): 35---41

Digital Library

[2]

Ballan L, Bertini M, Del Bimbo A, Meoni M, Serra G (2010) Tag suggestion and localization in user-generated videos based on social knowledge. In: Proceedings of ACM SIGMM Workshop on Social Media (WSM). Firenze

[3]

Ballan L, Bertini M, Del Bimbo A, Serra G (2011) Enriching and localizing semantic tags in internet videos. In: Proceedings of ACM international conference on multimedia (ACM MM), pp 1541---1544.

[4]

Choi H, Varian H (2011) Predicting the present with Google Trends. Tech. rep., Google

[5]

Chu W T, Li C J (2011) Tag suggestion and localization for web videos by bipartite graph matching. In: Proceedings of ACM SIGMM Workshop on Social Media (WSM). New York

[6]

Chua T S, Tang J, Hong R, Li H, Luo Z, Zheng Y (2009) NUS-WIDE: a real-world web image database from National University of Singapore. In: Proceedings of ACM CIVR

[7]

Cohen J (1988) Statistical power analysis for the behavioral sciences. Routledge Academic

[8]

Ginsberg J, Mohebbi M H, Patel R S, Brammer L, Smolinski M S, Brilliant L (2009) Detecting influenza epidemics using search engine query data. Nature 457(7232): 1012---1014

[9]

Guillaumin M, Mensink T, Verbeek J, Schmid C (2009) Tagprop: discriminative metric learning in nearest neighbor models for image auto-annotation. In: Proceedings of ICCV

[10]

Huiskes M J, Lew MS (2008), The MIR Flickr retrieval evaluation. In: Proceeding of ACM MIR

[11]

Huiskes MJ, Thomee B, Lew MS (2010) New trends and ideas in visual concept detection: the MIR Flickr retrieval evaluation initiative. In: Proceedings of ACM MIR, pp 527---536

[12]

Jin X, Gallagher A, Cao L, Luo J, Han J (2010) The wisdom of social multimedia: using Flickr for prediction and forecast. In: Proceedings of ACM MM, pp 1235---1244

[13]

Kennedy L S, Chang S F, Kozintsev I V (2006) To search or to label? Predicting the performance of search-based automatic image classifiers. In: Proceedings of ACM MIR

[14]

Kennedy L S, Slaney M, Weinberger K (2009) Reliable tags using image similarity: mining specificity and expertise from large-scale multimedia databases. In: Proceedings of ACM-MM Workshop on Web-Scale Multimedia Corpus. Beijing

[15]

Kim G, Xing EP (2013) Time-sensitive web image ranking and retrieval via dynamic multi-task regression. In: Proceedings of ACM WSDM, pp 163---172

[16]

Kim G, Xing EP, Torralba A (2010) Modeling and analysis of dynamic behaviors of web image collections. In: Proceedings of ECCV, pp 85---98

[17]

Kim G, Fei-Fei L, Xing EP (2012) Web image prediction using multivariate point processes. In: Proceedings of ACM SIGKDD, pp 1068---1076

[18]

Li G, Wang M, Zheng Y T, Chua T S (2011) ShotTagger: tag location for internet videos. In: Proceedings of ACM ICMR

[19]

Li H, Yi L, Guan Y, Zhang H (2013) DUT-WEBV: a benchmark dataset for performance evaluation of tag localization for web video. In: Proceedings of MMM

[20]

Li X, Snoek C G M, Worring M (2009) Learning social tag relevance by neighbor voting. IEEE Trans Multimed 11(7): 1310---1322

Digital Library

[21]

Li X, Snoek C G M, Worring M (2010a) Unsupervised multi-feature tag relevance learning for social image retrieval. In: Proceedings of ACM CIVR

[22]

Li Z, Liu J, Zhu X, Liu T, Lu H (2010b) Image annotation using multi-correlation probabilistic matrix factorization. In: Proceedings of the international conference on multimedia, MM'10. ACM, New York, pp 11871190

[23]

Liu D, Hua X S, Yang L, Wang M, Zhang HJ (2009) Tag ranking. In: Proceedings of WWW

[24]

Liu D, Hua X S, Wang M, Zhang HJ (2010) Image retagging. In: Proceedings of ACM multimedia

[25]

Liu D, Hua X S, Zhang H J (2011a) Content-based tag processing for internet social images. Multimed Tools Appl 51(1): 723---738

Digital Library

[26]

Liu D, Yan S, Hua X S, Zhang H J (2011b) Image retagging using collaborative tag propagation. IEEE Trans Multimed 13(4): 702---712

Digital Library

[27]

Liu Y, Jin R, Yang L (2006) Semi-supervised multi-label learning by constrained non-negative matrix factorization. In: AAAI-06: proceedings of the ninth national conference on artificial intelligence, vol 21. AAAI Press, p 421

[28]

Makadia A, Pavlovic V, Kumar S (2008) A new baseline for image annotation. In: Proceedings of ECCV

[29]

Min H S, Choi J, De Neve W, Ro Y M, Plataniotis K N (2009) Semantic annotation of personal video content using an image folksonomy. In: Proceedings of IEEE ICIP

[30]

Paisitkriangkrai S, Mei T, Zhang J, Hua X S (2010) Scalable clip-based near-duplicate video detection with ordinal measure. In: Proceedings of ACM CIVR

[31]

Rattenbury T, Good N, Naaman M (2007) Towards automatic extraction of event and place semantics from flickr tags. In: Proceedings of ACM SIGIR, pp 103---110

[32]

Salakhutdinov R, Mnih A (2008) Probabilistic matrix factorization. Adv. Neural Info Process Syst 20: 1257---1264

[33]

Sang J, Xu C, Liu J (2012) User-aware image tag refinement via ternary semantic analysis. IEEE Trans Multimed 14(3): 883---895

Digital Library

[34]

Shao J, Yin W, Ma S, Zhuang Y (2010) Topic discovery of web video using star-structured k-partite graph. In: Proceedings of ACM multimedia

[35]

Sigurbjörnsson B, van Zwol R (2008) Flickr tag recommendation based on collective knowledge. In: Proceedings of WWW, pp 327---336

[36]

Thomee B, Bakker EM, Lew MS (2010) TOP-SURF: a visual words toolkit. In: Proceedings of ACM multimedia.

[37]

Tsai D, Jing Y, Liu Y, Rowley H A, Ioffe S, Rehg J M (2011) Large-scale image annotation using visual synset. In: 2011 IEEE International conference on computer vision (ICCV). IEEE, pp 611---618

[38]

Ulges A, Schulze C, Koch M, Breuel T M (2010) Learning automatic concept detectors from online video. Comp Vision Image Underst 114(4): 429---438

Digital Library

[39]

Verbeek J, Guillaumin M, Mensink T, Schmid C (2010) Image annotation with TagProp on the MIRFLICKR set. In: Proceedings of ACM MIR

[40]

von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Proceedings of ACMCHI

[41]

Wang C, Jing F, Zhang L, Zhang HJ (2007) Content-based image annotation refinement. In: Proceedings of CVPR

[42]

Zhang ML, Zhou ZH (2004) Improve multi-instance neural networks through feature selection. Neural Process Lett 19(1):1---10.

Digital Library

[43]

Zhu G, Yan S, Ma Y (2010) Image tag refinement towards low-rank. In: Proceedings of ACM multimedia

Cited By

Wu CWang CZhou YWu DChen MWang JQin J(2020)Exploiting user reviews for automatic movie taggingMultimedia Tools and Applications10.1007/s11042-019-08513-079:17-18(11399-11419)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s11042-019-08513-0
Deng XLi YWeng JZhang J(2019)Feature selection for text classificationMultimedia Tools and Applications10.1007/s11042-018-6083-578:3(3797-3816)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11042-018-6083-5
Cui CShen JMa JLian T(2017)Social tag relevance learning via ranking-oriented neighbor votingMultimedia Tools and Applications10.1007/s11042-016-3512-176:6(8831-8857)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11042-016-3512-1
Show More Cited By

Index Terms

Data-driven approaches for social image and video tagging
1. Information systems
  1. Information retrieval

Index terms have been assigned to the content through auto-classification.

Recommendations

A data-driven approach for tag refinement and localization in web videos

Our approach locates the temporal positions of tags in videos at the keyframe level.We deal with a scenario in which there is no pre-defined set of tags.We report experiments about the use of different web sources (Flickr, Google, Bing).We show state-of-...
Tagging photos using users' vocabularies

Online social image share websites such as Flickr and Panoramio allow users to manually annotate their images with their own words, which can be used to facilitating image retrieval and other image applications. The smart-phones have made it possible ...
Tagboards for video tagging
MM '08: Proceedings of the 16th ACM international conference on Multimedia

There is an explosion of media content, especially user generated content, on the Internet. Efficient search of media data still remains a distant goal. Tagging has been used recently to bridge the "semantic gap" for images. Tagging of video data poses ...

Comments

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 74, Issue 4

February 2015

393 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 February 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu CWang CZhou YWu DChen MWang JQin J(2020)Exploiting user reviews for automatic movie taggingMultimedia Tools and Applications10.1007/s11042-019-08513-079:17-18(11399-11419)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s11042-019-08513-0
Deng XLi YWeng JZhang J(2019)Feature selection for text classificationMultimedia Tools and Applications10.1007/s11042-018-6083-578:3(3797-3816)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s11042-018-6083-5
Cui CShen JMa JLian T(2017)Social tag relevance learning via ranking-oriented neighbor votingMultimedia Tools and Applications10.1007/s11042-016-3512-176:6(8831-8857)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11042-016-3512-1
Ellis JKaraman SLi HShim HChang SHanjalic ASnoek CWorring MBulterman DHuet BKelliher AKompatsiaris YLi J(2016)Placing Broadcast News Videos in their Social Media Context Using HashtagsProceedings of the 24th ACM international conference on Multimedia10.1145/2964284.2970929(684-688)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1145/2964284.2970929
Li XUricchio TBallan LBertini MSnoek CBimbo A(2016)Socializing the Semantic GapACM Computing Surveys10.1145/290615249:1(1-39)Online publication date: 6-Jun-2016
https://dl.acm.org/doi/10.1145/2906152
Ballan LBertini MSerra GBimbo A(2015)A data-driven approach for tag refinement and localization in web videosComputer Vision and Image Understanding10.1016/j.cviu.2015.05.009140:C(58-67)Online publication date: 1-Nov-2015
https://dl.acm.org/doi/10.1016/j.cviu.2015.05.009

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents