tutorial

Predicting Viewer Affective Comments Based on Image Content in Social Media

Authors:

Winston H. Hsu,

Hong-Yuan Mark Liao,

Shih-Fu ChangAuthors Info & Claims

ICMR '14: Proceedings of International Conference on Multimedia Retrieval

Pages 233 - 240

https://doi.org/10.1145/2578726.2578756

Published: 01 April 2014 Publication History

Abstract

Visual sentiment analysis is getting increasing attention because of the rapidly growing amount of images in online social interactions and several emerging applications such as online propaganda and advertisement. Recent studies have shown promising progress in analyzing visual affect concepts intended by the media content publisher. In contrast, this paper focuses on predicting what viewer affect concepts will be triggered when the image is perceived by the viewers. For example, given an image tagged with "yummy food," the viewers are likely to comment "delicious" and "hungry," which we refer to as viewer affect concepts (VAC) in this paper. To the best of our knowledge, this is the first work explicitly distinguishing intended publisher affect concepts and induced viewer affect concepts associated with social visual content, and aiming at understanding their correlations. We present around 400 VACs automatically mined from million-scale real user comments associated with images in social media. Furthermore, we propose an automatic visual based approach to predict VACs by first detecting publisher affect concepts in image content and then applying statistical correlations between such publisher affect concepts and the VACs. We demonstrate major benefits of the proposed methods in several real-world tasks - recommending images to invoke certain target VACs among viewers, increasing the accuracy of predicting VACs by 20.1% and finally developing a social assistant tool that may suggest plausible, content-specific and desirable comments when users view new images.

References

[1]

I. Arapakis, J. M. Jose, and P. D. Gray. Affective feedback: An investigation into the role of emotions in the information seeking process. In SIGIR, 2008.

Digital Library

[2]

Bird, Steven, E. Loper, and E. Klein. Natural language processing with python. 2009.

Digital Library

[3]

D. Borth, R. Ji, T. Chen, T. Breuel, and S.-F. Chang. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In ACM Multimedia, 2013.

Digital Library

[4]

R. F. Bruce and J. M. Wiebe. Recognizing subjectivity: A case study of manual tagging. Natural Language Engineering, 1999.

Digital Library

[5]

R. Datta, D. Joshi, J. Li, and J. Z. Wang. Studying aesthetics in photographic images using a computational approach. In ECCV, 2006.

Digital Library

[6]

A. Esuli and F. Sebastiani. Sentiwordnet: A publicly available lexical resource for opinion mining. In LREC, 2006.

[7]

A. Hanjalic. Extracting moods from pictures and sounds: Towards truly personalized tv. IEEE Signal Processing Magazine, 2006.

[8]

M. Hu and B. Liu. Mining opinion features in customer reviews. In AAAI, 2004.

Digital Library

[9]

P. Isola, J. Xiao, A. Torralba, and A. Oliva. What makes an image memorable? In CVPR, 2011.

Digital Library

[10]

P. Lang, M. Bradley, and B. Cuthbert. International affective picture system (iaps): Affective ratings of pictures and instruction manual. Technical Report A-8. University of Florida, Gainesville, FL, 2008.

[11]

D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. In NIPS, 2001.

Digital Library

[12]

J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In ACM Multimedia, 2010.

Digital Library

[13]

A. McCallum and K. Nigam. A comparison of event models for naive bayes text classification. In AAAI Workshop on Learning for Text Categorization, 1998.

[14]

M. J. Pazzani and D. Billsus. Content-based recommendation systems. In The Adaptive Web: Methods and Strategies of Web Personalization. Volume 4321 of Lecture Notes in Computer Science, 2007.

Digital Library

[15]

R. Plutchik. Emotion: A psychoevolutionary synthesis. Harper & Row, Publishers, 1980.

[16]

N. Sebe, I. Cohen, T. Gevers, and T. S. Huang. Emotion recognition based on joint visual and audio cues. In ICPR, 2006.

Digital Library

[17]

S. Siersdorfer, S. Chelaru, W. Nejdl, and J. San Pedro. How useful are your comments?: Analyzing and predicting youtube comments and comment ratings. In WWW, 2010.

Digital Library

[18]

W. Wang and Q. He. A survey on emotional semantic image retrieval. In ICIP, 2008.

[19]

J. M. Wiebe, R. F. Bruce, and T. P. O'Hara. Development and use of a gold-standard data set for subjectivity classifications. In ACL, 1999.

Digital Library

[20]

J. Yuan, Q. You, S. McDonough, and J. Luo. Sentribute: Image sentiment analysis from a mid-level perspetive. In Workshop on Sentiment Discovery and Opinion Mining, 2013.

Digital Library

[21]

L. Zhuang, F. Jing, and X.-Y. Zhu. Movie review mining and summarization. In CIKM, 2006.

Digital Library

Cited By

Moroto YYe YMaeda KOgawa THaseyama M(2024)Zero-Shot Visual Sentiment Prediction via Cross-Domain Knowledge DistillationIEEE Open Journal of Signal Processing10.1109/OJSP.2023.33440795(177-185)Online publication date: 2024
https://doi.org/10.1109/OJSP.2023.3344079
Stray JHalevy AAssar PHadfield-Menell DBoutilier CAshar ABakalar CBeattie LEkstrand MLeibowicz CMoon Sehat CJohansen SKerlin LVickrey DSingh SVrijenhoek SZhang AAndrus MHelberger NProutskova PMitra TVasan N(2023)Building Human Values into Recommender Systems: An Interdisciplinary SynthesisACM Transactions on Recommender Systems10.1145/36322972:3(1-57)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3632297
Das RSingh T(2023)Multimodal Sentiment Analysis: A Survey of Methods, Trends, and ChallengesACM Computing Surveys10.1145/358607555:13s(1-38)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3586075
Show More Cited By

Index Terms

Predicting Viewer Affective Comments Based on Image Content in Social Media
1. Information systems
  1. Information systems applications

Recommendations

Identification of useful user comments in social media: a case study on flickr commons
JCDL '13: Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries

Cultural institutions are increasingly opening up their repositories and contribute digital objects to social media platforms such as Flickr. In return they often receive user comments containing information that could be incorporated in their catalog ...
Towards social imagematics: sentiment analysis in social multimedia
MDMKDD '13: Proceedings of the Thirteenth International Workshop on Multimedia Data Mining

Online social networks have attracted attention of people from both the academia and real world. In particular, the rich multimedia information accumulated in recent years provides an easy and convenient way for more active communication between people. ...
Sentiment and Emotion Analysis for Social Multimedia: Methodologies and Applications
MM '16: Proceedings of the 24th ACM international conference on Multimedia

Online social networks have attracted the attention from both the academia and real world. In particular, the rich multimedia information accumulated in recent years provides an easy and convenient way for more active communication between people. This ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMR '14: Proceedings of International Conference on Multimedia Retrieval

April 2014

564 pages

ISBN:9781450327824

DOI:10.1145/2578726

Conference Chairs:
Mohan Kankanhalli
National University of Singapore
,
Stefan Rueger
The Open University, UK
,
R. Manmatha
A9.com, USA
,
General Chairs:
Joemon Jose
University of Glasgow, UK
,
Keith van Rijsbergen
University of Glasgow, UK

Copyright © 2014 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

In-Cooperation

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2014

Check for updates

Author Tags

Qualifiers

Tutorial
Research
Refereed limited

Conference

ICMR '14

ICMR '14: International Conference on Multimedia Retrieval

April 1 - 4, 2014

Glasgow, United Kingdom

Acceptance Rates

ICMR '14 Paper Acceptance Rate 21 of 111 submissions, 19%;

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
567
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Moroto YYe YMaeda KOgawa THaseyama M(2024)Zero-Shot Visual Sentiment Prediction via Cross-Domain Knowledge DistillationIEEE Open Journal of Signal Processing10.1109/OJSP.2023.33440795(177-185)Online publication date: 2024
https://doi.org/10.1109/OJSP.2023.3344079
Stray JHalevy AAssar PHadfield-Menell DBoutilier CAshar ABakalar CBeattie LEkstrand MLeibowicz CMoon Sehat CJohansen SKerlin LVickrey DSingh SVrijenhoek SZhang AAndrus MHelberger NProutskova PMitra TVasan N(2023)Building Human Values into Recommender Systems: An Interdisciplinary SynthesisACM Transactions on Recommender Systems10.1145/36322972:3(1-57)Online publication date: 13-Nov-2023
https://dl.acm.org/doi/10.1145/3632297
Das RSingh T(2023)Multimodal Sentiment Analysis: A Survey of Methods, Trends, and ChallengesACM Computing Surveys10.1145/358607555:13s(1-38)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3586075
Zhang MLuo GMa YLi SQian ZZhang XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)VCMaster: Generating Diverse and Fluent Live Video Comments Based on Multimodal ContextsProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612078(4688-4696)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612078
Li YHe ZLi YHuang TLiu Z(2023)Keep it real: Assessing destination image congruence and its impact on tourist experience evaluationsTourism Management10.1016/j.tourman.2023.10473697(104736)Online publication date: Aug-2023
https://doi.org/10.1016/j.tourman.2023.104736
Ye YMoroto YMaeda KOgawa THaseyama MJiang SAizawa KChen PYanai K(2022)Affective Embedding Framework with Semantic Representations from Tweets for Zero-Shot Visual Sentiment PredictionProceedings of the 4th ACM International Conference on Multimedia in Asia10.1145/3551626.3564950(1-7)Online publication date: 13-Dec-2022
https://dl.acm.org/doi/10.1145/3551626.3564950
Dwivedi-Yu JWang YQin LCanton-Ferrer CHalevy AZhang ARangwala H(2022)Affective Signals in a Social Media Recommender SystemProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3534678.3539054(2831-2841)Online publication date: 14-Aug-2022
https://dl.acm.org/doi/10.1145/3534678.3539054
Ye YMoroto YMaeda KOgawa THaseyama M(2022)Visual Sentiment Prediction Using Cross-Way Few-Shot Learning Based on Knowledge Distillation2022 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP46576.2022.9897216(3838-3842)Online publication date: 16-Oct-2022
https://doi.org/10.1109/ICIP46576.2022.9897216
Kaur HPannu HMalhi A(2019)Multimedia blog volume prediction using adaptive neuro fuzzy inference system and evolutionary algorithmsMultimedia Tools and Applications10.1007/s11042-019-07903-878:22(31673-31707)Online publication date: 24-Jul-2019
https://doi.org/10.1007/s11042-019-07903-8
Li ZFan YJiang BLei TLiu W(2019)A survey on sentiment analysis and opinion mining for social multimediaMultimedia Tools and Applications10.1007/s11042-018-6445-z78:6(6939-6967)Online publication date: 1-Mar-2019
https://dl.acm.org/doi/10.1007/s11042-018-6445-z
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten