Abstract
Background
Sentiment analysis (SA) has turned out to be a new pattern in social networking, avidly helping people to realize views expressed in user-generated content and conventional platforms of social media. For performing numerous social media analytics tasks, SA of online user-produced content is vital. The performance of the sentiment classifiers utilizing a single modality, i.e., visual or textual, is still not matured because of the wide variety of data platforms.
Methods
In this paper, we propose a new framework called VIsual-TExtual SA (VITESA) that carries out visual analysis and textual analysis for polarity classification. In the VITESA framework, Brownian Movement-based Meerkat Clan Algorithm-centered DenseNet (BMMCA-DenseNet) is proposed that integrates textual and visual information for robust SA. In the visual phase, the images that are in the Flickr dataset are taken as input, and the operations: (1) preprocessing (2) feature extraction, and (3) feature selection utilizing Improved Coyote Optimization Algorithm (ICOA) are executed. In the textual phase, the user comments as of the Twitter dataset are taken as input, and the operations: (1) preprocessing (2) word embedding using adaptive Embedding for Language Models (ELMo), (3) emoticon and non-emoticon feature extraction, and SentiWordNet polarity assignment is carried out. The final stages of both phases are given as input to the proposed BMMCA-DenseNet classifier and intended to categorize the data into positive and negative polarity. The performance of BMMCA-DenseNet is compared with certain existing algorithms, and various performance metrics are evaluated.
Results
The proposed BMMCA-DenseNet classifier performs the polarity classification of the visual-textual data into two classes: positive or negative. The classifier categorizes the polarity of visual-textual data comprising 97% of accuracy, 94.44% of precision, 94.41% of recall, 94.41% of F-measure, 91.75% of Matthew’s Correlation Coefficient, 94.43% of sensitivity, 97.13% of specificity, and also minimal error.
Conclusions
The experiment is performed to evaluate the performance of the proposed method. The outcomes exhibit that BMMCA-DenseNet attains remarkable performance over other existing techniques. The result enhances the textual-visual communication systematically to improve sentiment prediction utilizing both information sources.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Chakraborty K, Bhatia S, Bhattacharyya S, Platos J, Bag R, Hassanien AE. Sentiment analysis of covid-19 tweets by deep learning classifiers–a study to show how popularity is affecting accuracy in social media. Appl Soft Comput. 2020;97:106754. https://doi.org/10.1016/j.asoc.2020.106754.
Huang F, Zhang X, Zhao Z, Xu J, Li Z. Image-text sentiment analysis via deep multimodal attentive fusion. Knowl-Based Syst. 2019;167:26–37. https://doi.org/10.1016/j.knosys.2019.01.019.
Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y. Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput. 2019;80:387–99. https://doi.org/10.1016/j.asoc.2019.04.010.
Kumar A, Srinivasan K, Cheng WH, Zomaya AY. Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data. Inf Process Manag. 2020;57(1):102141. https://doi.org/10.1016/j.ipm.2019.102141.
Poria S, Cambria E, Howard N, Huang GB, Hussain A. Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing. 2016;174:50–9. https://doi.org/10.1016/j.neucom.2015.01.095.
Vashishtha S, Susan S. Fuzzy rule based unsupervised sentiment analysis from social media posts. Expert Syst Appl. 2019;138:112834. https://doi.org/10.1016/j.eswa.2019.112834.
Zhao Z, Zhu H, Xue Z, Liu Z, Tian J, Chua MCH, Liu M. An image-text consistency driven multimodal sentiment analysis approach for social media. Inf Process Manag. 2019;56(6):102097. https://doi.org/10.1016/j.ipm.2019.102097.
Yang C, Zhang H, Jiang B, Li K. Aspect-based sentiment analysis with alternating coattention networks. Inf Process Manag. 2019;56(3):463–78. https://doi.org/10.1016/j.ipm.2018.12.004.
Asif M, Ishtiaq A, Ahmad H, Aljuaid H, Shah J. Sentiment analysis of extremism in social media from textual information. Telematics Inform. 2020;48:101345. https://doi.org/10.1016/j.tele.2020.101345.
Wu C, Wu F, Wu S, Yuan Z, Liu J, Huang Y. Semi-supervised dimensional sentiment analysis with variational autoencoder. Knowl-Based Syst. 2019;165:30–9. https://doi.org/10.1016/j.knosys.2018.11.018.
Yadav A, Vishwakarma DK. Sentiment analysis using deep learning architectures: a review. Artif Intell Rev. 2020;53(6):4335–85. https://doi.org/10.1007/s10462-019-09794-5.
Onan A, Korukoğlu S, Bulut H. A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification. Expert Syst Appl. 2016;62:1–16. https://doi.org/10.1016/j.eswa.2016.06.005.
Paolanti M, Kaiser C, Schallner R, Frontoni E, Zingaretti P. Visual and textual sentiment analysis of brand-related social media pictures using deep convolutional neural networks. In: International conference on image analysis and processing, pp. 402–413. Springer (2017). https://doi.org/10.1007/978-3-319-68560-1_36
Ullah MA, Marium SM, Begum SA, Dipa NS. An algorithm and method for sentiment analysis using the text and emoticon. ICT Express. 2020;6(4):357–60. https://doi.org/10.1016/j.icte.2020.07.003.
Xu J, Li Z, Huang F, Li C, Philip SY. Social image sentiment analysis by exploiting multimodal content and heterogeneous relations. IEEE Trans Ind Inf. 2020;17(4):2974–82. https://doi.org/10.1109/TII.2020.3005405.
Li Z, Fan Y, Liu W, Wang F. Image sentiment prediction based on textual descriptions with adjective noun pairs. Multimedia Tools Appl. 2018;77(1):1115–32. https://doi.org/10.1007/s11042-016-4310-5.
Song K, Yao T, Ling Q, Mei T. Boosting image sentiment analysis with visual attention. Neurocomputing. 2018;312:218–28. https://doi.org/10.1016/j.neucom.2018.05.104.
Abid F, Li C, Alam M. Multi-source social media data sentiment analysis using bidirectional recurrent convolutional neural networks. Comput Commun. 2020;157:102–15. https://doi.org/10.1016/j.comcom.2020.04.002.
Dashtipour K, Gogate M, Li J, Jiang F, Kong B, Hussain A. A hybrid persian sentiment analysis framework: integrating dependency grammar based rules and deep neural networks. Neurocomputing. 2020;380:1–10. https://doi.org/10.1016/j.neucom.2019.10.009.
Jindal K, Aron R. A systematic study of sentiment analysis for social media data. In: Materials today: proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.048.
Basiri ME, Nemati S, Abdar M, Cambria E, Acharya UR. Abcdm: An attention-based bidirectional cnn-rnn deep model for sentiment analysis. Futur Gener Comput Syst. 2021;115:279–94. https://doi.org/10.1016/j.future.2020.08.005.
Hajek P, Barushka A, Munk M. Opinion mining of consumer reviews using deep neural networks with word-sentiment associations. In: IFIP international conference on artificial intelligence applications and innovations, pp. 419–429. Springer (2020). https://doi.org/10.1007/978-3-030-49161-1_35.
Han Y, Liu M, Jing W. Aspect-level drug reviews sentiment analysis based on double bigru and knowledge transfer. IEEE Access. 2020;8:21314–25. https://doi.org/10.1109/ACCESS.2020.2969473.
Kumar A, Narapareddy VT, Srikanth VA, Malapati A, Neti LBM. Sarcasm detection using multi-head attention based bidirectional lstm. IEEE Access. 2020;8:6388–97. https://doi.org/10.1109/ACCESS.2019.2963630.
Xiong H, Liu Q, Song S, Cai Y. Region-based convolutional neural network using group sparse regularization for image sentiment classification. EURASIP J Image Video Process. 2019;2019(1):1–9. https://doi.org/10.1186/s13640-019-0433-8.
Susanto Y, Cambria E, Ng BC, Hussain A. Ten years of sentic computing. Cognit Comput. 2021. https://doi.org/10.1007/s12559-021-09824-x.
Cambria E, Speer R, Havasi C, Hussain A. Senticnet: A publicly available semantic resource for opinion mining. In: AAAI fall symposium: commonsense knowledge, vol. 10. Citeseer (2010).
Cambria E, Havasi C, Hussain A. Senticnet 2: A semantic and affective resource for opinion mining and sentiment analysis. In: Twenty-Fifth international FLAIRS conference (2012).
Cambria E, Olsher D, Rajagopal D. Senticnet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis. In: Proceedings of the AAAI conference on artificial intelligence, vol. 28 (2014).
Cambria E, Poria S, Bajpai R, Schuller B. Senticnet 4: A semantic resource for sentiment analysis based on conceptual primitives. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers, pp. 2666–2677 (2016).
Cambria E, Poria S, Hazarika D, Kwok K. Senticnet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In: Proceedings of the AAAI conference on artificial intelligence, vol. 32 (2018).
Vilares D, Peng H, Satapathy R, Cambria E. Babelsenticnet: a commonsense reasoning framework for multilingual sentiment analysis. In: 2018 IEEE symposium series on computational intelligence (SSCI), pp. 1292–1298. IEEE (2018). https://doi.org/10.1109/SSCI.2018.8628718.
Dragoni M, Donadello I, Cambria E. Ontosenticnet 2: enhancing reasoning within sentiment analysis. IEEE Intell Syst. 2021;36:5.
Cambria E, Li Y, Xing FZ, Poria S, Kwok K. Senticnet 6: Ensemble application of symbolic and subsymbolic AI for sentiment analysis. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp. 105–114 (2020). https://doi.org/10.1145/3340531.3412003.
Akhtar S, Ghosal D, Ekbal A, Bhattacharyya P, Kurohashi S. All-in-one: Emotion, sentiment and intensity prediction using a multi-task ensemble framework. IEEE Trans Affect Comput. 2019. https://doi.org/10.1109/TAFFC.2019.2926724.
Akhtar MS, Ekbal A, Cambria E. How intense are you? predicting intensities of emotions and sentiments using stacked ensemble [application notes]. IEEE Comput Intell Mag. 2020;15(1):64–75. https://doi.org/10.1109/MCI.2019.2954667.
Stappen L, Baird A, Cambria E, Schuller BW. Sentiment analysis and topic recognition in video transcriptions. IEEE Intell Syst. 2021;36(2):88–95. https://doi.org/10.1109/MIS.2021.3062200.
Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowl-Based Syst. 2018;161:124–33. https://doi.org/10.1016/j.knosys.2018.07.041.
Wöllmer M, Weninger F, Knaup T, Schuller B, Sun C, Sagae K, Morency LP. Youtube movie reviews: sentiment analysis in an audio-visual context. IEEE Intell Syst. 2013;28(3):46–53. https://doi.org/10.1109/MIS.2013.34.
Poria S, Peng H, Hussain A, Howard N, Cambria E. Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing. 2017;261:217–30. https://doi.org/10.1016/j.neucom.2016.09.117.
Xu G, Li W, Liu J. A social emotion classification approach using multi-model fusion. Futur Gener Comput Syst. 2020;102:347–56. https://doi.org/10.1016/j.future.2019.07.007.
Dashtipour K, Gogate M, Cambria E, Hussain A. A novel context-aware multimodal framework for persian sentiment analysis. Neurocomputing. 2021. https://doi.org/10.1016/j.neucom.2021.02.020.
SIFT Image Features. https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/MURRAY/SIFT.html.
Pierezan J, dos Santos Coelho L, Mariani VC, de Vasconcelos Segundo EH, Prayogo D. Chaotic coyote algorithm applied to truss optimization problems. Comput Struct. 2021;242:106353. https://doi.org/10.1016/j.compstruc.2020.106353.
Liu F, Zheng L, Zheng J. Hienn-dwe: a hierarchical neural network with dynamic word embeddings for document level sentiment classification. Neurocomputing. 2020;403:21–32. https://doi.org/10.1016/j.neucom.2020.04.084.
Hamouda A, Rohaim M. Reviews classification using sentiwordnet lexicon. In: World congress on computer science and information technology, vol. 23, pp. 104–105. sn (2011).
https://www.baeldung.com/cs/sentiment-analysis-dictionaries.
Ruiz P. Understanding and visualizing DenseNets (2018). https://towardsdatascience.com/understanding-and-visualizing-densenets-7f688092391a.
Al-Obaidi ATS, Abdullah HS, Ahmed ZO. Meerkat clan algorithm: a new swarm intelligence algorithm. Indones J Electr Eng Comput Sci. 2018;10(1):354–60. https://doi.org/10.11591/ijeecs.v10.i1.pp354-360.
Fake News. https://kaggle.com/c/fake-news.
Flickr8K. https://kaggle.com/shadabhussain/flickr8k.
Funding
This work has no funding resource.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics approval and consent to participate
This article does not contain any studies with human participants or animals performed by any of the authors.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jindal, K., Aron, R. A Novel Visual-Textual Sentiment Analysis Framework for Social Media Data. Cogn Comput 13, 1433–1450 (2021). https://doi.org/10.1007/s12559-021-09929-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12559-021-09929-3