research-article

Open access

Multi-channel Convolutional Neural Network for Precise Meme Classification

Authors:

Victoria Sherratt,

Kevin Pimbblet,

Nina DethlefsAuthors Info & Claims

ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

Pages 190 - 198

https://doi.org/10.1145/3591106.3592275

Published: 12 June 2023 Publication History

All formats PDF

Abstract

This paper proposes a multi-channel convolutional neural network (MC-CNN) for classifying memes and non-memes. Our architecture is trained and validated on a challenging dataset that includes non-meme formats with textual attributes, which are also circulated online but rarely accounted for in meme classification tasks. Alongside a transfer learning base, two additional channels capture low-level and fundamental features of memes that make them unique from other images with text. We contribute an approach which outperforms previous meme classifiers specifically in live data evaluation, and one that is better able to generalise ‘in the wild’. Our research aims to improve accurate collation of meme content to support continued research in meme content analysis, and meme-related sub-tasks such as harmful content detection.

Supplemental Material

ZIP File

Details of dataset compilation, Twitter URLs for live evaluation and parameters for generating training data for meme specific optical character recognition model.

Download
23.44 MB

References

[1]

Tariq Habib Afridi, Aftab Alam, Muhammad Numan Khan, Jawad Khan, and Young-Koo Lee. 2020. A Multimodal Memes Classification: A Survey and Open Research Issues. (Sept. 2020). https://doi.org/10.48550/arXiv.2009.08395 arXiv:https://arxiv.org/abs/2009.08395

[2]

Library of Congress American Folklore Centre. [n. d.]. Meme Generator: collected datasets. Available at: https://www.loc.gov/item/2018655320/ (2022-05-10).

[3]

Kate Barnes, Tiernon Riesenmy, Minh Duc Trinh, Eli Lleshi, Nóra Balogh, and Roland Molontay. 2021. Dank or not? Analyzing and predicting the popularity of memes on Reddit. Applied Network Science 6, 1 (March 2021). https://doi.org/10.1007/s41109-021-00358-7

[4]

David M. Beskow, Sumeet Kumar, and Kathleen M. Carley. 2020. The evolution of political memes: Detecting and characterizing internet memes with multi-modal deep learning. Information Processing & Management 57, 2 (March 2020), 102170. https://doi.org/10.1016/j.ipm.2019.102170 Number: 2.

Digital Library

[5]

Tanmoy Chakraborty and Sarah Masud. 2022. Nipping in the bud: detection, diffusion and mitigation of hate speech on social media. ACM SIGWEB NewsletterWinter (2022), 1–9.

Digital Library

[6]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46.

[7]

Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021. Detecting Propaganda Techniques in Memes. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 6603–6617. https://doi.org/10.18653/v1/2021.acl-long.516

[8]

Yuhao Du, Muhammad Aamir Masood, and Kenneth Joseph. 2020. Understanding Visual Memes: An Empirical Analysis of Text Superimposed on Memes Shared on Twitter. Proceedings of the International AAAI Conference on Web and Social Media 14, 1 (May 2020), 153–164. https://doi.org/10.1609/icwsm.v14i1.7287

[9]

Abhimanyu Dubey, Esteban Moro, Manuel Cebrian, and Iyad Rahwan. 2018. MemeSequencer: Sparse Matching for Embedding Image Macros. In Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW ’18. ACM Press, Lyon, France, 1225–1235. https://doi.org/10.1145/3178876.3186021

Digital Library

[10]

Marta Dynel. 2016. “I has seen image macros!” Advice animals memes as visual-verbal jokes. International Journal of Communication 10 (2016), 29.

[11]

Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Flickr8k Dataset. Journal of Artificial Intelligence Research 47 (2013), 853–899.

Digital Library

[12]

Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. 2017. Automatic understanding of image and video advertisements. In Proceedings of the IEEE conference on computer vision and pattern recognition (Honolulu, HI, USA). IEEE, 1705–1715. https://doi.org/10.1109/CVPR.2017.123

[13]

JaidedOCR. 2022. EasyOCR. Available at: https://www.jaided.ai/easyocr/ (2022-05-10).

[14]

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. In Advances in Neural Information Processing Systems NIPS’20 (Vancouver, BC, Canada), Vol. 33. Curran Associates, Inc., Red Hook, NY, USA, Article 220, 2611–2624 pages.

[15]

Hannah Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff-Sotta, Aleksandar Shtedritski, and Yuki M Asano. 2021. Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021). Association for Computational Linguistics, Online, 26–35. https://doi.org/10.18653/v1/2021.woah-1.4

[16]

Michele Knobel and Colin Lankshear. 2007. Online memes, affinities, and cultural production. A new literacies sampler 29 (2007), 199–227. Publisher: New York.

[17]

Christos Koutlis, Manos Schinas, and Symeon Papadopoulos. 2023. MemeTector: Enforcing deep focus for meme detection. International Journal of Multimedia Information Retrieval (Jan. 2023). https://doi.org/10.5281/zenodo.7554267

[18]

Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 497–506.

Digital Library

[19]

Martina Miliani, Giulia Giorgi, Ilir Rama, Guido Anselmi, and Gianluca E. Lebani. 2020. DANKMEMES @ EVALITA 2020: The Memeing of Life: Memes, Multimodality and Politics. In EVALITA. http://ceur-ws.org/Vol-2765/paper174.pdf

[20]

Ankit Kumar Mishra and Sunil Saumya. 2021. IIIT_DWD@EACL2021: Identifying Troll Meme in Tamil using a hybrid deep learning approach. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics, Kyiv, 243–248. https://aclanthology.org/2021.dravidianlangtech-1.33

[21]

Lawankorn Mookdarsanit and Pakpoom Mookdarsanit. 2021. Combating the hate speech in Thai textual memes. Indonesian Journal of Electrical Engineering and Computer Science 21, 3 (March 2021), 1493–1502. https://doi.org/10.11591/ijeecs.v21.i3.pp1493-1502 Number: 3.

[22]

Fausto Morales. 2019. Keras-OCR. Available at: https://keras-ocr.readthedocs.io/ (2022-02-05).

[23]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543.

[24]

Jesus Perez-Martin, Benjamin Bustos, and Magdalena Saldana. 2020. Semantic Search of Memes on Twitter. (Feb. 2020). https://doi.org/10.48550/arXiv.2002.01462 arXiv:https://arxiv.org/abs/2002.01462v4

[25]

Shraman Pramanick, Dimitar Dimitrov, Rituparna Mukherjee, Shivam Sharma, Md. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021. Detecting Harmful Memes and Their Targets. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online, 2783–2796. https://doi.org/10.18653/v1/2021.findings-acl.246

[26]

Joshua Roesslein. 2020. Tweepy: Twitter for Python!URL: https://github.com/tweepy/tweepy (2020).

[27]

Giorgio Roffo and Alessandro Vinciarelli. 2016. Personality in computational advertising: A benchmark. In 4th Workshop on Emotions and Personality in Peronalized Systems. 18.

[28]

Richard Rogers and Giulia Giorgi. 2023. What is a meme, technically speaking?Information, Communication & Society 0, 0 (2023), 1–19. https://doi.org/10.1080/1369118X.2023.2174790 arXiv:https://doi.org/10.1080/1369118X.2023.2174790

[29]

Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas PYKL, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck. 2020. SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!. In Proceedings of the Fourteenth Workshop on Semantic Evaluation. International Committee for Computational Linguistics, Barcelona (online), 759–773. https://doi.org/10.18653/v1/2020.semeval-1.99

[30]

Chhavi Sharma and Viswanath Pulabaigari. 2020. A Curious Case of Meme Detection: An Investigative Study. In Proceedings of the 16th International Conference on Web Information Systems and Technologies. SCITEPRESS - Science and Technology Publications, Budapest, Hungary, 327–338. https://doi.org/10.5220/0010110203270338

[31]

Chhavi Sharma, Viswanath Pulabaigari, and Amitava Das. 2020. Meme vs. Non-meme Classification using Visuo-linguistic Association:. In Proceedings of the 16th International Conference on Web Information Systems and Technologies. SCITEPRESS - Science and Technology Publications, Budapest, Hungary, 353–360. https://doi.org/10.5220/0010176303530360

[32]

Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dimitar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and Understanding Harmful Memes: A Survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, 5597–5606. https://doi.org/10.24963/ijcai.2022/781 Survey Track.

[33]

Limor Shifman. 2012. An anatomy of a YouTube meme. New media & society 14, 2 (2012), 187–203. Publisher: Sage Publications Sage UK: London, England.

[34]

Limor Shifman. 2013. Memes in a digital world: Reconciling with a conceptual troublemaker. Journal of computer-mediated communication 18, 3 (2013), 362–377. Publisher: Oxford University Press Oxford, UK.

[35]

Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR abs/1409.1556 (2014). http://arxiv.org/abs/1409.1556

[36]

Ray Smith. 2007. An overview of the Tesseract OCR engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), Vol. 2. IEEE, 629–633.

Digital Library

[37]

Shardul Suryawanshi and Bharathi Raja Chakravarthi. 2021. Findings of the Shared Task on Troll Meme Classification in Tamil. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics, Kyiv, 126–132. https://aclanthology.org/2021.dravidianlangtech-1.16

[38]

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67–78.

Cited By

Verma BMeel PVishwakarma D(2024)Visual Sentiment Recognition via Popular Deep Models on the Memotion Dataset2024 IEEE 9th International Conference for Convergence in Technology (I2CT)10.1109/I2CT61223.2024.10543569(1-7)Online publication date: 5-Apr-2024
https://doi.org/10.1109/I2CT61223.2024.10543569

Index Terms

Multi-channel Convolutional Neural Network for Precise Meme Classification
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object recognition
    2. Natural language processing
  2. Machine learning
    1. Learning paradigms
      1. Multi-task learning
        Transfer learning
2. Information systems
  1. World Wide Web
    1. Web applications
      1. Social networks

Recommendations

Multi-class pattern classification using neural networks

Multi-class pattern classification has many applications including text document classification, speech recognition, object recognition, etc. Multi-class pattern classification using neural networks is not a trivial extension from two-class neural ...
Hashtag-Guided Low-Resource Tweet Classification
WWW '23: Proceedings of the ACM Web Conference 2023

Social media classification tasks (e.g., tweet sentiment analysis, tweet stance detection) are challenging because social media posts are typically short, informal, and ambiguous. Thus, training on tweets is challenging and demands large-scale human-...
Class-switching neural network ensembles

This article investigates the properties of class-switching ensembles composed of neural networks and compares them to class-switching ensembles of decision trees and to standard ensemble learning methods, such as bagging and boosting. In a class-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval

June 2023

694 pages

ISBN:9798400701788

DOI:10.1145/3591106

Editors:
Ioannis (Yiannis) Kompatsiaris
Centre for Research and Technology Hellas, Greece
,
Jiebo Luo
University of Rochester,USA
,
Nicu Sebe
University of Trento, Italy
,
Angela Yao
National University of Singapore, Singapore
,
Vasileios Mezaris
Centre for Research and Technology Hellas, Greece
,
Symeon Papadopoulos
Centre for Research and Technology Hellas, Greece
,
Adrian Popescu
CEA LIST, France
,
Zi (Helen) Huang
University of Queensland, Australia

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Data Availability

Details of dataset compilation, Twitter URLs for live evaluation and parameters for generating training data for meme specific optical character recognition model. https://dl.acm.org/doi/10.1145/3591106.3592275#Supplemental Materials.zip

Conference

ICMR '23

Sponsor:

SIGMM

ICMR '23: International Conference on Multimedia Retrieval

June 12 - 15, 2023

Thessaloniki, Greece

Acceptance Rates

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
675
Total Downloads

Downloads (Last 12 months)475
Downloads (Last 6 weeks)62

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Verma BMeel PVishwakarma D(2024)Visual Sentiment Recognition via Popular Deep Models on the Memotion Dataset2024 IEEE 9th International Conference for Convergence in Technology (I2CT)10.1109/I2CT61223.2024.10543569(1-7)Online publication date: 5-Apr-2024
https://doi.org/10.1109/I2CT61223.2024.10543569

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents