short-paper

Multi-Modal Summary Generation using Multi-Objective Optimization

Authors:

Anubhav Jangra,

Mohammad HasanuzzamanAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1745 - 1748

https://doi.org/10.1145/3397271.3401232

Published: 25 July 2020 Publication History

Abstract

Significant development of communication technology over the past few years has motivated research in multi-modal summarization techniques. A majority of the previous works on multi-modal summarization focus on text and images. In this paper, we propose a novel extractive multi-objective optimization based model to produce a multi-modal summary containing text, images, and videos. Important objectives such as intra-modality salience, cross-modal redundancy and cross-modal similarity are optimized simultaneously in a multi-objective optimization framework to produce effective multi-modal output. The proposed model has been evaluated separately for different modalities, and has been found to perform better than state-of-the-art approaches.

Supplementary Material

MP4 File (3397271.3401232.mp4)

In this video we describe a novel multi-modal multi-objective optimization based summarization framework, that uses differential evolution as its underlying strategy. We also discuss the benefits of multi-modal summarization, discuss some empirical observations, and present a generated summary.

Download
67.06 MB

References

[1]

Rasim Alguliev, Ramiz Aliguliyev, and Makrufa Hajirahimova. 2010. Multi-document summarization model based on integer linear programming. Intelligent Control and Automation, Vol. 1, 02 (2010), 105.

[2]

David Arthur and Sergei Vassilvitskii. 2006. k-means+: The advantages of careful seeding. Technical Report. Stanford.

[3]

Swagatam Das, Ajith Abraham, and Amit Konar. 2007. Automatic clustering using an improved differential evolution algorithm. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, Vol. 38, 1 (2007), 218--237.

Digital Library

[4]

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, Vol. 6, 2 (2002), 182--197.

[5]

Kalyanmoy Deb and Santosh Tiwari. 2008. Omni-optimizer: A generic evolutionary algorithm for single and multi-objective optimization. European Journal of Operational Research, Vol. 185, 3 (2008), 1062--1087.

[6]

Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Jour. of artif. intel. res., Vol. 22 (2004), 457--479.

[7]

Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. Artificial Intelligence Review, Vol. 47, 1 (2017), 1--66.

Digital Library

[8]

Anubhav Jangra, Adam Jatowt, Mohammad Hasanuzzaman, and Sriparna Saha. 2020. Text-Image-Video Summary Generation Using Joint Integer Linear Programming. In European Conference on Information Retrieval. Springer, 190--198.

[9]

Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. 2014. Fisher vectors derived from hybrid gaussian-laplacian mixture models for image annotation. arXiv preprint arXiv:1411.7399 (2014).

[10]

Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2017. Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 1092--1102.

[11]

Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. 404--411.

[12]

Malay K Pakhira, Sanghamitra Bandyopadhyay, and Ujjwal Maulik. 2004. Validity index for crisp and fuzzy clusters. Pattern recognition, Vol. 37, 3 (2004), 487--501.

[13]

Naveen Saini, Sriparna Saha, Anubhav Jangra, and Pushpak Bhattacharyya. 2019. Extractive single document summarization using multi-objective optimization: Exploring self-organized differential evolution, grey wolf optimizer and water cycle algorithm. Knowledge-Based Systems, Vol. 164 (2019), 45--67.

[14]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In International Conference on Learning Representations .

[15]

Naushad UzZaman, Jeffrey P Bigham, and James F Allen. 2011. Multimodal summarization of complex sentences. In Proceedings of the 16th international conference on Intelligent user interfaces. ACM, 43--52.

Digital Library

[16]

Liwei Wang, Yin Li, and Svetlana Lazebnik. 2016. Learning deep structure-preserving image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5005--5013.

[17]

Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2017. Recent advances in document summarization. Knowledge and Information Systems, Vol. 53, 2 (2017), 297--336.

Digital Library

[18]

Yong Zhang, Meng Joo Er, Rui Zhao, and Mahardhika Pratama. 2016. Multiview convolutional neural networks for multidocument extractive summarization. IEEE transactions on cybernetics, Vol. 47, 10 (2016), 3230--3242.

[19]

Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. MSMO: Multimodal Summarization with Multimodal Output. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4154--4164.

Cited By

Kumar RPrakash DSaha SSharma S(2024)IndicBART Alongside Visual Element: Multimodal Summarization in Diverse Indian LanguagesDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70552-6_16(264-280)Online publication date: 11-Sep-2024
https://doi.org/10.1007/978-3-031-70552-6_16
Jangra AMukherjee SJatowt ASaha SHasanuzzaman M(2023)A Survey on Multi-modal SummarizationACM Computing Surveys10.1145/358470055:13s(1-36)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3584700
Roha VSaini NSaha SMoreno J(2023)MOO-CMDS+NER: Named Entity Recognition-Based Extractive Comment-Oriented Multi-document SummarizationAdvances in Information Retrieval10.1007/978-3-031-28238-6_49(580-588)Online publication date: 17-Mar-2023
https://doi.org/10.1007/978-3-031-28238-6_49
Show More Cited By

Index Terms

Recommendations

Multi-Modal Supplementary-Complementary Summarization using Multi-Objective Optimization
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Large amounts of multi-modal information online make it difficult for users to obtain proper insights. In this paper, we introduce and formally define the concepts of supplementary and complementary multi-modal summaries in the context of the overlap of ...
Multi-objective optimization using teaching-learning-based optimization algorithm

Two major goals in multi-objective optimization are to obtain a set of nondominated solutions as closely as possible to the true Pareto front (PF) and maintain a well-distributed solution set along the Pareto front. In this paper, we propose a teaching-...
Multi-objective optimization using BFO algorithm
ICIC'11: Proceedings of the 7th international conference on Intelligent Computing: bio-inspired computing and applications

This paper describes a novel bacterial foraging optimization (BFO) approach to multi-objective optimization, called Multi-objective Bacterial Foraging Optimization (MBFO). The search for Pareto optimal set of multi-objective optimization problems is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

© 2020 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Visvesvaraya PhD scheme for Electronics and IT, Ministry ofElectronics and Information Technology (MeitY), Government of India

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
309
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)8

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kumar RPrakash DSaha SSharma S(2024)IndicBART Alongside Visual Element: Multimodal Summarization in Diverse Indian LanguagesDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70552-6_16(264-280)Online publication date: 11-Sep-2024
https://doi.org/10.1007/978-3-031-70552-6_16
Jangra AMukherjee SJatowt ASaha SHasanuzzaman M(2023)A Survey on Multi-modal SummarizationACM Computing Surveys10.1145/358470055:13s(1-36)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3584700
Roha VSaini NSaha SMoreno J(2023)MOO-CMDS+NER: Named Entity Recognition-Based Extractive Comment-Oriented Multi-document SummarizationAdvances in Information Retrieval10.1007/978-3-031-28238-6_49(580-588)Online publication date: 17-Mar-2023
https://doi.org/10.1007/978-3-031-28238-6_49
Ma CZhang WGuo MWang HSheng Q(2022)Multi-document Summarization via Deep Learning Techniques: A SurveyACM Computing Surveys10.1145/352975455:5(1-37)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3529754
Roha VSaini NSaha SMoreno JWagner M(2022)Unsupervised framework for comment-based multi-document extractive summarizationProceedings of the Genetic and Evolutionary Computation Conference10.1145/3512290.3528691(574-582)Online publication date: 8-Jul-2022
https://dl.acm.org/doi/10.1145/3512290.3528691
Lu QZhu CYe X(2022)Research on Multimodal Summarization by Integrating Visual and Text Modal Information2022 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA)10.1109/AEECA55500.2022.9919012(882-889)Online publication date: 20-Aug-2022
https://doi.org/10.1109/AEECA55500.2022.9919012
Jain RMavi VJangra ASaha S(2022)WIDAR - Weighted Input Document Augmented ROUGEAdvances in Information Retrieval10.1007/978-3-030-99736-6_21(304-321)Online publication date: 5-Apr-2022
https://doi.org/10.1007/978-3-030-99736-6_21
Jangra ASaha SJatowt AHasanuzzaman MDiaz FShah CSuel TCastells PJones RSakai T(2021)Multi-Modal Supplementary-Complementary Summarization using Multi-Objective OptimizationProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462877(818-828)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3462877

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents