Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Multimodal Learning for Automatic Summarization: A Survey

  • Conference paper
  • First Online:
Advanced Data Mining and Applications (ADMA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14177))

Included in the following conference series:

  • 752 Accesses

Abstract

With the widespread availability of multiple data sources, such as image, audio-video, and text data, automatic summarization of multimodal data is becoming an important technology in decision support. This paper presents a comprehensive survey and summary of the main articles in the field of multimodal summarization techniques in recent years. Firstly, we define multimodal summarization and briefly describe the development process. Then, we survey existing techniques and their applicability in different domains. Additionally, we provide an analysis of their results and discuss the insights of those approaches, along with the challenges and future research directions. Based on our study, we found that the encoder-decoder approach is currently the best approach for automated summarization. In the future, we believe that the applications of multimodal summarization could develop rapidly in many different fields, particularly in medicine. In our case studies, we demonstrate that multimodal learning is a promising research direction for providing timely and accurate summarizations compared to unimodal approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Allawadi, S., Rana, V., Jain, M., et al.: Multimedia data summarization using joint integer linear programming. In: 2021 5th International Conference on Computing Methodologies and Communication (ICCMC), pp. 1462–1466. IEEE (2021)

    Google Scholar 

  2. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  3. Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)

    Article  Google Scholar 

  4. Bateman, J.A.: Multimodality and Genre. Palgrave Macmillan UK, London (2008). https://doi.org/10.1057/9780230582323

    Book  Google Scholar 

  5. Bian, J., Yang, Y., Chua, T.S.: Multimedia summarization for trending topics in microblogs. In: Proceedings of the 22nd ACM International Conference on Information and Knowledge Management, pp. 1807–1812 (2013)

    Google Scholar 

  6. Bian, J., Yang, Y., Zhang, H., Chua, T.S.: Multimedia summarization for social events in microblog stream. IEEE Trans. Multimedia 17(2), 216–228 (2014)

    Article  Google Scholar 

  7. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003)

    MATH  Google Scholar 

  8. Boudin, F., Mougard, H., Favre, B.: Concept-based summarization using integer linear programming: from concept pruning to multiple optimal solutions. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) 2015 (2015)

    Google Scholar 

  9. Chali, Y., Tanvee, M., Nayeem, M.T.: Towards abstractive multi-document summarization using submodular function-based framework, sentence compression and merging. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 418–424 (2017)

    Google Scholar 

  10. Chen, J., Zhuge, H.: Abstractive text-image summarization using multi-modal attentional hierarchical RNN. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 4046–4056 (2018)

    Google Scholar 

  11. Chen, J., Zhuge, H.: Extractive text-image summarization using multi-modal RNN. In: 2018 14th International Conference on Semantics, Knowledge and Grids (SKG), pp. 245–248. IEEE (2018)

    Google Scholar 

  12. Chen, J., Zhuge, H.: News image captioning based on text summarization using image as query. In: 2019 15th International Conference on Semantics, Knowledge and Grids (SKG), pp. 123–126. IEEE (2019)

    Google Scholar 

  13. Erol, B., Lee, D.S., Hull, J.: Multimodal summarization of meeting recordings. In: 2003 International Conference on Multimedia and Expo. ICME 2003. Proceedings (Cat. No. 03TH8698), vol. 3, pp. III-25. IEEE (2003)

    Google Scholar 

  14. Evangelopoulos, G., et al.: Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Trans. Multimedia 15(7), 1553–1568 (2013)

    Article  Google Scholar 

  15. Evangelopoulos, G., et al.: Video event detection and summarization using audio, visual and text saliency. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3553–3556. IEEE (2009)

    Google Scholar 

  16. Fan, F., et al.: A semantic-based medical image fusion approach. arXiv preprint arXiv:1906.00225 (2019)

  17. Gao, X., Jiang, X., Wu, Q., Wang, X., Lyu, C., Lyu, L.: Gt-SimNet: improving code automatic summarization via multi-modal similarity networks. J. Syst. Softw. 194, 111495 (2022)

    Article  Google Scholar 

  18. Gao, Y., Lyu, C.: M2TS: multi-scale multi-modal approach based on transformer for source code summarization. In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp. 24–35 (2022)

    Google Scholar 

  19. Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: a search space odyssey. IEEE Trans. Neural Networks Learn. Syst. 28(10), 2222–2232 (2016)

    Article  MathSciNet  Google Scholar 

  20. Hahn, U., Mani, I.: The challenges of automatic summarization. Computer 33(11), 29–36 (2000)

    Article  Google Scholar 

  21. Jangra, A., Jatowt, A., Hasanuzzaman, M., Saha, S.: Text-image-video summary generation using joint integer linear programming. In: Jose, J.M., et al. (eds.) ECIR 2020. LNCS, vol. 12036, pp. 190–198. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_24

    Chapter  Google Scholar 

  22. Khullar, A., Arora, U.: Mast: multimodal abstractive summarization with trimodal hierarchical attention. arXiv preprint arXiv:2010.08021 (2020)

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  24. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  25. Li, H., Yuan, P., Xu, S., Wu, Y., He, X., Zhou, B.: Aspect-aware multimodal summarization for Chinese e-commerce products. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8188–8195 (2020)

    Google Scholar 

  26. Li, H., Zhu, J., Liu, T., Zhang, J., Zong, C., et al.: Multi-modal sentence summarization with modality attention and image filtering. In: IJCAI, pp. 4152–4158 (2018)

    Google Scholar 

  27. Li, H., Zhu, J., Ma, C., Zhang, J., Zong, C.: Multi-modal summarization for asynchronous collection of text, image, audio and video. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1092–1102 (2017)

    Google Scholar 

  28. Li, Z., Tang, J., Wang, X., Liu, J., Lu, H.: Multimedia news summarization in search. ACM Trans. Intell. Syst. Technol. 7(3), 1–20 (2016)

    Google Scholar 

  29. Libovickỳ, J., Palaskar, S., Gella, S., Metze, F.: Multimodal abstractive summarization for open-domain videos. In: Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NIPS (2018)

    Google Scholar 

  30. Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)

    Google Scholar 

  31. Lin, H., Bilmes, J.: Multi-document summarization via budgeted maximization of submodular functions. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 912–920 (2010)

    Google Scholar 

  32. Liu, Y., Chen, X., Cheng, J., Peng, H.: A medical image fusion method based on convolutional neural networks. In: 2017 20th International Conference on Information Fusion (Fusion), pp. 1–7. IEEE (2017)

    Google Scholar 

  33. Ma, Z., Gao, Y., Lyu, L., Lyu, C.: MMF3: neural code summarization based on multi-modal fine-grained feature fusion. In: Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 171–182 (2022)

    Google Scholar 

  34. Mani, I.: Automatic Summarization, vol. 3. John Benjamins Publishing (2001)

    Google Scholar 

  35. McCowan, I., et al.: Modeling human interaction in meetings. In: 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP 2003), vol. 4, pp. IV-748. IEEE (2003)

    Google Scholar 

  36. McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264(5588), 746–748 (1976)

    Article  Google Scholar 

  37. Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404–411 (2004)

    Google Scholar 

  38. Modani, N., et al.: Summarizing multimedia content. In: Cellary, W., Mokbel, M., Wang, J., Wang, H., Zhou, R., Zhang, Y. (eds.) Web Information Systems Engineering-WISE 2016: 17th International Conference, Shanghai, China, November 8–10, 2016, Proceedings, LNCS, Part II 17, pp. 340–348. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48743-4_27

  39. Nallapati, R., Zhai, F., Zhou, B.: SummaRuNNer: a recurrent neural network based sequence model for extractive summarization of documents. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  40. O’Halloran, K.L.: Interdependence, interaction and metaphor in multisemiotic texts. Soc. Semiot. 9(3), 317–354 (1999)

    Article  Google Scholar 

  41. Palaskar, S., Libovický, J., Gella, S., Metze, F.: Multimodal abstractive summarization for how2 videos. arXiv preprint arXiv:1906.07901 (2019)

  42. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)

    Google Scholar 

  43. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)

  44. Sanabria, M., Sherly, Precioso, F., Menguy, T.: A deep architecture for multimodal summarization of soccer games. In: Proceedings Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, pp. 16–24 (2019)

    Google Scholar 

  45. Schinas, M., Papadopoulos, S., Kompatsiaris, Y., Mitkas, P.A.: MGraph: multimodal event summarization in social media using topic models and graph-based ranking. Int. J. Multimedia Inf. Retrieval 5, 51–69 (2016)

    Article  Google Scholar 

  46. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  47. Song, X., Jing, L., Lin, D., Zhao, Z., Chen, H., Nie, L.: V2P: vision-to-prompt based multi-modal product summary generation. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 992–1001 (2022)

    Google Scholar 

  48. Sun, Y., Lei, L., Tan, X., Guan, D., Wu, J., Kuang, G.: Structured graph based image regression for unsupervised multimodal change detection. ISPRS J. Photogramm. Remote. Sens. 185, 16–31 (2022)

    Article  Google Scholar 

  49. Tiwari, A., Weth, C.V.D., Kankanhalli, M.S.: Multimodal multiplatform social media event summarization. ACM Trans. Multimedia Comput. Commun. Appl. 14(2s), 1–23 (2018)

    Article  Google Scholar 

  50. Tjondronegoro, D., Tao, X., Sasongko, J., Lau, C.H.: Multi-modal summarization of key events and top players in sports tournament videos. In: 2011 IEEE Workshop on Applications of Computer Vision (WACV), pp. 471–478. IEEE (2011)

    Google Scholar 

  51. Torres, C., Rose, K., Fried, J.C., Manjunath, B.: Summarization of ICU patient motion from multimodal multiview videos. arXiv preprint arXiv:1706.09430 (2017)

  52. Tsai, Y.H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.P., Salakhutdinov, R.: Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the Conference. Association for Computational Linguistics. Meeting, vol. 2019, p. 6558. NIH Public Access (2019)

    Google Scholar 

  53. Wadagave, P., Garg, B.: A heterogeneous data summarization system to generate automatic summary of data using twitter API using tweets, images etc. Harbin Gongye Daxue Xuebao J. Harbin Inst. Technol. 54(6), 167–172 (2022)

    Google Scholar 

  54. Yuhas, B.P., Goldstein, M.H., Sejnowski, T.J.: Integration of acoustic and visual speech signals using neural networks. IEEE Commun. Mag. 27(11), 65–71 (1989)

    Article  Google Scholar 

  55. Zhu, J., Li, H., Liu, T., Zhou, Y., Zhang, J., Zong, C.: MSMO: multimodal summarization with multimodal output. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 4154–4164 (2018)

    Google Scholar 

  56. Zhu, J., Xiang, L., Zhou, Y., Zhang, J., Zong, C.: Graph-based multimodal ranking models for multimodal summarization. Trans. Asian Low Resour. Lang. Inf. Process. 20(4), 1–21 (2021)

    Article  Google Scholar 

  57. Zhu, J., Zhou, Y., Zhang, J., Li, H., Zong, C., Li, C.: Multimodal summarization with guidance of multimodal reference. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 9749–9756 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Queensland, Brisbane, QLD, 4072, Australia

    Zhicheng Zhang, Yibo Sun & Shiyan Su

Authors
  1. Zhicheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yibo Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiyan Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhicheng Zhang .

Editor information

Editors and Affiliations

  1. Northeastern University, Shenyang, China

    Xiaochun Yang

  2. The University of Indonesia, Depok, Indonesia

    Heru Suhartanto

  3. Beijing Institute of Technology, Beijing, China

    Guoren Wang

  4. Northeastern University, Shenyang, China

    Bin Wang

  5. University of Technology Sydney, Sydney, NSW, Australia

    Jing Jiang

  6. Agency for Science, Technology and Research (A*STAR), Singapore, Singapore

    Bing Li

  7. Sun Yat-sen University, Guangzhou, China

    Huaijie Zhu

  8. Anhui University, Hefei, China

    Ningning Cui

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, Z., Sun, Y., Su, S. (2023). Multimodal Learning for Automatic Summarization: A Survey. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14177. Springer, Cham. https://doi.org/10.1007/978-3-031-46664-9_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-46664-9_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-46663-2

  • Online ISBN: 978-3-031-46664-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation