research-article

Multi-level Connection Enhanced Representation Learning for Script Event Prediction

Authors:

Shenghai Zhong,

Chen LiAuthors Info & Claims

WWW '21: Proceedings of the Web Conference 2021

Pages 3524 - 3533

https://doi.org/10.1145/3442381.3449894

Published: 03 June 2021 Publication History

Abstract

Script event prediction (SEP) aims to choose a correct subsequent event from a candidate list, given a chain of ordered context events. Event representation learning has been proposed and successfully applied to this task. Most previous methods learning representations mainly focus on coarse-grained connections at event or chain level, while ignoring more fine-grained connections between events. Here we propose a novel framework which can enhance the representation learning of events by mining their connections at multiple granularity levels, including argument level, event level and chain level. In our method, we first employ a masked self-attention mechanism to model the relations between the components of events (i.e. arguments). Then, a directed graph convolutional network is further utilized to model the temporal or causal relations between events in the chain. Finally, we introduce an attention module to the context event chain, so as to dynamically aggregate context events with respect to the current candidate event. By fusing threefold connections in a unified framework, our approach can learn more accurate argument/event/chain representations, and thus leads to better prediction performance. Comprehensive experiment results on public New York Times corpus demonstrate that our model outperforms other state-of-the-art baselines. Our code is available in https://github.com/YueAWu/MCer.

References

[1]

Niranjan Balasubramanian, Stephen Soderland, and Oren Etzioni. 2013. Generating coherent event schemas at scale. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013. 1721–1731.

[2]

Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised Learning of Narrative Event Chains. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics, ACL 2008. 789–797.

[3]

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014. 1724–1734.

[4]

Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555(2014).

[5]

James R Curran, Stephen Clark, and Johan Bos. 2007. Linguistically Motivated Large-Scale NLP with C&C and Boxer. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. 33–36.

[6]

Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan. 2019. Event Representation Learning Enhanced with External Commonsense Knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. 4893–4902.

[7]

David Graff and Christopher Cieri. 2003. English gigaword. In Linguistic Data Consortium, Philadelphia 4(1). 34.

[8]

Mark Granroth-Wilding and Stephen Christopher Clark. 2016. What happens next? event prediction using a compositional neural network model. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016. 2727–2733.

Digital Library

[9]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735–1780.

Digital Library

[10]

Linmei Hu, Juanzi Li, Liqiang Nie, Xiaoli Li, and Chao Shao. 2017. What Happens Next? Future Subevent Prediction Using Contextual Hierarchical LSTM. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 2018. 3450–3456.

Digital Library

[11]

Bram Jans, Steven Bethard, Ivan Vulić, and Marie Francine Moens. 2012. Skip n-grams and ranking functions for predicting script events. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2012. 336–344.

[12]

Thomas N Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017.

[13]

I-Ta Lee and Dan Goldwasser. 2019. Multi-Relational Script Learning for Discourse Relations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019. 4214–4226.

[14]

Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Constructing Narrative Event Evolutionary Graph for Script Event Prediction. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. 4201–4207.

[15]

Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A Structured Self-attentive Sentence Embedding. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017.

[16]

Shangwen Lv, Wanhui Qian, Longtao Huang, Jizhong Han, and Songlin Hu. 2019. SAM-Net: Integrating Event-Level and Chain-Level Attentions to Predict What Happens Next. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAI, 2019. 6802–6809.

Digital Library

[17]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, NIPS 2013. 3111–3119.

[18]

Ashutosh Modi. 2016. Event embeddings for semantic script modeling. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016. 75–83.

[19]

Bryan Perozzi, Rami Alrfou, and Steven Skiena. 2014. Deepwalk: online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2014. 701–710.

Digital Library

[20]

Karl Pichotta and Raymond J Mooney. 2014. Statistical script learning with multi-argument events. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014. 220–229.

[21]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners, 2019.

[22]

Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015. 1681–1686.

[23]

Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015. 379–389.

[24]

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-To-End Memory Networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, NIPS 2015. 2440–2448.

[25]

Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmspropz: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.

[26]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N.Gomez, and Lukasz Kaiser. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, NIPS 2017. 5998–6008.

[27]

Zhongqing Wang, Yue Zhang, and Ching Yun Chang. 2017. Integrating Order Information and Event Relation for Script Event Prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. 57–67.

[28]

Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory Networks. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015.

Cited By

Li QLi JSheng JCui SWu JHei YPeng HGuo SWang LBeheshti AYu P(2024)A Survey on Deep Learning Event Extraction: Approaches and ApplicationsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321316835:5(6301-6321)Online publication date: May-2024
https://doi.org/10.1109/TNNLS.2022.3213168
Zhou PWu BWang CHe L(2024)An improved hierarchical neural network model with local and global feature matching for script event predictionExpert Systems with Applications10.1016/j.eswa.2024.125325(125325)Online publication date: Sep-2024
https://doi.org/10.1016/j.eswa.2024.125325
Zhang YTang X(2023)News event prediction by trigger evolution graph and event segmentJournal of Systems Engineering and Electronics10.23919/JSEE.2023.00008334:3(615-626)Online publication date: Jun-2023
https://doi.org/10.23919/JSEE.2023.000083
Show More Cited By

Recommendations

Script event prediction based on pre-trained model with tail event enhancement
CSAI '21: Proceedings of the 2021 5th International Conference on Computer Science and Artificial Intelligence

Script event prediction is a big challenge and its goal is to predict the subsequent event based on the observed events. Since an event is described by text, the pre-trained models have been applied for event representation. However, the embedding based ...
Script Event Prediction via Multilingual Event Graph Networks
Predicting what happens next in text plays a critical role in building NLP applications. Many methods including count-based and neural-network-based have been proposed to tackle the task called script event prediction: predicting the most suitable ...
Integrating rich event-level and schema-level information for script event prediction

A script consists of a series of structured event sequences extracted from the texts. Given historical scripts, script event prediction aims to predict the subsequent event. The critical aspect in script event prediction is how to effectively represent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '21: Proceedings of the Web Conference 2021

April 2021

4054 pages

ISBN:9781450383127

DOI:10.1145/3442381

Editors:
Jure Leskovec
Stanford
,
Marko Grobelnik
Jožef Stefan Institute
,
Marc Najork
Google
,
Jie Tang
Tsinghua University
,
Leila Zia
Wikimedia Foundation

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '21

Sponsor:

SIGWEB

WWW '21: The Web Conference 2021

April 19 - 23, 2021

Ljubljana, Slovenia

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
322
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)3

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li QLi JSheng JCui SWu JHei YPeng HGuo SWang LBeheshti AYu P(2024)A Survey on Deep Learning Event Extraction: Approaches and ApplicationsIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.321316835:5(6301-6321)Online publication date: May-2024
https://doi.org/10.1109/TNNLS.2022.3213168
Zhou PWu BWang CHe L(2024)An improved hierarchical neural network model with local and global feature matching for script event predictionExpert Systems with Applications10.1016/j.eswa.2024.125325(125325)Online publication date: Sep-2024
https://doi.org/10.1016/j.eswa.2024.125325
Zhang YTang X(2023)News event prediction by trigger evolution graph and event segmentJournal of Systems Engineering and Electronics10.23919/JSEE.2023.00008334:3(615-626)Online publication date: Jun-2023
https://doi.org/10.23919/JSEE.2023.000083
Yang SZha DXue C(2023)MSK-Net: Multi-source Knowledge Base Enhanced Networks for Script Event PredictionNeural Information Processing10.1007/978-981-99-1648-1_6(64-76)Online publication date: 15-Apr-2023
https://doi.org/10.1007/978-981-99-1648-1_6
Liu YDing KGuo FLiu MLiu LWang BSun Y(2023)Improving Event Representation for Script Event Prediction via Data Augmentation and IntegrationNatural Language Processing and Chinese Computing10.1007/978-3-031-44696-2_52(666-677)Online publication date: 8-Oct-2023
https://doi.org/10.1007/978-3-031-44696-2_52
Zhou PWu BWang CPeng HYue JXiao S(2022)What happens next? Combining enhanced multilevel script learning and dual fusion strategies for script event predictionInternational Journal of Intelligent Systems10.1002/int.2302537:11(10001-10040)Online publication date: 26-Sep-2022
https://dl.acm.org/doi/10.1002/int.23025

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten