Article

Dynamic pooling and unfolding recursive autoencoders for paraphrase detection

Authors:

Richard Socher,

Jeffrey Pennington,

Christopher D. ManningAuthors Info & Claims

NIPS'11: Proceedings of the 24th International Conference on Neural Information Processing Systems

Pages 801 - 809

Published: 12 December 2011 Publication History

Abstract

Paraphrase detection is the task of examining two sentences and determining whether they have the same meaning. In order to obtain high accuracy on this task, thorough syntactic and semantic analysis of the two statements is needed. We introduce a method for paraphrase detection based on recursive autoencoders (RAE). Our unsupervised RAEs are based on a novel unfolding objective and learn feature vectors for phrases in syntactic trees. These features are used to measure the word- and phrase-wise similarity between two sentences. Since sentences may be of arbitrary length, the resulting matrix of similarity measures is of variable size. We introduce a novel dynamic pooling layer which computes a fixed-sized representation from the variable-sized matrices. The pooled representation is then used as input to a classifier. Our method outperforms other state-of-the-art approaches on the challenging MSRP paraphrase corpus.

References

[1]

E. Marsi and E. Krahmer. Explorations in sentence fusion. In European Workshop on Natural Language Generation, 2005.

[2]

P. Clough, R. Gaizauskas, S. S. L. Piao, and Y. Wilks. METER: MEasuring TExt Reuse. In ACL, 2002.

[3]

C. Callison-Burch. Syntactic constraints on paraphrases extracted from parallel corpora. In Proceedings of EMNLP, pages 196-205, 2008.

[4]

B. Dolan, C. Quirk, and C. Brockett. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. In COLING, 2004.

[5]

Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3, March 2003.

[6]

R. Collobert and J. Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, 2008.

[7]

Y. Bengio, J. Louradour, Collobert R, and J. Weston. Curriculum learning. In ICML, 2009.

[8]

J. Turian, L. Ratinov, and Y. Bengio. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384-394, 2010.

[9]

J. B. Pollack. Recursive distributed representations. Artificial Intelligence, 46, November 1990.

[10]

T. Voegtlin and P. Dominey. Linear Recursive Distributed Representations. Neural Networks, 18(7), 2005.

[11]

J. L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7(2-3), 1991.

[12]

R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In EMNLP, 2011.

[13]

C. Goller and A. Küchler. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of the International Conference on Neural Networks (ICNN-96), 1996.

[14]

D. Klein and C. D. Manning. Accurate unlexicalized parsing. In ACL, 2003.

[15]

D. Das and N. A. Smith. Paraphrase identification as probabilistic quasi-synchronous recognition. In In Proc. of ACL-IJCNLP, 2009.

[16]

V. Rus, P. M. McCarthy, M. C. Lintean, D. S. McNamara, and A. C. Graesser. Paraphrase identification with lexico-syntactic graph subsumption. In FLAIRS Conference, 2008.

[17]

R. Mihalcea, C. Corley, and C. Strapparava. Corpus-based and Knowledge-based Measures of Text Semantic Similarity. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1, 2006.

[18]

A. Islam and D. Inkpen. Semantic Similarity of Short Texts. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2007), 2007.

[19]

L. Qiu, M. Kan, and T. Chua. Paraphrase recognition via dissimilarity significance classification. In EMNLP, 2006.

[20]

S. Fernando and M. Stevenson. A semantic similarity approach to paraphrase detection. Proceedings of the 11th Annual Research Colloquium of the UK Special Interest Group for Computational Linguistics, 2008.

[21]

S. Wan, M. Dras, R. Dale, and C. Paris. Using dependency-based features to take the "para-farce" out of paraphrase. In Proceedings of the Australasian Language Technology Workshop 2006, 2006.

[22]

R. Barzilay and L. Lee. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In NAACL, 2003.

[23]

Y. Zhang and J. Patrick. Paraphrase identification by text canonicalization. In Proceedings of the Australasian Language Technology Workshop 2005, 2005.

[24]

Z. Kozareva and A. Montoyo. Paraphrase Identification on the Basis of Supervised Machine Learning Techniques. In Advances in Natural Language Processing, 5th International Conference on NLP, FinTAL, 2006.

[25]

L. Bottou. From machine learning to machine reasoning. CoRR, abs/1102.1808, 2011.

[26]

H. Larochelle, Y. Bengio, J. Louradour, and P. Lamblin. Exploring strategies for training deep neural networks. JMLR, 10, 2009.

[27]

R. Socher, C. D. Manning, and A. Y. Ng. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, 2010.

[28]

R. Socher, C. Lin, A. Y. Ng, and C.D. Manning. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML, 2011.

Cited By

Samandi VTiňo PBahsoon R(2023)Real-Time Workflow Scheduling in Cloud with Recursive Neural Network and List SchedulingHybrid Artificial Intelligent Systems10.1007/978-3-031-40725-3_21(244-255)Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-40725-3_21
Ebrahimi FTushev MMahmoud A(2021)Classifying Mobile Applications Using Word EmbeddingsACM Transactions on Software Engineering and Methodology10.1145/347482731:2(1-30)Online publication date: 17-Nov-2021
https://dl.acm.org/doi/10.1145/3474827
Shao JWang YGao HShen HLi YCheng XDemartini GZuccon GCulpepper JHuang ZTong H(2021)Locate Who You AreProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482134(3413-3417)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482134
Show More Cited By

Index Terms

Dynamic pooling and unfolding recursive autoencoders for paraphrase detection

Recommendations

Urdu Short Paraphrase Detection at Sentence Level
Paraphrase detection systems uncover the relationship between two text fragments and classify them as paraphrased when they convey the same idea; otherwise non-paraphrased. Previously, the researchers have mainly focused on developing resources for the ...
PKU Paraphrase Bank: A Sentence-Level Paraphrase Corpus for Chinese
Natural Language Processing and Chinese Computing
Abstract
One of the main challenges of conducting research on paraphrase is the lack of large-scale, high-quality corpus, which is particularly serious for non-English investigations. In this paper, we present a simple and effective unsupervised learning ...
English- Vietnamese Cross-Language Paraphrase Identification Method
SoICT '17: Proceedings of the 8th International Symposium on Information and Communication Technology

Paraphrase identification is a very important problem and is used in many natural language processing tasks such as machine translation, bilingual information retrieval, plagiarism detection, etc. With the development of information technology and the ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NIPS'11: Proceedings of the 24th International Conference on Neural Information Processing Systems

December 2011

2752 pages

ISBN:9781618395993

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 12 December 2011

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Samandi VTiňo PBahsoon R(2023)Real-Time Workflow Scheduling in Cloud with Recursive Neural Network and List SchedulingHybrid Artificial Intelligent Systems10.1007/978-3-031-40725-3_21(244-255)Online publication date: 5-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-40725-3_21
Ebrahimi FTushev MMahmoud A(2021)Classifying Mobile Applications Using Word EmbeddingsACM Transactions on Software Engineering and Methodology10.1145/347482731:2(1-30)Online publication date: 17-Nov-2021
https://dl.acm.org/doi/10.1145/3474827
Shao JWang YGao HShen HLi YCheng XDemartini GZuccon GCulpepper JHuang ZTong H(2021)Locate Who You AreProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482134(3413-3417)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482134
Zhou QHui TWang RHu HLiu S(2021)Attentive Excitation and Aggregation for Bilingual Referring Image SegmentationACM Transactions on Intelligent Systems and Technology10.1145/344634512:2(1-17)Online publication date: 26-Feb-2021
https://dl.acm.org/doi/10.1145/3446345
Quamer WJain PRai ASaravanan VPamula RKumar C(2021)SACNN: Self-attentive Convolutional Neural Network Model for Natural Language InferenceACM Transactions on Asian and Low-Resource Language Information Processing10.1145/342688420:3(1-16)Online publication date: 16-Jun-2021
https://dl.acm.org/doi/10.1145/3426884
Kumar SRoy SPathak VVarma VKambhampati SBhattacharya ANatarajan SRoy R(2020)A Hybrid Distributed Model for Learning Representation of Short Texts with Attribute LabelsProceedings of the 7th ACM IKDD CoDS and 25th COMAD10.1145/3371158.3371195(244-248)Online publication date: 5-Jan-2020
https://dl.acm.org/doi/10.1145/3371158.3371195
Azaria ANivasch KElkind EVeloso MAgmon NTaylor M(2019)The Multimodal Correction Detection ProblemProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3306127.3331918(1784-1786)Online publication date: 8-May-2019
https://dl.acm.org/doi/10.5555/3306127.3331918
Feher GSpitz AGertz MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Retrieving Multi-Entity AssociationsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331366(1169-1172)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331366
Qu CJi FQiu MYang LMin ZChen HHuang JCroft WCulpepper JMoffat ABennett PLerman K(2019)Learning to Selectively TransferProceedings of the Twelfth ACM International Conference on Web Search and Data Mining10.1145/3289600.3290978(699-707)Online publication date: 30-Jan-2019
https://dl.acm.org/doi/10.1145/3289600.3290978
Wu YWu WXu CLi ZMcIlraith SWeinberger K(2018)Knowledge enhanced hybrid neural network for text matchingProceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence10.5555/3504035.3504720(5586-5593)Online publication date: 2-Feb-2018
https://dl.acm.org/doi/10.5555/3504035.3504720
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents