research-article

Evaluating Textual Entailment Recognition for University Entrance Examinations

Authors:

Hiroshi Kanayama,

Teruko MitamuraAuthors Info & Claims

ACM Transactions on Asian Language Information Processing (TALIP), Volume 11, Issue 4

Article No.: 13, Pages 1 - 16

https://doi.org/10.1145/2382593.2382595

Published: 01 December 2012 Publication History

Abstract

The present article addresses an attempt to apply questions in university entrance examinations to the evaluation of textual entailment recognition. Questions in several fields, such as history and politics, primarily test the examinee’s knowledge in the form of choosing true statements from multiple choices. Answering such questions can be regarded as equivalent to finding evidential texts from a textbase such as textbooks and Wikipedia. Therefore, this task can be recast as recognizing textual entailment between a description in a textbase and a statement given in a question. We focused on the National Center Test for University Admission in Japan and converted questions into the evaluation data for textual entailment recognition by using Wikipedia as a textbase. Consequently, it is revealed that nearly half of the questions can be mapped into textual entailment recognition; 941 text pairs were created from 404 questions from six subjects. This data set is provided for a subtask of NTCIR RITE (Recognizing Inference in Text), and 16 systems from six teams used the data set for evaluation. The evaluation results revealed that the best system achieved a correct answer ratio of 56%, which is significantly better than a random choice baseline.

References

[1]

Androutsopoulos, I. and Malakasiotis, P. 2010. A survey of paraphrasing and textual entailment methods. J. Artif. Intell. Res. 38, 135--187.

[2]

Angele, J., Moench, E., Oppermann, H., Staab, S., and Wenke, D. 2003. Ontology-based query and answering in chemistry: OntoNova Project Halo. In Proceedings of the 2nd International Semantic Web Conference (ISWC’03).

[3]

Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., and Magnini, B. 2006. The second PASCAL recognizing textual entailment challenge. In Proceedings of the 2nd PASCAL Challenges Workshop on Recognizing Textual Entailment (RTE-2’06).

[4]

Bentivogli, L., Dagan, I., Dang, H. T., Giampiccolo, D., and Magnini, B. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC’09).

[5]

Bentivogli, L., Clark, P., Dagan, I., Dang, H., and Giampiccolo, D. 2010. The sixth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC’10).

[6]

Bos, J., Zanzotto, F. M., and Pennacchiotti, M. 2009. Textual entailment at EVALITA 2009. In Proceedings of the Conference on Evaluation of NLP and Speech Tools for Italian (EVALITA’09).

[7]

Dagan, I., Glickman, O., and Magnini, B. 2006. The PASCAL recognizing textual entailment challenge. In Machine Learning Challenges. Lecture Notes in Computer Science, vol. 3944, Springer-Verlag, 177--190.

Digital Library

[8]

Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing (ACL-PASCAL’07).

Digital Library

[9]

Giampiccolo, D., Dang, H. T., Magnini, B., Dagan, I., and Dolan, B. 2009. The fourth PASCAL recognizing textual entailment challenge. In Proceedings of the Text Analysis Conference (TAC’08).

[10]

Ishikawa, D., Joho, H., Kando, N., Kato, T., Sakai, T., Sugimoto, M., Sumita, E., Akiba, T., Geva, S., Gey, F., Goto, I., Lu, B., Shima, H., Tang, E., and Trotman, A., Eds. 2011. Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[11]

Ji, H. and Grishman, R. 2011. Knowledge base population: Successful approaches and challenges. In Proceedings of the Association of Computational Linguistics (ACL’11). 1148--1158.

Digital Library

[12]

Kasahara, K., Taira, H., and Nagata, M. 2010. On the possibility of applying textual entailment recognition to multi-sentence reading comprehension tasks. In Proceedings of the 16th Annual Conference on Natural Language Processing in Japan (JapTAL’10). 780--783. (In Japanese).

[13]

Mehdad, Y., Negri, M., and Federico, M. 2010. Towards cross-lingual textual entailment. In Proceedings of Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics (HLT-NAACL’10).

Digital Library

[14]

Mehdad, Y., Negri, M., and Federico, M. 2011. Using bilingual parallel corpora for cross-lingual textual entailment. In Proceedings of the Association of Computational Linguistics (ACL’11).

Digital Library

[15]

Odani, M., Shibata, T., Kurohashi, S., and Nakata, T. 2008. Building data of Japanese text entailment and recognition of inferencing relations based on automatic achieved similar expressions. In Proceedings of the 14th Annual Conference on Natural Language Processing in Japan (JapTAL’08). 1140--1143. (In Japanese.)

[16]

Pakray, P., Neogi, S., Bandyopadhyay, S., and Gelbukh, A. 2011. A textual entailment system using web based machine translation system. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[17]

Peñas, A., Hovy, E., Forner, P., Rodrigo, A., Sutcliffe, R., Forascu, C., and Sporleder, C. 2011. Overview of QA4MRE at CLEF 2011: Question answering for machine reading evaluation. In Proceedings of the Workshop on Cross-Language Evaluation Forum (CLEF’11). 19--22.

[18]

Pham, Q. N. M., Nguyen, L. M., and Shimazu, A. 2011. A machine learning based textual entailment recognition system of JAIST team for the NTCIR9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[19]

Shibata, T. and Kurohashi, S. 2011. Predicate-argument structure based textual entailment recognition system of KYOTO team for the NTCIR9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[20]

Shima, H., Kanayama, H., Lee, C.-W., Lin, C.-J., Mitamura, T., Miyao, Y., Shi, S., and Takeda, K. 2011a. Overview of the NTCIR-9 RITE: Recognizing inference in text. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11). 291--301.

[21]

Shima, H., Li, Y., Orii, N., and Mitamura, T. 2011b. LTI’s textual entailment recognizer system at NTCIR-9 RITE. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[22]

Tsuboi, Y., Kanayama, H., Ohno, M., and Unno, Y. 2011. Syntactic difference based approach for the NTCIR-9 RITE task. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

[23]

Watanabe, Y., Mizuno, J., Nichols, E., Narisawa, K., Nabeshima, K., and Inui, K. 2011. TU group at NTCIR9-RITE: Leveraging diverse lexical resources for recognizing textual entailment. In Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11).

Cited By

Cheng GZhu WWang ZChen JQu Y(2016)Taking up the gaokao challengeProceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence10.5555/3060832.3060968(2479-2485)Online publication date: 9-Jul-2016
https://dl.acm.org/doi/10.5555/3060832.3060968
Arai N(2015)The impact of AI—can a robot get into the University of Tokyo?National Science Review10.1093/nsr/nwv0112:2(135-136)Online publication date: 30-Apr-2015
https://doi.org/10.1093/nsr/nwv011
KOTLERMAN LDAGAN IMAGNINI BBENTIVOGLI L(2015)Textual entailment graphsNatural Language Engineering10.1017/S135132491500010821:5(699-724)Online publication date: 23-Jun-2015
https://doi.org/10.1017/S1351324915000108
Show More Cited By

Index Terms

Evaluating Textual Entailment Recognition for University Entrance Examinations
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing

Recommendations

Leveraging Diverse Lexical Resources for Textual Entailment Recognition
Special Issue on RITE

Since the problem of textual entailment recognition requires capturing semantic relations between diverse expressions of language, linguistic and world knowledge play an important role. In this article, we explore the effectiveness of different types of ...
Recognizing Textual Entailment Based on WordNet
IITA '08: Proceedings of the 2008 Second International Symposium on Intelligent Information Technology Application - Volume 02

Textual Entailment Recognition (RTE) is one of the fundamental problems in many natural language processing applications. This paper proposes a new method for lexical entailment measure which is based on exploiting the information in the WordNet ...
Chinese Textual Entailment Recognition Enhanced with Word Embedding
Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data
Abstract
Textual entailment has been proposed as a unifying generic framework for modeling language variability and semantic inference in different Natural Language Processing (NLP) tasks. By evaluating on NTCIR-11 RITE3 Simplified Chinese subtask data set,...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian Language Information Processing

ACM Transactions on Asian Language Information Processing Volume 11, Issue 4

Special Issue on RITE

December 2012

130 pages

ISSN:1530-0226

EISSN:1558-3430

DOI:10.1145/2382593

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2012

Accepted: 01 August 2012

Revised: 01 July 2012

Received: 01 May 2012

Published in TALIP Volume 11, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
292
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cheng GZhu WWang ZChen JQu Y(2016)Taking up the gaokao challengeProceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence10.5555/3060832.3060968(2479-2485)Online publication date: 9-Jul-2016
https://dl.acm.org/doi/10.5555/3060832.3060968
Arai N(2015)The impact of AI—can a robot get into the University of Tokyo?National Science Review10.1093/nsr/nwv0112:2(135-136)Online publication date: 30-Apr-2015
https://doi.org/10.1093/nsr/nwv011
KOTLERMAN LDAGAN IMAGNINI BBENTIVOGLI L(2015)Textual entailment graphsNatural Language Engineering10.1017/S135132491500010821:5(699-724)Online publication date: 23-Jun-2015
https://doi.org/10.1017/S1351324915000108
Kawazoe AMiyao YMatsuzaki TYokono HArai N(2013)World History Ontology for Reasoning Truth/Falsehood of Sentences: Event Classification to Fill in the Gaps Between Knowledge Resources and Natural Language TextsNew Frontiers in Artificial Intelligence10.1007/978-3-319-10061-6_3(42-50)Online publication date: 27-Oct-2013
https://dl.acm.org/doi/10.1007/978-3-319-10061-6_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents