research-article

Arabic Span Extraction-based Reading Comprehension Benchmark (ASER) and Neural Baseline Models

Authors:

Mariam M. Biltawi,

Sara TedmoriAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 22, Issue 5

Article No.: 127, Pages 1 - 29

https://doi.org/10.1145/3579047

Published: 08 May 2023 Publication History

Abstract

Machine reading comprehension (MRC) requires machines to read and answer questions about a given text. This can be achieved through either predicting answers or extracting them. Extracting answers from text involves predicting the first and last index of the answer span within the paragraph. Training machines to answer questions requires datasets that are created for such a purpose. The lack of availability of benchmarking datasets for the Arabic language has hindered research into machine reading comprehension from Arabic text. The aim of this article is to propose an Arabic Span-Extraction-based Reading Comprehension Benchmark (ASER) and complement it with neural baseline models for performance evaluations. Detailed steps are depicted for building and evaluating ASER, which is an Arabic dataset created manually for the task of machine reading comprehension. It contains 10,000 records from different domains and is divided into training and testing sets. The results of ASER evaluation led to the conclusion that it is a challenging benchmark since the answers have varying lengths and human performance resulted in an exact match of 42%. On the other hand, two main baseline models were the focus of ASER experimentation: the sequence-to-sequence (Seq2Seq) model with different neural networks and the bidirectional attention flow (BIDAF) model. These experiments were implemented using different embeddings, and the results showed an exact match with lower values than human performance.

References

[1]

Laith Mohammad Qasim Abualigah. 2019. Feature Selection and Enhanced Krill Herd Algorithm for Text Document Clustering. Springer.

[2]

Laith Mohammad Abualigah, Ahamad Tajudin Khader, and Essam Said Hanandeh. 2018. A combination of objective functions and hybrid krill herd algorithm for text document clustering analysis. Engineering Applications of Artificial Intelligence 73 (2018), 111–125.

[3]

Eman Albilali, Nora Altwairesh, and Manar Hosny. 2021. What does BERT learn from Arabic machine reading comprehension datasets? In Proceedings of the 6th Arabic Natural Language Processing Workshop.

[4]

Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the LREC 2020 Workshop Language Resources and Evaluation Conference.

[5]

Mariam M. Biltawi, Sara Tedmori, and Arafat Awajan. 2021. Arabic question answering systems: Gap analysis. IEEE Access 9 (2021), 63876–63904.

[6]

Mariam Biltawi, Arafat Awajan, and Sara Tedmori. 2017. Towards building a frame-based ontology for the arabic language. In Proceedings of the International Arab Conference on Information Technology.

[7]

Mariam Biltawi, Arafat Awajan, and Sara Tedmori. 2020. Towards building an open-domain corpus for arabic reading comprehension. In Proceedings of the 35th IBlMA Conference.

[8]

Danqi Chen. 2018. Neural Reading Comprehension and Beyond. Stanford University.

Digital Library

[9]

Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2358–2367.

[10]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN Encoder–Decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.

[11]

Yimin Cui, Wanxiang Che, Ziqing Yang, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2022. Interactive gated decoder for machine reading comprehension. Transactions on Asian and Low-resource Language Information Processing 21, 4 (2022), 1–19.

Digital Library

[12]

Daria Dzendzik, Carl Vogel, and Jennifer Foster. 2021. English machine reading comprehension datasets: A survey. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 8784–8804.

[13]

Ahmad Eid, Nagwa M. El-Makky, and Khaled Nagi. 2019. Towards machine comprehension of arabic text. In Proceedings of the KDIR.

[14]

Aysu Ezen-Can. 2020. A comparison of LSTM and BERT for small corpus. arXiv:2009.05451. Retrieved from https://arxiv.org/abs/2009.05451

[15]

Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the 11th International Conference on Language Resources and Evaluation.

[16]

Dana Halabi, Ebaa Fayyoumi, and Arafat Awajan. 2021. I3rab: A new arabic dependency treebank based on arabic grammatical theory. Transactions on Asian and Low-resource Language Information Processing 21, 2 (2021), 1–32.

[17]

Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in Neural Information Processing Systems 28 (2015).

[18]

Sepp Hochreiter, and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9, 8 (1997), 1735–1780.

Digital Library

[19]

Xiaolin Hu and P. Balasubramaniam. 2008. (Eds.). Recurrent Neural Networks, Vol. 400. I-Tech Education and Publishing KG.

[20]

Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022. Biomedical question answering: A survey of approaches and challenges. ACM Computing Surveys 55, 2 (2022), 1–36.

[21]

Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. In Proceedings of the International Conference on Learning Representations.

[22]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26 (2013).

[23]

Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. Neural arabic question answering. In Proceedings of the 4rth Arabic Natural Language Processing Workshop. 108–118.

[24]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoCo@ NIPS 2640 (2016), 660.

[25]

Jianquan Ouyang and Mengen Fu. 2022. Improving machine reading comprehension with multi-task learning and self-training. Mathematics 10, 3, (2022), 310.

[26]

Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. 2017. MEMEN: Multi-layer embedding with memory networks for machine comprehension. In Proceedings of the AAAI Conference on Artificial Intelligence, the 30th innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence.

[27]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2383–2392.

[28]

Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bi-directional attention flow for machine comprehension. International Conference on Learning Representations.

[29]

Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. AraVec: A set of arabic word embedding models for use in arabic NLP. Procedia Computer Science 117 (2017), 256–265.

[30]

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems 27 (2014).

[31]

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP. 191–200.

[32]

Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019. Lost in translation: Loss and decay of linguistic richness in machine translation. In Proceedings of the Machine Translation Summit XVII. European Association for Machine Translation.

[33]

Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In Proceedings of the ICLR 2017: International Conference on Learning Representations. Toulon. 1–15.

[34]

W. Wang. 2017. R-Net: machine reading comprehension with self-matching networks. Natural Language Computer Group, Microsoft Reserach.

[35]

Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.

[36]

Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-Perspective context matching for machine comprehension. arXiv:1612.04211. Retrieved from https://arxiv.org/abs/1612.04211

[37]

Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning. 271–280.

[38]

Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text. 94–106.

[39]

Pengtao Xie and Eric Xing. 2017. A constituent-centric neural architecture for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.

Cited By

Biltawi MAwajan ATedmori S(2024)Improved bidirectional attention flow (BIDAF) model for Arabic machine reading comprehensionNatural Language Processing10.1017/nlp.2024.46(1-29)Online publication date: 31-Oct-2024
https://doi.org/10.1017/nlp.2024.46
Obeidat RAl-Harbi MAl-Ayyoub MAlawneh L(2024)ArQuAD: An Expert-Annotated Arabic Machine Reading Comprehension DatasetCognitive Computation10.1007/s12559-024-10248-616:3(984-1003)Online publication date: 11-Mar-2024
https://doi.org/10.1007/s12559-024-10248-6
Makahleh MObiedat RQuwaider M(2023)Leveraging Pre-trained Language Models for Arabic Machine Reading Comprehension with Unanswerable questions2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS)10.1109/SNAMS60348.2023.10375456(1-7)Online publication date: 21-Nov-2023
https://doi.org/10.1109/SNAMS60348.2023.10375456

Index Terms

Arabic Span Extraction-based Reading Comprehension Benchmark (ASER) and Neural Baseline Models
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Language resources

Recommendations

Nateq Reading Arabic Text for Visually Impaired People
Universal Access in Human-Computer Interaction. Methods, Technologies, and Users
Abstract
Nateq is a system developed to aid visually impaired people in their daily life tasks. Nateq allows blind users to read text written on papers and labels using their mobile phones. It uses two sources to read text from, either from camera or photo ...
Arabic Reading Machine for Visually Impaired People Using TTS and OCR
ISMS '13: Proceedings of the 2013 4th International Conference on Intelligent Systems, Modelling and Simulation

This paper suggests a standalone Arabic Reading Machine using TTS (Text-to-speech) and OCR (Optical Character Recognition) software built in a user friendly way for Visually Impaired People. In the Arab world, the assistive reading technology for ...
Neural Reading Comprehension and Beyond

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 22, Issue 5

May 2023

653 pages

ISSN:2375-4699

EISSN:2375-4702

DOI:10.1145/3596451

Editor:
Imed Zitouni
Google, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2023

Online AM: 10 January 2023

Accepted: 19 December 2022

Revised: 13 November 2022

Received: 14 July 2021

Published in TALLIP Volume 22, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
158
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)8

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Biltawi MAwajan ATedmori S(2024)Improved bidirectional attention flow (BIDAF) model for Arabic machine reading comprehensionNatural Language Processing10.1017/nlp.2024.46(1-29)Online publication date: 31-Oct-2024
https://doi.org/10.1017/nlp.2024.46
Obeidat RAl-Harbi MAl-Ayyoub MAlawneh L(2024)ArQuAD: An Expert-Annotated Arabic Machine Reading Comprehension DatasetCognitive Computation10.1007/s12559-024-10248-616:3(984-1003)Online publication date: 11-Mar-2024
https://doi.org/10.1007/s12559-024-10248-6
Makahleh MObiedat RQuwaider M(2023)Leveraging Pre-trained Language Models for Arabic Machine Reading Comprehension with Unanswerable questions2023 Tenth International Conference on Social Networks Analysis, Management and Security (SNAMS)10.1109/SNAMS60348.2023.10375456(1-7)Online publication date: 21-Nov-2023
https://doi.org/10.1109/SNAMS60348.2023.10375456

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents