article

Accuracy evaluation of methods and techniques in Web-based question answering systems: a survey

Authors:

Sri Devi Ravana,

Maizatul Akmar IsmailAuthors Info & Claims

Knowledge and Information Systems, Volume 58, Issue 3

Pages 611 - 650

https://doi.org/10.1007/s10115-018-1203-0

Published: 01 March 2019 Publication History

Abstract

Question answering (QA) systems answer the queries of users efficiently in the least amount of time. A researcher has to decide which among various methods and techniques available will be used to retrieve accurate answers when developing a QA system. This step creates an overhead before making a selection. The study highlights the methods and techniques that perform well in terms of the accuracy of answers provided. Nine Web-based question answering systems were consulted, and the applied methods and techniques evaluated on the basis of the percentage of questions correctly answered and the mean reciprocal rank evaluation measures. Results were discussed using three key stages involved in a QA system: answer extraction, scoring of answers, and answer aggregation. Results show some techniques have higher accuracy of answers than others. Not all methods in QA systems can improve the accuracy of answers individually, but the methods used in combination obtain greater effect. The results can be used to select methods and techniques optimal for producing highly accurate scores without spending time on benchmarking.

References

[1]

Hirschman L, Gaizauskas R (2001) Natural language question answering: the view from here. Nat Lang Eng 7(04):275---300

Digital Library

[2]

Gupta P, Gupta V (2012) A survey of text question answering techniques. Int J Comp Appl 53(4):1---8

[3]

McCallum A (2005) Information extraction: distilling structured data from unstructured text. Queue 3(9):48---57

Digital Library

[4]

Ng JP, Kan M-Y (2010) QANUS: an open-source question-answering platform. Unfiled technical report: (Posted to cs-IR and cs-CL). Preprint arXiv:1501.00311. Accessed 14th June 2015

[5]

Marriam-Webster Dictionary (2006) Merriam-Webster's collegiate dictionary, 11th edn. Merriam-Webster, Springfield

[6]

Kolomiyets O, Moens M-F (2011) A survey on question answering technology from an information retrieval perspective. Inf Sci 181(24):5412---5434

Digital Library

[7]

Wang M (2006) A survey of answer extraction techniques in factoid question answering. Comput Linguist 1(1)

[8]

Shim B, Ko Y, Seo J (2005) Extracting and utilizing of IS-A relation patterns for question answering systems. In: Lee G, Yamada A, Meng H, Myaeng S (eds) Information retrieval technology, vol 3689. Lecture notes in computer science. Springer, Heidelberg, pp 697---702.

Digital Library

[9]

Li X, Roth D (2002) Learning question classifiers. In: Proceedings of the 19th international conference on computational linguistics. Association for Computational Linguistics, Taipei, 1072378, pp 1---7.

Digital Library

[10]

Molino P, Lops P, Semeraro G, de Gemmis M, Basile P (2015) Playing with knowledge: a virtual player for "Who Wants to Be a Millionaire?" that leverages question answering techniques. Artif Intell 222:157---181

Digital Library

[11]

Abney S, Collins M, Singhal A (2000) Answer extraction. In: Proceedings of the sixth conference on Applied natural language processing, Seattle, Washington, 2000. Association for Computational Linguistics, 974188, pp 296---301.

Digital Library

[12]

Chen J, Diekema AR, Taffet MD, McCracken N, Ozgencil NE (2000) Question answering: CNLP at the TREC-10 question answering track

[13]

Harabagiu SM, Moldovan DI, Paşca M, Mihalcea R, Surdeanu M, Bunescu R, Gîrju CR, Rus V, Moră?rescu P (2000) Falcon: boosting knowledge for answer engines

[14]

Hovy EH, Gerber L, Hermjakob U, Junk M, Lin C-Y (2000) Question answering in Webclopedia. In: TREC, pp 53---56

[15]

Hovy EH, Hermjakob U, Lin C-Y (2001) The use of external knowledge of factoid QA. In: Proceedings of the TREC-10 conference, NIST, Gaithersburg, MD, 2001, pp 166---174

[16]

Pasca MA, Harabagiu SM (2001) High performance question/answering. In: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, New Orleans, Louisiana, USA, 2001. ACM, 384025, pp 366---374.

Digital Library

[17]

Prager J, Chu-Carroll J, Brown EW, Czuba K (2006) Question answering by predictive annotation. In: Advances in open domain question answering. Springer, Berlin, pp 307---347

[18]

Ferrucci D, Brown E, Chu-Carroll J, Fan J, Gondek D, Kalyanpur AA, Lally A, Murdock JW, Nyberg E, Prager J (2010) Building Watson: an overview of the DeepQA project. AI Mag 31(3):59---79

Digital Library

[19]

Wu M, Marian A (2011) A framework for corroborating answers from multiple web sources. Inf Syst 36(2):431---449

Digital Library

[20]

Yang H, Chua T-S (2003) QUALIFIER: question answering by lexical fabric and external resources. In: Copestake A, Hajiă? J (eds) Proceedings of the tenth conference on European chapter of the association for computational linguistics, Budapest, Hungary, 2003. Association for Computational Linguistics, 1067855, pp 363---370.

Digital Library

[21]

Wu M, Marian A (2007) Corroborating answers from multiple web sources. In: Proceedings of the 10th international workshop on the web and databases, ACM Special Interest Group on Management of Data, Beijing

[22]

Kwok C, Etzioni O, Weld DS (2001) Scaling question answering to the web. ACM Trans Inf Syst 19(3):242---262

Digital Library

[23]

Dumais S, Banko M, Brill E, Lin J, Ng A (2002) Web question answering: is more always better? In: Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval. ACM, Tampere, 564428, pp 291---298.

Digital Library

[24]

Yang H, Chua T-S (2002) The integration of lexical knowledge and external resources for question answering. In: Voorhees EM, Buckland LP (eds) Text retrieval conference. Department of Commerce, National Institute of Standards and Technology, Gaithersburg

[25]

Zhang D, Lee WS A (2003) Web-based question answering system. In: Proceedings of the Singapore-MIT alliance annual symposium. Singapore-MIT Alliance, Singapore, pp 1---5

[26]

Radev D, Fan W, Qi H, Wu H, Grewal A (2005) Probabilistic question answering on the web. J Am Soc Inform Sci Technol 56(6):571---583

Digital Library

[27]

Liu Z-J, Wang X-L, Chen Q-C, Zhang Y-Y, Xiang Y (2014) A Chinese question answering system based on web search. In: International conference on machine learning and cybernetics. IEEE Computer Society, Lanzhou, pp 816---820.

[28]

Oh H-J, Ryu P-M, Kim H (2012) Which is the best? re-ranking answers merged from multiple web sources. J Emerg Technol Web Intell 4(1):35---42

[29]

Oh H-J, Yoon Y-C, Kim HK (2013) Finding more trustworthy answers: various trustworthiness factors in question answering. J Inf Sci 39(4):509---522

Digital Library

[30]

Fader A, Zettlemoyer L, Etzioni O (2014) Open question answering over curated and extracted knowledge bases. In: Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 1156---1165

Digital Library

[31]

Mollá-Aliod D, Vicedo J-L (2010) Question answering. In: Indurkhya N, Damerau FJ (eds) Handbook of natural language processing, 2nd Edn. Chapman and Hall/CRC, pp 485---510

[32]

Webber B, Webb N (2010) Question answering. In: Clark A, Fox C, Lappin S (eds) The handbook of computational linguistics and natural language processing, vol 1. Wiley-Blackwell, West Sussex, pp 630---654

[33]

Allam AMN, Haggag MH (2012) The question answering systems: a survey. Int J Res Rev Inf Sci (IJRRIS) 2(3):211---221

[34]

Bouziane A, Bouchiha D, Doumi N, Malki M (2015) Question answering systems: survey and trends. Proc Comput Sci 73:366---375

[35]

Lopez V, Uren V, Sabou M, Motta E (2011) Is question answering fit for the semantic web? a survey. Semant Web 2(2):125---155

Digital Library

[36]

Kalaivani S, Duraiswamy K (2012) Comparison of question answering systems based on ontology and semantic web in different environment. J Comput Sci 8(9):1407---1413

[37]

Mollá D, Vicedo JL (2007) Question answering in restricted domains: an overview. Comput Linguist 33(1):41---61

Digital Library

[38]

Simmons RF (1965) Answering English questions by computer: a survey. Commun ACM 8(1):53---70

Digital Library

[39]

Green BF Jr, Wolf AK, Chomsky C, Laughery K (1961) Baseball: an automatic question-answerer. In: Papers presented at the May 9---11, 1961, western joint IRE-AIEE-ACM computer conference. ACM, pp 219---224

Digital Library

[40]

Woods WA, Kaplan RM, Nash-Webber B (1972) The lunar sciences: natural language information system: final report. Bolt Beranek and Newman, Cambridge

[41]

Black FS (1964) A deductive question answering system. Harvard University, Cambridge

[42]

Copestake A, Jones KS (1990) Natural language interfaces to databases. Knowl Eng Rev 5(04):225---249

[43]

Androutsopoulos I, Ritchie GD, Thanisch P (1995) Natural language interfaces to databases---an introduction. Nat Lang Eng 1(01):29---81

[44]

Copestake A, Jones KS (1990) Natural language interfaces to databases. Knowl Eng Rev 5(4):225---249.

[45]

Pazos RRA, González BJJ, Aguirre LMA, Martínez FJA, Fraire HHJ (2013) Natural language interfaces to databases: an analysis of the state of the art. In: Castillo O, Melin P, Kacprzyk J (eds) Recent advances on hybrid intelligent systems. Studies in Computational Intelligence, vol 451. Springer, Berlin, Heidelberg, pp 463---480

[46]

Nguyen A, Le H (2008) Natural language interface construction using semantic grammars. In: PRICAI 2008: trends in artificial intelligence, pp 728---739

Digital Library

[47]

Owda M, Bandar Z, Crockett K (2007) Conversation-based natural language interface to relational databases. In: Proceedings of the 2007 IEEE/WIC/ACM international conferences on web intelligence and intelligent agent technology-workshops. IEEE Computer Society, pp 363---367

Digital Library

[48]

Reddy V, Neumeier K, McFarlane J, Cothren J, Thompson CW (2007) Extending a natural language interface with geospatial queries. IEEE Internet Comput 11(6):82---85

Digital Library

[49]

Popescu A-M, Armanasu A, Etzioni O, Ko D, Yates A (2004) Modern natural language interfaces to databases: composing statistical parsing with semantic tractability. In: Proceedings of the 20th international conference on computational linguistics. Association for Computational Linguistics, p 141

Digital Library

[50]

Winograd T (1971) Procedures as a representation for data in a computer program for understanding natural language. DTIC document

[51]

Shortliffe E (2012) Computer-based medical consultations: MYCIN, vol 2. Elsevier, Amsterdam

[52]

Tablan V, Damljanovic D, Bontcheva K (2008) A natural language query interface to structured information. In: The semantic web: research and applications, pp 361---375

Digital Library

[53]

Damljanovic D, Agatonovic M, Cunningham H (2010) Natural language interfaces to ontologies: combining syntactic analysis and ontology-based lookup through the user interaction. In: The semantic web: research and applications, pp 106---120

Digital Library

[54]

Kaufmann E, Bernstein A (2010) Evaluating the usability of natural language query languages and interfaces to semantic web knowledge bases. Web Semant Sci Serv Agents World Wide Web 8(4):377---393

Digital Library

[55]

Brill E, Lin JJ, Banko M, Dumais ST, Ng AY (2001) Data-intensive question answering. In: Proceedings of the TREC-10 conference, NIST, Gaithersburg, MD, 2001, pp 183---189

[56]

Clarke CL, Cormack GV, Lynam TR, Li C, McLearn G (2001) Web reinforced question answering (MultiTest experiments for TREC 2001). In: TREC

[57]

Katz B (1997) Annotating the world wide web using natural language. In: Computer-assisted information searching on internet. Le Centre de Hautes Etudes Internationales D'informatique Documentaire, pp 136---155

Digital Library

[58]

Voorhees EM (1999) The TREC-8 question answering track report. In: Trec, pp 77---82

[59]

Alfonseca E, De Boni M, Jara-Valencia J-L, Manandhar S (2001) A prototype question answering system using syntactic and semantic information for answer retrieval. In: TREC

[60]

Li X (2003) Syntactic features in question answering. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 383---384

Digital Library

[61]

Cui H, Sun R, Li K, Kan M-Y, Chua T-S (2005) Question answering passage retrieval using dependency relations. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 400---407

Digital Library

[62]

Kitchenham B (2004) Procedures for performing systematic reviews. Keele University and NICTA, Keele

[63]

Moldovan D, Clark C, Harabagiu S, Maiorano S (2003) Cogex: a logic prover for question answering. In: Proceedings of the 2003 conference of the North American chapter of the association for computational linguistics on human language technology--volume 1. Association for Computational Linguistics, pp 87---93

Digital Library

[64]

Hovy E, Hermjakob U, Ravichandran D (2002) A question/answer typology with surface text patterns. In: Proceedings of the second international conference on human language technology research. Morgan Kaufmann Publishers Inc., pp 247---251

Digital Library

[65]

Wu Y, Zhang R, Hu X, Kashioka H (2007) Learning unsupervised SVM classifier for answer selection in web question answering

[66]

Brill E, Dumais S, Banko M (2002) An analysis of the AskMSR question-answering system. In: Proceedings of the ACL-02 conference on empirical methods in natural language processing--volume 10. Association for Computational Linguistics, pp 257---264

Digital Library

[67]

Somasundaran S, Wilson T, Wiebe J, Stoyanov V (2007) QA with attitude: exploiting opinion type analysis for improving question answering in on-line discussions and the news. In: ICWSM

[68]

Charniak E (2000) A maximum-entropy-inspired parser. In: Proceedings of the 1st North American chapter of the association for computational linguistics conference. Association for Computational Linguistics, pp 132---139

Digital Library

[69]

Ittycheriah A, Franz M, Zhu W-J, Ratnaparkhi A, Mammone RJ (2000) IBM's statistical question answering system. In: TREC

[70]

Yang H, Chua T-S, Wang S, Koh C-K (2003) Structured use of external knowledge for event-based open domain question answering. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 33---40

Digital Library

[71]

Manning CD, Surdeanu M, Bauer J, Finkel J, Bethard SJ, McClosky D (2014) The stanford core NLP natural language processing toolkit. In: Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations. Association for Computational Linguistics, Baltimore, June 22---27, 2014, pp 55---60

[72]

Aggarwal S, Van Oostendorp H, Reddy YR, Indurkhya B (2014) Providing web credibility assessment support. In: Proceedings of the 2014 European conference on cognitive ergonomics. ACM, p 29

Digital Library

[73]

Soubbotin MM, Soubbotin SM (2001) Patterns of potential answer expressions as clues to the right answers. In: TREC

[74]

Ravichandran D, Hovy E (2002) Learning surface text patterns for a question answering system. In: Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pp 41---47

Digital Library

[75]

Shih C-W, Day M-Y, Tsai T-H, Jiang T-J, Wu C-W, Sung C-L, Chen Y-R, Wu S-H, Hsu W-L (2005) ASQA: academia sinica question answering system for NTCIR-5 CLQA. In: NTCIR-5 workshop, Tokyo, pp 202---208

[76]

Barker K, Chaudhri VK, Chaw SY, Clark P, Fan J, Israel DJ, Mishra S, Porter BW, Romero P, Tecuci D (2004) A question-answering system for ap chemistry: assessing KR&R technologies. In: KR, pp 488---497

Digital Library

[77]

Toutanova K, Manning CD (2000) Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In: Proceedings of the 2000 joint SIGDAT conference on empirical methods in natural language processing and very large corpora: held in conjunction with the 38th annual meeting of the association for computational linguistics--volume 13. Association for Computational Linguistics, pp 63---70

Digital Library

[78]

Ittycheriah A, Franz M, Zhu W-J, Ratnaparkhi A, Mammone RJ (2001) Question answering using maximum entropy components. In: Proceedings of the second meeting of the North American chapter of the association for computational linguistics on language technologies. Association for Computational Linguistics, pp 1---7

Digital Library

Cited By

Wu YPan XLi JDou SWang X(2024)Interpretable answer retrieval based on heterogeneous network embeddingPattern Recognition Letters10.1016/j.patrec.2024.03.023182:C(9-16)Online publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.patrec.2024.03.023
Qamar FLatif SShah A(2023)Techniques, datasets, evaluation metrics and future directions of a question answering systemKnowledge and Information Systems10.1007/s10115-023-02019-w66:4(2235-2268)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1007/s10115-023-02019-w
Li ZHu HWang HCai LZhang HZhang K(2022)Why does the president tweet this? Discovering reasons and contexts for politicians’ tweets from news articlesInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10289259:3Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.ipm.2022.102892
Show More Cited By

Recommendations

A Yes/No Answer Generator Based on Sentiment-Word Scores in Biomedical Question Answering

Background and Objective: Yes/no question answering QA in open-domain is a longstanding challenge widely studied over the last decades. However, it still requires further efforts in the biomedical domain. Yes/no QA aims at answering yes/no questions, ...
Question Answering System Based on Web
ICICTA '12: Proceedings of the 2012 Fifth International Conference on Intelligent Computation Technology and Automation

This paper summarizes the classification, implementation and evaluation of question answering system (QA). QA is divided into four categories: chat robot, QA based knowledge base, QA retrieval system and QA based on free text. Web QA system is composed ...
Question answering system for semantic web: a review

Searching huge and heterogeneous contents from the web has become an increasingly challenging task with the contemporary growth of semantic web. In order to make the vision of semantic web a reality, some user-friendly interfaces are needed. Because of ...

Comments

Information & Contributors

Information

Published In

cover image Knowledge and Information Systems

Knowledge and Information Systems Volume 58, Issue 3

March 2019

240 pages

ISSN:0219-1377

Issue’s Table of Contents

Copyright © Copyright © 2019 Springer-Verlag London Ltd., part of Springer Nature.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 March 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu YPan XLi JDou SWang X(2024)Interpretable answer retrieval based on heterogeneous network embeddingPattern Recognition Letters10.1016/j.patrec.2024.03.023182:C(9-16)Online publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.patrec.2024.03.023
Qamar FLatif SShah A(2023)Techniques, datasets, evaluation metrics and future directions of a question answering systemKnowledge and Information Systems10.1007/s10115-023-02019-w66:4(2235-2268)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1007/s10115-023-02019-w
Li ZHu HWang HCai LZhang HZhang K(2022)Why does the president tweet this? Discovering reasons and contexts for politicians’ tweets from news articlesInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10289259:3Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.ipm.2022.102892
Cortes EWoloszyn VBarone DMöller SVieira R(2022)A systematic review of question answering systems for non-factoid questionsJournal of Intelligent Information Systems10.1007/s10844-021-00655-858:3(453-480)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1007/s10844-021-00655-8
Chen XCheng JLiu JXu WHua STang ZSheng V(2022)A Survey of Multi-label Text Classification Based on Deep LearningArtificial Intelligence and Security10.1007/978-3-031-06794-5_36(443-456)Online publication date: 15-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-06794-5_36

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents