research-article

Predicting web searcher satisfaction with existing community-based answers

Authors:

Eugene Agichtein,

Evgeniy Gabrilovich,

Idan SzpektorAuthors Info & Claims

SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Pages 415 - 424

https://doi.org/10.1145/2009916.2009974

Published: 24 July 2011 Publication History

Abstract

Community-based Question Answering (CQA) sites, such as Yahoo! Answers, Baidu Knows, Naver, and Quora, have been rapidly growing in popularity. The resulting archives of posted answers to questions, in Yahoo! Answers alone, already exceed in size 1 billion, and are aggressively indexed by web search engines. In fact, a large number of search engine users benefit from these archives, by finding existing answers that address their own queries. This scenario poses new challenges and opportunities for both search engines and CQA sites. To this end, we formulate a new problem of predicting the satisfaction of web searchers with CQA answers. We analyze a large number of web searches that result in a visit to a popular CQA site, and identify unique characteristics of searcher satisfaction in this setting, namely, the effects of query clarity, query-to-question match, and answer quality. We then propose and evaluate several approaches to predicting searcher satisfaction that exploit these characteristics. To the best of our knowledge, this is the first attempt to predict and validate the usefulness of CQA archives for external searchers, rather than for the original askers. Our results suggest promising directions for improving and exploiting community question answering services in pursuit of satisfying even more Web search queries.

References

[1]

E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne. Finding high-quality content in social media. In Proc. of WSDM, pages 183--194, 2008.

Digital Library

[2]

S. Amer-Yahia and M. Lalmas. XML search: languages, INEX and scoring. SIGMOD Rec., 35:16--23, December 2006.

Digital Library

[3]

M. Bendersky, E. Gabrilovich, V. Josifovski, and D. Metzler. The anatomy of an ad: Structured indexing and retrieval for sponsored search. In WWW'10, April 2010.

Digital Library

[4]

J. Bian, Y. Liu, E. Agichtein, and H. Zha. Finding the right facts in the crowd: factoid question answering over social media. In WWW'08, 2008.

Digital Library

[5]

X. Cao, G. Cong, B. Cui, C. S. Jensen, and C. Zhang. The use of categorization information in language models for question retrieval. In CIKM, 2009.

Digital Library

[6]

S. Cronen-Townsend, Y. Zhou, and W. B. Croft. Predicting query performance. In SIGIR, 2002.

Digital Library

[7]

D. Donato, F. Bonchi, T. Chi, and Y. Maarek. Do you want to take notes?: identifying research missions in Yahoo! Search Pad. In WWW'10, 2010.

Digital Library

[8]

H. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In Proc. of SIGIR, pages 34--41, 2010.

Digital Library

[9]

J. L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378 -- 382, 1971.

[10]

S. Goel, A. Broder, E. Gabrilovich, and B. Pang. Anatomy of the long tail: Ordinary people with extraordinary tastes. In WSDM, 2010.

Digital Library

[11]

F. Harper, D. Moy, and J. Konstan. Facts or friends?: distinguishing informational and conversational questions in social Q &A sites. In CHI, 2009.

Digital Library

[12]

F. M. Harper, D. Raban, S. Rafaeli, and J. A. Konstan. Predictors of answer quality in online Q &A sites. In CHI, pages 865--874, 2008.

Digital Library

[13]

A. Hassan, R. Jones, and K. Klinkner. Beyond DCG: User behavior as a predictor of a successful search. In Proc. of WSDM, pages 221--230, 2010.

Digital Library

[14]

D. Horowitz and S. Kamvar. The anatomy of a large-scale social search engine. In WWW, 2010.

Digital Library

[15]

S. B. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In SIGIR, pages 567--574, 2007.

Digital Library

[16]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20:422--446, October 2002.

Digital Library

[17]

J. Jeon, W. B. Croft, J. H. Lee, and S. Park. A framework to predict the quality of answers with non-textual features. In SIGIR, 2006.

Digital Library

[18]

G. Kazai and A. Doucet. Overview of the INEX 2007 book search track: Booksearch'07. SIGIR Forum, 42(1):2--15, 2008.

Digital Library

[19]

Y. Liu, J. Bian, and E. Agichtein. Predicting information seeker satisfaction in community question answering. In SIGIR, 2008.

Digital Library

[20]

B. Long, O. Chapelle, Y. Zhang, Y. Chang, Z. Zheng, and B. L. Tseng. Active learning for ranking through expected loss optimization. In SIGIR, pages 267--274, 2010.

Digital Library

[21]

M. Morris, J. Teevan, and K. Panovich. A Comparison of Information Seeking Using Search Engines and Social Networks. In ICWSM, 2010.

[22]

J. Nielsen. User interface directions for the web. Commun. ACM, 42:65--72, January 1999.

Digital Library

[23]

D. Raban. Self-presentation and the value of information in Q &A web sites. JASIST, 60(12):2465--2473, 2009.

Digital Library

[24]

H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400--407, 1951.

[25]

S. Robertson, H. Zaragoza, and M. Taylor. Simple BM25 extension to multiple weighted fields. In Proceedings of CIKM, pages 42--49, 2004.

Digital Library

[26]

D. E. Rose and D. Levinson. Understanding user goals in web search. In WWW, pages 13--19, 2004.

Digital Library

[27]

M. Sanderson and I. Soboroff. Problems with kendall's tau. In SIGIR, 2007.

Digital Library

[28]

C. Shah and J. Pomerantz. Evaluating and predicting answer quality in community QA. In SIGIR, 2010.

Digital Library

[29]

Y.-I. Song, C.-Y. Lin, Y. Cao, and H.-C. Rim. Question utility: A novel static ranking of question search. In AAAI, pages 1231--1236, 2008.

Digital Library

[30]

J. C. Spall. Introduction to Stochastic Search and Optimization. John Wiley & Sons, 2003.

Digital Library

[31]

K. Sun, Y. Cao, X. Song, Y.-I. Song, X. Wang, and C.-Y. Lin. Learning to recommend questions based on user ratings. In CIKM, pages 751--758, 2009.

Digital Library

[32]

M. Surdeanu, M. Ciaramita, and H. Zaragoza. Learning to rank answers on large online qa collections. In ACL, pages 719--727, 2008.

[33]

M. A. Suryanto, E. P. Lim, A. Sun, and R. H. L. Chiang. Quality-aware collaborative question answering: methods and evaluation. In WSDM, 2009.

Digital Library

[34]

J. Teevan, S. T. Dumais, and D. J. Liebling. To personalize or not to personalize: modeling queries with variation in user intent. In SIGIR, 2008.

Digital Library

[35]

A. Tsotsis. Just because google exists doesn't mean you should stop asking people things. TechCrunch, Oct 2010. http://techcrunch.com/2010/10/23/google-vs-humans/.

[36]

X. Wang, X. Tu, D. Feng, and L. Zhang. Ranking community answers by modeling question-answer relationships via analogical reasoning. In SIGIR, 2009.

Digital Library

[37]

Y. Wang and E. Agichtein. Query ambiguity revisited: clickthrough measures for distinguishing informational and ambiguous queries. In ACL, 2010.

Digital Library

[38]

F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1:80--83, 1945.

[39]

X. Xue, J. Jeon, and W. Croft. Retrieval models for question and answer archives. In SIGIR, 2008.

Digital Library

[40]

E. Yom-Tov, S. Fine, D. Carmel, and A. Darlow. Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval. In SIGIR, 2005.

Digital Library

Cited By

Banjar AShaheen AAmjad TAlharbey RDaud A(2024)Users’ satisfaction based ranking for Yahoo AnswersMultimedia Tools and Applications10.1007/s11042-024-18433-383:28(71265-71284)Online publication date: 7-Feb-2024
https://doi.org/10.1007/s11042-024-18433-3
Keyvan KHuang J(2022)How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and ChallengesACM Computing Surveys10.1145/353496555:6(1-40)Online publication date: 7-Dec-2022
https://dl.acm.org/doi/10.1145/3534965
Tavakoli LZamani HScholer FCroft WSanderson M(2022)Analyzing clarification in asynchronous information‐seeking conversationsJournal of the Association for Information Science and Technology10.1002/asi.2456273:3(449-471)Online publication date: 7-Feb-2022
https://dl.acm.org/doi/10.1002/asi.24562
Show More Cited By

Index Terms

Predicting web searcher satisfaction with existing community-based answers
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing

Recommendations

Evaluating and predicting answer quality in community QA
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Question answering (QA) helps one go beyond traditional keywords-based querying and retrieve information in more precise form than given by a document or a list of documents. Several community-based QA (CQA) services have emerged allowing information ...
Question Retrieval with High Quality Answers in Community Question Answering
CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

This paper studies the problem of question retrieval in community question answering (CQA). To bridge lexical gaps in questions, which is regarded as the biggest challenge in retrieval, state-of-the-art methods learn translation models using answers ...
Predicting information seeker satisfaction in community question answering
SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

Question answering communities such as Naver and Yahoo! Answers have emerged as popular, and often effective, means of information seeking on the web. By posting questions for other participants to answer, information seekers can obtain specific answers ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

July 2011

1374 pages

ISBN:9781450307574

DOI:10.1145/2009916

General Chairs:
Wei-Ying Ma
Microsoft Research Asia, China
,
Jian-Yun Nie
University of Montreal, Canada
,
Program Chairs:
Ricardo Baeza-Yates
Yahoo! Research, Spain
,
Tat-Seng Chua
National University of Singapore
,
W. Bruce Croft
University of Massachusetts, Amherst, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '11

Sponsor:

SIGIR

SIGIR '11: The 34th International ACM SIGIR conference on research and development in Information Retrieval

July 24 - 28, 2011

Beijing, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
604
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Banjar AShaheen AAmjad TAlharbey RDaud A(2024)Users’ satisfaction based ranking for Yahoo AnswersMultimedia Tools and Applications10.1007/s11042-024-18433-383:28(71265-71284)Online publication date: 7-Feb-2024
https://doi.org/10.1007/s11042-024-18433-3
Keyvan KHuang J(2022)How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and ChallengesACM Computing Surveys10.1145/353496555:6(1-40)Online publication date: 7-Dec-2022
https://dl.acm.org/doi/10.1145/3534965
Tavakoli LZamani HScholer FCroft WSanderson M(2022)Analyzing clarification in asynchronous information‐seeking conversationsJournal of the Association for Information Science and Technology10.1002/asi.2456273:3(449-471)Online publication date: 7-Feb-2022
https://dl.acm.org/doi/10.1002/asi.24562
Moutidis IWilliams H(2021)Community evolution on Stack OverflowPLOS ONE10.1371/journal.pone.025301016:6(e0253010)Online publication date: 17-Jun-2021
https://doi.org/10.1371/journal.pone.0253010
Tavakoli Ld'Aquin MDietze SHauff CCurry ECudre Mauroux P(2020)Generating Clarifying Questions in Conversational Search SystemsProceedings of the 29th ACM International Conference on Information & Knowledge Management10.1145/3340531.3418513(3253-3256)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3340531.3418513
He XWang LZhang WZhang P(2019)Research on the Quality Prediction of Online Chinese Question Answering Community Answers Based on CommentsProceedings of the 2nd International Conference on Big Data Technologies10.1145/3358528.3358592(114-120)Online publication date: 28-Aug-2019
https://dl.acm.org/doi/10.1145/3358528.3358592
Wei XHuang HNie LFeng FHong RChua T(2018)Quality mattersProceedings of the 27th International Joint Conference on Artificial Intelligence10.5555/3304222.3304393(4482-4488)Online publication date: 13-Jul-2018
https://dl.acm.org/doi/10.5555/3304222.3304393
Yulianti EChen RScholer FCroft WSanderson MCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Ranking Documents by Answer-Passage QualityThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210028(335-344)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210028
Yulianti EChen RScholer FCroft WSanderson M(2018)Document Summarization for Answering Non-Factoid QueriesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2017.275437330:1(15-28)Online publication date: 1-Jan-2018
https://doi.org/10.1109/TKDE.2017.2754373
Yao YTong HXu FLu J(2017)On the Measurement and Prediction of Web Content UtilityACM SIGKDD Explorations Newsletter10.1145/3166054.316605619:2(1-12)Online publication date: 21-Nov-2017
https://dl.acm.org/doi/10.1145/3166054.3166056
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents