research-article

A Study of Snippet Length and Informativeness: Behaviour, Performance and User Experience

Authors:

Leif Azzopardi,

Yashar MoshfeghiAuthors Info & Claims

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 135 - 144

https://doi.org/10.1145/3077136.3080824

Published: 07 August 2017 Publication History

Abstract

The design and presentation of a Search Engine Results Page (SERP) has been subject to much research. With many contemporary aspects of the SERP now under scrutiny, work still remains in investigating more traditional SERP components, such as the result summary. Prior studies have examined a variety of different aspects of result summaries, but in this paper we investigate the influence of result summary length on search behaviour, performance and user experience. To this end, we designed and conducted a within-subjects experiment using the TREC AQUAINT news collection with 53 participants. Using Kullback-Leibler distance as a measure of information gain, we examined result summaries of different lengths and selected four conditions where the change in information gain was the greatest: (i) title only; (ii) title plus one snippet; (iii) title plus two snippets; and (iv) title plus four snippets. Findings show that participants broadly preferred longer result summaries, as they were perceived to be more informative. However, their performance in terms of correctly identifying relevant documents was similar across all four conditions. Furthermore, while the participants felt that longer summaries were more informative, empirical observations suggest otherwise; while participants were more likely to click on relevant items given longer summaries, they also were more likely to click on non-relevant items. This shows that longer is not necessarily better, though participants perceived that to be the case - and second, they reveal a positive relationship between the length and informativeness of summaries and their attractiveness (i.e. clickthrough rates). These findings show that there are tensions between perception and performance when designing result summaries that need to be taken into account.

References

[1]

M. Ageev, D. Lagun, and E. Agichtein. Improving search result summaries by using searcher behavior data. In Proc. 35th ACM SIGIR, pages 13--22, 2013.

Digital Library

[2]

H. Ali, F. Scholer, J. A. Thom, and M. Wu. User interaction with novel web search interfaces. In Proc. 21st OZCHI, pages 301--304.

Digital Library

[3]

L. Azzopardi, D. Kelly, and K. Brennan. How query cost affects search behavior. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in IR, pages 23--32, 2013.

Digital Library

[4]

H. Bast and M. Celikik. Efficient index-based snippet generation. ACM Trans. Inf. Syst., 32(2):6:1--6:24, Apr. 2014.

Digital Library

[5]

H. Bota, K. Zhou, and J. M. Jose. Playing your cards right: The effect of entity cards on search behaviour and workload. In Proc. 1st ACM CHIIR, pages 131--140, 2016.

Digital Library

[6]

H. Chen and S. Dumais. Bringing order to the web: Automatically categorizing search results. In Proc. 18th ACM CHI, pages 145--152, 2000.

Digital Library

[7]

M. C. Chen, J. R. Anderson, and M. H. Sohn. What can a mouse cursor tell us more? Correlation of eye/mouse movements on web browsing. In Proc. 19th ACM CHI Extended Abstracts, pages 281--282, 2001.

Digital Library

[8]

F. Chierichetti, R. Kumar, and P. Raghavan. Optimizing two-dimensional search results presentation. In Proc. 4th ACM WSDM, pages 257--266, 2011.

Digital Library

[9]

C. L. A. Clarke, E. Agichtein, S. Dumais, and R. W. White. The influence of caption features on clickthrough patterns in web search. In Proc. 30th ACM SIGIR, pages 135--142, 2007.

Digital Library

[10]

N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. In Proc. 1st ACM WSDM, pages 87--94, 2008.

Digital Library

[11]

E. Cutrell and Z. Guan. What are you looking for? An eye-tracking study of information usage in web search. In Proc. 25th ACM CHI, pages 407--416, 2007.

[12]

S. Dumais, E. Cutrell, and H. Chen. Optimizing search by showing results in context. In Proc. 19th ACM CHI, pages 277--284, 2001.

Digital Library

[13]

H. Feild, R. Jones, R. Miller, R. Nayak, E. Churchill, and E. Velipasaoglu. Logging the search self-efficacy of amazon mechanical turkers. In Proc. CSE SIGIR Workshop, pages 27--30, 2010.

[14]

J. He, P. Duboue, and J.-Y. Nie. Bridging the gap between intrinsic and perceived relevance in snippet generation. In Proc. of COLING 2012, pages 1129--1146, 2012.

[15]

M. Hearst. Search user interfaces. Cambridge University Press, 2009.

[16]

P. Ingwersen and K. Järvelin. The Turn: Integration of Information Seeking and Retrieval in Context. 2005.

[17]

T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proc. 28th ACM SIGIR, pages 154--161, 2005.

Digital Library

[18]

H. Joho and J. M. Jose. A comparative study of the effectiveness of search result presentation on the web. In Proc. 28th ECIR, pages 302--313, 2006.

Digital Library

[19]

M. Kaisser, M. A. Hearst, and J. B. Lowe. Improving search results quality by customizing summary lengths. In Proc. 46th ACL, pages 701--709, 2008.

[20]

Y. Kammerer and P. Gerjets. How the interface design influences users' spontaneous trustworthiness evaluations of web search results: comparing a list and a grid interface. In Proc. of the Symp. on Eye-Tracking Research & Applications, pages 299--306, 2010.

Digital Library

[21]

T. Kanungo and D. Orr. Predicting the readability of short web summaries. In Proc. 2nd ACM WSDM, pages 202--211, 2009.

Digital Library

[22]

D. Kelly and L. Azzopardi. How many results per page? A study of serp size, search behavior and user experience. In Proc. 38th ACM SIGIR, pages 183--192, 2015.

[23]

D. Kelly, K. Gyllstrom, and E. W. Bailey. A comparison of query and term suggestion features for interactive searching. In Proc. 32nd ACM SIGIR, pages 371--378, 2009.

Digital Library

[24]

J. Kim, P. Thomas, R. Sankaranarayana, and T. Gedeon. Comparing scanning behaviour in web search on small and large screens. In Proc. 17th ADCS, pages 25--30, 2012.

Digital Library

[25]

J. Kim, P. Thomas, R. Sankaranarayana, T. Gedeon, and H.-J. Yoon. Eye-tracking analysis of user behavior and performance in web search on large and small screens. J. of the Assoc. for Information Science and Technology, 2014.

[26]

J. Kim, P. Thomas, R. Sankaranarayana, T. Gedeon, and H.-J. Yoon. Pagination versus scrolling in mobile web search. In Proc. 25th ACM CIKM, pages 751--760, 2016.

Digital Library

[27]

J. Kim, P. Thomas, R. Sankaranarayana, T. Gedeon, and H.-J. Yoon. What snippet size is needed in mobile web search? In Proc. 2nd ACM CHIIR, pages 97--106, 2017.

Digital Library

[28]

S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22:79--86, 1951.

[29]

J. Kupiec, J. Pedersen, and F. Chen. A trainable document summarizer. In Proc. 18th ACM SIGIR, pages 68--73, 1995.

Digital Library

[30]

T. Landauer, D. Egan, J. Remde, M. Lesk, C. Lochbaum, and D. Ketchum. Enhancing the usability of text through computer delivery and formative evaluation: the superbook project. Hypertext: A psychological perspective, pages 71--136, 1993.

[31]

L. Leal-Bando, F. Scholer, and A. Turpin. Query-biased summary generation assisted by query expansion. J. Assoc. for Info. Sci. and Tech., 66(5):961--979, 2015.

Digital Library

[32]

Q. Li and Y. P. Chen. Personalized text snippet extraction using statistical language models. Pattern Recogn., 43(1):378--386, Jan. 2010.

Digital Library

[33]

G. Linden. Marissa mayer at web 2.0, November 2006. http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html.

[34]

Marcos, M-C. and Gavin, F. and Arapakis, I. Effect of snippets on user experience in web search. In Proc. 16th Intl. Conf. on HCI, pages 47:1--47:8, 2015.

Digital Library

[35]

A. Muralidharan, Z. Gyongyi, and E. Chi. Social annotations in web search. In Proc. 30th ACM CHI, pages 1085--1094, 2012.

Digital Library

[36]

V. Navalpakkam, L. Jentzsch, R. Sayres, S. Ravi, A. Ahmed, and A. Smola. Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In Proc. 22nd WWW, pages 953--964, 2013.

Digital Library

[37]

A. Oulasvirta, J. Hukkinen, and B. Schwartz. When more is less: The paradox of choice in search engine use. In Proc. 32nd ACM SIGIR, pages 516--523, 2009.

Digital Library

[38]

T. Paek, S. Dumais, and R. Logan. Wavelens: A new view onto internet search results. In Proc. 22nd ACM CHI, pages 727--734, 2004.

Digital Library

[39]

J. Pedersen, D. Cutting, J. Tukey, et al. Snippet search: A single phrase approach to text access. In Proc. 1991 Joint Statistical Meetings, 1991.

[40]

M. L. Resnick, C. Maldonado, J. M. Santos, and R. Lergier. Modeling on-line search behavior using alternative output structures. In Proc. Human Factors and Ergonomics Soc. Annual Meeting, volume 45, pages 1166--1170, 2001.

[41]

D. E. Rose, D. Orr, and R. G. P. Kantamneni. Summary attributes and perceived search quality. In Proc. 16th WWW, pages 1201--1202, 2007.

Digital Library

[42]

D. Savenkov, P. Braslavski, and M. Lebedev. Search snippet evaluation at yandex: lessons learned and future directions. Multilingual & Multimodal Information Access Evaluation, pages 14--25, 2011.

[43]

B. Schwartz. The Paradox of Choice: Why More Is Less. Harper Perennial, 2005.

[44]

M. Smucker, X. Guo, and A. Toulis. Mouse movement during relevance judging: Implications for determining user attention. In Proc. 37th ACM SIGIR, pages 979--982, 2014.

Digital Library

[45]

N. V. Spirin, A. S. Kotov, K. G. Karahalios, V. Mladenov, and P. A. Izhutov. A comparative study of query-biased and non-redundant snippets for structured search on mobile devices. In Proc. 25th ACM CIKM, pages 2389--2394, 2016.

Digital Library

[46]

K. M. Svore, J. Teevan, S. T. Dumais, and A. Kulkarni. Creating temporally dynamic web search snippets. In Proc. 35th ACM SIGIR, pages 1045--1046, 2012.

Digital Library

[47]

J. Teevan, E. Cutrell, D. Fisher, S. M. Drucker, G. Ramos, P. André, and C. Hu. Visual snippets: Summarizing web pages for search and revisitation. In Proc. 27th ACM CHI, pages 2023--2032, 2009.

Digital Library

[48]

A. Tombros and M. Sanderson. Advantages of query biased summaries in information retrieval. In Proc. 21st ACM SIGIR, pages 2--10, 1998.

Digital Library

[49]

A. Turpin, Y. Tsegay, D. Hawking, and H. E. Williams. Fast generation of result snippets in web search. In Proc. 30th ACM SIGIR, pages 127--134, 2007.

Digital Library

[50]

E. M. Voorhees. Overview of the trec 2005 robust track. In Proc. TREC-14, 2006.

[51]

R. W. White, J. M. Jose, and I. Ruthven. A task-oriented study on the influencing effects of query-biased summarisation in web searching. Info. Processing & Management, 39(5):707--733, 2003.

Digital Library

[52]

A. Woodruff, R. Rosenholtz, J. B. Morrison, A. Faulring, and P. Pirolli. A comparison of the use of text summaries, plain thumbnails, and enhanced thumbnails for web search tasks. J. Am. Soc. Inf. Sci. Technol., 53(2):172--185, 2002.

Digital Library

[53]

Wu, W-C. and Kelly, D. and Sud, A. Using information scent and need for cognition to understand online search behavior. In Proceedings of the 37th International ACM SIGIR Conference, SIGIR '14, pages 557--566, 2014.

Digital Library

[54]

G. Zuccon, T. Leelanupab, S. Whiting, E. Yilmaz, J. Jose, and L. Azzopardi. Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Information Retrieval, 16(2):267--305, 2013.

Digital Library

Cited By

Azzopardi LLiu JSakai TIshita EOhshima HRadboud FMao JJose J(2024)Evaluating Cognitive Biases in Conversational and Generative IIR: A TutorialProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698437(287-290)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698437
Sasaoka TYamakata YTajima K(2024)Search Result Presentation for Non-Native Language DocumentsCompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645224(89-94)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645224
Pathak KAzzopardi LHalvey M(2024)The Influence of Presentation and Performance on User SatisfactionProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638335(77-86)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638335
Show More Cited By

Index Terms

A Study of Snippet Length and Informativeness: Behaviour, Performance and User Experience
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces

Recommendations

Engaged or Frustrated?: Disambiguating Emotional State in Search
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

One of the primary ways researchers have characterized engagement during information search is by increases in search behaviors, such as queries and clicks. However, studies have shown that frustration is also characterized by increases in these same ...
Query biased snippet generation in XML search
SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data

Snippets are used by almost every text search engine to complement ranking scheme in order to effectively handle user searches, which are inherently ambiguous and whose relevance semantics are difficult to assess. Despite the fact that XML is a standard ...
Exploring the Use of Query Auto Completion: Search Behavior and Query Entry Profiles
CHIIR '16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval

Query auto completion (QAC) is nearly ubiquitous in modern search systems, however, there are few published studies on how searchers use QAC query suggestions. This study describes the use of QAC by 29 searchers working on eight assigned search topics ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

August 2017

1476 pages

ISBN:9781450350228

DOI:10.1145/3077136

General Chairs:
Noriko Kando
National Institute of Informatics
,
Tetsuya Sakai
Waseda University
,
Hideo Joho
University of Tsukuba
,
Program Chairs:
Hang Li
Huawei Noah's Ark Lab
,
Arjen P. de Vries
Radboud University
,
Ryen W. White
Microsoft Cortana

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

SIGIR '17

Sponsor:

SIGIR

SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval

August 7 - 11, 2017

Tokyo, Shinjuku, Japan

Acceptance Rates

SIGIR '17 Paper Acceptance Rate 78 of 362 submissions, 22%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
684
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)9

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Azzopardi LLiu JSakai TIshita EOhshima HRadboud FMao JJose J(2024)Evaluating Cognitive Biases in Conversational and Generative IIR: A TutorialProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698437(287-290)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698437
Sasaoka TYamakata YTajima K(2024)Search Result Presentation for Non-Native Language DocumentsCompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645224(89-94)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645224
Pathak KAzzopardi LHalvey M(2024)The Influence of Presentation and Performance on User SatisfactionProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638335(77-86)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638335
Gienapp LScells HDeckers NBevendorff JWang SKiesel JSyed SFröbe MZuccon GStein BHagen MPotthast MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Generative Ad Hoc Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657849(1916-1929)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657849
Chen QChen JZhou XCheng GHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Enhancing Dataset Search with Compact Data SnippetsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657837(1093-1103)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657837
Kameni JBatchakui BNkambou R(2024)Optimization of Information Retrieval Systems for Learning ContextsInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00415-zOnline publication date: 8-Jul-2024
https://doi.org/10.1007/s40593-024-00415-z
Yilmaz MCicekli N(2024)Snippet Generation Using Local Alignment for Information Retrieval (LAIR)Innovative Methods in Computer Science and Computational Applications in the Era of Industry 5.010.1007/978-3-031-56322-5_6(63-78)Online publication date: 6-Apr-2024
https://doi.org/10.1007/978-3-031-56322-5_6
Pathak KAzzopardi LHalvey M(2024)Ranking Heterogeneous Search Result Pages Using the Interactive Probability Ranking PrincipleAdvances in Information Retrieval10.1007/978-3-031-56060-6_7(96-110)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56060-6_7
Deng ZDou ZWen J(2023)DeepQFM: a deep learning based query facets mining methodInformation Retrieval10.1007/s10791-023-09427-026:1-2Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1007/s10791-023-09427-0
Allen GBeijen MMaxwell DGadiraju U(2023)In a Hurry: How Time Constraints and the Presentation of Web Search Results Affect User Behaviour and ExperienceWeb Engineering10.1007/978-3-031-34444-2_16(221-235)Online publication date: 16-Jun-2023
https://doi.org/10.1007/978-3-031-34444-2_16
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents