research-article

Open access

Featured Snippets and their Influence on Users’ Credibility Judgements

Authors:

Steven Zimmerman,

David ElsweilerAuthors Info & Claims

CHIIR '22: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

Pages 113 - 122

https://doi.org/10.1145/3498366.3505766

Published: 14 March 2022 Publication History

All formats PDF

Abstract

Search engines often provide featured snippets, which are boxed and placed above other results with the aim of directly answering user queries. To learn about how users judge the credibility of such results and how they influence search outcomes, a controlled web-based user study (N = 96) was conducted. Using resources made available by scholars in the community, we study featured snippets in a medical context with participants being tasked with determining whether a named treatment is helpful for a specified medical condition both before and after viewing the search results. Experimental conditions varied the presence and credibility of featured snippets. Our findings indicate that participants tend to overestimate the credibility of information in featured snippets. Featured snippets are, moreover, shown to often change users’ opinion about a topic, especially if they are uncertain. Showing correct information inside featured snippets helped participants make more accurate decisions, whereas incorrect or contradicting information led to more harmful outcomes.

References

[1]

Leif Azzopardi. 2021. Cognitive Biases in Search: A Review and Reflection of Cognitive Biases in Information Retrieval. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval(CHIIR ’21). 27–37.

Digital Library

[2]

Michael S. Bernstein, Jaime Teevan, Susan Dumais, Daniel Liebling, and Eric Horvitz. 2012. Direct Answers for Search Queries in the Long Tail. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’12). 237–246.

Digital Library

[3]

Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information research 8, 3 (2003), 8–3.

[4]

Stuart K. Card, Peter Pirolli, Mija Van Der Wege, Julie B. Morrison, Robert W. Reeder, Pamela K. Schraedley, and Jenea Boshart. 2001. Information Scent as a Driver of Web Behavior Graphs: Results of a Protocol Analysis Method for Web Usability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’01). 498–505.

Digital Library

[5]

Lydia B. Chilton and Jaime Teevan. 2011. Addressing People’s Information Needs Directly in a Web Search Result Page. In Proceedings of the 20th International Conference on World Wide Web(WWW ’11). 27–36.

[6]

Aleksandr Chuklin and Pavel Serdyukov. 2012. Good Abandonments in Factoid Queries. In Proceedings of the 21st International Conference on World Wide Web(WWW ’12 Companion). 483–484.

Digital Library

[7]

Charles L. A. Clarke, Eugene Agichtein, Susan Dumais, and Ryen W. White. 2007. The Influence of Caption Features on Clickthrough Patterns in Web Search. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’07). 135–142.

Digital Library

[8]

Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic press.

[9]

Lubna Daraz, Allison S. Morrow, Oscar J. Ponce, Bradley Beuschel, Magdoleen H. Farah, Abdulrahman Katabi, Mouaz Alsawas, Abdul M. Majzoub, Raed Benkhadra, Mohamed O. Seisa, 2019. Can patients trust online health information? A meta-narrative systematic review addressing the quality of health information on the internet. Journal of general internal medicine 34, 9 (2019), 1884–1891.

[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).

[11]

Marcos Fernández-Pichel, David E Losada, Juan C Pichel, and David Elsweiler. 2021. Comparing Traditional and Neural Approaches for Detecting Health-Related Misinformation. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, 78–90.

[12]

B. J. Fogg. 2003. Prominence-Interpretation Theory: Explaining How People Assess Credibility Online. In CHI ’03 Extended Abstracts on Human Factors in Computing Systems(CHI EA ’03). 722–723.

[13]

Susannah Fox and Maeve Duggan. 2013. Health online 2013. Health 2013(2013), 1–55.

[14]

Alexandru L Ginsca, Adrian Popescu, and Mihai Lupu. 2015. Credibility in information retrieval. Foundations and Trends in Information Retrieval 9, 5 (2015), 355–475.

Digital Library

[15]

Carolin Hahnel, Frank Goldhammer, Ulf Kröhne, and Johannes Naumann. 2018. The role of reading skills in the evaluation of online information gathered from search engine environments. Computers in Human Behavior 78 (2018), 223–234.

Digital Library

[16]

Eszter Hargittai, Lindsay Fullerton, Ericka Menchen-Trevino, and Kristin Yates Thomas. 2010. Trust online: Young adults’ evaluation of web content. International journal of communication 4 (2010), 27.

[17]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. Evaluating the Accuracy of Implicit Feedback from Clicks and Query Reformulations in Web Search. ACM Trans. Inf. Syst. (April 2007), 7–es.

Digital Library

[18]

Markus Kattenbeck and David Elsweiler. 2019. Understanding credibility judgements for web search snippets. Aslib Journal of Information Management(2019).

[19]

Joshua Klayman and Young-Won Ha. 1987. Confirmation, disconfirmation, and information in hypothesis testing.Psychological review 94, 2 (1987), 211.

[20]

Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2017. Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing(CSCW ’17). 417–432.

Digital Library

[21]

Jane Li, Scott Huffman, and Akihito Tokuda. 2009. Good Abandonment in Mobile and PC Internet Search. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’09). 43–50.

Digital Library

[22]

D. Harrison McKnight and Charles J. Kacmar. 2007. Factors and Effects of Information Credibility. In Proceedings of the Ninth International Conference on Electronic Commerce(ICEC ’07). 423–432.

[23]

Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging Joint Interactions for Credibility Analysis in News Communities. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management(CIKM ’15). 353–362.

Digital Library

[24]

Raymond S. Nickerson. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175–220.

[25]

Alexandra Olteanu, Stanislav Peshterliev, Xin Liu, and Karl Aberer. 2013. Web Credibility: Features Exploration and Credibility Prediction. In Proceedings of the 35th European Conference on Advances in Information Retrieval(ECIR’13). 557–568.

Digital Library

[26]

Sandeep Pandey, Kedar Dhamdhere, and Christopher Olston. 2004. WIC: A General-Purpose Algorithm for Monitoring Web Information Sources. In Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30(VLDB ’04). 360–371.

[27]

Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Charles L.A. Clarke. 2017. The Positive and Negative Influence of Search Results on People’s Decisions about the Efficacy of Medical Treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval(ICTIR ’17). 209–216.

Digital Library

[28]

Martin Potthast, Matthias Hagen, and Benno Stein. 2021. The Dilemma of the Direct Answer. SIGIR Forum 54, 1, Article 14 (2021), 12 pages.

[29]

Prolific. 2021. Using attention checks as a measure of data quality. https://researcher-help.prolific.co/hc/en-gb/articles/360009223553-Using-attention-checks-as-a-measure-of-data-quality

[30]

Kristen Purcell, Lee Rainie, and Joanna Brenner. 2012. Search engine use 2012. (2012).

[31]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683(2019).

[32]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250(2016).

[33]

Louis M. Rea and Richard A. Parker. 2014. Designing and conducting survey research: A comprehensive guide. John Wiley & Sons.

[34]

Kristina Sam-Martin. 2020. Google’s Featured Snippets in the Context of Strategic Content Marketing. (2020). https://sam-martin.at/downloads/featured-snippets-and-strategic-content-marketing-ebook-2020/

[35]

Julia Schwarz and Meredith Morris. 2011. Augmenting Web Pages and Search Results to Support Credibility Assessment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’11). 1245–1254.

Digital Library

[36]

Milad Shokouhi, Ryen White, and Emine Yilmaz. 2015. Anchoring and Adjustment in Relevance Estimation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’15). 963–966.

Digital Library

[37]

Parikshit Sondhi, V. G. Vinod Vydiswaran, and Cheng Xiang Zhai. 2012. Reliability Prediction of Webpages in the Medical Domain. In Proceedings of the 34th European Conference on Advances in Information Retrieval(ECIR’12). 219–231.

Digital Library

[38]

Sofia Stamou and Efthimis N. Efthimiadis. 2010. Interpreting User Inactivity on Search Results. In Proceedings of the 32nd European Conference on Advances in Information Retrieval(ECIR’2010). 100–113.

Digital Library

[39]

Artur Strzelecki and Paulina Rutecka. 2020. Direct Answers in Google Search Results. IEEE Access 8(2020), 103642–103654.

[40]

Artur Strzelecki and Paulina Rutecka. 2020. Featured snippets results in Google Web search: An exploratory study. In Marketing and Smart Technologies. Springer, 9–18.

[41]

Danny Sullivan. 2021. A reintroduction to Google’s featured snippets. https://blog.google/products/search/reintroduction-googles-featured-snippets

[42]

R Core Team 2013. R: A language and environment for statistical computing. (2013).

[43]

Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science 185, 4157 (1974), 1124–1131.

[44]

Julian Unkel and Alexander Haas. 2017. The effects of credibility cues on the selection of search engine results. Journal of the Association for Information Science and Technology 68, 8(2017), 1850–1862.

Digital Library

[45]

Ryen White. 2013. Beliefs and Biases in Web Search. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’13). 3–12.

Digital Library

[46]

Ryen W. White. 2014. Belief Dynamics in Web Search. J. Assoc. Inf. Sci. Technol.(2014), 2165–2178.

[47]

Ryen W. White and Ahmed Hassan. 2014. Content Bias in Online Health Search. ACM Trans. Web (2014), 33 pages.

[48]

Kyle Williams, Julia Kiseleva, Aidan C. Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. 2016. Detecting Good Abandonment in Mobile Search. In Proceedings of the 25th International Conference on World Wide Web(WWW ’16). 495–505.

Digital Library

[49]

Kyle Williams, Julia Kiseleva, Aidan C. Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. 2016. Is This Your Final Answer? Evaluating the Effect of Answers on Good Abandonment in Mobile Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR ’16). 889–892.

[50]

Zhijing Wu, Mark Sanderson, B. Barla Cambazoglu, W. Bruce Croft, and Falk Scholer. 2020. Providing Direct Answers in Search Results: A Study of User Behavior. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management(CIKM ’20). 1635–1644.

Digital Library

[51]

Yusuke Yamamoto and Katsumi Tanaka. 2011. Enhancing Credibility Judgment of Web Search Results. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’11). 1235–1244.

Digital Library

[52]

Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond Position Bias: Examining Result Attractiveness as a Source of Presentation Bias in Clickthrough Data. In Proceedings of the 19th International Conference on World Wide Web(WWW ’10). 1011–1018.

Digital Library

[53]

Wei Zha and H Denis Wu. 2014. The Impact of Online Disruptive Ads on Users’ Comprehension, Evaluation of Site Credibility, and Sentiment of Intrusiveness.American Communication Journal 16, 2 (2014).

[54]

Yiming Zhao, Jin Zhang, Xue Xia, and Taowen Le. 2019. Evaluation of Google question-answering quality. Library Hi Tech (2019).

[55]

Steven Zimmerman, Alistair Thorpe, Chris Fox, and Udo Kruschwitz. 2019. Privacy Nudging in Search: Investigating Potential Impacts. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval(CHIIR ’19). 283–287.

Digital Library

Cited By

Oliveira BTeixeira Lopes C(2023)The Evolution of Web Search User Interfaces - An Archaeological Analysis of Google Search Engine Result PagesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578320(55-68)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578320
Oliveira BTeixeira Lopes C(2023)From 10 Blue Links Pages to Feature-Full Search Engine Results Pages - Analysis of the Temporal Evolution of SERP FeaturesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578307(338-345)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578307
Draws TNatesan Ramamurthy KBaldini IDhurandhar APadhi ITimmermans BTintarev N(2023)Explainable Cross-Topic Stance Detection for Search ResultsProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578296(221-235)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578296
Show More Cited By

Index Terms

Featured Snippets and their Influence on Users’ Credibility Judgements

Index terms have been assigned to the content through auto-classification.

Recommendations

Investigating the Influence of Featured Snippets on User Attitudes
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and Retrieval

Featured snippets that attempt to satisfy users’ information needs directly on top of the first search engine results page (SERP) have been shown to strongly impact users’ post-search attitudes and beliefs. In the context of debated but scientifically ...
Providing Direct Answers in Search Results: A Study of User Behavior
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

To study the impact of providing direct answers in search results on user behavior, we conducted a controlled user study to analyze factors including reading time, eye-tracked attention, and the influence of the quality of answer module content. We also ...
Individual Judgments Versus Consensus: Estimating Query-URL Relevance

Query-URL relevance, measuring the relevance of each retrieved URL with respect to a given query, is one of the fundamental criteria to evaluate the performance of commercial search engines. The traditional way to collect reliable and accurate query-URL ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHIIR '22: Proceedings of the 2022 Conference on Human Information Interaction and Retrieval

March 2022

399 pages

ISBN:9781450391863

DOI:10.1145/3498366

General Chairs:
David Elsweiler
University of Regensburg, Bavaria, Germany
,
Udo Kruschwitz
University of Regensburg, Bavaria, Germany
,
Bernd Ludwig
University of Regensburg, Bavaria, Germany

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHIIR '22

Sponsor:

CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval

March 14 - 18, 2022

Regensburg, Germany

Acceptance Rates

Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
945
Total Downloads

Downloads (Last 12 months)412
Downloads (Last 6 weeks)45

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oliveira BTeixeira Lopes C(2023)The Evolution of Web Search User Interfaces - An Archaeological Analysis of Google Search Engine Result PagesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578320(55-68)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578320
Oliveira BTeixeira Lopes C(2023)From 10 Blue Links Pages to Feature-Full Search Engine Results Pages - Analysis of the Temporal Evolution of SERP FeaturesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578307(338-345)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578307
Draws TNatesan Ramamurthy KBaldini IDhurandhar APadhi ITimmermans BTintarev N(2023)Explainable Cross-Topic Stance Detection for Search ResultsProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578296(221-235)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578296
Hashavit AWang HStern TKraus SChen HDuh WHuang HKato MMothe JPoblete B(2023)Not Just Skipping: Understanding the Effect of Sponsored Content on Users' Decision-Making in Online Health SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591744(1056-1065)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591744
Wu ZDraws TCau FBarile FRieger ATintarev N(2023)Explaining Search Result Stances to Opinionated PeopleExplainable Artificial Intelligence10.1007/978-3-031-44067-0_29(573-596)Online publication date: 21-Oct-2023
https://doi.org/10.1007/978-3-031-44067-0_29
Häußler H(2023)Trust in Search Engines: Developing a Trust Measure and Applying It in an ExperimentProceedings of the Association for Information Science and Technology10.1002/pra2.82260:1(597-602)Online publication date: 22-Oct-2023
https://doi.org/10.1002/pra2.822
Häußler HSchultheiß SLewandowski D(2023)Is googling risky? A study on risk perception and experiences of adverse consequences in web searchJournal of the Association for Information Science and Technology10.1002/asi.2480275:5(567-580)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1002/asi.24802

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents