research-article

Understanding and Mitigating Bias in Online Health Search

Authors:

Sarit KrausAuthors Info & Claims

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 265 - 274

https://doi.org/10.1145/3404835.3462930

Published: 11 July 2021 Publication History

Abstract

Search engines are perceived as a reliable source for general information needs. However, finding the answer to medical questions using search engines can be challenging for an ordinary user. Content can be biased and results may present different opinions. In addition, interpreting medically related content can be difficult for users with no medical background. All of these can lead users to incorrect conclusions regarding health related questions.

In this work we address this problem from two perspectives. First, to gain insight on users' ability to correctly answer medical questions using search engines, we conduct a comprehensive user study. We show that for questions regarding medical treatment effectiveness, participants struggle to find the correct answer and are prone to overestimating treatment effectiveness. We analyze participants' demographic traits according to age and education level and show that this problem persists in all demographic groups. We then propose a semi-automatic machine learning approach to find the correct answer to queries on medical treatment effectiveness as it is viewed by the medical community. The model relies on the opinions presented in medical papers related to the queries, as well as features representing their impact. We show that, compared to human behaviour, our method is less prone to bias. We compare various configurations of our inference model and a baseline method that determines treatment effectiveness based solely on the opinion of medical papers. The results bolster our confidence that our approach can pave the way to developing automatic bias-free tools that can help mediate complex health related content to users.

Supplementary Material

MP4 File (sigir.mp4)

Presentation Video for the SIGIR paper: Understanding and Mitigating Bias in Online Health Search

Download
28.32 MB

References

[1]

[n.d.]. Cochrane | Trusted evidence. Informed decisions. Better health. https://www.cochrane.org/. (Accessed on 02/06/2021).

[2]

2019. Majority of Adults Look Online for Health Information | PEW ResearchCenter. https://www.pewresearch.org/fact-tank/2013/02/01/majority-of-adults-look-online-for-health-information/. https://www.pewresearch.org/fact-tank/2013/02/01/majority-of-adults-look-online-for-health-information/

[3]

Asma Ben Abacha and Pierre Zweigenbaum. 2015. MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies. Information processing & management, Vol. 51, 5 (2015), 570--594.

[4]

Lutz Bornmann and Hans-Dieter Daniel. 2007. What do we know about the h index? Journal of the American Society for Information Science and technology, Vol. 58, 9 (2007), 1381--1385.

Digital Library

[5]

US Census Bureau. 2020. Educational Attainment in the United States: 2018. https://www.census.gov/data/tables/2018/demo/education-attainment/cps-detailed-tables.html

[6]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, Vol. 16 (2002), 321--357.

[7]

Amira Ghenai and Yelena Mejova. 2017. Catching Zika Fever: Application of Crowdsourcing and Machine Learning for Tracking Health Misinformation on Twitter. (jul 2017). arxiv: 1707.03778 http://arxiv.org/abs/1707.03778

[8]

Amira Ghenai and Yelena Mejova. 2018. Fake Cures: User-centric Modeling of Health Misinformation in Social Media., Vol. 2, November (2018). arxiv: 1809.00557 http://arxiv.org/abs/1809.00557

Digital Library

[9]

Patty Kostkova, Vino Mano, Heidi J. Larson, and William S. Schulz. 2016. VAC Medi+ board. In Proceedings of the 6th International Conference on Digital Health Conference - DH '16. ACM Press, New York, New York, USA, 163--164. https://doi.org/10.1145/2896338.2896370

Digital Library

[10]

Minsuk Lee, James Cimino, Hai Ran Zhu, Carl Sable, Vijay Shanker, John Ely, and Hong Yu. 2006. Beyond information retrieval-medical question answering. In AMIA annual symposium proceedings, Vol. 2006. American Medical Informatics Association, 469.

[11]

Jimmy Lin. 2008. PageRank without hyperlinks: Reranking with related article networks for biomedical text retrieval. BMC bioinformatics, Vol. 9, 1 (2008), 270.

[12]

Susan Michie, James Thomas, Johnston, et al. 2017. The Human Behaviour-Change Project: Harnessing the power of artificial intelligence and machine learning for evidence synthesis and interpretation. Implementation Science, Vol. 12 (12 2017). https://doi.org/10.1186/s13012-017-0641--5

[13]

Frances A Pogacar, Amira Ghenai, Mark D Smucker, and Charles LA Clarke. 2017. The Positive and Negative Influence of Search Results on People's Decisions about the Efficacy of Medical Treatments. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. ACM, 209--216.

Digital Library

[14]

Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2007. An introduction to information retrieval .Cambridge University Press,.

[15]

Joseph D Tucker, Suzanne Day, Weiming Tang, and Barry Bayus. 2019. Crowdsourcing in medical research: concepts and applications. PeerJ, Vol. 7 (2019), e6762.

[16]

Yuxi Wang, Martin McKee, Aleksandra Torbica, and David Stuckler. 2019. Systematic literature review on the spread of health-related misinformation on social media. Social Science & Medicine, Vol. 240 (2019), 112552.

[17]

Ryen White. 2013. Beliefs and biases in web search. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 3--12.

Digital Library

[18]

Ryen W White and Ahmed Hassan. 2014. Content bias in online health search. ACM Transactions on the Web (TWEB), Vol. 8, 4 (2014), 1--33.

Digital Library

[19]

Elliot J Yates and Louise C Dixon. 2015. PageRank as a method to rank biomedical literature by importance. Source code for biology and medicine, Vol. 10, 1 (2015), 16.

[20]

Bei Yu, Matt Willis, Peiyuan Sun, and Jun Wang. 2013. Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk. Journal of medical Internet research, Vol. 15, 6 (2013), e108.

[21]

Haijun Zhai, Todd Lingren, Louise Deleger, Qi Li, Megan Kaiser, Laura Stoutenborough, and Imre Solti. 2013. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of medical Internet research, Vol. 15, 4 (2013), e73.

Cited By

Lockett AZhang JBorji STowne GSingh SMobley‐Lee JConway LMcConnell MStanton AWilson MNovin A(2024)Health Insurance Literacy Among Young AdultsProceedings of the Association for Information Science and Technology10.1002/pra2.116861:1(1005-1007)Online publication date: 15-Oct-2024
De Jonge THiemstra D(2023)UNFair: Search Engine Manipulation, Undetectable by Amortized InequityProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594046(830-839)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594046
Hashavit AWang HStern TKraus SChen HDuh WHuang HKato MMothe JPoblete B(2023)Not Just Skipping: Understanding the Effect of Sponsored Content on Users' Decision-Making in Online Health SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591744(1056-1065)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591744
Show More Cited By

Index Terms

Understanding and Mitigating Bias in Online Health Search
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Retrieval tasks and goals
      1. Clustering and classification

Recommendations

Content Bias in Online Health Search

Search engines help people answer consequential questions. Biases in retrieved and indexed content (e.g., skew toward erroneous outcomes that represent deviations from reality), coupled with searchers' biases in how they examine and interpret search ...
Quality of health-related online search results

Consumers are increasingly searching for health information online and using that information to inform their decisions and behavior. Because the negative consequences of basing decisions on inaccurate or untrustworthy health information may be ...
Beliefs and biases in web search
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

People's beliefs, and unconscious biases that arise from those beliefs, influence their judgment, decision making, and actions, as is commonly accepted among psychologists. Biases can be observed in information retrieval in situations where searchers ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2021

2998 pages

ISBN:9781450380379

DOI:10.1145/3404835

General Chairs:
Fernando Diaz
(Google)
,
Chirag Shah
University of Washington
,
Torsten Suel
New York University
,
Program Chairs:
Pablo Castells
Universidad Autónoma de Madrid, Amazon
,
Rosie Jones
Spotify
,
Tetsuya Sakai
Waseda University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF IIS
Israel Innovation Authority

Conference

SIGIR '21

Sponsor:

SIGIR

SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2021

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
266
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)5

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lockett AZhang JBorji STowne GSingh SMobley‐Lee JConway LMcConnell MStanton AWilson MNovin A(2024)Health Insurance Literacy Among Young AdultsProceedings of the Association for Information Science and Technology10.1002/pra2.116861:1(1005-1007)Online publication date: 15-Oct-2024
De Jonge THiemstra D(2023)UNFair: Search Engine Manipulation, Undetectable by Amortized InequityProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594046(830-839)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594046
Hashavit AWang HStern TKraus SChen HDuh WHuang HKato MMothe JPoblete B(2023)Not Just Skipping: Understanding the Effect of Sponsored Content on Users' Decision-Making in Online Health SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591744(1056-1065)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591744
Liu JLiu J(2023)Behavioral Economics in IRA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_6(155-180)Online publication date: 18-Feb-2023
https://doi.org/10.1007/978-3-031-23229-9_6
Häußler HSchultheiß SLewandowski D(2023)Is googling risky? A study on risk perception and experiences of adverse consequences in web searchJournal of the Association for Information Science and Technology10.1002/asi.2480275:5(567-580)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1002/asi.24802
Song QJiang J(2022)How Misinformation Density Affects Health Information SearchProceedings of the ACM Web Conference 202210.1145/3485447.3512141(2668-2677)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3512141
Zhang DVakili Tahami AAbualsaud MSmucker MAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Learning Trustworthy Web Sources to Derive Correct Answers and Reduce Health Misinformation in SearchProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531812(2099-2104)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531812
Srba IPecher BTomlein MMoro RStefancova ESimko JBielikova MAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Monant Medical Misinformation DatasetProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531726(2949-2959)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531726
Bondarenko AShirshakova EDriker MHagen MBraslavski PDemartini GZuccon GCulpepper JHuang ZTong H(2021)Misbeliefs and Biases in Health-Related SearchesProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482141(2894-2899)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482141

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents