research-article

Public Access

Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

Authors:

Md Momen Bhuiyan,

Connie Moon Sehat,

Tanushree MitraAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 4, Issue CSCW2

Article No.: 93, Pages 1 - 26

https://doi.org/10.1145/3415164

Published: 15 October 2020 Publication History

Abstract

Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.

References

[1]

William RL Anderegg, James W Prall, Jacob Harold, and Stephen H Schneider. 2010. Expert credibility in climate change. Proceedings of the National Academy of Sciences, Vol. 107, 27 (2010), 12107--12109.

[2]

Lora Aroyo and Chris Welty. 2015. Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation. AI Magazine, Vol. 36, 1 (2015), 15--24.

Digital Library

[3]

Mahmoudreza Babaei, Abhijnan Chakraborty, Juhi Kulshrestha, Elissa M Redmiles, Meeyoung Cha, and Krishna P Gummadi. 2019. Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking. In FAT. 139.

[4]

Mevan Babakar. 2018. Crowdsourced Factchecking.

[5]

Betsy Jane Becker. 1994. Combining significance levels. The handbook of research synthesis (1994), 215--230.

[6]

Joshua Becker, Ethan Porter, and Damon Centola. 2019. The wisdom of partisan crowds. Proceedings of the National Academy of Sciences of the United States of America, Vol. 166, 22 (2019), 10717--10722. https://doi.org/10.1073/pnas.1817195116

[7]

Brooke Borel. 2015. The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec 2015). https://www.theguardian.com/media/2015/dec/30/problem-with-science-journalism-2015-reality-kevin-folta

[8]

Alexandre Bovet and Hernán A Makse. 2019. Influence of fake news in Twitter during the 2016 US presidential election. Nature communications, Vol. 10, 1 (2019), 7.

[9]

Mohamad Adam Bujang and Nurakmal Baharum. 2016. Sample size guideline for correlation analysis. World, Vol. 3, 1 (2016).

[10]

Cody Buntain and Jennifer Golbeck. 2017. Automatically Identifying Fake News in Popular Twitter Threads. Proceedings - 2nd IEEE International Conference on Smart Cloud, SmartCloud 2017 (2017), 208--215. https://doi.org/10.1109/SmartCloud.2017.40 arxiv: 1705.01613

[11]

Davide Ceolin. 2019. Conference Presentation: On the Quality of Crowdsourced Information Quality Assessments. https://drive.google.com/a/hackshackers.com/file/d/1AJmFmRqEhdhSIZLwhXT_1bzStXfV-hVf/view?usp=drive_open&usp=embed_facebook

[12]

Steven H Chaffee. 1982. Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter/media: Interpersonal communication in a media world, Vol. 57 (1982), 77.

[13]

Shelly Chaiken. 1987. The heuristic model of persuasion. In Social influence: the ontario symposium, Vol. 5. Hillsdale, NJ: Lawrence Erlbaum, 3--39.

[14]

Roy De Maesschalck, Delphine Jouan-Rimbaud, and Désiré L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems, Vol. 50, 1 (2000), 1--18.

[15]

Jaap J Dijkstra, Wim BG Liebrand, and Ellen Timminga. 1998. Persuasiveness of expert systems. Behaviour & Information Technology, Vol. 17, 3 (1998), 155--163.

[16]

Ziv Epstein, Gordon Pennycook, and David Rand. 2020. Will the Crowd Game the Algorithm? Using Layperson Judgments to Combat Misinformation on Social Media by Downranking Distrusted Sources. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, 1--11. https://doi.org/10.1145/3313831.3376232

Digital Library

[17]

Jonathan St BT Evans. 2008. Dual-processing accounts of reasoning, judgment, and social cognition. Annu. Rev. Psychol., Vol. 59 (2008), 255--278.

[18]

Facebook. 2020. Fact-Checking on Facebook: What Publishers Should Know. https://www.facebook.com/help/publisher/182222309230722. (Accessed on 01/14/2020).

[19]

FactCheckEU. [n.d.]. FactCheckEU - 19 European media outlets are fact-checking the May 2019 European elections. https://www.factcheckeu.info/en/. (Accessed on 05/14/2020).

[20]

Andrew J Flanagin and Miriam J Metzger. 2008. Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility (2008), 5--27.

[21]

Fabrice Florin. 2010. Crowdsourced Fact-Checking? What We Learned from Truthsquad. Mediashift (2010).

[22]

Brian J Fogg. 2003. Prominence-interpretation theory: Explaining how people assess credibility online. In CHI'03 extended abstracts on human factors in computing systems. Citeseer, 722--723.

Digital Library

[23]

Brian J Fogg and Hsiang Tseng. 1999. The elements of computer credibility. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 80--87.

Digital Library

[24]

American Press Institute & The AP-NORC Center for Public Affairs Research. 2018. Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project (2018).

[25]

Cary Funk, Meg Hefferon, Brian Kennedy, and Courtney Johnson. 2019. Trust and Mistrust in Americans? Views of Scientific Experts. Pew Research Center. https://www. pewresearch. org/science/2019/08/02/trust-and-mistrust-inamericans-views-of-scientific-experts (2019).

[26]

Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly, Vol. 63, 3 (1986), 451--462.

[27]

Emma Grillo. 2020. What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr 2020). https://www.nytimes.com/2020/04/05/reader-center/coronavirus-sports-reporting.html

[28]

Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on Twitter during the 2016 U.S. presidential election. Science, Vol. 363, 6425 (Jan 2019), 374--378.

[29]

Naeemul Hassan, Mohammad Yousuf, Mahfuzul Haque, Javier A Suarez Rivas, and Md Khadimul Islam. 2017. Towards A Sustainable Model for Fact-checking Platforms: Examining the Roles of Automation, Crowds and Professionals. https://doi.org/10.1145/3308560.3316734

[30]

Brian Hilligoss and Soo Young Rieh. 2008. Developing a unifying framework of credibility assessment: Construct, heuristics, and interaction in context. Information Processing & Management, Vol. 44, 4 (2008), 1467--1484.

Digital Library

[31]

Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. (2017), 759--766. arxiv: 1703.09398 http://arxiv.org/abs/1703.09398

[32]

Carl Iver Hovland, Irving Lester Janis, and Harold H Kelley. 1953. Communication and persuasion. (1953).

[33]

Rebecca Iannucci and Bill Adair. 2017. Reporters? Lab Study Results: Effective News Labeling and Media Literacy.

[34]

Jonathan Kennedy. 2019. Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data. European Journal of Public Health, Vol. 29, 3 (Jun 2019), 512--516. https://doi.org/10.1093/eurpub/ckz004

[35]

Gary King and Richard Nielsen. 2019. Why propensity scores should not be used for matching. Political Analysis, Vol. 27, 4 (2019), 435--454.

[36]

Spiro Kiousis. 2001. Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society, Vol. 4, 4 (2001), 381--403.

[37]

Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.

Digital Library

[38]

Michael Lucibella. 2009. Science Journalism Faces Perilous Times. American Physical Society (APS) News, Vol. 18, 4 (Apr 2009). http://www.aps.org/publications/apsnews/200904/journalism.cfm

[39]

Albert Mannes, Jack Soll, and Richard Larrick. 2014. The Wisdom of Select Crowds. Journal of personality and social psychology (2014).

[40]

Albert E. Mannes, Richard P. Larrick, and Jack B. Soll. 2012. The social psychology of the wisdom of crowds.

[41]

Aaron M. McCright, Katherine Dentzman, Meghan Charters, and Thomas Dietz. 2013. The influence of political ideology on trust in science. Environmental Research Letters, Vol. 8, 4 (Nov 2013), 044029.

[42]

Miriam J Metzger. 2007. Making sense of credibility on the Web: Models for evaluating online information and recommendations for future research. Journal of the American Society for Information Science and Technology, Vol. 58, 13 (2007), 2078--2091.

[43]

Miriam J Metzger, Ethan H Hartsell, and Andrew J Flanagin. 2015. Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research (2015), 0093650215613136.

[44]

Philip Meyer. 1988. Defining and measuring credibility of newspapers: Developing an index. Journalism quarterly, Vol. 65, 3 (1988), 567--574.

[45]

Amy Mitchell, Jeffrey Gottfried, Michael Barthel, and Nami Sumida. 2018. Can Americans Tell Factual From Opinion Statements in the News?

[46]

Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations. In Proc. ICWSM'15.

[47]

Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proc. CHI'15. ACM, 1345--1354.

Digital Library

[48]

Kevin Munger, Mario Luca, Jonathan Nagler, and Joshua Tucker. 2019. Age matters: Sampling strategies for studying digital media effects.

[49]

American Society of Newspaper Editors. 1975. ASNE Statement of Principles. https://members.newsleaders.org/content.asp?pl=24&sl=171&contentid=171. (Accessed on 01/14/2020).

[50]

Daniel J O'Keefe. 2008. Persuasion. The International Encyclopedia of Communication (2008).

[51]

Sheila O'Riordan, Gaye Kiely, Bill Emerson, and Joseph Feller. 2019. Do you have a source for that? Understanding the Challenges of Collaborative Evidence-based Journalism. In Proceedings of the 15th International Symposium on Open Collaboration. 1--10.

Digital Library

[52]

Gordon Pennycook, Tyrone Cannon, and David G. Rand. 2018. Prior Exposure Increases Perceived Accuracy of Fake News. Number ID 2958246. https://papers.ssrn.com/abstract=2958246

[53]

Gordon Pennycook and David G. Rand. 2019 a. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences, Vol. 116, 7 (Feb 2019), 2521--2526.

[54]

Gordon Pennycook and David G. Rand. 2019 b. Who Falls for Fake News? The Roles of Bullshit Receptivity, Overclaiming, Familiarity, and Analytic Thinking. Number ID 3023545. https://papers.ssrn.com/abstract=3023545

[55]

Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion. Springer, 1--24.

[56]

The Trust Project. 2017. Collaborator Materials. https://thetrustproject.org/collaborator-materials/. (Accessed on 01/14/2020).

[57]

Soo Young Rieh and David R Danielson. 2007. Credibility: A multidisciplinary framework. Annual review of information science and technology, Vol. 41, 1 (2007), 307--364.

[58]

Robert M. Ross, David G. Rand, and Gordon Pennycook. 2019. Beyond 'fake news': The role of analytic thinking in the detection of inaccuracy and partisan bias in news headlines. (2019), 1--22.

[59]

Linda Schamber. 1991. Users' Criteria for Evaluation in a Multimedia Environment. In Proceedings of the ASIS Annual Meeting, Vol. 28. ERIC, 126--33.

[60]

Dietram A Scheufele and Nicole M Krause. 2019. Science audiences, misinformation, and fake news. Proceedings of the National Academy of Sciences, Vol. 116, 16 (2019), 7662--7669.

[61]

Tracy Jia Shen, Robert Cowell, Aditi Gupta, Thai Le, Amulya Yadav, and Dongwon Lee. 2019. How Gullible Are You?: Predicting Susceptibility to Fake News. In Proceedings of the 10th ACM Conference on Web Science (WebSci '19). ACM, 287--288. https://doi.org/10.1145/3292522.3326055 event-place: Boston, Massachusetts, USA.

Digital Library

[62]

Art Silverblatt, Donald C. Miller, Julie Smith, and Nikole Brown. 2014. Media Literacy: Keys to Interpreting Media Messages, 4th Edition: Keys to Interpreting Media Messages .ABC-CLIO.

[63]

Henry Silverman. 2019. Helping Fact-Checkers Identify False Claims Faster - About Facebook. https://about.fb.com/news/2019/12/helping-fact-checkers/. (Accessed on 01/10/2020).

[64]

Julianne Stanford, Ellen R Tauber, BJ Fogg, and Leslie Marable. 2002. Experts vs. online consumers: A comparative credibility study of health and finance Web sites .Consumer Web Watch.

[65]

Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research, Vol. 17 (1994), 273--85.

[66]

S Shyam Sundar. 1999. Exploring receivers' criteria for perception of print and online news. Journalism & Mass Communication Quarterly, Vol. 76, 2 (1999), 373--386.

[67]

S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility, Vol. 73100 (2008).

[68]

Cass R. Sunstein. 2006. When Crowds Aren't Wise. Harvard Business Review (Sep 2006). https://hbr.org/2006/09/when-crowds-arent-wise

[69]

James Surowiecki. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations .Doubleday.

Digital Library

[70]

Jan-Willem van Prooijen, André P. M. Krouwel, and Thomas V. Pollet. 2015. Political Extremism Predicts Belief in Conspiracy Theories. Social Psychological and Personality Science, Vol. 6, 5 (Jul 2015), 570--578. https://doi.org/10.1177/1948550614567356

[71]

Christian Wagner and Ayoung Suh. 2014. The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan 2014), 594--603. https://doi.org/10.1109/HICSS.2014.80

[72]

Lorraine Whitmarsh. 2011. Scepticism and uncertainty about climate change: Dimensions, determinants and change over time. Global Environmental Change, Vol. 21, 2 (May 2011), 690--700.

[73]

Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science, Vol. 330, 6004 (Oct 2010), 686--688. https://doi.org/10.1126/science.1193147

[74]

Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Jennifer Lee, et almbox. 2018. A structured response to misinformation: Defining and annotating credibility indicators in news articles. In Companion Proceedings of The Web Conference 2018. 603--612.

Digital Library

Cited By

Agarwal DShahid FVashistha A(2024)Conversational Agents to Facilitate Deliberation on Harmful Content in WhatsApp GroupsProceedings of the ACM on Human-Computer Interaction10.1145/36870308:CSCW2(1-32)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687030
Chuai YTian HPröllochs NLenzini G(2024)Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?Proceedings of the ACM on Human-Computer Interaction10.1145/36869678:CSCW2(1-52)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686967
Yaqub WKay JGoldwater M(2024)Foundations for Enabling People to Recognise Misinformation in Social Media News based on Retracted ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36373358:CSCW1(1-38)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637335
Show More Cited By

Index Terms

Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI

Recommendations

Birds of a feather don’t fact-check each other: Partisanship and the evaluation of news in Twitter’s Birdwatch crowdsourced fact-checking program
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

There is a great deal of interest in the role that partisanship, and cross-party animosity in particular, plays in interactions on social media. Most prior research, however, must infer users’ judgments of others’ posts from engagement data. Here, we ...
Crowdsourced Fact-checking: Does It Actually Work?
Abstract
There is an important ongoing effort aimed to tackle misinformation and to perform reliable fact-checking by employing human assessors at scale, with a crowdsourcing-based approach. Previous studies on the feasibility of employing crowdsourcing ...
Highlights
- We extend past studies on crowdsourced fact-checking.
- We employ various methods to assess the reliability of crowd workers.
- We observe improved agreement between crowd workers and experts in studies.
GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 4, Issue CSCW2

CSCW

October 2020

2310 pages

EISSN:2573-0142

DOI:10.1145/3430143

Editor:
Jeff Nichols
Apple Inc.

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2020

Published in PACMHCI Volume 4, Issue CSCW2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
1,363
Total Downloads

Downloads (Last 12 months)484
Downloads (Last 6 weeks)55

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Agarwal DShahid FVashistha A(2024)Conversational Agents to Facilitate Deliberation on Harmful Content in WhatsApp GroupsProceedings of the ACM on Human-Computer Interaction10.1145/36870308:CSCW2(1-32)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3687030
Chuai YTian HPröllochs NLenzini G(2024)Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter?Proceedings of the ACM on Human-Computer Interaction10.1145/36869678:CSCW2(1-52)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686967
Yaqub WKay JGoldwater M(2024)Foundations for Enabling People to Recognise Misinformation in Social Media News based on Retracted ScienceProceedings of the ACM on Human-Computer Interaction10.1145/36373358:CSCW1(1-38)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637335
Heuer HGlassman E(2024)Reliability Criteria for News WebsitesACM Transactions on Computer-Human Interaction10.1145/363514731:2(1-33)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3635147
Hughes EWang RJuneja PLi TMitra TZhang A(2024)Viblio: Introducing Credibility Signals and Citations to Video-Sharing PlatformsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642490(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642490
Wang JMakowski SCieślik ALv HLv Z(2024)Fake News in Virtual Community, Virtual Society, and Metaverse: A SurveyIEEE Transactions on Computational Social Systems10.1109/TCSS.2022.322042011:4(4828-4842)Online publication date: Aug-2024
https://doi.org/10.1109/TCSS.2022.3220420
Bhayyar AKiran P(2024)Analysing Perception of Credibility of a Message in Online Social Network using SmartPLS2024 IEEE 5th India Council International Subsections Conference (INDISCON)10.1109/INDISCON62179.2024.10744321(1-6)Online publication date: 22-Aug-2024
https://doi.org/10.1109/INDISCON62179.2024.10744321
Drolsbach CSolovev KPröllochs N(2024)Community notes increase trust in fact-checking on social mediaPNAS Nexus10.1093/pnasnexus/pgae2173:7Online publication date: 31-May-2024
https://doi.org/10.1093/pnasnexus/pgae217
Soprano MRoitero KLa Barbera DCeolin DSpina DDemartini GMizzaro S(2024)Cognitive Biases in Fact-Checking and Their CountermeasuresInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10367261:3Online publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103672
Salah IJouini KPachon CKorbaa O(2024)Connecting the dots between stance and fake news detection with blockchain, proof of reputation, and the Hoeffding boundCluster Computing10.1007/s10586-024-04637-727:9(13395-13405)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/s10586-024-04637-7
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents