Assessing Credibility Factors of Short-Form Social Media Posts: A Crowdsourced Online Experiment
Authors: 
Junhao Li, Miikka Kuutila, Eetu Huusko, Nimantha Kariyakarawana, Marko Savic, Nazanin Nakhaie Ahooie, Simo Hosio, Mika MäntyläAuthors Info & Claims
Junhao Li, Miikka Kuutila, Eetu Huusko, Nimantha Kariyakarawana, Marko Savic, Nazanin Nakhaie Ahooie, Simo Hosio, Mika MäntyläAuthors Info & ClaimsArticle No.: 9, Pages 1 - 14
Abstract
People commonly turn to the Internet and social media for their information needs. Most popular social media platforms focus on short-form content that can be consumed rapidly. Given how fast such content spreads online, its trustworthiness and credibility have become important research areas. We investigate how different factors of social media posts influence their perceived credibility. We generated health-themed short-form social media posts, varied specific aspects of those posts, and deployed the variations on three different online crowdsourcing platforms for credibility assessment. Our quantitative data analysis reveals, for instance, how author professions related to healthcare and science increase the perceived credibility of health-themed posts. Moreover, a higher number of likes and shares increased the credibility in two out of the three platforms. Our qualitative results based on questionnaires highlight personal filtering strategies and critical thinking skills as factors that influence post credibility online. Consequently, our results encourage experts to provide information on social media and to be part of correcting any misinformation as they have higher credibility. Our work strengthens the previous body of work on the credibility of online content in general and acts as a starting point for further studies on social media post content by demonstrating a systematic, crowdsourced, and scalable approach.
References
[1]
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives 31, 2 (2017), 211–36.
[2]
Majed Alrubaian, Muhammad Al-Qurishi, Atif Alamri, Mabrook Al-Rakhami, Mohammad Mehedi Hassan, and Giancarlo Fortino. 2018. Credibility in online social networks: A survey. IEEE Access 7 (2018), 2828–2855.
[3]
Cory L Armstrong and Melinda J McAdams. 2009. Blogs of information: How gender cues and individual motivations influence perceptions of credibility. Journal of Computer-Mediated Communication 14, 3 (2009), 435–456.
[4]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
[5]
Pew Research Center. 2015. Pew Research Center Demographic Questions Web or Mail Mode 12-29-2015. https://assets.pewresearch.org/wp-content/uploads/sites/12/2015/03/Demographic-Questions-Web-and-Mail-English-3-20-2015.pdf, Accessed: 13-8-2021.
[6]
Nyamragchaa Chimedtseren, Bridget Kelly, Anne-Therese McMahon, and Heather Yeatman. 2020. Prevalence and Credibility of Nutrition and Health Claims: Policy Implications from a Case Study of Mongolian Food Labels. International Journal of Environmental Research and Public Health 17, 20 (2020), 7456.
[7]
Roxanne Connelly, Vernon Gayle, and Paul S Lambert. 2016. Ethnicity and ethnic group measures in social survey research. Methodological Innovations 9 (2016), 2059799116642885.
[8]
Judith Donath and Danah Boyd. 2004. Public displays of connection. BT technology Journal 22, 4 (2004), 71–82.
[9]
Alexey Drutsa, Valentina Fedorova, Dmitry Ustalov, Olga Megorskaya, Evfrosiniya Zerminova, and Daria Baidakova. 2020. Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing. In Proceedings of the 13th International Conference on Web Search and Data Mining. 873–876.
[10]
Axel G Ekström, Diederick C Niehorster, and Erik J Olsson. 2022. Self-imposed filter bubbles: Selective attention and exposure in online search. Computers in Human Behavior Reports 7 (2022), 100226.
[11]
Nicole B Ellison, Charles Steinfield, and Cliff Lampe. 2007. The benefits of Facebook “friends:” Social capital and college students’ use of online social network sites. Journal of computer-mediated communication 12, 4 (2007), 1143–1168.
[12]
Nicole B Ellison, Charles Steinfield, and Cliff Lampe. 2011. Connection strategies: Social capital implications of Facebook-enabled communication practices. New media & society 13, 6 (2011), 873–892.
[13]
Nicole B Ellison, Jessica Vitak, Rebecca Gray, and Cliff Lampe. 2014. Cultivating social resources on social network sites: Facebook relationship maintenance behaviors and their role in social capital processes. Journal of Computer-Mediated Communication 19, 4 (2014), 855–870.
[14]
Anna Fanoberova and Hanna Kuczkowska. 2016. Effects of source credibility and information quality on attitudes and purchase intentions of apparel products: A quantitative study of online shopping among consumers in Sweden.
[15]
Claudia Flores-Saviaga and Saiph Savage. 2021. Fighting disaster misinformation in Latin America: the# 19S Mexican earthquake case study. Personal and Ubiquitous Computing 25, 2 (2021), 353–373.
[16]
Brian J Fogg, Cathy Soohoo, David R Danielson, Leslie Marable, Julianne Stanford, and Ellen R Tauber. 2003. How do users evaluate the credibility of Web sites? A study with over 2,500 participants. In Proceedings of the 2003 conference on Designing for user experiences. 1–15.
[17]
Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1631–1640.
[18]
Eun Go, Eun Hwa Jung, and Mu Wu. 2014. The effects of source cues on online news perception. Computers in Human Behavior 38 (2014), 358–367.
[19]
Angela R Gover, Shannon B Harper, and Lynn Langton. 2020. Anti-Asian hate crime during the COVID-19 pandemic: Exploring the reproduction of inequality. American journal of criminal justice 45 (2020), 647–667.
[20]
Melissa A Hardy. 1993. Regression with dummy variables. Vol. 93. Sage.
[21]
E. K. Hämäläinen, C. Kiili, M. Marttunen, E. Rikknen, and Pht Leppänen. 2020. Promoting sixth graders’ credibility Evaluation of Web pages: An intervention study. Computers in Human Behavior (2020), 106372.
[22]
Thomas J Johnson and Barbara K Kaye. 1998. Cruising is believing?: Comparing Internet and traditional sources on media credibility measures. Journalism & Mass Communication Quarterly 75, 2 (1998), 325–340.
[23]
Thomas J Johnson and Barbara K Kaye. 2016. Some like it lots: The influence of interactivity and reliance on credibility. Computers in Human Behavior 61 (2016), 136–145.
[24]
Nicholas A Jones and Michael Bentley. 2017. Overview of the 2015 national content test analysis report on race & ethnicity. US Census Bureau, Suitland-Silver Hill, MD (2017).
[25]
Minjeong Kang. 2010. Measuring social media credibility: A study on a measure of blog credibility. Institute for Public Relations (2010), 59–68.
[26]
Jinyoung Kim and Andrew Gambino. 2016. Do we trust the crowd or information system? Effects of personalization and bandwagon cues on users’ attitudes and behavioral intentions toward a restaurant recommendation website. Computers in Human Behavior 65 (2016), 369–379.
[27]
Lars König and Regina Jucks. 2019. Hot topics in science communication: Aggressive language decreases trustworthiness and credibility in scientific debates. Public understanding of science 28, 4 (2019), 401–416.
[28]
Lars König and Regina Jucks. 2020. Effects of Positive Language and Profession on Trustworthiness and Credibility in Online Health Advice: Experimental Study. Journal of medical Internet research 22, 3 (2020).
[29]
Lars König, Regina Jucks, 2019. Influence of enthusiastic language on the credibility of health information and the trustworthiness of science communicators: Insights from a between-subject web-based experiment. Interactive Journal of Medical Research 8, 3 (2019), e13619.
[30]
Junhao Li, Ville Paananen, Sharadhi Alape Suryanarayana, Eetu Huusko, Miikka Kuutila, Mika Mäntylä, and Simo Hosio. 2023. It is an online platform and not the real world, I don’t care much: Investigating Twitter Profile Credibility With an Online Machine Learning-Based Tool. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval. 117–127.
[31]
Isabelle Lid Rosenholm. 2022. Health Communication on TikTok: A Qualitative Study of Credibility on A Humorous Platform.
[32]
Mufan Luo, Jeffrey T Hancock, and David M Markowitz. 2022. Credibility perceptions and detection accuracy of fake news headlines on social media: Effects of truth-bias and endorsement cues. Communication Research 49, 2 (2022), 171–195.
[33]
Debbie S Ma, Joshua Correll, and Bernd Wittenbrink. 2015. The Chicago face database: A free stimulus set of faces and norming data. Behavior research methods 47, 4 (2015), 1122–1135.
[34]
Mark MacCarthy. 2020. Transparency requirements for digital social media platforms: Recommendations for policy makers and industry. Transatlantic Working Group (2020).
[35]
Scott R Maier, Marcus Mayorga, and Paul Slovic. 2017. Personalized news stories affect men as well as women. Newspaper Research Journal 38, 2 (2017), 172–186.
[36]
Scott R Maier, Paul Slovic, and Marcus Mayorga. 2017. Reader reaction to news of mass suffering: Assessing the influence of story form and emotional response. Journalism 18, 8 (2017), 1011–1029.
[37]
Kelly Mathews, Jessica Phelan, Nicholas A Jones, Sarah Konya, Rachel Marks, Beverly M Pratt, Julia Coombs, and Michael Bentley. 2015. National Content Test: Race and ethnicity analysis report. US Department of Commerce, Economics and Statistics Administration, US Census Bureau (2015).
[38]
Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23.
[39]
Davood Mehrabi, Musa Abu Hassan, and Muhamad Sham Shahkat Ali. 2009. News media credibility of the Internet and television. European journal of social sciences 11, 1 (2009), 136–148.
[40]
Miriam J Metzger and Andrew J Flanagin. 2015. Psychological approaches to credibility assessment online. The handbook of the psychology of communication technology 32 (2015), 445–466.
[41]
Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020. Creativity on paid crowdsourcing platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
[42]
Zizi Papacharissi. 2009. The virtual geographies of social networks: a comparative analysis of Facebook, LinkedIn and ASmallWorld. New media & society 11, 1-2 (2009), 199–220.
[43]
Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153–163.
[44]
Richard E Petty, John T Cacioppo, and Rachel Goldman. 1981. Personal involvement as a determinant of argument-based persuasion.Journal of personality and social psychology 41, 5 (1981), 847.
[45]
Kevin Roitero, Michael Soprano, Beatrice Portelli, Massimiliano De Luise, Damiano Spina, Vincenzo Della Mea, Giuseppe Serra, Stefano Mizzaro, and Gianluca Demartini. 2021. Can the crowd judge truthfulness? A longitudinal study on recent misinformation about COVID-19. Personal and Ubiquitous Computing (2021), 1–31.
[46]
Madison Elizabeth Sauls. 2018. Perceived Credibility of Information on Internet Health Forums. Ph. D. Dissertation. Clemson University.
[47]
Laura Sbaffi and Jennifer Rowley. 2017. Trust and credibility in web-based health information: a review and agenda for future research. Journal of medical Internet research 19, 6 (2017), e218.
[48]
Charles C Self. 2014. Credibility. In An integrated approach to communication theory and research. Routledge, 449–470.
[49]
Patric R Spence, Kenneth A Lachlan, David Westerman, and Stephen A Spates. 2013. Where the gates matter less: Ethnicity and perceived source credibility in social media health messages. Howard Journal of Communications 24, 1 (2013), 1–16.
[50]
David Sterrett, Dan Malato, Jennifer Benz, Liz Kantor, Trevor Tompson, Tom Rosenstiel, Jeff Sonderman, and Kevin Loker. 2019. Who shared it?: Deciding what news to trust on social media. Digital Journalism 7, 6 (2019), 783–801.
[51]
C Strijbos, M Schluck, J Bisschop, T Bui, I De Jong, M Van Leeuwen, M von Tottleben, and SG van Breda. 2016. Consumer awareness and credibility factors of health claims on innovative meat products in a cross-sectional population study in the Netherlands. Food Quality and Preference 54 (2016), 13–22.
[52]
Cass R Sunstein. 2006. Infotopia: How many minds produce knowledge. Oxford University Press.
[53]
Kyle A Thomas and Scott Clifford. 2017. Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior 77 (2017), 184–197.
[54]
Sonja Utz. 2015. The function of self-disclosure on social network sites: Not only intimate, but also positive and entertaining self-disclosures increase the feeling of connection. Computers in Human Behavior 45 (2015), 1–10.
[55]
Hans CM Van Trijp and Ivo A Van der Lans. 2007. Consumer perceptions of nutrition and health claims. Appetite 48, 3 (2007), 305–324.
[56]
Amy Beth Warriner, Victor Kuperman, and Marc Brysbaert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior research methods 45, 4 (2013), 1191–1207.
[57]
Senuri Wijenayake, Danula Hettiachchi, Simo Johannes Hosio, Vassilis Kostakos, and Jorge Goncalves. 2020. Effect of Conformity on Perceived Trustworthiness of News in Social Media. IEEE Internet Computing (2020).
[58]
Dmitri Williams. 2006. On and off the’Net: Scales for social capital in an online era. Journal of computer-mediated communication 11, 2 (2006), 593–628.
[59]
Anja Wölker and Thomas E Powell. 2018. Algorithms in the newsroom? News readers’ perceived credibility and selection of automated journalism. Journalism (2018), 1464884918757072.
[60]
Min Xiao, Rang Wang, and Sylvia Chan-Olmsted. 2018. Factors affecting YouTube influencer marketing credibility: a heuristic-systematic model. Journal of Media Business Studies 15 (07 2018), 1–26. https://doi.org/10.1080/16522354.2018.1501146
[61]
Jia Zhou, Honglian Xiang, and Bingjun Xie. 2022. Better safe than sorry: a study on older adults’ credibility judgments and spreading of health misinformation. Universal Access in the Information Society (2022), 1–10.
Index Terms
- Assessing Credibility Factors of Short-Form Social Media Posts: A Crowdsourced Online Experiment
Recommendations
NudgeCred: Supporting News Credibility Assessment on Social Media Through Nudges
CSCW2Struggling to curb misinformation, social media platforms are experimenting with design interventions to enhance consumption of credible news on their platforms. Some of these interventions, such as the use of warning messages, are examples of nudges---a ...
Parasocial relationship via reality TV and social media: its implications for celebrity endorsement
TVX '14: Proceedings of the ACM International Conference on Interactive Experiences for TV and Online VideoThe purpose of this study was to explore the ways in which audiences build parasocial relationships with media characters via reality TV and social media, and its implications for celebrity endorsement and purchase intentions. Using an online survey, ...
Social media self-efficacy and information evaluation online
We introduce the concept of social media self-efficacy (SMSE).We examine this concept using survey data (N=3568) of Internet users.Users with higher SMSE find online social information to be more trustworthy.They also use others' opinions and social ...
Comments
Information & Contributors
Information
Published In
September 2023
416 pages
ISBN:9798400708060
DOI:10.1145/3605390
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 September 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Academy of Finland, Strategic Research Council, CRITICAL
Conference
CHItaly 2023
CHItaly 2023: 15th Biannual Conference of the Italian SIGCHI Chapter
September 20 - 22, 2023
Torino, Italy
Acceptance Rates
Overall Acceptance Rate 109 of 242 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 474Total Downloads
- Downloads (Last 12 months)474
- Downloads (Last 6 weeks)106
Reflects downloads up to 26 Jul 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in