"Are Crowdsourcing Platforms Reliable for Video Game-related Research?" A Case Study on Amazon Mechanical Turk
Pages 56 - 63
Abstract
Video games are becoming increasingly popular in research, and abundant prior work has investigated this domain by means of user studies. However, carrying out user studies whose population encompasses a large and diverse set of participants is challenging. Crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), represent a cost-effective solution to address this problem. Yet, prior efforts scrutinizing the data-quality (unrelated to gaming) collected via AMT raises a concern: is AMT reliable for game studies?
In this paper, we are the first to tackle this question. We carry out three user studies (n=302) through which we evaluate the overall validity of the responses—pertaining to 14 popular video games—we received via AMT. We adopt strict verification mechanisms, which are trivial to “bypass” by real gamers, but costly for non-gamers. We found that the percentage of valid responses ranges from 5% (for WoW) to 28% (for PUBG). We hence advocate future research to carefully scrutinize the validity of responses collected via AMT.
Supplemental Material
MP4 File - 1061#-video/1061#-video.mp4
Short summary of the paper.
- Download
- 7.17 MB
References
[1]
2024. Amazon Mechanical Turk (Wikipedia). https://web.archive.org/web/20240517120039/https://en.wikipedia.org/wiki/Amazon_Mechanical_Turk.
[2]
2024. Crowdsourcing (Wikipedia). https://web.archive.org/web/20240715201145/https://en.wikipedia.org/wiki/Crowdsourcing.
[3]
2024. Repository of this paper. https://github.com/hihey54/chiplay24.
[4]
Tahir Abbas and Ujwal Gadiraju. 2022. Goal-setting behavior of workers on crowdsourcing platforms: An exploratory study on mturk and prolific. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 10. 2–13.
[5]
Hadi Abdullah, Muhammad Sajidur Rahman, Washington Garcia, Kevin Warren, Anurag Swarnim Yadav, Tom Shrimpton, and Patrick Traynor. 2021. Hear" no evil", see" kenansville": Efficient and transferable black-box attacks on speech recognition and voice identification systems. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 712–729.
[6]
Maribel Acosta, Amrapali Zaveri, Elena Simperl, Dimitris Kontokostas, Fabian Flöck, and Jens Lehmann. 2018. Detecting linked data quality issues via crowdsourcing: A dbpedia study. Semantic web 9, 3 (2018), 303–335.
[7]
Jon Agley, Yunyu Xiao, Rachael Nolan, and Lilian Golzarri-Arroyo. 2022. Considerations for conducting alcohol research with the USAUDIT on Mechanical Turk (mTurk). Journal of studies on alcohol and drugs 83, 1 (2022), 159–161.
[8]
Amazon. 2018. MTurk’s Acceptable Use Policy. https://web.archive.org/web/20240712184710/https://www.mturk.com/acceptable-use-policy
[9]
Amazon. 2020. Participation Agreement. https://web.archive.org/web/20240712184709/https://www.mturk.com/participation-agreement
[10]
Michael Bailey, David Dittrich, Erin Kenneally, and Doug Maughan. 2012. The Menlo report. IEEE Security & Privacy (2012).
[11]
Nicole A Beres, Julian Frommel, Elizabeth Reid, Regan L Mandryk, and Madison Klarkowski. 2021. Don’t you know that you’re toxic: Normalization of toxicity in online gaming. In Proc. CHI. 1–15.
[12]
Kelly Bergstrom and Nathaniel Poor. 2021. Reddit gaming communities during times of transition. Social Media+ Society 7, 2 (2021).
[13]
Max V. Birk, Maximilian A. Friehs, and Regan L. Mandryk. 2017. Age-Based Preferences and Player Experience: A Crowdsourced Cross-sectional Study. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. Association for Computing Machinery, 157–170.
[14]
Beate Braun, Juliane M Stopfer, Kai W Müller, Manfred E Beutel, and Boris Egloff. 2016. Personality and video gaming: Comparing regular gamers, non-gamers, and gaming addicts and differentiating between game genres. Computers in Human Behavior (2016).
[15]
Gabriel A. Brooks and Luke Clark. 2019. Associations between loot box use, problematic gaming and gambling, and gambling-related cognitions. Addictive Behaviors 96 (2019), 26–34. https://doi.org/10.1016/j.addbeh.2019.04.009
[16]
Xiaowei Cai, Javier Cebollada, and Mónica Cortiñas. 2022. Self-report measure of dispositional flow experience in the video game context: Conceptualisation and scale development. International Journal of Human-Computer Studies 159 (2022), 102746.
[17]
Isaac Cheah, Anwar Sadat Shimul, and Ian Phau. 2022. Motivations of playing digital games: A review and research agenda. Psychology & Marketing (2022).
[18]
Evgenia Christoforou, Gianluca Demartini, and Jahna Otterbacher. 2024. Generative AI in Crowdwork for Web and Social Media Research: A Survey of Workers at Three Platforms. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 18. 2097–2103.
[19]
European Commission. 2024. Sensitive Data. https://web.archive.org/web/20240204163656/https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/sensitive-data_en.
[20]
Gillian Dale and C Shawn Green. 2017. The changing face of video games and video gamers: Future directions in the scientific study of video game play and cognitive performance. Journal of Cognitive Enhancement 1 (2017), 280–294.
[21]
Robert O Deaner, Lucretia C Dunlap, and April Bleske-Rechek. 2022. Sex Differences in Competitiveness in Massively Multiplayer Online Role-Playing Games (MMORPGs). Evolutionary Psychology 20, 2 (2022), 14747049221109388.
[22]
Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In Proceedings of the eleventh ACM international conference on web search and data mining. 135–143.
[23]
Mary Jo Dondlinger. 2007. Educational video game design: A review of the literature. Journal of applied educational technology 4, 1 (2007), 21–31.
[24]
Benjamin D Douglas, Patrick J Ewell, and Markus Brauer. 2023. Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. Plos one 18, 3 (2023), e0279720.
[25]
Adam Enfroy. 2024. 7 Best Crowdsourcing Platforms of 2024 (Ultimate Guide). https://www.adamenfroy.com/crowdsourcing-platform
[26]
Finding Five. 2020. Tutorial: Mechanical Turk Best Practices. https://web.archive.org/web/20240225025247/https://news.findingfive.com/2020/05/17/mechanical-turk-best-practices/. Accessed: May, 2024.
[27]
GDPR.eu. 2019. What is considered personal data under the EU GDPR?https://web.archive.org/web/20240513154019/https://gdpr.eu/eu-gdpr-personal-data/
[28]
Erica A Giammarco, Travis J Schneider, Julie J Carswell, and William S Knipe. 2015. Video game preferences and their relation to career interests. Personality and Individual Differences 73 (2015), 98–104.
[29]
Edward Goh, Omar Al-Tabbaa, and Zaheer Khan. 2023. Unravelling the complexity of the Video Game Industry: An integrative framework and future research directions. Telematics and Informatics Reports (2023), 100100.
[30]
Isabela Granic, Adam Lobel, and Rutger CME Engels. 2014. The benefits of playing video games.American psychologist 69, 1 (2014), 66.
[31]
David Halbhuber, Niels Henze, and Valentin Schwind. 2021. Increasing player performance and game experience in high latency systems. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY (2021), 1–20.
[32]
Stuart Hallifax, Maximilian Altmeyer, Kristina Kölln, Maria Rauschenberger, and Lennart E Nacke. 2023. From Points to Progression: A Scoping Review of Game Elements in Gamification Research with a Content Analysis of 280 Research Papers. Proceedings of the ACM on Human-Computer Interaction 7, CHI PLAY (2023), 748–768.
[33]
Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. All Those Wasted Hours: On Task Abandonment in Crowdsourcing. In WSDM.
[34]
Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–14.
[35]
Megumi Ichimiya, Hope Muller-Tabanera, Jennifer Cantrell, Jeffrey B Bingenheimer, Raquel Gerard, Elizabeth C Hair, Dante Donati, Nandan Rao, and W Douglas Evans. 2023. Evaluation of response to incentive recruitment strategies in a social media-based survey. Digital Health (2023).
[36]
Wooyoung William Jang and Kevin K Byon. 2020. Antecedents of esports gameplay intention: Genre as a moderator. Comp. Human Behavior (2020).
[37]
Davinder Kaur, Suleyman Uslu, Kaley J Rittichier, and Arjan Durresi. 2022. Trustworthy artificial intelligence: a review. ACM CSUR (2022).
[38]
Linda K Kaye, Rachel Kowert, and Sally Quinn. 2017. The role of social identity and online social capital on psychosocial outcomes in MMO players. Computers in Human Behavior 74 (2017), 215–223.
[39]
Jonathan W Kelly, Lucia A Cherep, Alex F Lim, Taylor Doty, and Stephen B Gilbert. 2021. Who are virtual reality headset owners? a survey and comparison of headset owners and non-owners. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 687–694.
[40]
Daniel L King, Alex MT Russell, Paul H Delfabbro, and Dean Polisena. 2020. Fortnite microtransaction spending was associated with peers’ purchasing behaviors but not gaming disorder symptoms. Addictive Behaviors 104 (2020), 106311.
[41]
Rachel Kowert, Alexi Martel, and William B Swann. 2022. Not just a game: Identity fusion and extremism in gaming cultures. Frontiers in Communication 7 (2022), 1007128.
[42]
Rachel Kowert and Julian A Oldmeadow. 2013. (A) Social reputation: Exploring the relationship between online video game involvement and social competence. Computers in Human Behavior 29, 4 (2013), 1872–1878.
[43]
Andrey Krekhov, Sebastian Cmentowski, and Jens Krüger. 2019. The illusion of animal body ownership and its potential for virtual reality games. In 2019 IEEE Conference on Games (CoG). IEEE, 1–8.
[44]
Jacob Leon Kröger, Philip Raschke, Jessica Percy Campbell, and Stefan Ullrich. 2023. Surveilling the gamers: Privacy impacts of the video game industry. Entertainment Computing 44 (2023), 100537.
[45]
Chanel J Larche, Katrina Chini, Christopher Lee, and Mike J Dixon. 2023. To pay or just play? Examining individual differences between purchasers and earners of loot boxes in overwatch. Journal of Gambling Studies 39, 2 (2023), 625–643.
[46]
Pascal Lessel, Maximilian Altmeyer, Julian Sahner, and Antonio Krüger. 2022. Streamer’s Hell-Investigating Audience Influence in Live-Streams Beyond the Game. Proceedings of the ACM on Human-Computer Interaction 6, CHI PLAY (2022), 1–27.
[47]
Wen Li Anthony, Devin J Mills, Jackie F Stanmyre, and Lia Nower. 2023. Facets of mindfulness among video game players: A latent profile analysis.Psychology of Addictive Behaviors (2023).
[48]
Jian Lu, Wei Li, Qingren Wang, and Yiwen Zhang. 2020. Research on data quality control of crowdsourcing annotation: A survey. In IEEE DASC/PiCom/CBDCom/CyberSciTech.
[49]
Alexandra Sasha Luccioni and David Rolnick. 2023. Bugs in the data: How ImageNet misrepresents biodiversity. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 14382–14390.
[50]
Catherine C Marshall, Partha SR Goguladinne, Mudit Maheshwari, Apoorva Sathe, and Frank M Shipman. 2023. Who broke Amazon Mechanical Turk? An analysis of crowdsourcing data quality over time. In Proceedings of the 15th ACM Web Science Conference 2023. 335–345.
[51]
Tara McAllister Byun, Peter F. Halpin, and Daniel Szeredi. 2015. Online crowdsourcing for efficient rating of speech: A validation study. Journal of Communication Disorders (2015).
[52]
Francesco Neri, Carmelo Luca Smeralda, Davide Momi, Giulia Sprugnoli, Arianna Menardi, Salvatore Ferrone, Simone Rossi, Alessandro Rossi, Giorgio Di Lorenzo, and Emiliano Santarnecchi. 2021. Personalized adaptive training improves performance at a professional first-person shooter action videogame. Frontiers in Psychology 12 (2021), 598410.
[53]
US Department of the Treasury. 2023. Sensitive Personal Data. https://web.archive.org/web/20231130015633/https://home.treasury.gov/taxonomy/term/7651.
[54]
College of Wooster Human Subject Research Committee. 2023. Guidance for Using MTurk. https://web.archive.org/web/20230329032500/https://inside.wooster.edu/hsrc/guidance-for-using-mturk/
[55]
Vincent Huard Pelletier, Arianne Lessard, Florence Piché, Charles Tétreau, and Martin Descarreaux. 2020. Video games and their associations with physical health: A scoping review. BMJ open sport & exercise medicine 6, 1 (2020), e000832.
[56]
Cristiano Politowski, Fabio Petrillo, and Yann-Gaël Guéhéneuc. 2021. A survey of video game testing. In IEEE/ACM AST.
[57]
Prolific. 2024. What is your pricing?researcher-help.prolific.com/hc/en-gb/articles/360009223533-What-is-your-pricing. Accessed in May 2024.
[58]
Lisa Raith, Julie Bignill, Vasileios Stavropoulos, Prudence Millear, Andrew Allen, Helen M Stallman, Jonathan Mason, Tamara De Regt, Andrew Wood, and Lee Kannis-Dymand. 2021. Massively multiplayer online games and well-being: A systematic literature review. Frontiers in Psychology 12 (2021), 698799.
[59]
Charles Reynaldo, Ryan Christian, Hansel Hosea, and Alexander AS Gunawan. 2021. Using video games to improve capabilities in decision making and cognitive skill: a literature review. Procedia Computer Science 179 (2021), 211–221.
[60]
Jonathan Robinson, Cheskie Rosenzweig, Aaron J Moss, and Leib Litman. 2019. Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PloS one 14, 12 (2019), e0226394.
[61]
Antonios Saravanos, Stavros Zervoudakis, Dongnanzi Zheng, Neil Stott, Bohdan Hawryluk, and Donatella Delfino. 2021. The hidden cost of using Amazon Mechanical Turk for research. In HCI.
[62]
Johannes Schneider and Giovanni Apruzzese. 2023. Dual adversarial attacks: Fooling humans and classifiers. JISA (2023).
[63]
Peter G Schrader, Mark C Carroll, Michael P McCreery, and Danielle L Head. 2020. Mixed methods for human–computer interactions research: An iterative study using Reddit and social media. Journal of Educational Computing Research 58, 4 (2020), 818–841.
[64]
David Sharek and Eric Wiebe. 2014. Measuring video game engagement through the cognitive and affective dimensions. Simulation & Gaming 45, 4-5 (2014), 569–592.
[65]
Zhiguo Shi, Guang Yang, Xiaowen Gong, Shibo He, and Jiming Chen. 2021. Quality-aware incentive mechanisms under social influences in data crowdsourcing. IEEE/ACM Transactions on Networking 30, 1 (2021), 176–189.
[66]
Pier Paolo Tricomi, Lisa Facciolo, Giovanni Apruzzese, and Mauro Conti. 2023. Attribute inference attacks in online multiplayer video games: A case study on Dota2. In Proceedings of the Thirteenth ACM Conference on Data and Application Security and Privacy. 27–38.
[67]
Kellie Vella, Daniel Johnson, Vanessa Wan Sze Cheng, Tracey Davenport, Jo Mitchell, Madison Klarkowski, and Cody Phillips. 2019. A sense of belonging: Pokémon GO and social connectedness. Games and Culture 14, 6 (2019), 583–603.
[68]
Zhao Wang, Anna Sapienza, Aron Culotta, and Emilio Ferrara. 2019. Personality and behavior in role-based online games. In IEEE CoG.
[69]
Kerri Wazny. 2017. “Crowdsourcing” ten years in: A review. Journal of global health 7, 2 (2017).
[70]
Donghee Yvette Wohn, Rabindra Ratan, and Leticia Cherchiglia. 2020. Gender and Genre Differences in Multiplayer Gaming Motivations. In HCI in Games.
[71]
Lin Zhu. 2021. The psychology behind video games during COVID-19 pandemic: A case study of Animal Crossing: New Horizons. Human Behavior and Emerging Technologies 3, 1 (2021), 157–159.
Index Terms
- "Are Crowdsourcing Platforms Reliable for Video Game-related Research?" A Case Study on Amazon Mechanical Turk
Recommendations
Evaluating the accessibility of crowdsourcing tasks on Amazon's mechanical turk
ASSETS '14: Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibilityCrowd work web sites such as Amazon Mechanical Turk enable individuals to work from home, which may be useful for people with disabilities. However, the web sites for finding and performing crowd work tasks must be accessible if people with disabilities ...
Investigating the Accessibility of Crowdwork Tasks on Mechanical Turk
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsCrowdwork can enable invaluable opportunities for people with disabilities, not least the work flexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how engagement in crowdwork tasks is ...
Who Broke Amazon Mechanical Turk?: An Analysis of Crowdsourcing Data Quality over Time
WebSci '23: Proceedings of the 15th ACM Web Science Conference 2023We present the results of a survey fielded in June of 2022 as a lens to examine recent data reliability issues on Amazon Mechanical Turk. We contrast bad data from this survey with bad data from the same survey fielded among US workers in October 2013, ...
Comments
Information & Contributors
Information
Published In
October 2024
500 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Hilti
Conference
CHI PLAY '24
Sponsor:
CHI PLAY '24: The Annual Symposium on Computer-Human Interaction in Play
October 14 - 17, 2024
Tampere, Finland
Acceptance Rates
Overall Acceptance Rate 421 of 1,386 submissions, 30%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 57Total Downloads
- Downloads (Last 12 months)57
- Downloads (Last 6 weeks)10
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format