Abstract
Amazon Mechanical Turk, an online marketplace designed for crowdsourcing tasks to other people for compensation, is growing in popularity as a platform for gathering research data within the social sciences. Sociology, compared to some other social sciences, has not been as quick to adopt this form of data collection. Therefore, in this paper I overview the basics of Mechanical Turk research and suggest its pros and cons, both in general and in relation to different sociological data-collection methods and research needs. While Mechanical Turk is currently the most popular crowdsourcing website for research, I present general concepts, patterns, and suggestions that can be applied beyond Mechanical Turk to other crowdsourcing and online research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Some others include ProlificAcademic, SocialSci, CrowdFlower, ClickWorker, CrowdSource. Some websites are specific to crowdsourcing particular types of tasks (e.g., graphic design) while others are general project markets like mturk.
Mturk’s motto is “Artificial Artificial Intelligence.”
I searched in mid to late 2014 within any journals classified as (a) in the top 50 sociology journal by ISI Impact Factor, (b) an American Sociological Association official journal, (c) a regional US sociology association official journal, or (d) listed as a common sociology journal on several websites with such lists. Search criteria included using the words “mturk” or “mechanical turk.”
The fifteen research articles were in Social Psychology Quarterly (1 count), Social Forces (2), Journal for the Scientific Study of Religion (2), Work and Occupations (1), International Journal of Intercultural Relations (1), Social Science Research (4), and American Behavioral Scientist (4). Additionally, four articles in Contexts (1), Politics and Society (1), Social Science Research (1), and American Behavioral Scientist (1) mentioned mturk, but did not use data collected from it. For a comparison, psychology’s Journal of Personality and Social Psychology has 68 articles that at least mention mturk.
References
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: amazon.com’s mechanical Turk. Political Analysis, 20(3), 351–368. doi:10.1093/pan/mpr057.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.
Chandler, D., & Kapelner, A. (2013). Breaking monotony with meaning: motivation in crowdsourcing markets. Journal of Economic Behavior & Organization, 90, 123–133. doi:10.1016/j.jebo.2013.03.003.
Crump, M. J., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research. PloS One, 8(3), e57410.
Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5(3), 159–163.
Hart, J. (2014). Did hurricane sandy influence the 2012 US presidential election? Social Science Research, 46, 1–8.
Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425.
Hunzaker, M. B. F. (2014). Making sense of misfortune: cultural schemas, victim redefinition, and the perpetuation of stereotypes. Social Psychology Quarterly, 77(2), 166–184. doi:10.1177/0190272514521219.
Johnson, M. K., Labouff, J. P., Rowatt, W. C., Patock‐Peckham, J. A., & Carlisle, R. D. (2012). Facets of right-wing authoritarianism mediate the relationship between religious fundamentalism and attitudes toward Arabs and African americans. Journal for the Scientific Study of Religion, 51(1), 128–142.
Kuwabara, K., & Sheldon, O. (2012). Temporal dynamics of social exchange and the development of solidarity:“testing the waters” versus “taking a leap of faith”. Social Forces, 91(1), 253–273.
Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s mechanical Turk. Behavior Research Methods, 44(1), 1–23.
Munsch, C. L., Ridgeway, C. L., & Williams, J. C. (2014). Pluralistic ignorance and the flexibility bias: understanding and mitigating flextime and flexplace bias at work. Work and Occupations, 41(1), 40–62.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk. Judgment and Decision Making, 5(5), 411–419.
Rand, D. G. (2012). The promise of mechanical Turk: how online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172–179. doi:10.1016/j.jtbi.2011.03.004.
Ritter, R. S., & Preston, J. L. (2013). Representations of religious words: insights for religious priming research. Journal for the Scientific Study of Religion, 52(3), 494–507.
Shepherd, H. (2012). Crowdsourcing. Contexts, 11(2), 10–11.
Simpson, B., Harrell, A., & Willer, R. (2013). Hidden paths from morality to cooperation: moral judgments promote trust and trustworthiness. Social Forces, 91(4), 1529–1548.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shank, D.B. Using Crowdsourcing Websites for Sociological Research: The Case of Amazon Mechanical Turk. Am Soc 47, 47–55 (2016). https://doi.org/10.1007/s12108-015-9266-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12108-015-9266-9