research-article

Curiosity Killed the Cat, but Makes Crowdwork Better

Authors:

Michael A. Terry, and

Krzysztof Z. GajosAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

Pages 4098 - 4110

https://doi.org/10.1145/2858036.2858144

Published: 07 May 2016 Publication History

Abstract

Crowdsourcing systems are designed to elicit help from humans to accomplish tasks that are still difficult for computers. How to motivate workers to stay longer and/or perform better in crowdsourcing systems is a critical question for designers. Previous work have explored different motivational frameworks, both extrinsic and intrinsic. In this work, we examine the potential for curiosity as a new type of intrinsic motivational driver to incentivize crowd workers. We design crowdsourcing task interfaces that explicitly incorporate mechanisms to induce curiosity and conduct a set of experiments on Amazon's Mechanical Turk. Our experiment results show that curiosity interventions improve worker retention without degrading performance, and the magnitude of the effects are influenced by both personal characteristics of the worker and the nature of the task.

References

[1]

Anderson, S. Applying curiosity to interaction design: Tell me something I don't know, 2009. Last Retrieved January 7, 2016 from http://johnnyholland.org/2009/08/ curiosity-and-interaction-design/.

[2]

Ariely, D., Kamenica, E., and Prelec, D. Man's search for meaning: The case of legos. Journal of Economic Behaviour and Organization 67, 3 (2008), 671-677.

[3]

Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., and Kraut, R. E. Using social psychology to motivate contributions to online communities. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, CSCW '04, ACM (NY, NY, USA, 2004), 212-221.

Digital Library

[4]

Berlyne, D. An experimental study of human curiosity. British Journal of Psychology 45 (1954), 256-265.

[5]

Berlyne, D. A theory of human curiosity. British Journal of Psychology 45 (1954), 180-191.

[6]

Berlyne, D. Conflict, arousal and curiosity. McGraw-Hill, London, 1960.

[7]

Bowser, A., Hansen, D., He, Y., Boston, C., Reid, M., Gunnell, L., and Preece, J. Using gamification to inspire new citizen science volunteers. In Proceedings of the First International Conference on Gameful Design, Research, and Applications, Gamification '13, ACM (NY, NY, USA, 2013), 18-25.

Digital Library

[8]

Caplin, A., and Leahy, J. Psychological expected utility theory and anticipatory feelings. Quarterly Journal of Economics (2001), 55-79.

[9]

Chandler, D., and Kapelner, A. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behaviour and Organization 90 (2013), 123-133.

[10]

Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., and Players, F. Predicting protein structures with a multiplayer online game. Nature 466 (August 2010), 756-760.

[11]

Dai, P., Rzeszotarski, J. M., Paritosh, P., and Chi, E. H. And now for something completely different: Improving crowdsourcing workflows with micro-diversions. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work, CSCW '15, ACM (NY, NY, USA, 2015), 628-638.

Digital Library

[12]

Easley, D., and Ghosh, A. Incentives, gamification, and game theory: An economic approach to badge design. In Proceedings of the Fourteenth ACM Conference on Electronic Commerce, EC '13, ACM (NY, NY, USA, 2013), 359-376.

Digital Library

[13]

Festinger, L. A theory of social comparison processes. Human Relations 7 (1954), 117-140.

[14]

Feyisetan, O., Simperl, E., Van Kleek, M., and Shadbolt, N. Improving paid microtasks through gamification and adaptive furtherance incentives. In Proceedings of the 24th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee (2015), 333-343.

Digital Library

[15]

Frank Bretz, T. H., and Westfall, P. Multiple Comparisons Using R. Chapman & Hall/CRC, 2011.

[16]

Gaver, W. W., Beaver, J., and Benford, S. Ambiguity as a resource for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '03, ACM (NY, NY, USA, 2003), 233-240.

Digital Library

[17]

Gollub, L. Information on conditioned reinforcement. Journal of the Experimental Analysis of Behaviour (2001), 361-372.

[18]

Grevet, C., Mankoff, J., and Anderson, S. D. Design and evaluation of a social visualization aimed at encouraging sustainable behavior. In Proceedings of the 2010 43rd Hawaii International Conference on System Sciences, HICSS '10, IEEE Computer Society (Washington, DC, USA, 2010), 1-8.

Digital Library

[19]

Harris, C. You're hired! an examination of crowdsourcing incentive models in human resource tasks. In ACM International Conference on Web Search and Data Mining, WSDM '11 (2011), 15-18.

[20]

Ho, C.-J., Slivkins, A., Suri, S., and Vaughan, J. W. Incentivizing high quality crowdwork. In Proceedings of the 24th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee (2015), 419-429.

Digital Library

[21]

Horton, J. J., and Chilton, L. B. The labor economics of paid crowdsourcing. In Proceedings of the 11th ACM Conference on Electronic Commerce, EC '10, ACM (NY, NY, USA, 2010), 209-218.

Digital Library

[22]

Hosmer, D. W., and Lemeshow, S. Goodness of fit tests for the multiple logistic regression model. Communications in Statistics: Theory and Methods 9, 10 (1980), 1043 - 1069.

[23]

Huang, S.-W., and Fu, W.-T. Don't hide in the crowd!: Increasing social transparency between peer workers improves crowdsourcing outcomes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13, ACM (NY, NY, USA, 2013), 621-630.

Digital Library

[24]

Jones, S. Curiosity and knowledge. Psychological Reports 45 (1979), 639-642.

[25]

Kang, M., Hsu, M., Krajbich, I., Loewenstein, G., McClure, S., Wang, J., and Camerer, C. The wick in the candle of learning: epistemic curiosity activates reward circuitry and enhances memory. Psychological Science 20, 8 (2009), 963-973.

[26]

Kinnaird, P., Dabbish, L., Kiesler, S., and Faste, H. Co-worker transparency in a microtask marketplace. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW '13, ACM (NY, NY, USA, 2013), 1285-1290.

Digital Library

[27]

Koffka, K. Principles of Gestalt Psychology. Harcourt-Brace, NY, 1935.

[28]

Kruskal, W., and Wallis, W. A. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 47, 260 (1952), 583-621.

[29]

Langer, K. Suspenseful Design: Engaging Emotionally with Complex Applications through Compelling Narratives. Master's thesis, University of Waterloo, Waterloo, Ontario, 2014.

[30]

Law, E., and von Ahn, L. Human Computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan and Claypool, June 2011.

Digital Library

[31]

Litman, J. Curiosity and the pleasures of learning: Wanting and liking new information. Cognition and Emotion 19, 6 (2005), 793-814.

[32]

Litman, J., Hutchins, T., and Russon, R. Epistemic curiosity, feeling-of-knowing, and exploratory behavior. Cognition and Emotion 19, 4 (2005), 559-582.

[33]

Locke, E., and Latham, G. Building a practically useful theory of goal setting and task motivation: A 35 year odyssey. American Psychologist 57, 9 (2002), 705-717.

[34]

Loewenstein, G. The psychology of curiosity: A review and reinterpretation. Cognition and Emotion 19, 6 (2005), 793-814.

[35]

Malone, T. Toward a theory of intrinsically motivating instruction. Cognitive Science 4 (1981), 333-369.

[36]

Markey, A., and Lowenstein, G. Curiosity. In International Handbook of Emotions in Eduction, R. Pekrun and L. Linnenbrink-Garcia, Eds. Rutledge, NY, 2014.

[37]

Marlow, J., and Dabbish, L. A. The effects of visualizing activity history on attitudes and behaviors in a peer production context. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work, CSCW '15, ACM (NY, NY, USA, 2015), 757-764.

Digital Library

[38]

Mason, W., and Watts, D. Financial incentives and the "performance of crowd". In ACM SIGKDD Workshop on Human Computation (HCOMP) (2009), 77--85.

Digital Library

[39]

McCullagh, P., and Nelder, J. Generalized Linear Modesl, 2 ed. Chapman & Hall/CRC, 1989.

[40]

Menon, S., and Soman, D. Managing the power of curiosity for effective web advertising strategies. Journal of Advertising 31, 3 (2002), 1-14.

[41]

Metzger, W. Laws of Seeing. MIT Press, Cambridge, MA, 2006.

[42]

Miller, N. Liberalization of basic s-r concepts: Extensions to conflict behavior, motivation and social learning. Psychology: the study of a science (1959), 196-292.

[43]

Naylor, F. A state-trait curiosity inventory. Australian Psychology 16 (1981), 172-183.

[44]

Osius, G., and Rojek, D. Normal goodness-of-fit tests for multinomial models with large degrees of freedom. Journal of the American Statistical Association 87, 140 (1992), 1145 - 1152.

[45]

Pluck, G., and Johnson, H. Stimulating curiosity to enhance learning. GESJ: Education Sciences and Psychology 2, 19 (2011), 1512-1801.

[46]

Raddick, M., Bracey, G., Gay, P., Lintott, C., Cardamone, C., Murray, P., Schawinski, K., Szalay, A., and Vandenberg, J. Galaxy zoo: Motivations of citizen scientists. Astronomy Education Review 12, 1 (2013).

[47]

Rashid, A. M., Ling, K., Tassone, R. D., Resnick, P., Kraut, R., and Riedl, J. Motivating participation by displaying the value of contribution. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '06, ACM (NY, NY, USA, 2006), 955-958.

Digital Library

[48]

Rogstadius, J., Kostakos, V., Kittur, A., Smus, B., Laredo, J., and Vukovic, M. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In AAAI Conference on Weblogs and Social Media (2011).

[49]

Rotman, D., Preece, J., Hammock, J., Procita, K., Hansen, D., Parr, C., Lewis, D., and Jacobs, D. Dynamic changes in motivation in collaborative citizen-science projects. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) '12 (2012), 217-226.

Digital Library

[50]

Ryan, R. Control and information in the interpersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology 42 (1982), 450-461.

[51]

Schmitt, F., and Lahroodi, R. The epistemic value of curiosity. Education Theory 8 (2008), 125-148.

[52]

Segal, A., Simpson, R., Gal, Y., Homsy, V., Heartwood, M., Page, K., and Jirotka, M. Improving productivity in citizen science through controlled intervention. In WWW (2015), 331-337.

Digital Library

[53]

Shaw, A. D., Horton, J. J., and Chen, D. L. Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, CSCW '11, ACM (NY, NY, USA, 2011), 275-284.

Digital Library

[54]

Shirk, J., Ballard, H., Wilderman, C., Phillips, T., Wiggins, A., Jordan, R., and Bonney, R. Public participation in scientific research: A framework for intentional design. Ecology and Society 17, 2 (2012), 29-48.

[55]

Silvia, P. J. Curiosity and motivation. In Oxford Handbook of Motivation, R. M. Ryan, Ed. Oxford University Press, NY, 2014.

[56]

Speilberger, C., and Starr, L. Curiosity and exploratory behavior. Motivation, Theory and Research (1994), 221-243.

[57]

Teodoro, R., Ozturk, P., Naaman, M., Mason, W., and Lindqvist, J. The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work, CSCW '14, ACM (NY, NY, USA, 2014), 236-247.

Digital Library

[58]

Tieben, R., Bekker, T., and Schouten, B. Curiosity and interaction: Making people curious through interactive systems. In Proceedings of the 25th BCS Conference on Human-Computer Interaction, BCS-HCI '11, British Computer Society (Swinton, UK, UK, 2011), 361-370.

Digital Library

[59]

von Ahn, L., and Dabbish, L. Designing games with a purpose. Communications of the ACM 51, 8 (2008), 58-67.

Digital Library

[60]

Wasserman, L. All of Statistics: A Concise Course in Statistical Inference. Springer-Verlag, Berlin, 2003.

Digital Library

[61]

Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C., Durham, M., and Rothermel, G. Harnessing curiosity to increase correctness in end-user programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '03, ACM (NY, NY, USA, 2003), 305-312.

Digital Library

[62]

Yin, M., and Chen, Y. Bonus or not? learn to reward in crowdsourcing. In Proceedings of the 24th International Joint Conference on Artificial Intelligence, IJCAI '15 (2015), 201-207.

Digital Library

[63]

Yin, M., Chen, Y., and Sun, Y. The effects of performance-contingent financial incentives in online labor markets. In Conference on Artificial Intelligence, AAAI '13 (2013), 1191-1197.

Digital Library

[64]

Zeiler, M. Schedules of reinforcement. In Handbook of Operant Behaviour, W. Hong and J. Staddon, Eds. Prentice Hall, Englewood Cliffs, NJ, 1977.

[65]

Zhu, H., Kraut, R., and Kittur, A. Organizing without formal organization: Group identification, goal setting and social modeling in directing online production. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW '12, ACM (NY, NY, USA, 2012), 935-944.

Digital Library

[66]

Zion, M., and Sadeh, I. Curiosity and open inquiry learning. Journal of Biological Education 41 (2007), 162-8.

Cited By

Amorim ARibeiro AArouca MMeijon ICavalheiro VPestana MVieira V(2024)Gamification Techniques and Contribution Filtering in Crowdsourcing Micro-Task ApplicationsJournal on Interactive Systems10.5753/jis.2024.372715:1(401-416)Online publication date: 15-May-2024
https://doi.org/10.5753/jis.2024.3727
de Leon Pereira RTremblay-Savard O(2024)FORGE: A Framework for Organizing Rewards in Gamified EnvironmentsGames and Culture10.1177/15554120241241555Online publication date: 31-Mar-2024
https://doi.org/10.1177/15554120241241555
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Show More Cited By

Index Terms

Curiosity Killed the Cat, but Makes Crowdwork Better
1. Human-centered computing

Recommendations

"Why would anybody do this?": Understanding Older Adults' Motivations and Challenges in Crowd Work
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Diversifying participation in crowd work can benefit the worker and requester. Increasing numbers of older adults are online, but little is known about their awareness of or how they engage in mainstream crowd work. Through an online survey with 505 ...
Read More
Investigating the Accessibility of Crowdwork Tasks on Mechanical Turk
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Crowdwork can enable invaluable opportunities for people with disabilities, not least the work flexibility and the ability to work from home, especially during the current Covid-19 pandemic. This paper investigates how engagement in crowdwork tasks is ...
Read More
Incentivizing high quality crowdwork

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI PLAY '24

Sponsor:
sigchi

The Annual Symposium on Computer-Human Interaction in Play

October 14 - 17, 2024

Tampere , Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
1,336
Total Downloads

Downloads (Last 12 months)95
Downloads (Last 6 weeks)7

Other Metrics

View Author Metrics

Citations

Cited By

Amorim ARibeiro AArouca MMeijon ICavalheiro VPestana MVieira V(2024)Gamification Techniques and Contribution Filtering in Crowdsourcing Micro-Task ApplicationsJournal on Interactive Systems10.5753/jis.2024.372715:1(401-416)Online publication date: 15-May-2024
https://doi.org/10.5753/jis.2024.3727
de Leon Pereira RTremblay-Savard O(2024)FORGE: A Framework for Organizing Rewards in Gamified EnvironmentsGames and Culture10.1177/15554120241241555Online publication date: 31-Mar-2024
https://doi.org/10.1177/15554120241241555
Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Sun LZhang HLiu ELiu MVaccaro K(2024)NewsGuesser: Using Curiosity to Reduce Selective ExposureProceedings of the ACM on Human-Computer Interaction10.1145/36373768:CSCW1(1-22)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637376
Perin MMelo Lde Souza Cde Almeida AFigueira Filho F(2024)Developing and validating a scale for motivation in participating in time-bounded collaborative eventsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103223185:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103223
PELIT EKATIRCIOGLU E(2023)INVESTIGATING THE EFFECTS OF EPISTEMIC CURIOSITY ON INNOVATIVE WORK BEHAVIOUR: A STUDY ON HOTEL EMPLOYEESTourism and hospitality management10.20867/thm.29.1.629:1(73-85)Online publication date: Mar-2023
https://doi.org/10.20867/thm.29.1.6
Maddalena EIbáñez LReeves NSimperl E(2023)Qrowdsmith: Enhancing Paid Microtask Crowdsourcing with Gamification and Furtherance IncentivesACM Transactions on Intelligent Systems and Technology10.1145/360494014:5(1-26)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3604940
Pandey VKhan NGupta AGajos K(2023)Accuracy and Reliability of At-Home Quantification of Motor Impairments Using a Computer-Based Pointing Task with Children with Ataxia-TelangiectasiaACM Transactions on Accessible Computing10.1145/358179016:1(1-25)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3581790
Ramesh DHenning CEscher NZhu HLee MBanovic N(2023)Ludification as a Lens for Algorithmic Management: A Case Study of Gig-Workers’ Experiences of Ambiguity in Instacart WorkProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596004(638-651)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596004
Manca MPalumbo VPaternò FSantoro C(2023)The Transparency of Automatic Web Accessibility Evaluation Tools: Design Criteria, State of the Art, and User PerceptionACM Transactions on Accessible Computing10.1145/355697916:1(1-36)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3556979
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents