column

Public Access

Running user studies with crowd workers

Authors:

Brian McInnis,

Gilly LeshedAuthors Info & Claims

Interactions, Volume 23, Issue 5

Pages 50 - 53

https://doi.org/10.1145/2968077

Published: 23 August 2016 Publication History

All formats PDF

References

[1]

Kittur, A., Chi, E., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. of CHI '08. 2008, 453--456.

Digital Library

Google Scholar

[2]

McGinn, J. and LaRoche, C. Fast, cheap, and powerful user research. Interactions 21, 3 (May 2014), 62--65.

Digital Library

Google Scholar

[3]

Mason, W. and Suri, S. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods 44, 1 (2012), 1--23.

Crossref

Google Scholar

[4]

Berinsky, A.J., Huber, G.A., and Lenz, G.S. Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis 20 (2012), 351--368.

Crossref

Google Scholar

[5]

McInnis, B., Nam, C., Cosley, D., and Leshed, G. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. Proc. of CHI '16. 2016.

Digital Library

Google Scholar

[6]

Irani, L.C. and Silberman, M.S. Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk. Proc. of CHI '13. 2013, 611--620.

Digital Library

Google Scholar

[7]

McInnis, B., Murnane, E., Epstein, D., Cosley, D., and Leshed, G. One and done: Factors affecting one-time contributors to ad-hoc online communities. Proc. of CSCW '16. 2016, 609--623.

Digital Library

Google Scholar

Cited By

View all

McInnis BPindus RKareem DGamboa SNebeker C(2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637330
McInnis BLeshed GCosley D(2018)Crafting Policy Discussion Prompts as a Task for NewcomersProceedings of the ACM on Human-Computer Interaction10.1145/32743902:CSCW(1-23)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1145/3274390
McInnis BCosley DBaumer ELeshed GForte APrilla MVivacqua AMüller CRobert L(2018)Effects of Comment Curation and Opposition on Coherence in Online Policy DiscussionProceedings of the 2018 ACM International Conference on Supporting Group Work10.1145/3148330.3148348(347-358)Online publication date: 7-Jan-2018
https://dl.acm.org/doi/10.1145/3148330.3148348
Show More Cited By

Index Terms

Running user studies with crowd workers
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
2. Information systems
  1. World Wide Web
    1. Web applications
      1. Crowdsourcing
        Trust

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Optimal Testing for Crowd Workers
AAMAS '16: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems

Requesters on crowdsourcing platforms, such as Amazon Mechanical Turk, routinely insert gold questions to verify that a worker is diligent and is providing high-quality answers. However, there is no clear understanding of when and how many gold ...
Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms
Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a ...

Comments

Information & Contributors

Information

Published In

Interactions Volume 23, Issue 5

September + October 2016

67 pages

ISSN:1072-5520

EISSN:1558-3449

DOI:10.1145/2991131

Editors:
Ron Wakkary
Association for Computing Machinery, New York, NY
,
Erik Stolterman
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 August 2016

Published in INTERACTIONS Volume 23, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Column
Popular
Pre-selected

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
1,089
Total Downloads

Downloads (Last 12 months)240
Downloads (Last 6 weeks)44

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

McInnis BPindus RKareem DGamboa SNebeker C(2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637330
McInnis BLeshed GCosley D(2018)Crafting Policy Discussion Prompts as a Task for NewcomersProceedings of the ACM on Human-Computer Interaction10.1145/32743902:CSCW(1-23)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1145/3274390
McInnis BCosley DBaumer ELeshed GForte APrilla MVivacqua AMüller CRobert L(2018)Effects of Comment Curation and Opposition on Coherence in Online Policy DiscussionProceedings of the 2018 ACM International Conference on Supporting Group Work10.1145/3148330.3148348(347-358)Online publication date: 7-Jan-2018
https://dl.acm.org/doi/10.1145/3148330.3148348
Miller MTang JVenolia GWilkinson GInkpen KMark GFussell SLampe Cschraefel mHourcade JAppert CWigdor D(2017)Conversational Chat CirclesProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3025621(2394-2404)Online publication date: 2-May-2017
https://dl.acm.org/doi/10.1145/3025453.3025621
Lazar JFeng JHochheiser H(2017)Online and ubiquitous HCI researchResearch Methods in Human Computer Interaction10.1016/B978-0-12-805390-4.00014-5(411-453)Online publication date: 2017
https://doi.org/10.1016/B978-0-12-805390-4.00014-5

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Digital Edition

View this article in digital edition.

Digital Edition

HTML Format

View this article in HTML Format.

HTML Format

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

References

Cited By

Index Terms

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

Optimal Testing for Crowd Workers

Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Digital Edition

HTML Format

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations