research-article

CrowdPickUp: Crowdsourcing Task Pickup in the Wild

Authors:

Jorge Goncalves,

Niels van Berkel,

Vassilis KostakosAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 1, Issue 3

Article No.: 51, Pages 1 - 22

https://doi.org/10.1145/3130916

Published: 11 September 2017 Publication History

Abstract

We develop and evaluate a new ubiquitous crowdsourcing platform called CrowdPickUp, that combines the advantages of mobile and situated crowdsourcing to overcome their respective limitations. In a 19-day long field study with 70 participants, we evaluate the quality of work that CrowdPickUp produces. In particular, we measure quality in terms of worker performance in a variety of tasks (requiring local knowledge, location-based, general) while using a number of different quality control mechanisms, and also capture workers’ perceptions of the platform. Our findings show that workers of CrowdPickUp contributed data of comparable quality to previously presented crowdsourcing deployments while at the same time allowing for a wide breadth of tasks to be deployed. Finally, we offer insights towards the continued exploration of this research agenda.

References

[1]

Florian Alt, Alireza A. S. Shirazi, Albrecht Schmidt, Urs Kramer and Zahid Nawaz. 2010. Location-based Crowdsourcing: Extending Crowdsourcing to the Real World. In NordiCHI, ACM, 13-22.

Digital Library

[2]

Mustafa M. Y. Arslan, Indrajeet Singh, Shailendra Singh, Harsha H. V. Madhyastha, Karthikeyan Sundaresan and Srikanth S. V. Krishnamurthy. 2012. Computing while charging: building a distributed computing infrastructure using smartphones. In Proceedings of the 8th international conference on Emerging networking experiments and technologies, ACM, 193-204.

Digital Library

[3]

Jeffrey J. A. Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy and Mani M. B. Srivastava. 2006. Participatory sensing. In First Workshop on World-Sensor-Web: Mobile Device Centric Sensory Networks and Applications at Sensys ‘06.

[4]

Chris Callison-Burch. 2009. Fast, cheap, and creative: evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, 286-295.

Digital Library

[5]

Manuel Cebrian, Lorenzo Coviello, Andrea Vattani and Panagiotis Voulgaris. 2012. Finding red balloons with split contracts: crobustness to individuals’ selfishness. In Proceedings of the Forty-Fourth Annual ACM Symposium on Theory of Computing (STOC 2012), 775-788.

Digital Library

[6]

Zhao Chen, Rui Fu, Ziyuan Zhao, Zheng Liu, Leihao Xia, Lei Chen, Peng Cheng, Caleb C. C. Cao, Yongxin Tong and Chen C. J. Zhang. 2014. gMission: A General Spatial Crowdsourcing Platform. Proc. VLDB Endow. 7, 13: 1629-1632.

Digital Library

[7]

Donkor B. On Social Sentiment and Sentiment Analysis. Retrieved 13/05/2016 from http://brnrd.me/social-sentiment-sentiment-analysis/

[8]

N. Eagle. 2009. txteagle: Mobile Crowdsourcing. In Proceedings of the 3rd International Conference on Internationalization, Design and Global Development: Held as Part of HCI International 2009, Springer-Verlag, 447-456.

Digital Library

[9]

Stephen Figlewski. 1979. Subjective information and market efficiency in a betting market. Journal of Political Economy 87, 1: 75-88.

[10]

Jorge Goncalves, Michael Feldman, Subingqian Hu, Vassilis Kostakos and Abraham Bernstein. 2017. Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities. In Proceedings of the 26th International Conference on World Wide Web, 1023-1031.

Digital Library

[11]

Jorge Goncalves, Denzil Ferreira, Simo Hosio, Yong Liu, Jakob Rogstadius, Hannu Kukka and Vassilis Kostakos. 2013. Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 753-762.

Digital Library

[12]

Jorge Goncalves, Simo Hosio, Denzil Ferreira and Vassilis Kostakos. 2014. Game of Words: Tagging Places through Crowdsourcing on Public Displays. In Designing Interactive Systems, 705-714.

Digital Library

[13]

Jorge Goncalves, Simo Hosio and Vassilis Kostakos. 2017. Eliciting Structured Knowledge from Situated Crowd Markets. ACM Transactions on Internet Technology, 17(2), article 14.

Digital Library

[14]

Jorge Goncalves, Simo Hosio, Yong Liu and Vassilis Kostakos. 2016. Worker performance in a situated crowdsourcing market. Interacting with Computers 28, 5: 612-624.

[15]

Jorge Goncalves, Vassilis Kostakos, Simo Hosio, Evangelos Karapanos and Olga Lyra. 2013. IncluCity: Using Contextual Cues to Raise Awareness on Environmental Accessibility. In Proceedings of the ACM SIGACCESS International Conference on Computers and Accessibility, article 17.

Digital Library

[16]

Aakar Gupta, William Thies, Edward Cutrell and Ravin Balakrishnan. 2012. mClerk: enabling mobile crowdsourcing in developing regions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1843-1852.

Digital Library

[17]

Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh and Björn Hartmann. 2012. Community Sourcing: engaging local crowds to perform expert work via physical kiosks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1539-1548.

Digital Library

[18]

Simo Hosio, Jorge Goncalves, Vili Lehdonvirta, Denzil Ferreira and Vassilis Kostakos. 2014. Situated Crowdsourcing Using a Market Model. In User Interface Software and Technology, ACM, 55-64.

Digital Library

[19]

Simo Hosio, Jorge Goncalves, Niels van Berkel, Simon Klakegg. 2016. Crowdsourcing situated 8 subjective knowledge for decision support. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, ACM, 1478-1483.

Digital Library

[20]

Yi-Ching Huang. 2015. Designing a Micro-Volunteering Platform for Situated Crowdsourcing. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work 8 Social Computing, ACM, 73-76.

Digital Library

[21]

Yun Huang, Alain Shema and Huichuan Xia. 2016. A proposed genome of mobile and situated crowdsourcing and its design implications for encouraging contributions. International Journal of Human-Computer Studies.

Digital Library

[22]

Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. 2014. Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd International Conference on World Wide Web, 143-154.

Digital Library

[23]

Panagiotis G. Ipeirotis, Foster Provost and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, 64-67.

Digital Library

[24]

Thivya Kandappu, Nikita Jaiman, Randy Tandriansyah, Archan Misra, Shih-Fen Cheng, Cen Chen, Hoong Chuin Lau, Deepthi Chander and Koustov Dasgupta. 2016. TASKer: behavioral insights via campus-based experimental mobile crowd-sourcing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 392-402.

Digital Library

[25]

Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, ACM, 1301-1318.

Digital Library

[26]

Shin’ichi Konomi, Niwat Thepvilojanapong, Ryohei Suzuki, Susanna Pirttikangas, Kaoru Sezaki and Yoshito Tobe. 2009. Askus: Amplifying Mobile Actions. In International Conference on Pervasive Computing, Springer Berlin Heidelberg, 202-219.

Digital Library

[27]

Nira Liberman, Yaacov Trope and Elena Stephan. 2007. Psychological distance. Social psychology: Handbook of basic principles 2: 353-383.

[28]

Yong Liu, Denzil Ferreira, Jorge Goncalves, Simo Hosio, Pratyush Pandab and Vassilis Kostakos. 2017. Donating Context Data to Science: The Effects of Social Signals and Perceptions on Action-Taking. Interacting with Computers 29, 2: 132-146.

[29]

Jan Lorenz, Heiko Rauhut, Frank Schweitzer and Dirk Helbing. 2011. How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences 108, 22: 9020-9025.

[30]

Thomas Ludwig, Christoph Kotthaus, Christian Reuter, Sören S. v. Dongen and Volkmar Pipek. 2016. Situated crowdsourcing during disasters: Managing the tasks of spontaneous volunteers through public displays. International Journal of Human-Computer Studies.

Digital Library

[31]

Thomas W. Malone, Robert Laubacher and Chrysanthos Dellarocas. 2010. The collective intelligence genome. MIT Sloan Management Review 51, 3: 21.

[32]

Jörg Müller, Florian Alt, Daniel Michelis and Albrecht Schmidt. 2010. Requirements and design space for interactive public displays. In Proceedings of the international conference on Multimedia, ACM, 1285-1294.

Digital Library

[33]

Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni and Bjoern Hartmann. 2011. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. In AAAI Workshop on Human Computation, 121-123.

Digital Library

[34]

Eric Paulos, R. J. Honicky and Ben Hooker. 2008. Citizen science: Enabling participatory urbanism. In Handbook of Research on Urban Informatics: The Practice and Promise of the Real-Time City Markus Foth (eds.). IGI Global, Hershey, PA, 414-436.

[35]

Yvonne Rogers, Helen Sharp and Jenny Preece. 2011. Interaction design: beyond human-computer interaction. John Wiley 8 Sons.

Digital Library

[36]

Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, J, Laredo and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In International AAAI Conference on Web and Social Media, 321-328.

[37]

Reijo Savolainen. 2006. Time as a context of information seeking. Library 8 Information Science Research 28, 1: 110-127.

[38]

Clay Shirky. 2010. Cognitive surplus: How technology makes consumers into collaborators. Penguin.

[39]

Rannie Teodoro, Pinar Ozturk, Mor Naaman, Winter Mason and Janne Lindqvist. 2014. The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing, ACM, 236-247.

Digital Library

[40]

Jacob Thebault-Spieker, Loren G. Terveen and Brent Hecht. 2015. Avoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work 8 Social Computing, ACM, 265-275.

Digital Library

[41]

Mirjam A. Tuk, Peeter W.J. Verlegh, A. Smidts and Daniel H.J. Wigboldus. 2009. Sales and sincerity: The role of relational framing in word-of-mouth marketing. Journal of Consumer Psychology 19, 1: 38-47.

[42]

Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung and Michael S. Bernstein. 2014. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, 3645-3654.

Digital Library

[43]

Heli Väätäjä, Teija Vainio, Esa Sirkkunen and Kari Salo. 2011. Crowdsourced news reporting: supporting news content creation with mobile phones. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, ACM, 435-444.

Digital Library

[44]

Maja Vukovic, Soundar Kumara and Ohad Greenshpan. 2010. Ubiquitous crowdsourcing. In Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing - Adjunct, ACM, 523-526.

Digital Library

[45]

Jeroen Vuurens, Arjen A. P. de Vries and Carsten Eickhoff. 2011. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR11), 21-26.

Cited By

Chiang YHsu JLiu CHuang TChiu HChang Y(2023)Investigating Users' Inclination of Leveraging Mobile Crowdsourcing to Obtain Verifying vs. Supplemental Information when Facing Inconsistent Smat-city Sensor InformationCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607001(338-342)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3607001
Hettiachchi DKostakos VGoncalves J(2023)A Survey on Task Assignment in CrowdsourcingACM Computing Surveys10.1145/349452255:3(1-35)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3494522
Primbs JIlg WThierfelder ASeveritt BHohnecker CAlt APascher AWörz ULautenbacher HHollmann KBarth GRenner TMenth M(2022)The SSTeP-KiZ System—Secure Real-Time Communication Based on Open Web Standards for Multimodal Sensor-Assisted Tele-PsychotherapySensors10.3390/s2224958922:24(9589)Online publication date: 7-Dec-2022
https://doi.org/10.3390/s22249589
Show More Cited By

Index Terms

CrowdPickUp: Crowdsourcing Task Pickup in the Wild
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
  2. Ubiquitous and mobile computing

Recommendations

Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web Companion

Appropriate task routing and assignment is an important, but often overlooked, element in crowdsourcing research and practice. In this paper, we explore and evaluate a mechanism that can enable matching crowdsourcing tasks to suitable crowd-workers ...
Crowdsourcing GO: Effect of Worker Situation on Mobile Crowdsourcing Performance
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

The increasing popularity of mobile crowdsourcing platforms has enabled crowd workers to accept jobs wherever/whenever they are, and also provides opportunity for task requesters to order time/location specific tasks to workers. Since workers on mobile ...
Do extra dollars paid-off?: an exploratory study on topcoder
CSI-SE '18: Proceedings of the 5th International Workshop on Crowd Sourcing in Software Engineering

In general crowdsourcing, different task requesters employ different pricing strategies to balance task cost and expected worker performance. While most existing studies show that increasing incentives tend to benefit crowdsourcing outcomes, i.e. ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 1, Issue 3

September 2017

2023 pages

EISSN:2474-9567

DOI:10.1145/3139486

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 September 2017

Accepted: 01 July 2017

Revised: 01 May 2017

Received: 01 February 2017

Published in IMWUT Volume 1, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
348
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chiang YHsu JLiu CHuang TChiu HChang Y(2023)Investigating Users' Inclination of Leveraging Mobile Crowdsourcing to Obtain Verifying vs. Supplemental Information when Facing Inconsistent Smat-city Sensor InformationCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3607001(338-342)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3607001
Hettiachchi DKostakos VGoncalves J(2023)A Survey on Task Assignment in CrowdsourcingACM Computing Surveys10.1145/349452255:3(1-35)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3494522
Primbs JIlg WThierfelder ASeveritt BHohnecker CAlt APascher AWörz ULautenbacher HHollmann KBarth GRenner TMenth M(2022)The SSTeP-KiZ System—Secure Real-Time Communication Based on Open Web Standards for Multimodal Sensor-Assisted Tele-PsychotherapySensors10.3390/s2224958922:24(9589)Online publication date: 7-Dec-2022
https://doi.org/10.3390/s22249589
Wu DZeng ZShi FYu WWu TLiu Q(2022)Human as a Service: Towards Resilient Parking Search System With Sensorless SensingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2021.313371323:8(13863-13877)Online publication date: Aug-2022
https://doi.org/10.1109/TITS.2021.3133713
Alorwu Avan Berkel NGoncalves JOppenlaender JLópez MSeetharaman MHosio S(2022)Crowdsourcing sensitive data using public displays—opportunities, challenges, and considerationsPersonal and Ubiquitous Computing10.1007/s00779-020-01375-626:3(681-696)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1007/s00779-020-01375-6
Hettiachchi DSanderson MGoncalves JHosio SKazai GLease MSchaekermann MYilmaz E(2021)Investigating and Mitigating Biases in Crowdsourced DataCompanion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3462204.3481729(331-334)Online publication date: 23-Oct-2021
https://dl.acm.org/doi/10.1145/3462204.3481729
Chiang CChen YLin FFeng FWu HLee HYang CChang YKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)“I Got Some Free Time”: Investigating Task-execution and Task-effort Metrics in Mobile Crowdsourcing TasksProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445477(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445477
Medina Rios LRuiz-Correa SSantani DGatica-Perez D(2021)Who Sees What? Examining Urban Impressions in Global South CitiesHuman Perception of Visual Information10.1007/978-3-030-81465-6_10(263-292)Online publication date: 22-Jul-2021
https://doi.org/10.1007/978-3-030-81465-6_10
Hettiachchi Dvan Berkel NKostakos VGoncalves J(2020)CrowdCogProceedings of the ACM on Human-Computer Interaction10.1145/34151814:CSCW2(1-22)Online publication date: 15-Oct-2020
https://dl.acm.org/doi/10.1145/3415181
Hettiachchi DSarsenbayeva ZAllison Fvan Berkel NDingler TMarini GKostakos VGoncalves JBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)"Hi! I am the Crowd Tasker" Crowdsourcing through Digital Voice AssistantsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376320(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376320
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents