research-article

Crowdsourced mobile data collection: lessons learned from a new study methodology

Authors:

Evan Welbourne,

Emmanuel Munguia-TapiaAuthors Info & Claims

HotMobile '14: Proceedings of the 15th Workshop on Mobile Computing Systems and Applications

Article No.: 2, Pages 1 - 6

https://doi.org/10.1145/2565585.2565608

Published: 26 February 2014 Publication History

Abstract

In this paper we explore a scalable data collection methodology that simultaneously achieves low cost and a high degree of control. We use popular online crowdsourcing platforms to recruit 63 subjects for a 90-day data collection that resulted in over 75K hours of data. The total cost of data collection was dramatically lower than for alternative methodologies, with total subject compensation under $3.5K US, and a total of less than 10 hours/week spent by researchers managing the study. At the same time, our methodology enhances control and enables richer study protocols by allowing direct contact with subjects. We were able to conduct surveys, exchange messages, and debug remotely with feedback from subjects. In addition to reporting on study details, we also discuss interesting findings and offer lessons learned.

References

[1]

ACRA https://github.com/ACRA Oct 2013

[2]

Aharony, N. et al. Social fMRI: Investigating and shaping social mechanisms in the real world. Pervasive and Mobile Computing 7 643--659 (2011)

Digital Library

[3]

Amazon Mechanical Turk. http://www.mturk.com Oct 2013

[4]

Amazon EC2. http://aws.amazon.com/ec2/ Oct 2013

[5]

Apache. http://httpd.apache.org/ Oct 2013

[6]

CastingWords. https://castingwords.com/ Oct 2013

[7]

Django. https://www.djangoproject.com/ Oct 2013

[8]

Elance. http://www.elance.com Oct 2013

[9]

Funf Open Sensing Framework. http://funf.org/ Oct 2013

[10]

Kocchar, S. et al. The Anatomy of a Large-Scale Human Computation Engine. KDD-HCOMP '10, 2010

Digital Library

[11]

Laurila, J. K. The Mobile Data Challenge: Big Data for Mobile Computing Research. Mobile Data Challenge Workshop in conjunction with Pervasive '12, 2012

[12]

Miluzzo, E. et al. Research in the App Store Era: Experiences from the CenceMe App Deployment on the iPhone. Ubicomp '10, 2010

[13]

Nielsen. Who's Winning the U.S. Smartphone Market? http://www.nielsen.com/us/en/newswire.html Aug 2013

[14]

ODesk. http://www.odesk.com Oct 2013

[15]

Paolacci, G. et al. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 2010

[16]

PhoneLab. http://www.phone-lab.org Jan 2014

[17]

Ra, M. et al. Medusa: A Programming Framework for Crowd-Sensing Applications. MobiSys'12, 2012

Digital Library

[18]

Ruby on Rails. http://rubyonrails.org/ Oct 2013

[19]

Schmidt, L. A. Crowdsourcing for Human Subjects Research. CrowdConf 2010, 2010

[20]

Smith, A. Smartphone Ownership -- 2013 Update. Pew Research Center Report, Jun 2013

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Tosida EHerdiyeni YSuprehatin SMarimin (2020)The Potential for Implementing a Big Data Analytic-based Smart Village in Indonesia2020 International Conference on Computer Science and Its Application in Agriculture (ICOSICA)10.1109/ICOSICA49951.2020.9243265(1-10)Online publication date: 16-Sep-2020
https://doi.org/10.1109/ICOSICA49951.2020.9243265
Li YPandis IGuo Y(2016)Enabling Virtual Sensing as a ServiceInformatics10.3390/informatics30200033:2(3)Online publication date: 29-Mar-2016
https://doi.org/10.3390/informatics3020003
Show More Cited By

Index Terms

Crowdsourced mobile data collection: lessons learned from a new study methodology
1. Applied computing
  1. Computers in other domains
    1. Personal computers and PC applications
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Web-based interaction

Recommendations

Age-based garbage collection

Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Age-based garbage collection
OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications

Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Garbage collection hints
HiPEAC'05: Proceedings of the First international conference on High Performance Embedded Architectures and Compilers

This paper shows that Appel-style garbage collectors often make suboptimal decisions both in terms of when and how to collect. We argue that garbage collection should be done when the amount of live bytes is low (in order to minimize the collection cost)...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

HotMobile '14: Proceedings of the 15th Workshop on Mobile Computing Systems and Applications

February 2014

134 pages

ISBN:9781450327428

DOI:10.1145/2565585

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 February 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

HotMobile '14

Sponsor:

SIGMOBILE

HotMobile '14: 15th Workshop on Mobile Computing Systems and Applications

February 26 - 27, 2014

California, Santa Barbara

Acceptance Rates

HotMobile '14 Paper Acceptance Rate 22 of 72 submissions, 31%;

Overall Acceptance Rate 96 of 345 submissions, 28%

Upcoming Conference

HOTMOBILE '25

Sponsor:
sigmobile

The 26th International Workshop on Mobile Computing Systems and Applications

February 26 - 27, 2025

La Quinta , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
249
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Tosida EHerdiyeni YSuprehatin SMarimin (2020)The Potential for Implementing a Big Data Analytic-based Smart Village in Indonesia2020 International Conference on Computer Science and Its Application in Agriculture (ICOSICA)10.1109/ICOSICA49951.2020.9243265(1-10)Online publication date: 16-Sep-2020
https://doi.org/10.1109/ICOSICA49951.2020.9243265
Li YPandis IGuo Y(2016)Enabling Virtual Sensing as a ServiceInformatics10.3390/informatics30200033:2(3)Online publication date: 29-Mar-2016
https://doi.org/10.3390/informatics3020003
Tangmunarunkit HHsieh CLongstaff BNolen SJenkins JKetcham CSelsky JAlquaddoomi FGeorge DKang JKhalapyan ZOoms JRamanathan NEstrin D(2015)OhmageACM Transactions on Intelligent Systems and Technology10.1145/27173186:3(1-21)Online publication date: 21-Apr-2015
https://dl.acm.org/doi/10.1145/2717318
Assemi BSchlagwein DSafi HMesbah M(2015)Crowdsourcing as a Method for the Collection of Revealed Preferences DataProceedings of the 2015 IEEE Symposium on Service-Oriented System Engineering10.1109/SOSE.2015.52(378-382)Online publication date: 30-Mar-2015
https://dl.acm.org/doi/10.1109/SOSE.2015.52
Lakar PSamal SMuthupandi SPatil N(2015)Smart web miner - extending web browser with mining framework based on user behavior & web-of-thing patterns for web personalizationProceedings of the 2015 International Conference on Green Computing and Internet of Things (ICGCIoT)10.1109/ICGCIoT.2015.7380520(522-527)Online publication date: 8-Oct-2015
https://dl.acm.org/doi/10.1109/ICGCIoT.2015.7380520
Welbourne ETapia EBrush AFriday AKientz JScott JSong J(2014)CrowdSignalsProceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication10.1145/2638728.2641309(873-877)Online publication date: 13-Sep-2014
https://dl.acm.org/doi/10.1145/2638728.2641309
Srinivasan VMoghaddam SMukherji ARachuri KXu CTapia EBrush AFriday AKientz JScott JSong J(2014)MobileMinerProceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/2632048.2632052(389-400)Online publication date: 13-Sep-2014
https://dl.acm.org/doi/10.1145/2632048.2632052

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents