research-article

CrowdFill: collecting structured data from the crowd

Authors:

Jennifer WidomAuthors Info & Claims

SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

Pages 577 - 588

https://doi.org/10.1145/2588555.2610503

Published: 18 June 2014 Publication History

Abstract

We present CrowdFill, a system for collecting structured data from the crowd. While a typical microtask-based approach would pose specific questions to each worker and assemble the answers, CrowdFill shows a partially-filled table to all participating workers. Workers contribute by filling in empty cells, as well as upvoting and downvoting data entered by other workers. The system's synchronization scheme, based on a careful model of primitive operations, enables workers to collaboratively complete the table without latency overhead. CrowdFill allows the specification of constraints on the collected data, and has mechanisms for resolving inconsistencies. Its compensation scheme takes into account each worker's contribution to the final table, and the varying difficulty of data entry tasks. The paper includes some preliminary experimental results.

References

[1]

Amazon Mechanical Turk. http://mturk.com/.

[2]

Amazon Mechanical Turk Developer Sandbox. https://requestersandbox.mturk.com/.

[3]

Google Docs. http://docs.google.com/.

[4]

MongoDB. http://www.mongodb.org/.

[5]

Mongolab. http://www.mongolab.com/.

[6]

Nodejitsu. http://www.nodejitsu.com/.

[7]

Node.js. http://www.nodejs.org/.

[8]

Socket.IO. http://socket.io/.

[9]

C. Berge. Two theorems in graph theory. PNAS, 43(9):842--844, 1957.

[10]

C. A. Ellis and S. J. Gibbs. Concurrency control in groupware systems. In SIGMOD, 1989.

Digital Library

[11]

M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. CrowdDB: Answering queries with crowdsourcing. In SIGMOD, 2011.

Digital Library

[12]

J. Horton and L. Chilton. The labor economics of paid crowdsourcing. In EC, 2010.

Digital Library

[13]

G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Turkit: tools for iterative tasks on mechanical turk. In HCOMP, 2009.

Digital Library

[14]

W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". In HCOMP, 2009.

Digital Library

[15]

A. Parameswaran and N. Polyzotis. Answering queries using humans, algorithms and databases. In CIDR, 2011.

[16]

H. Park, R. Pang, A. Parameswaran, H. Garcia-Molina, N. Polyzotis, and J. Widom. An overview of the deco system: data model and query language; query processing and optimization. SIGMOD Record, 41(4):22--27, 2012.

Digital Library

[17]

H. Park, A. Parameswaran, and J. Widom. Query processing over crowdsourced data, http://ilpubs.stanford.edu:8090/1052/. Technical report, Stanford InfoLab, 2012.

[18]

O. Scekic, H. L. Truong, and S. Dustdar. Incentives and rewarding in social computing. Communications of the ACM, 56(6):72--82, 2013.

Digital Library

[19]

A. D. Shaw, J. J. Horton, and D. L. Chen. Designing incentives for inexpert human raters. In CSCW, 2011.

Digital Library

[20]

Y. Singer and M. Mittal. Pricing mechanisms for crowdsourcing markets. In WWW, 2013.

Digital Library

[21]

A. Singla and A. Krause. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In WWW, 2013.

Digital Library

[22]

C. Sun, X. Jia, Y. Zhang, Y. Yang, and D. Chen. Achieving convergence, causality preservation, and intention preservation in real-time cooperative editing systems. ACM Transactions on Computer-Human Interaction, 5(1):63--108, 1998.

Digital Library

[23]

B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. In ICDE, 2013.

Digital Library

[24]

J. D. Ullman. Principles of Database and Knowledge-Base Systems, Volume II. Computer Science Press, 1989.

Digital Library

[25]

P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis. Max algorithms in crowdsourcing environments. In WWW, 2012.

Digital Library

Cited By

Xiong KXu XFu SWeng DWang YWu Y(2024)JsonCurer: Data Quality Management for JSON Based on an Aggregated SchemaIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338855630:6(3008-3021)Online publication date: Jun-2024
https://doi.org/10.1109/TVCG.2024.3388556
Xi YWang NZhang YChen X(2024)CrowdDA: Difficulty-aware crowdsourcing task optimization for cleaning web tablesExpert Systems with Applications10.1016/j.eswa.2023.122139238(122139)Online publication date: Mar-2024
https://doi.org/10.1016/j.eswa.2023.122139
Bastanfard AShahabipour MAmirkhani D(2023)Crowdsourcing of labeling image objects: an online gamification application for data collectionMultimedia Tools and Applications10.1007/s11042-023-16325-683:7(20827-20860)Online publication date: 4-Aug-2023
https://doi.org/10.1007/s11042-023-16325-6
Show More Cited By

Index Terms

CrowdFill: collecting structured data from the crowd
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing theory, concepts and paradigms
      1. Computer supported cooperative work
2. Social and professional topics
  1. Professional topics
    1. Computing and business
      1. Computer supported cooperative work

Recommendations

CrowdFill: a system for collecting structured data from the crowd
WWW '14 Companion: Proceedings of the 23rd International Conference on World Wide Web

CrowdFill is a system for collecting structured data from the crowd. Unlike a typical microtask-based approach, CrowdFill shows an entire partially-filled table to all participating workers; workers collaboratively complete the table by filling in empty ...
Crowdsourcing as a Method for the Collection of Revealed Preferences Data
SOSE '15: Proceedings of the 2015 IEEE Symposium on Service-Oriented System Engineering

Crowdsourcing has been used widely for the collection of stated preference data (e.g., responses in a survey) by researchers. However, the use of crowdsourcing for collection of revealed preference data (e.g., real-life data collected in natural ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

June 2014

1645 pages

ISBN:9781450323765

DOI:10.1145/2588555

General Chairs:
Curtis Dyreson
Utah State University, USA
,
Feifei Li
University of Utah, USA
,
Program Chair:
M. Tamer Özsu
University of Waterloo, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SIGMOD/PODS'14

Sponsor:

SIGMOD

SIGMOD/PODS'14: International Conference on Management of Data

June 22 - 27, 2014

Utah, Snowbird, USA

Acceptance Rates

SIGMOD '14 Paper Acceptance Rate 107 of 421 submissions, 25%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
506
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xiong KXu XFu SWeng DWang YWu Y(2024)JsonCurer: Data Quality Management for JSON Based on an Aggregated SchemaIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338855630:6(3008-3021)Online publication date: Jun-2024
https://doi.org/10.1109/TVCG.2024.3388556
Xi YWang NZhang YChen X(2024)CrowdDA: Difficulty-aware crowdsourcing task optimization for cleaning web tablesExpert Systems with Applications10.1016/j.eswa.2023.122139238(122139)Online publication date: Mar-2024
https://doi.org/10.1016/j.eswa.2023.122139
Bastanfard AShahabipour MAmirkhani D(2023)Crowdsourcing of labeling image objects: an online gamification application for data collectionMultimedia Tools and Applications10.1007/s11042-023-16325-683:7(20827-20860)Online publication date: 4-Aug-2023
https://doi.org/10.1007/s11042-023-16325-6
Chai CWang JLuo YNiu ZLi G(2022)Data Management for Machine Learning: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3148237(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3148237
Wallace SPapoutsaki ATan NGuo HHuang J(2021)Case Studies on the Motivation and Performance of Contributors Who Verify and Maintain In-Flux Tabular DatasetsProceedings of the ACM on Human-Computer Interaction10.1145/34795925:CSCW2(1-25)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479592
Suneja SZheng YZhuang YLaredo JMorari A(2021)Towards Reliable AI for Source Code UnderstandingProceedings of the ACM Symposium on Cloud Computing10.1145/3472883.3486995(403-411)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1145/3472883.3486995
Rahman PNandi AHebert C(2020)Amplifying Domain Expertise in Clinical Data PipelinesJMIR Medical Informatics10.2196/196128:11(e19612)Online publication date: 5-Nov-2020
https://doi.org/10.2196/19612
Rahman PHebert CNandi A(2019)ICARUSProceedings of the VLDB Endowment10.14778/3275366.328497011:13(2263-2276)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.14778/3275366.3284970
Rekatsinas TDeshpande AParameswaran AZhu WTao DCheng XCui PRundensteiner ECarmel DHe QXu Yu J(2019)CRUXProceedings of the 28th ACM International Conference on Information and Knowledge Management10.1145/3357384.3357976(841-850)Online publication date: 3-Nov-2019
https://dl.acm.org/doi/10.1145/3357384.3357976
Dolatshah MTeoh MWang JPei J(2018)Cleaning crowdsourced labels using oracles for statistical classificationProceedings of the VLDB Endowment10.14778/3297753.329775812:4(376-389)Online publication date: 1-Dec-2018
https://dl.acm.org/doi/10.14778/3297753.3297758
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents