research-article

CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling

Authors:

Ju Fan,

Xiaoyong DuAuthors Info & Claims

SIGMOD '19: Proceedings of the 2019 International Conference on Management of Data

Pages 1957 - 1960

https://doi.org/10.1145/3299869.3320221

Published: 25 June 2019 Publication History

Get Access

Abstract

Large-scale data labeling has become a major bottleneck for many applications, such as machine learning and data integration. This paper presents CrowdGame, a crowdsourcing system that harnesses the crowd to gather data labels in a cost-effective way. CrowdGame focuses on generating high-quality labeling rules to largely reduce the labeling cost while preserving quality. It first generates candidate rules, and then devises a game-based crowdsourcing approach to select rules with high coverage and accuracy. CrowdGame applies the generated rules for effective data labeling. We have implemented CrowdGame and provided a user-friendly interface for users to deploy their labeling applications. We will demonstrate CrowdGame in two representative data labeling scenarios, entity matching and relation extraction.

References

[1]

J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248--255, 2009.

Crossref

Google Scholar

[2]

J. Fan, G. Li, B. C. Ooi, K. Tan, and J. Feng. icrowd: An adaptive crowdsourcing framework. In SIGMOD, pages 1015--1030, 2015.

Digital Library

Google Scholar

[3]

D. Gao, Y. Tong, J. She, T. Song, L. Chen, and K. Xu. Top-k team recommendation and its variants in spatial crowdsourcing. Data Science and Engineering, 2(2):136--150, 2017.

Crossref

Google Scholar

[4]

A. Ratner, S. H. Bach, H. R. Ehrenberg, J. A. Fries, S. Wu, and C. Ré . Snorkel: Rapid training data creation with weak supervision. PVLDB, 11(3):269--282, 2017.

Digital Library

Google Scholar

[5]

Y. Tong, L. Chen, Z. Zhou, H. V. Jagadish, L. Shou, and W. Lv. SLADE: A smart large-scale task decomposer in crowdsourcing. IEEE Trans. Knowl. Data Eng., 30(8):1588--1601, 2018.

Digital Library

Google Scholar

[6]

P. Varma and C. Ré . Snuba: Automating weak supervision to label training data. PVLDB, 12(3):223--236, 2018.

Digital Library

Google Scholar

[7]

J. Yang, J. Fan, Z. Wei, G. Li, T. Liu, and X. Du. Cost-effective data annotation using game-based crowdsourcing. PVLDB, 12(1):57--70, 2018.

Digital Library

Google Scholar

Cited By

View all

Liu TFan JLi GTang NDu X(2023)Tabular data synthesis with generative adversarial networks: design space and optimizationsThe VLDB Journal10.1007/s00778-023-00807-y33:2(255-280)Online publication date: 15-Aug-2023
https://doi.org/10.1007/s00778-023-00807-y
Chai CWang JLuo YNiu ZLi G(2022)Data Management for Machine Learning: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3148237(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3148237
Li KLi GWang YHuang YLiu ZWu Z(2021)CrowdRL: An End-to-End Reinforcement Learning Framework for Data Labelling2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00032(289-300)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00032
Show More Cited By

Index Terms

CrowdGame: A Game-Based Crowdsourcing System for Cost-Effective Data Labeling

Recommendations

A game-based framework for crowdsourced data labeling
Abstract
Data labeling, which assigns data with multiple classes, is indispensable for many applications, such as machine learning and data integration. However, existing labeling solutions either incur expensive cost for large datasets or produce noisy ...
Learning rules for multi-label classification: a stacking and a separate-and-conquer approach

Dependencies between the labels are commonly regarded as the crucial issue in multi-label classification. Rules provide a natural way for symbolically describing such relationships. For instance, rules with label tests in the body allow for representing ...
A Cluster-then-label Approach for Few-shot Learning with Application to Automatic Image Data Labeling
Few-shot learning (FSL) aims at learning to generalize from only a small number of labeled examples for a given target task. Most current state-of-the-art FSL methods typically have two limitations. First, they usually require access to a source dataset (...

Comments

Information & Contributors

Information

Published In

SIGMOD '19: Proceedings of the 2019 International Conference on Management of Data

June 2019

2106 pages

ISBN:9781450356435

DOI:10.1145/3299869

General Chairs:
Peter Boncz
CWI & Vrije Universiteit Amsterdam, The Netherlands
,
Stefan Manegold
CWI & Universiteit Leiden, The Netherlands
,
Program Chairs:
Anastasia Ailamaki
EPFL, Switzerland
,
Amol Deshpande
University of Maryland, USA
,
Tim Kraska
MIT, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
the 973 Program of China
the Humanities and Social Sciences Base Foundation of MOE of China

Conference

SIGMOD/PODS '19

Sponsor:

SIGMOD

SIGMOD/PODS '19: International Conference on Management of Data

June 30 - July 5, 2019

Amsterdam, Netherlands

Acceptance Rates

SIGMOD '19 Paper Acceptance Rate 88 of 430 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
435
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)2

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Liu TFan JLi GTang NDu X(2023)Tabular data synthesis with generative adversarial networks: design space and optimizationsThe VLDB Journal10.1007/s00778-023-00807-y33:2(255-280)Online publication date: 15-Aug-2023
https://doi.org/10.1007/s00778-023-00807-y
Chai CWang JLuo YNiu ZLi G(2022)Data Management for Machine Learning: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3148237(1-1)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3148237
Li KLi GWang YHuang YLiu ZWu Z(2021)CrowdRL: An End-to-End Reinforcement Learning Framework for Data Labelling2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00032(289-300)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00032
Heo GRoh YHwang SLee DWhang S(2020)Inspector gadgetProceedings of the VLDB Endowment10.14778/3421424.342142914:1(28-36)Online publication date: 27-Oct-2020
https://dl.acm.org/doi/10.14778/3421424.3421429
Fan JChen JLiu TShen YLi GDu X(2020)Relational data synthesis using generative adversarial networksProceedings of the VLDB Endowment10.14778/3407790.340780213:12(1962-1975)Online publication date: 14-Sep-2020
https://dl.acm.org/doi/10.14778/3407790.3407802

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A game-based framework for crowdsourced data labeling

Learning rules for multi-label classification: a stacking and a separate-and-conquer approach

A Cluster-then-label Approach for Few-shot Learning with Application to Automatic Image Data Labeling