poster

An analysis of crowd workers mistakes for specific and complex relevance assessment task

Authors:

Jesse Anderton,

Maryam Bashir,

Virgil Pavlu,

Javed A. AslamAuthors Info & Claims

CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge Management

Pages 1873 - 1876

https://doi.org/10.1145/2505515.2507884

Published: 27 October 2013 Publication History

Get Access

Abstract

The TREC 2012 Crowdsourcing track asked participants to crowdsource relevance assessments with the goal of replicating costly expert judgements with relatively fast, inexpensive, but less reliable judgements from anonymous online workers. The track used 10 "ad-hoc" queries, highly specific and complex (as compared to web search). The crowdsourced assessments were evaluated against expert judgments made by highly trained and capable human analysts in 1999 as part of ad hoc track collection construction. Since most crowdsourcing approaches submitted to the TREC 2012 track produced assessment sets nowhere close to the expert judgements, we decided to analyze crowdsourcing mistakes made on this task using data we collected via Amazon's Mechanical Turk service. We investigate two types of crowdsourcing approaches: one that asks for nominal relevance grades for each document, and the other that asks for preferences on many (not all) pairs of documents.

References

[1]

C. Buckley, D. Dimmick, I. Soboroff, and E. M. Voorhees. Bias and the limits of pooling. In SIGIR, pages 619--620, 2006.

Digital Library

Google Scholar

[2]

B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there. In ECIR, 2008.

Crossref

Google Scholar

[3]

M. Hosseini, I. J. Cox, N. Mili--c-Frayling, G. Kazai, and V. Vinay. On aggregating labels from multiple crowd workers to infer relevance of documents. In ECIR. Springer-Verlag, 2012.

Digital Library

Google Scholar

[4]

P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In SIGKDD Workshop on Human Computation. ACM, 2010.

Digital Library

Google Scholar

[5]

M. D. Smucker, G. Kazai, and M. Lease. Overview of the trec 2012 crowdsourcing track. In TREC 2012.

Google Scholar

Cited By

View all

Xu JHan LSadiq SDemartini G(2023)On the role of human and machine metadata in relevance judgment tasksInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10317760:2Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.ipm.2022.103177
Roitero KMaddalena EMizzaro SScholer F(2021)On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10268858:6Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.ipm.2021.102688
Daniel FKucherbaev PCappiello CBenatallah BAllahbakhsh M(2018)Quality Control in CrowdsourcingACM Computing Surveys10.1145/314814851:1(1-40)Online publication date: 4-Jan-2018
https://dl.acm.org/doi/10.1145/3148148
Show More Cited By

Index Terms

An analysis of crowd workers mistakes for specific and complex relevance assessment task
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization
SIGIR '12: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval

In crowdsourced relevance judging, each crowd worker typically judges only a small number of examples, yielding a sparse and imbalanced set of judgments in which relatively few workers influence output consensus labels, particularly with simple ...
Using crowdsourcing for TREC relevance assessment

Crowdsourcing has recently gained a lot of attention as a tool for conducting different kinds of relevance evaluations. At a very high level, crowdsourcing describes outsourcing of tasks to a large group of people instead of assigning such tasks to an ...

Comments

Information & Contributors

Information

Published In

CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge Management

October 2013

2612 pages

ISBN:9781450322638

DOI:10.1145/2505515

General Chairs:
Qi He
LinkedIn, USA
,
Arun Iyengar
IBM T.J. Watson Research Center, USA
,
Program Chairs:
Wolfgang Nejdl
L3S Research Center, Germany
,
Jian Pei
Simon Fraser University, Canada
,
Rajeev Rastogi
Amazon, India

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster

Conference

CIKM'13

Sponsor:

CIKM'13: 22nd ACM International Conference on Information and Knowledge Management

October 27 - November 1, 2013

California, San Francisco, USA

Acceptance Rates

CIKM '13 Paper Acceptance Rate 143 of 848 submissions, 17%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
244
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Xu JHan LSadiq SDemartini G(2023)On the role of human and machine metadata in relevance judgment tasksInformation Processing and Management: an International Journal10.1016/j.ipm.2022.10317760:2Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.ipm.2022.103177
Roitero KMaddalena EMizzaro SScholer F(2021)On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluationInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10268858:6Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1016/j.ipm.2021.102688
Daniel FKucherbaev PCappiello CBenatallah BAllahbakhsh M(2018)Quality Control in CrowdsourcingACM Computing Surveys10.1145/314814851:1(1-40)Online publication date: 4-Jan-2018
https://dl.acm.org/doi/10.1145/3148148
Maddalena EMizzaro SScholer FTurpin A(2017)On Crowdsourcing Relevance Magnitudes for Information Retrieval EvaluationACM Transactions on Information Systems10.1145/300217235:3(1-32)Online publication date: 4-Jan-2017
https://dl.acm.org/doi/10.1145/3002172

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

Inferring missing relevance judgments from crowd workers via probabilistic matrix factorization

Using crowdsourcing for TREC relevance assessment