Article

"Dr. Detective": combining gamication techniques and crowdsourcing to create a gold standard in medical text

Authors:

Anca Dumitrache,

Robert-Jan Sips,

Anthony LevasAuthors Info & Claims

CrowdSem'13: Proceedings of the 1st International Conference on Crowdsourcing the Semantic Web - Volume 1030

Pages 16 - 31

Published: 19 October 2013 Publication History

Abstract

This paper proposes a design for a gamified crowdsourcing workflow to extract annotation from medical text. Developed in the context of a general crowdsourcing platform, Dr. Detective is a game with a purpose that engages medical experts into solving annotation tasks on medical case reports, tailored to capture disagreement between annotators. It incorporates incentives such as learning features, to motivate a continuous involvement of the expert crowd. The game was designed to identify expressions valuable for training NLP tools, and interpret their relation in the context of medical diagnosing. In this way, we can resolve the main problem in gathering ground truth from experts - that the low inter-annotator agreement is typically caused by different interpretations of the text. We report on the results of a pilot study assessing the usefulness of this game. The results show that the quality of the annotations by the expert crowd are comparable to those of an NLP parser. Furthermore, we observed that allowing game users to access each others' answers increases agreement between annotators.

References

[1]

Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building Watson: An Overview of the DeepQA Project. AI Magazine 31 (2010) 59-79.

[2]

Kalyanpur, A., Boguraev, B., Patwardhan, S., Murdock, J.W., Lally, A., Welty, C., Prager, J.M., Coppola, B., Fokoue-Nkoutche, A., Zhang, L., et al.: Structured data and inference in DeepQA. IBM Journal of Research and Development 56(3.4) (2012) 10-1.

[3]

Aroyo, L., Welty, C.: Harnessing disagreement for event semantics. Detection, Representation, and Exploitation of Events in the Semantic Web 31.

[4]

Aroyo, L., Welty, C.: Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013. ACM (2013).

[5]

Inel, O., Aroyo, L., Welty, C., Sips, R.J.: Exploiting Crowdsourcing Disagreement with Various Domain-Independent Quality Measures. Technical report, VU University Amsterdam (July 2013). http://crowd-watson.nl/tech-reports/20130702.pdf

[6]

Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (2004) 319-326.

[7]

Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., Blum, M.: recaptcha: Human-based character recognition via web security measures. Science 321(5895) (2008) 1465-1468.

[8]

Lintott, C.J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M.J., Nichol, R.C., Szalay, A., Andreescu, D., et al.: Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 389(3) (2008) 1179-1189.

[9]

Siorpaes, K., Hepp, M.: Games with a Purpose for the Semantic Web. Intelligent Systems, IEEE 23(3) (2008) 50-60.

[10]

Chamberlain, J., Poesio, M., Kruschwitz, U.: Phrase Detectives - A Web-based Collaborative Annotation Game. In: Proceedings of I-Semantics. (2008).

[11]

Scharl, A., Sabou, M., Gindl, S., Rafelsberger, W., Weichselbraun, A.: Leveraging the Wisdom of the Crowds for the Acquisition of Multilingual Language Resources. In: Proc. 8th LREC - International Conference on Language Resources and Evaluation. (2012).

[12]

Lin, H.: Crowd Watson: Crowdsourced Text Annotations. Technical report, VU University Amsterdam (July 2013). http://crowd-watson.nl/tech-reports/20130704.pdf

[13]

Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research 32(suppl 1) (2004) D267-D270.

[14]

Goldberg, H.S., Hsu, C., Law, V., Safran, C.: Validation of clinical problems using a UMLS-based semantic parser. In: Proceedings of the AMIA Symposium, American Medical Informatics Association (1998) 805.

[15]

Aroyo, L., Welty, C.: Measuring crowd truth for medical relation extraction. In: AAAI 2013 Fall Symposium on Semantics for Big Data (in print). (2013).

[16]

Soberón, G., Aroyo, L., Welty, C., Inel, O., Lin, H., Overmeen, M.: Crowd Truth Metrics. Technical report, VU University Amsterdam (July 2013). http://crowd-watson.nl/tech-reports/20130703.pdf

[17]

Tokarchuk, O., Cuel, R., Zamarian, M.: Analyzing crowd labor and designing incentives for humans in the loop. IEEE Internet Computing 16(5) (2012) 0045-51.

[18]

McLaughlin, G.H.: SMOG grading: A new readability formula. Journal of reading 12(8) (1969) 639-646.

[19]

Doak, C.C., Doak, L.G., Root, J.H.: Teaching patients with low literacy skills. AJN The American Journal of Nursing 96(12) (1996) 16M.

[20]

Von Ahn, L., Dabbish, L.: Designing games with a purpose. Communications of the ACM 51(8) (2008) 58-67.

[21]

Sherry, J.L.: Flow and media enjoyment. Communication Theory 14(4) (2004) 328-347.

Cited By

Kicikoglu DBartle RChamberlain JPoesio MDeterding SKhosmood FPirker JApperley T(2019)WormingoProceedings of the 14th International Conference on the Foundations of Digital Games10.1145/3337722.3341868(1-7)Online publication date: 26-Aug-2019
https://dl.acm.org/doi/10.1145/3337722.3341868
Amorim MSaleme EAssis Neto FSantos CGhinea G(2019)Crowdsourcing authoring of sensory effects on videosMultimedia Tools and Applications10.1007/s11042-019-7312-278:14(19201-19227)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1007/s11042-019-7312-2
Afentoulidis GSzlávik ZYang JBozzon AAkkermans HFontaine KVermeulen IHouben GWeber M(2018)Social Gamification in Enterprise CrowdsourcingProceedings of the 10th ACM Conference on Web Science10.1145/3201064.3201094(135-144)Online publication date: 15-May-2018
https://dl.acm.org/doi/10.1145/3201064.3201094
Show More Cited By

"Dr. Detective": combining gamication techniques and crowdsourcing to create a gold standard in medical text
1. Human-centered computing

Recommendations

Constructing Sentiment Lexicon with Game for Annotation Collection
Statistical Language and Speech Processing
Abstract
While research of sentiment analysis became very popular on the global scope, in Slovak language as an under-resourced language there are still many issues to be tackled, especially the lack of resources. In this paper, we introduce a sentiment ...
Inducing Gazetteer for Chinese Named Entity Recognition Based on Local High-Frequent Strings
FITME '09: Proceedings of the 2009 Second International Conference on Future Information Technology and Management Engineering

Gazetteers, or entity dictionaries, are important for named entity recognition (NER). Although the dictionaries extracted automatically by the previous methods from a corpus, web or Wikipedia are very huge, they also misses some entities, especially the ...
Building a semantically annotated corpus of clinical texts

In this paper, we describe the construction of a semantically annotated corpus of clinical texts for use in the development and evaluation of systems for automatically extracting clinically significant information from the textual component of patient ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

CrowdSem'13: Proceedings of the 1st International Conference on Crowdsourcing the Semantic Web - Volume 1030

October 2013

95 pages

Editors:
Maribel Acosta
KIT, Germany
,
Lora Aroyo
VUA, Netherlands
,
Abraham Bernstein
University of Zurich, Switzerland
,
Jens Lehmann
University of Leipzig, Germany
,
Natasha Noy
Stanford University

Publisher

CEUR-WS.org

Aachen, Germany

Publication History

Published: 19 October 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
14
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kicikoglu DBartle RChamberlain JPoesio MDeterding SKhosmood FPirker JApperley T(2019)WormingoProceedings of the 14th International Conference on the Foundations of Digital Games10.1145/3337722.3341868(1-7)Online publication date: 26-Aug-2019
https://dl.acm.org/doi/10.1145/3337722.3341868
Amorim MSaleme EAssis Neto FSantos CGhinea G(2019)Crowdsourcing authoring of sensory effects on videosMultimedia Tools and Applications10.1007/s11042-019-7312-278:14(19201-19227)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1007/s11042-019-7312-2
Afentoulidis GSzlávik ZYang JBozzon AAkkermans HFontaine KVermeulen IHouben GWeber M(2018)Social Gamification in Enterprise CrowdsourcingProceedings of the 10th ACM Conference on Web Science10.1145/3201064.3201094(135-144)Online publication date: 15-May-2018
https://dl.acm.org/doi/10.1145/3201064.3201094
Assis Neto FSantos C(2018)Understanding crowdsourcing projectsInformation Processing and Management: an International Journal10.1016/j.ipm.2018.03.00654:4(490-506)Online publication date: 1-Jul-2018
https://dl.acm.org/doi/10.1016/j.ipm.2018.03.006
Dumitrache A(2015)Crowdsourcing Disagreement for Collecting Semantic AnnotationProceedings of the 12th European Semantic Web Conference on The Semantic Web. Latest Advances and New Domains - Volume 908810.1007/978-3-319-18818-8_43(701-710)Online publication date: 31-May-2015
https://dl.acm.org/doi/10.1007/978-3-319-18818-8_43

View Options

View options

Figures

Tables

Media

View Table of Conten