research-article

Wormingo: a 'true gamification' approach to anaphoric annotation

Authors:

Doruk Kicikoglu,

Richard Bartle,

Jon Chamberlain,

Massimo PoesioAuthors Info & Claims

FDG '19: Proceedings of the 14th International Conference on the Foundations of Digital Games

Article No.: 75, Pages 1 - 7

https://doi.org/10.1145/3337722.3341868

Published: 26 August 2019 Publication History

Abstract

In this paper we present Wormingo, ¹ a new Game-with-a-Purpose for anaphoric annotation. It introduces the motivation-annotation paradigm which uses linguistic puzzles and other widely known gamification techniques and word game mechanics to motivate players to carry out anaphoric annotation tasks. In a preliminary experiment, the game was tested on 270 players recruited through the Reddit platform, achieving promising results.

References

[1]

Tyler Baron and Ashish Amresh. 2015. Word towers: Assessing domain knowledge with non-traditional genres. In Proceedings of the European Conference on Games-based Learning, Vol. 2015-January. Dechema e.V., 638--645.

[2]

Luke Barrington, Damien O'Malley, Douglas Turnbull, and Gert Lanckriet. 2009. User-centered Design of a Social Game to Tag Music. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09). ACM, New York, NY, USA, 7--10.

Digital Library

[3]

Karën Fort Bruno Guillaume and Nicolas Lefebvre. 2016. Crowdsourcing complex language resources: Playing to annotate dependency syntax. In Proc. of COLING.

[4]

J. Chamberlain. 2016. Harnessing Collective Intelligence on Social Networks. Ph.D. Dissertation. University of Essex, School of Computer Science and Electronic Engineering.

[5]

Jon Chamberlain, Richard Bartle, Udo Kruschwitz, Chris Madge, and Massimo Poesio. 2017. Metrics of games-with-a-purpose for NLP applications. (2017), 2.

[6]

Jon Chamberlain, Karën Fort, Udo Kruschwitz, Mathieu Lafourcade, and Massimo Poesio. 2013. Using Games to Create Language Resources: Successes and Limitations of the Approach. 3--44. <sub>1</sub>

[7]

Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase Detectives - A Web-based Collaborative Annotation Game. In In Proceedings of I-Semantics.

[8]

Seth Cooper, Firsas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and the Foldit Players. 2010. Predicting protein structures with a multiplayer online game. Nature 466 (2010), 756--760.

[9]

Greg Costikyan. 2013. Uncertainty in Games. The MIT Press.

Digital Library

[10]

Anca Dumitrache, Lora Aroyo, Chris Welty, Robert-Jan Sips, and Anthony Levas. 2013. "Dr. Detective": combining gamification techniques and crowdsourcing to create a gold standard in medical text. In Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web, Sydney, Australia, October 19, 2013. 16--31. http://ceur-ws.org/Vol-1030/paper-02.pdf

Digital Library

[11]

Dagmara Dziedzic. 2016. Use of the Free to Play model in games with a purpose: The RoboCorp game case study. Bio-Algorithms and Med-Systems 12 (11 2016), 187--197.

[12]

L. Gu and A. L. Jia. 2018. Player Activity and Popularity in Online Social Games and their Implications for Player Retention. In 2018 16th Annual Workshop on Network and Systems Support for Games (NetGames). 1--6.

Digital Library

[13]

Barbora Hladka, Jirí Mírovský, and Pavel Schlesinger. 2009. Play the Language: Play Coreference. 209--212.

Digital Library

[14]

David Jurgens and Roberto Navigli. 2014. It's All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation. Transactions of the Association for Computational Linguistics 2 (2014), 449--464. https://www.aclweb.org/anthology/Q14-1035

[15]

Jesper Juul. 2008. The Magic Circle and the Puzzle Piece. (01 2008), 56--67.

[16]

Firas Khatib, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popović, David Baker, and Foldit Players. 2011. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108, 47 (2011), 18949--18953. arXiv:https://www.pnas.org/content/108/47/18949.full.pdf

[17]

Mathieu Lafourcade. 2007. Making people play for Lexical Acquisition with the JeuxDeMots prototype. In Proc. of SNLP.

[18]

Mathieu Lafourcade, Alain Joubert, and Nathalie Le Brun. 2015. Games with a Purpose (GWAPS) (Focus Series in Cognitive Science and Knowledge Management). Wiley-ISTE.

[19]

Chris J. Lintott, Kate Land, Kevin Schawinski, Anže Slosar, Daniel Thomas, Robert C. Nichol, Steven Bamford, Alex Szalay, Jan Vandenberg, M. Jordan Raddick, Dan Andreescu, and Phil Murray. 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey*. Monthly Notices of the Royal Astronomical Society 389, 3 (09 2008), 1179--1189. arXiv:http://oup.prod.sis.lan/mnras/article-pdf/389/3/1179/3325962/mnras0389-1179.pdf

[20]

Allan Paivio, T. B. Rogers, and Padric C. Smythe. 1968. Why are pictures easier to recall than words? Psychonomic Science 11, 4 (01 Apr 1968), 137--138.

[21]

José Pedro Pinto and Paula Viana. 2013. TAG4VD: A Game for Collaborative Video Annotation. In Proceedings of the 2013 ACM International Workshop on Immersive Media Experiences (ImmersiveMe '13). ACM, New York, NY, USA, 25--28.

Digital Library

[22]

Massimo Poesio. 2016. Linguistic and Cognitive Evidence About Anaphora. In Anaphora Resolution: Algorithms, Resources and Applications, M. Poesio, R. Stuckardt, and Y. Versley (Eds.). Springer, Chapter 2.

[23]

Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing Collective Intelligence for Internet-scale Language Resource Creation. ACM Trans. Interact. Intell. Syst. 3, 1, Article 3 (April 2013), 44 pages.

Digital Library

[24]

Pramila Rani, Nilanjan Sarkar, and Changchun Liu. 2005. Maintaining optimal challenge in computer games through real-time physiological feedback. In Proceedings of the 11th international conference on human computer interaction, Vol. 58. 22--27.

[25]

Maria Rasmusson and Monica Eklund. 2013. "It's easier to read on the Internet---you just click on what you want to read...". Education and Information Technologies 18, 3 (01 Sep 2013), 401--419.

Digital Library

[26]

Noortje J. Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for Word Sense Labeling. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) - Short Papers. Association for Computational Linguistics, Potsdam, Germany, 397--403. https://www.aclweb.org/anthology/W13-0215

[27]

Luis Von Ahn. 2006. Games with a purpose. Computer 39, 6 (2006), 92--94.

Digital Library

[28]

Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 319--326.

Digital Library

[29]

Luis von Ahn and Laura Dabbish. 2008. Designing Games with a Purpose. Commun. ACM 51, 8 (Aug. 2008), 58--67.

Digital Library

[30]

Gabe Zichermann and Christopher Cunningham. 2011. Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps (1st ed.). O'Reilly Media, Inc.

Digital Library

Cited By

Hinterreiter SSpinde TOberdörfer SEchizen ILatoschik M(2024)News Ninja: Gamified Annotation of Linguistic Bias in Online NewsProceedings of the ACM on Human-Computer Interaction10.1145/36770928:CHI PLAY(1-29)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677092
Aliady WPoesio M(2024)Master the Linguistic Landscape: Puzzle Integration in a 3D NLP Game2024 IEEE Conference on Games (CoG)10.1109/CoG60054.2024.10645588(1-8)Online publication date: 5-Aug-2024
https://doi.org/10.1109/CoG60054.2024.10645588
Althani FMadge CPoesio M(2023)The Onboarding Phase in a Game for Text Labelling: Comparing the Effect of Animated vs. Textual Onboarding on Player Experience and Accuracy2023 IEEE Conference on Games (CoG)10.1109/CoG57401.2023.10333246(1-8)Online publication date: 21-Aug-2023
https://doi.org/10.1109/CoG57401.2023.10333246
Show More Cited By

Wormingo: a 'true gamification' approach to anaphoric annotation

Recommendations

Assessing the benefits of gamification in mathematics for student gameful experience and gaming motivation
Abstract
This study addresses the question of how to use game design elements to raise students' motivation to engage in mathematics learning activities. Four conditions of mathematics learning activities were designed and assessed: 1. a ...
Highlights
- Effect of four gamified activities on game experience and motivation was checked.
Got skillz?: player matching, mastery, and engagement in skill-based games
Gamification '13: Proceedings of the First International Conference on Gameful Design, Research, and Applications

In this paper we explore mastery, a powerful intrinsic motivator, and its relationship to player engagement in a mobile skill-based game platform (Skillz.com). More specifically, we present a unique Elo-based rating system for quickly matching players ...
Forty simple computer games and what they could mean to educators

The purpose of this study was to evaluate computer games or components of the games that would lend themselves for use in an educational setting. Participants included 20 men and 20 women who volunteered to participate. All participants played four ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

FDG '19: Proceedings of the 14th International Conference on the Foundations of Digital Games

August 2019

822 pages

ISBN:9781450372176

DOI:10.1145/3337722

Conference Chair:
Sebastian Deterding
University of York
,
General Chair:
Foaad Khosmood
California Polytechnic State University
,
Program Chairs:
Johanna Pirker
Graz University of Technology
,
Thomas Apperley
CEGCS Tampere University

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 August 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Research Council

Conference

FDG '19

FDG '19: The Fourteenth International Conference on the Foundations of Digital Games

August 26 - 30, 2019

California, San Luis Obispo, USA

Acceptance Rates

FDG '19 Paper Acceptance Rate 46 of 124 submissions, 37%;

Overall Acceptance Rate 152 of 415 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
182
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)1

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hinterreiter SSpinde TOberdörfer SEchizen ILatoschik M(2024)News Ninja: Gamified Annotation of Linguistic Bias in Online NewsProceedings of the ACM on Human-Computer Interaction10.1145/36770928:CHI PLAY(1-29)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677092
Aliady WPoesio M(2024)Master the Linguistic Landscape: Puzzle Integration in a 3D NLP Game2024 IEEE Conference on Games (CoG)10.1109/CoG60054.2024.10645588(1-8)Online publication date: 5-Aug-2024
https://doi.org/10.1109/CoG60054.2024.10645588
Althani FMadge CPoesio M(2023)The Onboarding Phase in a Game for Text Labelling: Comparing the Effect of Animated vs. Textual Onboarding on Player Experience and Accuracy2023 IEEE Conference on Games (CoG)10.1109/CoG57401.2023.10333246(1-8)Online publication date: 21-Aug-2023
https://doi.org/10.1109/CoG57401.2023.10333246
Madge CBrightmore JKicikoglu DAlthani FBartle RChamberlain JKruschwitz UPoesio M(2022)LingoTowns: A Virtual World For Natural Language Annotation and Language LearningExtended Abstracts of the 2022 Annual Symposium on Computer-Human Interaction in Play10.1145/3505270.3558323(57-62)Online publication date: 2-Nov-2022
https://dl.acm.org/doi/10.1145/3505270.3558323
Bonetti FTonelli S(2021)Measuring Orthogonal Mechanics in Linguistic Annotation GamesProceedings of the ACM on Human-Computer Interaction10.1145/34746925:CHI PLAY(1-16)Online publication date: 6-Oct-2021
https://dl.acm.org/doi/10.1145/3474692
Hinterreiter S(2021)A Gamified Approach To Automatically Detect Biased Wording And Train Critical Reading2021 International Conference on Data Mining Workshops (ICDMW)10.1109/ICDMW53433.2021.00141(1085-1086)Online publication date: Dec-2021
https://doi.org/10.1109/ICDMW53433.2021.00141

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents