research-article

Public Access

Opening the Black Box of Play: Strategy Analysis of an Educational Game

Authors:

Yetunde Folajimi,

Casper HarteveldAuthors Info & Claims

CHI PLAY '16: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play

Pages 142 - 153

https://doi.org/10.1145/2967934.2968109

Published: 15 October 2016 Publication History

Abstract

A significant issue in research on educational games lies in evaluating their educational impact. Although game analytics is often leveraged in the game industry, it can also provide insight into player actions, strategy development, and the learning process in educational games separate from external evaluation measures. This paper explores the potential of game analytics for learning by analyzing player strategies of an educational game that is designed to support algorithmic thinking. We analyze player strategies from nine cases in our data, combining quantitative and qualitative game analysis techniques: hierarchical player clustering, game progression visualizations, playtraces, and think-aloud data. Results suggest that this combination of data analysis techniques provides insights into level progression and learning strategies that may have been otherwise overlooked.

References

[1]

Mike Ambinder. 2009. Valve's approach to playtesting: The application of empiricism.

[2]

Erik Andersen, Sumit Gulwani, and Zoran Popović. 2013. A Trace-Based Framework for Analyzing and Synthesizing Educational Progressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 773--782.

Digital Library

[3]

Erik Andersen, Yun-En Liu, Ethan Apter, Francois Boucher-Genesse, and Zoran Popović. 2010. Gameplay Analysis Through State Projection. ACM, 1--8.

Digital Library

[4]

CodeCombat. 2016. CodeCombat: Learn to Code by Playing a Game. (2016).

[5]

Adele Cutler and Leo Breiman. 1994. Archetypal analysis. Technometrics 36, 4 (1994).

[6]

Phillip DeRosa. 2007. Tracking player feedback to improve game design. Gamasutra (2007).

[7]

Nicholas Duchenaut, Nick Yee, Eric Nickel, and Robert J. Moore. 2006. Building an MMO with mass appeal: A look at gameplay in World of Warcraft. Game Developer Magazine 1 (2006), 281--317.

[8]

Magy Seif El-Nasr, Anders Drachen, and Alessandro Canossa. 2013. Game analytics: Maximizing the value of player data. Springer Science & Business Media.

Digital Library

[9]

Yetunde Folajimi, Britton Horn, Jacqueline Barnes, Amy Hoover, Gillian Smith, and Casper Harteveld. 2016. A Cross-Cultural Evaluation of a Computer Science Teaching Game. In Proceedings of Games+Learning+Society. ETC Press, Pittsburgh, PA.

[10]

Tracy Fullerton. 2008. Game Design Workshop: A playcentric Approach to Creating Innovative Games (2nd ed.). Morgan Kaufmann.

[11]

Bradley S Greenberg, John Sherry, Kenneth Lachlan, Kristen Lucas, and Amanda Holmstrom. 2010. Orientations to video games among gender and age groups. Simulation & Gaming 41 (2010), 238--259.

Digital Library

[12]

Erik Harpstead and Vincent Aleven. Using Empirical Learning Curve Analysis to Inform Design in an Educational Game. In Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY 2015). ACM Press, 197--207.

Digital Library

[13]

Erik Harpstead, Christopher J MacLellan, Vincent Aleven, and Brad A Myers. 2015. Replay Analysis in Open-Ended Educational Games. In Serious Games Analytics, Christian Sebastian Loh, Yanyan Sheng, and Dirk Ifenthaler (Eds.). Springer International Publishing, 381--399.

[14]

Casper Harteveld. 2011. Triadic game design: Balancing reality, meaning and play. Springer Science & Business Media.

Digital Library

[15]

Casper Harteveld. 2012. Making Sense of Virtual Risks. IOS Press.

[16]

Casper Harteveld, Gillian Smith, Gail Carmichael, Elisabeth Gee, and Carolee Stewart-Gardiner. 2014. A Design-Focused Analysis of Games Teaching Computer Science. In Proceedings of Games+Learning+Society 10. Madison, WI.

[17]

Casper Harteveld and Steven Sutherland. 2015. The goal of scoring: Exploring the role of game performance. In Proceedings of the 2015 ACM Conference on Computer-Human Interaction.

Digital Library

[18]

Eric Hazan. 2013. Contextualizing Data. In Game Analytics. Springer, 477--496.

[19]

David Hilbert and David Redmiles. 2000. Extracting usability information from user interface events. Comput. Surveys 32, 4 (2000), 384--421.

Digital Library

[20]

Britton Horn, Christopher Clark, Oskar Strom, Hilery Chao, Amy J. Stahl, Casper Harteveld, and Gillian Smith. 2016. Design insights into the creation and evaluation of a computer science educational game. In Proceedings of the 47th ACM Technical Symposium on Computer Science Education (SIGCSE). ACM Press, Memphis, TN.

Digital Library

[21]

Ioanna Iacovides, Anna L Cox, Ara Avakian, and Thomas Knoll. Player Strategies: Achieving Breakthroughs and Progressing in Single-player and Cooperative Games. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (CHI PLAY 2014). ACM Press, 131--140.

Digital Library

[22]

Amanda Lenhart and Pew Internet & American Life Project. 2008. Teens, Video Gaming and Civics. Technical Report.

[23]

Vladimir I Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady 10, 8 (1966), 707--710.

[24]

Frederick WB Li and Christopher Watson. 2011. Game-based concept visualization for learning programming. In Proceedings of the third international ACM workshop on Multimedia technologies for distance learning. ACM, 37--42.

Digital Library

[25]

Conor Linehan, George Bellord, Ben Kirman, Zachary H Morford, and Bryan Roche. Learning Curves: Analysing Pace and Challenge in Four Successful Puzzle Games. In Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (CHI PLAY 2014). ACM Press, 181--190.

Digital Library

[26]

Yun-En Liu, Erik Andersen, Richard Snider, Seth Cooper, and Zoran Popović. 2011. Feature-based Projections for Effective Playtrace Analysis. In Proceedings of the 6th International Conference on Foundations of Digital Games (FDG '11). ACM, New York, NY, USA, 69--76.

Digital Library

[27]

Christian Sebastian Loh, Yanyan Sheng, and Dirk Ifenthaler. 2015. Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement. Springer.

[28]

Truong-Huy Dinh Nguyen, Magy Seif El-Nasr, and Alessandro Canossa. 2015. Glyph: Visualization Tool for Understanding Problem Solving Strategies in Puzzle Games. Foundations of Digital Games (FDG) (2015).

[29]

Jackie O'Kelly and J Paul Gibson. 2006. RoboCode & problem-based learning: a non-prescriptive approach to teaching programming. ACM SIGCSE Bulletin 38, 3 (2006), 217--221.

Digital Library

[30]

Joseph C Osborn, Ben Samuel, Joshua Allen McCoy, and Michael Mateas. 2014. Evaluating play trace (dis)similarity metrics. In 10th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment.

[31]

Mikki H Phan, Jo R Jardina, Sloane Hoyle, and Barbara S Chaparro. 2012. Examining the Role of Gender in Video Game Usage, Preference, and Behavior. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. SAGE Publications, 1496--1500.

[32]

Robert C. Prim. 1957. Shortest Connection Networks And Some Generalizations. Bell System Technical Journal 36 (Nov. 1957), 1389--1401.

[33]

Ramon Romero. 2008. Successful instrumentation: Tracking attitudes and behviors to improve games. In Game Developers' Conference.

[34]

Angel Serrano-Laguna, Javier Torrente, Pablo Moreno-Ger, and Baltasar Fernández-Man. 2014. Application of Learning Analytics in Educational Videogames. Entertainment Computing 5 (2014), 313--322.

[35]

David Williamson Shaffer and James Paul Gee. 2012. The Right Kind of GATE: Computer games and the future of assessment. Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Charlotte, NC: Information Age Publishing (2012).

[36]

Valerie Shute and Matthew Ventura. 2013. Stealth assessment: Measuring and supporting learning in video games. MIT Press, Cambridge, MA.

Digital Library

[37]

George Siemens and Ryan S J d Baker. 2012. Learning Analytics and Educational Data Mining: Towards Communication and Collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. ACM, 252--254.

Digital Library

[38]

Catherine A Sugar and Gareth M James. 2003. Finding the Number of Clusters in a Dataset: An Information-Theoretic Approach. J. Amer. Statist. Assoc. (2003), 750--763.

[39]

Robert L Thorndike. 1953. Who belongs in the family? Psychometrika 18 (Dec. 1953), 267--276.

[40]

Allen Tucker, Fadi Deek, Jill Jones, Dennis McCowan, Chris Stephenson, and Anita Verno. 2006. ACM K-12 CS Model Curriculum. Computer Science Teachers Association.

Cited By

Zhou SBakhir N(2024)Interdisciplinary Art Learning Through Artistic Digital Game-Based Learning (DGBL): Evaluating Learning Outcomes and Processes Among Science and Engineering StudentsEmpirical Studies of the Arts10.1177/02762374241246948Online publication date: 16-Apr-2024
https://doi.org/10.1177/02762374241246948
Kleinman EXu JPfau JSeif El-Nasr M(2024)"Trust the Process": An Exploratory Study of Process Visualizations for Self-Reflection in League of LegendsProceedings of the ACM on Human-Computer Interaction10.1145/36771118:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677111
Marques FLignani LQuadros JAmorim MViana WOgasawara Edos Santos J(2024)ProBee: A Provenance-based Design for an Educational Game Analytics ModelTechnology, Knowledge and Learning10.1007/s10758-024-09758-xOnline publication date: 4-Jul-2024
https://doi.org/10.1007/s10758-024-09758-x
Show More Cited By

Index Terms

Opening the Black Box of Play: Strategy Analysis of an Educational Game

Recommendations

A Gateway Game to Make Computational Problem Solving Accessible and Engaging (Abstract Only)
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

In this work, we seek to better understand how game design principles impact the effectiveness of a video game learning environment for computational problem solving. Video games naturally encourage problem solving and incremental learning of their game ...
Clashroom: A Game to Enhance the Classroom Experience (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

Clashroom is a web-based fantasy game that was created to enhance the motivation and learning of a traditional classroom. It was created to address several drawbacks of other educational games by being broadly applicable to any course, requiring no ...
Evaluation of the user experience and intrinsic motivation with educational and mainstream digital games
CLIHC '15: Proceedings of the Latin American Conference on Human Computer Interaction

Many studies of digital games have been developed to investigate a series of attributes to improve the games industry and to know the impacts of the use of digital games on users. Many arguments exist stating that the appeal of mainstream digital games ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI PLAY '16: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play

October 2016

424 pages

ISBN:9781450344562

DOI:10.1145/2967934

General Chairs:
Anna Cox
University College London, UK
,
Phoebe O. Toups Dugas
New Mexico State University, USA
,
Program Chairs:
Regan L. Mandryk
University of Saskatchewan, Canada
,
Paul Cairns
University of York, UK

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CHI PLAY '16

Sponsor:

SIGCHI

CHI PLAY '16: The annual symposium on Computer-Human Interaction in Play

October 16 - 19, 2016

Texas, Austin, USA

Acceptance Rates

CHI PLAY '16 Paper Acceptance Rate 36 of 124 submissions, 29%;

Overall Acceptance Rate 421 of 1,386 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
760
Total Downloads

Downloads (Last 12 months)117
Downloads (Last 6 weeks)25

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou SBakhir N(2024)Interdisciplinary Art Learning Through Artistic Digital Game-Based Learning (DGBL): Evaluating Learning Outcomes and Processes Among Science and Engineering StudentsEmpirical Studies of the Arts10.1177/02762374241246948Online publication date: 16-Apr-2024
https://doi.org/10.1177/02762374241246948
Kleinman EXu JPfau JSeif El-Nasr M(2024)"Trust the Process": An Exploratory Study of Process Visualizations for Self-Reflection in League of LegendsProceedings of the ACM on Human-Computer Interaction10.1145/36771118:CHI PLAY(1-28)Online publication date: 15-Oct-2024
https://dl.acm.org/doi/10.1145/3677111
Marques FLignani LQuadros JAmorim MViana WOgasawara Edos Santos J(2024)ProBee: A Provenance-based Design for an Educational Game Analytics ModelTechnology, Knowledge and Learning10.1007/s10758-024-09758-xOnline publication date: 4-Jul-2024
https://doi.org/10.1007/s10758-024-09758-x
Giannakoulas AXinogalos S(2023)Studying the effects of educational games on cultivating computational thinking skills to primary school students: a systematic literature reviewJournal of Computers in Education10.1007/s40692-023-00300-z11:4(1283-1325)Online publication date: 21-Nov-2023
https://doi.org/10.1007/s40692-023-00300-z
Joshi AMousas CHarrell DKao D(2022)Exploring the Influence of Demographic Factors on Progression and Playtime in Educational GamesProceedings of the 17th International Conference on the Foundations of Digital Games10.1145/3555858.3555873(1-15)Online publication date: 5-Sep-2022
https://dl.acm.org/doi/10.1145/3555858.3555873
Perez-Colado IPerez-Colado VMartinez-Ortiz IFreire MFernandez-Manjon B(2022)Using e-Learning Standards to Improve Serious Game Deployment and Evaluation2022 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON52537.2022.9766573(2077-2083)Online publication date: 28-Mar-2022
https://doi.org/10.1109/EDUCON52537.2022.9766573
Han SLiu MPan ZCai YShao P(2022)Making FAQ Chatbots More Inclusive: An Examination of Non-Native English Users’ Interactions with New Technology in Massive Open Online CoursesInternational Journal of Artificial Intelligence in Education10.1007/s40593-022-00311-433:3(752-780)Online publication date: 1-Sep-2022
https://doi.org/10.1007/s40593-022-00311-4
Kleinman EVillareale JShergadwala MTeng ZBryant AZhu JEl-Nasr M(2022)Towards an Understanding of How Players Make Meaning from Post-Play Process VisualizationsEntertainment Computing – ICEC 202210.1007/978-3-031-20212-4_4(47-58)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-20212-4_4
Esclamado MRodrigo M(2022)Are All Who Wander Lost? An Exploratory Analysis of Learner Traversals of Minecraft WorldsArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium10.1007/978-3-031-11647-6_48(263-266)Online publication date: 26-Jul-2022
https://doi.org/10.1007/978-3-031-11647-6_48
Pan ZCui YLeighton JCutumisu M(2022)Insights into computational thinking from think‐aloud interviews: A systematic reviewApplied Cognitive Psychology10.1002/acp.402937:1(71-95)Online publication date: 23-Dec-2022
https://doi.org/10.1002/acp.4029
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten