Abstract
This chapter introduces the approach taken by the Games for Learning Institute (G4LI) to assess learning and related learner variables, with a focus on the use of metrics obtained during game play and simulation exploration. Learning is fundamental to all games (Gee 2008). At minimum, players must learn the basics of a game’s mechanics to play. Additionally, players must uncover what these mechanics are for, and what the game designer wants them to do (Cook 2006). Feedback mechanisms are an example of how game designers encourage (reward) or discourage (punish) a behavior. Game mechanics for learning must incorporate all of these aspects, from the moment-to-moment activities in which players engage, to reward and punishment systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We thank Bill Shribman, one of our very thoughtful reviewers, for raising these questions.
Recommended Readings
Iseli, M. R., Koenig, A. D., Lee, J. J., & Wainess, R. (2010). Automatic assessment of complex task performance in games and simulations (CRESST Report 775). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., & Wixon, D. (2008). Tracking real-time user experience (TRUE): A comprehensive instrumentation solution for complex systems. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems (pp. 443–452). Florence: ACM. doi:10.1145/1357054.1357126
Plass, J. L., O’Keefe, P., Homer, B. D., Case, J., Hayward, E. O., Stein, M., & Perlin, K. (in press). Motivational and cognitive outcomes associated with individual, competitive, and collaborative game play. Special issue on advanced learning technologies. Journal of Educational Psychology.
Plass, J. L., Homer, B. D., Kinzer, C., Frye, J., & Perlin, K. (2011, September 30). Learning mechanics and assessment mechanics for games for learning (G4LI White Paper # 01/2011 Version 0.1). Available online at www.g4li.org
Shute, V. J., Ventura, M., Bauer, M. I., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In U. Ritterfeld, M. Cody, & P. Vorderer (Eds.), Serious games: Mechanisms and effects (pp. 295–321). Mahwah: Routledge/Taylor & Francis.
References
Aleven, V., & Koedinger, K. R. (2000). Limitations of student control: Do students know when they need help? In Gauthier, G., Frasson, C., & VanLehn, K. (Eds.), Proceedings of the 5th international conference on intelligent tutoring systems, ITS 2000 (pp. 292–303). Berlin: Springer.
Ambinder, M. (2009). Valve’s approach to playtesting: The application of empiricism. Presented at the 2009 Game Developers Conference, San Francisco.
America’s Army. (2012). America’s Army: Honor and advances – Army values. Retrieved July 3, 2012 from http://manual.americasarmy.com/index.php/Honor_and_Advancement
Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students’ learning strategies and motivation processes. Journal of Educational Psychology, 80(3), 260–267.
Anderson, A., & Bavelier, D. (2011). Action game play as a tool to enhance perception, attention and cognition. In S. Tobias & J. D. Fleycher (Eds.), Computer games and instruction (pp. 307–330). Charlotte: Information Age Publishing: IAP.
Brown, T. L. (1997). Task analysis strategies and practices (Practices Application Brief). Washington, DC: Office of Educational Research and Improvement. (ERIC Document Reproduction Service No. ED 404 571).
Chang, Y. K. (2010). Examining metacognitive processes in exploratory computer-based learning environments using activity log analysis. Unpublished Doctoral Dissertation, New York University, New York.
Chang, Y. K., & Plass, J. L. (2012). Assessment of the metacognitive processes from the behavioral data. (G4LI White Paper # 01/2012). Available online at www.g4li.org
Chang, Y. K., Plass, J. L., & Homer, B. D. (2008, October). Development and validation of a behavioral measure of metacognitive processes (BMMP). Featured research presentation at the annual convention of the Association for Educational Communication and Technology (AECT), Orlando, FL.
Cognition and Technology Group at Vanderbilt. (1990). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19, 2–10.
Cognition and Technology Group at Vanderbilt. (1993). Anchored instruction and situated cognition revisited. Educational Technology, 33(3), 52–70.
Collins, A. (1988). Cognitive apprenticeship and instructional technology. Hillsdale: Lawrence Erlbaum.
Cook, D. (2006). What are game mechanics? lostgarden.com. Retrieved May 23, 2010 from http://lostgarden.com/2006/10/what-are-game-mechanics.html
Cordingley, E. S. (1989). Knowledge elicitation techniques for knowledge-based systems. In D. Diaper (Ed.), Knowledge elicitation: Principles, techniques, and application (pp. 89–175). New York: Wiley.
DeRosa, P. (2007, August 7). Tracking player feedback to improve gamedesign. Gamasutra.
Domagk, S., Schwartz, R., & Plass, J. L. (2010). Defining interactivity in multimedia learning. Computers in Human Behavior, 26, 1024–1033. doi:10.1016/j.chb.2010.03.003.
Ducheneaut, N., Moore, R. J., & Nickell, E. (2004). Designing for sociability in massively multiplayer games: An examination of the “third places” of SWG. In J. H. Smith & M. Sicart (Eds.), Proceedings of the Other Players Conference. Copenhagen, Denmark.
Efklides, A. (2002). They systemic nature of metacognitive experiences: Feelings, judgment, and their interrelation. In M. Izaute, P. Chambres, & P.-J. Marescaux (Eds.), Metacognition: Process, function, and use (pp. 19–34). Dordrecht: Kluwer.
Elliot, A. J. (2005). A conceptual history of the achievement goal construct. In A. J. Elliot & C. S. Dweck (Eds.), Handbook of competence and motivation (pp. 52–72). New York: Guilford Publications.
Gee, J. P. (2008). What video games have to teach us about learning and literacy (Rev. and updated). Basingstoke: Palgrave Macmillan.
Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15, 1–38.
Heeter, C., Magerko, B., Medler, B., & Fitzgerald, J. (2009). Game design and the challenge-avoiding ‘Validator’ player type. International Journal of Gaming and Computer-Mediated Simulations, 1(3), 53–67.
Hill, J. R., & Hannafin, M. J. (2001). Teaching and learning in digital environments: The resurgence of resource-based learning. Educational Technology Research and Development, 49, 15–26.
Hunicke, R., LeBlanc, M., & Zubek, R. (2004). MDA: A formal approach to game design and game research. In Proceedings of the Challenges in Game AI Workshop, 19th National Conference on Artificial Intelligence, AAAI’04, San Jose, CA. Vancouver: AAAI Press.
Isbister, K., & Schaffer, N. (2008). Game usability. Advice from the experts for advancing the payer experience. New York: Morgan Kaufman.
Isbister, K., Flanagan, M., & Hash, C. (2010). Designing games for learning: Insights from conversations with designers. In Proceedings of CHI (Conference on human factors in computing), Atlanta, GA.
Järvinen, A. (2008). Games without frontiers: Theories and methods for game studies and design. Tampere: Tampere University Press.
Juul, J. (2003). The game, the player, the world: Looking for a heart of gameness. In M. Copier & J. Raessens (Eds.), Level up: Digital games research conference proceedings (pp. 30–45). Utrecht: Utrecht University.
Lajoie, S. P., Azevedo, R., & Fleiszer, D. M. (1998). Cognitive tools for assessment and learning in a high flow information environment. Journal of Educational Computing Research, 18(3), 205–235.
Lave, J., & Wenger, E. (1990). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.
Liu, M. (1998). A study of engaging high-school students as multimedia designers in a cognitive apprenticeship-style learning environment. Computers in Human Behavior, 14(3), 387–415.
McCombs, B. L. (2001). Self-regulated learning and academic achievement: A phenomenological view. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (pp. 67–123). Hillsdale: Lawrence Erlbaum Associates.
Mislevy, R. J., & Gitomer, D. H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253–282.
Mislevy, R. J., & Riconscente, M. (2005). Evidence-centered assessment design: Layers, structures, and terminology (PADI Technical Report 9). Menlo Park: SRI International.
Mislevy, R. J., Steinberg, L. S., Breyer, F. J., Almond, R. G., & Johnson, L. (1999). A cognitive task analysis with implications for designing a simulation-based assessment system. Computers in Human Behavior, 15, 335–374.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based language assessment. Language Testing, 19, 477–496.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67.
Nelson, B., Erlandson, B., & Denham, A. (2010). Global channels for learning and assessment in complex game environments. British Journal of Educational Technology. Published online, January 2010. To appear in print, June 2010.
Perry, N. E., VandeKamp, K. O., Mercer, L. K., & Nordby, C. J. (2002). Investigating teacher-student interactions that foster self-regulated learning. Educational Psychologist, 37(1), 5–15.
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801–813.
Plass, J. L., O’Keefe, P., Homer, B. D., Case, J., Hayward, E. O., Stein. M., & Perlin, K. (in press). Motivational and educational outcomes associated with individual, competitive, and collaborative game play. Journal of Educational Psychology.
Plass, J. L., Homer, B. D., Hayward, E. O., Frye, J., Huang, T. T., Biles, M., Stein, M., & Perlin, K. (2012, September 18–20). The effect of learning mechanics design on learning outcomes in a computer-based geometry game. Paper presented at GameDays 2012 held, at Fraunhofer IGD TU Darmstadt, Darmstadt, Germany.
Roth, E. M., & Woods, D. D. (1989). Cognitive task analysis: An approach to knowledge acquisition of intelligent system design. In G. Guida & C. Tasso (Eds.), Topics in expert system design (pp. 233–264). New York: Elsevier Science.
Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment, 8(4). Retrieved June 1, 2011 from /http://www.jtla.org
Salen, K., & Zimmerman, E. (2003). Rules of play: Game design fundamentals. Cambridge: MIT Press.
Schraagen, J. M., Chipman, S. F., & Shalin, V. L. (Eds.). (2000). Cognitive task analysis. Mahwah: Lawrence Erlbaum Associates.
Schraw, G., & Moshman, D. (1995). Metacognitive theories. Educational Psychology Review, 7(4), 351–371.
Schwartz, D. L., & Black, J. B. (1990). The induction of rules from analog, mental models. Paper presented at the annual meeting of the American Educational Research Association, Boston, MA.
Shaffer, D. W. (2006). How computer games help children learn. New York: Palgrave Macmillan.
Shute, V. J. (2010). Innovative assessment for the 21st century: Supporting educational needs. New York: Springer.
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–524). Charlotte: Information Age.
Shute, V. J., & Kim, Y. J. (2011). Does playing the World of Goo facilitate learning? In D. Y. Dai (Ed.), Design research on learning and thinking in educational settings: Enhancing intellectual growth and functioning (pp. 243–267). New York: Routledge Books.
Shute, V. J., & Torreano, L. A. (2003). Formative evaluation of an automated knowledge elicitation and organization tool. In Murray, T., Ainsworth, S., & Blessing, S. (Eds.), Authoring tools for advanced technology learning environments: Toward cost-effective adaptive, interactive, and intelligent educational software (pp. 149-180). The Netherlands: Kluwer Academic Publishers.
Sicart, M. (2008). Defining game mechanics. Game Studies, 8(2). Retrieved June 1, 2011 from http://gamestudies.org/0802/articles/sicart
Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. In V. Patel (Ed.), Proceedings of the 10th annual conference of the cognitive science society. Hillsdale: Erlbaum.
Spiro, R. J., & Jehng, J. C. (1990). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In Nix, D. & Spiro, R. (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology (pp. 163–205). Hillsdale: Erlbaum.
Swink, S. (2008). Game feel: A game designer’s guide to virtual sensation. New York: Morgan Kaufmann.
Tychsen, A., & Canossa, A. (2008). Defining personas in games using metrics. In Proceedings of the 2008 conference on future play: Research, play, share (pp. 73–80). New York/Boston: Association Computing Machinery.
Um, E., Plass, J. L., Hayward, E. O., & Homer, B. D. (2011). Emotional design in multimedia learning. Journal of Educational Psychology, 104(2), 485–498.
Williamson, D., Bauer, M., Steinberg, L. S., Mislevy, R. J., Behrens, J., & DeMark, S. (2004). Design rationale for a complex performance assessment. International Journal of Testing, 4, 303–332.
Winne, P. H. (1982). Minimizing the black box problem to enhance the validity of theories about instructional effects. Instructional Science, 11, 13–28.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Additional information
About the Authors
Jan L. Plass, Ph.D. is Paulette Goddard Professor of Digital Media and Learning Sciences at NYU Steinhardt. He is the founding director of the Consortium for Research and Evaluation of Advanced Technology in Education (CREATE), and co-director of the Games for Learning Institute (G4LI).
Bruce D. Homer, Ph.D. is an Associate Professor of Educational Psychology (Learning, Development & Instruction Sub-Program) and Training Director for the Interdisciplinary Postdoctoral Research Training program at the Graduate Center at the City University of New York (CUNY).
Charles K. Kinzer, Ph.D. is a Professor of Communication and Education at Teachers College Columbia University where he is the program coordinator at Computing, Communication and Technology in Education (CCTE).
Yoo Kyung Chang, Ph.D. is a Lecturer in CCTE at Teachers College Columbia University and former research assistant at the CREATE lab at New York University.
Jonathan Frye is the Technology Coordinator and a Research Assistant for the CREATE lab. He is currently a doctoral candidate in the Educational Communications and Technology program at New York University.
Walter Kaczetow is currently a doctoral student in CUNY Graduate Center’s program in Educational Psychology. When not studying Walter can be found teaching mathematics as an adjunct professor at New Jersey City University.
Katherine Isbister, Ph.D. is an Associate Professor of both Computer Science Engineering and Digital Media at NYU’s Polytechnic Institute. She is the founding director of the Social Game Lab.
Ken Perlin, Ph.D. is a professor in the Department of Computer Science at New York University. In addition to being the director of the Games For Learning Institute, he was also founding director of the Media Research Laboratory and director of the NYU Center for Advanced Technology.
Rights and permissions
Copyright information
© 2013 Springer-Verlag London
About this chapter
Cite this chapter
Plass, J.L. et al. (2013). Metrics in Simulations and Games for Learning. In: Seif El-Nasr, M., Drachen, A., Canossa, A. (eds) Game Analytics. Springer, London. https://doi.org/10.1007/978-1-4471-4769-5_31
Download citation
DOI: https://doi.org/10.1007/978-1-4471-4769-5_31
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-4768-8
Online ISBN: 978-1-4471-4769-5
eBook Packages: Computer ScienceComputer Science (R0)