research-article

Developing a pedagogical framework for designing a multisensory serious gaming environment

Authors:

Monica GoriAuthors Info & Claims

MIE 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education

Pages 1 - 9

https://doi.org/10.1145/3139513.3139517

Published: 13 November 2017 Publication History

Abstract

The importance of multisensory interaction for learning has increased with improved understanding of children’s sensory development, and a flourishing interest in embodied cognition. The potential to foster new forms of multisensory interaction through various sensor, mobile and haptic technologies is promising in providing new ways for young children to engage with key mathematical concepts. However, designing effective learning environments for real world classrooms is challenging, and requires a pedagogically, rather than technologically, driven approach to design. This paper describes initial work underpinning the development of a pedagogical framework, intended to inform the design of a multisensory serious gaming environment. It identifies the theoretical basis of the framework, illustrates how this informs teaching strategies, and outlines key technology research driven perspectives and considerations important for informing design. An initial table mapping mathematical concepts to design, a framework of considerations for design, and a process model of how the framework will continue to be developed across the design process are provided.

References

[1]

Abrahamson, D. and Sánchez-García, R. 2016. Learning Is Moving in New Ways: The Ecological Dynamics of Mathematics Education. Journal of the Learning Sciences. 25, 2 (2016), 203–239.

[2]

Abrahamson, D. and Trninic, D. 2011. Toward an embodiedinteraction design framework for mathematical concepts. Proceedings of the 10th International Conference on Interaction Design and Children - IDC ’11 (Ann Arbor, Michigan, 2011), 1–10.

Digital Library

[3]

Alibali, M.W. and Nathan, M.J. 2012. Embodiment in Mathematics Teaching and Learning: Evidence From Learners’ and Teachers’ Gestures. Journal of the Learning Sciences. 21, 2 (2012), 247–286.

[4]

Almeida e Costa, F. and Mateus Rocha, L. 2005. Embodied and Situated Cognition. Artificial Life. 11, (2005), 5–11.

Digital Library

[5]

Anderson, A.A. 2005. The community builder’s approach to theory of change.

[6]

Antle, A. 2007. The CTI framework: informing the design of tangible systems for children. Proceedings of the 1st international conference on Tangible and embedded interaction - TEI ’07 (Baton Rouge, LA, 2007), 195–202.

Digital Library

[7]

Barsalou, L.W. 2008. Grounded Cognition. Annual Review of Psychology. 59, (Jan. 2008), 617–645.

[8]

Benford, S. et al. 2000. Designing storytelling technologies to encourage collaboration detween young children. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’00 (New York, New York, USA, 2000), 556–563.

Digital Library

[9]

Benton, L. et al. 2016. Building mathematical knowledge with programming: insights from the ScratchMaths project. Constructionism in Action 2016 (2016), 26–33.

[10]

Chahine, I.C. 2013. The impact of using multiple modalities on students’ acquisition of fractional knowledge: An international study in embodied mathematics across semiotic cultures. Journal of Mathematical Behavior. 32, 3 (2013), 434–449.

[11]

Charsky, D. 2010. From Edutainment to Serious Games: A Change in the Use of Game Characteristics. Games and Culture. 5, 2 (2010), 177– 198.

[12]

Clark, A. and Chalmers, D. 1998. The Extended Mind. Analysis. 58, 1 (1998), 7–19.

[13]

Cramer, E.S. and Antle, A.N. 2015. Button Matrix: How Tangible Interfaces Can Structure Physical Experiences for Learning. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction TEI ’15 (Stanford, CA, 2015), 301–304.

Digital Library

[14]

Cuendet, S. et al. 2012. Tangible vs. virtual representations. Proceedings of the 7th Nordic Conference on Human-Computer Interaction Making Sense Through Design - NordiCHI ’12 (Copenhagen, 2012), 99–108.

Digital Library

[15]

Curzon, P. et al. 2009. Enthusing and inspiring with reusable kinaesthetic activities. Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education - ITiCSE ’09 (Paris, 2009), 94–98.

Digital Library

[16]

Decety, J. and Grèzes, J. 2006. The power of simulation: Imagining one’s own and other’s behavior. Brain Research. 1079, 1 (2006), 4–14.

[17]

Department for Education 2013. Mathematics programmes of study: key stages 1 and 2 National curriculum in England.

[18]

Dourish, P. 2001. Where the Action Is: The Foundations of Embodied Interaction. MIT Press.

Digital Library

[19]

Duffy, S. et al. 2017. WeDraw  : using multisensory serious games to explore concepts in primary mathematics. Proceedings of the 13th International Conference on Technology in Mathematics Teaching - ICTMT 13 (Lyon, 2017), in press.

[20]

Elia, I. et al. 2014. The role of gestures in making connections between space and shape aspects and their verbal representations in the early years: findings from a case study. Mathematics Education Research Journal. 26, 4 (2014), 735–761.

[21]

Garbarini, F. and Adenzato, M. 2004. At the root of embodied cognition: Cognitive science meets neurophysiology. Brain and Cognition.

[22]

Goldin-Meadow, S. et al. 2009. Gesturing Gives Children New Ideas About Math. Psychological Science. 20, 3 (2009), 267–272.

[23]

Gori, M. 2015. Multisensory Integration and Calibration in Children and Adults with and without Sensory and Motor Disabilities. Multisensory Research. 28, 1–2 (Apr. 2015), 71–99.

[24]

Gori, M. et al. 2008. Young Children Do Not Integrate Visual and Haptic Form Information. Current Biology. 18, 9 (May 2008), 694–698.

[25]

Habib, M. et al. 2016. Music and Dyslexia: A New Musical Training Method to Improve Reading and Related Disorders. Frontiers in psychology. 7, 26 (2016).

[26]

Heath, C. and Lehn, D. Vom 2008. Configuring “Interactivity”: Enhancing Engagement in Science Centres and Museums. Social Studies of Science. 38, 1 (Feb. 2008), 63–91.

[27]

Hohenwarter, M. and Fuchs, K. 2004. Combination of dynamic geometry, algebra and calculus in the software system GeoGebra. Computer algebra systems and dynamic geometry systems in mathematics teaching conference 2004 (2004).

[28]

Hornecker, E. et al. 2008. Collaboration and interference: Awareness with Mice or Touch Input. ACM 2008 Conference on Computer Supported Cooperative Work (CSCW’08) (San Diego, CA, 2008), 167– 176.

Digital Library

[29]

Hornecker, E. and Buur, J. 2006. Getting a grip on tangible interaction. Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI ’06. (2006), 437.

Digital Library

[30]

Hornecker, E. and Dünser, A. 2009. Of pages and paddles: Children’s Developing a Pedagogical Framework for Designing… MIE’17, November 13, 2017, Glasgow, UK expectations and mistaken interactions with physical-digital tools. Interacting with Computers. 21, 1–2 (Jan. 2009), 95–107.

Digital Library

[31]

Hostetter, A.B. and Alibali, M.W. 2008. Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review. 15, 3 (2008), 495– 514.

[32]

Huizenga, J. et al. 2009. Mobile game-based learning in secondary education: engagement, motivation and learning in a mobile city game. Journal of Computer Assisted Learning. 25, 4 (2009), 332–344.

[33]

Hutto, D.D. et al. 2015. The enactive roots of STEM: Rethinking educational design in mathematics. Educational Psychology Review. 27, 3 (2015), 371–389.

[34]

Johnson, D.W. and Johnson, R.T. 1999. Learning together and alone  : cooperative, competitive, and individualistic learning. Allyn and Bacon.

[35]

Ke, F. and Grabowski, B. 2007. Gameplaying for maths learning: Cooperative or not? British Journal of Educational Technology. 38, 2 (Mar. 2007), 249–259.

[36]

Kessell, A. and Tversky, B. 2011. Visualizing space, time, and agents: Production, performance, and preference. Cognitive Processing. 12, (2011), 43–52.

[37]

Kim, M. et al. 2011. Children’s gestures and the embodied knowledge of geometry. International Journal of Science and Mathematics Education. 9, 1 (2011), 207–238.

[38]

Klemmer, S.R. et al. 2006. How Bodies Matter: Five Themes for Interaction Design. Proceedings of the 6th ACM conference on Designing Interactive systems - DIS ’06 (Pennsylvania, 2006), 140–149.

Digital Library

[39]

Laanpere, M. et al. 2014. Pedagogy-driven Design Of Digital Learning Ecosystems. Computer Science and Information Systems. 11, 1 (2014), 419–442.

[40]

Lakoff, G. and Núñez, R.E. 2000. Preface. Where mathematics comes from: How the embodied mind brings mathematics into being. Basic Books. XI–XVII.

[41]

Ma, J.Y. 2017. Multi-Party, Whole-Body Interactions in Mathematical Activity. Cognition and Instruction. 8, April (2017), 1–24.

[42]

Manches, A. and O’Malley, C. 2016. Cognition and Instruction The Effects of Physical Manipulatives on Children’s Numerical Strategies. Cognition and Instruction. 34, 1 (2016), 27–50.

[43]

Martin, T. and Schwartz, D.L. 2005. Physically Distributed Learning: Adapting and Reinterpreting Physical Environments in the Development of Fraction Concepts. Cognitive Science. 29, 4 (Jul. 2005), 587–625.

[44]

Meluso, A. et al. 2012. Enhancing 5th graders’ science content knowledge and self-efficacy through game-based learning. Computers & Education. 59, 2 (Sep. 2012), 497–504.

Digital Library

[45]

Merleau-Ponty, M. 1962. Phenomenology of Perception. Routledge and Kegan Paul Ltd.

[46]

Nakamura, J. and Csikszentmihalyi, M. 2002. The concept of flow. Handbook of Positive Psychology. Oxford University Press.

[47]

Nemirovsky, R. et al. 2012. When the Classroom Floor Becomes the Complex Plane: Addition and Multiplication as Ways of Bodily Navigation. Journal of the Learning Sciences. 21, 2 (2012), 287–323.

[48]

Nicolopoulou, A. 1993. Play, cognitive development, and the social world: Piaget, vygotsky, and beyond. Human Development. 36, 1 (1993), 1–23.

[49]

Nunes, T. et al. 2009. Key understandings in mathematics learning. Nuffield Foundation.

[50]

Park, Y. 2011. A Pedagogical Framework for Mobile Learning: Categorizing Educational Applications of Mobile Technologies into Four Types. International Review of Research in Open and Distance Learning. 12, 2 (2011).

[51]

Piaget, J. 1972. The principles of genetic epistemology. Routledge and Kegan Paul Ltd.

[52]

Prensky, M. 2005. Computer games and learning: Digital Game-Based Learning. Handbook of computer game studies. 18, (2005), 97– 122.

[53]

Price, S. 2008. A representation approach to conceptualizing tangible learning environments. Proceedings of the 2nd international conference on Tangible and embedded interaction TEI 08 (Bonn, Germany, 2008), 151–158.

Digital Library

[54]

Price, S. et al. 2010. Action and representation in tangible systems: implications for design of learning interactions. Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction - TEI ’10 (New York, New York, USA, 2010), 145–152.

Digital Library

[55]

Price, S. and Falcão, T.P. 2009. Designing for physical-digital correspondence in tangible learning environments. Proceedings of the 8th International Conference on Interaction Design and Children - IDC ’09 (Como, Italy, 2009), 194–197.

Digital Library

[56]

Radcliffe, D. 2008. A Pedagogy-Space-Technology (PST) Framework for Designing and Evaluating Learning Places. Proceedings of the Next Generation Learning Spaces 2008 Colloquium (Brisbane, 2008), 9– 16.

[57]

Ritterfeld, U. et al. 2009. Serious games: Mechanisms and effects. Routledge.

[58]

Roschelle, J. and Pea, R. 2002. A walk on the WILD side: How wireless handhelds may change computer-supported collaborative learning. International Journal of Cognition and Technology. 1, 1 (Jan. 2002), 145–168.

[59]

Sakr, M. et al. 2016. Mobile Experiences of Historical Place: A Multimodal Analysis of Emotional Engagement. Journal of the Learning Sciences. 25, 1 (Jan. 2016), 51–92.

[60]

Sellar, S. and Cormack, P. 2007. Framing pedagogies: a review of frameworks and research designed to promote effective approaches to teaching and learning.

[61]

Shaer, O. and Hornecker, E. 2009. Tangible User Interfaces: Past, Present, and Future Directions. Foundations and Trends in Human– Computer Interaction. 3, 1–2 (2009).

Digital Library

[62]

Shaun Gallagher 2005. How the Body Shapes the Mind. Clarendon Press.

[63]

Shoval, E. 2011. Using mindful movement in cooperative learning while learning about angles. Instructional Science. 39, 4 (2011), 453– 466.

[64]

Siegler, R.S. and Ramani, G.B. 2008. Playing linear numerical board games promotes low-income children’s numerical development. Developmental Science. 11, 5 (Sep. 2008), 655–661.

[65]

Smith, C.P. et al. 2014. Learning angles through movement: Critical actions for developing understanding in an embodied activity. Journal of Mathematical Behavior. 36, November 2016 (2014), 95–108.

[66]

Smith, L. and Gasser, M. 2005. The Development of Embodied Cognition: Six Lessons from Babies. Artificial Life. 11, 1–2 (2005), 13– 29.

Digital Library

[67]

Starr-Glass, D. 2016. Participation in Online Distance Learning Environments: Proxy, Sign, or a Means to an End? Handbook of research on strategic management of interaction, presence, and participation in online courses. L. Kyei-Blankson et al., eds. IGI Global. 611.

[68]

Suh, J. and Moyer, P.S. 2007. Developing students’ representational fluency using virtual and physical algebra balances. Journal of Computers in Mathematics and Science Teaching. 26, 2 (2007), 155– 173.

[69]

Vygotsky, L.S. 1978. Mind in society  : the development of higher psychological processes. Harvard University Press.

[70]

Walkington, C. et al. 2012. Grounding Mathematical Justifications in Concrete Embodied Experience: The Link between Action and Cognition. The Annual meeting of the American Educational Research Association (Vancouver, 2012), 1–36.

[71]

Wiemers, M. et al. 2014. Spatial interferences in mental arithmetic: Evidence from the motion-arithmetic compatibility effect. The quarterly journal of experimental psychology. 67, 8 (Aug. 2014), 1557– 1570.

[72]

Wilson, M. 2002. Six views of embodied cognition. Psychonomic bulletin & review. 9, 4 (Dec. 2002), 625–36.

[73]

Winograd, T. and Flores, F. 1986. Understanding Computers and Cognition  : A New Foundation for Design. Ablex Publishing Corporation.

Digital Library

[74]

Young, M.F. et al. 2012. Our Princess Is in Another Castle: A Review of Trends in Serious Gaming for Education. Review of Educational Research. 82, 1 (2012), 61–89.

Cited By

Yu MWang LCai MZhang MYu CYang XZhang J(2024)MakeBronzeInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103201183:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103201
Fernández-Sánchez MGonzález-Fernández AAcevedo-Borrega J(2023)Conceptual Approach to the Pedagogy of Serious GamesInformation10.3390/info1402013214:2(132)Online publication date: 17-Feb-2023
https://doi.org/10.3390/info14020132
Debloos JJost CArchambault D(2023)ScenaConnect: an original device to enhance experiences with multisensorialityProceedings of the 2023 ACM International Conference on Interactive Media Experiences10.1145/3573381.3597225(356-358)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3573381.3597225
Show More Cited By

Index Terms

Developing a pedagogical framework for designing a multisensory serious gaming environment

Recommendations

Pedagogical Strategies for Reflection in Project-based HCI Education with End Users
DIS '21: Proceedings of the 2021 ACM Designing Interactive Systems Conference

As HCI pedagogy research grows, so too does an emerging set of evidence-based teaching and curricular recommendations. Yet, few studies have implemented and examined these recommendations in the classroom. In this paper, we present a Research Through ...
Designing Together?: Group Dynamics in Participatory Digital Badge Design with Teens
IDC '17: Proceedings of the 2017 Conference on Interaction Design and Children

Balancing the dynamics between industry, academia, and stakeholders in a participatory design (PD) project can be challenging, particularly with teens as design partners. In this reflective case study of digital badge design, we attempt to untangle ...
HCD mobile health project: post collaboration reflection of researcher and designer
APCHIUX '15: Proceedings of the Asia Pacific HCI and UX Design Symposium

This paper constitutes a single case study documenting the design of a mobile health application relying on human-centred design approaches. The project required collaboration between a group of medical health researchers and a human-centred design ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MIE 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education

November 2017

75 pages

ISBN:9781450355575

DOI:10.1145/3139513

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13, 2017

Glasgow, UK

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
409
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu MWang LCai MZhang MYu CYang XZhang J(2024)MakeBronzeInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103201183:COnline publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103201
Fernández-Sánchez MGonzález-Fernández AAcevedo-Borrega J(2023)Conceptual Approach to the Pedagogy of Serious GamesInformation10.3390/info1402013214:2(132)Online publication date: 17-Feb-2023
https://doi.org/10.3390/info14020132
Debloos JJost CArchambault D(2023)ScenaConnect: an original device to enhance experiences with multisensorialityProceedings of the 2023 ACM International Conference on Interactive Media Experiences10.1145/3573381.3597225(356-358)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3573381.3597225
Cuturi LCooney SCappagli GNewell FGori M(2023)Primary Schoolers' Response to a Multisensory Serious Game on Cartesian Plane Coordinates in Immersive Virtual RealityCyberpsychology, Behavior, and Social Networking10.1089/cyber.2022.020926:8(648-656)Online publication date: 1-Aug-2023
https://doi.org/10.1089/cyber.2022.0209
Boucher HLierse SMarzano GMarzano G(2022)A Dialogical View on R. Murray Schafer's Theories and Creative Approaches in 21st Century Music EducationSustaining Creativity and the Arts in the Digital Age10.4018/978-1-7998-7840-7.ch011(292-325)Online publication date: 24-Jun-2022
https://doi.org/10.4018/978-1-7998-7840-7.ch011
Olugbade TNewbold JJohnson RVolta EAlborno PNiewiadomski RDillon MVolpe GBianchi-Berthouze N(2022)Automatic Detection of Reflective Thinking in Mathematical Problem Solving Based on Unconstrained Bodily ExplorationIEEE Transactions on Affective Computing10.1109/TAFFC.2020.297806913:2(944-957)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TAFFC.2020.2978069
Jost CDebloos JArchambault DLe Pévédic BSagot JSohier RAlbert Tijus CTruck IUzan G(2021)PRIM Project: Playing and Recording with Interactivity and MultisensorialityProceedings of the 2021 ACM International Conference on Interactive Media Experiences10.1145/3452918.3465487(223-227)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3452918.3465487
Guevara CCoronel D(2020)Multisensory Learning System Applying Augmented RealityAdvances in Human Factors in Training, Education, and Learning Sciences10.1007/978-3-030-50896-8_48(336-342)Online publication date: 1-Jul-2020
https://doi.org/10.1007/978-3-030-50896-8_48
Cuturi LTonelli ACappagli GGori M(2019)Coarse to Fine Audio-Visual Size Correspondences Develop During Primary School AgeFrontiers in Psychology10.3389/fpsyg.2019.0206810Online publication date: 12-Sep-2019
https://doi.org/10.3389/fpsyg.2019.02068
Jost CPévédic BUzan G(2019)MulseBoxProceedings of the 31st Conference on l'Interaction Homme-Machine10.1145/3366550.3372251(1-13)Online publication date: 10-Dec-2019
https://dl.acm.org/doi/10.1145/3366550.3372251
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents