research-article

Algorithm Visualization and the Elusive Modality Effect

Authors:

Albina Zavgorodniaia,

Artturi Tilanterä,

Otto Seppälä,

Juha SorvaAuthors Info & Claims

ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education Research

Pages 368 - 378

https://doi.org/10.1145/3446871.3469747

Published: 17 August 2021 Publication History

Abstract

The modality effect in multimedia learning suggests that pictures are best accompanied by audio explanations rather than text, but this finding has not been replicated in computing education. We investigate which instructional modality works best as an accompaniment for algorithm visualizations. In a randomized controlled trial, learners were split into three conditions who viewed an instructional video on Dijkstra’s algorithm, with diagrams accompanied by audio, text, or both. We find neither a modality effect in favor of the audio condition nor a verbal redundancy effect in favor of using only a single modality rather than both. Taken together with earlier research, our findings suggest that the modality effect is difficult to apply reliably and computing educators should not rush to integrate audio into visualizations in expectation of the effect. We discuss theoretical viewpoints that future research should attend to; these include alternative part-explanations of the modality effect and attention-based models of working memory, among others.

References

[1]

Eryn J. Adams, Anh T. Nguyen, and Nelson Cowan. 2018. Theories of working memory: Differences in definition, degree of modularity, role of attention, and purpose. Language, Speech, and Hearing Services in Schools 49, 3 (2018), 340–355.

[2]

Katherine Ann Austin. 2009. Multimedia learning: Cognitive individual diffences and display design techniques predict transfer learning with multimedia learning modules. Computers & Education 53, 4 (2009), 1339–1354.

Digital Library

[3]

Alan Baddeley. 1992. Working memory. Science 255, 5044 (Jan. 1992), 556–559.

[4]

Alan Baddeley, Graham Hitch, and Richard Allen. 2021. A Multicomponent Model of Working Memory. In Working Memory. Oxford University Press, Oxford, United Kingdom, 10–43.

[5]

Pierre Barrouillet, Sophie Bernardin, and Valérie Camos. 2004. Time Constraints and Resource Sharing in Adults' Working Memory Spans.Journal of Experimental Psychology: General 133, 1 (2004), 83–100.

[6]

Pierre Barrouillet and Valérie Camos. 2021. The Time-Based Resource-Sharing Model of Working Memory. In Working Memory. Oxford University Press, Oxford, United Kingdom, 85–115.

[7]

Mireille Bétrancourt and André Bisseret. 1998. Integrating textual and pictorial information via pop-up windows: An experimental study. Behaviour & Information Technology 17, 5 (Jan. 1998), 263–273.

[8]

Cynthia J. Brame. 2016. Effective Educational Videos: Principles and Guidelines for Maximizing Student Learning from Video Content. CBE—Life Sciences Education 15, 4 (Dec. 2016), es6.

[9]

Dan Breakiron, Cliff Shaffer, Jackson Wonderly, sublime09, Eric Fouh, Kasper Hellström, Samnyeong Heo, Hossameldin Shahin, and Alex Hicks. 2021. Compiling Book Instances: OpenDSA Configuration. https://github.com/OpenDSA/OpenDSA/blob/master/Doc/source/Configuration.rst

[10]

Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H. Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In IEEE International Conference on Program Comprehension. 255–265.

Digital Library

[11]

Juan C. Castro-Alonso and John Sweller. 2019. The Modality Effect of Cognitive Load Theory. In Advances in Intelligent Systems and Computing. Springer International Publishing, Berlin Heidelberg, Germany, 75–84.

[12]

Paul Chandler and John Sweller. 1991. Cognitive Load Theory and the Format of Instruction. Cognition and Instruction 8, 4 (Dec. 1991), 293–332.

[13]

James M. Clark and Allan Paivio. 1991. Dual coding theory and education. Educational Psychology Review 3, 3 (Sept. 1991), 149–210.

[14]

Robert Coe. 2002. It’s the effect size, stupid: What effect size is and why it is important. http://www.leeds.ac.uk/educol/documents/00002182.htm In 2002 Annual Conference of the British Educational Research Association, Exeter, England.

[15]

Steven M. Crooks, Jongpil Cheon, Fethi Inan, Fatih Ari, and Raymond Flores. 2012. Modality and cueing in multimedia learning: Examining cognitive and perceptual explanations for the modality effect. Computers in Human Behavior 28, 3 (2012), 1063–1071.

Digital Library

[16]

Holger Danielsiek, Wolfgang Paul, and Jan Vahrenhold. 2012. Detecting and Understanding Students’ Misconceptions Related to Algorithms and Data Structures. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education(SIGCSE ’12). ACM, New York, NY, USA, 21–26.

Digital Library

[17]

Fatma Deniz, Anwar O. Nunez-Elizalde, Alexander G. Huth, and Jack L. Gallant. 2019. The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality. The Journal of Neuroscience 39, 39 (Aug. 2019), 7722–7736.

[18]

Ann Depoorter and André Vandierendonck. 2009. Evidence for Modality-Independent Order Coding in Working Memory. Quarterly Journal of Experimental Psychology 62, 3 (March 2009), 531–549.

[19]

E. W. Dijkstra. 1959. A note on two problems in connexion with graphs. Numer. Math. 1(1959), 269–271.

Digital Library

[20]

Barbara Ericson, Steven Moore, Briana B. Morrison, and Mark Guzdial. 2015. Usability and usage of interactive features in an online ebook for CS teachers. In WiPSCE ’15. ACM.

[21]

Mohammed F. Farghally, Kyu Han Koh, Jeremy V. Ernst, and Clifford A. Shaffer. 2017. Towards a Concept Inventory for Algorithm Analysis Topics. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’17). ACM, New York, NY, USA, 207–212.

Digital Library

[22]

Clive Frankish. 1995. Intonation and auditory grouping in immediate serial recall. Applied Cognitive Psychology 9, 7 (1995), S5–S22.

[23]

Judith Gal-Ezer and Ela Zur. 2004. The efficiency of algorithms—misconceptions. Computers & Education 42, 3 (2004), 215 – 226.

Digital Library

[24]

Paul Ginns. 2005. Meta-analysis of the modality effect. Learning and Instruction 15, 4 (2005), 313–331.

[25]

Tina Götschi, Ian Sanders, and Vashti Galpin. 2003. Mental Models of Recursion. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’03). ACM, New York, NY, USA, 346–350.

Digital Library

[26]

Kevin Greenberg, Robert Zheng, Michael Gardner, and Matthew Orr. 2020. Individual differences in visuospatial working memory capacity influence the modality effect. (2020). Preprint in the Journal of Computer Assisted Learning.

[27]

Michael Hansen, Robert L. Goldstone, and Andrew Lumsdaine. 2013. What makes code hard to understand?https://arxiv.org/abs/1304.5257

[28]

Angela Hildyard and David R. Olson. 1978. Memory and inference in the comprehension of oral and written discourse*. Discourse Processes 1, 2 (April 1978), 91–117.

[29]

Slava Kalyuga. 2011. Instructional benefits of spoken words: A review of cognitive load factors. Educational Research Review 7 (2011), 145–159.

[30]

Slava Kalyuga, Paul Chandler, and John Sweller. 1999. Managing split-attention and redundancy in multimedia instruction. Applied Cogn. Psychology(1999), 351–371.

[31]

Ville Karavirta, Petri Ihantola, and Teemu Koskinen. 2013. Service-Oriented Approach to Improve Interoperability of E-Learning Systems. In 2013 IEEE 13th International Conference on Advanced Learning Technologies. IEEE, 341–345.

Digital Library

[32]

Ville Karavirta, Ari Korhonen, and Otto Seppälä. 2013. Misconceptions in Visual Algorithm Simulation Revisited: On UI’s Effect on Student Performance, Attitudes, and Misconceptions. In 2013 Learning and Teaching in Computing and Engineering. IEEE, Piscataway, New Jersey, United States, 62–69.

[33]

Ville Karavirta and Clifford A. Shaffer. 2016. Creating Engaging Online Learning Material with the JSAV JavaScript Algorithm Visualization Library. IEEE Transactions on Learning Technologies 9, 2 (4 2016), 171–183.

[34]

Ari Korhonen and Lauri Malmi. 2000. Algorithm Simulation with Automatic Assessment. SIGCSE Bull. 32, 3 (July 2000), 160–163.

Digital Library

[35]

Wayne Leahy and John Sweller. 2016. Cognitive load theory and the effects of transient information on the modality effect. Instructional Science 44, 1 (2016), 107–123.

[36]

Stefanie Lindow, Heather M. Fuchs, Anne Fürstenberg, Janet Kleber, Judith Schweppe, and Ralf Rummer. 2011. On the robustness of the modality effect: Attempting to replicate a basic finding. Zeitschrift fur Pädagogische Psychologie 25, 4 (2011), 231–243.

[37]

Robert H. Logie, Clément Belletier, and Jason M. Doherty. 2021. Integrating Theories of Working Memory. In Working Memory: State of the Science, Robert H. Logie, Valérie Camos, and Nelson Cowan(Eds.). Oxford University Press, Oxford, United Kingdom, 389–430.

[38]

Robert H. Logie, Valérie Camos, and Nelson Cowan(Eds.). 2021. Working Memory: State of the Science. Oxford University Press, Oxford, United Kingdom.

[39]

Lauri Malmi, Ville Karavirta, Ari Korhonen, Jussi Nikander, Otto Seppälä, and Panu Silvasti. 2004. Visual Algorithm Simulation Exercise System with Automatic Assessment: TRAKLA2. Informatics in Education 3 (10 2004), 267–288.

[40]

Giacomo Mariani. 2020. Design of an Application to Collect Data and Create Animations from Visual Algorithm Simulation Exercises. https://aaltodoc.aalto.fi/handle/123456789/44448 Master’s Thesis.

[41]

Richard E. Mayer (Ed.). 2014. The Cambridge Handbook of Multimedia Learning (2nd ed.). Cambridge University Press.

[42]

Richard E. Mayer. 2020. Multimedia Learning(3rd ed.). Cambridge University Press.

[43]

Roxana Moreno and Richard E. Mayer. 1999. Cognitive principles of multimedia learning: The role of modality and contiguity.Journal of Educational Psychology 91, 2 (1999), 358–368.

[44]

Briana B. Morrison. 2017. Dual modality code explanations for novices: Unexpected results. In The 2017 ACM Conference on International Computing Education Research(ICER ’17). ACM, Tacoma, Washington, USA, 226–235.

Digital Library

[45]

Thomas L. Naps, Guido Rößling, Vicki Almstrum, Wanda Dann, Rudolf Fleischer, Chris Hundhausen, Ari Korhonen, Lauri Malmi, Myles McNally, Susan Rodger, and J. Ángel Velázquez-Iturbide. 2002. Exploring the Role of Visualization and Engagement in Computer Science Education. SIGCSE Bull. 35, 2 (June 2002), 131–152.

Digital Library

[46]

Kensuke Okada. 2017. Negative estimate of variance-accounted-for effect size: How often it is obtained, and what happens if it is treated as zero. Behavior Research Methods 49, 3 (2017), 979–987.

[47]

Nesrin Özdener. 2008. A comparison of the misconceptions about the time-efficiency of algorithms by various profiles of computer-programming students. Computers & Education 51, 3 (2008), 1094 – 1102.

Digital Library

[48]

Allan Paivio. 1986. Mental representations : a dual coding approach. Oxford University Press, Oxford, United Kingdom.

[49]

Wolfgang Paul and Jan Vahrenhold. 2013. Hunting High and Low: Instruments to Detect Misconceptions Related to Algorithms and Data Structures. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education(SIGCSE ’13). ACM, New York, NY, USA, 29–34.

Digital Library

[50]

Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code - A Replication Experiment. In 11th International Conference on Augmented Cognition. 206–218.

[51]

Norman Peitek, Janet Siegmund, and Sven Apel. 2020. What drives the reading order of programmers? An eye tracking study. In ICPC.

[52]

Marian Petre and Alan F. Blackwell. 1999. Mental imagery in program design and visual programming. Int. J. of Human Computer Studies 51, 1 (1999), 7–30.

Digital Library

[53]

Joachim Reinwein. 2012. Does the modality effect exist? And if so, which modality effect?Journal of Psycholinguistic Research 41, 1 (2012), 1–32.

[54]

Donald L. Rubin, Teresa Hafer, and Kevin Arata. 2000. Reading and listening to oral-based versus literate-based discourse. Communication Education 49, 2 (April 2000), 121–133.

[55]

Ralf Rummer, Judith Schweppe, Anne Fürstenberg, Katharina Scheiter, and Antje Zindler. 2011. The perceptual basis of the modality effect in multimedia learning.Journal of Experimental Psychology: Applied 17, 2 (2011), 159–173.

[56]

Wolfgang Schnotz. 2011. Colorful bouquets in multimedia research: A closer look at the modality effect. Zeitschrift fur Pädagogische Psychologie 25, 4 (2011), 269–276.

[57]

Anne Schüler, Katharina Scheiter, Ralf Rummer, and Peter Gerjets. 2012. Explaining the modality effect in multimedia learning: Is it due to a lack of temporal contiguity with written text and pictures?Learning and Instruction 22, 2 (2012), 92–102.

[58]

Anne Schüler, Katharina Scheiter, and Florian Schmidt-Weigand. 2011. Boundary conditions and constraints of the modality effect. Zeitschrift fur Pädagogische Psychologie 25, 4(2011), 211–220.

[59]

Stoo Sepp, Steven J. Howard, Sharon Tindall-Ford, Shirley Agostinho, and Fred Paas. 2019. Cognitive Load Theory and Human Movement: Towards an Integrated Model of Working Memory. Educational Psychology Review 31, 2 (Feb. 2019), 293–317.

[60]

Otto Seppälä, Lauri Malmi, and Ari Korhonen. 2006. Observations on Student Misconceptions—A Case Study of the Build – Heap Algorithm. Computer Science Education 16, 3 (2006), 241 – 255.

[61]

Tina Seufert, Maren Schütze, and Roland Brünken. 2009. Memory characteristics and modality in multimedia learning: An aptitude–treatment–interaction study. Learning and Instruction 19, 1 (2009), 28–42.

[62]

Clifford A. Shaffer, Matthew L. Cooper, Alexander Joel D. Alon, Monika Akbar, Michael Stewart, Sean Ponce, and Stephen H. Edwards. 2010. Algorithm Visualization: The State of the Field. ACM Trans. Comput. Educ. 10, 3, Article 9 (Aug. 2010), 22 pages.

Digital Library

[63]

Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A review of generic program visualization systems for introductory programming education. ACM Transactions on Computing Education 13, 4 (2013), 1–64.

Digital Library

[64]

Susan Elizabeth Thompson. 2003. Text-structuring metadiscourse, intonation and the signalling of organisation in academic lectures. Journal of English for Academic Purposes 2, 1 (Jan. 2003), 5–20.

[65]

Artturi Tilanterä. 2020. Towards Automatic Advice in Visual Algorithm Simulation. http://urn.fi/URN:NBN:fi:aalto-202005243300 Master’s thesis. Aalto University, 2020.

[66]

André Vandierendonck. 2021. Multicomponent Working Memory System with Distributed Executive Control. In Working Memory. Oxford University Press, Oxford, United Kingdom, 150–174.

[67]

Jackson Wonderly and Ehsan Elgendi. 2017. Narration by jwonderly · Pull Request #207 · vkaravir/JSAV · GitHub. https://github.com/vkaravir/JSAV/pull/207

[68]

Zhen Xu, Albert D. Ritzhaupt, Karthikeyan Umapathy, Yang Ning, and Chin-Chung Tsai. 2021. Exploring college students’ conceptions of learning computer science: A draw-a-picture technique study. Computer Science Education 31, 1 (2021), 60–82.

[69]

Albina Zavgorodniaia, Rodrigo Duran, Arto Hellas, Otto Seppälä, and Juha Sorva. 2020. Should explanations of program code use audio, text, or both? A replication study. In Proceedings of the 20th Koli Calling International Conference on Computing Education Research(Koli Calling ’20). ACM, New York, NY, USA.

Digital Library

[70]

Rui Zhi. 2019. Design and Evaluation of Instructional Supports for Novice Programming Environments. Doctoral dissertation. North Carolina State University.

Cited By

Kim GGreen DMatthews S(2024)Evaluating the Efficacy of Peer-Created Worked-Example Videos in a Computer Systems CourseJournal of Computing Sciences in Colleges10.5555/3665609.366561539:8(83-97)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.5555/3665609.3665615
Sanders KVahrenhold JMcCartney R(2023)How Do Computing Education Researchers Talk About Threats and Limitations?Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600114(381-396)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600114
Muldner KJennings JChiarelli V(2022)A Review of Worked Examples in Programming ActivitiesACM Transactions on Computing Education10.1145/356026623:1(1-35)Online publication date: 29-Dec-2022
https://dl.acm.org/doi/10.1145/3560266

Recommendations

Long-term modality effect in multimedia learning
NordiCHI '14: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational

Cognitive theories of multimedia are seeking the best way of creating materials to enhance learning outcomes. The so-called modality effect accords that learning outcomes are better if visual material such as images is presented together with auditory ...
Modality effects on cognitive load and performance in high-load information presentation
IUI '09: Proceedings of the 14th international conference on Intelligent user interfaces

In this study, we argue that modality planning in multimodal presentation systems needs to consider the modality characteristics at not only the presentational level but also the cognitive level, especially in a situation where the information load is ...
Is spoken text always better? Investigating the modality and redundancy effect with longer text presentation

The reported study investigated the influence of longer text presentation on the modality and the redundancy effect. In particular, the auditory-recency-effect explanation, which predicts that both effects should disappear when longer texts are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER 2021: Proceedings of the 17th ACM Conference on International Computing Education Research

August 2021

451 pages

ISBN:9781450383264

DOI:10.1145/3446871

Editors:
Amy J. Ko
University of Washington, Seattle, USA
,
Jan Vahrenhold
Westfälische Wilhelms-Universität Münster, Germany
,
Renée McCauley
College of Charleston, USA
,
Matthias Hauswirth
Università della Svizzera italiana, Switzerland

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICER 2021

Sponsor:

SIGCSE

ICER 2021: ACM Conference on International Computing Education Research

August 16 - 19, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 189 of 803 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
270
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)5

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kim GGreen DMatthews S(2024)Evaluating the Efficacy of Peer-Created Worked-Example Videos in a Computer Systems CourseJournal of Computing Sciences in Colleges10.5555/3665609.366561539:8(83-97)Online publication date: 17-May-2024
https://dl.acm.org/doi/10.5555/3665609.3665615
Sanders KVahrenhold JMcCartney R(2023)How Do Computing Education Researchers Talk About Threats and Limitations?Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600114(381-396)Online publication date: 7-Aug-2023
https://dl.acm.org/doi/10.1145/3568813.3600114
Muldner KJennings JChiarelli V(2022)A Review of Worked Examples in Programming ActivitiesACM Transactions on Computing Education10.1145/356026623:1(1-35)Online publication date: 29-Dec-2022
https://dl.acm.org/doi/10.1145/3560266
Duran RZavgorodniaia ASorva J(2022)Cognitive Load Theory in Computing Education Research: A ReviewACM Transactions on Computing Education10.1145/348384322:4(1-27)Online publication date: 15-Sep-2022
https://dl.acm.org/doi/10.1145/3483843

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents