research-article

Public Access

Students’ Verbalized Metacognition During Computerized Learning

Authors:

Jaclyn Ocumpaugh,

Gautam BiswasAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 680, Pages 1 - 12

https://doi.org/10.1145/3411764.3445809

Published: 07 May 2021 Publication History

All formats PDF

Abstract

Students in computerized learning environments often direct their own learning processes, which requires metacognitive awareness of what should be learned next. We investigated a novel method of measuring verbalized metacognition by applying natural language processing (NLP) to transcripts of interviews conducted in a classroom with 99 middle school students who were using a computerized learning environment. We iteratively adapted the NLP method for the linguistic characteristics of these interviews, then applied it to study three research questions regarding the relationships between verbalized metacognition and measures of 1) learning, 2) confusion, and 3) metacognitive problem-solving strategies. Verbalized metacognition was not directly related to learning, but was related to confusion and metacognitive problem-solving strategies. Results also suggested that interviews themselves may improve learning by encouraging metacognition. We discuss implications for designing computerized environments that support self-regulated learning through metacognition.

References

[1]

Rakefet Ackerman and Morris Goldsmith. 2011. Metacognitive regulation of text learning: On screen versus on paper. Journal of Experimental Psychology: Applied 17, 1 (2011), 18–32.

[2]

Icek Ajzen and Martin Fishbein. 2005. The influence of attitudes on behavior. In The handbook of attitudes, Dolores Albarracín, Blair T. Johnson and Mark P. Zanna (eds.). Lawrence Erlbaum Associates Publishers, Mahwah, NJ, 173–221.

[3]

Alexandra Andres, Jaclyn Ocumpaugh, Ryan S. Baker, Stefan Slater, Luc Paquette, Yang Jiang, Nigel Bosch, Anabil Munshi, Allison L. Moore, and Gautam Biswas. 2019. Affect sequences and learning in Betty's Brain. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK19), ACM, New York, NY, 383–390.

Digital Library

[4]

Roger Azevedo and Vincent Aleven. 2013. Metacognition and learning technologies: An overview of current interdisciplinary research. In International Handbook of Metacognition and Learning Technologies, Roger Azevedo and Vincent Aleven (eds.). Springer, New York, NY, 1–16.

[5]

Roger Azevedo, Amy Johnson, Amber Chauncey, and Candice Burkett. 2010. Self-regulated learning with MetaTutor: Advancing the science of learning with metacognitive tools. In New Science of Learning: Cognition, Computers and Collaboration in Education, Myint Swe Khine and Issa M. Saleh (eds.). Springer, New York, NY, 225–247.

[6]

Gautam Biswas, Krittaya Leelawong, Daniel Schwartz, Nancy Vye, and The Teachable Agents Group at Vanderbilt. 2005. Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence 19, 3–4 (March 2005), 363–392.

[7]

Gautam Biswas, James R. Segedy, and Kritya Bunchongchit. 2016. From design to implementation to practice - A learning by teaching system: Betty's Brain. International Journal of Artificial Intelligence in Education 26, 1 (March 2016), 350–364.

[8]

A.F. Blackwell. 1996. Metacognitive theories of visual programming: What do we think we are doing? In Proceedings 1996 IEEE Symposium on Visual Languages, IEEE, 240–246.

[9]

Benjamin S. Bloom. 1968. Learning for mastery. Evaluation Comment 1, 2 (1968).

[10]

Ivar Bråten and Marit S. Samuelstuen. 2007. Measuring strategic processing: Comparing task-specific self-reports to traces. Metacognition Learning 2, 1 (April 2007), 1–20.

[11]

Gwendolyn E. Campbell, Natalie B. Steinhauser, Myroslava Dzikovska, Johanna D. Moore, Charles B. Callaway, and Elaine Farrow. 2009. Metacognitive awareness versus linguistic politeness: Expressions of confusion in tutorial dialogues. Naval Air Warfare Center Training Systems Div. Retrieved September 3, 2020 from https://apps.dtic.mil/sti/citations/ADA530033

[12]

Michelene T. H. Chi, Miriam Bassok, Matthew W. Lewis, Peter Reimann, and Robert Glaser. 1989. Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science 13, 2 (April 1989), 145–182.

[13]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, (1960), 37–46.

[14]

Scotty D. Craig, Xiangen Hu, Arthur C. Graesser, Anna E. Bargagliotti, Allan Sterbinsky, Kyle R. Cheney, and Theresa Okwumabua. 2013. The impact of a technology-based mathematics after-school program using ALEKS on student's knowledge and behaviors. Computers & Education 68, (October 2013), 495–504.

[15]

Scotty Craig, Art Graesser, Jeremiah Sullins, and Barry Gholson. 2004. Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media 29, 3 (2004), 241–250.

[16]

Betsy DiSalvo. 2016. Participatory design through a learning science lens. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, New York, NY, USA, 4459–4463.

Digital Library

[17]

Sidney K. D'Mello and Art Graesser. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2 (April 2012), 145–157.

[18]

Sidney K. D'Mello and Arthur C. Graesser. 2014. Confusion. In International Handbook of Emotions in Education, Reinhard Pekrun and Lisa Linnenbrink-Garcia (eds.). New York, NY: Routledge, 289–310.

[19]

Sidney K. D'Mello, Blair Lehman, Reinhard Pekrun, and Art Graesser. 2014. Confusion can be beneficial for learning. Learning and Instruction 29, 1 (2014), 153–170.

[20]

Yehudit Judy Dori, Zemira R. Mevarech, and Dale R. Baker (Eds.). 2018. Cognition, Metacognition, and Culture in STEM Education. Springer, Cham, CH. Retrieved from https://doi.org/10.1007/978-3-319-66659-4

[21]

Jason E. Dowd, Ives Araujo, and Eric Mazur. 2015. Making sense of confusion: Relating performance, confidence, and self-efficacy to expressions of confusion in an introductory physics class. Phys. Rev. ST Phys. Educ. Res. 11, 1 (March 2015), 010107.

[22]

David Dunning. 2011. Chapter five - The Dunning–Kruger effect: On being ignorant of one's own ignorance. In Advances in Experimental Social Psychology, James M. Olson and Mark P. Zanna (eds.). Academic Press, 247–296.

[23]

Anastasia Efklides. 2006. Metacognition and affect: What can metacognitive experiences tell us about the learning process? Educational Research Review 1, 1 (January 2006), 3–14.

[24]

Anastasia Efklides. 2008. Metacognition: Defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist 13, 4 (January 2008), 277–287.

[25]

John H. Flavell. 1976. Metacognitive aspects of problem solving. In The Nature of Intelligence, L. B. Resnick (ed.). Erlbaum, Hillsdale, NJ, 231–236.

[26]

John H. Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist 34, 10 (1979), 906–911.

[27]

Matthias Gamer, Jim Lemon, Ian Fellows, and Puspendra Singh. 2019. irr: Various coefficients off interrater reliability and agreement.

[28]

Peter M. Gollwitzer and Bernd Schaal. 1998. Metacognition in action: The importance of implementation intentions. Personality and Social Psychology Review 2, 2 (May 1998), 124–136.

[29]

George M. Harrison and Lisa M. Vallin. 2018. Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition and Learning 13, 1 (April 2018), 15–38.

[30]

Neil T. Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human earning and teaching. Int J Artif Intell Educ 24, 4 (2014), 470–497.

[31]

Eddie Huang, Hannah Valdiviejas, and Nigel Bosch. 2019. I'm sure! Automatic detection of metacognition in online course discussion forums. In Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII 2019), IEEE, Piscataway, NJ, 241–247.

[32]

Yang Jiang, Nigel Bosch, Ryan S. Baker, Luc Paquette, Jaclyn Ocumpaugh, Juliana Ma. Alexandra L. Andres, Allison L. Moore, and Gautam Biswas. 2018. Expert feature-engineering vs. deep neural networks: Which is better for sensor-free affect detection? In Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018), Springer, Cham, CH, 198–211.

[33]

Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman. 2017. Explaining the gap: Visualizing one's predictions improves recall and comprehension of data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), Association for Computing Machinery, New York, NY, USA, 1375–1386.

Digital Library

[34]

John S. Kinnebrew, James R. Segedy, and Gautam Biswas. 2014. Analyzing the temporal evolution of students’ behaviors in open-ended learning environments. Metacognition Learning 9, 2 (August 2014), 187–215.

[35]

James A. Kulik and J. D. Fletcher. 2016. Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research 86, 1 (March 2016), 42–78.

[36]

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159–174.

[37]

Blair Lehman and Art Graesser. 2015. To resolve or not to resolve? That is the big question about confusion. In Artificial Intelligence in Education (Lecture Notes in Computer Science), Springer International Publishing, Cham, CH, 216–225.

[38]

Diane Litman and Kate Forbes-Riley. 2013. Towards improving (meta)cognition by adapting to student uncertainty in tutorial dialogue. In International Handbook of Metacognition and Learning Technologies, Roger Azevedo and Vincent Aleven (eds.). Springer, New York, NY, 385–396.

[39]

Dastyni Loksa, Amy J. Ko, Will Jernigan, Alannah Oleson, Christopher J. Mendez, and Margaret M. Burnett. 2016. Programming, problem solving, and self-awareness: Effects of explicit guidance. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, New York, NY, USA, 1449–1461.

Digital Library

[40]

Richard E. Mayer. 2016. The role of metacognition in STEM games and simulations. In Using Games and Simulations for Teaching and Assessment: Key Issues, Harold F. O'Neil, Eva L. Baker and Ray S. Perez (eds.). Routledge, 183–205.

[41]

Anabil Munshi, Ramkumar Rajendran, Jaclyn Ocumpaugh, Gautam Biswas, Ryan S. Baker, and Luc Paquette. 2018. Modeling learners’ cognitive and affective states to scaffold SRL in open-ended learning environments. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization (UMAP ’18), Association for Computing Machinery, New York, NY, USA, 131–138.

Digital Library

[42]

Joseph D. Novak and Alberto J. Cañas. 2008. The theory underlying concept maps and how to construct and use them. Florida Institute for Human and Machine Cognition (IHMC).

[43]

Ernesto Panadero. 2017. A review of self-regulated learning: Six Models and four directions for research. Frontiers in Psychology 8, (2017).

[44]

Nancy E. Perry and Philip H. Winne. 2006. Learning from learning kits: gStudy traces of students’ self-regulated engagements with computerized content. Educ Psychol Rev 18, 3 (September 2006), 211–228.

[45]

James Prather, Raymond Pettit, Brett A. Becker, Paul Denny, Dastyni Loksa, Alani Peters, Zachary Albrecht, and Krista Masci. 2019. First things first: Providing metacognitive scaffolding for interpreting problem prompts. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE ’19), Association for Computing Machinery, New York, NY, USA, 531–537.

Digital Library

[46]

James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive difficulties faced by novice programmers in automated assessment tools. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER ’18), Association for Computing Machinery, New York, NY, USA, 41–50.

Digital Library

[47]

R Core Team. 2013. R: A language and environment for statistical computing. (2013).

[48]

Steve Ritter, Michael Yudelson, Stephen E. Fancsali, and Susan R. Berman. 2016. How mastery learning works at scale. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (L@S ’16), ACM, New York, NY, 71–79.

Digital Library

[49]

Rod D. Roscoe and Michelene T. H. Chi. 2007. Understanding tutor learning: Knowledge-building and knowledge-telling in peer tutors’ explanations and questions. Review of Educational Research 77, 4 (December 2007), 534–574.

[50]

Jonathan W. Schooler, Stellan Ohlsson, and Kevin Brooks. 1993. Thoughts beyond words: When language overshadows insight. Journal of Experimental Psychology: General 122, 2 (1993), 166–183.

[51]

Gregory Schraw. 2009. A conceptual analysis of five measures of metacognitive monitoring. Metacognition Learning 4, 1 (April 2009), 33–45.

[52]

Gregory Schraw, Kent J. Crippen, and Kendall Hartley. 2006. Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education 36, 1 (March 2006), 111–139.

[53]

Gregory Schraw and Rayne Sperling Dennison. 1994. Assessing metacognitive awareness. Contemporary Educational Psychology 19, 4 (1994), 460–475.

[54]

James R. Segedy, John S. Kinnebrew, and Gautam Biswas. 2015. Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics 2, 1 (May 2015), 13–48–13–48.

[55]

Valerie J. Shute, Matthew Ventura, and Yoon Jeon Kim. 2013. Assessment and learning of qualitative physics in Newton's Playground. The Journal of Educational Research 106, 6 (2013), 423–430.

[56]

Kimberly D. Tanner. 2012. Promoting student metacognition. CBE—Life Sciences Education 11, 2 (June 2012), 113–120.

[57]

Marcel V. J. Veenman, Laura Bavelaar, Levina De Wolf, and Marieke G. P. Van Haaren. 2014. The on-line assessment of metacognitive skills in a computerized learning environment. Learning and Individual Differences 29, (January 2014), 123–130.

[58]

Marcel V. J. Veenman, Bernadette H. A. M. Van Hout-Wolters, and Peter Afflerbach. 2006. Metacognition and learning: Conceptual and methodological considerations. Metacognition Learning 1, 1 (April 2006), 3–14.

[59]

Kim-Phuong L. Vu, Gerard L. Hanley, Thomas Z. Strybel, and Robert W. Proctor. 2000. Metacognitive processes in human-computer Interaction: Self-assessments of knowledge as predictors of computer expertise. International Journal of Human–Computer Interaction 12, 1 (May 2000), 43–71.

[60]

Thiemo Wambsganss, Christina Niklaus, Matthias Cetto, Matthias Söllner, Siegfried Handschuh, and Jan Marco Leimeister. 2020. AL: An adaptive learning support system for argumentation skills. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[61]

Margaret C. Wang, Geneva D. Haertel, and Herbert J. Walberg. 1990. What influences learning? A content analysis of review literature. The Journal of Educational Research 84, 1 (September 1990), 30–43.

[62]

Shang Wang, Deniz Sonmez Unal, and Erin Walker. 2019. MindDot: Supporting effective cognitive behaviors in concept map-based learning environments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[63]

Shang Wang, Erin Walker, and Ruth Wylie. 2017. What matters in concept mapping? Maps learners create or how they create them. In Artificial Intelligence in Education (Lecture Notes in Computer Science), Springer International Publishing, Cham, CH, 406–417.

[64]

Yingbin Zhang, Luc Paquette, Ryan S. Baker, Jaclyn Ocumpaugh, Nigel Bosch, Anabil Munshi, and Gautam Biswas. 2020. The relationship between confusion and metacognitive strategies in Betty's Brain. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK20), ACM, New York, NY, 276–284.

Digital Library

Cited By

Lee HBosch N(2024)Subtopic-specific heterogeneity in computer-based learning behaviorsInternational Journal of STEM Education10.1186/s40594-024-00519-x11:1Online publication date: 24-Dec-2024
https://doi.org/10.1186/s40594-024-00519-x
Goslen AVandenberg JZambrano ANasiar NHutt SOcumpaugh JRowe JDorodchi MZhange MCooper S(2024)Student Perspectives on Expressing Academic Emotions in Digital Game-Based LearningProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 210.1145/3649409.3691087(316-317)Online publication date: 5-Dec-2024
https://dl.acm.org/doi/10.1145/3649409.3691087
Wu WZhou XShi XWu PLin XYang JHe L(2024)The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2332031(1-28)Online publication date: 16-Apr-2024
https://doi.org/10.1080/10447318.2024.2332031
Show More Cited By

Index Terms

Students’ Verbalized Metacognition During Computerized Learning

Index terms have been assigned to the content through auto-classification.

Recommendations

Can Adaptive Pedagogical Agents' Prompting Strategies Improve Students' Learning and Self-Regulation?
ITS 2016: Proceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 9684

This study examines whether an ITS that fosters the use of metacognitive strategies can benefit from variations in its prompts based on learners' self-regulatory behaviors. We use log files and questionnaire data from 116 participants who interacted ...
Are Pedagogical Agents' External Regulation Effective in Fostering Learning with Intelligent Tutoring Systems?
ITS 2016: Proceedings of the 13th International Conference on Intelligent Tutoring Systems - Volume 9684

In this study we tested whether external regulation provided by artificial pedagogical agents PAs was effective in facilitating learners' self-regulated learning SRL and can therefore foster complex learning with a hypermedia-based intelligent tutoring ...
The design of an electronic self-regulation skill notebook for the development of meta-cognitive strategies and self-assessment in digital game-based learning environments
SpringSim '08: Proceedings of the 2008 Spring simulation multiconference

Since the introduction of lantern slides in the early 1900s, educators have sought to enhance the teaching and learning process through the use of technology. We have pursued television, film, overhead projectors, computer-based instruction, and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
813
Total Downloads

Downloads (Last 12 months)270
Downloads (Last 6 weeks)39

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lee HBosch N(2024)Subtopic-specific heterogeneity in computer-based learning behaviorsInternational Journal of STEM Education10.1186/s40594-024-00519-x11:1Online publication date: 24-Dec-2024
https://doi.org/10.1186/s40594-024-00519-x
Goslen AVandenberg JZambrano ANasiar NHutt SOcumpaugh JRowe JDorodchi MZhange MCooper S(2024)Student Perspectives on Expressing Academic Emotions in Digital Game-Based LearningProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 210.1145/3649409.3691087(316-317)Online publication date: 5-Dec-2024
https://dl.acm.org/doi/10.1145/3649409.3691087
Wu WZhou XShi XWu PLin XYang JHe L(2024)The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2332031(1-28)Online publication date: 16-Apr-2024
https://doi.org/10.1080/10447318.2024.2332031
Earle-Randell TWiggins JMa YCelepkolu MBounajim DGao ZRuiz JBoyer KIsrael MLynch CWiebe E(2024)The impact of near-peer virtual agents on computer science attitudes and collaborative dialogueInternational Journal of Child-Computer Interaction10.1016/j.ijcci.2024.10064640(100646)Online publication date: Jun-2024
https://doi.org/10.1016/j.ijcci.2024.100646
Liu XGurung ABaker RBarany A(2024)Understanding the Impact of Observer Effects on Student AffectAdvances in Quantitative Ethnography10.1007/978-3-031-76332-8_7(79-94)Online publication date: 2-Nov-2024
https://doi.org/10.1007/978-3-031-76332-8_7
Baker RHutt SBosch NOcumpaugh JBiswas GPaquette LAndres JNasiar NMunshi A(2023)Detector-driven classroom interviewing: focusing qualitative researcher time by selecting cases in situEducational technology research and development10.1007/s11423-023-10324-y72:5(2841-2863)Online publication date: 22-Dec-2023
https://doi.org/10.1007/s11423-023-10324-y
Ludwig SRausch A(2022) The relationship between problem‐solving behaviour and performance – Analysing tool use and information retrieval in a computer‐based office simulation Journal of Computer Assisted Learning10.1111/jcal.1277039:2(617-643)Online publication date: 26-Dec-2022
https://doi.org/10.1111/jcal.12770

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents