research-article

Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning

Authors:

Patrick J. Donnelly,

Sidney K. D'MelloAuthors Info & Claims

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 13

https://doi.org/10.1145/3313831.3376418

Published: 23 April 2020 Publication History

Abstract

Like anyone, teachers need feedback to improve. Due to the high cost of human classroom observation, teachers receive infrequent feedback which is often more focused on evaluating performance than on improving practice. To address this critical barrier to teacher learning, we aim to provide teachers with detailed and actionable automated feedback. Towards this end, we developed an approach that enables teachers to easily record high-quality audio from their classes. Using this approach, teachers recorded 142 classroom sessions, of which 127 (89%) were usable. Next, we used speech recognition and machine learning to develop teacher-generalizable computer-scored estimates of key dimensions of teacher discourse. We found that automated models were moderately accurate when compared to human coders and that speech recognition errors did not influence performance. We conclude that authentic teacher discourse can be recorded and analyzed for automatic feedback. Our next step is to incorporate the automatic models into an interactive visualization tool that will provide teachers with objective feedback on the quality of their discourse.

References

[1]

Alibali, M.W. et al. 2014. How teachers link ideas in mathematics instruction using speech and gesture: a corpus analysis. Cognition and Instruction. 32, 1 (2014), 65--100. 1.

[2]

Archer, J. et al. 2016. Better Feedback for Better Teaching: A Practical Guide to Improving Classroom Observations.

[3]

Aslan, S. et al. 2019. Investigating the impact of a real-time, multimodal student engagement analytics technology in authentic classrooms. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19 (New York, New York, USA, 2019), 1--12.

Digital Library

[4]

Azevedo, R. and Bernard, R.M. 1995. A metaanalysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research. 13, 2 (1995), 111--127.

[5]

Ball, D.L. and Cohen, D.K. 1999. Developing practice, developing practitioners: toward a practicebased theory of professional education. Teaching as the learning profession: Handbook of policy and practice. G. Sykes and L. Darling-Hammond, eds. Jossey-Bass Inc. 3--32.

[6]

Birman, B.F. et al. 2007. State and local implementation of the no child left behind act. Volume II - Teacher Quality Under NCLB: Interim Report. U.S. Department of Education.

[7]

Blanchard, N. et al. 2016. Automatic detection of teacher questions from audio in live classrooms. Proceedings of the 9th International Conference on Educational Data Mining (EDM 2016) (2016).

[8]

Borko, H. 2004. Professional development and teacher learning: mapping the terrain. Educational Researcher. 33, 8 (2004), 3--15.

[9]

Bransford, J.D. et al. eds. 2000. How people learn: brain, mind, experience, and school. National Academy Press.

[10]

Brill, E. 1992. A simple rule-based part of speech tagger. Proceedings of the Workshop on Speech and Natural Language (Harriman, NY, 1992), 112--116.

Digital Library

[11]

Cantrell, S. and Kane, T.J. 2013. Ensuring Fair and Reliable Measures of Effective Teaching: Culminating Findings from the MET Project's Three-Year Study.

[12]

Church, K.W. and Hanks, P. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics. 16, 1 (1990), 76--83.

Digital Library

[13]

Cook, C. et al. 2018. An open vocabulary approach for estimating teacher use of authentic questions in classroom discourse. Proceedings of the 11th International Conference on Educational Data Mining (2018).

[14]

D'Mello, S.K. et al. 2010. Expert tutors' feedback is immediate, direct, and discriminating. Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference, FLAIRS23 (Menlo Park, CA, 2010), 504--509.

[15]

D'Mello, S.K. et al. 2015. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction - ICMI '15 (New York, New York, USA, 2015), 557--566.

Digital Library

[16]

Danielson, C. 2012. Observing classroom practice. Educational Leadership. 70, 2 (2012), 32--37.

[17]

Donnelly, P.J. et al. 2016. Automatic teacher modeling from live classroom audio. Proceedings of the 24th Conference on User Modeling, Adaptation, and Personalization (UMAP 2016) (2016).

Digital Library

[18]

Donnelly, P.J. et al. 2017. Words matter: automatic detection of questions in classroom discourse using linguistics, paralinguistics, and context. LAK '17: Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (2017).

[19]

Duckworth, A.L. et al. 2010. Deliberate practice spells success: why grittier competitors triumph at the national spelling bee. Social Psychological and Personality Science. 2, 2 (2010), 174--181.

[20]

Duke, N. et al. 2012. Reading and writing genre with purpose in a k-8 classroom. Heinemann.

[21]

Ericsson, K.A. 2006. The influence of experience and deliberate practice on the development of superior expert performance. The Cambridge Handbook of Expertise and Expert Performance. K.A. Ericsson et al., eds. Cambridge University Press. 685--706.

[22]

Ericsson, K.A. et al. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological Review. 100, 3 (1993), 363--406.

[23]

Eyben, F. et al. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. Proceedings of the 21st ACM international conference on Multimedia - MM '13 (New York, New York, USA, 2013), 835--838.

Digital Library

[24]

Fadde, P.J. and Klein, G.A. 2010. Deliberate performance: accelerating expertise in natural settings. Performance Improvement. 49, 9 (2010), 5-- 14.

[25]

Ford, M. et al. 2008. The LENA Language Environment Analysis System.

[26]

Gamoran, A. and Nystrand, M. 1992. Taking students seriously. Student Engagement and Achievement in American Secondary Schools. F.M. Newmann, ed. Teachers College Press. 40--61.

[27]

Garet, M.S. et al. 2011. Middle School Mathematics Professional Development Impact Study: Findings After the Second Year of Implementation.

[28]

Garet, M.S. et al. 2008. The Impact of Two Professional Development Interventions on Early Reading Instruction and Achievement.

[29]

Goldman, R. et al. eds. Video research in the learning sciences. Erlbaum.

[30]

Graesser, A.C. et al. 2009. What is a good question? Threads of coherence in research on the development of reading ability. M.G. McKeown and L. Kucan, eds. Guilford Press. 112--141.

[31]

Grossman, P. et al. 2013. Measure for measure: the relationship between measures of instructional practice in middle school english language arts and teachers' value-added scores. American Journal of Education. 119, 3 (2013), 445--470.

[32]

Gwet, K.L. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology. 61, (2008), 29--48.

[33]

Hill, H.C. et al. 2008. Mathematical knowledge for teaching and the mathematical quality of instruction: an exploratory study. Cognition and Instruction. 26, 4 (2008), 430--511.

[34]

Ho, A.D. and Kane, T.J. 2013. The Reliability of Classroom Observations by School Personnel.

[35]

Hochreiter, S. and Schmidhuber, J. 1997. Long short-term memory. Neural Computation. 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[36]

Holstein, K. et al. 2018. Student learning benefits of a mixed-reality teacher awareness tool in aienhanced classrooms. Artificial Intelligence in Education (2018), 154--168.

[37]

Juzwik, M.M. et al. 2013. Inspiring dialogue: talking to learn in the english classroom. Teachers College Press.

[38]

Kelly, S. et al. 2018. Automatically measuring question authenticity in real-world classrooms. Educational Researcher. 47, 7 (2018), 451--464.

[39]

Kelly, S. 2007. Classroom discourse and the distribution of student engagement. Social Psychology of Education. 10, 3 (2007), 331--352.

[40]

Lemaître, G. et al. 2017. Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. The Journal of Machine Learning Research. 18, (2017), 559--563. ume18/16--365/16--365.pdf.

[41]

Mikolov, T. et al. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems (2013), 3111--3119.

[42]

Molenaar, I. and Knoop-van Campen, C. 2017. Teacher dashboards in practice: usage and impact. 12th European Conference on Technology Enhanced Learning, EC-TEL 2017 (Tallinn, Estonia, 2017), 125--138.

[43]

Molenaar, I. and Knoop-van Campen, C.A.N. 2019. How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies. 12, 3 (Jul. 2019), 347--355.

Digital Library

[44]

Murphy, P.K. et al. 2009. Examining the effects of classroom discussion on students' comprehension of text: a meta-analysis. Journal of Educational Psychology. 101, 3 (2009), 740--764.

[45]

Newman, F.M. et al. 1992. The significance and sources of student engagement. Student Engagement and Achievement in American Secondary Schools. F.M. Newman, ed. Teachers College Press. 11--39.

[46]

Nystrand, M. et al. 1997. Opening dialogue: understanding the dynamics of language and learning in the english classroom. Teachers College Press.

[47]

Nystrand, M. et al. 2003. Questions in time: investigating the structure and dynamics of unfolding classroom discourse. Discourse Processes. 35, 2 (Mar. 2003), 135--198.

[48]

Olney, A.M. et al. 2017. Assessing the dialogic properties of classroom discourse: proportion models for imbalanced classes. Proceedings of the 10th International Conference on Educational Data Mining (2017), 162--167.

[49]

Olney, A.M. et al. 2003. Utterance classification in autotutor. Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing (2003), 1--8.

Digital Library

[50]

Pedregosa, F. et al. 2011. Scikit-learn: machine learning in python. Journal of Machine Learning Research. 12, (2011), 2825--2830.

[51]

Pennington, J. et al. 2014. Glove: global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (2014), 1532--1543.

[52]

Ramakrishnan, A. et al. 2019. Toward automated classroom observation?: predicting positive and negative climate. IEEE Conference on Automatic Face and Gesture Recognition (2019).

[53]

Raudenbush, S.W. et al. 1993. Higher order instructional goals in secondary schools: class, teacher, and school influences. American Educational Research Journal. 30, 3 (Jan. 1993), 523--553.

[54]

Resnick, L.B. and Schantz, F. 2015. Re-thinking intelligence: schools that build the mind. European Journal of Education. 50, 3 (2015), 340--349.

[55]

Rodriguez-Triana, M.J. et al. 2017. Monitoring, awareness and reflection in blended technology enhanced learning: a systematic review. IJTEL. 9, 2-- 3 (2017), 126--150.

[56]

Samei, B. et al. 2014. Domain independent assessment of dialogic properties of classroom discourse. Proceedings of the 7th International Conference on Educational Data Mining (EDM). (2014), 233--236.

[57]

Samei, B. et al. 2015. Modeling classroom discourse: do models that predict dialogic instruction properties generalize across populations? Proceedings of the 8th International Conference on Educational Data Mining (2015), 444--447.

[58]

Saon, G. et al. 2015. The ibm 2015 english conversational telephone speech recognition system. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH (Dresden, Germany, 2015), 3140-- 3144.

[59]

Schuller, B. et al. 2009. The interspeech 2009 emotion challenge. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. (2009), 312--315.

[60]

Shernoff, D.J. 2013. Optimal learning environments to promote student engagement. Springer New York.

[61]

Shernoff, D.J. et al. 2016. Student engagement as a function of environmental complexity in high school classrooms. Learning and Instruction. 43, (2016), 52--60. 003.

[62]

Shernoff, D.J. et al. 2003. Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly. 18, 2 (2003), 158.

[63]

Shute, V.J. 2008. Focus on formative feedback. Review of Educational Research. 78, 1 (2008), 153-- 189.

[64]

Stigler, J.W. et al. 1999. The TIMSS Videotape Classroom Study: Methods and Findings from an Exploratory Research Project on Eighth-Grade Mathematics Instruction in Germany, Japan, and the United States. A Research and Development Report.

[65]

Stigler, J.W. and Miller, K.F. 2018. Expertise and expert performance in teaching. The Cambridge Handbook of Expertise and Expert Performance. K.A. Ericsson et al., eds. Cambridge University Press. 431--452.

[66]

Stone, C. et al. 2019. Utterance-level modeling of indicators of engaging classroom discourse. The 12th International Conference on Educational Data Mining (Montreal, Canada, 2019), 420--425.

[67]

Suresh, A. et al. 2019. Automating analysis and feedback to improve mathematics' teachers' classroom discourse. Proceedings of the Ninth Symposium on Educational Advances in Artificial Intelligence (EAAI) (2019).

[68]

Suresh, A. et al. 2018. Using deep learning to automatically detect talk moves in teachers'mathematics lessons. Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. (2018), 5445--5447. .

[69]

Taylor, B.M. et al. 2003. The influence of teacher practices that encourage cognitive engagement in literacy learning. The Elementary School Journal. 104, 1 (2003), 3--28.

[70]

TNTP 2015. The Mirage: Confronting the Hard Truth About Our Quest for Teacher Development.

[71]

Vaswani, A. et al. 2017. Attention is all you need. Advances in neural information processing systems (2017), 5998--6008.

[72]

Wang, Z. et al. 2013. Using the lena in teacher training: promoting student involvement through automated feedback. Unterrichtswissenshaft. 4, (2013), 290--305.

[73]

Xhakaj, F. et al. 2017. Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. EC-TEL. 315--329.

Cited By

Ramya R(2024)Teaching, Learning, and Coaching Using Generative AI ToolsImproving Student Assessment With Emerging AI Tools10.4018/979-8-3693-6170-2.ch014(387-414)Online publication date: 29-Nov-2024
https://doi.org/10.4018/979-8-3693-6170-2.ch014
Murgai AAndure VValiba DThorat R(2024)THE ROLE OF AI IN EDUCATION: TRANSFORMING LEARNING THROUGH CHATGPT AND EMERGING TECHNOLOGIESShodhKosh: Journal of Visual and Performing Arts10.29121/shodhkosh.v5.i6.2024.28345:6Online publication date: 30-Jun-2024
https://doi.org/10.29121/shodhkosh.v5.i6.2024.2834
Peña-Torres J(2024)Towards an improved of teaching practice using Sentiment Analysis in Student EvaluationIngeniería y Competitividad10.25100/iyc.v26i2.1375926:2Online publication date: 20-Jun-2024
https://doi.org/10.25100/iyc.v26i2.13759
Show More Cited By

Index Terms

Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Usability testing

Recommendations

Automatic Teacher Modeling from Live Classroom Audio
UMAP '16: Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization

We investigate automatic analysis of teachers' instructional strategies from audio recordings collected in live classrooms. We collected a data set of teacher audio and human-coded instructional activities (e.g., lecture, question and answer, group work)...
Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

We focus on data collection designs for the automated analysis of teacher-student interactions in live classrooms with the goal of identifying instructional activities (e.g., lecturing, discussion) and assessing the quality of dialogic instruction (e.g.,...
Multi-sensor modeling of teacher instructional segments in live classrooms
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

We investigate multi-sensor modeling of teachers’ instructional segments (e.g., lecture, group work) from audio recordings collected in 56 classes from eight teachers across five middle schools. Our approach fuses two sensors: a unidirectional ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

10688 pages

ISBN:9781450367080

DOI:10.1145/3313831

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
1,296
Total Downloads

Downloads (Last 12 months)233
Downloads (Last 6 weeks)23

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ramya R(2024)Teaching, Learning, and Coaching Using Generative AI ToolsImproving Student Assessment With Emerging AI Tools10.4018/979-8-3693-6170-2.ch014(387-414)Online publication date: 29-Nov-2024
https://doi.org/10.4018/979-8-3693-6170-2.ch014
Murgai AAndure VValiba DThorat R(2024)THE ROLE OF AI IN EDUCATION: TRANSFORMING LEARNING THROUGH CHATGPT AND EMERGING TECHNOLOGIESShodhKosh: Journal of Visual and Performing Arts10.29121/shodhkosh.v5.i6.2024.28345:6Online publication date: 30-Jun-2024
https://doi.org/10.29121/shodhkosh.v5.i6.2024.2834
Peña-Torres J(2024)Towards an improved of teaching practice using Sentiment Analysis in Student EvaluationIngeniería y Competitividad10.25100/iyc.v26i2.1375926:2Online publication date: 20-Jun-2024
https://doi.org/10.25100/iyc.v26i2.13759
L’Enfant J(2024)AI as a reflective coach in graduate ESL practicum: activity theory insights into student-teacher developmentEuropean Journal of Open, Distance and E-Learning10.2478/eurodl-2024-000326:s1(1-19)Online publication date: 8-Nov-2024
https://doi.org/10.2478/eurodl-2024-0003
A. SGulfisha SA. KMane S(2024)Exploring the Fundamental Aspects of Artificial Intelligence: A Comprehensive OverviewAI Innovations in Drug Delivery and Pharmaceutical Sciences; Advancing Therapy through Technology10.2174/9789815305753124010005(28-61)Online publication date: 14-Nov-2024
https://doi.org/10.2174/9789815305753124010005
Moreau-Pernet BTian YSawaya SFoltz PCao JMilne BChristie TJoyner DKim MWang XXia M(2024)Classifying Tutor Discursive Moves at Scale in Mathematics Classrooms with Large Language ModelsProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664664(361-365)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664664
Booth BJacobs JBush JMilne BFischaber TDMello S(2024)Human-tutor Coaching Technology (HTCT): Automated Discourse Analytics in a Coached Tutoring ModelProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636937(725-735)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636937
Zhao LEcheverria VSwiecki ZYan LAlfredo RLi XGasevic DMartinez-Maldonado R(2024)Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team LearningProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636855(90-100)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636855
Ngoon TSushil SStewart ALee UVenkatraman SThawani NMitra PClarke SZimmerman JOgan A(2024)ClassInSight: Designing Conversation Support Tools to Visualize Classroom Discussion for Personalized Teacher Professional DevelopmentProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642487(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642487
Zhao LGašević DSwiecki ZLi YLin JSha LYan LAlfredo RLi XMartinez‐Maldonado R(2024)Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analyticsBritish Journal of Educational Technology10.1111/bjet.13476Online publication date: 30-May-2024
https://doi.org/10.1111/bjet.13476
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents