What’s Most Broken? Design and Evaluation of a Tool to Guide Improvement of an Intelligent Tutor
Pages 283 - 295
Abstract
Intelligent Tutoring Systems (ITS) have great potential to change the educational landscape by bringing scientifically tested one-to-one tutoring to remote and under-served areas. However, effective ITSs are too complex to perfect. Instead, a practical guiding principle for ITS development and improvement is to fix what’s most broken. This paper presents SPOT (Statistical Probe of Tutoring), a tool that mines data logged by an ITS to identify ‘hot spots’ most detrimental to its efficiency and effectiveness in terms of its software reliability, usability, task difficulty, student engagement, and other criteria. SPOT uses heuristics and machine learning to discover, characterize, and prioritize such hot spots in order to focus ITS refinement on what matters most. We applied SPOT to data logged by RoboTutor, an ITS that teaches children basic reading, writing and arithmetic. A panel-of-experts experimental evaluation found SPOT’s selected video clips of RoboTutor’s hot spots as significantly more informative than video clips selected randomly.
References
[1]
Aleven V, McLaren BM, Sewall J, and Koedinger KR Ikeda M, Ashley KD, and Chan T-W The cognitive tutor authoring tools (CTAT): preliminary evaluation of efficiency gains Intelligent Tutoring Systems 2006 Heidelberg Springer 61-70
[2]
Apté C and Weiss S Data mining with decision trees and decision rules Future Gener. Comput. Syst. 1997 13 2–3 197-210
[3]
Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-task behavior in the cognitive tutor classroom: when students game the system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 383–390. ACM (2004)
[4]
Baker RSJ, Corbett AT, Koedinger KR, and Roll I Ikeda M, Ashley KD, and Chan T-W Generalizing detection of gaming the system across a tutoring curriculum Intelligent Tutoring Systems 2006 Heidelberg Springer 402-411
[5]
d Baker, R.S., et al.: Towards sensor-free affect detection in cognitive tutor algebra. Int. Educ. Data Min. Soc. (2012)
[6]
Baker, R.S.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1059–1068. ACM (2007)
[7]
Baker, R.S., de Carvalho, A., Raspat, J., Aleven, V., Corbett, A.T., Koedinger, K.R.: Educational software features that encourage and discourage “gaming the system”. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 475–482 (2009)
[8]
Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and regression trees (1984)
[9]
Chrysafiadi K and Virvou M Damiani E, Jeong J, Howlett RJ, and Jain LC Usability factors for an intelligent tutoring system on computer programming New Directions in Intelligent Interactive Multimedia Systems and Services-2 2009 Heidelberg Springer 339-347
[10]
Chughtai, R., Zhang, S., Craig, S.D.: Usability evaluation of intelligent tutoring system: its from a usability perspective. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, pp. 367–371, SAGE Publications, Los Angeles (2015)
[11]
Cocea, M., Hershkovitz, A., Baker, R.S.: The impact of off-task and gaming behaviors on learning: immediate or aggregate? In: Proceeding of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling. IOS Press (2009)
[12]
Cocea, M., Weibelzahl, S.: Can log files analysis estimate learners’ level of motivation? In: LWA. University of Hildesheim, Institute of Computer Science (2006)
[13]
Cocea M and Weibelzahl S Duval E, Klamma R, and Wolpers M Cross-system validation of engagement prediction from log files Creating New Learning Experiences on a Global Scale 2007 Heidelberg Springer 14-25
[14]
Goel AL Software reliability models: assumptions, limitations, and applicability IEEE Trans. Softw. Eng. 1985 12 1411-1423
[15]
Granić, A., Glavinić, V.: An approach to usability evaluation of an intelligent tutoring system. In: Mastorakis, N., Kluev, V. (eds.) Advances in Multimedia, Video and Signal Processing Systems (2002)
[16]
Henry, G., Lin, J., Park, C.Y.: SPOT: refining robotutor. HCI senior capstone project presentation, Carnegie Mellon University (2017)
[17]
Joseph E Engagement tracing: using response times to model student disengagement Artif. Intell. Educ.: Support. Learn. Through Intell. Soc. Inf. Technol. 2005 125 88
[18]
Karunanithi N, Whitley D, and Malaiya YK Prediction of software reliability using connectionist models IEEE Trans. Softw. Eng. 1992 18 7 563-574
[19]
Koedinger KR, Booth JL, and Klahr D Instructional complexity and the science needed to constrain it Science 2014 342 935-937
[20]
LearnSphere: LearnSphere (2017). http://learnsphere.org/
[21]
Mostow, J., Beck, J., Cen, H., Cuneo, A., Gouvea, E., Heiner, C.: An educational data mining tool to browse tutor-student interactions: time will tell. In: Proceedings of the Workshop on Educational Data Mining, National Conference on Artificial Intelligence, pp. 15–22. AAAI Press (2005)
[22]
Murthy SK Automatic construction of decision trees from data: a multi-disciplinary survey Data Min. Knowl. Discov. 1998 2 4 345-389
[23]
Pedregosa F et al. Scikit-learn: machine learning in python J. Mach. Learn. Res. 2011 12 Oct 2825-2830
[24]
Play, G.: AZ Screen Recorder (2018). https://play.google.com/store/apps/details?id=com.hecorat.screenrecorder.free
[25]
RoboTutor: RoboTutor (2015). http://robotutor.org
[26]
Romero C and Ventura S Educational data mining: a review of the state of the art IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 2010 40 6 601-618
[27]
XPRIZE: Global Learning XPRIZE (2015). http://learning.xprize.org
Recommendations
What's most broken? a tool to assist data-driven iterative improvement of an Intelligent Tutoring System
AAAI'19/IAAI'19/EAAI'19: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial IntelligenceIntelligent Tutoring Systems (ITS) have great potential to change the educational landscape by bringing scientifically tested one-to-one tutoring to remote and under-served areas. However, effective ITSs are too complex to perfect. Instead, a practical ...
Tutorial on Rapid Development of Intelligent Tutors using the Cognitive Tutor Authoring Tools (CTAT)
ICALT '06: Proceedings of the Sixth IEEE International Conference on Advanced Learning TechnologiesIntelligent Tutoring Systems (ITS) can both help improve student learning and serve as useful platforms for experiments in learning science [1,2]. But the difficulty of building or customizing ITSs has hindered their acceptance among educators and ...
Comments
Information & Contributors
Information
Published In
Jun 2019
591 pages
ISBN:978-3-030-23203-0
DOI:10.1007/978-3-030-23204-7
© Springer Nature Switzerland AG 2019.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 25 June 2019
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Feb 2025