Abstract
This paper presents the results of three evaluation studies performed during 1998 and 1999 on SQL-Tutor, an intelligent tutoring system for the SQL database language. We have evaluated the system in the context of genuine courses, and used the results to further refine the system. The main goal of our research has been the exploration and extension of Constraint-Based Modeling (CBM), a student modeling approach proposed by Ohlsson (1994). SQL-Tutor provided us with experiences of using CBM, and we used it to extend the approach in several important ways. The main goal of all three evaluation studies was to determine how well CBM supported student learning. We have obtained positive results. The students who learnt with SQL-Tutor in the first study performed significantly better than those who did not when assessed by a subsequent classroom examination. Furthermore, the analysis of students' learning shows that CBM has a sound psychological foundation.
Besides the evaluation of CBM, we also evaluated the improvements in terms of student assessments of the usefulness of the system and evaluated various techniques used in SQL-Tutor. In the second study, we evaluated the effectiveness of feedback provided to the students. This study showed that high-level advice is most beneficial to students' learning. The focus of the third study was different. We extended CBM to support long-term modeling of student knowledge, and used this extension to develop an adaptive problem-selection strategy. The study revealed the benefits of this strategy in comparison with a simple heuristic strategy. We also reflect on our experiences in evaluating SQL-Tutor.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Albacete, P. L. and Van Lehn, K.: 2000, The Conceptual Helper: an Intelligent Tutoring System for Teaching Fundamental Physics Concepts. 5th In ternational Conference on Intelligent Tutoring Systems, Montreal, pp. 564–573.
Anderson, J. R, Corbett, A. T., Koedinger, K. R. and Pelletier, R.: 1995, Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences 4(2), 167–207.
Bloom, B. S.: 1984, The 2 Sigma Problem: the Search for Methods of Group Instruction as Effective as one-to-one Tutoring. Educational Researcher 13, 4–16.
Chao-Lin Liu and Wellman, M.: 1998, Incremental Tradeoff Resolution in Qualitative Probability Networks. 14th Conference on Uncertainty in Artificial Intelligence, Madison, WI, pp. 346–353.
Charniak, E.: 1991, Bayesian Networks without Tears. AI Magazine, Winter 1991, 50–63.
Chin, D. N.: 2000, Empirical Evaluation of User Models and User-adapted Systems. User-Modeling and User Adapted Interaction, 12(2-3), 1–5.
Collins, J. A., Greer, J. and Huang, S. X.: 1996, Adaptive Assessment Using Granularity Hierarchies and Bayesian Nets. 3rd International Conference on Intelligent Tutoring Systems, Montreal, pp. 569-577.
Conati, C., Gertner, A., Van Lehn, K. and Druzdzel, M. J.: 1997, On-Line Student Modeling for Coached Problem Solving Using Bayesian Networks. 6th International Conference on User Modeling, Sardinia, pp. 231–242.
Forgy, C. L.: 1982, Rete: a Fast Algorithm for theMany Pattern/Many Object Pattern Match Problem. Artificial Intelligence 19, 17–37.
Gertner, A. S.: 1998, Providing Feedback to Equation Entries in an Intelligent Tutoring System for Physics. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 254–263.
Mayo. M.: 2000, A Normative Framework for Intelligent Tutoring Systems, 5th International Conference on Intelligent Tutoring Systems, Montreal (poster).
Mayo, M., Mitrovic, A. and McKenzie, J.: 2000, CAPIT: an Intelligent Tutoring System for Capitalization and Punctuation. International Workshop on Advanced Learning Technologies, Palmerston North, NZ, pp. 151–154.
Mayo, M. and Mitrovic, A.: 2001, Optimising ITS Behavior with Bayesian Networks and Decision Theory. Accepted for International Journal on Artificial Intelligence in Education.
Millan, E. and Perez-de-la-Cruz, J. L.: 2001, A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation. UMUAI, this volume.
Mitrovic, A.: 1998a, A Knowledge-Based Teaching System for SQL. ED-MEDIA'98, Freiburg, pp. 1027–1032.
Mitrovic, A.: 1998b, Experiences in Implementing Constraint-Based Modeling in SQL-Tutor. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 414–423.
Mitrovic, A. and Hausler, K.: 2000, Porting SQL-Tutor to the Web. ITS'2000 Workshop on Adaptive and Intelligent Web-based Education Systems, Montreal, pp. 37–44.
Mitrovic, A. and Ohlsson, S.: 1999, Evaluation of a Constraint-Based Tutor for a Database Language. Int. Journal. on Artificial Intelligence in Education 10(3–4), 238–256.
Mitrovic, A. and Suraweera, P.: 2000, Evaluating an Animated Pedagogical Agent. 5th International Conference on Intelligent Tutoring Systems, Montreal, pp. 73–82.
Newell, A. and Rosenblum, P. S.: 1981, Mechanisms of skill acquisition and the law of practice. In: J.R. Anderson (ed.), Cognitive Skills and Their Acquisition, pp. 1–55.
Ohlsson, S.: 1994, Constraint-based Student Modeling. In: J. E. Greer and G.I. McCalla (eds.), Student Modeling: the Key to Individualized Knowledge-based Instruction, pp. 167–189.
Ohlsson, S.: 1996, Learning from Performance Errors. Psychological Review 103(2) 241–262.
Pearl, J.: 1988, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.
Reye, J.: 1998. Two-Phase Updating of Student Models Based on Dynamic Belief Networks. 4th International Conference on Intelligent Tutoring Systems, San Antonio, TX, pp. 274–283.
Stern, L. and Sterling, L.: 1997, Teaching AI Algorithms using Animations Reinforced by Interactive Exercises. Australian Computer Science Education Conference, Sydney, pp. 78–83.
Suraweera, P. and Mitrovic, A.: 2001,Designing a Constraint-based Tutor for Database Design. To be presented at the 9th Int. Conf. on Human-Computer Interaction, New Orleans.
Vigotsky, L. S.: 1978, The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Mitrovic, A., Martin, B. & Mayo, M. Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor. User Modeling and User-Adapted Interaction 12, 243–279 (2002). https://doi.org/10.1023/A:1015022619307
Issue Date:
DOI: https://doi.org/10.1023/A:1015022619307