column

The sum is greater than the parts: ensembling models of student knowledge in educational software

Authors:

Zachary A, Pardos,

Sujith M. Gowda,

Ryan S.J.d. Baker,

Neil T. HeffernanAuthors Info & Claims

ACM SIGKDD Explorations Newsletter, Volume 13, Issue 2

Pages 37 - 44

https://doi.org/10.1145/2207243.2207249

Published: 01 May 2012 Publication History

Abstract

Many competing models have been proposed in the past decade for predicting student knowledge within educational software. Recent research attempted to combine these models in an effort to improve performance but have yielded inconsistent results. While work in the 2010 KDD Cup data set showed the benefits of ensemble methods, work in the Genetics Tutor failed to show similar benefits. We hypothesize that the key factor has been data set size. We explore the potential for improving student performance prediction with ensemble methods in a data set drawn from a different tutoring system, the ASSISTments Platform, which contains 15 times the number of responses of the Genetics Tutor data set. We evaluated the predictive performance of eight student models and eight methods of ensembling predictions. Within this data set, ensemble approaches were more effective than any single method with the best ensemble approach producing predictions of student performance 10% better than the best individual student knowledge model.

References

[1]

Baker, R.S.J.D., Corbett, A.T., Aleven, V., 2008. More Accurate Student Modeling Through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing. In: Proc. of the 9th International Conference on Intelligent Tutoring Systems, 406--415.

Digital Library

[2]

Baker, R.S.J.D., Corbett, A.T., Gowda, S.M., Wagner, A.Z., Maclaren, B.M., Kauffman, L.R., Mitchell, A.P., Giguere, S. 2010. Contextual Slip and Prediction of Student Performance After Use of an Intelligent Tutor. In: Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, 52--63.Baker, R.S.J.D., Corbett, A.T., Gowda, S.M., Wagner, A.Z., Maclaren, B.M., Kauffman, L.R., Mitchell, A.P., Giguere, S. 2010. Contextual Slip and Prediction of Student Performance After Use of an Intelligent Tutor. In: Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, 52--63

Digital Library

[3]

Baker, R.S.J.D., Pardos, Z., Gowda, S., Nooraei, B., Heffernan, N., 2011. Ensembling Predictions of Student Knowledge within Intelligent Tutoring Systems. Proceedings of 19th International Conference on User Modeling, Adaptation, and Personalization, 13--24.

Digital Library

[4]

Breiman, L., 1996. Bagging predictors. Machine Learning, 24, 123--14.

[5]

Breiman. L., 2001. Statistical modeling: the two cultures. Stat Sci 2001;16:199--231

[6]

Brusilovsky, P., Millán, E., 2007. User models for adaptive hypermedia and adaptive educational systems. In: P. Brusilovsky, A. Kobsa and W. Neidl (eds.): The Adaptive Web: Methods and Strategies of Web Personalization. Lecture Notes in Computer Science, Vol. 4321, Berlin Heidelberg New York: Springer-Verlag, pp. 3--53.

Digital Library

[7]

Caruana, R., Niculescu-Mizil, A, 2004. Ensemble selection from libraries of models. In: Proceedings of the 21st International Conference on Machine Learning (ICML'04).

Digital Library

[8]

Chang, K.-M., Beck, J., Mostow, J., Corbett, A, 2006. A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 104--113.

Digital Library

[9]

Corbett, A.T., Anderson, J.R., 1995. Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253--278.

[10]

Dietterich, T. G., 2000. Ensemble Methods in Machine Learning, Proceedings of the First International Workshop on Multiple Classifier Systems, p.1--15, June 21-23

Digital Library

[11]

Domingos, P., 1997. Why does bagging work? A Bayesian account and its implications. In D. Heckerman,H. Mannila, D. Pregibon, & R. Uthurusamy (Eds.), Proceedings of the Third International Conference onKnowledge Discovery and Data Mining (pp. 155--158). AAAI Press.

[12]

Freund, Y. & Schapire, R. E., 1996. Experiments with a new boosting algorithm. In Proceedings of the 13th International Conference on Machine Learning, pp. 148--146. Morgan Kaufmann.

Digital Library

[13]

Goldstein, I.J. (1979) The genetic graph: a representation for the evolution of procedural knowledge. International Journal of Man-Machine Studies, 11 (1), 51--77.

[14]

Gong, Y., Beck, J.E., Heffernan, N.T., 2010. Comparing Knowledge Tracing and Performance Factor Analysis by Using Multiple Model Fitting Procedures. In: Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 35--44.

Digital Library

[15]

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Newbury Park, CA: Sage Press.

[16]

Hanley, J. A., & Mcneil, B. J., 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29--36.

[17]

Haykin, S., 1998. Neural Networks: A Comprehensive Foundation. New York, NY: Macmillan.

Digital Library

[18]

Koedinger, K. R., Corbett, A. T., 2006. Cognitive tutors: Technology bringing learning science to the classroom. In K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 61--78). New York: Cambridge University Press.

[19]

Koedinger, K.R., Pavlik, P.I., Stamper, J., Nixon, T., Ritter, S., 2010. Avoiding Problem Selection Thrashing with Conjunctive Knowledge Tracing. Proceedings of the 3rd International Conference on Educational Data Mining, 91--100.

[20]

Langley, P., Iba, W., 1992. An Analysis of Bayesian Classifiers. Proceedings of the 10th National Conference on Artificial Intelligence, 223--228.

Digital Library

[21]

Martin, J., Vanlehn, K. (1995). Student Assessment Using Bayesian Nets. International Journal of Human-Computer Studies, 42, 575--591.

Digital Library

[22]

Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Comparison of Traditional Homework with Computer Supported Homework. Journal of Research on Technology in Education, 41(3), 331--359.

[23]

Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., Euler, T. 2006. YALE: Rapid Prototyping for Complex Data Mining Tasks. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2006), 935--940.

Digital Library

[24]

Nooraei B., B., Pardos, A. Z., Heffernan, N. T., Baker, R.S.J.D., 2011. Less is More: Improving the Speed and Prediction Power of Knowledge Tracing by Using Less Data. Proceedings of the 4th International Conference on Educational Data Mining, 101--110.

[25]

Pardos, Z. A., Heffernan, N. T., 2010a. Modeling Individualization in a Bayesian Networks Implementation of Knowledge Tracing. In P. De Bra, A. Kobsa, and D. Chin (Eds.): UMAP 2010, LNCS 6075, 225--266. Springer-Verlag: Berlin

Digital Library

[26]

Pardos, Z. A., Heffernan, N. T., 2010b. Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm. In: Proc. of the 3rd International Conference on Educational Data Mining, 161--170.

[27]

Pardos, Z.A., Heffernan, N. T., Using HMMs and bagged decision trees to leverage rich features of user and skill from an intelligent tutoring system dataset. To appear in Journal of Machine Learning Research W & CP.

[28]

Pavlik, P.I., Cen, H., Koedinger, K.R., 2009a. Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models. In: Proceedings of the 2nd International Conference on Educational Data Mining, 121--130.

[29]

Pavlik, P.I., Cen, H., Koedinger, K.R., 2009b. Performance Factors Analysis -- A New Alternative to Knowledge Tracing. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, 531--538. Version of paper used online at http://eric.ed.gov/PDFS/ED506305.pdf, retrieved 1/26/2011. This version has minor differences from the printed version of this paper.

Digital Library

[30]

Quinlan, J.R., 1993. C4.5: Programs for Machine Learning. San Francisco, CA: Morgan Kaufmann.

Digital Library

[31]

Quinlan, J.R., 1996. Bagging, boosting, and c4.5. Proceedings of the Thirteenth National Conference on Artificial Intelligence (pp. 725--730). AAAI Press and the MIT Press.

Digital Library

[32]

Rai, D, Gong, Y, Beck, J. E, 2009. Using Dirichlet priors to improve model parameter plausibility. In: Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 141--148.

[33]

Razzaq, L., Heffernan, N.T., Feng, M., Pardos, Z.A., 2007. Developing Fine-Grained Transfer Models in the ASSISTment System. Journal of Technology, Instruction, Cognition, and Learning, Vol. 5. Number 3. Old City Publishing, Philadelphia, PA. 2007. pp. 289--304.

[34]

Reye, J., 2004. Student modeling based on belief networks. International Journal of Artificial Intelligence in Education 14, 1--33.

Digital Library

[35]

Thissen, D., Mislevy, R. 2000. Testing Algorithms. In H. Wainer, N.J. Dorans (Eds.) Computerized Adaptive Testing: A Primer (pp. 101--134). Hillsdale, NJ: Lawrence Erlbaum.

[36]

Vanlehn, K., 1988. Student Modeling. In M.C. Polson & J.J. Richardson (Eds.) Foundations of Intelligent Tutoring Systems. London, UK: Psychology Press.

[37]

Yu, H-F., Lo, H-Y., Hsieh, H-P., Lou, J-K., Mckenzie, T.G., Chou, J-W., et al., 2010 Feature Engineering and Classifier Ensemble for KDD Cup 2010. Proc. of the KDD Cup 2010 Workshop, 1--16.

Cited By

Qiu YDjemili KElezi DSrazali APérez-Ortiz MYilmaz EShawe-Taylor JBulathwela SWooldridge MDy JNatarajan S(2024)A toolbox for modelling engagement with educational videosProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i21.30358(23128-23136)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i21.30358
Strukova SRuipérez-Valiente JMármol F(2023)Adapting Knowledge Inference Algorithms to Measure Geometry Competencies through a Puzzle GameACM Transactions on Knowledge Discovery from Data10.1145/361443618:1(1-23)Online publication date: 6-Sep-2023
https://dl.acm.org/doi/10.1145/3614436
Shen YWang S(2023)Explore Testing Performance and Learning BehaviorsMeasurement: Interdisciplinary Research and Perspectives10.1080/15366367.2021.200083021:4(203-231)Online publication date: 23-Oct-2023
https://doi.org/10.1080/15366367.2021.2000830
Show More Cited By

Recommendations

Approximate Reasoning by Parts: An Introduction to Rough Mereology
Wisdom in Sum of Parts: Multi-Platform Activity Prediction in Social Collaborative Sites
WebSci '18: Proceedings of the 10th ACM Conference on Web Science

In this paper, we proposed a novel framework which uses user interests inferred from activities (a.k.a., activity interests) in multiple social collaborative platforms to predict users' platform activities. Included in the framework are two prediction ...
Object Detection Using the Statistics of Parts

In this paper we describe a trainable object detector and its instantiations for detecting faces and cars at any size, location, and pose. To cope with variation in object orientation, the detector uses multiple classifiers, each spanning a different ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGKDD Explorations Newsletter

ACM SIGKDD Explorations Newsletter Volume 13, Issue 2

December 2011

101 pages

ISSN:1931-0145

EISSN:1931-0153

DOI:10.1145/2207243

Issue’s Table of Contents

Copyright © 2012 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 2012

Published in SIGKDD Volume 13, Issue 2

Check for updates

Qualifiers

Column

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
450
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qiu YDjemili KElezi DSrazali APérez-Ortiz MYilmaz EShawe-Taylor JBulathwela SWooldridge MDy JNatarajan S(2024)A toolbox for modelling engagement with educational videosProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i21.30358(23128-23136)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i21.30358
Strukova SRuipérez-Valiente JMármol F(2023)Adapting Knowledge Inference Algorithms to Measure Geometry Competencies through a Puzzle GameACM Transactions on Knowledge Discovery from Data10.1145/361443618:1(1-23)Online publication date: 6-Sep-2023
https://dl.acm.org/doi/10.1145/3614436
Shen YWang S(2023)Explore Testing Performance and Learning BehaviorsMeasurement: Interdisciplinary Research and Perspectives10.1080/15366367.2021.200083021:4(203-231)Online publication date: 23-Oct-2023
https://doi.org/10.1080/15366367.2021.2000830
Bulathwela SPérez-Ortiz MYilmaz EShawe-Taylor J(2022)Power to the Learner: Towards Human-Intuitive and Integrative Recommendations with Open Educational ResourcesSustainability10.3390/su14181168214:18(11682)Online publication date: 17-Sep-2022
https://doi.org/10.3390/su141811682
Denaro KKranzfelder POwens MSato BZuckerman AHardesty RSignorini AAebersold AVerma MLo S(2022)Predicting implementation of active learning by tenure-track teaching faculty using robust cluster analysisInternational Journal of STEM Education10.1186/s40594-022-00365-99:1Online publication date: 28-Jul-2022
https://doi.org/10.1186/s40594-022-00365-9
Ahmad SEl-Affendi MAnwar MIqbal R(2022)Potential Future Directions in Optimization of Students’ Performance Prediction SystemComputational Intelligence and Neuroscience10.1155/2022/68649552022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/6864955
Carlon MMoon SOkazaki NCross J(2022)Predicting Guesses and Slips Through Question Encoding with Complexity Hints2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)10.1109/TALE54877.2022.00054(284-291)Online publication date: Dec-2022
https://doi.org/10.1109/TALE54877.2022.00054
Denaro KSato BHarlow AAebersold AVerma M(2021)Comparison of Cluster Analysis Methodologies for Characterization of Classroom Observation Protocol for Undergraduate STEM (COPUS) DataCBE—Life Sciences Education10.1187/cbe.20-04-007720:1(ar3)Online publication date: Mar-2021
https://doi.org/10.1187/cbe.20-04-0077
Jensen EUmada THunkins NHutt SHuggins-Manley AD'Mello SScheffel MDowell NJoksimovic SSiemens G(2021)What You Do Predicts How You DoLAK21: 11th International Learning Analytics and Knowledge Conference10.1145/3448139.3448151(121-131)Online publication date: 12-Apr-2021
https://dl.acm.org/doi/10.1145/3448139.3448151
Kanchana JAmarasinghe GNanayakkara VPerera A(2021)A Data Mining Approach for Early Prediction Of Academic Performance of Students2021 IEEE International Conference on Engineering, Technology & Education (TALE)10.1109/TALE52509.2021.9678558(01-08)Online publication date: 5-Dec-2021
https://doi.org/10.1109/TALE52509.2021.9678558
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents