research-article

Tracking knowledge proficiency of students with calibrated Q-matrix

Authors:

Xiangchun HeAuthors Info & Claims

Volume 192, Issue C

https://doi.org/10.1016/j.eswa.2021.116454

Published: 15 April 2022 Publication History

Highlights

•

A new calibrated Q-matrix-based framework is designed for knowledge tracing.

•

The relationships between exercises and knowledge points are deeply explored.

•

Three Q-matrix calibration methods are proposed to eliminate the bias in Q-matrix.

•

The diagnosis results are applied on three educational tasks.

Abstract

With the emergence of intelligent educational systems, numerous research works are dedicated to Knowledge Tracing (KT), which refers to the issue of diagnosing students’ changing knowledge proficiency in exercises. Recent developments in KT have yielded immense success on this task and they mainly use sophisticated and flexible deep neural network-based models to fully exploit the interaction information between students and response logs. However, these models either ignore the significance of Q-matrix associated exercises with knowledge concepts (KCs) or fail to avoid the subjective tendency of experts within the Q-matrix. To tackle these problems, in this paper, we devise a novel Calibrated Q-matrix-based Knowledge Tracing (CQKT) framework to track knowledge proficiency of students dynamically in KT. To be specific, for the original Q-matrix, we primarily strive to capture the high-order connectivity between exercises and KCs to obtain potential KCs of each exercise by utilizing graph convolution network. Then, three Q-matrix calibration methods based on a pairwise Bayesian treatment equipped with potential KCs are adopted to refine and calibrate the raw Q-matrix so that the subjective tendency of the Q-matrix defined by domain experts can be weakened. After that, the embedding of each exercise aggregated the calibrated Q-matrix with historical student interactions is injected into the Long Short-Term Memory (LSTM) network to trace students’ knowledge states. Extensive experiments are conducted on three real-world benchmark datasets and the results show the superiority of CQKT. In particular, we demonstrate its practicability via applying it to three fundamental educational tasks, including score prediction, knowledge state estimation, and diagnosis result visualization.

References

[1]

Chen, Y., Liu, Q., Huang, Z., Le, W., Chen, E., Wu, R. Z., Su, Y., & Hu, G. (2017, November). Tracking Knowledge Proficiency of Students with Educational Priors. In Proceedings of the 26th ACM on Conference on Information and Knowledge Management (pp. 989-998), Singapore, Singapore.

[2]

S.E. Embretson, S.P. Reise, Item response theory, Psychology Press, 2013.

[3]

F. Fouss, A. Pirotte, J.-M. Renders, M. Saerens, Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation, IEEE Transactions on Knowledge and Data Engineering 19 (3) (2007) 355–369.

Digital Library

[4]

Glorot, X., & Bengio, Y. (2010, March). Understanding the Difficulty of Training Deep Feedforward Neural Networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (pp. 249-256), Sardinia, Italy.

[5]

X. He, K. Deng, X. Wang, Y. Li, Y.D. Zhang, M. Wang, LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation, in: In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, China, 2020, pp. 639–648.

[6]

Z. Huang, Q.i. Liu, Y. Chen, L.e. Wu, K. Xiao, E. Chen, …., G. Hu, Learning or forgetting? a dynamic approach for tracking the knowledge proficiency of students, ACM Transactions on Information Systems 38 (2) (2020) 1–33.

[7]

Y. Huo, D.F. Wong, L.M. Ni, L.S. Chao, J. Zhang, HeTROPY: Explainable learning diagnostics via heterogeneous maximum-entropy and multi-spatial knowledge representation, Knowledge-Based Systems 207 (2020) 106389,.

[8]

Y. Huo, D.F. Wong, L.M. Ni, L.S. Chao, J. Zhang, Knowledge modeling via contextualized representations for LSTM-based personalized exercise recommendation, Information Sciences 523 (2020) 266–278.

[9]

D.L.T. Jimmy, DINA model and parameter estimation: A didactic, Journal of Nutrition Education and Behavioral Statistics 523 (2009) 266–278.

[10]

D.L.T. Jimmy, C.-Y. Chiu, A general method of empirical Q-matrix validation, Psychometrika 81 (2) (2016) 253–273.

[11]

Jimmy, D. L. T. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199.

[12]

Kingma, D., & Ba, J. (2015, May). Adam: A method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representation, San Diego, CA.

[13]

Liu, Y., Yang, Y., Chen, X., Shen., Zhang, H., & Yu, Y. (2020, July). Improving Knowledge Tracing via Pre-training Question Embeddings. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (pp. 1577-1583), Yokohama, Japan.

[14]

Q.i. Liu, Z. Huang, Y.u. Yin, E. Chen, H. Xiong, Y.u. Su, G. Hu, EKT: Exercise-aware knowledge tracing for student performance prediction, IEEE Transactions on Knowledge and Data Engineering 33 (1) (2021) 100–115.

Digital Library

[15]

Q. Liu, R.Z. Wu, E. Chen, G. Xu, Y. Su, Z. Chen, G. Hu, Fuzzy cognitive diagnosis for modelingexaminee performance, ACM Transactions on Intelligent Systems and Technology 9 (4) (2018) 1–26.

[16]

J. Liu, G. Xu, Z. Ying, Data-driven learning of Q-matrix, Applied Psychological Measurement 36 (7) (2012) 548–564.

[17]

W. Ma, D.L.T. Jimmy, An empirical Q-matrix validation method for the sequential generalized DINA model, British Journal of Mathematical and Statistical Psychology 73 (1) (2020) 142–163.

[18]

W. Ma, D.L.T. Jimmy, GDINA: An R package for cognitive diagnosis modeling, Journal of Statistical Software 93 (14) (2020) 1–26.

[19]

Piech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., & Jascha, S. D. (2015 December). Deep Knowledge Tracing. In Proceedings of the 28th International Conference on Neural Information Processing Systems (pp. 505-513), Bali, Indonesia.

[20]

André A. Rupp, Jonathan Templin, The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model, Educational and Psychological Measurement 68 (1) (2008) 78–96.

[21]

Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016, June). Meta-learning with Memory-augmented Neural Networks. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1842-1850), New York, NY.

[22]

X. Song, J. Li, S. Sun, H. Yin, P. Dawson, R. Doss, SEPN: A sequential engagement based academic performance prediction model, IEEE Intelligent Systems 36 (1) (2020) 46–53.

[23]

X. Song, J. Li, Y. Tang, T. Zhao, Y. Chen, Z. Guan, JKT: A joint graph convolutional network based deep knowledge tracing, Information Sciences 580 (2021) (2021) 510–523.

[24]

Su, Y., Liu, Q., Liu, Q., Huang. Z., Yin, Y., Chen, E., Ding, C., Wei, S., & Hu, G. (2018, February). Exercise-enhanced Sequential Modeling for Student Performance Prediction. In Proceedings of 32nd AAAI Conference on Artificial Intelligence (pp. 2435-2443), New Orleans, LA.

[25]

J.L. Templin, R.A. Henson, Measurement of psychological disorders using cognitive diagnosis models, Psychological Methods 11 (3) (2006) 287–305.

[26]

D. Wang, Y. Cai, D. Tu, Q-matrix estimation methods for cognitive diagnosis models: Based on Partial Known Q-matrix, Multivariate Behavioral Research 2020 (2020) 1–13.

[27]

Wang, X., He, X., Wang, M., Feng, F., & Chua, T. S. (2019, July). Neural Graph Collaborative Filtering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 165-174), Paris, France.

[28]

Wang, C., Zhu, H., Zhu, Chen., Zhang, X., Chen, E., & Xiong, H. (2020, April). Personalized Employee Training Course Recommendation with Career Development Awareness. In Proceedings of the 29th Internationa World Wide Web Conference (pp. 1648-1659), Taipei, China.

[29]

Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., & Wang, S. (2020, February). Neural Cognitive Diagnosis for Intelligent Education Systems. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (pp. 6153-6161), New York, NY.

[30]

Wang, W., Ma, H., Zhao, Y., Li, Z., & He, X. (2021 September). Relevance-Aware Q-matrix Calibration for Knowledge Tracing. In 30th International Conference on Artificial Neural Networks (pp. 101-112), Bratislava, Slovakia.

[31]

Gongjun Xu, Zhuoran Shang, Identifying latent structures in restricted latent class models, Journal of the American Statistical Association 113 (523) (2018) 1284–1295.

[32]

Yeung, C. K., & Yeung, D. Y. (2018, June). Addressing Two Problems in Deep Knowledge Tracing via Prediction-Consistent Regularization. In Proceedings of the 5th Annual ACM Conference on Learning at Scale (pp. 1-10), London, UK.

[33]

Zhang, J., Shi, X., King, I., & Y, D. Y. (2017, April). Dynamic Key-Value Memory Networks for Knowledge Tracing. In Proceedings of the 36th International Conference on World Wide Web, Perth, Australia.

[34]

Zhang, M., Zhu, X., Zhang, C., Ji, Y., Pan, F., & Yin, C. (2021). Multi-factors aware dual-attentional knowledge tracing. arXiv preprint arXiv:2108.04741.

[35]

Yuwen Zhou, Changqin Huang, Qintai Hu, Jia Zhu, Yong Tang, Personalized learning full-path recommendation model based on LSTM neural networks, Information Sciences 444 (2018) 135–152.

Cited By

Wang WMa HZhao YLi Z(2024)Pre-training Question Embeddings for Improving Knowledge Tracing with Self-supervised Bi-graph Co-contrastive LearningACM Transactions on Knowledge Discovery from Data10.1145/363805518:4(1-20)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1145/3638055
Du HPalaoag TGuo T(2023)Personalized Learning Resource Recommendation Framework Based on Knowledge MapProceedings of the 2023 International Conference on Advances in Artificial Intelligence and Applications10.1145/3603273.3634709(132-136)Online publication date: 18-Nov-2023
https://dl.acm.org/doi/10.1145/3603273.3634709
Zhou JWu ZYuan CZeng L(2023)A Causality-Based Interpretable Cognitive Diagnosis ModelNeural Information Processing10.1007/978-981-99-8067-3_16(214-226)Online publication date: 20-Nov-2023
https://dl.acm.org/doi/10.1007/978-981-99-8067-3_16
Show More Cited By

Index Terms

Tracking knowledge proficiency of students with calibrated Q-matrix

Index terms have been assigned to the content through auto-classification.

Recommendations

Tracking Knowledge Proficiency of Students with Educational Priors
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management

Diagnosing students' knowledge proficiency, i.e., the mastery degrees of a particular knowledge point in exercises, is a crucial issue for numerous educational applications, e.g., targeted knowledge training and exercise recommendation. Educational ...
Dynamic Key-Value Memory Networks for Knowledge Tracing
WWW '17: Proceedings of the 26th International Conference on World Wide Web

Knowledge Tracing (KT) is a task of tracing evolving knowledge state of students with respect to one or more concepts as they engage in a sequence of learning activities. One important purpose of KT is to personalize the practice sequence to help ...
Relevance-Aware Q-matrix Calibration for Knowledge Tracing
Artificial Neural Networks and Machine Learning – ICANN 2021
Abstract
Knowledge tracing (KT) lies at the core of intelligent education, which aims to diagnose students’ changing knowledge level over time based on their historical performance. Most of the existing KT models either ignore the significance of Q-matrix ...

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 192, Issue C

Apr 2022

831 pages

ISSN:0957-4174

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 15 April 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang WMa HZhao YLi Z(2024)Pre-training Question Embeddings for Improving Knowledge Tracing with Self-supervised Bi-graph Co-contrastive LearningACM Transactions on Knowledge Discovery from Data10.1145/363805518:4(1-20)Online publication date: 12-Feb-2024
https://dl.acm.org/doi/10.1145/3638055
Du HPalaoag TGuo T(2023)Personalized Learning Resource Recommendation Framework Based on Knowledge MapProceedings of the 2023 International Conference on Advances in Artificial Intelligence and Applications10.1145/3603273.3634709(132-136)Online publication date: 18-Nov-2023
https://dl.acm.org/doi/10.1145/3603273.3634709
Zhou JWu ZYuan CZeng L(2023)A Causality-Based Interpretable Cognitive Diagnosis ModelNeural Information Processing10.1007/978-981-99-8067-3_16(214-226)Online publication date: 20-Nov-2023
https://dl.acm.org/doi/10.1007/978-981-99-8067-3_16
Gao WMa HZhao YWang JTian Q(2023)Modeling Portraits of Students and Exercises for Exercise RecommendationAdvanced Intelligent Computing Technology and Applications10.1007/978-981-99-4752-2_19(226-236)Online publication date: 10-Aug-2023
https://dl.acm.org/doi/10.1007/978-981-99-4752-2_19
Wang WMa HZhao YYang FChang L(2022)PERM: Pre-training Question Embeddings via Relation Map for Improving Knowledge TracingDatabase Systems for Advanced Applications10.1007/978-3-031-00129-1_22(281-288)Online publication date: 11-Apr-2022
https://dl.acm.org/doi/10.1007/978-3-031-00129-1_22

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents