Just Accepted
Model-Agnostic Adaptive Testing for Intelligent Education Systems via Meta-learned Gradient Embeddings
Accepted on 04 April 2024
Abstract
The field of education has undergone a significant revolution with the advent of intelligent systems and technology, which aim to personalize the learning experience, catering to the unique needs and abilities of individual learners. In this pursuit, a fundamental challenge is designing proper test for assessing the students’ cognitive status on knowledge and skills accurately and efficiently. One promising approach, referred to as Computerized Adaptive Testing (CAT), is to administrate computer-automated tests that alternately select the next item for each examinee and estimate their cognitive states given their responses to the selected items. Nevertheless, existing CAT systems suffer from inflexibility in item selection and ineffectiveness in cognitive state estimation, respectively. In this paper, we propose a Model-Agnostic adaptive testing framework via Meta-leaned Gradient Embeddings, MAMGE for short, improving both item selection and cognitive state estimation simultaneously. For item selection, we design a Gradient Embedding based Item Selector (GEIS) which incorporates the concept of gradient embeddings to represent items and selects the best ones that are both informative and representative. For cognitive state estimation, we propose a Meta-learned Cognitive State Estimator (MCSE) to automatically control the estimation process by learning to learn a proper initialization and dynamically inferred updates. Both MCSE and GEIS are inherently model-agnostic, and the two modules have an ingenious connection via meta-learned gradient embeddings. Finally, extensive experiments evaluate the effectiveness and flexibility of MAMGE.
References
[1]
Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. 2022. Sharp-maml: Sharpness-aware model-agnostic meta learning. In International conference on machine learning. PMLR, 10–32.
[2]
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. In International Conference on Learning Representations.
[3]
Philip Bachman, Alessandro Sordoni, and Adam Trischler. 2017. Learning algorithms for active learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 301–310.
[4]
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. 2020. Meta-learning with adaptive hyperparameters. Advances in neural information processing systems 33 (2020), 20755–20765.
[5]
Sungyong Baik, Seokil Hong, and Kyoung Mu Lee. 2020. Learning to forget for meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2379–2387.
[6]
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. 2007. Margin based active learning. In International Conference on Computational Learning Theory. Springer, 35–50.
[7]
Md Abul Bashar and Richi Nayak. 2021. Active learning for effectively fine-tuning transfer learning to downstream task. ACM Transactions on Intelligent Systems and Technology (TIST) 12, 2 (2021), 1–24.
[8]
Haoyang Bi, Haiping Ma, Zhenya Huang, Yu Yin, Qi Liu, Enhong Chen, Yu Su, and Shijin Wang. 2020. Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing. In 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 42–51.
[9]
Hua-Hua Chang. 2015. Psychometrics behind computerized adaptive testing. Psychometrika 80, 1 (2015), 1–20.
[10]
Hua-Hua Chang and Zhiliang Ying. 1996. A global information approach to computerized adaptive testing. Applied Psychological Measurement 20, 3 (1996), 213–229.
[11]
Seung W Choi, Matthew W Grady, and Barbara G Dodd. 2011. A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement 71, 1 (2011), 37–53.
[12]
Susan E Embretson and Steven P Reise. 2013. Item response theory. Psychology Press.
[13]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. PMLR, 1126–1135.
[14]
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In International Conference on Machine Learning. PMLR, 1183–1192.
[15]
Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. 2021. RCD: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 501–510.
[16]
Aritra Ghosh and Andrew Lan. 2021. BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing. In International Joint Conference on Artificial Intelligence.
[17]
Stephen M Haley, Pengsheng Ni, Ronald K Hambleton, Mary D Slavin, and Alan M Jette. 2006. Computer adaptive testing improved accuracy and precision of scores over random item selection in a physical functioning item bank. Journal of clinical epidemiology 59, 11 (2006), 1174–1182.
[18]
Yuting Hong, Shiwei Tong, Wei Huang, Yan Zhuang, Qi Liu, Enhong Chen, Xin Li, and Yuanjing He. 2023. Search-Efficient Computerized Adaptive Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 773–782.
[19]
Giles Hooker, Matthew Finkelman, and Armin Schwartzman. 2009. Paradoxical results in multidimensional item response theory. Psychometrika 74, 3 (2009), 419–442.
[20]
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2021. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence 44, 9 (2021), 5149–5169.
[21]
H Jiaji, C Rewon, R Vinay, L Hairong, S Sanjeev, and C Adam. 2016. Active learning for speech recognition: The power of gradients. In The 30th Conference on Neural Information Processing Systems, NIPS. Barcelona, Spain. 1–5.
[22]
Weisen Jiang, James Kwok, and Yu Zhang. 2022. Subspace learning for effective meta-learning. In International Conference on Machine Learning. PMLR, 10177–10194.
[23]
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. 2015. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, Vol. 2. Lille.
[24]
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. 2017. Learning active learning from data. Advances in neural information processing systems 30 (2017).
[25]
Elisabeth Lex and Markus Schedl. 2022. Psychology-informed recommender systems tutorial. In Proceedings of the 16th ACM Conference on Recommender Systems. 714–717.
[26]
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 (2017).
[27]
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1 (2019), 100–115.
[28]
Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. 2018. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST) 9, 4 (2018), 1–26.
[29]
Shuo Liu, Junhao Shen, Hong Qian, and Aimin Zhou. 2024. Inductive Cognitive Diagnosis for Fast Student Learning in Web-Based Online Intelligent Education Systems. In Proceedings of the ACM Web Conference 2024. Singapore, Singapore.
[30]
Wenhe Liu, Xiaojun Chang, Ling Chen, Dinh Phung, Xiaoqin Zhang, Yi Yang, and Alexander G Hauptmann. 2020. Pair-based uncertainty and diversity promoting early active learning for person re-identification. ACM Transactions on Intelligent Systems and Technology 11, 2 (2020), 1–15.
[31]
Frederic M Lord. 1980. Applications of item response theory to practical testing problems. Routledge.
[32]
David Magis, Duanli Yan, and Alina A Von Davier. 2017. Computerized adaptive and multistage testing with R: Using packages catr and mstr. Springer.
[33]
Andrew J Martin and Goran Lazendic. 2018. Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of educational psychology 110, 1 (2018), 27.
[34]
Craig N Mills and Manfred Steffen. 2000. The GRE computer adaptive test: Operational issues. In Computerized adaptive testing: Theory and practice. Springer, 75–99.
[35]
Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In International Conference on Machine Learning. PMLR, 2554–2563.
[36]
Darkhan Nurakhmetov. 2019. Reinforcement Learning Applied to Adaptive Classification Testing. Theoretical and Practical Advances in Computer-based Educational Measurement (2019), 325.
[37]
Martin Plajner and Jirí Vomlel. 2016. Student Skill Models in Adaptive Testing. In Probabilistic Graphical Models. 403–414.
[38]
Lawrence M Rudner. 2009. Implementing the graduate management admission test computerized adaptive test. In Elements of adaptive testing. Springer, 151–165.
[39]
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In International conference on machine learning. PMLR, 1842–1850.
[40]
Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017).
[41]
Burr Settles. 2011. From theories to queries: Active learning in practice. In Active learning and experimental design workshop in conjunction with AISTATS 2010. JMLR Workshop and Conference Proceedings, 1–18.
[42]
Junhao Shen, Hong Qian, Wei Zhang, and Aimin Zhou. 2024. Symbolic Cognitive Diagnosis via Hybrid Optimization for Intelligent Education Systems. In Proceedings of the 38th AAAI Conference on Artificial Intelligence. Vancouver, Canada, 14928–14936.
[43]
Shuanghong Shen, Qi Liu, Zhenya Huang, Yonghe Zheng, Minghao Yin, Minjuan Wang, and Enhong Chen. 2024. A Survey of Knowledge Tracing: Models, Variants, and Applications. IEEE Transactions on Learning Technologies (2024), 1–22.
[44]
Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 (2017).
[45]
Yu Su, Zeyu Cheng, Jinze Wu, Yanmin Dong, Zhenya Huang, Le Wu, Enhong Chen, Shijin Wang, and Fei Xie. 2022. Graph-based cognitive diagnosis for intelligent tutoring systems. Knowledge-Based Systems 253 (2022), 109547.
[46]
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1199–1208.
[47]
Daniel Ting and Eric Brochu. 2018. Optimal subsampling with influence functions. In Advances in Neural Information Processing Systems. 3650–3659.
[48]
Wim J van der Linden and Cees AW Glas. 2010. Elements of adaptive testing. Springer.
[49]
Jill-Jênn Vie, Fabrice Popineau, Éric Bruillard, and Yolaine Bourda. 2017. A review of recent advances in adaptive assessment. In Learning analytics: fundaments, applications, and trends. Springer, 113–142.
[50]
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems 29 (2016), 3630–3638.
[51]
Howard Wainer, Neil J Dorans, Ronald Flaugher, Bert F Green, and Robert J Mislevy. 2000. Computerized adaptive testing: A primer. Routledge.
[52]
Chun Wang and Hua-Hua Chang. 2011. Item selection in multidimensional computerized adaptive testing—Gaining information from different angles. Psychometrika 76, 3 (2011), 363–384.
[53]
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. 2023. NeuralCD: A General Framework for Cognitive Diagnosis. IEEE Transactions on Knowledge and Data Engineering 35, 8 (2023), 8312–8327.
[54]
Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. 2021. Hypersorec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. ACM Transactions on Information Systems (TOIS) 40, 2 (2021), 1–28.
[55]
Le Wu, Qi Liu, Enhong Chen, Nicholas Jing Yuan, Guangming Guo, and Xing Xie. 2016. Relevance meets coverage: A unified framework to generate diversified recommendations. ACM Transactions on Intelligent Systems and Technology (TIST) 7, 3 (2016), 1–30.
[56]
Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen. 2017. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In International conference on medical image computing and computer-assisted intervention. Springer, 399–407.
[57]
Lei Zhang, Wuji Zhang, Likang Wu, Ming He, and Hongke Zhao. 2023. SHGCN: Socially enhanced heterogeneous graph convolutional network for multi-behavior prediction. ACM Transactions on the Web 18, 1 (2023), 1–27.
[58]
Ye Zhang, Matthew Lease, and Byron Wallace. 2017. Active discriminative text representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.
[59]
Zheng Zhang, Le Wu, Qi Liu, Jiayu Liu, Zhenya Huang, Yu Yin, Yan Zhuang, Weibo Gao, and Enhong Chen. 2023. Understanding and Improving Fairness in Cognitive Diagnosis. SCIENCE CHINA Information Sciences (2023).
[60]
Chuang Zhao, Hongke Zhao, Xiaomeng Li, Ming He, Jiahui Wang, and Jianping Fan. 2023. Cross-domain recommendation via progressive structural alignment. IEEE Transactions on Knowledge and Data Engineering (2023).
[61]
Chanjin Zheng. 2015. Some practical item selection algorithms in cognitive diagnostic computerized adaptive testing–smart diagnosis for smart learning. Ph. D. Dissertation. University of Illinois at Urbana-Champaign.
[62]
Yan Zhuang, Qi Liu, Zhenya Huang, Zhi Li, Binbin Jin, Haoyang Bi, Enhong Chen, and Shijin Wang. 2022. A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 416–426.
[63]
Yingtian Zou, Fusheng Liu, and Qianxiao Li. 2021. Unraveling model-agnostic meta-learning via the adaptation learning rate. In International Conference on Learning Representations.
Index Terms
- Model-Agnostic Adaptive Testing for Intelligent Education Systems via Meta-learned Gradient Embeddings
Recommendations
Comprehensive Fair Meta-learned Recommender System
KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data MiningIn recommender systems, one common challenge is the cold-start problem, where interactions are very limited for fresh users in the systems. To address this challenge, recently, many works introduce the meta-optimization idea into the recommendation ...
Meta-Learning with Adaptive Weighted Loss for Imbalanced Cold-Start Recommendation
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementSequential recommenders have made great strides in capturing a user's preferences. Nevertheless, the cold-start recommendation remains a fundamental challenge as they typically involve limited user-item interactions for personalization. Recently, ...
CoMeta: Enhancing Meta Embeddings with Collaborative Information in Cold-Start Problem of Recommendation
Knowledge Science, Engineering and ManagementAbstractThe cold-start problem is quite challenging for existing recommendation models. Specifically, for the new items with only a few interactions, their ID embeddings are trained inadequately, leading to poor recommendation performance. Some recent ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 23 May 2024
Accepted: 04 April 2024
Revised: 16 March 2024
Received: 14 August 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 107Total Downloads
- Downloads (Last 12 months)107
- Downloads (Last 6 weeks)35
Reflects downloads up to 10 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in