research-article

Open access

Improving Model Fairness with Time-Augmented Bayesian Knowledge Tracing

Authors:

Kobi GalAuthors Info & Claims

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

Pages 46 - 54

https://doi.org/10.1145/3636555.3636849

Published: 18 March 2024 Publication History

All formats PDF

Abstract

Modelling student performance is an increasingly popular goal in the learning analytics community. A common method for this task is Bayesian Knowledge Tracing (BKT), which predicts student performance and topic mastery using the student’s answer history. While BKT has strong qualities and good empirical performance, like many machine learning approaches it can be prone to bias. In this study we demonstrate an inherent bias in BKT with respect to students’ income support levels and gender, using publicly available data. We find that this bias is likely a result of the model’s ‘slip’ parameter disregarding answer speed when deciding if a student has lost mastery status. We propose a new BKT model variation that directly considers answer speed, resulting in a significant fairness increase without sacrificing model performance. We discuss the role of answer speed as a potential cause of BKT model bias, as well as a method to minimise bias in future implementations.

References

[1]

Ghodai Abdelrahman, Qing Wang, and Bernardo Nunes. 2023. Knowledge Tracing: A Survey. ACM Comput. Surv. 55, 11, Article 224 (feb 2023), 37 pages. https://doi.org/10.1145/3569576

Digital Library

[2]

Fangzhe Ai, Yishuai Chen, Yuchun Guo, Yongxiang Zhao, Zhenzhu Wang, Guowei Fu, and Guangyan Wang. 2019. Concept-aware deep knowledge tracing and exercise recommendation in an online learning system. In The 12th International Conference on Educational Data Mining. ERIC, 240–245.

[3]

Anirudhan Badrinath, Frederic Wang, and Zachary Pardos. 2021. pybkt: An accessible python library of bayesian knowledge tracing models. arXiv preprint arXiv:2105.00385 (2021).

[4]

Ryan S Baker and Aaron Hawn. 2021. Algorithmic bias in education. International Journal of Artificial Intelligence in Education (2021), 1–41. https://doi.org/10.35542/osf.io/pbmvz

[5]

George Clement Bond. 1981. Social Economic Status and Educational Achievement: A Review Article. https://doi.org/10.1525/aeq.1981.12.4.05x1811q

[6]

Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 134–148. https://doi.org/10.1145/3555101

Digital Library

[7]

Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4 (1994), 253–278. https://doi.org/10.1007/BF01099821

[8]

Yossi Ben David, Avi Segal, and Ya’akov (Kobi) Gal. 2016. Sequencing Educational Content in Classrooms Using Bayesian Knowledge Tracing. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (Edinburgh, United Kingdom) (LAK ’16). Association for Computing Machinery, New York, NY, USA, 354–363. https://doi.org/10.1145/2883851.2883885

Digital Library

[9]

Shayan Doroudi and Emma Brunskill. 2019. Fairer but Not Fair Enough On the Equitability of Knowledge Tracing. Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 335–339. https://doi.org/10.1145/3303772.3303838

Digital Library

[10]

Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226. https://doi.org/10.1145/2090236.2090255

Digital Library

[11]

Matt Faus. 2014. Khan Academy Mastery Mechanics. https://mattfaus.com/2014/07/03/khan-academy-mastery-mechanics/

[12]

Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. 2009. Addressing the Assessment Challenge with an Online System That Tutors as it Assesses. User Modeling and User-Adapted Interaction 19 (2009), 243–266. https://doi.org/10.1007/s11257-009-9063-7

Digital Library

[13]

Dawn Forman, Lovemore Nyatanga, and Terry Rich. 2002. E-learning and educational diversity. Nurse Education Today 22, 1 (2002), 76–82. https://doi.org/10.1054/nedt.2001.0740

[14]

Josh Gardner, Christopher Brooks, and Ryan Baker. 2019. Evaluating the fairness of predictive student models through slicing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge. 225–234. https://doi.org/10.1145/3303772.3303791

Digital Library

[15]

Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, 3323–3331. https://doi.org/10.5555/3157382.3157469

Digital Library

[16]

Kenneth Holstein and Shayan Doroudi. 2021. Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?arXiv preprint arXiv:2104.12920 (2021). https://doi.org/10.48550/arXiv.2104.12920

[17]

https://www.eedi.com. [n. d.]. NeurIPS 2020 Education Challenge. https://eedi.com/projects/neurips-education-challenge

[18]

Qian Hu and Huzefa Rangwala. 2020. Towards Fair Educational Data Mining: A Case Study on Detecting At-Risk Students. International Educational Data Mining Society (2020).

[19]

Stephen Hutt, Margo Gardner, Angela L Duckworth, and Sidney K D’Mello. 2019. Evaluating Fairness and Generalizability in Models Predicting On-Time Graduation from College Applications. International Educational Data Mining Society (2019).

[20]

Weijie Jiang and Zachary A Pardos. 2021. Towards equity and algorithmic fairness in student grade prediction. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 608–617. https://doi.org/10.1145/3461702.3462623

Digital Library

[21]

Young J Juhn, Euijung Ryu, Chung-Il Wi, Katherine S King, Momin Malik, Santiago Romero-Brufau, Chunhua Weng, Sunghwan Sohn, Richard R Sharp, and John D Halamka. 2022. Assessing socioeconomic bias in machine learning algorithms in health care: a case study of the HOUSES index. Journal of the American Medical Informatics Association 29, 7 (04 2022), 1142–1151. https://doi.org/10.1093/jamia/ocac052

[22]

Slava Kalyuga. 2007. Enhancing instructional efficiency of interactive e-learning environments: A cognitive load perspective. Educational psychology review 19 (2007), 387–399. https://doi.org/10.1007/s10648-007-9051-6

[23]

Kim Kelly, Yan Wang, Tamisha Thompson, and Neil Heffernan. 2016. Defining mastery: Knowledge tracing versus n-consecutive correct responses. Student Modeling From Different Aspects, 39.

[24]

Lynn Kettell. 2020. Young adult carers in higher education: the motivations, barriers and challenges involved – a UK study. Journal of Further and Higher Education 44, 1 (2020), 100–112. https://doi.org/10.1080/0309877X.2018.1515427

[25]

Mohammad Khajah, Robert V Lindsey, and Michael C Mozer. 2016. How deep is knowledge tracing?arXiv preprint arXiv:1604.02416 (2016).

[26]

Michael Madaio, Su Lin Blodgett, Elijah Mayfield, and Ezekiel Dixon-Román. 2021. Confronting structural inequities in AI for education. arXiv preprint arXiv:2105.08847 (2021).

[27]

Kate Malleson. 2018. Equality Law and the Protected Characteristics. Modern Law Review 81, 4 (2018), 598–621. https://doi.org/10.1111/1468-2230.12353

[28]

Chric Warwick-Evans. Michelle Chance. 2020. Social Class Discrimination – Time for a New Protected Characteristic for a Post-Covid Britain? (2020). https://www.rosenblatt-law.co.uk/insight/social-class-discrimination-time-for-a-new-protected-characteristic-for-a-post-covid-britain/

[29]

Ayesha Nadeem, Olivera Marjanovic, Babak Abedin, 2022. Gender bias in AI-based decision-making systems: a systematic literature review. Australasian Journal of Information Systems 26 (2022). https://doi.org/10.3127/ajis.v26i0.3835

[30]

OECD. 2015. The ABC of Gender Equality in Education. 35–61 pages. https://doi.org/10.1787/9789264229945-en

[31]

Radek Pelánek. 2017. Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27 (2017), 313–350. https://doi.org/10.1007/s11257-017-9193-2

Digital Library

[32]

Radek Pelánek. 2018. Conceptual Issues in Mastery Criteria: Differentiating Uncertainty and Degrees of Knowledge, Vol. 19. Artificial Intelligence in Education, 450–461. https://doi.org/10.1007/978-3-319-93843-1_33

[33]

Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems 28 (2015). https://doi.org/10.5555/2969239.2969296

Digital Library

[34]

Yumeng Qiu, Yingmei Qi, Hanyuan Lu, Zachary Pardos, and Neil Heffernan. 2011. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing. EDM 2011 - Proceedings of the 4th International Conference on Educational Data Mining, 139–148.

[35]

Kit T Rodolfa, Hemank Lamba, and Rayid Ghani. 2021. Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policy. Nature Machine Intelligence 3, 10 (2021), 896–904. https://doi.org/10.1038/s42256-021-00396-x

[36]

Salisu Muhammad Sani, Abdullahi Baffa Bichi, and Shehu Ayuba. 2016. Artificial intelligence approaches in student modeling: Half decade review (2010-2015). IJCSN-International Journal of Computer Science and Network 5, 5.

[37]

Lele Sha, Mladen Rakovic, Alexander Whitelock-Wainwright, David Carroll, Victoria M Yew, Dragan Gasevic, and Guanliang Chen. 2021. Assessing algorithmic fairness in automatic classifiers of educational forum posts. In Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14–18, 2021, Proceedings, Part I 22. Springer, 381–394. https://doi.org/10.1007/978-3-030-78292-4_31

Digital Library

[38]

Sunny Shrestha and Sanchari Das. 2022. Exploring gender biases in ML and AI academic research through systematic literature review. Frontiers in artificial intelligence 5 (2022), 976838. https://doi.org/10.3389/frai.2022.976838

[39]

Sebastian Tschiatschek, Maria Knobelsdorf, and Adish Singla. 2022. Equity and Fairness of Bayesian Knowledge Tracing. Proceedings of the 15th International Conference on Educational Data Mining, 578–582. https://doi.org/10.5281/zenodo.6853011

[40]

Jan Van Bavel, Christine R Schwartz, and Albert Esteve. 2018. The reversal of the gender gap in education and its consequences for family life. Annual review of sociology 44 (2018), 341–360. https://doi.org/10.1146/annurev-soc-073117-041215

[41]

Wim J Van Der Linden. 2009. Conceptual issues in response-time modeling. Journal of Educational Measurement 46, 3 (2009), 247–272. https://doi.org/10.1111/j.1745-3984.2009.00080.x

[42]

Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. 2021. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1748–1757. https://doi.org/10.1145/3447548.3467326

Digital Library

[43]

Sonja Bethune Wendy Conaway. 2015. Implicit Bias and First Name Stereotypes: What Are the Implications for Online Instruction?Journal of Asynchronous Learning Networks 19, 3. https://doi.org/10.24059/olj.v19i3.674

[44]

Renzhe Yu, Qiujie Li, Christian Fischer, Shayan Doroudi, and Di Xu. 2020. Towards Accurate and Fair Prediction of College Success: Evaluating Different Sources of Student Data. International Educational Data Mining Society (2020).

Cited By

Wu LChen XLiu FXie JXia CTan ZTian MLi JZhang KLian DHong RWang M(2025)EduStudio: towards a unified library for student cognitive modelingFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-40372-319:8Online publication date: 1-Aug-2025
https://dl.acm.org/doi/10.1007/s11704-024-40372-3

Index Terms

Improving Model Fairness with Time-Augmented Bayesian Knowledge Tracing
1. Applied computing
  1. Education
    1. E-learning
2. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms

Recommendations

Evaluating Bayesian Knowledge Tracing for Estimating Learner Proficiency and Guiding Learner Behavior
L@S '20: Proceedings of the Seventh ACM Conference on Learning @ Scale

Open navigation online learning systems allow learners to choose the next learning activity. These systems can be instrumented to provide learners with feedback to help them choose the next learning activity. One type of feedback is providing an ...
Toward the Evaluation of Educational Videos using Bayesian Knowledge Tracing and Big Data
L@S '15: Proceedings of the Second (2015) ACM Conference on Learning @ Scale

Along with the advent of MOOCs and other online learning platforms such as Khan Academy, the role of online education has continued to grow in relation to that of traditional on-campus instruction. Rather than tackle the problem of evaluating large ...
Structured Knowledge Tracing Models for Student Assessment on Coursera
L@S '16: Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Massive Open Online Courses (MOOCs) provide an effective learning platform with various high-quality educational materials accessible to learners from all over the world. However, current MOOCs lack personalized learning guidance and intelligent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference

March 2024

962 pages

ISBN:9798400716188

DOI:10.1145/3636555

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

LAK '24

LAK '24: The 14th Learning Analytics and Knowledge Conference

March 18 - 22, 2024

Kyoto, Japan

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
547
Total Downloads

Downloads (Last 12 months)547
Downloads (Last 6 weeks)64

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu LChen XLiu FXie JXia CTan ZTian MLi JZhang KLian DHong RWang M(2025)EduStudio: towards a unified library for student cognitive modelingFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-024-40372-319:8Online publication date: 1-Aug-2025
https://dl.acm.org/doi/10.1007/s11704-024-40372-3

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten