Leveraging Pedagogical Theories to Understand Student Learning Process with Graph-based Reasonable Knowledge Tracing

Early KT methods in machine learning, such as Bayesian Knowledge Tracing (BKT) (Corbett and Anderson, 1994), initially showcased reasonable results due to their transparent and interpretable internal structure. BKT utilizes Hidden Markov Models (HMMs) to probabilistically represent the student learning process. It transitions knowledge mastery and emits probabilities of correct responses, while also considering guessing and slipping behaviors. Subsequent KT methods expanded upon BKT by incorporating additional pedagogical factors such as question difficulty (Pardos and Heffernan, 2011) or prior student information (Yudelson et al., 2013).

However, traditional methods show inferior prediction performance when compared to subsequent emerging DLKT methods (Piech et al., 2015; Liu et al., 2019; Shen et al., 2022; Pandey and Karypis, 2019; Ghosh et al., 2020; Choi et al., 2020; Cui et al., 2024), which reach high prediction performance due to the power of neural networks. Even so, these DLKT methods fail to produce reasonable knowledge tracing results due to their inherently opaque structures. Efforts have been made to tackle this challenge. Shen et al. (Shen et al., 2021) proposed Learning Process-consistent Knowledge Tracing (LPKT), which utilizes student response duration and interval time to capture learning and forgetting behaviors. However, it only focuses on knowledge learning and forgetting and does not model the interplay of knowledge mastery changes between KCs, limiting its reasonability. Similarly, Yin et al. (Yin et al., 2023) introduced the Diagnostic Transformer (DTransformer), which diagnoses student knowledge mastery from each tackled question and employs a contrastive learning framework to produce more stable knowledge tracing. While this stability enhances reasonability to some extent, its transformer-based structures do not adequately reflect the transition of knowledge mastery between continuous student responses. Therefore, while these approaches improve model reasonability from specific angles, they do not offer a comprehensive method to generate reasonable knowledge tracing results covering both KC relations and continuous learning processes.

We address this gap with our proposed GRKT, which utilizes GNNs to model KC relations and introduces a three-stage learning process to capture evolving knowledge mastery. By integrating these techniques, GRKT achieves high prediction performance while also generating more reasonable knowledge tracing results.

Graph Neural Networks (GNNs) (Scarselli et al., 2008) serve as an efficient tool to capture intricate relations between instances in real-world scenarios. Their message aggregation and propagation operations on graphs yield deep representations for node features, enhancing performance in various downstream tasks across different domains. In the context of KT, researchers explore various structures to harness the power of GNNs. Nakagawa et al. (Nakagawa et al., 2019) pioneered the incorporation of GNNs into KT by reformulating it as a time-series node-level classification problem based on KC relation graphs. Gan et al. (Gan et al., 2022) leveraged this structure to enhance graph representation learning, generating more informative question and concept embeddings. Except for KC relations, question-question and question-KC relations are also widely considered. For instance, Bi-CLKT (Song et al., 2022) applied contrastive learning to question-KC and KC-KC graphs to generate question embeddings enriched with question and KC structural information. Another work (Yang et al., 2021) leveraged question-KC relations to address question sparsity and multi-skill problems. In our paper, we specifically focus on utilizing GNNs to model the mutual effects of KCs during students’ knowledge leveraging and changing, constructing a more reasonable approach to knowledge tracing.

It is worth noting that some other GNN-based or memory-based methods (e.g., GKT (Nakagawa et al., 2019) and DKVMN (Zhang et al., 2017)) also update mastery between KCs. However, their knowledge state updating is still potentially performed by the erase-followed-by-add mechanism, which uses GRU/LSTM cells unable to solve the reasonability issues such as not guaranteeing the direction of consistency change between KCs.

Knowledge tracing aims to trace the dynamic evolution of students’ knowledge mastery throughout their learning processes characterized by their responses to questions. Suppose there are a student set $\mathcal{U}$, a question set $\mathcal{Q}$, and a KC set $\mathcal{C}$. Each student $u\in\mathcal{U}$ has a historical response sequence $\mathcal{H}^{u}=\{r^{u}_{1},r^{u}_{2},\cdots,r^{u}_{|\mathcal{H}^{u}|}\}$, where each response $r^{u}_{t}=\left(q^{u}_{t},a^{u}_{t},c^{u}_{t},T^{u}_{t}\right)$ comprises the involved question $q^{u}_{t}\in\mathcal{Q}$, the correctness $a^{u}_{t}\in\{0,1\}$ (where $a^{u}_{t}=1$ means a correct response), the KC $c_{t}^{u}\in\mathcal{C}$ examined by the question, and the timestamp $T_{t}^{u}$ of the response. It is worth noting that there could be multiple KCs associated with one question. To be concise, we use the notations with just one KC to describe the task setting and the proposed method, but our method is easily extended to the setting of multiple KCs (e.g., averaging the KC representations as mentioned in Section 4.3). The objective is to track and monitor the evolving knowledge mastery of $u$ after each response, $\mathcal{M}^{u}=\{\textbf{m}^{u}_{1},\textbf{m}^{u}_{2},\cdots,\textbf{m}^{u}_{|\mathcal{H}^{u}|}\}$ where $\textbf{m}^{u}_{t}$ is stacked with $\{m^{u}_{c_{i},t}|c_{i}\in\mathcal{C}\}$ and $m^{u}_{c_{i},t}$ signifies the student’s knowledge mastery of the KC $c_{i}$ at time step $t$. A higher value denotes a superior level of mastery. However, the absence of annotated mastery levels necessitates researchers to resort to the student performance prediction task as a surrogate measure (Liu et al., 2021). In this paradigm, given $\mathcal{H}^{u}$, the objective is to predict whether student $u$ can correctly answer a new question $q^{u}_{|\mathcal{H}^{u}|+1}$, with its associated KC $c^{u}_{|\mathcal{H}^{u}|+1}$ at timestamp $T^{u}_{|\mathcal{H}^{u}|+1}$. This hinges on the monotonicity assumption (Embretson and Reise, 2013), which posits that higher knowledge mastery leads to a higher probability of answering questions correctly. For brevity, we omit the superscript $u$ in the later method description.

Based on the transfer of learning theory (Perkins et al., 1992), we introduce KC relation-based GNNs to transfer the knowledge leveraging and changing throughout the three-stage learning process, as shown in Figure 2. To avoid repetition, we first elaborate on a prototype of KC relation-based GNNs in this section and highlight differences when applied to different stages in the subsequent sections.

Due to the lack of KC relation annotations, we follow previous works (Nakagawa et al., 2019; Song et al., 2022) that construct KC relations based on the data statistics. Details could be referred to in Appendix B. Besides, we focus on the two most common relations, prerequisite and similarity and extend three relation graphs ${\mathcal{P},\mathcal{S},\mathcal{R}}$, whose edges denote one KC being prerequisite/subsequent/relevant (similar) to another one. This is because the forward and backward message passed along the unidirectional prerequisite relation should be differentiated. Based on this, we design the KC relation-based GNNs with multiple layers. They receive KC node features such as knowledge memory, knowledge gain/loss, or knowledge learnt in the three stages, which would be introduced later. To capture the graph information, each layer first aggregates the features of each node’s neighbors for each graph $\mathcal{G}\in\{\mathcal{P},\mathcal{S},\mathcal{R}\}$ from the last layer as

where $\textbf{W}^{\mathcal{G},(l)}_{proto}\in\mathbb{R}^{d_{l-1}\times d_{l-1}}$ and $\textbf{O}_{proto}^{\mathcal{G},(l)}\in\mathbb{R}^{d_{l-1}\times d_{l}}$ are the learnable weight matrices in this layer. $\mathcal{G}(\cdot)$ is the neighbor function of $\mathcal{G}$. $\text{ReLU}(\cdot)$ is an activation function to introduce non-linearity to enhance model representability. $\beta^{\mathcal{G}}_{c_{i},c_{j}}$ is the correlation score of KC $c_{i}$ and $c_{j}$ on graph $\mathcal{G}$, obtained by

$\textbf{k}_{c_{i}},\textbf{k}_{c_{j}}\in\mathbb{R}^{1\times d_{e}}$ are the two KCs’ embeddings where $d_{e}$ is the number of embedding dimensions. $\textbf{W}^{\mathcal{G}}_{cor}\in\mathbb{R}^{d_{e}\times d_{e}}$ is the trainable matrix for $\mathcal{G}$, and $\sigma(\cdot)$ denotes the sigmoid function, which regularizes the score in $(0,1)$. We then fuse the aggregated features from the three graphs by

where we apply a residual connection (Szegedy et al., 2017) when $d_{l-1}=d_{l}$ to stabilize the training process. In this prototype, we denote the input features of all KCs as $\tilde{\textbf{F}}^{(0)}\in\mathbb{R}^{|C|\times d_{0}}$ and one of them as $\tilde{\textbf{f}}^{(0)}_{c_{i}}\in\mathbb{R}^{1\times d_{0}}$ for KC $c_{i}$, and the output features as $\tilde{\textbf{F}}^{(L)}\in\mathbb{R}^{|C|\times d_{L}}$ and $\tilde{\textbf{f}}_{c_{i}}^{(L)}\in\mathbb{R}^{1\times d_{L}}$, where $d_{0},d_{L}$ are the numbers of input and output feature dimensions. Then this prototype GNN is formulated as:

This prototype is then extended for different student learning stages to construct reasonable knowledge tracing based on the transfer of learning theory. Besides, the number of layers $L$ controls the number of hops the feature propagates on the graphs, which clarifies the boundary between related and non-related KCs.

GRKT aims to model the process of student retrieving and learning knowledge with their memory. Therefore, we employ a dynamic knowledge memory bank denoted as $\textbf{H}\in\mathbb{R}^{|C|\times d_{k}}$, where each row $\textbf{h}_{c_{i}}$ encodes the current knowledge memory of KC $c_{i}$ for the student. Here, $d_{k}$ signifies the number of memory dimensions. This memory bank evolves alongside the student’s learning process, represented as $\textbf{H}_{t}$, with a learnable initial state $\textbf{H}_{0}$ representing their prior knowledge before engaging in any learning behavior. To track the knowledge mastery of a specific KC, we apply a non-negative projection vector $\textbf{w}_{h}\in\mathbb{R}^{d_{k}\times 1}_{\geq 0}$ to $\textbf{h}_{c_{i},t}$ using the equation:

which yields the mastery of KC $c_{i}$ at time step $t$. The non-negative constraint on the network weights guarantees the monotonic relationship between mastery and each memory dimension. This technique has been widely adopted in numerous studies (Wang et al., 2022, 2021) to satisfy the monotonicity assumption. Moreover, we leverage this constraint to establish a foundation for reasonable knowledge tracing, which would be gradually refined in subsequent descriptions.

In this stage, students retrieve stored knowledge from memory to solve given questions, a mechanism explained by memory theory (Melton, 1963). Additionally, the transfer of learning theory (Perkins et al., 1992) suggests that learners transfer knowledge from similar fields to tackle problems. Leveraging this insight, we employ a KC relation-based GNN to model knowledge transfer from related KCs. Specifically, we aggregate the knowledge memory of the given KC $c_{t}$ to solve its corresponding question $q_{t}$ before time step $t$ (represented as $t^{-}$):

with initializing $\tilde{\textbf{h}}^{(0)}_{c_{t},t^{-}}=\textbf{h}_{c_{t},t^{-}}$. Recognizing that different questions have different mastery requirements of KCs, we incorporate question-KC correlation scores into the aggregation process in this GNN, which are calculated by:

where $\textbf{e}_{q_{i}}\in\mathbb{R}^{d_{e}\times 1}$ and $\textbf{k}_{c_{j}}$ are the embeddings of $q_{i}$ and $c_{j}$, and $\textbf{W}_{req}\in\mathbb{R}^{d_{e}\times d_{e}}$ is a learnable matrix. Then, the graph message aggregation process of Equation 8 is actually

We also remove the non-linear feed-forward process and restrict $\textbf{W}^{\mathcal{G},(l)}_{rtv}\in\mathbb{R}^{d_{k}\times d_{k}}_{\geq 0}$ to ensure higher values of the related KCs’ memory bring higher knowledge mastery. After getting the aggregating knowledge memory from this GNN, we get the knowledge mastery as Equation 7 and compare it with the question difficulty $d_{q_{t}}$ to generate the predictive probability of solving the question:

For multi-KC questions, we average the KCs’ memory. The difficulty $d_{q_{t}}$ of question $q_{t}$ is generated by a Multi-Layer Perception (MLP):

Here, $\bar{\textbf{e}}_{q_{t}}=[\textbf{k}_{c_{t}}\oplus\textbf{e}_{q_{t}}]$ is the concatenated representation of $q_{t}$ and its examined KC $c_{t}$’s embeddings. For multi-KC questions, we use the KCs’ average embedding. $\textbf{W}_{diff}^{(1)}\in\mathbb{R}^{2d_{e}\times d_{h}}$, $\textbf{W}_{diff}^{(2)}\in\mathbb{R}^{d_{h}\times 1}$, $\textbf{b}_{diff}^{(1)}\in\mathbb{R}^{1\times d_{h}}$, and $\textbf{b}_{diff}^{(2)}\in\mathbb{R}^{1\times 1}$ are learnable matrices and vectors. $d_{h}$ is the number of hidden dimensions. We denote the process of this two-layer MLP as $d_{q_{t}}=\text{MLP}_{diff}(\bar{\textbf{e}}_{q_{t}}|2d_{e},d_{h},1)$, and a similar notation is applied for brevity in subsequent descriptions. Hereinafter, we accurately model the process whereby students retrieve knowledge from memory to answer new questions.

The testing effect theory (Roediger III and Karpicke, 2006) reveals that a correct retrieval strengthens the storage of knowledge in memory, while an unsuccessful retrieval can lead to incorrect strengthening. Without correction or active learning after the error, this may reduce knowledge mastery (Kornell et al., 2009). In this stage, we determine the memory strengthening process based on whether the examined KC is correctly retrieved to solve the question, resulting in either knowledge gain or loss. Additionally, these knowledge changes are propagated to related KCs based on the transfer of learning theory. To enhance memory from a correct response to question $q_{t}$, we first combine and input the current memory $\textbf{h}_{c_{t},t^{-}}$ of KC $c_{t}$ and the question information $\bar{\textbf{e}}_{q_{t}}$ into an MLP to obtain an initial memory feature:

For multi-KC question, we calculate all the associated KCs’ features. This feature serves as a spark to propagate knowledge changes via another KC relation-based GNN. Specifically, by initializing an input feature matrix $\tilde{\textbf{G}}^{(0)}_{t}$, where $\tilde{\textbf{g}}^{(0)}_{c_{i},t}=\textbf{g}_{c_{t},t}$ if $c_{i}=c_{t}$ and $\tilde{\textbf{g}}^{(0)}_{c_{i},t}=\textbf{0}$ if $c_{i}\neq c_{t}$, the knowledge gain for all KCs is obtained as follows:

The $\text{ReLU}(\cdot)$ activation function ensures that the knowledge gain to be positive. Moreover, due to the zero feature initialization except for the examined KC, the knowledge gain is only propagated to KCs within $L$ hops, delineating a boundary between related and unrelated KCs. Similarly, we could derive the negative knowledge loss $\tilde{\textbf{L}}_{t}^{(L)}$ when students provide incorrect responses and wrongly strengthen their memory, by using a similar network $\text{GNN}_{loss}(\cdot)$.

Subsequently, we update the knowledge memory bank with respect to the response $a_{t}$ as follows:

It is worth noting that different questions also have different effects on strengthening students’ memory of KCs. Therefore, similar to Equation 10, these two GNNs also add the question-KC correlation scores during message passing. Henceforth, the second stage, memory strengthening, is reasonably modeled based on the testing effect and the transfer of learning.

After students answer questions, their subsequent actions vary depending on the feedback received. They may review their correct answers or correct their mistakes. Besides, they might prepare for the next question’s KC they would encounter. These active learning behaviors contribute to improving their knowledge mastery, which we model as the knowledge learning process in this stage. Concretely, the KC of the last question and the next question both influence the student’s learning target. Therefore, we use an MLP to determine if the student actively learns them based on his/her current knowledge memory and the involved questions’ information. For KC $c_{i}\in\{c_{t},c_{t+1}\}$ (or more involved KCs for multi-KC questions), the two-dimension policy distribution is calculated by:

Here, $\text{argmax }\pi_{c_{i},t}=0$ indicates that the first dimension is bigger. We suppose there is no active learning. Contrarily, $\text{argmax }\pi_{c_{i},t}=1$ indicates the student would learn $c_{i}$. Under this circumstance, we calculate the progress of learning $c_{i}$ in a similar way:

Based on the transfer of learning theory, this progress is also propagated to related KCs using another KC relation-based GNN. After initializing $\tilde{\textbf{P}}^{(0)}_{t}$ where $\tilde{\textbf{p}}^{(0)}_{c_{i},t}=\textbf{p}_{c_{i},t}$ for $c_{i}\in\{c_{t},c_{t+1}\}$ with $\text{argmax }\pi_{c_{i},t}=1$, and $\tilde{\textbf{p}}^{(0)}_{c_{i},t}=\textbf{0}$ otherwise, we compute

This active learning process continues until the student answers the next question, allowing us to model each KC’s progress $\tilde{\textbf{p}}_{c_{i},t}^{(L)},c_{i}\in\mathcal{C}$ with a KC-specific time-aware kernel function to update:

where $\Delta T_{t+1}=T_{t+1}-T_{t}$ is the time duration until the next question. According to the learning curve (Yelle, 1979), the efficiency of students in learning a specific KC tends to be high initially and gradually decreases over both the learning time and frequency. Therefore, we design the kernel function in an exponential form:

where $\odot$ is the Hadamard product. $n_{c_{i},t}$ is the number of times that $c_{i}$ has been learned by the student. $\tilde{\boldsymbol{\gamma}}_{c_{i}}^{(L)}$ represents the KC-specific kernel parameters of $c_{i}$ generated by another KC relation-based GNN. It leverages the property of graph homophily that makes related KCs have similar learning ratios:

with initializing $\tilde{\boldsymbol{\gamma}}_{c_{i}}^{(0)}=\textbf{k}_{c_{i}}$ which is $c_{i}$’s embedding. Here, $\text{softplus}(\cdot)$ is an activation function to restrict the parameter to be positive. On the other hand, for KCs that students have acquired before but they do not choose to learn, we introduce the knowledge forgetting process. Therefore, for the KCs students do not make progress on (i.e., $\tilde{\textbf{p}}^{(L)}_{c_{i},t}=\textbf{0}$), their previously acquired knowledge fades over time:

where $\Delta\textbf{h}_{c_{i},t}=\textbf{h}_{c_{i},t}-\textbf{h}_{c_{i},0}$ represents the total knowledge acquisition the student has accumulated. According to the forgetting curve (Ebbinghaus, 1885), the speed that students forget knowledge follows a pattern of initially rapid decay and then a gradual decrease over time and the review frequency. Therefore, we similarly design KC-specific forgetting kernel functions in an exponential form:

where the kernel parameters $\tilde{\boldsymbol{\theta}}_{c_{i}}^{(L)}$ are similarly generated by another KC relation-based GNN:

with initializing $\tilde{\boldsymbol{\theta}}_{c_{i}}^{(0)}=\textbf{k}_{c_{i}}$. Consequently, based on the learning and forgetting curves, we have derived the updated knowledge memory $\textbf{H}_{(t+1)^{-}}$ in this stage, which is recursively used for answering the next question.

The three-stage modeling is recurrent along the student response sequence. After learning/forgetting knowledge in the third stage, the updated knowledge memory is prepared for the first stage to answer the next question. This makes GRKT an end-to-end style so we directly train the model by the binary cross-entropy loss, aligning the predictive probability $\hat{a}^{u}_{t}$ from Equation 11 with the ground-truth response correctness label $a^{u}_{t}$:

Here, we omit the averaging notation for brevity. Besides, we also apply the $l_{2}$ normalization to the model parameters during the training process to avoid the over-fitting issue.