Predicting Long-Term Student Outcomes from Short-Term EdTech Log Data

Can’t Wait to Learn (CWTL) is an educational program developed by the nonprofit organization War Child to support children impacted by or residing in conflict affected areas who may have challenges accessing quality education. The educational technology is a curriculum aligned, self-paced, autonomous learning program that aims to teach foundational numeracy and literacy skills, and prior and ongoing work suggests that it can have a positive impact on student learning outcomes (e.g.  [8]). The program is delivered on a tablet and targets learning objectives from grades 1-3. Based on the context, the program can be used as a standalone or a supplemental educational program. CWTL has been used in many settings, including South Sudan, Sudan, Lebanon, Jordan, Chad, Bangladesh, Ukraine and Uganda.

In this paper we used CWTL data collected across 30 schools from September 2021 to December 2022. We have access to log data from 739 students learning literacy (referred as CWTLReading in this paper), who participated in the study with both pre- and post-test scores. The post-test scores were calculated by averaging of sub-task scores, including letter knowledge, phonemic awareness, reading fluency, and reading comprehension in an assessment.

MATHia (previously Cognitive Tutor [29]) is an intelligent tutoring system for middle school mathematics. The anonymized MATHia data used in this study was collected from 2644 students using the software from August 2020 to June 2021 in a mid-western US state. The predicted post-test score is assessed by a state end-of-year test. The test has both an integer score, and 5 achievement categories: levels 3 to 5 correspond to a passing score.

iReady is an online program for K-8 reading and/or mathematics that provides an adaptive diagnostic, and online instruction. The iReady data used in this study contains 428 Grade 7 students in mathematics courses with complete records including both pre- and post-test scores from August 2022 to June 2023. The predicted post-test score is assessed by the Smarter Balanced Assessment System (SBAC) which is a standardized test consortium and creates Common Core State Standards-aligned tests to be used in multiple states of the US. The students were part of another research study with a particular focus on schools with more struggling students.

In this section, we will outline our approach for estimating delayed external assessments using short-term log data as students work on an educational technology product. We will overview the features extracted from log data, before describing the machine learning algorithms used, and the evaluations of our proposed methods

We first note an important detail, which is how to define usage time. Though it may seem trivial, defining usage time often requires several decisions, including whether to define things as available usage time (the time in a classroom a student had the opportunity to use educational software), the wall clock time a student was logged in to an educational software tool, as well as the active time a student spent doing problems. The first definition (potential usage time) is often very hard to know, since it requires information around attendance. The second one (wall clock or logged in time) is easier to extract for most systems that include a session variable, though not all software systems do – in particular, our CWTL data did not include a session variable. Even in other systems this can be a challenge as it is common for some software products to automatically log out a student given a sufficiently long inactive time period. For example, if a student logs in at 9:10am, and then is logged out at 9:25am, and then logs in again at 9:40am, and logs out for the day at 10:00am, one could count this as a 50 minutes of usage, or 35 minutes. The 50-minute view might reveal important aspects of disengagement that would be missed if it was treated as 35 minutes. For the MATHia and iReady data, for which we do have session identifiers, we consider wall clock time spent within each session. In contrast, for CWTL, which lacks session identifiers in our available data, we instead accumulate active time on each played minigame. This is possible because the start and end time on each minigame is recorded. Note that in general this may underestimate the total amount of time spent by a student, since it does not include watching videos or time between completing activities. In our discussion we will further reflect on the potential impact of our particular choices around usage time definitions.

To understand the potential signal from using student log data given various lengths of usage since the start of data collection, we train and test each machine learning prediction model with cumulative log data on varied lengths of horizons, i.e., 1, 2, 3, 4, 5, 12 hours, as well as the full length of data (denoted as H). The prediction models’ RMSE and R² are presented in Figure 1. The x-axis represents the length of horizon for used cumulative log data to train models and y-axis represents the prediction results averaged from 5-fold cross validation.

Interestingly, we observe that for all three educational products and datasets, there exists a machine learning model built using short-horizon of log data that is nearly as or equally effective as a the best machine learning model built using the full horizon log data.

The second thing to note is that the short horizon time period with the best performance (as measured by RMSE and R² varies slightly by datasets/setting. For example, for CWTLReading, the random forest machine learning model has notably better performance with 5 hours of interactive usage (0.13 RMSE and 0.34 R²) but worse performance using log data from 12 hours and full log data. In MATHia, two hours of log data is the best of the early horizon (up to 5 hours) exceeds the performance of the full horizon log data, and for iReady strong early performance (measured by R² and RMSE) among the first 5 hours is obtained at 3 hours, though the performance is quite similar over teh full range.

We also note that performance not always monotonically improve with longer horizon log data– machine learning predictors for both CWTLReading and MATHia seem to slightly decrease in performance at the longest horizon lengths. It is known from the economics surrogate literature that using a surrogate measure, when surrogacy is satisfied, can result in a lower variance estimate [5], since additional data beyond the surrogate window can introduce additional noise. Though we would not expect to surrogacy to necessarily always hold in real educational data, this may be a reason for why we see, sometimes, higher accuracy estimates using a shorter horizon.

We next assess if the machine learning algorithm used has a significant impact on resulting performance, which we visualize in Figure 1. In general, all machine learning algorithms are quite similar. For CWTLReading, random forest (RF) achieves the lowest RMSE across different horizons, and achieves the highest overall R². Support vector regression (SVR) were slightly less consistent, and linear regression (LR) performs relatively more stable across lengths of horizon and performs better than random forest at some early horizons in terms of R². For MATHia, RMSE for all three models (LR, SVR, RF) is similar, and the difference of R² between the three methods is minimal on given short horizon log data. For iReady, random forest consistently achieves the lowest RMSE and highest R², followed by support vector regression, and then linear regression.

All differences are relatively minor, though random forest overall seems to have a slight noticeable benefit in the CWTLReading and iReady datasets. The baseline of simply predicting the average score in the training data performs poorly across all horizons across datasets.

We also note the predictive models have lower accuracy in CWTLReading than the other two datasets. A potential reason is that the records across students of CWTLReading are relatively more sparse than the other two datasets. On average, the numbers of logged events over the best short and full data are 270 (5hr) and 1245 for CWTLReading, 586 (2hr) and 8738 for MATHia, 1162 (3hr) and 5042 for iReady. This likely merely reflects the differences across different educational technology designs and instrumentation, and it may have an impact on the density of data available for prediction.

Using interpretable features from log data for our machine learning predictors has the potential to capture interactions that reflect underlying learning processes of students, and allow us to understand how such processes may be similar or different across platforms and temporal periods. Towards such insights, we first identify features that are important in our machine learning model, and then also consider if we can use such features to formulate a much simpler model which may replace our more complex machine learning models.

Important features shared across educational contexts. To quantify important features, we consider top 5 features selected by a random forest with 100 decision trees (with maximum depth as 10) trained on each dataset, both using a selected short horizon and full data. The results are shown Table 1. Overall, there is a significant overlap across contexts, and across both horizon lengths shown within each context. Specifically, across all three datasets, the percentage of times the student succeeded at a problem, and the average number of times a student attempted a problem, are frequently selected as important features, in some cases being the most important feature for a dataset. We also note that within a dataset, the important features often significantly overlap between the short and long term important features. For example, the average number of attempts per problem and counts of when a student took a long time to complete a problem, are top features in CWTLReading using both short (i.e., 5 hours) and full horizon. Two features, the percentage of times the student succeeded at a problem and the average time to finish a successful problem, are selected in the top 5 features on MATHia using both short (i.e., 2 hours) and full horizon. Three features, including the average number of times a student attempted a problem, the average time to finish a successful problem, and how long a student persisted unsuccessfully to finish a problem, are selected in the top 5 features on iReady using both short (i.e., 3 hours) and full horizon. The features are generally consistent across datasets and horizon settings, suggests that one might build a general predictive model across educational contexts using the same feature set.

Prediction effects using single feature vs. the entire set of features. In part motivated by the prior observation in the last paragraph, we investigated the effectiveness of using a single feature compared to using a set of features extracted from log data. We train linear regression using the percent success per problem, and the average attempts per problem, respectively, using short horizons as noted in prior sections (i.e., 5 hours for CWTLReading, 2 hours for MATHia, 3 hours for iReady). Table 2 shows the comparison results in terms of RMSE and R². Our results show that while a single feature machine learning model still performs much better than the baseline predicting the average performance in a held out set, our machine learning models using the full set of available log features outperform it, in terms of both RMSE and R². This suggests more complex machine learning models can offer a useful advantage.

As mentioned, though population level prediction accuracy is the most common reported measure, another important motivation of predicting outcomes using short-horizon data is to help understand and inform additional support and challenge for students in need. We divide students into performance subgroups, by two strategies: 1) using quintile division sorted by their test outcomes (represented by Q1 to Q5 in ascending order); 2) using context-specific binary clusters such as pass and fail (that we refer as ‘on track’ and ‘not on track’, respectively), which are defined by the given assessment.

Quintile subgroups sorted by performance. Figure 2 shows heat maps of confusion matrices that represent the prediction performance of the three machine learning methods (linear regression, support vector regression and random forest) on three datasets using short horizon. The x-axis shows the true performance subgroups in training set (sorted by post-test scores in ascending order), and the y-axis shows the predicted performance groups. Overall, linear regression (LR) performs better in general at having a stronger diagonal on the confusion matrices. Across all three methods, the accuracy is worst for students predicted in the middle (i.e., Q3 are frequently over- and under-estimated).

The highest precision is for students at either extreme. For example, for CWTLReading, when predicting students in the lowest performance quintile (Q1), the best model (random forest) is accurate 77% of the time when it predicts someone is going to be Q1, and is accurate 72% of the time if it predicts someone is going to be Q5. Random forest is also the best at Q1 and Q5 for iReady, with an accuracy of 57% at predicting Q1, and 88% at predicting Q5. For MATHia support vector machines has the slight edge over the other models at predictive accuracy for its extreme predictions, reaching an accuracy for predicted Q1 of 70%, and for Q5 is around 61%. We note in all cases the other models do worse, but similarly. The models are most inaccurate at predicting quintiles 2 to 5. Note though that the ranges of these quintiles are not evenly spaced (CWTLReading: [0.11, 0.42), [0.42, 0.53), [0.53, 0.62), [0.62, 0.74), [0.74, 1]; MATHia: [0, 0.37), [0.37, 0.48), [0.48, 0.55), [0.55, 0.63), [0.63, 1]; iReady: [0, 0.28), [0.28, 0.42), [0.42, 0.52), [0.52, 0.64), [0.64, 1]).

While we generally see correlation with the true quantile, the predicted models often overestimate poor performance and underestimate high performance, suggesting it is not yet well-prepared for individual or subgroup level performance predictions. However, the models’ relatively high precision at the extremes suggest they might be able to highlight when students might benefit from additional challenge or support, though further investigation and validation is key before employing such predictions for any important interventions.

Predicting subgroups on or off track. In MATHia, the end-of-year state test also has 5 achievement categories where level 3 to 5 correspond to a passing score. Thus, we consider cluster students into two subgroups, ‘on track’ subgroup whose achievement categories are in level 3 to 5, and ‘not on track’ subgroup whose achievement categories are in level 1 to 2. The prediction process is conducted over the original achievement categories and then we calculate a confusion matrix based on the two clusters. Using the short horizon (2hr) log data, we find that the model correctly classifies as "on track" 1541 students, correctly classifies as "not on track" 423 students, and misclassifies 680 students. In particular, the model has low recall for the ‘not on track’ subgroup (0.44, true predicted ‘not on track’ divided by actual ‘not on track’), indicating it misses a large number of students who are not on track. On the plus side, similar to our results for the quintile analysis, the model has reasonable precision: if it predicts a student is not on track, 74% of those instances did have a post assessment that indicates they are not on track.

Prediction over population using pre-assessment scores combining with a set of log data features. Tables 3 &  4 show: 1) using pre-test scores only; 2) combining pre-test with log data features on short horizon; and 3) combining pre-test with full log data features. In general, pre-assessment or pre-test score is a powerful indicator for long-term outcomes, that can be stand-alone and outperforms using log data features only. With pre-test scores, all models show similar performance with RMSE around 0.14 for CWTLReading and around 0.12 for iReady. And RMSE is not significantly improved compared to using log data features in CWTLReading. But R² values are higher when pre-test score is included, peaking at 0.44 for LR and SVR in the Short+Pre-Test (CWTLReading) and 0.66 for SVR in Short+Pre-Test (iReady), which indicates a better fit model when pre-test scores are integrated. Overall, unsurprizingly, it is beneficial to include a pre-test score in the model if available.

We further investigate the effects of combining single feature with pre-test scores. Table 5 shows the prediction results on CWTLReading and iReady using LR. Pre-test can substantially enhance LR’s performance with single feature. And even when two features have differed effects on model performance (e.g., 0.197 RMSE using PS only vs. 0.179 RMSE using AA only), combining with pre-test could fill the gap by improving prediction performance to the similar level (e.g., 0.123 RMSE using PS+Pre-Test vs. 0.122 RMSE using AA+Pre-Test). Those indicate the effectiveness of pre-test scores in prediction.

Prediction over subgroups using pre-assessment scores combining with a set of log data features. We calculate confusion matrices on quintile division of subgroups using both with and without pre-test scores on selected short horizon. Figure 3 shows the heat maps of the confusion matrices for CWTLReading and iReady. For lower performers Q1 and Q2 in CWTLReading, using log data features with pretest (Short+Pre-Test) shows more concentrated on overestimating cases (with less on predicting Q1 as Q3-Q5). And for highest performers Q5, Short+Pre-Test shows more concentrated on underestimating cases (with less on predicting Q5 as Q1-Q3). For lowest performers Q1 in iReady, similar to what we observe from CWTLReading, introducing pre-test to log data features helps LR less overestimate. And for highest performers Q5 in iReady, Pre-Test also help LR less underestimate the performers as Q1-Q3.

To summarize, if pre-test score is available, one can use that alone or combine it with a single feature. If we don’t have pre-test score and only have log data, there is still a gain to using more than one feature extracted from log data.