Are Working Habits Different Between Well-Performing and at-Risk Students in Online Project-Based Courses?

We analyze differences in working habits between well-performing and at-risk students using highly-granular data collected from two semesters of an online project-based, upper-level course on cloud computing at a US institution of higher education. Such differentiating metrics may provide deeper insights than interim grades, which are oftentimes the only quantifiable data that is captured and available to an instructor as a proxy for students' learning. Interim grades provide little insight into students' broader work habits and may mask unsustainable learning strategies that result in shallow learning or quickly-forgotten skills/knowledge. The adoption of technology-enhanced learning tools for course delivery, automatic feedback, and grading enable data-informed insight and reflection into students' working habits. This data could allow the detection of early signs of under-prepared students or students in crisis. We empirically assess what working habits, if any, differ among well-performing and at-risk students. From clickstream and other activity data, we derive 22 metrics such as time spent reading project write-ups, timing of starting and finishing work, or break-taking. We also calculate two measures of consistency of each metric measured by a coefficient of variance and a variance of ranking over the semester as well as outlier behavior of a student. Using Z-test and Kolmogorov-Smirnov test, we confirm differences in multiple behavior patterns. Notably, our data suggest that well-performing students start and finish working on a project earlier than at-risk students but they also tend to have fewer submissions which indicate they are more thoughtful about feedback.