Combining Local Testing with Automatic Commits: Benefits for Progress Tracking and CS2 Students' Learning Experience

Many instructors in introductory programming courses experience high dropout and failure rates. Identifying struggling students early is a prerequisite to target this problem. To this end, instructors and learning analytics researchers may leverage version control by analyzing the students' commit histories. This approach relies on frequent pushes to the version control platform, which many instructors incentivize by offering test results each time a student pushes a commit. However, instructors who provide test cases that can be run locally (i.e., without creating a commit) may face coarse-grained commit histories.

In this study, we analyze a CS2 course which offers both local and remote testing. Students were provided with tools that automatically create and push a commit on each local test run. We investigate to what extent these automatically created snapshots contribute to obtaining fine-grained commit histories and early initial push events. Our analysis uncovers distinct commit patterns among high- and low-performing students. Furthermore, we find that despite the commit automation and encouraging students to start early, many students pushed their first commit late. We triangulate this observation with survey results which confirm the late start of many students. The survey also identified reasons for students to opt out of automatic commit creation. Moreover, many students expressed a positive attitude towards testing their programs locally. Thus, our survey results underline that instructors should strive for providing students with comprehensive feedback that students can conveniently obtain.