An Analysis Scheme to Interpret Students' Cognitive Process in Error Finding Test

Finding errors in the source code is a typical kinds of test in the programming courses, which is helpful to assess the students' ability in program comprehension and debugging. In this paper, we propose an analysis scheme to interpret the measured line-level eye movement in an error-finding test of C language. An eye-tracking measurement system is implemented to collect the student's eye movement; during the test, the students select the error lines by mouse click in the web pages. We divide the students' eye movement into multiple cognitive tasks according to their decision actions, i.e. the mouse click events. For each cognitive task, we formulate the line-level program execution sequence around the code line selected by the students. A new metric, namely Matched Movement Ratio (MMR), is proposed to describe the proportion of line-level eye-focus jumping events that matches with the program execution sequence. We validate the calculation of MMR by two case studies, and further utilize this metric to conduct group study for 19 students in a test about bubble-sorting algorithm. Experiment results show that, MMR can differentiate the students by their code tracing performance in error-finding missions. Although our scheme can not infer the whole cognitive process in test, MMR can work as a reference quantifying the students' cognitive efforts tracing the code lines. Our results provide evidences for the teacher to diagnose the weakness of students and provide improvement suggestions in programming courses.