Quantifying the Corona Effect
How much the pandemic-induced switch from face-to-face to online teaching increased students' self-efficacy – a practical report.

The measures and the limitations due to Corona had a significant, but hard to quantify effect on teaching practices worldwide [18]. By and large, learning in classrooms has been replaced with online learning approaches. In computer science education the overall effect of this systemic shift is somewhat unclear. On the one hand, social learning practices, such as pair programming or discussions may have been replaced with more difficult to use online mechanisms. On the other hand, however, the pandemic has removed some of the resistance towards technology [5].

Scalable Game Design (SGD) is a K-12 computer science education strategy teaching Computational Thinking (CT) [22] by creating 2D and 3D games and simulations [21]. The strategy was used since 2017 in primary teacher education [17]. As an emergency measure Corona forced a change in the learning design. The face-to-face (f2f) teaching with many cooperative and collaborative interactions among students were replaced by online teaching practices. The SGD teaching practices has been developed and improved since 2017 using Design Based Implementation Research (DBIR). DBIR is an iterative process employing pre/post research instruments to assess the efficacy of the learning design [9] with respect to computational thinking learning outcomes. Over five years the learning design evolved to the point where it was able to provide evidence of efficacy quantifying cognitive as well as affective [19] learning gains as effect sizes [4, 13].

The Corona situation, forcing a significant shift in teaching practice, changed the learning design dramatically in a short period of time. This kind of change was not anticipated by the DBIR process. Instead of fine tuning the next iteration of a learning design based on previous efficacy measures this change was simply forced upon teaching practice for highly pragmatic concerns. Some of these changes included adjustments that, without the forcing function of the measures against the pandemic, we would not have contemplated. For instance, the shift to online learning resulted in a de-emphasis of social learning such as peer programming. We prepared to search for mitigation techniques to overcome negative impact onto the learning outcomes. However, it was a surprise that nearly in all aspects of learning evaluation, the results improved in distance learning.

This experience report attempts to quantify the Corona Effect by employing previous iterations of these courses as an efficacy baseline. The report first highlights the most salient features of the changes to the learning design, discusses related work, contrasts pre-Corona and during Corona conditions through effect sizes and discusses possible explanations that could have led to this result.

Design-Based Research (DBR) [1] and DBIR are relevant research methodologies because they were used to establish a baseline efficacy of a large scale computational thinking course. The idea of design-based research was first mentioned by Ann Brown in 1999 [3]. She was looking for a research approach that investigates learning phenomena not in the laboratory, but in real situations. “Design research is not defined by methodology. All sorts of methods may be employed. What defines design research is its purpose: sustained innovative development.” [2] DBIR grew out of the DBR approach to education. The difference between the two approaches is well described by Kevin Crowley [6] and by Fishman et al.[9]. In DBIR teachers develop and implement the design of an activity together with researchers. Based on the evaluated data, they create a Δ design at each iteration to enhance the desired learning outcomes.

Effect sizes [4] are suitable indicators to quantify the efficacy of educational interventions. Hattie for instance, in his seminal work on visible learning [12], employed effect sizes to rank the efficacy of a very large number of teaching practices. To help researchers with interpreting effect sizes, Hill developed a set of benchmarks establishing typical effect sizes derived from large scale standardized tests such as the improvement of reading comprehension of typical 1^st graders in school [13].

To understand the roots of a potential Corona-effect, it's important to understand shifts in social learning practices typical in face-to-face versus online education. Kozar distinguishes between cooperation and collaboration [16]. Collaborative Learning can be defined [8] as “a situation in which two or more people learn or attempt to learn something together.” Research on collaborative learning, for instance [11], includes both “collaborate to learn” and “learn to collaborate.” Schneider et al. summarize in their paper approaches to measure the effects of collaborative learning [24]. They point out that there is neither a generally accepted definition of collaborative nor cooperative learning but different approaches to analyze the issue. Johnson/Johnson compared in their meta-study eight methods of cooperative learning [14]. They conclude that cooperative learning contributes to a small positive learning effect across all 194 studies reviewed. But what if students don't program together in a way of collaborative learning but simply split the work? This collaborative practice of splitting the work would typically be considered a cooperative teamwork [17].

Studies conducted during Corona suggest that under some circumstances collaborative learning methods are less effective than learning individually. Garrote et al.[10] showed that during the lockdown in Switzerland, learning gains in language or mathematics were higher than in a classroom situation. They attribute this increase in efficacy to predominantly individual work caused by the lockdown.

Beyond cognitive shifts there are also attitudinal ones. Crick et al. [5] found an affective impact specific to computer science education. They found significantly more positive attitudes towards online learning in the UK educational workforce in CS compared to other disciplines.

Computer science education is mandatory for all primary pre-service teachers who are undergraduate students at the School of Education Northwestern Switzerland (PH FHNW). These pre-service teachers must take two courses in their first year. In the first semester they take a science seminar learning about computational thinking and programming. The second seminar focuses on didactics, but it is not considered in this report.The science seminar consists of three pillars:

• Motivation and Learning Strategy: Scalable Game Design. Motivation is at the center of this course. SGD describes the creation of games and simulations and their continuous adaptation to new ideas [22]. 
  
• Tools designed specifically to teach and support Computational Thinking (CT) at the elementary school level. Students create their projects using CT tools optimized for use with children and youth. They do not need to learn the syntax of a programming language and are supported in the process of abstraction, code creation and debugging [21]. In addition, they work with tools they can use directly in their future teaching.
  
• The 7 big ideas of Computer Science from Computer Science Principles. The mapping of the SGD strategy onto the computer science part of the Swiss curriculum called Lehrplan 21 [7] was straightforward. The three main Lehrplan 21 CS topics (data, algorithms, and systems) were identified as a subset of the 7 big ideas found in the Advanced Placement (AP) Computer Science Principles [20] framework: creativity, abstraction, data, algorithms, programming, networks and global impact.
  

Table 1 summarizes the shift in learning design from 2019 (before Corona) to 2020 (during Corona). The six sections below describe the shift in more detail.

• Programming activities before Corona. Students worked in small teams of 2 or 3 people to design, and program games. Students largely decided on their own how to organize the collaborative activities. Many of these games produced collaboratively were rooted in existing 2D 1980 arcade game play ideas such as Pac-Man, or Frogger. More sophisticated kinds of games were Tetris (including non-trivial geometric transformations of compound shapes). Other projects were 3D adventure games where a player needs to roam through 3D world to find clues to make it to a final goal. During the f2f lecture students engaged in pair programming [17]. However, in many cases, during lecture or when working at home, the collaborative learning often devolved into cooperative work [16] where two students would simply split up design and programming tasks. Worse, often contributions of the team members were not balanced.
  
• Programming activities during Corona: In online learning, all students had to develop all projects themselves. This would lead to a much larger number of projects. Various tutorials and a weekly online office hour meeting were available for support.
  
• Peer review before Corona. Before Corona peer-review was used informally. While programming in the classroom, teachers periodically asked students to share and exchange their projects in the spirit of Resnick's creativity spiral [23].
  
• Peer review during Corona. In online learning, peer review had to be formalized. Students had to test and evaluate three projects of fellow students. The assessment was criteria-based using a questionnaire. Through this measure, each student saw not just one - but at least three other projects.
  
• Grading before Corona consisted in the 2019 course a written exam and a collaborative or cooperative project. In the written test, students were asked about the theory, and they had to solve small programming tasks. Beside the evaluation of their competencies one important function of this exam was to motivate (extrinsic) students to engage with the content. After the written test, the students received an assignment to develop a computer game. They could work on this in teams of two or three. The game was presented to and tested by the students in the final session. Half of the grade was based on the exam and half on the project.
  
• Grading during Corona: In online learning, the written exam was cancelled for practical reasons. On the one hand this was because of the massively increased volume of projects which would at least double the project grading time. On the other hand, one purpose of the exam was to create individual grades and to catch students who did not sufficiently contribute to projects. With individual projects this was no longer necessary. The previous collaborative project presentation of the project (a computer game) at the end of the course was replaced by a code walk video. The code walk is a 5-minute video in which every student had to present his/her game and explain the written code.
  

The DBIR was conducted at the School of Education Northwestern Switzerland (PH FHNW). To compare the two years, both the results of the survey (table 3) and the quality of the final projects were considered.

In both years, nearly 400 students started this mandatory course in 16 different groups of approximately 25 students. The course was led by 4 different instructors and counts as 2 Credits (60 working hours). The students' mean age is 22. 75% of them are female.

At the conclusion of the course, students were asked to complete a survey. The participation in the surveys was anonymous and voluntary. The course was taken by approximately 400 students (pre-service teachers) mostly in their first semester of their bachelor programs. The 2020 response rate described in table 2 was slightly lower than in 2019, but still above 50%.

The survey asked about self-efficacy in CS knowledge/skills questions. It used a four-point Likert scale (4 = strongly agree / 1 = strongly disagree). Only 4 scale levels were defined to avoid the tendency toward the middle [15].

The questionnaire of the online seminar was the same as the year before. Only two questions about the special situation were added. We asked them whether the computer or computer science had become more relevant for them or for schools in general. The questionnaires also included a qualitative course evaluation part, in which students could write an open-ended text response. Second, the instructors grading all projects were asked about the quality of the final projects. This was done informally in an open discussion after the final projects were evaluated. The analysis of these data is not part of this article but will be used to design the next iteration of the seminar. In the discussion section, a few quotes are used to support the conclusion.

As mentioned in section 3 (Δ Design), the instructors were well prepared for teaching online. Therefore, it was hoped that the quality of teaching would only slightly decrease. But the self-efficacy scores increased. The magnitude in shift is captured by two measures. The first one is shift in self-efficacy described in Table 3. The second one is an increase in the project quality described in section 5.2.

5.2 Quality of Artefacts

In addition to the questionnaire serving as a self-efficacy instrument we also looked at the quality of the final projects. Figure 1 and Figure 2 below show two exemplary games produced. The first one is a 2D adventure style game with 22 agent classes, 218 IF/THEN branches, and 51 methods. The second one is a 3D first person adventure game with 36 agent classes, 64 IF/THEN branches and 40 methods.

Screenshot of a 2D Adventure with Hero, Elf, Innkeeper and Knights.

Screenshot of a 3D Agentcube Dungeoncrawler Game.

Instructors assessed the quality of games using the same criteria grid. All instructors (who had to grade a total of 331 projects) unanimously shared the impression that the level of sophistication of the projects was higher and the game design was better in the online seminar. In both years, students had the same amount of time to create the projects. But they had to work alone and at the same time, by eliminating the written exam, the grade relevancy of the project increased.

Using the pre-Corona efficacy as a baseline the Corona effect can be quantified. However, while the data indicate an overall positive shift, we cannot attribute this shift to specific changes in the learning design. In this discussion section we speculate about the most salient aspects that could have caused the shift.

The discussion focuses on the elements that were changed in the online seminar: programming activities, peer review, and grading. As mentioned, it is likely that with the increased relevance of computers and computer science during the pandemic, students were generally more motivated and engaged. One could also explain the improvement by the fact that creative work in computer science is more suited to online learning than in other disciplines such as music, design, or sports. There were various feedbacks from students indicating this: 

“Huge compliments to the organization! Compared to other courses, this one was very well designed and structured. The provision of weekly office hours was also very helpful. The workload was sometimes extremely high :) but it was fun.”

“I found the course super structured, and it is one of the most motivating subjects, because you can do a lot by yourself. I find the certificate of achievement great but also time consuming and would have liked more time to be able to implement everything I wanted.”

The impact of Corona onto courses teaching computer science to pre-service elementary school teachers was quantified through effect sizes. Replacing a collaborative f2f course, providing no scaffolding for successful collaborative practices, with a non-collaborative online course had an overall small but positive effect onto self-efficacy and improved the quality of projects. Without proper scaffolding project-based group work is likely to devolve into cooperative instead of collaborative learning practice. Cooperative learning, the practice where students split a project up into separate chunks, can be inferior to non-collaborative learning in terms of self-efficacy and product quality.