Affective Walkthroughs and Heuristics: Evaluating Minecraft Hour of Code

Abstract

This paper presents an evaluation of Code.org’s Minecraft Hour of Code that was created to encourage and support people initial learning of computer programming. In particular, this web-based software uses a spatial model world, where the learner’s programs manipulate the world. We applied the Affective Walkthrough and the Affective Heuristic Evaluation, proposed evaluation methods for affective learning in multimodal educational software. Our findings provided illumination about the Minecraft Hour of Code approach, highlighting some aspects that are successful, and others where improvement appears necessary. We also gained insight about the evaluation methods and their effectiveness.

1 Introduction

This paper presents an evaluation of Code.org’s Minecraft Hour of Code. The software was created to encourage and support people initial learning of computer programming. In particular, this web-based software uses a spatial model world, where the learner’s programs manipulates the world. We applied the Affective Walkthrough and the Affective Heuristic Evaluation proposed earlier [4] and refined with the previous case studies [5, 6]. Our previous studies were conducted with participants to refine our methods, and inform us about the collaborative processes. In this new study, we simply applied this knowledge and conducted the evaluations ourselves. Our findings provided illumination about the Minecraft Hour of Code approach, highlighting some aspects that are successful, and others where improvement appears necessary. We also gained insight about the evaluation methods and their effectiveness.

Code.org is a non-profit organization that encourages learning of computer science, especially programming and computation thinking. It’s “Hour of Code” is an initiative that emphasizes short introductory tutorials aimed at total beginners, and has been widely used, claiming almost 100 million learners, including US President Barack Obama [14].

The Hour of Code approach is strongly visual in nature, both for the program itself, and for what the program does. This is reasonable, as it is established that such visual environments might assist in novice engagement, and also support them demonstrate skills and strategies in a familiar or easily understood context [1].

Fig. 1.

The initial interface of the Minecraft Hour of Code tutorials.

One aspect of Hour of Code’s visual nature relates to the program itself, and it uses a variant of the “jigsaw” or “block” approach used in Scratch [11], Alice [2] and other learning systems. The other aspect is the spatial model world, where such worlds include graphical elements that serve as integral parts of a computational context, problem, and solution. The Hour of Code tutorials include many variants of this, for example including ones resembling many popular games, e.g. “Angry Birds”, “Plants vs. Zombies”. In this paper we focus on the “Minecraft” variant — see Figs. 1 and 2. This is of special interest because Minecraft itself involves programming as a core part of the appeal of the “game”.

Our interest is in affective learning in multimodal educational software, focusing on the emotional elements in the software. This reflects calls for more attention to emotion in the domain of Human-Computer Interaction (HCI) and education [15]. In this paper we present our application of two such methods, the affective walkthrough and the affective heuristic evaluation proposed earlier to the Hour of Code Minecraft software.

Fig. 2.

The Hour of Code Minecraft interface, with model world on the left, and the program on the right.

2 Overview of Hour of Code Minecraft

The Hour of Code Minecraft tutorial was developed to engage students in learning programming. It introduces players to basic programming concepts, allowing them to navigate, mine, craft, manipulate and explore in a 2D Minecraft world by plugging together blocks to complete all actions and generate computer code.

The system provides a sequence of tutorials, illustrated by a horizontal line with nodes representing each tutorial step, as shown in Fig. 1.

For each tutorial, the learner uses an interface such as shown in Fig. 2. On the left top is the model world (“play space”), resembling the rectilinear blocks as used in the full Minecraft game, representing objects such as grass, trees, sheep, etc. On the left bottom is a panel with instructions to the learner. When a learner begins the tutorials, they select an avatar, either “Alex” or “Steve”. The chosen avatar then appears in the model world, and the program controls their actions within the world.

On the right is an area consisting of a “toolbox” of jigsaw-like pieces, and a “workspace”. The pieces in the toolbox are for simple statements or commands, as well as loop (“repeat”) and conditional (“if”) statements, and the shape of the pieces shows how they connect with other pieces. These can be copied and dragged into the workspace, and assembled in sequence to form programs.

When assembled, the jigsaw pieces do resemble traditional (textual) code, and in most Hour of Code this is the only code shown. In the Minecraft tutorial, however, whenever a tutorial step is completed satisfactorily, a pop-up window shows the equivalent code in JavaScript, which is what is actually used to program Minecraft.

In Hour of Code Minecraft, the tutorial steps reflect teaching and learning goals, in sequence. The sequence is fairly traditional, beginning with sequences of commands, moving on to loops, and then conditional structures. At first, the introduction is strongly guided, and then more freedom allowed to encourage a program solving approach.

In the interface and the tutorial steps, several modalities are involved. Firstly, the interface is graphical and spatial, with the model world and the jigsaw piece program. The character of the graphics is appealing, with a coarse lo-fi and fun game-like appearance that might help engagement. Secondly, the model world suggests a “world of action”, like a board game, where the blocks might be moved and interact. Thirdly, and related, is that the tutorial instructions build narrative elements where the player’s avatar should accomplish goals by moving and acting in the model world. Over the sequence of the steps, these build into an overall story. This approach would appear to help learner motivation, understanding, and possibly supporting reflection and creativity about what other possibilities might be programmed. Finally, there are audio elements with calm background music and occasional effects to emphasize events.

3 Minecraft Hour of Code and Affective Walkthrough

The affective walkthrough proposed earlier [4] was based on Wharton et al.’s cognitive walkthrough [13], and Dormann and Biddle’s affective walkthrough [3]. Following case studies, it was refined [6] to de-emphasize normal usability issues, and to acknowledge the contextual role of teaching goals and potential modality benefits. It follows Kort et al.’s affective model [7]. This identifies four phases of learning and the affective character of each. The first phase is encouraging exploration with positive affect. The second phase introduces challenges, and negative affect is expected. The third phase is to support overcoming challenges and reduce the negative affect, and the fourth phase is to affirm learning and restore positive affect.

3.1 Method

As with the earlier cognitive walkthrough by Wharton et al., the main idea is to select user tasks and then step through the software considering key questions, and making notes and observations, as shown in Table 1. The walkthrough was conducted by us, role-playing evaluators. We used a large screen display to collaborate in the same manner as done by participants described in [6].

Table 1. The affective walkthrough (version 3).

The objective of our study was to evaluate affective learning in Hour of Code Minecraft tutorials, with special attention to the supporting modalities involved. We presented, above, the basic design of the tutorials and the interfaces, and identified the modalities involved and their potential benefits. We then outlined the basis of the affective walkthrough, and the steps involved.

To select the tasks, we simply used the Hour of Code Minecraft tutorial steps, although we grouped the 14 steps into 4 larger tasks, bringing together steps that had strongly related educational purposes.

We then followed the walkthough as shown in Table 1, and considered each question in turn. We made Walkthrough Evaluation Sheets to record our answers and related observations. In the section below, we review the nature of the taskes, and the results of our walkthroughs.

3.2 Findings

First Task. To begin, we select a character. We have two choices, e.g. we can pick either Alex or Steve. The first task is about “commands”. The steps are shown in Table 2 and the results of the walkthrough are shown is Table 3.

Table 2. First task: walk around.

By completing the first part of Task 1, the software gave emotional feedback such as happy sounds, a jumping character, green highlight on the achievement bar and a pop-up window providing feedback. Figure 3 shows the pop-up window with the achievement bar and our character reaction at the bottom. If we were not able to complete the step with the minimum lines of code, the software will let us know with a challenging part, but it does not help us to complete the step with the minimum lines of code (see Fig. 4). But if it was really going to encourage exploration, it should allow the user to see all of the options he/she could select and do, but it does not.

Table 3. First task walkthrough results: walk around.

Fig. 3.

Task 1 - Step one.

Table 4. Second task walkthrough results: build a house.

Fig. 4.

Task completion but not with minimum lines of code.

For every task, the software only provides us with the number of blocks we need. Therefore, they are not encouraging exploration all that much. They are not providing us with other alternatives. That is a bit of a criticism if we are encouraging people to explore. If we want to encourage exploration, we do not constrain people. We give them a bunch of different things, so that they explore to see what happens. We are surprised they did not give all the blocks that are relevant to let us explore a bit (e.g. shear sheep and cut down tree). We were a bit confused that we did not know all the possible things or at least some small subset of the possible things.

In the second part of the first task, the software took awaycut down, and added shear.

It provides the user with more information about the reason of doing the task, e.g.“wood is a very important resource. Many things are made from it.” The third part of the first task uses “shear” command instead of “destroy block”.

As we are not using repeat loops in this task, the last part of task one needed lots of “moveForward()” blocks. It would be easier if we could just tell the computer to perform the move forward command the number of times we needed. It would be much easier to transition to repeat loops, if we had told the computer to “move forward” a specific number of times. We will need thousands and thousands of blocks to create a new world in Minecraft.

Second Task. The second task is about building the rest of a house from the material available. The “repeat” command will come in handy. For the last part of the task we can select “Easy”, “Medium” or “Hard”. The steps are shown in Table 5 and the results of the walkthrough are shown is Table 4.

Table 5. Second task: build a house.

The second and third tasks are about how to come up with algorithms, giving a problem and solve it using repeat loops. For the first part of task two, the software provides the reason and time to complete this part by saying: “We need to build a house before the sun goes down. Houses require a lot of wood.” For the second part of this task it suggests: “Every house starts with a wall.”

This part of the software is not telling the user how much wood he/she gets, and why we need to chop down all the trees. We do not have variables. We do not have any idea how much wood we have. Also we can not really build a house; we can just build some walls, so the story is not correct.

Table 6. Third task: plan some crops.

Table 7. Third task walkthrough results: plan some crops.

Third Task. This task is again about using “Repeat loops” and “commands”, but in a more challenging way using some creativity and freedom. It has four steps and each step takes you to different interactive spaces including: 1. Plant crops on both sides of water, 2. Move past the Creepers and reach safety at home, 3. Move underground placing torches and mine coal, 4. Avoid walking into molten and place cobblestone to create a bridge, and mine iron blocks. The steps are shown in Table 6 and the results of the walkthrough are shown is Table 7. This task has gamification steps to bring more engagement to the user.

Forth Task. Finally, the last task is about “commands”, “repeat loops”, and learning about “if” statements; a fundamental part of learning to program. It introduces us to a concept, which requires more problem solving. “if” statements help all computers make decisions. We are able to use “if” statement in the code to make our character react to what they see in the world, e.g. if there is rock in front of her, she can turn left or right and watch her steps. It has two steps; first step is simple and second step is more challenging, which are including: 1. Lava is hiding beneath some of these blocks, which you’ll need to cover up before moving forward, 2. Mine redstone but don’t fall in the lava by placing cobblestone over any lava you uncover. The steps are shown in Table 8 and the results of the walkthrough are shown is Table 9. This task has gamification steps to bring more engagement to the user.

Table 8. Fourth task: “if” statement.

Table 9. Fourth task walkthrough results: “if” statement.

3.3 Discussion

Reviewing the results of our walkthroughs, we are able to make several general observations. Most simply, we were pleased with several strengths: good visual effects and artwork, and sensible audio. Beyond those, however, we found more insight, described in sections below.

World and Story. Perhaps our main positive finding was the interplay between the model world of the “playspace” and the narrative aspects of the programming tasks given in the instructions. We felt it was clear and motivating when the learn was asked to create programs to move the avatar, chop down the tree, build a wall, and so on. The world and the story seemed to go together well, almost the way that narratives often work in computer games. This worked very well here, and showed some limitations in tutorials in Scratch, for example, which typically start with a blank canvas, rather than a world ready for a story.

While acclaiming this aspect, we found two weakness. One is that there was often little motivation for the actions requested in the instructions. The learner is asked to chop down trees, for example, before any mention of building walls for a house. Second is that there was no overall story, no eventual goal to accomplish, despite this being so common in computer games. Even in Mario, the player knows they are not done until the princess is saved.

Challenge but Not Overcome. We very much appreciate the way in which the tutorials helped the learner get started, offering a clear narrative goal, exactly the right tools to accomplish it, and providing affirming feedback when done. In several places, however, it seemed that this limited exploration: it encouraged simple constrained programming, but did not add the context of a large number of possibilities in the “toolbox”. Learning to choose a strategy in the presence of many possibilities, with different strengths, is essential to learning about problem-solving.

We also liked the ideas of greater challenge by suggesting it might be done with fewer steps: we felt it really would make the learner try to do better. But whereas a real teacher might monitor and provide hints when the learner gets frustrated, the tutorial offered nothing equivalent. Considering Kort’s cycle, there was support to challenge, but not to overcome. As we show in Fig. 5, we see the support for the cycle strong in some places (Challenge, Affirm) but weak in others (Explore, Overcome).

Fig. 5.

Minecraft and Kort et al. [7] steps.

Simple Programming. As the name “Hour of Code” suggests, the tutorial only addresses the simple beginnings of programming — “First Hour of Code” might be more appropriate. We noticed no real introduction to variables, for example, despite the potential of having them associated with objects in the model world. There was also no lead-in to object orientation, the most common practical paradigm for programming, again despite the potential of objects in a model world — this is exploited well in Alice, for example. In reviewing comments on official videos about the Hour of Code tutorial, this was a common criticism, that it wasn’t introducing “real” programming:

I’m not convinced that this kind of a tutorial could help anybody in understanding of programming paradigms. Modern programming is mostly object-oriented, and this film could at most give a little of vision, how sequential programming should look alike. IMO it’s not proper to teach only sequential programming, without any code and without even telling, that another styles of programming exists. Moreover, I think that beginning learning of programming with sequential programming instead of object-oriented programming could make OOP harder to understand Youtube Comment

However it is still an open question as to whether OOP or sequential programming is best for absolute beginners. Similarly, other programming paradigms are also not addressed. These issues are beyond our limited scope in this study.

One last point is perhaps more relevant. By basing the tutorial on Minecraft, many beginners might hope to get started with actually programming in Minecraft, which is done in JavaScript. The tutorials do show pop-up windows showing JavaScript code equivalent to the jigsaw blocks, but go no further. We understand that the complexities and dependences in real JavaScript would present great difficulties to address within the Hour of Code framework, but we also anticipate that some beginners might find this disappointing.

Table 10. The affective heuristics (version 3).

4 Minecraft Hour of Code and Affective Heuristic Evaluation

This section addresses the use of the affective heuristic evaluation [5] to evaluate affect in the user interface, educational design, and content of Minecraft Hour of Code to see if the software supports the educational objectives, narrative and persuasion to make the learners more engaged in learning programming. The affective heuristic evaluation is based on Nielsen and Molich [9], and adapted with affect supported by Norman [10] and Kort et al.’s emotional cycle of learning model [7] as well as multimodal design based on Sankey [12]. We have explained the full rational in more detail earlier [4].

4.1 Method

The study was performed in the same laboratory as earlier. In this study, we applied the evaluation method as discussed earlier. We reviewed the heuristics, explored, and then reflected on the software. We then wrote comments about the environment filling in a table with respect to the different heuristics. For heuristic severity we adapted Nielsen’s severity ratings for usability problems [8], changing their emphasis to reflect emotional educational impact. We explicitly emphasized on the learning objective and the modality that was employed.

For the learning objective we considered the following. This system was designed to be fun for a student working alone or in a classroom where the instructor has minimal preparation or computer science background. It has fourteen tutorials to learn the basics of JavaScript programming concepts that are: use of commands, repeat loops and if statements. It aims to create a quick and enjoyable experience for students and instructors who are new to computer science.

The modality involved a 2D block world with programming using jigsaw pieces. There is also an avatar to represent the user, animation of the block world, and engaging audio samples. Moreover, it used a narrative and persuasion quasi-modalities.

We went through the interface systematically. For example we added blocks to the workspace, trying to think of the intended user. We checked in each step if the system state to consider the learning objective and the modality employed. At the end of the suggested tasks, we filled in a table with issues based on the ten heuristics. Table 10 shows the affective heuristic evaluation. This is the set of heuristics following revisions discussed in [5].

4.2 Findings

In this study, the evaluators read the heuristics, explored, and then reflected on the environment. They then wrote comments including the interface element that was violated, problems that illustrated poor considering of affect and modality, suggestions and recommendations for solutions, and the severity based on Jakob Nielsen’s five-step rating scale [8]. Tables 11 and 12 show the affective heuristic evaluation results. Each of the ten heuristics led to a useful comment at least once.

Table 11. The affective heuristic evaluation H1 to H5.

Table 12. The affective heuristic evaluation H6 to H10.

4.3 Discussion

The heuristics invited reflection on modality as well as affect. Our findings show that the affective heuristic evaluation led to identification of many problems and potential solutions as shown in the tables. By reviewing these results, we can make some general observations.

Narrative and Continuity. There was no clear continuity between the steps. One of the tutorial steps was to chop down trees (Fig. 6 top left), and the next one was to shear sheep (Fig. 6 top right). The wool that resulted from shearing was never used or mentioned again. It did say that wood was useful for building and later steps did build walls for a house, but there was no indication of the wood came from chopping down the trees. It could have been made more continuous making it clear that the trees were used to build the house and also by using the wool to make carpets etc. They are building a house without using the wood or the wool, and therefore, there is no strong continuity and storytelling between the steps (Fig. 6 bottom). This might also have been a useful opportunity to introduce quantities of wood or wool as variables.

Fig. 6.

Narrative and continuity.

Encouraging Exploration. There was not enough encouraging to explore at each step, and no clear story for exploration. The commands provided were specific to each step, meaning there were not all commands provided in each tutorial e.g. shear was provided in the sheep-shearing task but not in other tasks. The commands were changed for every tutorial, which is a real limit to learning as it restricts exploration, and people would expect specific commands for each programming situation. Figure 7 shows the command blocks for tutorials two and three. They can use “destroy block” in the second tutorial step but it is not seen in the third tutorial step where they have to use “shear” instead.

Fig. 7.

Encouraging exploration.

JavaScript Code. As we mentioned, the real Minecraft software is programmed using JavaScript. Minecraft Hour of Code uses the jigsaw command blocks. But when the task is complete, is shows the real JavaScript code. But a learner would not know what they have to do with this code; there is a poor connection between the JavaScript code and the block commands that represents that code, and there is no affective feedback provided to the learners. Therefore, the connection between them is not made; we can not edit or change the JavaScript code. They are not really learning the JavaScript language. The JavaScript code is not even nicely formatted, and there is no interaction with the learners, which can lead to disappointment for them (see Fig. 8).

Fig. 8.

Blocks with the generated code.

5 Conclusions

In this paper we presented our evaluation of the multimodal affective learning support in Code.org’s Hour of Code tutorial set based on the popular game Minecraft. We applied our two proposed evaluation methods called the affective walkthrough and the affective heuristic evaluation, modeled on the widely used cognitive walkthrough and heuristic evaluation. The new methods keep the procedural framework of the cognitive walkthrough and heuristic evaluation, but as the evaluator steps through, the questions and heuristics are about emotional support in education.

The Hour of Code Minecraft software applies various modalities to support learning, talking a visual approach with a engaging game-like graphics, a model world, and a programming language using jigsaw-like pieces (as do Scratch and Alice). The tutorial instructions add a narrative aspect, whereby the programming tasks involve acting out a story in the model world.

Our experience in conducting the evaluation was positive and enlightening. We recognized elements in the Hour of Code design that we would not otherwise have noticed, and also identified ways in which the design might be improved. We see as especially valuable the interplay between the spatial model world, and the narrative stemming from the instructions. On the other hand, we felt that some extra freedom in command choice might in several places be helpful. Also, where challenges were provided by suggesting shorter solutions were possible, it would be helpful to add a capability for the learner get hints so they can overcome the challenges if they are stuck. We also recognized the limits of the tutorials, which really only learners with the very early steps in learning to program.

We appreciated the focus of the affective walkthrough and affective heuristic evaluation focus on modality benefits and teaching goals, and most importantly its use of Kort’s model of the affective cycle in learning. We feel they offer a helpful and systematic approach to evaluating affective learning in multimodal software.