Designing Effective Playful Experiences for Sustainability Awareness in Schools and Makerspaces

Gamification, defined as “the use of game design elements in non-game contexts” in [16], is an approach that has been gaining popularity in both the fields of sustainability and educational research. The use of gamification in education was recently discussed in [17], where the need for additional systematic studies to understand the actual effect of this approach is supported by the authors, due to the fact that most of the studies focus on short-term effects and results, and do not consider how such interventions can bring long-term effects and system changes. In this context, [30] provides a review of recent bibliography in serious games and gamified interventions for sustainability, concluding that such approaches can help foster sustainability, while there is a wealth of different approaches and deployment strategies.

Regarding the use of gamification for energy saving, the work of Johnson et al. [24] provided a recent review of the use of such techniques, while Ro et al. [36] reports on positive outcomes of gamification on both sustainability awareness and energy savings. Zhao et al. [45] studied the interplay and effect between the various technologies used inside a building for monitoring/control, concluding that both technology advances in building equipment/devices and the ways humans interact with them can lead to energy savings. Moreover, as regards the use of competition in this context, the relation between competition and sustainable behavior was studied in [40], where the authors claim that utilizing competition in actions promoting sustainability is a valid “marketing” strategy, since it appeals to both groups that are less motivated and those that are more inclined to participate in such activities. This is also discussed in [44], where both competitive and collaborative approaches are studied, and which have led to similar energy saving results. The results presented in [34] seem to verify the value of competition to a certain extent, where results from long-term energy-saving competitions among 4 schools led to modest energy savings. Our work discussed here agrees with their findings, while we also provide results on the educational dimension of the schools’ participation.

Regarding the use of gamification for energy saving, a characteristic early example of the use of a serious game towards energy and sustainability awareness was Enercities [26], reporting increased awareness and more positive attitudes as regards energy saving. Opower [27] was one of the earliest commercial applications utilizing behavioral science-based mechanisms, aiming to produce energy savings in the real world by partnering with energy providers and offering consumers located in the same neighborhood targeted information. Real-world results point to 2-3% energy savings in the long term. The work of Johnson et al. [25] provided a recent review of the use of such techniques, while Ro et al. [37] reports on positive outcomes of gamification on both sustainability awareness and energy savings. A collection of sustainability-related serious games is available on Games4Sustainability [5].

Recent projects like Charged [19], Entropy [8], GreenSoul [10], Tribe [12], have pursued IoT- and game-based approaches towards increasing awareness on behavior-based energy efficiency in public buildings. However, although schools may be considered examples of public buildings, the software and hardware artefacts produced in these projects are not designed to be included in a school's curriculum; in fact, they target adults, making it difficult for teachers and parents to apply such materials with young students. With respect to gamification utilizing IoT in the context of sustainability, and specifically for energy and water, a recent survey on the subject is provided in [13].

Regarding the successful integration of gamification techniques for energy saving and sustainability awareness, while there is a large number of works that report on positive outcomes, e.g., [27, 31, 37], at the same time there are also works that report on less positive results and stress the acute need for taking action [41]. This controversy is partially based on the fact that there is no commonly acceptable way on how results are measured and reported; [38] provides a detailed discussion on common pitfalls and mistakes made by teams working on these issues. It concludes that there is a number of works that do not sufficiently back up their findings with adequate evidence, resulting in reports which are oftentimes disputable and provide contradicting results. For such evidence to be useful for the society and activate policy and practice, it needs to be connected with rigorous experiments and measurements. Dos Santos et al. [20] provided a comprehensive review of recent works on the use of gamification techniques for approaching learning activities related to sustainability. It provides a categorization of related serious games based on their focus, namely, educational and motivational games for sustainability. It concludes that existing results are somewhat partial and there is a need for a more rigorous and systematic approach in evaluating the effectiveness of such approaches. Another recent work reporting on the causes of the failure in practice of gamification approaches applied in an educational context is [41]. The authors present a set of “heuristics” to evaluate whether a gamification approach applied in education has a potential to be successful, such as “avoid obligatory uses”, “make the system flexible” and “align gamification with the goal of the activity in question”. In [42], the authors argue that a more thorough understanding of how gamification works in education and especially with respect to motivational processes and need-supporting game elements, i.e., “gamification in education that is designed to foster learners’ basic psychological needs satisfaction”.

Regarding the current state of the art in inclusion of sustainability and other related aspects in the educational domain, there is a lot of activity taking place with respect to inclusion of makerspace elements in school curricula, aided by the availability of IoT hardware as well. The work in [33] summarizes recent activity within the Maker Movement approach, presenting relevant recent findings and open issues in related research. Recent projects like Entropy [8] target diverse end-user communities and do not focus specifically on the educational community, or leave it out completely.

With respect to the work presented in this paper, we follow a different approach than the vast majority of the gamification approaches studied in the bibliography; instead of focusing solely on a gamified/playful strategy, we have placed the Challenge inside a set of tools to achieve our goals. In addition, the design of the Challenge is largely compatible with strategies outlined/proposed in works like [20] and [41], especially aligning gamification with the goal of the activity in question and offering a flexible approach, offering among others, a focused and seamless experience that goes hand by hand with curriculum goals and encouraging group activity to reach learning goals, instead of rewarding specific individuals. The gameful intervention discussed here was one of the pillars used to introduce students to sustainability concepts, keep them engaged and allow them to “compete” across schools. Regarding our own previous related work, [31] provides a discussion on the overall approach and strategy of GAIA and presents some early findings as regards both energy savings and feedback from students and educators. [32] discusses positive results received via large-scale pilot activities with groups of students using an IoT-based lab kit.

The Challenge is an online playful introduction to sustainability and energy saving, targeting mostly school students. Its architecture combines a quest map with a dashboard and uses sensor data coming from school buildings (Fig. 1). It consists of a number of missions (Fig. 2) and utilizes gamification mechanics to motivate students to engage in energy saving topics by collaboratively working on online “quests” and participating in real-life activities. Moreover, students experience their impact on the facilities’ energy consumption over the course of the Challenge, while also competing and comparing against other classes and schools. Overall, it has the following goals:

• Motivate children to engage with energy saving topics.
  
• Offer a variety of online quests (quest map).
  
• Allow children to participate in real life activities (class activities).
  
• Introduce competition elements to children via scoreboards with data from other schools and other countries.
  
• Allow students to share their achievements with their peer group.
  
• Scaffold an in-situ experience and engage children on their school's energy consumption.
  

Gamification in the GAIA Challenge is utilized in the following ways:

• Students and teachers pick up tasks from the Challenge and either work on these tasks directly in the Challenge, or they use it together with a number of other tools produced in the GAIA research project.
  
• Knowledge missions are played directly in the Challenge browser window, but may require additional research or actions from other GAIA applications, like a web-based building manager application.
  
• Findings, results and ideas can be shared in the form of “Snapshots”.
  
• For action missions, no direct interaction within the Challenge is required, but students instead have to engage with their school community, the school building and the other applications.
  
• The results of Action Missions can be documented in “Portfolios”, supplemented with photos/videos by students.
  
• Visualizations of sensor data, can be grabbed from a Building Manager App and embedded to Snapshots/Portfolios.
  

Activities using the Challenge took place within the GAIA project across Greece, Italy and Sweden. There was an initial small-scale pilot activity, mainly for testing purposes, during 2017. The design and the majority of the content was finalized in early 2018, while large-scale pilot activities took place in school years 2017-18 and 2018-19. By the end of the project, a total of 25 schools had participated in the pilots, with most schools being active in both school years.

To evaluate the effectiveness of the intervention and investigate its benefits, we collected data via 3 different sources: i) analytics generated by students’ activity during the Challenge, ii) responses to an online survey that was sent out to participating students, and iii) questionnaire responses from educators.

As mentioned, the collected data consists of 2 types, analytics and questionnaires. An appropriate data analysis was used for each different set of data. Per students’ analytics, we used descriptive statistics to understand the students’ overall engagement with the Challenge and aggregated time series visualization, in order to understand how their usage behavior changed over time. In addition, we used descriptive statistics to understand the students’ overall reported experience, as demonstrated from questionnaire responses. Regarding the analytics from the Challenge website, we include here results from the whole active period of the project, i.e., between the public opening of the Challenge on Spring 2017 and May 2019. As regards the players’ data from the Challenge database, we include here results from a representative “screenshot” on Spring 2018, which includes only the active players at that point, i.e., the school year 2017-18. We selected this period mainly due to having a larger user sample for that year.

By the end of the project, 73 teams appeared to be active in the Challenge, with 7 teams achieving a score above 11.200 points (the total of regular points each user can earn). This essentially means that they must have played not just the complete learning part of the challenge, but also achieved big portions of the limited time bonus (there is a maximum of 3780 bonus points). The actual number of active users is lesser than the total number of registered users, because we have placed a mechanism for auto-deleting inactive users after a period of 6 months, i.e., a large number of students were auto-deleted. This is consistent with our ethics and privacy policy, i.e., keeping as little information about the users as possible, and deleting it after the time period which is no longer required.

Visits per device: 28% of the Challenge usage comes from smartphones, although it was designed for big screens. The vast majority of visits comes from desktop/laptop users, as expected, with a sizable part of visits from tablets.

As mentioned, we focus on the “screenshot” of our database on May 2018. At that point, there were 1256 active users in the database (13 were researchers/test accounts). An interesting statistic was the mission completion rate for users who had started a mission at least once, which was on average 87.45% for all missions (87.2% for Knowledge missions). It can be concluded that almost 9 out of 10 users who started a mission then completed it at least once, i.e., they found it engaging enough to complete it through. Regarding the number of missions completed by the students, Fig. 8 shows the distribution of the number of completed knowledge missions per user. Almost half of the students appear to either not complete a single mission (16%), or got stuck/stopped at 1 (17%) or 2 (18%) missions. Since most players followed a linear route in gameplay, such users stopped at “level” 1 or 2. On the opposite end, 21% of the users completed all knowledge missions, or played missions in an additional language. In the space between, percentages are between 3-6%.

However, this picture changes when we look at the user statistics for each language in the Challenge. In general, the language used in the Challenge roughly corresponds to the country of origin of the user, although there are a few cases of users playing the Challenge in more than 1 language for some missions. Moreover, via the language used we can roughly categorize the users: for English the vast majority of the students were 17-18 years old, while for Italian the users’ age was 15-16, and for Greek it ranged between 10 to 15 years old. We can see that for English there is a quite steep drop in interest after missions 1 and 2, while for Greek results are quite different with a much smoother curve. For Italian, the curve is even flatter, and after mission 4, 60% continued to mission 9.

From our statistics, we found that in the Challenge there have been 22.429 started tasks altogether. The breakdown into the statistics for each knowledge and action mission is included in the following two tables. As a reminder, each mission has two tasks. A player can open a task multiple times, e.g., in order to improve mission completion statistics. The tasks in Action Missions can be started only by players, granted the Action Mission has been started beforehand by a Teacher for the Player's Mission Team. Not many users have started/opened the contents of Action Missions in the Challenge. There could be multiple reasons for this: a) not all players were part of a Mission Team, b) not all teachers started Action Missions for their Mission Teams, c) some students might have worked on Action Missions in groups.

A limitation of this work is that we did not conduct trials with study groups regarding the use of the Challenge. This was to a certain degree due to the fact that it was designed to serve as an introduction to sustainability and the project overall. Thus, student groups not using the Challenge or using it to a limited extent would change the overall design of the pilot activities. Moreover, among classes in the same school, in practice we saw that educators were reluctant to divide students to separate groups, arguing they would prefer that all students followed the same work plan.