On Evaluating Social Learning Outcomes of Serious Games to Collaboratively Address Sustainability Problems: A Literature Review

Abstract

Serious games are increasingly explored as collaborative tools to enhance social learning on sustainable management of land and natural resources. A systematic literature review was conducted to examine the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games. Forty-two publications were identified and included in the review following study selection and quality assessment steps. Extracted data from the publications were categorized in relation to five research questions. Approaches that were used to assess cognitive, normative, and relational learning outcomes of collaborative serious games were subsequently identified based on the categorizations. As a result, these approaches distinguished between the nature of learning in the assessment of collaborative serious games. Combined, these approaches provide an overview of how to assess social learning outcomes of collaborative serious games, including the methods and procedures that can be used, and may serve as a reference for scholars designing and evaluating collaborative serious games.

1. Introduction

Environmental sustainability problems are typically complex and multi-scale, concern inherent uncertainty, and affect multiple stakeholders and agencies. Solving sustainability problems concerns the management of land and natural resources in a way that creates and maintains prosperous social, economic, and ecological systems [1]. To address such problems, decision-making needs to be adaptive to deal with the uncertainties and needs to include the diversity of knowledge and values of all affected stakeholders. To achieve this, scholars have advocated active experimentation and continuous evaluation, summarized as learning-by-doing, in natural resources management [2,3,4,5]. Central to these approaches is collaboration between and learning among researchers, resource managers, and resource users in order to find sustainable solutions [1,6,7,8].

Learning, in particular social learning, is therefore seen as a prominent driver and normative goal in natural resources management [2,9,10,11,12]. Although there is debate to find a commonly shared definition for social learning (see e.g., [3,11,13,14]), scholars increasingly agree that social learning has occurred when a change in understanding—related to for example the system, problem at hand, agreement, and collective action—is achieved through interaction in collaborative and participatory settings [11,13,15,16,17]. Social learning outcomes therefore require deliberative interactions, where multiple stakeholders work together and build relationships, which should ultimately lead to collective action [14,15,18]. Baird et al. [17] define three types of learning outcomes in relation to social learning: cognitive, the acquisition of new or restructuring of existing knowledge; normative, a shift in viewpoints, values or paradigms; and relational, an improved understanding of others’ mind-sets and enhanced trust and ability to cooperate between stakeholders.

Serious games are increasingly explored as a method to establish social learning on sustainable natural resources management and urban planning [19,20,21,22]. Serious games are generally referenced to as games that have a primary purpose other than entertainment, such as educating, training or informing players [23,24,25]. More specifically towards policy-making, Mayer ([26], p. 825) defines games as “experi(m)ent(i)al, rule-based, interactive environments, where players learn by taking actions and by experiencing their effects through feedback mechanisms that are deliberately built into and around the game”. A recognized strength of such serious games is that games can include both the techno-physical complexity—the underlying physical elements of the system and its uncertainties—and the socio-political complexity—the strategic interactions between stakeholders in the policy arena—by combining role-play with in-game feedback mechanisms [20,21,26,27]. Therefore, serious games fit well with the learning-by-doing approach in natural resources management; serious games offer stakeholders a place to negotiate, deliberate, exchange perspectives used in decision-making and learn about the trade-offs between decisions in the safe experimentation environment of a game [26,27]. Moreover, multiplayer and multi-role serious games thereby offer the collaborative and participatory stakeholder interactions that are needed to establish social learning [20,22]. The term collaborative serious games is used in the remainder of the paper to refer to serious games that follow the above definition by Mayer [26], but are particularly focused to collaboratively—through mutual engagement of stakeholders in a coordinated effort to solve the problem ([28], p. 70)—explore sustainable management strategies.

The assumption is that any learning that occurs from playing collaborative serious games is transferable to the world outside the game [26,29]. A collaborative serious game can therefore be seen as an intervention or a transitional object [29] that may lead to social learning outcomes. While calls exist for more systematic assessment of learning through serious games (see e.g., [30]), assessing social learning outcomes of collaborative serious games is challenging. On a practical level, gathering data occurs in a collaborative setting that introduces many confounding variables such as players’ prior relations, players’ attitudes towards serious gaming, and facilitators who have to make decisions on the spot while guiding sessions [30,31,32]. On a higher level, the lack of consensus on the definition of social learning itself has made it difficult to assess [8], and indeed few studies empirically and directly assess the learning effects of interventions [3,12,17].

The research presented in this paper adds to this gap by answering the main research question: What is the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games? To this end, a systematic literature review was conducted to survey the empirical assessment of social learning through collaborative serious games. As such, the contributions of this paper include: (1) summarizing and categorizing different evaluation procedures applied and methods used to assess social learning outcomes of collaborative serious games; and (2) presenting a state-of-the-art overview of approaches to assess social learning outcomes of collaborative serious games.

The next section presents the method of the systematic review and describes all the steps taken in detail. Section 3 and Section 4 cover the search and categorization results of the review, respectively. Section 5 discusses approaches used to assess cognitive, normative, and relational learning outcomes of collaborative serious games. Section 6 finally sums up the conclusions drawn from the review and provides an overview of the identified assessment approaches.

2. Materials and Methods

To perform the literature review, the guidelines by Kitchenham and Charters [33]—an updated version of Kitchenham [34]—were used. These guidelines are an established procedure for conducting systematic reviews, particularly in software engineering. Although collaborative serious games are not exclusively digital—a board game can also be used to explore sustainable management solutions—the use of these guidelines is appropriate as many serious games do make use of software and the guidelines are based on guidelines used in other disciplines.

Kitchenham and Charters [33] prescribe three phases of a systematic review: planning, conducting, and reporting. The planning phase covers confirming the need for a review, defining the research questions, and producing a review protocol. The conducting phase involves developing a search strategy, executing as well as documenting the search, performing a study selection, and extracting data relevant to the research questions. The reporting phase then covers reporting and disseminating the results of the study. In the next subsections, all the steps taken in performing the systematic literature review are described.

2.1. Related Work

As a starting point in the study, a search was executed to confirm the need for a review. One literature review by Calderón and Ruiz [35] on the different methods and procedures used to evaluate serious games was identified in this step. Notably, the serious games included in this review are mostly evaluated using a questionnaire as the main assessment method and are mostly evaluated on the educational effectiveness—defined by Calderón and Ruiz [35] as the learning outcomes, usability, and user’s experience. The study is similar in terms of its scope and research questions covered, but lacks the focus on social learning and the policy setting. The study was therefore used as a reference for the overall search strategy, such as to sharpen the research questions and to formulate inclusion and exclusion criteria. Other reviews, both literature and meta reviews, focused on the effectiveness of serious games [36,37,38,39,40], not on the assessment procedures.

In addition to these studies, five review papers were identified which focus on serious games in relation to sustainable development [41], climate change [42,43,44], and sustainable water resources management [45]. While these do not review the assessment approaches of learning outcomes, the serious games reviewed in these papers do all relate to sustainability and may therefore be of interest to this review. These studies were therefore used to inform the search strategy, for example to define keywords, and as snowball resources.

Concluding, no review was found that focuses specifically on evaluating collaborative serious games and this study therefore adds an in-depth review with this explicit focus to the literature.

2.2. Research Questions

To examine the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games, the following research questions were addressed to analyse publications:

RQ1.

How is learning through collaborative serious games conceptualized?

RQ2.

When is data collected in the evaluation of learning through collaborative serious games?

RQ3.

What methods are used in the evaluation of learning through collaborative serious games?

RQ4.

Do evaluations of learning through collaborative serious games use quantitative, qualitative or a combination between quantitative and qualitative data?

RQ5.

What are the learning effects of collaborative serious games according to their evaluations in relation to social learning?

Learning is used over social learning in these questions to not exclude publications that use a different learning conceptualization but do fall within the scope of this review. For all research questions, data was extracted from the publications and subsequently categorized, which is further explained in the data extraction and data analysis Section 2.7 and Section 2.8.

2.3. Search Strategy

The search strategy defines the systematic approach to identify publications to include in the literature review. Two categories of keywords were used to create search combinations. The first category (A) covered three keywords to identify publications that covered a game-based approach: serious gam* (A1), simulation gam* (A2), and policy gam* (A3). The asterisk was used in each keyword to cover for both game and gaming. Simulation and policy gaming were added to the search terms as pilot searches showed this term, unlike alternatives such as applied gaming, returned search results that were relevant to the scope.

The second category (B) covered 18 keywords to identify publications in relation to sustainability and disciplines dealing with sustainability issues: sustainability (B1), sustainable use (B2), sustainable development (B3), climate change (B4), climate adaptation (B5), environmental change (B6), energy transition (B7), adaptive *management (B8), collaborative management (B9), natural resource management (B10), environmental management (B11), spatial planning (B12), urban planning (B13), urban development (B14), water management (B15), water resources (B16), river management (B17), and river basin (B18). These keywords were selected based on the five identified review papers on games and sustainability [41,42,43,44,45], pilot searches, and publications covering collaborative serious games already known to the authors. The asterisk in adaptive *management was added to also cover for both comanagement and co-management.

The keywords were subsequently used in search strings to search for all possible combinations between one keyword from category A and one keyword from category B. Using Boolean statements, the search strings therefore covered “(A1 OR A2 OR A3) AND (B1 OR B2 … OR B18)”.

Searches were limited to publications published since 2007 to review the state of the art; 2007 was chosen in order to cover the last ten years and as pilot searches found few publications providing sufficient information for the scope of the review before 2007—consistent with the results of Calderón and Ruiz [35].

Six databases were similarly determined through pilot searches for conducting the review: ACM Digital Library, IEEE Xplore Digital Library, ISI Web of Science, ScienceDirect, Scopus and SpringerLink. Searches were executed on title, abstract, and keywords.

2.4. Study Selection

To narrow down the publications obtained following the search strategy, an initial selection and a final selection step were performed following inclusion and exclusion criteria (

Table 1. Inclusion and exclusion criteria.

Inclusion criteria

Publications that describe a collaborative serious game on sustainable management of land or natural resources, or otherwise related to a complex sustainability issues (e.g., climate change) Publications that describe the assessment of learning outcomes of a collaborative serious game Publications that describe the procedure and methods applied to assess learning outcomes of a collaborative serious game

Exclusion criteria

Publications that describe a serious game, but which do not cover a collaborative serious game (e.g., a single-player serious game) Publications that describe a collaborative serious game, but the game is not in any way related to sustainability Publications that describe a collaborative serious game, but which do not assess its learning outcomes Publications that describe the evaluation of the learning outcomes from a collaborative serious game, but offer no information on how this assessment was performed Publications that describe the assessment of a larger learning process of which a collaborative serious game is a component that cannot be seen as separated from the larger process and its evaluation Publications that describe a formative evaluation that only focuses on improving the design of a collaborative serious game and do not assess learning outcomes in any way Publications that describe a serious game designed only for use in an educational setting Publications that are not written in English Publications that are only available as abstract 1

¹ Abstracts were still reviewed, if the abstract seemed applicable, an attempt was made to obtain the entire publication, exclusion occurred if this was not possible.

). In the initial selection, each study was reviewed based on its title, abstract, and keywords. This way, publications clearly not covering the scope of the literature review—i.e., a publication describing the economy and markets as a serious game—were excluded. In the final selection, the publications included in the initial selection were reviewed more rigorously according to the same criteria by scanning through each study and readings its conclusion. The entire study was read if necessary to determine inclusion or exclusion.

2.5. Study Quality Assessment

Following the study selection steps, the selected publications were looked at in more detail in relation to the research questions in a quality assessment step. Specifically, questions that can be answered with yes or no were formulated in relation to the research questions:

• Does the publication provide a conceptualization of learning? (RQ1)
• Does the publication state when data is collected in the evaluation of the collaborative serious game? (RQ2)
• Does the publication discuss a method, technique or theory used to evaluate the collaborative serious game? (RQ3)
• Does the publication discuss whether the evaluation used a quantitative or qualitative approach, or both in the evaluation of the collaborative serious game? (RQ4)
• Does the publication describe learning outcomes of the collaborative serious game following evaluation? (RQ5)

The publications included in the study selection step were assessed on each of these questions as yes or no, corresponding to a 1 or 0 score. If a publication included all relevant aspects in regard to the scope of the review, it would therefore score five out of five. Publications that scored four or higher were deemed of sufficient quality and included in the review. Publications that scored exactly three, and were thus excluded, were checked again after completing the quality assessment to confirm exclusion was the correct decision.

2.6. Snowballing

To cover for relevant publications missed in the database searches, a snowballing approach was used to identify further relevant publications [46]. Firstly, backward snowballing was applied by identifying possible relevant publications from the reference lists of publications passing the quality assessment step. Secondly, forward snowballing was applied by checking all publications citing the publications deemed of sufficient quality using Google Scholar. Thirdly, the five review papers on sustainable development, climate change, and sustainable water resources management [41,42,43,44,45] identified earlier were used to snowball from as serious games covered in these reviews may be of interest to the scope of this review. All papers identified through snowballing were subsequently reviewed following the same study selection and quality assessment steps.

2.7. Data Extraction

In the data extraction step, all the publications included in the study selection and deemed of sufficient quality were read in its entirety. Data was extracted from the publications in regard to the research questions. All extracted data, for all publications, were stored in a spreadsheet. At this point, if publications were identified covering the same collaborative serious game without providing additional insight into its evaluation—i.e., a conference paper covering a limited evaluation and a journal paper covering a more extensive evaluation by the same authors—the publication with the lower-quality assessment was excluded from the review.

2.8. Data Analysis and Categorization

All data was subsequently categorized in regard to each research question. An inductive approach was used to create categories based on the data retrieved by identifying common themes, for example applied methods as surveyed in RQ3. On the first research question, for example, in regard to the conceptualization of learning, categories included social learning and experiential learning. For RQ5, the typology on social learning by Baird et al. [17] (see

Table 2. Typology of learning effects (from Baird et al. [17]).

Type	Definition/Indicators of Learning Effects	Measures of Indicators
Cognitive learning	Acquisition of new knowledge; restructuring of existing knowledge	Test scores; change in centrality and specificity of knowledge presented in concept maps
Normative learning	Changes in norms; change in values; change in paradigms; convergence of group opinion	Change in, and convergence around, environmental beliefs; participant reflections; meeting proceedings
Relational learning	Improved understanding of mind sets of others; building of relationships; enhanced trust and cooperation	Change in social network structure; participant reflections

) was used as a framework to define categories that differentiate between the three types of learning. From an assessment perspective, the typology is beneficial as it views learning outcomes from their nature—cognitive, normative or relational—rather than their perceived value. Moreover, the typology separates relational learning as an explicit learning outcome, which is of particular interest in the multi-stakeholder context of sustainable land and natural resources management.

The categorizations were subsequently used to identify assessment approaches used in the evaluation of learning outcomes of collaborative serious games.

3. Search Results

The entire review, from developing the scope and performing the search, selection, data analysis, and synthesis, was executed between October 2017 and November 2018. Initial database searches were executed on or a few days before the 7th of January 2018, and all publications extracted through these searches were analysed. After processing all retrieved publications, all database searches were updated on the 7th of November 2018. Figure 1 provides an overview of the number of publications found and subsequently included after each step.

As Figure 1 shows, 78 publications retrieved through the database searches were included in the selection steps following the inclusion criteria, of which 41 publications were included following quality assessment. Sixty-six additional publications were identified through snowballing, 45 publications from the publications identified through database searches, and 21 from the five review papers [41,42,43,44,45]. Six of these 66 publications passed the same selection steps and quality assessment. Five publications were subsequently removed from the total publications as these covered the same collaborative serious game without providing additional insight into its evaluation. Forty-two publications were therefore included in the review.

Figure 2a shows an overview of how many publications included in the review were retrieved through each database as well as through snowballing. Some publications appeared in multiple databases and are added to these databases in the figure, explaining why the numbers added up are higher than 42. Scopus, as can be expected, provided the most publications included in the review. No publications retrieved from the IEEE database passed the selection criteria and quality assessment steps.

Figure 2b shows an overview of the publications included in the review in regard to their year of publication. The figure suggests an increasing trend in the use of collaborative serious games to explore sustainable management strategies. It must be noted that 2018 has the highest publications included in the review (10 publications), even though the review only covers the first 10 months of 2018.

Table 3. Overview of all games and publications included in the review.

Game Name	Reference
Aqua Republica	Jean, Medema, Adamowski, Chew, Delaney and Wals [48]
CauxOperation	Souchère, Millair, Echeverria, Bousquet, Le Page and Etienne [49]
Climate Game	Zhou, Bekebrede, Mayer, Warmerdam and Knepflé [50]
Community Cooperation Game	Tanwattana and Toyoda [51]
Forage Rummy	Martin [52]
FOWIS	Hertzog, Poussin, Tangara, Kouriba and Jamin [53]
Futura	Antle, Bevans, Tanenbaum, Seaborn and Wang [54]
GO2Zero	Bekebrede, van Bueren and Wenzler [55]
Grazing game	Villamor and Badmos [56]
Invitational Drought Tournament	Hill, Hadarits, Rieger, Strickert, Davies and Strobbe [57]
IUP & NECAP	Rumore, Schenk and Susskind [58]
KEEP COOL	Eisenack [59]
	Meya and Eisenack [60]
Lords of the Valley	Magnuszewski, et al. [61]
LottoSIM	Becu, et al. [62]
MAE SALAE RPG 1 & 2	Barnaud, Promburom, Trébuil and Bousquet [63]
Marine Spatial Planning Challenge	Mayer, Zhou, Lo, Abspoel, Keijser, Olsen, Nixon and Kannen [64]
	Keijser, Ripken, Mayer, Warmelink, Abspoel, Fairgrieve and Paris [65]
New-District	Becu, Frascaria-Lacoste and Latune [66]
REEFGAME	Cleland, Dray, Perez, Cruz-Trinidad and Geronimo [67]
ReHab	Page, Dray, Perez and Garcia [68]
RESORTES	Speelman, García-Barrios, Groot and Tittonell [69]
Shariva	Douven, Mul, Son, Bakker, Radosevich and Hendriks [70]
SimPhy	Ayadi, Le Bars, Le Grusse, Mandart, Fabre, Bouaziz and Bord [71]
SPRINTCITY	Duffhues, Mayer, Nefs and van der Vliet [72]
Sustainable Delta Game (and adaptations)	Valkering, van der Brugge, Offermans, Haasnoot and Vreugdenhil [73]
	Lawrence and Haasnoot [74]
	Van der Wal, De Kraker, Kroeze, Kirschner and Valkering [75]
	Van Pelt, Haasnoot, Arts, Ludwig, Swart and Biesbroek [76]
TADLA	Dionnet, Kuper, Hammani and Garin [77]
Ter’Aguas	Ducrot, Van Paassen, Barban, Daré and Gramaglia [78]
The Floodplain Management Game	Stefanska, Magnuszewski, Sendzimir, Romaniuk, Taillieu, Dubel, Flachner and Balogh [79]
The Multi-Hazard Tournament	Carson, et al. [80]
Unnamed game	Salvini, Van Paassen, Ligtenberg, Carrero and Bregt [81]
Unnamed game	Haug, Huitema and Wenzler [82]
Unnamed game	Meinzen-Dick, Janssen, Kandikuppa, Chaturvedi, Rao and Theis [83]
Unnamed game	Sausse, Le Bail, Lecroart, Remy and Messéan [84]
Unnamed game	Moser and Mußhoff [85]
WeShareIt	Onencan, Van de Walle, Enserink, Chelang’a and Kulei [86]
WeShareIt Nzoia	Onencan and Van de Walle [87]
WORLD CLIMATE	Sterman, Franck, Fiddaman, Jones, McCauley, Rice, Sawin, Siegel and Rooney-Varga [88]
WSP game	Ferrero, Bichai and Rusca [89]

provides an overview of the publications included in the review and the game each publication evaluates.

4. Categorization Results

This section covers the categorized findings in relation to the formulated research questions. Through an inductive approach, between three and 13 categories were identified for each research question. A complete overview of all the publications and assigned categories can be found in

Table A1. Overview of all publications included in the review, the quality assessment and the categorization of the data in relation to the research questions.

Game Name	Reference	Quality Assessment	Learning Conceptualization (RQ1)	Data Collection (RQ2)	Assessment Methods (RQ3)	Quantitative or Qualitative Data (RQ4)	Social Learning Effects (RQ5)
Aqua Republica	[48]	5	Boundary crossing	Pre & During & Post	Questionnaires & interaction analysis & social network analysis	Quantitative & qualitative	Understanding other perspectives Building relationships and trust
CauxOperation	[49]	5	Collective learning	Post	Debriefing	Qualitative	Raised awareness Increased system understanding Understanding other perspectives Building relationships and trust
Climate Game	[50]	5	Links to experiential learning	Pre & During & Post	Questionnaires & observations & debriefs	Quantitative & qualitative	Increased system understanding
Community Cooperation Game	[51]	5	Links to experiential learning	During	Observations	Qualitative	Increased system understanding Raised awareness
Forage Rummy	[52]	5	Knowledge co-production	Post	Direct feedback	Qualitative	Increased system understanding Raised awareness Building relationships and trust
FOWIS	[53]	4	Not conceptualized	During & Post	Observations & debriefs & Interviews	Qualitative	Increased system understanding
Futura	[54]	5	Experiential learning	During	Observations	Qualitative	Raised awareness
GO2Zero	[55]	4	Not conceptualized	Pre & During & Post	Questionnaires & Observations	Quantitative & qualitative	Increased system understanding Building relationships and trust
Grazing game	[56]	5	Social learning & Anticipatory learning	During & Post	Observations & debrief	Qualitative	Increased system understanding Understanding other perspectives
Invitational Drought Tournament	[57]	5	Social learning	Post	Questionnaires & debriefs	Qualitative	Increased system understanding
IUP & NECAP	[58]	5	Social learning	Pre & Post & Post-post	Questionnaires & debriefs & interviews	Quantitative & qualitative	Increased system understanding Raised awareness Understanding other perspectives
KEEP COOL	[59]	4	Not conceptualized	Pre & During & Post	Questionnaires & observations	Qualitative	Increased system understanding
	[60]	4	Not conceptualized	Pre & During & Post	Questionnaire & Observations	Quantitative	Increased system understanding Raised awareness Change in views
Lords of the Valley	[61]	4	Social learning	Pre & During & Post	Questionnaires & observations	Quantitative & qualitative	No learning effects reported
LottoSIM	[62]	5	Links to social learning & experiential learning	Post	Debriefs	Qualitative	Increased system understanding
MAE SALAE RPG 1 & 2	[63]	5	Collective learning	During & Post & Post-post	Observations & Interviews	Qualitative	Increased system understanding Understanding other perspectives
Marine Spatial Planning Challenge	[64]	5	Constructivist learning philosophy	During & Post	Questionnaires & observations	Quantitative & qualitative	Increased system understanding
	[65]	5	Policy-oriented learning	Pre & During & Post	Questionnaires & observations & data logging & interviews	Quantitative & qualitative	Increased system understanding Building relationships and trust
New-District	[66]	5	Experiential learning	Pre & Post	Questionnaires	Quantitative & qualitative	Understanding other perspectives
REEFGAME	[67]	5	Links to social learning	During & Post	Debriefs	Qualitative	Increased system understanding
ReHab	[68]	5	Social learning & experiential learning	During & Post	Observations & debriefings	Quantitative & qualitative	Increased system understanding
RESORTES	[69]	4	Not conceptualized	Pre & During & Post & Post-post	Questionnaires & observations & debrief & interview	Qualitative	Building relationships and trust
Shariva	[70]	5	Meaningful play	Pre & Post	Questionnaires	Quantitative & qualitative	Raised awareness Increased system understanding Understanding other perspectives
SimPhy	[71]	4	Not conceptualized	During	Data logging	Quantitative	Increased system understanding
SPRINTCITY	[72]	5	Policy oriented learning	During & Post	Questionnaires & observations & data logging & interviews	Qualitative	Raised awareness Increased system understanding
Sustainable Delta Game (and adaptations)	[73]	4	Not conceptualized	Pre & During & Post	Observations & debriefs & perspective mapping	Qualitative	Increased system understanding Understanding other perspectives
	[74]	5	Social learning	During & Post	Observations & interviews & debriefs	Qualitative	Increased system understanding Change in views
	[75]	5	Social learning & experiential learning	Pre & During & Post	Observations & data logging & interviews & perspective mapping	Quantitative & qualitative	Change in views
	[76]	4	Not conceptualized	Pre & During & Post & Post-post & Control Group	Questionnaires & observations & debriefs & interviews & control group	Quantitative & qualitative	Increased system understanding Raised awareness Establish experiential environment
TADLA	[77]	5	Individual learning & Social learning	Post & Post-post	Questionnaires & Interviews	Qualitative	Increased system understanding
Ter’Aguas	[78]	5	Social learning	Pre & During & Post & Post-post	Questionnaires & observations & debriefs & interviews	Qualitative	Increased system understanding Understanding other perspectives Change in views
The Floodplain Management Game	[79]	5	Social learning	During & Post	Observations & debriefs	Qualitative	Raised awareness
The Multi-Hazard Tournament (MHT)	[80]	5	Social learning	Pre & Post & Post-post	Questionnaires	Quantitative & qualitative	Increased system understanding Raised awareness Understanding other perspectives Building relationships and trust
Unnamed game	[81]	5	Social learning	Pre & Post	Interviews	Qualitative	Increased system understanding Building relationships and trust
Unnamed game	[82]	5	Social learning	Pre & During & Post & Post-post	Questionnaires & Observations & Debriefs & Interviews & Concept maps	Qualitative	Increased system understanding Understanding other perspectives
Unnamed game	[83]	5	Social learning	During & Post & Post-post & Control group	Questionnaires & data logging & interviews & real-world data	Quantitative & qualitative	Increased system understanding Real-world adoption
Unnamed game	[84]	4	Not conceptualized	During	Observations & data logging	Quantitative & Qualitative	Increased system understanding
Unnamed game	[85]	4	Not conceptualized	Pre & During	Questionnaires & data logging & control group	Quantitative	Increased system understanding
WeShareIt	[86]	5	Effective learning	Pre & During & Post	Questionnaires & observations & debrief	Quantitative & qualitative	Increased system understanding
WeShareIt Nzoia	[87]	5	Situational awareness	Pre & During & Post	Questionnaires & data logging	Quantitative	Raised awareness Increased system understanding
WORLD CLIMATE	[88]	5	Links to experiential learning	Pre & Post-post	Questionnaires	Quantitative	Increased system understanding Raised awareness Change in views
WSP game	[89]	5	Experiential learning	Post	Questionnaires & debrief	Qualitative	Raised awareness Building relationships and trust

in the Appendix A. An overview of the underlying data can be found as a spreadsheet in the Supplementary Materials.

The next subsections present the results of categorization for each research question.

4.1. How Is Learning through Collaborative Serious Games Conceptualized?

The aim of this question was to obtain an overview of how learning through collaborative serious games is conceptualized. Thirteen categories were identified in the analysis. In most cases, publications mentioned the categories explicitly and included relevant citations, while in some cases publications lacked these citations. In the latter case, publications were still assigned to the category if a definition or description was included that matches the category definition. Only the categories social learning and experiential learning covered more than one or two publications.

Table 4. Overview of categorization for Research Question 1 (RQ1) on the conceptualization of learning through collaborative serious games.

Category (Number of Publications)	Category Definition	Assigned to Publications
Social learning (16)	A change in understanding that goes beyond the individual to become situated within wider social units or communities of practice through social interactions between actors within social networks ([11], p. 6)	[56,57,58,61,62,67,68,74,75,77,78,79,80,81,82,83]
Experiential learning (9)	The process whereby knowledge is created through the transformation of experience ([90], p. 38)	[50,51,54,62,66,68,75,88,89]
Other (11)	Other learning conceptualization (see Table A1)	[48,49,52,56,63,64,65,70,72,86,87]
Not conceptualized (10)	Assessing whether or not playing the game led to led to actual changes beyond gameplay	[53,55,59,60,69,71,73,76,84,85]

provides the categories, their definitions, and the publications assigned to these categories. Categories covering only one or two publications are grouped into the other category in Table 4, but can be found in full in Table A1.

4.2. When is Data Collected in the Evaluation of Learning through Collaborative Serious Games?

The aim of this research question was to get an overview of when data is gathered in the assessment of learning through collaborative serious games. Categorizations were chosen based on game sessions as a reference point, with pre, during, post, and post-post data collection.

Table 5. Overview of categorization for RQ4 on when data is collected in the evaluation of collaborative serious games.

Category (Number of Publications)	Category Definition	Assigned to Publications
Pre (22)	Data gathered before game sessions	[48,50,55,58,59,60,61,64,66,69,70,73,75,76,78,80,81,82,85,86,87,88]
During (30)	Data gathered during game sessions	[48,50,51,53,54,55,56,59,60,61,63,64,65,67,68,69,71,72,73,74,75,76,78,79,82,83,84,86,87]
Post (37)	Data gathered immediately after game sessions	[48,49,50,52,53,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,72,73,74,75,76,77,78,79,80,81,82,83,86,87,89]
Post-post (10)	Data gathered well after game sessions; not on the same day	[58,63,69,76,77,78,80,82,83,88]

provides the categories, their definitions, and the publications assigned to these categories. Notably, 37 out of 42 publications gather data immediately after game sessions (post). Twenty-five of these 37 publications combine the post data collection with data collection during game sessions, while 21 of these combine it with a data collection before game sessions (pre) and 15 combine all three. Only 10 publications collected data well after game sessions (post-post).

4.3. What Methods are Used in the Evaluation of Learning through Collaborative Serious Games?

The aim of this research question was to obtain an overview of the methods and techniques that are used to evaluate collaborative serious games.

Table 6. Overview of categorization for RQ5 on the methods used in the evaluation of collaborative serious games.

Category (Number of Publications)	Category Definition	Assigned to Publications
Questionnaires (25)	Written or electronic questionnaires or surveys filled out by participants	[48,50,55,57,58,59,60,61,64,65,66,69,70,72,76,77,78,80,82,83,85,86,87,88,89]
Observations (24)	Observations performed in-situ or based on recordings	[50,51,53,54,55,56,59,60,61,63,64,65,68,69,72,73,74,75,76,78,79,82,84,86]
Debriefings (18)	Formal analysis of the game’s debriefing(s)	[49,50,53,56,57,58,62,67,68,69,73,74,76,78,79,82,86,89]
Interviews (14)	Interviews conducted with individual participants	[53,58,63,64,69,72,74,75,76,77,78,81,82,83]
Data Logging (8)	Data logged during gameplay, such as player decisions	[64,71,72,75,83,84,85,87]
Control group (3)	Data gathered through the use of a control group	[76,83,85]
Perspective mapping (2)	Classifying, interpreting and analysing different perspectives on promising paths for sustainable strategies [91]	[73,75]
Concept maps (1)	Participants drawing structural representation of associations in relation to a topic [92]	[82]
Interaction Analysis (1)	Analysis of human-to-human as well as human-environment interactions [93]	[48]
Social Network Analysis (1)	Analysis of the affordances and hindrances to knowledge co-creation by structuring relationships [94]	[48]
Real-world data (1)	Data gathered from the ‘real world’ research context or environment	[83]

provides the categories used to represent different methods, their definitions, and the publications assigned to these categories. Thirty-one out of the 42 publications used multiple methods—combinations of the five most used methods; questionnaires, observations, debriefings, interviews, and data logging.

4.4. Do Evaluations of Learning through Collaborative Serious Games Use Quantitative, Qualitative or a Combination between Quantitative and Qualitative Data?

The aim of this research question was to see whether the evaluation of collaborative serious games relied on qualitative or quantitative or qualitative and quantitative data.

Table 7. Overview of categorization for RQ6 on use of qualitative and/or quantitative data in the evaluation of collaborative serious games.

Category (Number of Publications)	Category Definition	Assigned to Publications
Qualitative (21)	Evaluations based only on qualitative data	[49,51,52,53,54,56,57,59,62,63,67,69,72,73,74,77,78,79,81,82,89]
Quantitative (5)	Evaluations based only on quantitative data	[60,71,85,87,88]
Qualitative and quantitative (16)	Evaluations based on both qualitative and quantitative data	[48,50,55,58,61,64,65,66,68,70,75,76,80,83,84,86]

provides an overview of which publications based evaluations on either of these three categories. Thirty-seven out of the 42 publications used qualitative data. Sixteen of these 37 publications combined the qualitative data with quantitative data. Qualitative data resulted from questionnaires (open questions), observations (unstructured observations by researchers), debriefings, and interviews. Quantitative data on the other hand resulted from questionnaires (closed questions using Likert scales), observations (structured observations e.g., noting and counting certain behaviour), and data logging (e.g., player choices and group decisions).

4.5. What Are the Learning Effects of Applying Collaborative Serious Games According to Their Evaluations in Relation to Social Learning?

The aim of this research question was to get an overview of the learning outcomes of collaborative serious games. Categories were formulated based on the data extracted from the publications and in relation to the learning typology by Baird et al. [17].

Table 8. Overview of categorization for RQ5 on the learning effects of applying collaborative serious games according to their evaluations in relation to social learning.

Category (Number of Publications)	Category Definition	Assigned to Publications
Increased system understanding (34)	Learning about the topic covered in the game, its complexity and possible trade-offs between decisions (cognitive learning)	[49,50,51,52,53,55,56,57,58,59,60,62,63,64,65,67,68,70,71,72,73,74,76,77,78,80,81,82,83,84,85,86,87,88]
Raised awareness (14)	Learning about the seriousness of an issue (cognitive learning)	[49,51,52,54,58,60,70,72,76,79,80,87,88,89]
Change in views (5)	Observing participants changing their viewpoints, values or paradigms (normative learning)	[60,74,75,78,88]
Understanding other roles and perspectives (13)	Learning about other participants’ roles, mind-sets and points of view (relational learning)	[48,49,56,58,63,64,66,70,74,78,80,82,89]
Building relationships and trust (7)	Enhancing participants’ ability to collaborate by building relationships and enhancing trust (relational learning)	[48,49,52,55,69,80,81]

provides the categories, their definitions and relation to the social learning typology, and the publications assigned to these categories. Thirty-seven out of 42 publications report on cognitive learning outcomes, increased system understanding or raised awareness or both. Only five publications report that playing a collaborative serious game led to a change in views, the category linked to normative learning. Seventeen publications report on relational learning outcomes in the form of understanding other perspectives or building relationships and trust or both.

To formulate and assign categories for this research question, the choice was made to assign publications to categories based only on the text as written in the publications—the data—regardless of whether or not publications described their results as representative for the entire population or as highly or weakly significant. For example, Keijser et al. [65] and Lawrence and Haasnoot [74] both noted results related to the category increased system understanding although Keijser et al. only noted it for those less familiar with the topic, yet both publications were assigned to this category. Making distinctions would otherwise add a selection mechanism susceptible to subjective interpretations of the publications. The results as presented here should therefore only be viewed in relation to the learning outcomes of collaborative serious games can have in general and should not be used to draw direct conclusions on individual publications.

5. Discussion

This section covers the findings on the approaches to assess social learning outcomes of collaborative serious games. The findings are discussed in relation to the three types of learning as defined by Baird et al. [17]; cognitive, relational, and normative learning. In addition, as social learning should lead to collective action, the real-world impact of collaborative serious games—as reported in the reviewed publications—are discussed. The section ends with a reflection on and limitations of the review.

Forty-two publications were included in the review. In relation to the first research question, social learning and experiential learning are the only conceptualizations of learning used by more than two publications. While some publications provide learning conceptualization related to social learning—i.e., collective learning [49,63] and boundary crossing [48]—other publications provide conceptualizations that do not explicitly include the relational learning component of social learning. This is reflected in the review as the latter publications, as well as the publications that do not provide an identifiable conceptualization of learning, mostly report on cognitive learning outcomes. The next subsections discuss the assessments of cognitive, relational, and normative learning outcomes of collaborative serious games in detail.

5.1. Cognitive Learning

Cognitive learning relates to the acquisition of new or the restructuring of existing knowledge. Almost all of the publications in the review—37 out of 42—report on cognitive learning outcomes of collaborative serious games. Two categories were linked to cognitive learning, increased system understanding, and raised awareness, although both can relate to acquiring new knowledge and restructuring existing knowledge. Three common approaches were identified in the assessment: (1) self-reflective questions after game sessions; (2) pre-post measurements of self-reported knowledge on issues; and (3) observed acquisition or restructuring of knowledge.

Firstly, 21 publications describe an approach where participants were asked self-reflective questions in questionnaires, interviews or debriefings after game sessions, sometimes in combination with other methods [49,50,52,55,57,58,62,63,64,67,68,72,73,74,76,77,78,80,82,86,89]. For example, Dionnet et al. [77] assessed their TADLA approach, which aims to facilitate farmers in the collective modernization of their irrigation system, by combining a questionnaire after game sessions with follow-up interviews. They show that their approach helped farmers to acquire a more comprehensive understanding of how the different system components work together. Becu et al. [62] asked self-reflective questions on what local policymakers learned during their structured, round-table debriefing of LottoSim, a game to support social learning on coastal risk prevention measures. According to their evaluation, the participants especially learned about the dynamics of water expansion in flooding events as a result of the game’s integrated simulations. Moreover, the game helped participants to discuss long-term strategies, although participants did not pursue other prevention measures than those already applied in practice. Keijser et al. [65] in turn assessed the ability of the Marine Spatial Planning Challenge to communicate the dynamic and complex interactions between shipping and spatial planning in the maritime environment. Questionnaires after game sessions were applied to assess self-reported learning through closed questions and a five-point Likert scale. Their results show that the game worked well as an introduction to marine spatial planning where participants gain a grasp of its complexities, particularly for those with limited knowledge on the topic.

Secondly, 12 publications analysed self-reported learning through pre-post measurements of participants’ knowledge on the issues addressed in the game [58,59,60,65,70,76,78,80,81,82,87,88]. Questionnaires were the common method in this approach, complemented with other methods, although Salvini et al. [81] and Haug et al. [82] instead used interviews and concept maps respectively. Salvini et al. [81] assessed the effects on social learning of their game, that aims to induce Brazilian farmers to explore agroforestry practices, by conducting interviews with participants. The interviews covered the same questions before and after game sessions and included questions on the farmers’ practices, knowledge of different systems, and opinions on forming cooperatives. The farmers learned about the technical aspects of agroforestry, such as the amount of investments needed for new infrastructure, and its benefits, such as increased productivity, product quality, and profitability. Haug et al. [82] assessed social learning outcomes through a policy game on EU climate change policy and the member states’ sharing arrangements. Participants were asked to draw concept maps [92] before and after game sessions. The analysis of the concept maps was complemented with self-reflective questions asked in questionnaires and interviews after game sessions. Their results show there were some cognitive learning outcomes as certain issues became more central and specific in the post session concept maps—an indication of restructuring of knowledge.

Pre-post measurements of cognitive learning outcomes through questionnaires were based on qualitative assessment of open questions or quantitative assessment of closed questions using Likert scales or a combination of the two. For example, Douven et al. [70] assessed the impact on both raising awareness and upgrading knowledge of playing the Shariva game, which aims to stimulate collaboration among water and related professionals and to resolve transboundary river basin issues. They used questionnaires with both open and closed questions before and after game sessions in this assessment. Their results show that participants acquired knowledge on addressing and resolving transboundary river basin issues. Onencan and Van de Walle [87] assessed whether or not participants gained increased situational awareness—the perception of system elements, comprehension of their meaning and projection of their near future status—from playing WeShareIt Nzoia. To this end, they applied questionnaires based on SART [95], using 10-dimensional subjective ratings, before and after game sessions in combination with in-game performance measures to measure the game’s effects. Their results show an increase in participants’ situational awareness, on all dependent variables, between the pre-test and post-test of local policymakers.

Thirdly, 10 publications assessed cognitive learning outcomes without asking participants, but rather assessed observed acquisition or restructuring of knowledge by analysing observations and data resulting from game sessions [51,53,54,56,61,71,79,83,84,85]. To assess whether or not participants improved their understanding of the complexity of planning for a sustainable future through their game Futura, Antle et al. [54] analysed field notes on participant behaviour and quotes by identifying themes through coding. Their results show that participants did not necessarily acquire new knowledge from playing the game, but did obtain a better sense of the sheer complexity of sustainable development. Sausse et al. [84] developed a game for local stakeholders to explore management of both genetically modified and conventional crops in an area. To analyse how farmers manage the coexistence of both crop types, they analysed data collected from gameplay, such as the simulation results, player actions and maps drawn by the participants, complemented with observations from notes and audio recordings, to understand why players made certain decisions and why certain events occurred. They show that farmers learned about how to technically implement management measures and to better assess the risk of coexistence of both genetically modified and conventional crops in fields next to each other.

5.2. Normative Learning

Normative learning relates to a shift in viewpoints, values or paradigms. Assessing changes in viewpoints and values is particularly challenging as these are difficult to measure and require reflection—and thus time—to occur. How normative learning has been assessed in collaborative serious games is therefore of particular interest. Five publications report on normative learning outcomes [60,74,75,78,88]. In addition, Haug et al. [82] did assess normative learning, but did not find evidence of it taking place. Their assessment was based on pre-post measurements, participants rating the extent they agreed or disagreed on propositions before and after game sessions, in combination with participant interviews after game sessions.

Three of the five publications that report on normative learning outcomes used either pre-post measurements or self-reflective questions after gameplay to assess individual changes in views and values. In the evaluation of Ter’Aguas, a game used to simulate negotiations related to land-use planning in a Brazilian municipality, Ducrot, et al. [78] applied a short-term and long-term learning assessment. In the long-term assessment, through interviews conducted eight months after the game sessions, they found that participants had changed their opinion on their role and position in relation to water management issues. For example, a participant noted that (s)he no longer threw away oil after becoming aware of a water quality issue through playing the game.

Two publications applied pre-post measurements that asked participants to indicate to what extent they agreed with propositions, an approach similar to Haug, et al. [82]. Meya and Eisenack [60] assessed how playing the board game KEEP COOL, which aims to enhance public understanding of climate change science and to raise awareness among public, scientific and environmental organizations, changes participants’ beliefs on international climate politics. To this end, they asked participants to rate their opinions on for example personal responsibility towards climate change mitigation and the confidence in politics to act against climate change on a five-point Likert scale in questionnaires before and after game sessions. Their results show that participants perceived increased responsibility and became more confident in the potential of politics to act. Similarly, Sterman et al. [88] used pre-post measurements of closed question questionnaires to assess changes in attitudes towards climate change from playing the game WORLD CLIMATE, which aims to help participants understand the dynamics and geopolitical implications of climate change. Their results show that participants became more worried about climate change, believed it to be more personally important, and were more likely to urge for immediate action.

The other two publications that report normative learning outcomes cover different versions of the Sustainable Delta Game, which aims to help participants learn about preparing water management strategies for an uncertain future. Lawrence and Haasnoot [74] applied an adapted version of the game in a regional water management project in New Zealand to test how dynamic pathways, pro-active planning to adapt to uncertain future developments, can be adopted in decision-making. Based on observations during game sessions complemented with analyses of the debriefings and interviews with participants after game sessions, they conclude that the game led to group convergence on the necessities to make decisions in uncertain conditions. Van der Wal et al. [75] in turn explicitly assessed such group convergence by measuring changes in group perspectives. They applied perspective mapping, a method rooted in Cultural Theory to classify, interpret, and analyse different individual perspectives [91]. Specifically, participants were asked to select statements they agreed with in relation to a topic during each phase of the game. The method is therefore quite similar to the agreement or disagreement ratings applied by Haug, et al. [82], Meya and Eisenack [60] and Sterman et al. [88]. Group convergence was subsequently assessed by comparing the agreement on statements between participants. The analysis was complemented with analysing recordings of game sessions to understand why perspectives changed and participant interviews after the game to validate that participants found the game and its underlying model credible. Their results show that individual perspectives changed during gameplay, leading to group convergence in most game sessions.

5.3. Relational Learning

Relational learning relates to obtaining an improved understanding of others’ mind-sets, enhanced trust and ability to cooperate. Around half of the publications—17 out of 42—report on relational learning outcomes. Two categories were identified in regard to relational learning outcomes: Understanding other roles and perspectives and building relationships and trust. The first category, reported by 13 publications, relates to participants acquiring an increased understanding on other participants’ roles, mind-sets, and points of view [48,49,56,58,63,64,66,70,74,78,80,82,89]. The second category, reported by seven publications, instead relates to enhancing participants’ ability to collaborate by building relationships and enhancing trust [48,49,52,55,69,80,81]. Both learning outcomes were assessed mostly qualitatively, based on self-reflective questions in questionnaires, interviews and debriefings conducted after game sessions or well after game sessions (post-post). However, building relationships and trust was reported more often when collaborative serious games were used in a community or an existing project where participants addressed a shared problem that directly impacts them.

For example, in the same long-term assessment as described in the previous section by Ducrot et al. [78], participants indicated in the interviews that they learned about the relationships between issues and stakeholders, and obtained a better understanding of other stakeholders’ positions. Similarly, Rumore et al. [58] interviewed participants four to six weeks after game sessions in the assessment of a tailored role-playing game to enhance engagement in the NECAP project, a project that aims to increase the climate change adaptation readiness of coastal communities in New England. In the interviews, participants showed increased empathy and appreciation for other perspectives as a result of playing the game. In particular, participants noted these learning outcomes as a result of taking on another role, looking at the issue from another perspective, and engaging openly with other stakeholders’ point of views. Different from these examples, Carson et al. [80] used questionnaires both directly after game sessions and three months later to assess the effect of the Multi Hazard Tournament on decision-making, social learning, and relationship building in watershed management. The post-game questionnaire showed that learning about other stakeholders’ perspectives was the main learning outcome. Moreover, participants indicated in the follow-up questionnaire that game sessions had led to them pursuing potential projects with other participants.

Exception to this approach is Jean et al. [48], who combined self-reflective questions with interaction and social network analysis to assess how playing Aqua Republica—a game focused on sustainable watershed management—enhanced collaboration and knowledge co-creation. In particular, participants were asked to rate the collaboration in their game session both during—after completed phases—and after the session. Interaction analysis and social network analysis were applied on session recordings of the game phases to analyse the amount of interactions between participants, the quality of interactions and the effect pre-existing relationships had on group dynamics. Their analysis shows that participants shared ideas and forged relationships during game sessions, providing evidence that collaborative serious games can help to enhance connections and develop a mutual understanding between stakeholders.

5.4. Collective Action

Apart from cognitive, relational, and normative learning outcomes, social learning should ultimately lead to collective action [14,15,18]. Arguably, the best indicator of collective action is showing that using a collaborative serious game led to real-world impact. Indeed, a few publications describe how game sessions within a project or community led to real-world initiatives and decisions. For example, the application of the Sustainable Delta Game in a water management project in New Zealand by Lawrence and Haasnoot [74] led to the adoption of adaptive pathways strategy in the project’s climate adaptation strategy. In the interviews conducted before game sessions by Salvini et al. [81], farmers indicated that they had had negative experiences with farmer cooperatives in the past. In interviews after game sessions, farmers individually indicated that they were considering to create a new cooperative together with other participants.

One publication however set out to measure the real-world impact of using a collaborative serious game. Meinzen-Dick et al. [83] used two different follow-ups after game sessions to improve local community understanding of groundwater interrelationships and to stimulate collective governance of groundwater in India: (1) interviewing locals who did not participate in game sessions to measure whether or not there were any spill-over learning effects; and (2) gathering community-level data of different communities in the region, both communities where the collaborative serious game was used and where it was not used, to evaluate whether or not the game led to changes in community practices. In relation to the latter, more communities—statistically significant—where the game was used adopted lessons from the game, leading Meinzen-Dick et al. [83] to conclude that playing the game led to real-world impact.

5.5. Reflection and Limitations

All literature reviews aim to provide the complete overview of relevant publications and the state-of-the-art. There are however some threats and limitations to note in that regard. Firstly, a literature review is always limited in the keywords that it uses in the document search. In this case, a combination of generic and specific keywords was used in relation to sustainability and disciplines dealing with sustainability issues. The generic keywords aimed to make sure collaborative serious games on a variety of sustainability topics were found. The specific keywords were added based on relevant review papers and collaborative serious games known to the researchers. A threat here is that the prior knowledge results in a bias. the generic keywords in combination with snowballing off identified publications however limited any influence of such a bias.

Another threat is that the analysis of the identified publications, and particularly during the study selection and quality assessment steps, contains some subjectivity. After all, the majority of these analysis steps were conducting by a single researcher, the first author of this paper. To limit this threat, a systematic approach to the review was applied, following the guidelines by Kitchenham and Charters [33]. Firstly, the study selection protocol enabled an initial selection that excluded publications clearly out of scope of the review, but included publications that required more scrutiny and in-depth assessment. Secondly, the quality assessment step added a structured way of selecting the publications relevant to include in the review and while answering the quality assessment questions may still have included some subjectivity, this could only affect publications that scored just short of the threshold and were thus excluded. These publications were carefully looked at again after completing the entire quality assessment to check whether or not exclusion was a correct conclusion. None of these publications were subsequently added to the review.

6. Conclusions

Serious games are increasingly explored as a method to establish social learning on sustainable natural resources management and urban planning. The aim of this review was to answer the main research question: What is the current state of the art of the different methods and procedures used to assess social learning outcomes of collaborative serious games? A systematic literature review was conducted, which reviewed 42 publications that applied serious gaming to collaboratively explore sustainable management strategies of land and natural resources. The publications were analysed in order to determine how three types of learning outcomes—cognitive, normative, and relational learning—were assessed. To this end, data was extracted from the publications in relation to five research questions and were categorized following an inductive approach. The categorizations made it possible to identify approaches used in relation to assessing cognitive, normative, and relational learning outcomes of collaborative serious games.

Most evaluations of collaborative serious games focus on assessing cognitive learning, which relates to the acquisition of new or the restructuring of existing knowledge, for example whether or not participants obtain a better understanding of the complexities of climate change. Few evaluations focus explicitly on normative learning, which relates to a shift in viewpoints, values or paradigms, for example on an individual’s feeling of responsibility towards climate change mitigation. That few evaluations assess normative learning is not surprising as it generally requires experience, reflection, and time before a person would shift viewpoints, values or paradigms. Nevertheless, the reviewed publications do show that collaborative serious games can lead to normative learning. About half of the reviewed publications report on relational learning outcomes, which relates to obtaining an improved understanding of others’ mind-sets, enhanced trust, and ability to cooperate. That only half assess this learning type seems to relate to whether or not relational learning is seen as part of the conceptualization of learning and whether or not the collaborative serious game is used in an existing project or community.

Table 9. Overview of approaches used to assess cognitive, relational and normative learning through collaborative serious games.

Type	Assessment Approach	Data Collection	Methods
Cognitive learning	Self-reflective questions	Post-game	Questionnaires (open or closed questions); interviews; debriefings
	Knowledge measurements	Pre- and post-game	Questionnaires (open or closed questions); interviews; concept maps
	Observed knowledge acquisition	In-game	Observations (structured or unstructured, in-situ or from recordings); Analysing data logged during gameplay
Normative learning	Self-reflective questions	Post- and post-post game	Interviews
	Shifts in opinion measurements	Pre- and post-game	Questionnaires (closed questions); perspective mapping
Relational learning	Self-reflective questions	Post- and post-post game	Questionnaires (open or closed question); interviews; debriefings
	Group dynamics analysis	In-game	Questionnaires (closed questions) & interaction analysis and social network analysis

provides an overview of the approaches used in the assessment of the three learning outcomes, including the common methods and procedures used.

The review shows that three common approaches are used to assess cognitive learning outcomes. Firstly, asking participants self-reflective questions such as what they learned after game sessions, either in a qualitative assessment through open questions or a quantitative assessment using closed questions. Secondly, applying measurements of participants’ knowledge before and after game sessions, also based on self-reporting by asking open or closed questions to participants. Thirdly, assessing observed acquisition or restructuring of knowledge by analysing observations, either qualitatively or quantitatively, and in-game collected data.

Two approaches are used to assess normative learning outcomes. Firstly, asking participants to rate their views on propositions in pre–post measurements, optionally complemented with such ratings during game sessions, through closed questions questionnaires. Next to assessing individual shifts in viewpoints, pre–post measured individual shifts can be compared within the group of participants to determine whether or not convergence of group opinion occurs. Secondly, interviews with participants well after game sessions—weeks to months—can be used to reflect on the impact of collaborative serious games on normative learning.

Two approaches are also used to assess relational learning outcomes. Firstly, asking participants self-reflective questions following game sessions—the approach used in all but one publication reviewed—in order to determine whether or not participants gain a better understanding of other stakeholders. Secondly, asking participants to self-report the level of collaboration in combination with analysing participants’ interactions during game sessions. This approach quantitatively analyses the group dynamics during game sessions in order to determine whether or not relationships are enhanced and a mutual understanding is developed.

Of course, evaluation approaches are not exclusive from one another. Many of the reviewed publications in fact combine approaches or use multiple methods to assess learning outcomes. A multi-method evaluation approach is also recognized to fit well with serious gaming in general. However, explicitly assessing cognitive, normative, and relational learning may help to separate the nature of learning through collaborative serious games. The overview provided in this paper may therefore serve as a reference for scholars designing and evaluating collaborative serious games, particularly those that address sustainability problems, but may also be useful in relation to serious games that aim to collaboratively address problems in complex systems in general.