The Use of Theory in Designing a Serious Game
for the Reduction of Cognitive Biases
Meg Barton, Carl Symborski, Mary Quinn, Carey K. Morewedge, Karim S.
Kassam, and James H. Korris
Transactions of the Digital Games Research Association
2016, Vol. 2, No. 3, pp. 61-87
ISSN 2328-9422
http://todigra.org
TEXT: Licensed under Creative Commons Attribution (CC BY-NC- ND
2.5) http://creativecommons.org/licenses/by-nc- nd/2.5/
IMAGES: All images appearing in this work are property of the respective
copyright owners, and are not released into the Creative Commons. The
respective owners reserve all rights.
ABSTRACT
In the current study, a serious game was developed to address a training
challenge: teaching players to recognize and mitigate their cognitive
biases. Cognitive biases, which are human tendencies to commit systematic errors in thinking that lead to irrational judgments, are deeply
ingrained and difficult to alter. This paper describes the theory-based
approach we employed to create a game for the mitigation of cognitive
biases – a challenging and abstract training topic. A cognitive bias
framework that relates the target cognitive biases, their causes, and effective bias mitigation techniques was developed and incorporated into
the game design. The resultant serious game, titled Missing: The Final
61
62 ToDiGRA
Secret (hereinafter: Missing), pairs the most promising mitigation strategies with the primary causes of the targeted cognitive biases and incorporates them into game-play. Further, we present preliminary results from a
game efficacy evaluation suggesting that Missing is an effective tool for
training cognitive bias recognition and mitigation.
Keywords
serious games, training, game design, cognitive bias, critical thinking
INTRODUCTION
The idea that digital games have valuable uses beyond entertainment
alone – such as training, education, and promoting social change – is
well-accepted. Games designed for purposes beyond pure entertainment
are known as “serious games” (Stapleton 2004). As the potential to
use digital games as vehicles for training has become apparent, serious
games have been deployed across diverse fields for diverse communities
of players.
Many of the serious games that have been developed in recent years
are designed to teach the player about a particular topic or a concrete
skill set. In educational settings, serious games have been incorporated
into lesson plans for a variety of academic subjects, such as history
(e.g., Stories from the History of Czechoslovakia; Šisler et al. 2012),
cell biology (e.g., Virtual Cell; McLean et al. 2001), and computer
programming (e.g., a Real-Time Strategy game developed by Muratet
et al. 2009). Other serious games have been developed in a variety
of domains (e.g., health, social activism) to educate players about a
topic with the intention of promoting behavior change; examples include
Re-Mission, designed to educate cancer patients about the disease and
thereby enhance treatment adherence and side effect management (Beale
et al. 2007), and Green My Place, designed to teach energy awareness
and increase energy saving behavior in players (Cowley et al. 2011).
Some serious games teach manual skills that can then be practiced in
Reduction of Cognitive Biases 63
the virtual environment. Examples include Skillmaster, which teaches
mechanics to assemble a car power generator (Woll et al. 2011), and a
serious game that familiarizes orthopedic surgical residents with the knee
replacement surgery procedure (Sabri et al. 2010).
While many serious games are designed to train concrete concepts or
skills, as in the examples above, others promote more abstract learning
outcomes; these games are far rarer than games designed to help players
acquire basic knowledge on a particular topic (Connolly et al. 2012).
For instance, Operation ARIES! (Millis et al. 2011) and Operation ARA
(Halpern et al. 2012) train players’ critical thinking and scientific inquiry
skills to be used when evaluating scientific research. Another example of
a serious game geared toward abstract learning outcomes is a mini-game
developed by Grappiolo et al. (2011) to train conflict resolution skills. A
third example, DREAD-ED, is a multiplayer game developed by Haferkamp et al. (2011) to teach communication and decision-making skills to
emergency management personnel.
In the current study, we sought to develop and evaluate a serious game
targeting a challenging and abstract training problem: the mitigation of
cognitive biases. Cognitive biases are the systematic errors and illogical thought processes to which all humans are prone (Kahneman 2011).
Though many social institutions (e.g., the legal system, the medical field,
the business world, political spheres) rely on the ability of human decision makers to render balanced and rational judgments, cognitive biases
are pervasive and notoriously difficult to mitigate (Croskerry, Singhal,
and Mamede 2013; Kahneman 2011). This paper is a follow-on to a previously published paper, which presented the game design and efficacy
evaluation of a serious game for the mitigation of three cognitive biases:
confirmation bias, the fundamental attribution error, and the bias blind
spot (Symborski et al. 2014). For the current study, we sought to replicate
these results by developing a serious game to target three different cognitive biases: anchoring bias, the representativeness heuristic, and projection bias (defined below).
64 ToDiGRA
In this paper, we describe the game design for Missing: The Final Secret,
a serious game to teach the recognition and mitigation of anchoring bias,
the representativeness heuristic, and projection bias. Given that cognitive biases are deeply ingrained and challenging to mitigate, we sought
to maximize the educational potential of the game by incorporating current research and theory on the causes and mitigations of cognitive biases
into the game design. The design of Missing integrates a “cognitive bias
framework,” based on the theory of dual-process systems of reasoning
(Kahneman 2011), that relates the target cognitive biases, their causes,
and effective bias mitigation techniques. In addition, we present preliminary results from an experiment to assess the game’s efficacy for training
the recognition and mitigation of the three biases.
GAME DESIGN
The serious game Missing: The Final Secret is a cross between an interactive storytelling game and an adventure game. Over the course of three
episodes, players interact with non-player characters (NPCs) and complete activities as they work toward solving the mystery driving the plot
of the story. In each game episode, the player is exposed to bias-invoking situations referred to as “bias vignettes,” during which the cognitive
biases demonstrated by the player are measured. After the conclusion
of each episode, an After Action Review (AAR) provides instruction on
each of the three target biases, offers feedback on game performance,
and provides practice examples for each bias.
In the following sections, the design of the in-game bias vignettes and
AARs is described with respect to a cognitive bias framework derived
from the literature. An outline of the structure of Missing, with specific
details as to how the cognitive bias framework was integrated into the
game elements, is also provided.
Missing and the Cognitive Bias Framework
The foundation of our game design is based on current research on the
Reduction of Cognitive Biases 65
cognitive processes that lead people to commit cognitive biases, based
on the theory of dual-process systems of reasoning. This theory asserts
that there are two systems of reasoning that we commonly employ when
making judgments: System 1 reasoning is characterized by automatic,
intuitive, and reactive thinking, while System 2 reasoning is characterized by logical reasoning and rule-governed thinking (Evans 2007;
Forster and Liberman 2007; Kahneman 2011; Morewedge and Kahneman 2010). According to the theory, cognitive biases commonly arise
when the automatic and intuitive processes of System 1 reasoning lead us
to generate faulty conclusions, which the logical, rule-based processes of
System 2 reasoning fail to identify and mitigate (Morewedge and Kahneman 2010).
To guide our game design process, we created a cognitive bias framework (see Figure 1, below) that defines the relationship between the
three cognitive biases being targeted, their theoretical causes (i.e., automatic System 1 reasoning processes), and mitigation approaches (i.e.,
logical System 2 reasoning processes). This framework allowed us to
design Missing such that players would be exposed to the System 1
causes of the target biases during game play, which would then be connected to the most promising System 2 mitigation strategies for each of
those causes in the AARs
66 ToDiGRA
Figure 1: Cognitive bias framework.
Given that there is some overlap with regard to common System 1 causes
and potential System 2 mitigation strategies for the three biases (see Figure 1), we were able to develop an efficient game that targets the origins of multiple biases at their common source and allows players to
generalize their learning across multiple biases. The following sections
define the three target biases and describe the theory-based causes, the
theory-based mitigation strategies, and the general structure of the game
vignettes for each bias.
Anchoring Bias
Anchoring bias is the tendency to place excessive weight, or “anchor,”
on a single piece of information when making a judgment or decision
(Kahneman 2011; Sapadin 2013). For example, in one study, even experienced real estate agents overestimated the value of a home after exposure to an overly high asking price (Northcraft and Neal 1987).
Theory-Based Cause: The literature suggests that there are two types
of anchoring with two different causes, depending on how the anchor
was provided: externally or internally (Epley and Gilovich 2001). When
we consider externally provided anchors (i.e., given to us through an
Reduction of Cognitive Biases 67
external source such as a price tag, another person, etc.), the selective
accessibility of information consistent with the anchor is increased.
Anchor-consistent information is then given excessive weight when we
make subsequent judgments, leading to bias (Mussweiler and Strack
2000). Internally generated anchors impact our judgment when we have
prior knowledge on a topic; for example, when asked to guess the freezing point of vodka, most people consciously anchor on the freezing
point of water (32°F/0°C) and recognize that vodka freezes at a lower
temperature than water. However, when searching for the correct answer,
most people do not adjust far enough away from the anchor toward the
correct answer (Epley and Gilovich 2001).
Theory-Based Mitigation: The effects of both selective accessibility and
conscious anchoring are reduced when we are asked to think of reasons
for rejecting the anchor as an estimate and to explicitly consider alternatives. Deliberately considering alternative information to the anchor
helps to reduce the biasing effects of the anchor (Chapman and Johnson
1999). In addition, the conscious anchoring that occurs with internally
generated anchors can be reduced by prompting the logical processes of
System 2 reasoning (Epley and Gilovich 2006; Simmons, LeBoeuf, and
Nelson 2010).
Anchoring Bias Game Vignettes: This bias is elicited during the game by
asking the player questions related to the game narrative and requiring
the player to respond with numerical answers. For externally provided
anchors, an anchor value is provided explicitly in the question or as part
of the game dialog immediately preceding the question. The extent to
which the player was influenced by the anchor is determined by comparing the player’s answer to the provided anchor. As a lesson for mitigation, players are guided to recognize when an anchor is present, to
determine the direction in which to adjust their answer away from the
anchor, and to adjust further away from the anchor than they think
they should. Players are also guided to consider additional information
beyond the anchor that might be relevant when making their judgment.
68 ToDiGRA
Representativeness Heuristic
The representativeness heuristic is not a unitary construct; rather, it is
comprised of multiple bias facets (Tversky and Kahneman 1974). In
essence, the representativeness heuristic boils down to relying on appearances or what “seems right” to make judgments, while neglecting to take
relevant principles of statistics and probability into account (Kahneman
2011). Missing covers the five facets of the representativeness heuristic
described in Table 1.
Table 1: Facets of the representativeness heuristic covered in Missing.
Sources: Kahneman 2011; Tversky and Kahneman 1974
Theory-Based Cause: Research suggests that the representativeness
heuristic is the result of the substitution of a similarity judgment (“this
seems right”) for a probability judgment, known as attribute substitution (Kahneman and Frederick 2002). This often entails a neglect of
base rates or probability information (Kahneman and Tversky 1972).
Theory-Based Mitigation: Prompting logical, rule-based System 2 reasoning can help to reduce attribute substitution and base rate neglect and,
therefore, is effective in mitigating errors in thinking resulting from the
representativeness heuristic. In addition, research indicates that people
with formal statistical training are less affected by the representativeness
heuristic; providing statistical training should help to increase attentiveness to base rates and probability information (Tversky and Kahneman 2002).
Reduction of Cognitive Biases 69
Representativeness Heuristic Game Vignettes: The game elicits the representativeness heuristic by having players make judgments about ingame characters and probabilities. For example, in one vignette, the
player is asked to speculate about the presumed nefarious activities of
the nemesis in the game, Arthur Flaherty, by selecting which activity(ies)
Arthur is likely involved in from a list provided. Players judging it more
likely that Arthur is involved in a conjunction of two activities than in
any one of the constituent activities will have committed the conjunction fallacy. Additional vignettes cover the other four of the five aforementioned representativeness heuristic facets (see Table 1). In the AAR,
basic statistical instruction about the conjunction of multiple events, base
rates, randomness of events, and sample sizes is provided.
Projection Bias
Projection bias is the tendency to overestimate the extent to which others
share our own emotional states, characteristics, thoughts, and values
(Epley et al. 2004). Missing covers two primary forms of projection bias:
the false consensus effect (Ross, Greene, and House 1977) and attributive similarity (Human and Biesanz 2011). The false consensus effect
occurs when we assume that more people share our beliefs than actually
do (Ross, Greene, and House 1977). Attributive similarity occurs when
we overestimate the extent to which others are likely to share our traits
or characteristics (Human and Biesanz 2011).
Theory-Based Cause: Projection bias stems from consciously anchoring on our own emotional states, thoughts, and values when evaluating
the emotional states, thoughts, and values of others (Epley et al. 2004).
This leads to an increase in the selective accessibility of information
consistent with our own perspective and the overweighting of that information when making a judgment about others’ views (Epley et al. 2004).
Theory-Based Mitigation: Explicit consideration of alternatives – in
other words, deliberately considering alternative points of view or
attempting to put oneself “in someone else’s shoes” – can help to reduce
70 ToDiGRA
projection bias, along with activating logical System 2 reasoning
processes (Epley, Morewedge, and Keysar 2004; Van Boven and
Loewenstein 2003).
Projection Bias Game Vignettes: The false consensus effect is elicited ingame by asking the player to indicate his/her view on a particular topic
(e.g., prefers cats versus prefers dogs). The player is then asked to estimate the percentage of people who agree with his/her opinion. Overestimating this percentage suggests the presence of projection bias. The
attributive similarity bias vignettes share a similar format, except that
players respond to questions on a continuous Likert scale and then estimate how the average person would respond to the question using the
same scale. Answering that the average person would answer identically
or very similarly to oneself suggests the presence of projection bias.
The Game Design of Missing and Theoretical Underpinnings
The bias framework and theory-based mitigation techniques are incorporated directly into game-play through the four major instructive phases
of the game: cognitive bias elicitation, bias measurement, participant
feedback, and cognitive reinforcement. These steps are repeated multiple
times in a given episode for each of the three game episodes, offering
repeat learning experiences for the target biases. Table 2 provides an
overview of each of the four instructive phases in Missing and specifies
during which game segment the phase occurs (episode versus AAR).
Table 2: The four major instructive phases in Missing.
Reduction of Cognitive Biases 71
In the sub-sections that follow, the structure of Missing will be described
with respect to the bias framework and the four instructional phases
defined in Table 2.
Opening Video
The player is drawn into the game through a brief video that presents the
highlights of the first game in the Missing series (Symborski et al. 2014).
Through a conversation between two unseen characters and a montage
of animations, the player is provided with a recapitulation of the first
game’s adventures. The player is then informed that, following these
adventures, (s)he and Terry have joined forces to create a hot news blog:
Manhattan Azimuth. As the game opens, Terry and the player are celebrating the 1,000,000th hit on their blog.
In the opening video and throughout the game, an interactive narrative
storyline is a key element of Missing (see Figure 2). A well-crafted storyline that draws the player into the game promotes player engagement
and feelings of immersion (“presence”) while maximizing the entertainment value of the game (McDaniel, Fiore, and Nicholson 2010).
Figure 2: Terry finds herself in trouble with the law (left); Terry and player sneak
into a warehouse (right)
Episodes
Missing is composed of three episodes, each of which is followed by an
AAR (described in the following section). The episodes are sequential
72 ToDiGRA
and present the story to the player, from exposition to denouement. Each
episode is punctuated with three to four bias vignettes that are woven
into the plot of the story. In order to illustrate the format of an episode of
Missing, several highlights of Episode 1 are described below.
In the opening of Episode 1, Terry and the player are celebrating the
1,000,000th hit on their news blog, Manhattan Azimuth, and plotting
how best to expand their success. In the context of this conversation, the
first bias vignette arises ([Phase 1] Cognitive bias elicitation). Terry
is considering the relative merits of developing a Facebook application
to help market their news blog. Terry remarks that Facebook apps seem
to be popular and that she spends around 30 hours per month browsing
Facebook on her cell phone. In order to gauge whether or not a Manhattan Azimuth Facebook app would be a valuable investment of time
and money, Terry asks the player to estimate the following: first, whether
the average mobile Facebook user spends more or less than 30 hours
per month looking at Facebook on his/her phone, and second, about how
many hours the average mobile Facebook user spends browsing Facebook on his/her phone (see Figure 3).
This is an example of an anchoring bias vignette. In this case, the externally provided anchor is Terry’s 30-hour estimate of the time she spends
on mobile Facebook each month. The player is first prompted to answer
whether (s)he thinks that the average mobile Facebook user spends more
or less than 30 hours a month on the app, increasing the selective accessibility of the anchor to the player and establishing which direction
the player will be adjusting from the anchor (i.e., higher or lower than
the anchor). The player is then asked to estimate the number of hours
the average user spends on mobile Facebook in a month. By analyzing
the player’s response relative to the anchor value (30 hours per month)
and to the correct answer (11 hours per month; Sternberg 2013), the
player’s anchoring bias can be assessed by the game mechanics ([Phase
2] Bias measurement). The closer the player’s answer has been “pulled”
toward the anchor of 30 hours and away from the correct answer, the
more anchoring bias the player has demonstrated; the closer the player’s
Reduction of Cognitive Biases 73
answer is to the true value of 11 hours, the less anchoring bias the player
has demonstrated. The player’s measured bias serves as the basis for the
feedback that the player will receive during the AAR for Episode 1.
Figure 3: Player estimates the number of hours the average user spends on mobile
Facebook in a month
Episode 1 also includes two bias vignettes for the representativeness
heuristic. In one of these vignettes, Terry and the player discuss the presumed nefarious activities of their nemesis, Arthur Flaherty. In the other
vignette, Mike, the quirky building superintendent, draws the player
into a game he likes to play to pass the time: pondering the nature of
those who live in the building. This time, Mike is wondering which
gym the new building tenant will join. There are three fitness options
within range of the apartment building: Rocky’s Gym is the cheap, nofrills option within a five-minute walk from the apartment; Entropy is
an upscale, spa-like facility around 10 minutes from the building; and
there’s always the option of staying at home on the couch watching television. According to Mike, 70% of the tenants go to Rocky’s Gym, 10%
go to Entropy, and 20% stay home and watch TV. Mike has observed that
Mary, the new tenant, is classy, well-dressed, and an avid museum-goer,
74 ToDiGRA
and queries the player as to which fitness option (s)he expects Mary to
select ([Phase 1] Cognitive bias elicitation).
If the player is engaging in attribute substitution, (s)he might rely on
the description of Mary’s qualities (classy, cultured, stylish) to make this
judgment, while neglecting the base rate information provided. The
player’s bias is measured as a function of the fitness program the player
assumes that Mary would select: Rocky’s, Entropy, or the couch ([Phase
2] Bias measurement; see Figure 4). While the player might assume
that Mary will sign up for Entropy, the upscale gym option that seems
more congruent with her personality, this choice neglects to consider the
base rate information that Mike provided: 70% of the tenants exercise
at Rocky’s, whereas only 10% go to Entropy; thus, Mary is more likely
to sign up at Rocky’s, from a probabilistic perspective. In the AAR for
Episode 1, the player is given feedback to this effect.
Finally, Episode 1 contains a projection bias vignette. In pursuit of generating content for their news blog, Terry and the player consider what
event they should cover in the upcoming week. Terry directs the player
to a set of invitations for different charity events (see Figure 4), one of
which is a fundraiser for “Friends of a Green New York.” After some
deliberation, Terry asserts that she is leaning toward this invitation, and
asks the player whether (s)he approves or disapproves of the government’s spending on parks and recreation. After the player has answered,
Terry asks the player to estimate what percentage of Americans agree
with the player’s opinion on the issue ([Phase 1] Cognitive bias elicitation).
This vignette is designed to elicit the false consensus effect. Players
impacted by projection bias are likely to consciously anchor on their
own opinion on government spending on parks and recreation in this situation, increasing the selective accessibility of information consistent
with their views. To assess player bias, the player’s estimate of the percentage of others who share their views is compared to the actual percentage of Americans who also approve or disapprove of government
Reduction of Cognitive Biases 75
spending on parks and recreation, based on polling data ([Phase 2] Bias
measurement). Overestimating the percentage of others who share one’s
views on an issue is indicative of projection bias, which is discussed with
the player during the feedback sections of the AARs.
Figure 4: Player guesses which gym Mary attends (left); Player inspects
invitations to charity functions (right)
After Action Reviews (AARs)
The AARs are composed of three main parts: defining the biases or bias
facets, providing the player with feedback, and reinforcing the player’s
learning with practice examples. Thus, for each bias vignette in the
game, there is a corresponding segment in the AAR during which the
bias is defined (if it has not been defined already), feedback specific to
that bias vignette is given, and the player is given practice examples of a
similar format to the bias vignette.
During AAR 1, before players are provided with feedback or practice
examples for any of the biases, each bias (anchoring, representativeness,
and projection) is defined in a brief, two- to three-minute video of a
subject matter expert explaining the bias in simple terms. Further, brief
explanations of each of the facets of the representativeness heuristic
appear throughout the AARs for Episodes 1, 2, and 3, before feedback
for the bias vignettes corresponding to those facets is given.
Along with basic definitions of the biases, the AARs provide feedback
for each of the bias vignettes in the game. For a given bias vignette,
the AAR segment begins with a flashback video clip that reminds the
76 ToDiGRA
player of the bias-eliciting situation in the game. The player’s response
to the situation is then highlighted and feedback is given as to whether or
not his/her answer demonstrated bias ([Phase 3] Participant feedback).
Research in the field of education has consistently found that feedback
is beneficial for learning (Hattie and Timperley 2007). Based on recommendations from the literature, the feedback provided by the game
was created with the following features: it is tailored to the player’s performance (i.e., it specifically addresses the player’s answer to the question, whether correct or incorrect); it is comprehensive yet brief, to avoid
placing excessive cognitive load on the player and to remain manageable; and it is designed to address the player’s misperceptions and incorrect answers, while respecting the player’s self-esteem and attempting
to avoid making him/her feel threatened or defensive (Hattie and Timperley 2007; Race n.d.). Specifically, while the feedback attributes unbiased/correct answers to the player (“Great job! You avoided bias…”),
biased/incorrect answers are separated from the player and addressed
tactfully (“Your answer [versus the player him/herself] might [versus
a more definitive and condemning verb] have demonstrated bias…”).
After providing feedback to the player, bias-specific mitigation strategies
(i.e., explicit consideration of alternatives, prompting System 2 reasoning, and statistical training) are provided.
After the player has received feedback and instruction on mitigation
strategies, one or more practice examples is/are presented. The practice
examples mirror the format of the in-game bias vignettes, and players
are given tailored feedback immediately after answering ([Phase 4]
Cognitive reinforcement). The importance of practice for learning and
improved performance has long been recognized (e.g., Ericsson,
Krampe, and Tesch-Romer 1993); further, presenting multiple analogous
examples allows the learner to generate a problem-solving schema that
is more likely to generalize to other contexts in the future (Gick and
Holyoak 1980).
Reduction of Cognitive Biases 77
Method and Results of Game Efficacy Evaluation
In order to evaluate whether Missing was effective at training the recognition and mitigation of cognitive biases, we conducted a test campaign
wherein participants were assigned either to play the game or to watch
an educational control video on cognitive bias. Assigning participants to
a game or control condition allowed us to compare the knowledge transfer from the game relative to a more traditional method of teaching about
cognitive biases (i.e., an educational video). The study consisted of a
pretest, exposure to the game or the video, an immediate posttest, and
a follow-up test 12 weeks later to assess longitudinal retention of bias
knowledge and mitigation. Use of a 12-week longitudinal period was an
increase from the eight-week period used in our previous study (Symborski et al. 2014), to further investigate knowledge retention over time.
Method
Participants
Participants were recruited via the Center for Behavioral Decision
Research (CBDR) website, operated by Carnegie Mellon University in
Pittsburgh, Pennsylvania. Recruitment was open to the general public;
however, the sample consisted primarily of students from nearby universities. A total of 238 participants were recruited; 156 were assigned to the
game/experimental condition and 82 were assigned to the video/control
condition. Of these, 126 of the participants in the game condition (80.8%
retention) and 66 of the participants in the video condition (80.5% retention) completed the follow-up test 12 weeks later.
Materials and Procedure
A standardized measure of cognitive bias knowledge and mitigation
was developed jointly by a MITRE-led team consisting of researchers
from Leidos, Applied Research Associates (ARA), and the University
of Albany. The bias assessment instrument was composed of two sections, one on the recognition and discrimination of the three target biases
78 ToDiGRA
and one on bias mitigation. Three different forms were developed to be
administered as pre-, post-, or follow-up tests in counterbalanced order.
Materials for the experiment included the bias assessment instrument;
copies of Missing, loaded onto computers that met the minimum specification to run the game (Intel® Core™ i7 processor; Windows 7 operating system; 4 GB Dual Channel DDR3 SDRAM 1333 MHz or greater
memory; 1 GB on board DDR3 RAM video card); and the control video.
The control video was a professionally produced, engaging video that
taught recognition and mitigation of anchoring bias, the representativeness heuristic, and projection bias.
During testing, participants would arrive at the CBDR lab site. They
would then be randomly assigned to the game or video condition and to
a pretest form. After taking the pretest, the participant would either play
the game or watch the video, which was followed by completion of the
posttest. Participants received an email with a personalized link to the
follow-up test 12 weeks after the study date and were given a week to
complete it.
Results
As in our previous study (Symborski et al. 2014), our analysis of the
data was guided by three primary research questions. First, was the game
effective in teaching the recognition of and discrimination between the
three target biases, and was this training effect retained over time? Second, was the game effective in training the mitigation of cognitive biases,
and was this training effect retained over time? Finally, was the game a
more effective training tool than an educational video?
We evaluated these research questions using t-tests to assess the statistical significance of the differences in the pretest to posttest scores, pretest
to follow-up test scores, and results for the game versus the video at
posttest and at follow-up. In addition, we report Cohen’s d as a measure
of effect size (Cohen 1992). As a standard guideline for interpretation
Reduction of Cognitive Biases 79
of effect sizes, Cohen (1992) suggested that d = 0.2 could be considered
small, d = 0.5 could be considered medium, and d = 0.8 could be considered a large effect size.
Game Efficacy: Training Recognition and Discrimination of Target Biases
Participants’ ability to accurately match the selected biases to their definitions (recognition) and to differentiate the biases from one another when
given a scenario and asked which bias the scenario represented (discrimination) was measured via the bias assessment instrument at pretest, at
immediate posttest, and at 12-week follow-up. There was a statistically
significant improvement in participants’ bias recognition and discrimination from pretest to posttest (t(155) = 17.75, p < .001). This improvement
in recognition and discrimination of biases at immediate posttest corresponded to a large effect size (d = 1.43). Following the 12-week longitudinal period, the improvement over pretest scores at follow-up remained
statistically significant (t(125) = 7.33, p < .001); though the effect size
decreased, as would be expected, it remained medium to large in magnitude (d = 0.66).
Game Efficacy: Training Mitigation of Target Biases
Participants’ ability to mitigate (i.e., avoid committing) the target biases
at pretest, at immediate posttest, and at 12-week follow-up was assessed.
Participants demonstrated statistically significant improvement in bias
mitigation capability (i.e., committed biases less often) from pretest to
immediate posttest (t(155) = 21.76, p < .001). The effect size, d = 1.75,
exceeded the threshold for a large effect size. After the 12-week longitudinal period, improvement in bias mitigation capability remained statistically significant and the effect size remained large (t(125) = 13.04, p <
.001; d = 1.17).
Game vs. Educational Video Comparison
Finally, the game’s efficacy was compared to that of an educational video
for training the mitigation of cognitive biases. At immediate posttest,
80 ToDiGRA
Missing was significantly more effective for teaching the mitigation of
cognitive biases than the educational control video (t(235) = 3.67, p
< .001; d = 0.24). After the 12-week longitudinal period, this effect
remained marginally significant (t(189) = 1.91, p = .057; d = 0.14).
Taken together, these results suggest that Missing: The Final Secret is
an effective teaching tool for the recognition and mitigation of cognitive
biases. Further, the results of this version of the game are similar to our
previous version that addressed three different cognitive biases (Symborski et al. 2014). Both games had a positive training effect, immediately after training and longitudinally. In addition, both Missing games
outperformed educational control videos in training cognitive bias mitigation, which lends support to the idea that serious games may be
more effective for training than standard approaches such as educational
videos or lectures.
Conclusion
Cognitive biases arise from human instincts and intuitions that are
deeply ingrained and difficult to alter. The challenge faced by the present
study was to utilize a serious game approach to make people aware of
cognitive biases and to provide training to reduce the occurrence of these
biases; in other words, to create a change in thinking, actions, and attitudes of the game player – a common challenge when creating serious
“games for change” to leverage game mechanics for social benefit.
Our approach to creating the game Missing was to guide the game design
using current literature on cognitive biases, which provided theoretical
bias causes and mitigation strategies. As the foundation of our game
design, we paired the most promising mitigation strategies with the primary causes of the cognitive biases to create the described bias framework. These concepts were incorporated into specific game mechanics
and story narrative. The game episodes were designed to expose the
player to the causes of the biases, and then to teach mitigation strategies
during the feedback sections between game episodes. This gives players
Reduction of Cognitive Biases 81
a chance to absorb these strategies and practice them in subsequent game
episodes.
Three cognitive biases were selected as targets for mitigation: anchoring
bias, the representativeness heuristic, and projection bias. An experiment
was conducted and the effects of Missing on bias recognition/discrimination and mitigation were measured. The immediate effect of Missing on
bias knowledge was large at d = 1.43. The immediate effect of Missing
on bias mitigation was also positive at d = 1.75. Both results are encouraging. Some decay in learning results was expected and observed regarding the effects of Missing on both bias knowledge and bias mitigation
when measured 12 weeks after game play. Bias knowledge improvement
reduced to d = 0.66 and mitigation improvement reduced to d = 1.17.
However, these results remain statistically significant compared to baseline scores and suggests that the knowledge gained by playing Missing is
internalized and retained.
In conclusion, the training results described above suggest that using relevant theory to guide the game design process is a promising approach
for building serious games that teach abstract topics. Future research
may seek to validate this design approach across diverse topic areas for
diverse communities of players.
ACKNOWLEDGMENTS
This work was supported by the Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory (AFRL)
contract number FA8650-11-C-7175. The views and conclusions contained herein are those of the authors and should not be interpreted
as necessarily representing the official policies or endorsements, either
expressed or implied, of IARPA, AFRL, or the U.S. Government.
BIBLIOGRAPHY
Beale, I.L., Kato, P.M., Marin-Bowling, V.M., Guthrie, N., and Cole,
82 ToDiGRA
S.W. “Improvement in cancer-related knowledge following use of a psychoeducational video game for adolescents and young adults with cancer,” in Journal of Adolescent Health vol. 41, no. 3 (September 2007),
pp. 263-270. doi: 10.1016/j.jadohealth.2007.04.006.
Chapman, G.B., and Johnson, E.J. “Anchoring, activation, and the construction of values,” in Organizational Behavior and Human Decision
Processes vol. 79 (1999), pp. 115–153.
Cohen, J. “A power primer,” in Psychological Bulletin vol. 112, no. 1
(1992), pp. 155-159. doi: 10.1037/0033-2909.112.1.155.
Connolly, T., Stansfield, M., and Hainey, T. “Development of a general
framework for evaluating games-based learning,” in Proceedings of the
2nd European Conference on Games-Based Learning (Barcelona, Spain,
2008), pp. 105-114.
Cowley, B., Moutinho, J.L., Bateman, C., and Oliveira, A. “Learning
principles and interaction design for Green My Place: A massively multiplayer serious game,” in Entertainment Computing vol. 2, no. 2 (2011),
pp. 103-113. doi: 10.1016/j.entcom.2011.01.001.
Croskerry, P., Singhal, G., and Mamede, S. “Cognitive debiasing 2:
Impediments to and strategies for change,” in BMJ Quality and Safety
vol. 22 (2013), pp. ii65-ii72. doi: 0.1136/bmjqs-2012-001713.
Epley, N., and Gilovich, T. “The anchoring and adjustment heuristic:
Why adjustments are insufficient,” in Psychological Science vol. 17
(2006), pp. 311–318.
Epley, N., and Gilovich, T. “Putting adjustment back in the anchoring
and adjustment heuristic: Divergent processing of self-generated and
experimenter-provided anchors,” in Psychological Science vol. 12
(2001), pp. 391-396.
Epley, N., Keysar, B., Van Boven, L., and Gilovich, T. “Perspective tak-
Reduction of Cognitive Biases 83
ing as egocentric anchoring and adjustment,” in Journal of Personality
and Social Psychology vol. 87 (2004), pp. 327-339.
Epley, N., Morewedge, C., and Keysar, B. “Perspective taking in children and adults: Equivalent egocentrism but differential correction,” in
Journal of Experimental Social Psychology vol. 40 (2004), pp. 760-768.
Ericsson, K.A., Krampe, R.T., and Tesch-Romer, C. “The role of deliberate practice in the acquisition of expert performance,” in Psychological
Review vol. 100, no. 3 (1993), pp. 363-406.
Evans, J.S.B.T. Hypothetical thinking: Dual processes in reasoning and
judgment, vol. 3. New York: Psychology Press, 2007.
Forster, J., and Liberman, N. “Knowledge activation,” in Social Psychology: Handbook of Basic Principles, 2nd ed., edited by A. W. Kruglanski
and T. E. Higgins, pp. 201-231. New York: Guilford Press, 2007.
Gick, M.L., and Holyoak, K.J. “Analogical problem solving,” in Cognitive Psychology vol. 12, no. 3 (1980), pp. 306-355.
Grappiolo, C., Cheong, Y. G., Togelius, J., Khaled, R., and Yannakakis,
G. N. “Towards player adaptivity in a serious game for conflict resolution,” in Games and Virtual Worlds for Serious Applications (VSGAMES 2011), Third International Conference (Athens, Greece, May
2011), pp. 192-198.
Haferkamp, N., Kraemer, N. C., Linehan, C., and Schembri, M. “Training disaster communication by means of serious games in virtual environments” in Entertainment Computing vol 2, no. 2 (2011), pp. 81-88.
doi: 10.1016/j.entcom.2010.12.009.
Halpern, D. F., Millis, K., Graesser, A. C., Butler, H., Forsyth, C., and
Cai, Z. “Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning,” in Thinking Skills and Creativity
vol. 7, no. 2 (2012), pp. 93-100. doi: 10.1016/j.tsc.2012.03.006.
84 ToDiGRA
Hattie, J., and Timperley, H. “The power of feedback,” in Review of
Educational Research vol. 77, no. 1 (2007), pp. 81-112. doi: 10.3102/
003465430298487.
Human, L.J., and Biesanz, J.C. “Through the looking glass clearly:
Accuracy and assumed similarity in well-adjusted individuals’ first
impressions,” in Journal of Personality and Social Psychology vol. 100
(2011), pp. 349-364. doi: 10.1037/a0021850.
Kahneman, D. Thinking, fast and slow. New York: Farrar, Straus and
Giroux, 2011.
Kahneman, D., and Frederick, S. “Representativeness revisited:
Attribute substitution in intuitive judgment,” in Heuristics and Biases:
The Psychology of Intuitive Judgment, edited by T. Gilovich, D. Griffin,
and D. Kahneman, pp. 49–81. New York: Cambridge University Press,
2002.
Kahneman, D., and Tversky, A. “Subjective probability: A judgment of
representativeness,” in Cognitive Psychology vol. 3 (1972), pp. 430-454.
McClean, P., Saini-Eidukat, B., Schwert, D., Slator, B., and White, A.
“Virtual worlds in large enrollment science classes significantly improve
authentic learning,” in Proceedings of the 12th International Conference
on College Teaching and Learning: Center for the Advancement of
Teaching and Learning (Jacksonville, FL, April 2001), pp. 111-118.
McDaniel, R., Fiore, S.M., and Nicholson, D. “Serious storytelling: Narrative considerations for serious games researchers and developers,” in
Serious Game Design and Development: Technologies for Training and
Learning, edited by J. Cannon-Bowers and C. Bowers, pp. 13-30. Hershey, PA: IGI Global, 2010. doi: 10.4018/978-1-61520-739-8.ch002.
Millis, K., Forsyth, C., Butler, H., Wallace, P., Graesser, A., and Halpern,
D. “Operation ARIES!: A serious game for teaching scientific inquiry,”
in Serious Games and Edutainment Applications, edited by M. Ma, A.
Reduction of Cognitive Biases 85
Oikonomou, and L.C. Jain, pp. 169-195. London, U.K.: Springer-Verlag,
2011. doi: 10.1007/978-1-4471-2161-9_10.
Morewedge, C.K., and Kahneman, D. “Associative processes in intuitive
judgment,” in Trends in Cognitive Sciences vol. 14, no. 10 (2010), pp.
435-440. doi: 10.1016/j.tics.2010.07.004.
Muratet, M., Torguet, P., Jessel, J. P., and Viallet, F. “Towards a serious
game to help students learn computer programming,” in International
Journal of Computer Games Technology vol. 2009 (2009), pp. 1-12. doi:
10.1155/2009/470590.
Mussweiler, T., and Strack, F. “The use of category and exemplar knowledge in the solution of anchoring tasks,” in Journal of Personality and
Social Psychology vol. 78 (2000), pp. 1038-1052.
Northcraft, G.B., and Neale, M.A. “Experts, amateurs, and real estate:
An anchoring-and-adjustment perspective on property pricing decisions,” in Organizational Behavior and Human Decision Processes vol.
39 (1987), pp. 84-97. doi: 10.1016/0749-5978(87)90046-X.
Race, P. “Using feedback to help students learn.” The Higher Education
Academy.
http://wap.rdg.ac.uk/web/FILES/EngageinFeedback/
Race_using_feedback_to_help_students_learn.pdf
Ross, L., Greene, D., and House, P. “The false consensus effect: An egocentric bias in social perception and attribution processes,” in Journal of
Experimental Social Psychology vol. 13, no. 3 (1977), pp. 279-301. doi:
10.1016/0022-1031(77)90049-X.
Sabri, H., Cowan, B., Kapralos, B., Porte, M., Backstein, D., and
Dubrowski, A. “Serious games for knee replacement surgery procedure
education and training,” in Procedia: Social and Behavioral Sciences vol.
2, no. 2 (2010), pp. 3483–3488. doi: 10.1016/j.sbspro.2010.03.539.
Sapadin, L. “The anchoring effect: How it impacts your everyday life.”
Psych Central. Last modified July 24, 2013.
86 ToDiGRA
http://psychcentral.com/blog/archives/2013/07/27/the-anchoring-effecthow-it-impacts-your-everyday-life/.
Simmons, J.P., LeBoeuf, R.A., and Nelson, L.D. “The effect of accuracy
motivation on anchoring and adjustment: Do people adjust from provided anchors?” in Journal of Personality and Social Psychology vol. 99
(2010), pp. 917-932.
Šisler, V., Brom, C., Cuhra, J., Činátl, K., and Gemrot, J. “Stories from
the History of Czechoslovakia, A serious game for teaching history of
the Czech lands in the 20th century: Notes on design concepts and design
process,” in Entertainment Computing-ICEC 2012, edited by M. Herrlich, R. Malaka, and M. Masuch, pp. 67-74. Berlin, Germany: Springer
Berlin-Heidelberg, 2012. doi: 10.1007/978-3-642-33542-6_6.
Stapleton, A. “Serious games: Serious opportunities.” Paper presented at
the Australian Game Developers’ Conference, Academic Summit, Melbourne, VIC, 2004.
Sternberg, J. “10 stats brands should know about Facebook.” Digiday.
March 22, 2013. http://digiday.com/brands/10-stats-brands-shouldknow-about-facebook/.
Symborski, C., Barton, M., Quinn, M., Morewedge, C., Kassam, K.,
and Korris, J. “Missing: A serious game for the mitigation of cognitive
biases,” in Proceedings of the Interservice/Industry Training, Simulation,
and Education Conference (I/ITSEC; Orlando, Florida, December 2014),
pp. 1-13. http://www.iitsec.org/about/PublicationsProceedings/Documents/BP_TRNG_14295_Paper.pdf.
Tversky, A., and Kahneman, D. “Extensional versus intuitive reasoning:
The conjunction fallacy in probability judgment” in Heuristics and
Biases: The Psychology of Intuitive Judgment, edited by T. Gilovich, D.
Griffin, and D. Kahneman, pp. 19-48. Cambridge, U.K.: Cambridge University Press, 2002.
Reduction of Cognitive Biases 87
Tversky, A., and Kahneman, D. “Judgment under uncertainty: Heuristics
and biases,” in Science vol. 185, no. 4157 (1974), pp. 1124–1131.
doi: 10.1126/science.185.4157.1124.
Van Boven, L., and Loewenstein, G. “Social projection of transient drive
states,” in Personality and Social Psychology Bulletin vol. 29, no. 9
(2003), pp. 1159-1168. doi: 10.1177/0146167203254597.
Woll, R., Damerau, T., Wrasse, K., and Stark, R. “Augmented reality in
a serious game for manual assembly processes,” in Science and Technology Proceedings of the 37th IEEE International Symposium on Mixed
and Augmented Reality (Basel, Switzerland, October 2011), IEEE
Xplore, pp. 37-39. doi: 10.1109/ISMAR-AMH.2011.6093654.
Wouters, P., van der Spek, E.D., and Van Oostendorp, H. “Current practices in serious game research: A review from a learning outcomes
perspective,” in Games-based learning advancements for multisensory
human computer interfaces: Techniques and effective practices, edited
by T. Connolly, M. Stansfield, and L. Boyle, pp. 232-255. Hershey,
PA: Information Science Reference, 2009. doi: 10.4018/
978-1-60566-360-9.ch014.