“What Went Right and What Went Wrong”: An Analysis of

155 Postmortems from Game Development

Michael Washburn Jr.1 , Pavithra Sathiyanarayanan1 , Meiyappan Nagappan1 ,

Thomas Zimmermann2 , Christian Bird2
1
Rochester Institute of Technology, Rochester, NY, USA
2
Microsoft Research, Redmond, WA, USA
{mdw7326, ps2908}@rit.edu, mei@se.rit.edu, {tzimmer, cbird}@microsoft.com

ABSTRACT created from scratch, and deadlines are incredibly tight [12].
In game development, software teams often conduct post- Therefore it is important to understand both the chal-
mortems to reflect on what went well and what went wrong lenges that game development efforts face as well as the
in a project. The postmortems are shared publicly on gam- best practices that teams use to build games more effec-
ing sites or at developer conferences. In this paper, we tively. The challenges are real problems faced by complex
present an analysis of 155 postmortems published on the software efforts and represent avenues for research for our
gaming site Gamasutra.com. We identify characteristics of community. Successes and best practices embody knowl-
game development, link the characteristics to positive and edge that can aid future game development efforts and in
negative experiences in the postmortems and distill a set of some cases may generalize to or can be adapted for software
best practices and pitfalls for game development. development in non-game contexts. Because game devel-
opment makes up a large slice of commercial software, a
non-trivial proportion of students in computer science and
Keywords software engineering programs will work on games during
Games, Postmortems, Qualitative analysis. their careers. An understanding of game development can
help educate and prepare such students.
Interestingly, game development has received very little
1. INTRODUCTION attention in the academic community, as only three of the
Over the past thirty years, the importance and market- 116 open and closed source projects studied in the major
share of video games in the world of software has grown by software engineering conferences in two years were games [15].
leaps and bounds. In lockstep with this growth, the scale of Thus, one might reasonably expect that getting an inside
work required to develop games, whether in terms of bud- view of game development is limited to a select few. Fortu-
get, size of codebase, or team makeup, has ballooned and is nately, the game development community has a unique prac-
on par with or exceeds any other software endeavors [13]. tice that belies this assumption. Development teams often
Games are arguably the most sophisticated and complex conduct postmortem retrospectives and share them publicly
forms of software [18]. on gaming sites such as Gamasutra.com and at gaming con-
Indeed, games have been the driving factors behind many ferences such as the Game Developers Conference (GDC).
technological advances including high performance graphics These postmortems offer an open and honest window into
cards, virtual reality, and distributed computing [16, 13]. the development of games, often sharing the mistakes, set-
Games also represent a substantial portion of software rev- backs, and wasted effort just as much as the successes and
enue; in 2013, video game revenue totaled over 93 billion heroics that go into game building.
dollars [21]! As such, the money, manpower, and effort put To address the limited study of this domain and shed light
into video game development is likely to continue to increase on the practice of game development we qualitatively and
in the coming years. quantitatively analyze 155 retrospective postmortems pub-
From a development perspective, games differ from more lished on Gamasutra.com over 16 years. These postmortems
traditional software projects in a number of ways. Require- cover games for PCs, mobile devices, and consoles and range
ments are more subjective (e.g. “must be fun”), maintain- from small independent efforts to large AAA game fran-
ability is often sacrificed for performance, testing and qual- chises. As such, this represents the largest and most diverse
ity assurance are approached completely differently (e.g. live study of game development to date and this data allows us
testers and few automated tests), most games require tools to draw conclusions from a broad spectrum of game devel-
opment. We make the following contributions in this paper.
Permission to make digital or hard copies of all or part of this work for personal or • We present an empirically derived taxonomy of char-
classroom use is granted without fee provided that copies are not made or distributed acteristics or dimensions of game development.
for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. Copyrights for components of this work owned by others than • We synthesize the best practices and commonly en-
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- countered challenges in game development and identify
publish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org. those areas that impact project outcomes the most.
ICSE ’16 Companion, May 14-22, 2016, Austin, TX, USA • We provide recommendations for future game devel-

c 2016 ACM. ISBN 978-1-4503-4205-6/16/05. . . $15.00 opment based on the experiences shared in over one
DOI: http://dx.doi.org/10.1145/2889160.2889253 hundred postmortems.
2. RELATED WORK
There are many books and articles which inspect the prac-
tices relating to the game development process [2, 5, 10, 11,
22, 3]. These studies primarily focus on how game devel-
opment should be in the industry and are primarily based
off the authors’ experience. However, our findings focus on
how game development is in practice based off of the post-
mortem reviews written by game developers and posted on
Gamasutra.com [1]. There have been several studies where
developers were interviewed or surveyed to understand sev-
eral specific characteristics of game development [4, 8, 20, 7,
14, 12]. These studies tend to focus more on software engi-
neering processes and methodologies in game development.
Additionally, these studies solicited information from game
developers by either conducting interviews or surveying de-
velopers. In contrast, we retrieved our data more organ-
ically by analyzing self reported postmortem reviews that
had already been written by developers. Similar to what we
do, there have been studies that examine what went wrong
during game development using bug data [9], and 20 post-
mortem reviews [17]. However, in this study, we not only
focus on what went wrong but also look at what went right Figure 1: Screenshot of the first of a six page post
in order to distill best practices and pitfalls. mortem for Pangalore’s Knightly Adventure on Gama-
The work most similar to ours is that of Ara Shirinian who sutra.com
conducted a study in which he analyzed 24 postmortem re- If a case was in fact a postmortem review for a game,
views written between 2008 and 2010 from Gamasutra.com and detailed what went right and what went wrong during
in order to see if there were any interesting trends occurring development, then it was used in this study. Out of the
in game development [19]. Also like this study, he analyzed original 215 post-mortem reviews, 60 cases were ignored,
the postmortems and grouped them into categories to de- leaving 155 postmortem reviews to be analyzed.
termine what went right and what went wrong while devel- Identifying Categories: Initially, we started with 12
oping games. In contrast, we analyzed a much larger set categories of common aspects of development. These cate-
of postmortem reviews, spanning from 1998 until 2015. In gories were based on the categories Ara Shirinian identified
addition to this, Shirinian categorized items from the post- in his analysis of postmortem reviews [19].
mortems into 7 categories of what went right and what went In order to identify additional categories, we performed 3
wrong, while we had a total of 22 categories in order to more iterations of analysis and identification. The first week, we
precisely determine the trends in game development (which each read and analyzed 3 postmortem reviews each, classi-
could be because of the increased set of postmortem reviews fying the items discussed in each section into the 12 prede-
that we analyzed). termined categories of common aspects that impact devel-
opment. While analyzing these reviews, we identified addi-
3. METHODOLOGY tional categories of items that went right or wrong during
development, and revisited the reviews we had already ana-
To conduct this case study, we analyzed 215 postmortem lyzed to update the categorization of items. For the next two
reviews listed before Jan 2014 on Gamasutra.com, a game weeks we repeated this process of analyzing postmortems
development news website. These reviews contain an intro- and identifying categories, analyzing 10 postmortems each
duction with some context, sections describing what went in week 2, and 15 postmortems each in week 3. After each it-
right and what went wrong during the development process, eration, we discussed the additional categories we identified,
and finally some more contextual information in a table. and determined if they were viable.
An example of a postmortem is shown in Figure 1. We Analysis: After our initial iterations for identifying addi-
(Michael Washburn and Pavithra Sathiyanarayanan) ana- tional categories, we had completed the analysis of 60 post-
lyzed each postmortem review, categorizing items discussed mortem reviews. We then stopped identifying new cate-
in the what went right and what went wrong sections into gories, and began analyzing postmortems at a combined rate
groups of common themes. We used the contextual informa- of about 40 postmortem reviews per week. After each week
tion to determine which platform the game was designed for, we reviewed what we had done to ensure we both had the
how many people were in the development team, how much same understanding of each category. This continued until
time did it take to develop the game among other things. we had analyzed all the postmortem reviews.
Ignoring Off-Topic Reviews: While analyzing the post-
mortem reviews, we found that some reviews were off-topic. 4. CONTEXTUAL INFORMATION
For example, some cases were just reviews of an individual’s The characteristics of the games mentioned in the post-
experience at the Game Developers Conference. Sometimes mortem reviews we inspected were diverse in areas such as
the case only described a specific tool or technology used their supported platforms, the size of the development team,
during development, rather than how development went. In the amount of time it took to develop them, the type of
other cases, authors focused on a particular aspect of their publisher they had, and more. We will further describe the
process, rather than their experience as a whole. Cases clas- context of these games, based on data we gathered, which
sified as off-topic were ignored in our study. is available in our online appendix [6].
 Platforms Used: The majority of the games created in during the making of American McGee’s Grimm Spicy Horse
these postmortem reviews were developed for the PC plat- did “a great job at keeping creative thinking alive throughout
form, with the next most common platforms being the Xbox the whole project.” They were able to inject large amounts
360, followed by the PlayStation 2 (PS2). Of the post- of creative content into their game by encouraging an open
mortem reviews we analyzed, 87% of them included the dialog within their company. They encouraged anyone with
platforms they were developed for. Games can be devel- an idea come forward and lead a discussion. Decisions were
oped for a single platform or for multiple platforms. Of the made by a democratic vote, to prevent any one person from
postmortems that indicated platform information, 37.86% taking control of the game.
of them were developed for a single platform, while 62.14% Example of what went wrong: Tim Turner, of developer
were developed for multiple platforms. Intergalactic Crime Prevention Unit, said that they chose
Length of Development: The majority of projects were to build an improvement of an existing game mechanic as
developed in less then 2 years. Of the 73% of developers who their debut title, and while it was “not a knockoff,” it was
put the length of development in their postmortem reviews, not “groundbreaking” either. He stated that they “wasted
39% of the games were developed in 1-2 years and 38% were many months on a title that, ultimately, we decided would
developed in 1 year or less. not serve us well as our initial commercial release.”
Team Size: Out of the 81% of postmortems which stated (c) Features: Many developers listed features as something
the number of developers the team had, 74% of the teams that went right when they had a lot of very unique and fun
contained 20 team members or less. The remaining 26% of content in their games, or they had one feature that went
the teams were of various sizes greater than 20. very well and drastically contributed to the quality of their
Publishers: Developers often have to make a choice be- game. Developers typically listed features as going wrong
tween whether they want to, or can, publish their game when they poorly implemented a feature, discovered after
themselves, or whether they should sign with a publisher. implementation that a feature was a bad idea, or when they
90% of developers listed the publisher of their game in their experienced feature creep.
postmortem. Out of these 74.4% used a third party pub- Example of what went right: Lionhead Studios’ Peter Mol-
lisher, while 25.6% decided to publish the game themselves. yneux stated that while they were developing Black & White
they were able to create an AI that was actually able to learn
5. IDENTIFIED CATEGORIES from the user and develop a personality such that no two
users’ AIs would ever have the same personalities. Molyneux
We identified the following categories as common aspects
described the AI as an “astonishing piece of work.”
of things that go right or wrong for teams during game de-
Example of what went wrong: John Cutler, of Cutler Cre-
velopment. Each of these categories are further divided into
ative, said that in Last Call, “Feature creep ruled our roost.”
sub-categories. In each sub-category, we give a description
They kept adding features to their game even though they
of them and an example. Note that all the categories are
had a deadline and budget to meet, which resulted in tension
present in both best practices and pitfalls. This is because a
between them and their publisher.
developer could talk about doing an activity like testing as
(d) Game Design: Developers listed this as something
a best practice, or talk about the lack of testing as a pitfall.
that went right or wrong when they had made good or bad
5.1 Product design decisions that impacted the quality of their game.
Example of what went right: Stuart Denman of Surreal
In this category, we present and discuss seven sub-categories
Software stated that “A good design will not only sell a
that are related to the artifacts that make the game.
game – it can also help smooth the development process.”
(a) Art: Game art is a contributing factor to the overall
During the making of Sanitarium, Surreal Software was able
quality of a game. Items that can go right or wrong within
to create a game design that was unique, kept the team
this category include artist skill, art development process,
interested, offered many possibilities for new features, and
art quality, the effect of art on gameplay, and the effect of
allowed some freedom for artists and designed to work with,
art on user experience.
according to Denman.
Example of what went right: Nicole Goodfellow of Torus
Example of what went wrong: Søren Lund of Deadline
Games said that while they were developing Scooby-Doo!
Games wrote that during the making of Chili Con Carnage,
First Frights they made the decision to pair level designers
they designed a game under the assumption that “players
with artists during development. According to Nicole, “They
would ‘get’ the game and [the developers’] desire to make
would sit together, eat together, live and breathe their level.”
players perfect their score.” Upon release they discovered
The entire team was impressed with the quality of the levels,
players weren’t interested in perfecting scores, but wanted
and each pair took great pride in the levels they created.
more depth to the game. They stated that they should spent
Example of what went wrong: Søren Lund of Deadline
more time to lengthen the story of the game.
Games said that they decided to focus less on creating unique
(e) Gameplay: How players interact with the game often
art content and had their artists focus on small areas of the
determines how easily they will learn the controls as well as
game instead when making Chili Con Carnage. The team
how entertaining it is for them.
recycled much of the game art from their previous title To-
Example of what went right: According to Prithvi Viras-
tal Overdose, under the assumption that few people would
inghe and Jeremy Mahler of Pipeworks Software, while cre-
play both games. However, the reviewers had played both
ating The Deadliest Warrior game (based on the popular
games, and gave the game bad reviews due to the lack of
TV show), they actually had some members of the team
unique content.
train with the weapons featured in the game. Additionally,
(b) Creativity: The creativity of the development team is
they said “The show also weighed in on the warrior and
often a main factor in the success or failure of a game.
weapons designs, as well as gameplay itself.”
Example of what went right: According to Wim Coveliers,
 Example of what went wrong: According to Bong Koo that this “allowed us [the developer] to get the core game
Shin of Gamevil, while developing Nom 2, they wanted to mechanics to the point where we wanted [them] to be.”
make the game very intense for the user in order to keep Example of what went wrong: Naked Sky Entertainment
them interested in the game. However, many users com- dove right into the development of RoboBlitz with almost
plained that the gameplay was too intense, going as far as no work done planning or designing the product. Accord-
to say that “their eyes hurt because the game is so fast.” ing to them, “When we started development on RoboBlitz,
(f ) Product Evolution: Software evolution in general in- we jumped into the level creation process with very little
dicates the changes happening in the software over time. In pre-production.” Unfortunately, this led to them designing
our study we link product evolution to those cases where the their levels before they had fully decided what they wanted
games evolve from one idea into another idea. the game mechanics to be like. Additionally, when the game
Example of what went right: Tom Leonard of Looking was nearly finished, the team had trouble making even mi-
Glass Studios said that the Thief: The Dark Project game nor design changes. They stated that “it became extremely
was originally called The Dark Project, but was renamed burdensome whenever we had to add new features or tweak
during development. Leonard stated that the renaming was character movements.”
“a seemingly minor decision that in truth gave the team a (b) Documents: Generally, developers who listed some-
concrete ideological focus.” thing in the documentation category as going right when
Example of what went wrong: Alyssa Finley of 2K Games they produced designs and other documentation which ben-
stated that while developing BioShock “The spec of BioShock efited the project in the long run. When developers listed
changed so much over the course of development that we documentation as going wrong, they usually suffered from a
spent the majority of the time making the wrong game.” The lack of proper documentation.
game also changed from being a RPG first person shooter to Example of what went right: Wu Dong Hao of Ubisoft said
being just a first person shooter. Finley said that “the real that they dedicated 3 months to preparing for the develop-
turning point for BioShock came when we had to present ment of Tom Clancy’s Splinter Cell. During this time, one of
the game to the outside world, which forced us to carefully the things they did was create technical design documents.
consider the story and takeaway message.” She concluded The team stated that “One of the most important steps we
by stating that they should have taken some time to develop took during the pre-production stage was to create Techni-
those ideas sooner. cal Design Documents (TDD), into which we poured all the
(g) Scope: The set of features that developers choose to knowledge we gleaned from our prototype.” The documents
include in their product is always going to make a difference also helped everyone to understand the major technical is-
in the quality of their product, and whether or not they will sues of their game. However, one team did mention that
complete it on time. they spent as little time doing documentation as possible,
Example of what went right: Brad Wardell of Stardock and “As long as the documentation provided a high-level
Entertainment stated that in Galactic Civilizations, they de- idea of the game and how it would play, we [the developers]
cided not to include a multiplayer mode. By limiting their dropped it like a hot potato and got back to work.” They
scope they were able add more advanced features to single could do this because they they strongly emphasized verbal
player mode. Wardell said “it’s much easier to add features communication during development, and used a very itera-
to a game when you don’t have to worry about how they’ll tive, agile approach to development.
impact issues like synchronization, latency, and game flow.” Example of what went wrong: According to Marek and
User feedback from beta testing also played a role in this Ondrej Spanel of Bohemia Interactive Studio, in Operation
decision. They concluded that making this decision made Flashpoint they didn’t document their design or have any
Galactic Civilizations a much better game. documentation on what features had already been built dur-
Example of what went wrong: Schadenfreude Interactive ing development. This led to problems in the late stages
was developing Age Of Ornithology and kept increasing the of development. At one point they said that “hours were
size of their scope. They stated that “Despite our attempts spent trying to investigate how something had originally
to cut back, the programmers were sneaking in Easter eggs been meant to work.” However, some teams also suffered
until the last minute.” This lack of control led to some player from over-documentation. Paul Jobling of Eutechnyx stated
confusion after the game was released. that during the development of Big Motha Truckers they
5.2 Development tried producing documentation that also doubled as an in-
house marketing tool in order to get people interested in the
In this category, we present six sub-categories that are
game. Jobling stated that “As a result, instead of concen-
related to the development process in building the game.
trating on the ‘hows’ and ‘whys’ of the game’s production,
(a) Development Process: The process teams use while
it was instead focused on the ‘whos’ and the ‘wheres.”’
developing always affects the quality of the product. This
(c) Obstacles: Teams often face obstacles during devel-
was one of the more common categories that developers often
opment. A situation will arise which makes it difficult for
listed in their postmortems.
the team to continue development, and how the team reacts
Example of what went right: Jeferson Valadares and Mikko
when faced with obstacles will impact their product for bet-
Kodisoja of Sumea Games stated that in Tower Bloxx, they
ter or for worse. Obstacles are more likely to have a negative
needed to improve their game mechanics and design. Their
impact on a team.
solution was to pair their Senior Developer and Senior De-
Example of what went right: Alberto Moreno and Carlos
signer together, and do small, rapid iterations. In these
Abril of Crocodile Entertainment stated that they didn’t
iterations the developer would make a change, then the de-
have proper office space for the development of Zack Zero, so
signer would give them feedback and say how it should be
they were forced to improve their process, communication,
fixed in the next iteration. Each iteration lasted less than 3
and task organization, which benefited them in the long run.
hours, and allowed a rapid evolution of the game. They said
 Example of what went wrong: John Li of Pixelogic stated (a) Budget: A team’s budget can play a huge role dur-
that during the development of The Italian Job game, their ing the development of any software. Often times in game
team member Rob was in a car accident and was unable to development, the budget affects other aspects of the game
work. After the injured team member was able to work, the such as marketing, which contractors get hired, and technol-
team sent his PC to his home, so that he could work when ogy choices. Typically when developers listed the budget as
he was feeling up to it. Overall, they said their schedule was something that went right, it wasn’t because they had mas-
delayed 2 weeks because of this incident. sive budgets, but rather they had average or limited budgets
(d) Team: Generally, developers had positive experiences which they were able to stick to during development. Con-
with their teams. However, some teams do have trouble versely, developers listed the budget as one of the things that
during development. went wrong when it was so limited that it hurt the quality
Example of what went right: During the development of of the game, or they were unable to stay on budget.
Dungeon Siege, Bartosz Kijanka of Gas Powered Games de- Example of what went right: Mike Goslin of VR Studios
scribed the development team as “the single best thing thing said that they were operating under a tight budget to min-
about Dungeon Siege.” They attribute this to the individu- imize risk during the creation of Toontown. They were still
als’ strong values, personalities, and respect. able to release the quality MMORPG by designing it to be
Example of what went wrong: Scott Bilas of Sierra Studios as low cost as possible. Goslin said “No in-game support is
said that the team who developed Gabriel Knight 3 suffered required, which eliminates a significant component of tra-
from bad role casting. The original team was not experi- ditional customer service costs for this genre of game. In
enced enough to implement the advanced features of the addition, we consume a fraction of the bandwidth of other
game, and so there was a lot of turnover within the team. MMORPGs. Both of these costs scale with the number of
Bilas said “Many of the problems with GK3 resulted from players, so they are important ones to minimize.”
developers being badly cast in their roles, usually because Example of what went wrong: During the development of
the project requirements were so severely underestimated.” RoboBlitz, Naked Sky Entertainment ran out of money mid-
(e) Testing: Developers put testing down under what went way through developing a game because they incorrectly es-
right when they performed some kind of testing during de- timated their schedule. Additionally they said “Because we
velopment such as unit/alpha/beta/usability testing, etc. weren’t willing to put out an inferior product, we just kept
When developers listed testing as something that went wrong, going until we were satisfied with the quality.” The develop-
they generally suffered from a lack of testing. ers were forced to borrow money from family members and
Example of what went right: Paul Dennen of Nayantara close friends in order to continue development.
Studios said that in Star Chamber they performed alpha (b) Hardware: Developers listed hardware as something
and beta testing. This allowed them to make adjustments that went right or wrong when either they were able to work
that they wouldn’t have identified otherwise. Specifically, with very good hardware, allowing them access to advanced
Dennen said “Long, meticulous early testing periods allowed features, or they were working with very limited hardware,
players to acquire deep understanding of the gameplay, and which hindered development. Developers were more likely
they were then able to provide well-informed feedback that to list this as something that went right or wrong if they
was invaluable in balancing the game.” were working on a single platform. In fact, about 90% of
Example of what went wrong: Ichiro Lambe, Dan Brain- the developers who listed this as something that went right
erd, and Leo Jaitley of Dejobaan Games stated that they or wrong were working on a single platform.
didn’t test Aaaaa! – A Reckless Disregard for Gravity enough. Example of what went right: Motohideo Eshiro and Ku-
Additionally, even though they did eventually bring in beta niomi Matsushita of Capcom described some of the Nintendo
testers, they said “when we did bring testers in to toss the DS hardware features they benefited from during the devel-
game around, we underused the feedback we received.” opment of Okamiden. In the PlayStation 2 and Wii prequels,
(f ) Tools: The tools you utilize to develop a game often im- Okami, users had some difficulty using the drawing feature
pact the ease of development, the complexity of the features, of the game because of the limitations of the PlayStation
and the quality of the end product. controller and Wii Remote. Additionally, Eshiro and Mat-
Example of what went right: Eric Peterson, Wayne Har- sushita stated that “with the Nintendo DS stylus, not only
vey, Mike Pearson, and Dave Ellis of Vicious Cycle Software is one able to draw even more complex lines precisely, but
said that while developing Dead Head Fred they had previ- we were able to create even better puzzles and missions that
ously developed their own tool, the Vicious Engine. They utilize the celestial brush mechanic.”
were able to implement all the game features for Dead Head Example of what went wrong: Bong Koo Shin of Gamevil
Fred using preexisting code in the Vicious Engine, allowing described the hardware limitations they encountered on Ko-
them not to write any game specific engine code. rean mobile devices for Nom 2. At the time, it was very
Example of what went wrong: Alexander Seropian of Wide- difficult for the devices to play two sounds at once. The
load Games said that in Stubbs the Zombie they decided to only way they were able to get this to work was to cut the
use the Halo engine. However, the Halo engine had never background music, then play the game sound, and resume
been licensed before, and therefore had little documenta- the music. They were forced to cut the sound effects, and
tion. Additionally, Seropian stated that “the learning curve just stick with background music, which had a small nega-
slowed us [the developers] down and wasted time.” tive effect on the experience of playing the game.
(c) Publisher Relations: If a developer chose to work
5.3 Resources with a publisher, then they often experienced benefits or
drawbacks from that relationship. In general, those who
In this subsection, we present four things that develop-
listed items in this category in their postmortems had good
ers talk about in their postmortems that are related to the
relationships with their publisher.
resources required to develop a game.
 Example of what went right: Linda Currie of Blue Fang a feature of your game.” They argue that when players of a
Games said that the team had a great relationship with Mi- game develop a sense of community, it’s like adding a whole
crosoft during the development of Zoo Tycoon 2: Marine other feature to that game.
Mania. Currie said that “the time we [Blue Fang Games] Example of what went wrong: Jamie Cheng of Klei En-
spent with them [Microsoft] both in person and in [dialog- tertainment said that they had a thriving online commu-
ging] via phone and email really paid off with both parties nity when they released their first preview of their game
being on the same page about product decisions.” Currie Eets: Hunger. It’s emotional. during development, which
also expressed that they were able to convey ideas to Mi- was merely a throwaway prototype of their game designed
crosoft openly and freely, and concluded that “Throughout to gather feedback from users. However, the community
[Zoo Tycoon 2: Marine Mania] we felt like we were working was begging for more content and when Klei Entertainment
with a helpful partner.” could not deliver, the community died and their game lost
Example of what went wrong: David McQueen of The popularity before it was even released. Cheng also said “A
Games Kitchen described the development of Wireless Pets large part of the problem was we didn’t know what to tell
under Digital Bridges. While McQueen did say that “DB them. Were we going to be on handheld? Self-published
[Digital Bridges] is not at all bad to work with,” there was online? Retail PC?”
some friction at times. Use of intellectual property, and dif- (b) Feedback: In most cases developers listed feedback as
ferences of opinion of decisions about the game strained their something that went right when they actively sought feed-
relationship. They overcame this in the future by laying out back during development to make sure they were building
very specific ground rules and processes to adhere to, and the right product, or when they received good feedback after
stated that they now work with Digital Bridges regularly. release. In general, developers listed feedback as something
(d) Schedule: Most developers who listed the schedule as that went wrong when they received bad or insufficient feed-
something that went right did it because they were able to back upon release.
meet their milestones, or because they were granted extra Example of what went right: Alyssa Finley of 2K Games
time to complete their milestones. Conversely, those who stated that while developing BioShock, 2K Games conducted
listed the schedule as something that went wrong usually tests with a focus group to see how they reacted to the game.
missed milestones or delivered them late. They described the feedback as “brutal.” Finley also said
Example of what went right: Wu Dong Hao of Ubisoft that “Based on this humbling feedback, we came to the re-
described their unique experience of having to port Tom alization that our own instincts were not serving us well,”
Clancy’s Splinter Cell from the Xbox to the PlayStation and they were able to make the necessary corrections.
2 within a 4 month schedule. Hao said they were able to Example of what went wrong: Ethan Einhorn of Other
make this work because “the company was willing to dedi- Ocean stated that after the release of Super Monkey Ball 2,
cate people and spend money to gain time.” They also did a the team realized that players were not interested in their
lot of preparation prior to when they were handed the Xbox game’s multiplayer mode when all of the players’ feedback
version of the game, which helped to organize the team. was on the single-player mode. Einhorn stated “if we had
Example of what went wrong: Brian Reynolds of Big Huge known that interest in the multiplayer would be so limited,
Games described the developing of Rise of Nations. They we may have dropped it.” If they had sought feedback prior
underestimated their schedule, which made them work long to release, they could have avoided this situation.
hours later on in the project. Reynolds stated that“We un- (c) Marketing: Typically, developers who published their
derestimated the amount of coding time necessary, which re- own game listed marketing more than those who had pub-
sulted in an extremely overworked programming staff.” Ad- lishers. About 70% of the developers that listed marketing
ditionally, overloaded their lead programmer, making him a as going right or wrong published their game themselves.
bottleneck for development. They also did not hire enough This is because in most cases where a publisher is present,
programmers to start. They stated that the root of their the developer won’t deal with the marketing aspect.
mistake was not listening to the postmortem reviews they Example of what went right: Frank Wilson and Josh Bear
had read in the Game Developer magazine. of Twisted Pixel Games described after developing Splosion
5.4 Customer facing Man, they were able to take their game to Microsoft’s Sum-
mer of Arcade promotional event. Attending the event defi-
Here we present four themes that are related to customer
nitely helped get the word out about their game, they said.
facing issues that developers discussed in their postmortems.
They stated that “The extra attention the game received
(a) Community Support: The support of a community
was something that would have been very hard for a small
can greatly benefit any game, whether it be an online com-
company like ours to get recognition for on our own.”
munity spreading the word about your game, or a group of
Example of what went wrong: Dave Gilbert of Wadjet
outside individuals who lend a helping hand during devel-
Eye Games stated that after releasing The Blackwell Con-
opment. Other aspects of community support that can go
vergence on various game portals (online game distribution
right or wrong for a team are the presence or lack of an on-
websites), the game was not getting noticed. Gilbert later
line community, publicity by users of message boards, a loyal
said “I learned my lesson very quickly: I could no longer rely
fan base. We found that developers listed this as something
on the game portals to sustain the majority of my income,”
that went well during development rather than something
and that in the future he would have to do some marketing
that went wrong.
and PR work for his games.
Example of what went right: Art Min of game developer
(d) Piracy/Licensing: Piracy and licensing has always
Multitude said that they created tools for the community to
been an issue in the gaming industry. Whether it be a game
utilize such within their game lobby and community website
losing sales because of piracy, or a lack of quality music
during the making of Fireteam. They stated that “Players
because the developer couldn’t get the license.
are not only a source of revenue for a project, but they are
 Example of clear vision: Paul Dennen of Nayantara Stu-
dios stated that “Focusing the design down one clearly plot-
ted path from start to finish resulted in a game with strength
of clarity and identity” during the creation of Start Chamber.
Takeaway: Game developers should create a well-defined
concept before beginning development as opposed to an ad
hoc method of game design. Developers should also focus
the design around capturing the attention of the player.
(b) Development Process: Development process was mar-
ked as something that went right in 43% of the postmortem
reviews, according to Figure 2. Among the teams that listed
the development process as something that went right when
making a game, there were 3 common practices teams did:
First, they spent time planning and preparing for the game
prior to development. Additionally, they often created pro-
totypes which they would use as a proof of concept for the
game in general, or a specific feature of the game. Many
teams also used their prototypes as a basis of development.
It was also common for these teams to use an iterative de-
Figure 2: What went right velopment process.
Example of planning: Brian Gilman of Full Sail described
Example of what went right: Blair Fraser and Brad Wardell
how before starting development on Romeo and Juliet, they
of developer Ironclad said that they chose to release Sins of a
were able to spend an extra month planning how they were
Solar Empire without any CD/DVD copy protection. They
going to complete the project. They said that “a single addi-
calculated that users would still buy the game to avoid the
tional month in pre-production can make all of the difference
inconvenience of pirating it. Therefore, they were able to
in the world.”
ignore the issue, and save a little time and money.
Example of prototyping: Wu Dong Hao of Ubisoft stated
Example of what went wrong: Alex Austin of Chronic
that they made a prototype for their title Tom Clancy’s
Logic stated that when they released Gish, many players
Splinter Cell which “helped determine resources and schedul-
started pirating their software after release. This took away
ing” for the whole project. They also said that the prototype
from their profits and they expressed much frustration at
helped them organize the production schedule and proved
watching so many users steal their game. They even said
that they could overcome the technical issues of the prob-
that in some cases there were “people even writing to us
lem. Additionally, they were able to use the prototype as a
[Chronic Logic] for tech support when their stolen keycode
base for production.
[didn’t] work with the newest patch.”
Takeaway: For a better development process, game de-
velopers should invest time in the beginning of the project
5.5 Other planning and designing. Game developers should also build
We categorized things into this category when they didn’t prototypes during development, and if possible continue build-
quite fit into the other categories. Examples include - teams ing off of these prototypes using an iterative process.
believing in themselves, a team self funding their own project, (c) Team: The team was listed as something that went
luck, and risks paying off. right in 40% of the postmortem reviews that we analyzed,
as depicted in Figure 2. Of the developers that listed the
6. RESULTS team as going right during development, many of them cred-
ited their experienced team members. Many developers also
6.1 Best Practices stated that their team members were very motivated.
Example of experienced team: According to Rade Sto-
Figure 2 shows the percentage of postmortem reviews which
jsavljevic, the team who built Command and Conquer: Tibe-
list each category as something that went right during de-
rian Sun was extremely experienced with real time strategy
velopment. These percentages do not add up to 100% be-
games. Many of the members had worked on at least 4 real
cause each postmortem lists multiple categories that went
time strategy games before, and according to Stojsavljevic
right during development. The four categories that most
“This level of experience was key in allowing the team to
frequently went right were game design, development pro-
conquer all the obstacles thrown in their path.”
cess, team, and art.
Example of motivated team: Deng Yi Wen of Ubisoft de-
(a) Game Design: As shown in Figure 2, Game Design
scribed how the team of Music Up – Summer Rainbow was
was stated to have went right in 50% of postmortems. In
highly motivated. He stated that “A willing crew is the most
general, teams who marked game design as something that
important factor shipping a title on time, and we had a team
went right emphasized having a “hook” to capture player
who [was] highly motivated by the chance to finally make a
interest, and having a clear vision. Having a complete story,
game for local players”
and developed characters were less prevalent themes.
Takeaway: Game developers should spend more time
Example of game hook: Dan Marshall stated that when
training their team members in the environment they will
making Gibbage, he emphasized variety in the different levels
be developing in. However, game development companies
of Gibbage, which helped maintain the player’s attention.
should focus on recruiting not only talented individuals, but
Marshall said “It’s this variety that keeps the gameplay fresh
they have to be motivated as well.
throughout each of its levels.”
 Takeaway: Developers working in newly formed teams
should participate in team building in order to minimize
risks during development related to unfamiliar teams. Ad-
ditionally, new companies and teams should subscribe to a
method of risk management, because they are more likely
to face obstacles than more seasoned teams.
(b) Schedule: In 25% of the postmortem reviews, develop-
ers said that the schedule was something that went wrong,
as show in Figure 3. These games faced common problems
in estimation and optimistic scheduling. Some teams also
faced problems with design changes late in development.
Example of bad estimation: Linda Currie of Blue Fang
Games described the scheduling problems surrounding their
game Zoo Tycoon: Marine Mania. According to Currie,
“we [Blue Fang Games] overlooked the fact that the young
guests needed their own specific animations.” Currie also
described additional scheduling problems, saying that “We
also underestimated some of the difficulty in transitioning
to 3D movement through water.”
Figure 3: What went wrong Example of bad optimistic scheduling: Peter Molyneux of
(d) Art: In Figure 2 it is shown that art went right in 39% Lionhead Studios described the scheduling difficulties they
of the postmortem reviews. Those developers who marked encountered during the creation of Black & White. Molyneaux
art as something that went right in development typically admitted “I have a reputation for being, shall we say, opti-
had creative artists on their teams, or were able to contract mistic about when the projects I’m working on will be com-
creative artists. pleted.” Molyneaux also stated that there were many people
Example of creative artists: Brad Wardell of Stardock En- who didn’t believe him when he finally told them the game
tertainment described their method of improving the quality was finished, and that they “were only convinced when they
of art in Galactic Civilizations 2: Dread Lords. According to saw a box with a CD in it.”
Wardell, “Anyone who’s followed Gamasutra for a long time Takeaway: To avoid schedule slippage, game develop-
and seen Stardock’s games mentioned in the past knows one ers need to spend more time to plan out all the work that
simple fact: Stardock’s games look like crap.” To improve needs to be done so that no tasks are overlooked when giv-
the art in the new Galactic Civilizations, they began work- ing estimates. By planning out tasks, you can also estimate
ing with popular artists like Paul Boyer and Mike Bryant. each task individually, and give a more accurate prediction,
Additionally, they increased the size of their in-staff art de- instead of an optimistic one.
partment. Wardell stated that “The result was that we were (c) Development Process: The development process was
able to make a game that had competitive graphics for any listed as something that went wrong in 24% of postmortem
AAA strategy title.” reviews. Many teams listed this as something that went
Takeaway: In order to build a higher quality game, game wrong when they did not spend a significant amount of time
studies should seek the help of professional artists in order planning before beginning development. There was a large
to make visuals that will capture the attention of the player. variety of things that went wrong in the development pro-
cesses, however, another less common theme within these
6.2 Pitfalls postmortems was mismanagement.
Example of lack of planning: Wayne Imlach of Muckyfoot
The percentages of each categories that went wrong in
Productions described a lack of up-front design work during
the postmortem reviews can be seen in Figure 3. These per-
the making of Startopia. The result was a lot of confusion
centages do not add up to 100% because each postmortem
among the team during development. Imlach said that they
mentions multiple categories that went wrong during devel-
dove in to development as soon as possible but that “we
opment. The 4 most occurring categories that went wrong
[Muckyfoot Productions] failed to realize at the time that
for teams during development are obstacles, schedule, devel-
everybody was carrying a slightly different picture in his
opment process, and game design.
head of what the final game would be.”
(a) Obstacles: Teams reported facing various obstacles in
Example of poor project management: Leonard Paul of
37% of the postmortem reviews, as shown in Figure 3. While
Moderngroove stated that Modern Groove – The Ministry
there was a large variety in the types of obstacles faced by
of Sound Edition suffered from poor project management
developers, a significant number of the teams and sometimes
because they had no dedicated project manager. Most of
companies were newly formed, and faced obstacles related
those responsibilities were given to the lead programmer,
to lack of team dynamic and unfamiliarity among the team.
which resulted in an over burdened team. Paul said “I be-
Example of newly formed team: Scott Alden of Dream-
lieve hiring a good manager early in the project and having
Forge Entertainment described the troubles in their newly
clearer deadlines would have definitely helped the project.”
formed team during the development of Sin. Alden said
Takeaway: In order to avoid conflicts during the devel-
“Our newly formed tribe felt little sense of cohesion, as most
opment process, teams need to have proper management.
members were basically strangers to each other.” He went
Developers also need to invest time upfront planning before
on to say that their team had trouble agreeing with each
beginning development.
other, which led to problems making decisions during the
(d) Game Design: According to Figure 3, game design
project.
is something that went wrong in 22% of the postmortem 7.2 Publisher vs. No Publisher
reviews. Many teams fell victim to overly ambitious game When looking at what went right and wrong in developers
designs which could not be implemented, or caused them to who had publishers versus developers who published their
go over their planned schedule. In addition to this, many own game, we identified a few categories in which one group
other teams designed games with concepts that confused the performed better than the other.
player. First, 30% of the developers without publishers listed test-
Example of ambitious design Jim Napier of Sierra Studios ing as something that went right during development, while
described how the design for SWAT 3: Close Quarters Bat- only 19% of those with publishers listed this as having gone
tle was extremely ambitious. Specifically, Napier said that right. The two groups were comparable in the number of
“Most of the individual features seemed doable, but added times they listed testing as something that went wrong, with
up they represented a tremendous amount of work.” They 11% of those without publishers and 13% of those with pub-
were able to release the game on time still by cutting out lishers listing it as having gone wrong. This shows that those
some of the features, which “was painful for everyone be- developers who published their own games performed better
cause we felt the game wouldn’t be nearly as fun without testing than the developers who had publishers.
the missing features.” Second, we found that while only 17% of developers who
Example of confusing design Bruce Chia and Desmond published their ow game listed tools as something that went
Wong of Singapore-MIT GAMBIT Game Lab described the right, 40% of developers with publishers listed tools as some-
confusion surrounding their original prototypes for a game. thing that went right. Both groups listed tools as having
The game was based around the concept that players would gone wrong in only 17% of cases. This indicates that devel-
be rewarded by failing to complete an action. However, they opers who have publishers often have better tools for devel-
said that “testers didn’t understand the logic behind these opment than developers who do not have publishers.
design decisions, and we spent a lot of time making a game Lastly, we found that 43% of the developers that pub-
that was not well thought-out and was boring to play.” lished their own game listed obstacles as something that
Takeaway: For a better game design, developers should went wrong, while only 35% of the developers with publish-
keep implementation in mind while creating a design, and ers ran into obstacles. This suggests that developers who
if a implementation cannot be pictured for a part of the publish their own games may be affected more severely by
design, contingencies should be made. Key game concepts obstacles than developers who utilize publishers.
should also be testing before release so the reactions of the
players can be verified. 7.3 Single Platform vs. Multiplatform
When looking at games developed for one platform versus
7. DISCUSSION games developed for multiple platforms, we found a few ad-
As described in Section 4, we collected the metadata avail- vantages and disadvantages to choosing one over the other.
able in each postmortem as well. Based on this metadata, First, 46% of developers that used multiple platforms listed
we split our postmortems based on team size, the presence or art as something that went right, while only 33% of devel-
absence of publisher, and single or multiplatform, to analyze opers that used a single platform listed this. Additionally,
our results further. 15% of the developers that used a single platform listed this
as something that went wrong, while only 17% of developers
7.1 Small Team vs. Big Team that used multiple platforms listed art as having gone wrong.
When looking at what went right and wrong in teams Therefore, the evidence shows us that games developed for
of 20 or less team members, and teams of more than 20 multiple platforms may be more likely to have better game
team members separately, we can see some clear differences art.
between the two sets. We also found that 20% of developers that worked on mul-
Firstly, 61% of teams with 20 or less members listed game tiple platforms listed marketing as having gone right, and
design as something that went right during development, only 12% of developers that worked on a single platform
while only 50% of teams with more than 20 members listed listed this as having gone right. There is not much varia-
this as something that went right during development. Con- tion between the two groups when looking at marketing as
versely, 25% of both small and large teams listed the team as something that went wrong. In fact, 11% of the developers
having gone wrong during development. This may indicate that used multiple platforms and 10% of the developers that
that smaller teams also produce a better game design. used single platforms listed marketing as going wrong. This
When looking at gameplay and different team sizes, we evidence suggests that marketing is often better in games
found that 42% of large teams listed gameplay as something developed for multiple platforms.
that went right during development, while only 27% of small
teams listed gameplay as having gone right. Supporting this, 8. THREATS TO VALIDITY
16% of small teams listed gameplay as something that went The three threats to validity of this research are: (1)
wrong, and only 9% of large teams listed this as something We could have made mistakes in qualitatively analyzing the
that went wrong. This indicates that having a larger team postmortems. We tried to address this issue by having two
may produce a game with better gameplay. authors discuss their qualitative tagging frequently. (2) The
Lastly, we found that 50% of small teams listed obsta- results were synthesized from self reported postmortems,
cles as something that went wrong, while only 26% of large which could be written by one particular set of game devel-
teams listed obstacles as something that went wrong during opers. However, we find that the games we analyzed were
development. This indicates that small teams may be much very diverse. (3) The authors of these postmortems may not
more likely to encounter obstacles during development than report what actually happened during development or hide
large teams. certain failures. However, on reading these postmortems, we
often find that the authors are very candid. Also in order to [9] Chris Lewis, Jim Whitehead, and Noah
make our experiments more transparent and reproducible, Wardrip-Fruin. What went wrong: a taxonomy of
we have made available all the data, including the actual video game bugs. In Proceedings of the fifth
raw postmortem reports, in our online appendix [6]. international conference on the foundations of digital
games, pages 108–115. ACM, 2010.
9. CONCLUSIONS [10] Morgan McGuire and Odest Chadwicke Jenkins.
We find that we were able to identify both best prac- Creating games: Mechanics, content, and technology.
tices and pitfalls in game development using the informa- CRC Press, 2008.
tion present in the postmortems. Such information on the [11] Mike McShaffry and David Graham. Game Coding
development of all kinds of software would be highly useful Complete. 4 edition.
too. Therefore we urge the research community to provide a [12] Emerson Murphy-Hill, Thomas Zimmermann, and
forum where postmortems on general software development Nachiappan Nagappan. Cowboys, ankle sprains, and
can be presented, and practitioners to report their retro- keepers of quality: How is video game development
spective thoughts in a postmortem. different from software development? In Proceedings of
Finally, based on our analysis of the data we collected, we the 36th International Conference on Software
make a few recommendations to game developers. First, be Engineering, ICSE 2014, pages 1–11, 2014.
sure to practice good risk management techniques. This will [13] Juergen Musil, Angelika Schweda, Dietmar Winkler,
help avoid some of the adverse effects of obstacles that you and Stefan Biffl. Improving video game development:
may encounter during development. Second, prescribe to Facilitating heterogeneous team collaboration through
an iterative development process, and utilize prototypes as flexible software processes. In Systems, Software and
a method of proving features and concepts before commit- Services Process Improvement, pages 83–94. Springer,
ting them to your design. Third, don’t be overly ambitious 2010.
in your design. Be reasonable, and take into account your [14] Juergen Musil, Angelika Schweda, Dietmar Winkler,
schedule and budget before adding something to your de- and Stefan Biffl. A survey on the state of the practice
sign. Building off of that, don’t be overly optimistic with in video game software development. Technical report,
your scheduling. If you make an estimate that initially feels Technical report, QSE-IFS-10/04, TU Wien, 2010.
optimistic to you, don’t give that estimate to your stake- [15] Meiyappan Nagappan, Thomas Zimmermann, and
holders. Revisit and reassess your design to form a better Christian Bird. Diversity in software engineering
estimation. research. In Proceedings of the 2013 9th Joint Meeting
on Foundations of Software Engineering, ESEC/FSE
10. REFERENCES 2013, pages 466–476, 2013.
[1] Gamasutra. http://www.gamasutra.com/. 2015-09-30.
[16] Randy J. Pagulayan, Kevin Keeker, Dennis Wixon,
[2] Erik Bethke. Game development and production. Ramon L. Romero, and Thomas Fuller. The
Wordware Publishing, 2003. human-computer interaction handbook. chapter
[3] Jonathan Blow. Game development: Harder than you User-centered Design in Games, pages 883–906. L.
think. Queue, 1(10):28–37, February 2004. Erlbaum Associates Inc., Hillsdale, NJ, USA, 2003.
[4] Thierry Burger-Helmchen and Patrick Cohendet. User [17] Fábio Petrillo, Marcelo Pimenta, Francisco Trindade,
communities and social software in the video game and Carlos Dietrich. What went wrong? a survey
industry. Long Range Planning, 44(5–6):317 – 343, of problems in game development. Comput.
2011. Social Software: Strategy, Technology, and Entertain., 7(1):13:1–13:22, February 2009.
Community. [18] Robert Purchese. Games are arguably the most
[5] Heather Maxwell Chandler. The Game Production sophisticated and complex forms of software out there
Handbook. Jones & Bartlett Learning, 2008. these days.
[6] Michael Washburn Jr., Pavithra Sathiyanarayanan, http://bit.ly/eurogamer-games-more-complex.
Meiyappan Nagappan, Thomas Zimmermann, and 2015-10-22.
Christian Bird. Appendix to “what went right and [19] Ara Shirinian. Dissecting the postmortem.
what went wrong”: An analysis of 155 postmortems http://gamasutra.com/view/feature/134679/
from game development. Technical Report dissecting the postmortem lessons .php. 2015-09-30.
MSR-TR-2016-6, February 2016. [20] Patrick Stacey and Joe Nandhakumar. A temporal
research.microsoft.com/apps/pubs/?id=262289. perspective of the computer game development
[7] Jussi Kasurinen, Jukka-Pekka Strandén, and Kari process. Information Systems Journal, 19(5):479–497,
Smolander. What do game developers expect from 2009.
development and design tools? In Proceedings of the [21] Rob van der Meulen and Janessa Rivera. Gartner Says
17th International Conference on Evaluation and Worldwide Video Game Market to Total $93 Billion in
Assessment in Software Engineering, pages 36–41. 2013. http://www.gartner.com/newsroom/id/2614915.
ACM, 2013. 2015-10-22.
[8] Annakaisa Kultima and Kati Alha. Hopefully [22] Michael Thornton Wyman. Making Great Games: An
everything i’m doing has to do with innovation: Insider’s Guide to designing and developing the
Games industry professionals on innovation in 2009. In World’s Greatest Games. CRC Press, 2012.
Games Innovations Conference (ICE-GIC), 2010
International IEEE Consumer Electronics Society’s,
pages 1–8, Dec 2010.