Advances in Computer Entertainment: Dennis Reidsma Haruhiro Katayose Anton Nijholt
Advances in Computer Entertainment: Dennis Reidsma Haruhiro Katayose Anton Nijholt
Advances in Computer Entertainment: Dennis Reidsma Haruhiro Katayose Anton Nijholt
Haruhiro Katayose
Anton Nijholt (Eds.)
LNCS 8253
Advances in
Computer Entertainment
10th International Conference, ACE 2013
Boekelo, The Netherlands, November 2013
Proceedings
123
Lecture Notes in Computer Science 8253
Commenced Publication in 1973
Founding and Former Series Editors:
Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board
David Hutchison
Lancaster University, UK
Takeo Kanade
Carnegie Mellon University, Pittsburgh, PA, USA
Josef Kittler
University of Surrey, Guildford, UK
Jon M. Kleinberg
Cornell University, Ithaca, NY, USA
Alfred Kobsa
University of California, Irvine, CA, USA
Friedemann Mattern
ETH Zurich, Switzerland
John C. Mitchell
Stanford University, CA, USA
Moni Naor
Weizmann Institute of Science, Rehovot, Israel
Oscar Nierstrasz
University of Bern, Switzerland
C. Pandu Rangan
Indian Institute of Technology, Madras, India
Bernhard Steffen
TU Dortmund University, Germany
Madhu Sudan
Microsoft Research, Cambridge, MA, USA
Demetri Terzopoulos
University of California, Los Angeles, CA, USA
Doug Tygar
University of California, Berkeley, CA, USA
Gerhard Weikum
Max Planck Institute for Informatics, Saarbruecken, Germany
Advances in
Computer Entertainment
10th International Conference, ACE 2013
Boekelo, The Netherlands, November 12-15, 2013
Proceedings
13
This copy belongs to 'VANC03'
Volume Editors
Dennis Reidsma
University of Twente, Human Media Interaction/Creative Technology
Drienerlolaan 5, 7522 NB Enschede, The Netherlands
E-mail: d.reidsma@utwente.nl
Haruhiro Katayose
Kwansei Gakuin University, School of Science and Technology
Department of Human System Interaction
Gakuen Sanda 2-1, Sanda 669-1337, Japan
E-mail: katayose@kwansei.ac.jp
Anton Nijholt
University of Twente, Human Media Interaction
Drienerlolaan 5, 7522 NB Enschede, The Netherlands
E-mail: anijholt@cs.utwente.nl
the proceedings of ACE 2012, creating has always been an important form of
entertainment. People paint for a hobby, play music, build model airplanes, or
write amateur poetry in their free time. Just for the fun of designing and creating
their own entertainment; the final result may be less important than the process.
Tinkering can also be a strong source of learning, something that has been known
at least since the seminal work of Seymour Papert. In a video lecture on Carnegie
Commons, John Seely Brown suggests that the role of a teacher partly shifts
from imparting knowledge to building a learning community. Clearly, tools for
programming and physical computing can serve as tinkering materials in such
a community, and maybe there are further roles that computer entertainment
technology can play in building and facilitating such a learning community.
These thoughts are not only reflected in a number of papers and extended
abstracts in these proceedings, but also in several of the additional activities
that were organized during this year’s conference. There were panels, workshops
in which the participants sit down together to actively make things or to discuss
the role (and challenges!) of tinkering in scientific education, the Kids’ Workshop
Track featuring activities for children making stories, animations, and elements
for games, and there were special efforts to include more students at various
levels in their education in the conference. All this took place at the beautiful
resort Bad Boekelo, situated in the pastoral countryside of Twente.
Of course, there cannot be a conference without the submission of many good
papers. This year, 133 papers were submitted to the various tracks. With an ac-
ceptance rate of 22% for long regular presentations, and 54% for all contributions
including extended abstracts for the poster presentations, these proceedings rep-
resent the very interesting and relevant work currently carried out by the ACE
community.
Like every year, many people worked hard to make this 10th edition of ACE a
success. To the Program Committee, reviewers, authors, track chairs, workshop
organizers, delegates visiting the conference, and the sponsors supporting the
conference in various ways: Thank you! We are proud to have served as this
year’s general and program chairs to bring everything together in the lovely
countryside of Boekelo, The Netherlands!
Steering Committee
Adrian David Cheok Keio University, Japan and NUS, Singapore
Masahiko Inami Keio University, Japan
Teresa Romão CITI, FCT, Universidade Nova de Lisboa,
Portugal
General Chair
Anton Nijholt University of Twente, The Netherlands
Program Chairs
Haruhiro Katayose Kwansei Gakuin University, Japan
Dennis Reidsma University of Twente, The Netherlands
Poster Chair
Günter Wallner University of Applied Arts, Vienna, Austria
Workshop Chair
Randy Klaassen University of Twente, The Netherlands
Local Chair
Gijs Huisman University of Twente, The Netherlands
Program Committee
A. Augusto Sousa Angelika Mader
Aderito Marcos Ann Morrison
Adrian Cheok Annika Waern
Adrian Clark Anton Nijholt
Akihiko Shirai Antonio Coelho
Alan Chatham Arjan Egges
Ana Veloso Athanasios Vasilakos
Andrei Sherstyuk Atsushi Hiyama
Additional Reviewers
Ana Tajadura Philip Voglreiter
André Pereira Philipp Grasmug
Andreas Hartl Raphael Grasset
Anton Eliens Rui Craveirinha
Christian Pirchheim Samuel Silva
Daniel Rea Simon Hoermann
Doros Polydorou Stefan Hauswiesner
Jens Grubert Stefan Liszio
Katharina Emmerich Takao Watanabe
Kening Zhu Tom Penney
Marielle Stoelinga Viridiana Silva-Rodriguez
Markus Steinberger Zsófia Ruttkay
Collaboration
ACE 2013 at the University of Twente, The Netherlands, was organized in part-
nership with the Centre for Telematics and Information Technology, The Nether-
lands Organisation for Scientific Research (NWO), the SIKS Graduate School,
and Springer Publishing.
Yvonne Rogers
Ali Israr
Disney Research
Arjan Egges and Kaśka Porayska-Pomsta
1 Session Overview
Over the last decade serious gaming has become a prominent and important
field of research. Serious games are increasingly used to support learning of and
training in diverse and traditionally unrelated domains. These domains range
from formal learning of traditional subjects such as mathematics, vocational
training for professions such as air pilots or dentists, coaching individuals in
acquiring better job interview skills, to therapeutic applications which aim to
support the development of skills associated with socio-emotional coping, e.g. in
schizophrenia or autism. Serious games leverage both the intrinsic motivation
associated with playing computer games as well as a serious intent to furnish
their players with skills that are useful in the real world. As such, these games
present their own set of challenges to game designers and developers. First, as
most serious games will have some sort of educational goal, the design of a game
should ensure that these educational goals are reached when someone plays the
game. Second, a serious game should be able to measure the success of the player
within the game itself. Although tracking a player’s progress is something that
any game should do, for serious games this is even more important to get right,
since the quality of the training in part determines the performance of the trainee
in the real world.
Serious games are also challenging from the technological and engineering
point of view. In many cases, serious games use specific hardware such as 3D
screens, plates that can measure exerted forces, motion trackers, or 3D sound
generators. Incorporating all of these modalities into a coherent and seamless
game environment is complex. Designing and developing serious games becomes
even more challenging when one wants to incorporate capabilities such as track-
ing of individual players over different sessions, allowing for simultaneous par-
ticipation of multiple players over a network connection. Furthermore, games
often need to be adapted to different languages and cultures—this process is
Virtual Human Technology Lab, Dept. of Information and Computing Sci-
ences, Utrecht University, the Netherlands. Website: http://vhtlab.nl. Email:
j.egges@uu.nl
London Knowledge Lab, Institute of Education, United Kingdom. Website:
http://http://www.lkl.ac.uk/. Email: K.Porayska-Pomsta@ioe.ac.uk
2 Program Committee
– Sylvester Arnab, Serious Games Institute, UK
– Jannicke Baalsrud Hauge, BIBA - Bremer Institut für Produktion und Lo-
gistik GmbH, Germany
– Francesco Bellotti, University of Genoa, Italy
– Ionut Damian, Augsburg University, Germany
– Cathy Ennis, Utrecht University, the Netherlands
– Igor Mayer, TU Delft, the Netherlands
– Nicolas Sabouret, LIMSI-CNRS, France
Long Presentations
Web Analytics: The New Purpose towards Predictive Mobile Games . . . . 1
Mathew Burns and Martin Colbert
An Author-Centric Approach to Procedural Content Generation . . . . . . . 14
Rui Craveirinha, Lucas Santos, and Licı́nio Roque
Providing Adaptive Visual Interface Feedback in Massively Multiplayer
Online Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Chris Deaker, Masood Masoodian, and Bill Rogers
Persuasive Elements in Videogames: Effects on Player Performance and
Physiological State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Luı́s Duarte and Luı́s Carriço
Evaluating Human-like Behaviors of Video-Game Agents Autonomously
Acquired with Biological Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Nobuto Fujii, Yuichi Sato, Hironori Wakama, Koji Kazai, and
Haruhiro Katayose
Comparing Game User Research Methodologies for the Improvement of
Level Design in a 2-D Platformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
Marcello Andres Gómez Maureira, Dirk P. Janssen,
Stefano Gualeni, Michelle Westerlaken, and Licia Calvi
Touch Me, Tilt Me – Comparing Interaction Modalities for Navigation
in 2D and 3D Worlds on Mobiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Wolfgang Hürst and Hector Cunat Nunez
Virtual Robotization of the Human Body via Data-Driven Vibrotactile
Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Yosuke Kurihara, Taku Hachisu, Katherine J. Kuchenbecker, and
Hiroyuki Kajimoto
BOLLOCKS!! Designing Pervasive Games That Play with the Social
Rules of Built Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Conor Linehan, Nick Bull, and Ben Kirman
Cuddly: Enchant Your Soft Objects with a Mobile Phone . . . . . . . . . . . . . 138
Suzanne Low, Yuta Sugiura, Kevin Fan, and Masahiko Inami
GuideMe: A Mobile Augmented Reality System to Display User
Manuals for Home Appliances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Lars Müller, Ilhan Aslan, and Lucas Krüßen
Short Presentations
The Art of Tug of War: Investigating the Influence of Remote Touch
on Social Presence in a Distributed Rope Pulling Game . . . . . . . . . . . . . . . 246
Thomas Beelen, Robert Blaauboer, Noraly Bovenmars, Bob Loos,
Lukas Zielonka, Robby van Delden, Gijs Huisman, and
Dennis Reidsma
Extended Abstracts
Development of a Full-Body Interaction Digital Game for Children to
Learn Vegetation Succession . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
Takayuki Adachi, Hiroshi Mizoguchi, Miki Namatame,
Fusako Kusunoki, Masanori Sugimoto, Keita Muratsu,
Etsuji Yamaguchi, Shigenori Inagaki, and Yoshiaki Takeda
How to Make Tangible Games and Not Die in the Attempt . . . . . . . . . . . . 513
Eva Cerezo, Javier Marco, and Sandra Baldassarri
1 Introduction
Prediction is the simple designation to a possible outcome which can lead to one of
two eventualities, right or wrong, win or lose [1]. Many businesses and theories ex-
ploit either possibility. In the instance of economics or gambling a prediction is sup-
ported on the supposed understanding of risk [2]. Placing a prediction in conjunction
with a commodity is nothing new. However this concept has previously not been
applied to the usage of internet traffic known as web analytics. Web analytics has
remained confined to the repetitive accumulation and conclusion of online activity
[3], where its predefined purpose has not yet been explored beyond.
The current wide acceptance and ever growing popularity of mobile technology
over many cultural and social boundaries has witnessed a level of unity with a user’s
life [4]. Mobile technologies ability to allow access to games and applications beyond
the geographical limitations of desktop computers presents a unique conduit to assess
the effective entertainment of an analytic based prediction game. The domestication
of users towards the technology has caused an “industrial revolution of data” [5]. The
scale of this data explosion can be observed in a singular instance such as social me-
dia. Facebook in a single day alone generates sixty terabytes of data [6]. Potentially
harnessing these scales of data in the form of a game presents an intriguing pursuit of
research.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 1–13, 2013.
© Springer International Publishing Switzerland 2013
The purpose of this paper is to explore how web analytics can be used beyond its
previous connotations by employing prediction to develop a playful and entertaining
mobile game for users. It specifically refers to the following objectives:
3 Web Analytics
change. Analytics have remained exclusively to these main areas of use and therefore
confined to this limited framework. This encourages new innovations to build upon
what analytics is currently defined as, side step its original use by considering how it
might be used in a previously unseen manner.
Tools for analytics reside heavily on the past for visitor behavior [14]. While the past
measurement and analysis of data can influence an improved future user experience,
analytics does not provide much of a future consideration. For instance, a result or
value based upon a future outcome. Generally analytics is not concerned with the
future behavior until collected in the present. Whilst analytics has remained rigid in
this regard, businesses have made use of the future towards a prediction.
It is not uncommon for businesses to attempt to predict outcomes for its benefit.
Google, Best Buy and P&G are known to analyze employee data in order to isolate
those that are most likely to gain advantages for the business such as sales. The same
can be gained for market segments with promotions and pricing strategies [15].
Both examples of analytic tools and businesses only focus on either the past or the
future. Neither makes use of both past and future predictions. The proposed mobile
game looks to meet this.
4 Gambling on Prediction
prediction is made [2]. Risk has been seen to have a profound effect on behavioral
patterns. In particular, a recent win can provide a strong influence to take further risk
[18]. The same can be true of losing causing a need to play [19]. It describes the im-
agery of an individual being drawn to place coins into a gambling machine but ulti-
mately it shows the potential that risk has towards playfulness with a prediction game.
Clearly, the commodity placed against the prediction and the risk involved holds a
certain level of playfulness to invoke continued use for industries such as gambling.
The similarity that placing a prediction against online traffic has to gambling empha-
sizes the need to consider these points within the design of a mobile game. It points
towards the consideration of how such a design might affect the user and specifically
the question, when does playfulness end and addiction begin?
vanishes and addiction issues takeover. Clearly, addiction has a negative connotation
associated with it, which is not in the interests of the user therefore it is noteworthy to
emphasize the reluctance to employ such concepts within a game design.
5 Data-Driven Games
It makes use of a normal skill of searching for an answer into a playful activity.
The pursuit of an answer encourages knowledge to be gained through discovery [27].
The development and change towards this new type of game has been as a result of
recent findings. A significant effect in a user’s ability to search effectively has been
highlighted when appropriately engaged on an educational level [28]. Learning in an
exploratory fashion can ensure that the search skills are retained far beyond the stage
it was taught. This discoverable concept of enforcing a skill has begun to be seen
amongst other business’s products such as Bing, which demonstrates the popularity to
engage with users in a playful manner [29]. While it has only recently begun to be
integrated into games, other applications have made use of the same concept but
through a different context.
Another project by Google known as the wonder wheel allowed users to visually
identify possible words or phrases that may relate to those entered.
From the word entered, a visual link to possibly related phrases could be displayed.
Users could follow a single word through a long list of corresponding meanings that
may appear within a search. It enabled marketers to identify the most relevant key-
word in order to gain the highest number of visitors. Essentially the user gained
knowledge through a process of discovery [27]. The application has since been rede-
signed and launched as the Contextual Targeting Tool [30] after inspiring academic
papers and success amongst users [27]. The new application has removed the visual
representation used to find a word [25], which appears to have removed the core of
discovering knowledge.
The commonality that the A Google A Day game and the Wonder Wheel applica-
tions shared was the obvious ability to allow the user to discover knowledge through
their own means. Certain contradictions could be observed between the two, where a
visual representation was a crucial feature whilst it was not relevant to the other as a
result of its context. Ultimately, each instance demonstrated the power of using data
and transforming it into an innovative product towards a certain context.
6 Design
making it viewable to mobile devices. The mobile game made use of a JavaScript
based native app as the front end to the user where results and data could be passed
back and forth between an online web service via Ajax calls. Commonly, both ver-
sions made use of the Google Analytics Core Reporting API to channel data from a
live analytics account to their users. The analytics account was linked to an online
blog being supplied data by regular visitors.
The clear dependence on design and development required the repetitive approval
of core aspects. A Rapid Application Development methodology was applied allow-
ing for referral and demonstration to various users. As a result, the initial development
became an evolutionary process moving from a web application to a game which was
more enjoyable, usable and playful.
As can be seen from Figure 1, there is a significant difference between the two ver-
sions. The web application provided the user with the necessary past data for the re-
view and investigation of possible outcomes that could allow for a prediction to be
made. The user could identify particular trends through the discovery of data. De-
pending upon the time scale selected, a user would be required to return to receive
their success or failure of their prediction much like how spread betting might be
used.
The mobile game made use of the web analytics in a different regard to time. In-
stead of waiting for occurrences to elapse in order to retrieve the verdict of a win or
loss, questions would only be asked of the user when online events had already trans-
pired therefore resembling a web design quiz.
The traits regarding risk and commodities have been included within both versions.
The fascinating psychological trait to continue to play in the face of risk when win-
ning or losing [18] [19] was intertwined with a commodity known as Stakes per Point.
This attempted to create the illusion of the potential to win/lose therefore presenting a
level of risk as can be seen in Figure 2.
When a prediction is submitted, a result is presented to the user. Through the form
of a win or a loss, the user is rewarded or punished for their prediction. In order to
gain the result a calculation was performed.
Simply, the calculation is represented by three key aspects in order to provide a re-
sult. As stated within (1) the overall outcome (a), the difference between the predic-
tion and the actual outcome (b) and the commodity placed (c) is required to provide a
result (d).
The calculation can be broken down further as stated within (2). An overall outcome
is gained by subtracting from the value that occurred within the time frame prior to
the prediction (b). For example, the user places a prediction within a certain week
therefore the week before shall be used. This value is subtracted from the actual value
(a) that transpired within the time frame selected.
As stated within (3) the prediction (c) is comprised of the actual value (a) in addi-
tion to the predicted change (f). The actual value (a) can then be subtracted from the
prediction (b). The value of this process is then multiplied by the commodity (d) to
provide the result (e) as shown below:
((a - b) – ((c) - a)) x d = e (2)
7 Methodology
The need to measure the success or failure of each version requires the collection
of qualitative as well as quantitative data. A usability methodology was employed
due to its inherent logical nature. Specifically, the Common Industry Format (CIF) to
aid testing as a formal methodology making it applicable for testing with users and
software [26].
At certain stages of the development process, formative feedback would be
gained from certain users in order to refine the design through RAD. At the summa-
tive stage, a field based empirical evaluation allowed for the constant reporting of
quantitative measurements to be gained. Building the necessary features into each
version, certain metrics such as time spent, success and path of the user could be con-
cluded. In conjunction, survey evaluation was utilized to cover the qualitative aspect
of testing after a user’s completion of tasks within an electronic format. This entire
process began when a participant showed an interest to take part within the study; an
electronic email was sent to them including the necessary instructions to carry out the
tasks.
In the same way that a user will make use of a website, an application or game
where the developer will not determine who necessarily uses it, random sampling was
used. As conveyed by its name, it acted as an appropriate means to keep testing ran-
dom and therefore unbiased.
The web application was tested against two random samples of five participants. In
order to test the validity of risk in regards to winning and losing, the calculation men-
tioned was offset to cause five participants of each outcome. Whether this caused an
effect to continue playing or not when winning or losing would be reflected within the
summative feedback provided within the survey evaluation.
The mobile game was tested against a singular group of five participants in accor-
dance to the previous versions methodology for consistency and reduced bias of
results. As such, the main difference within testing was the design of each version.
8 Findings
Learning from the issues of the web application, the mobile game was designed to
include an intuitive guide to step the user through making a prediction. Differences
applied like this and the alternate use of analytics in regard to time discussed pre-
viously, opens versions to be scrutinized comparatively.
Testing of the mobile game saw the five participant’s progress through each task
quicker than the last with individual means between 120 and 166 seconds whilst gain-
ing an accumulative mean of 386.6 seconds. It indicated participants learning through
repetition from the guided stages. It demonstrates that the mobile game was accepted
once the initial workings were understood. A design that is hard to use may not exhi-
bit this quality which may have been indicative of the web application.
When comparing the mobile game’s participant group against that of the web ap-
plication in order to identify participants taking a risk when winning (t(10 = 0.869; p
> 0.05) or losing (t(10 = 1.131; p > 0.05), there is no significance for the tasks carried
out. Within both accounts of each version, a significance cannot be found to prove a
possibility that users are drawn to play a game due to their trend of winning or losing.
The survey evaluations overall total for the mobile game (257 / 56.4%) and the
web application (Group 1 = 249 / 47.5%, Group 2 = 229 / 43.6%) provides the game
with a lead of 10.9%. It generally highlighted a higher level of satisfaction with the
mobile game compared to the web application. Additionally, an independent t-test
between the winning group for the web application against the mobile game has found
significances for certain questions regarding ‘addiction’ (t(10) = -2.530; p < 0.05) and
the ‘ability to find results’ (t(10) = -3.939; p < 0.05). Both the ‘addiction’ (M = 2.60,
S.D. = 0.548) and ‘finding results’ (M = 5.80, S.D. 1.304) for the mobile game were
significantly higher than the web application (M = 2.80, S.D. = 1.095) (M = 1.80,
S.D. = 0.447). It demonstrated a general preference to use the mobile game as well as
finding it easier to find results and play when guided through a staged design.
9 Conclusion
The design and development of two distinctive versions making use of web analytics
in a playful manner with prediction has identified:
• A potential demographic profile (18 – 24 and an expert user) has been formed of a
user that is most likely to find this type of mobile game playful.
• Confusion was associated with the web application. A method of guiding the user
through the stages as seen by the mobile game is required in order to simplify the
design and therefore encourage use.
• The literature [19] based around a win or a loss having the potential to encourage
further risks to be taken has been disproven.
• A mobile game making use of web analytics from events that have elapsed rather
than having the user return to gain an answer appears to be the most suitable.
The findings have supported the feasibility of a web analytics mobile game. This
research has demonstrated that web analytics can be broken out of its predefined pur-
pose and used for more unique and creative means.
Whilst some of the features identified to encourage use such as risk were disproven
for suitability, it was only tested against small participant groups. Carrying out addi-
tional tests could elaborate on aspects and potentials that placing predictions on web
analytics can bring for the user.
The findings present a strong basis for future research. Already, testing has begun
using an extensive dataset from Kingston University’s live web analytics to provide
questions and answers to drive predictions made from the mobile game. With a larger
scale of participants across the mobile marketplace, this research leaves the question,
what other contexts can web analytics be used for?
References
1. Danielson, M.: Prediction Markets. In: MAT, vol. 5900 (2010)
2. Hayden, B., Platt, M.: Gambling for Gatorade: Risk-sensitive Decision Making for Fluid
Rewards in Humans. Anim. Cogn. 12, 201–207 (2009)
3. Ledford, K., Teixeira, J., Tyler, M.: Google Analytics. John Wiley & Sons
4. Rose, K.: The Long and Short of it, Herptile
5. Hellerstein, J.: Programming a Parallel Future. In: EECS, vol. 144 (2008)
6. Ashish, T., Suresh, A., Namit, J., Raghotham, M., Zheng, S., Dhruba, B., Joydeep, S., Hao,
L.: Data Warehosuing and Analytics Infrastructure at Facebook. In: SIGMOD (2010)
7. Yechiam, E.: Risk Attitude in Decision Making: In Search of Trait Like Con-structs. Topic
in Cognitive Science 3, 166–186 (2011)
8. Kaushik, A.: Web Analytics: An Hour a Day. John Wiley & Sons (2007)
9. Liberatore, M., Luo, W.: The Analytics Movement: Implications for Opera-tions Research.
Informs., 1–12 (2010)
10. W3CTechs.: Usage of Traffic Analysis Tools for Websites (2012),
http://w3techs.com/technologies/overview/
traffic_analysis/all
11. Cutroni, J.: Google Analytics. O’Reilly Media (2010)
12. Lewis, C., Wardrip-Fruin, N.: Mining Game Statistics from Web Services: A World of
Warcraft Armoury Case Study. In: Proceedings of the Fifth Internationsal Conference on
the Foundations of Digital Games, pp. 100–107 (2010)
13. Kim, K., Gunn, D., Schuh, E., Phillips, B., Pagulayan, R., Wixon, D.: Tracking Real Time
User Experience. In: Proceedings of the Twenty Sixth Annaul Conference on Human Fac-
tors in Computing Systems, pp. 443–351 (2008)
14. Sanfilippo, A., Cowell, A., Malone, C., Riensche, R., Thomas, J., Unwin, S., Whitney, P.,
Wong, P.: Technosocial Predictive Analytics in Support of Natu-ralistic Decision Making.
In: Proceedings of NDM 2009 (2009)
15. Davenport, T., Harris, J., Shapiro, J.: Competing on Talent Analytics. In: The Big Idea.
Hardvard Business Review
16. Wolfers, J., Zitzewitz, E.: Prediction Markets in Theory and Practice. National Bureau of
Economic Research 12084, 1–16 (2006)
17. Nagar, Y., Malone, T.: Making Business Prediction By Combing Human and Machine In-
telligence in Prediction markets. In: Thirty Second International Conference on Informa-
tion Systems, pp. 1–16 (2011)
18. Hayden, B., Nair, A., McCoy, A., Platt, M.: Posterior Cingulate Cortex Me-diates Out-
come-Contingent Allocation of Behaviour. PMC 60, 19–25 (2009)
19. Ottaviani, M., Sorensen, P.: Noise, Information and the Favorite-Longshot Bi-as in Pari-
mutiel Prediction. American Economic Journal Microeconomics 2, 58–85 (2010)
20. Griffiths, M.: Technological Addictions. Clinical Psychology Forum 76, 14–19 (1995)
21. Kairouz, S., Paradis, C., Nadeau, L.: Are Online Gamblers More At Risk Than Offline
Gamblers. Cyberpsychology, Behaviour and Social Networking 1, 1–6 (2011)
22. Stekler, H., Sendor, D., Verlander, R.: Issues in Sports Forecasting. International Journal
of Forecasting 26, 606–621 (2010)
23. Jansen, B., Mullen, T.: Sponsored Search: An Overview of the Concept, History, and
Technology. Electronic Business 6, 114–131 (2008)
24. Google: A Trivia Game Where Using Google is Allowed,
http://googleblog.blogspot.co.uk/2011/04/trivia-game-where-
using-google-is.html
25. Google: The New a Google a Day, http://agoogleaday.com/
26. CSQI Wonder Wheel, http://www.hmtweb.com/marketing-blog/
27. Mirizzi, R., Ragone, A., Di Noia, T., Di Sciascio, E.: Semantic Wonder Cloud: Explorato-
ry Search in DBpedia. In: Daniel, F., Facca, F.M. (eds.) ICWE 2010. LNCS, vol. 6385, pp.
138–149. Springer, Heidelberg (2010)
28. Moraveji, N., Russel, D., Bien, J., Mease, D.: Measuring Improvement I User Search Per-
formance Resulting from Optimal Search Tips. In: Proceedings of the 34th International
Conference on Reaseagc and Development in Information Retrieval, pp. 355–364 (2011)
29. Bateman, S., Teevan, J., White, R.: The Search Dashboard: How Reflection and Compari-
son Impact Search Behaviour. In: CHI, pp. 1–10 (2012)
30. Google Contextual Targeting Tool,
http://www.google.com/ads/innovations/ctt.html
31. Bevan, N.: Extending Quality in Use to Provide a Framework for Usability Measurement.
In: Kurosu, M. (ed.) HCD 2009. LNCS, vol. 5619, pp. 13–22. Springer, Heidelberg (2009)
32. Silva, E., Pires, L., Sinderen, M.: On the Design of User-Centric Supporting Service Com-
position Environments. In: ITNF, pp. 1–6 (2010)
1 Introduction
Founding pillars of videogame theory come from different fields and are mostly
general in nature, and while certainly useful for guidance in game design, they
tend to lack a strong quantitative basis [Pedersen et al., 2010]. The past years
have seen the rise of methods – such as Gameplay Metrics [Kim et al., 2008] –
that can better inform creators during the videogame design process, by provid-
ing increasingly meaningful and objective data on how videogames are effectively
played and experienced. Furthermore, procedural content generation aimed at
improving user-experience (e.g. Experience-Driven Procedural Content Gener-
ation, EDPCG for short[Yannakakis and Togelius, 2011]) has become a reality,
allowing designers access to tools that provide them active support through gen-
eration of game content. These advances deliver potential for creative processes
that deliver more expressive designs. However, thus far, there are few examples
of how these methods can be meaningfully incorporated in design contexts, as
ways of facilitating or expanding production of videogame artifacts. What this
paper describes then is an alternative procedural content generation approach,
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 14–28, 2013.
c Springer International Publishing Switzerland 2013
created to help creative game design and production processes. It seeks to utilize
automatic generation and evaluation of artifacts to increase both the efficiency
and effectiveness of the design process, whilst empowering authorial exploration
of the design space and furthering designers’ creative agendas for their artifacts.
The remaining paper is thus structured: section 2 gives a brief overview of
the main EDPCG literature, and how it was framed in this research; section 3
details this research approach, its architectural details, and how it is intended
to use these new technologies, while section 4 details a first trial to implement
this solution. Finally, in the last section, results are discussed and future work
for this project is described.
2 Literature Review
Several benefits have been studied and verified in relation to user-centered de-
sign philosophies, namely in terms of increased system quality thanks to more ac-
curate requirements, heightened production efficiency, improved user acceptance,
amongst others [Kujala, 2003]. But we share with other researchers [Steen et al.,
2007] reasonable concerns over how extreme user-focused approaches can impair
innovation and creativity. Even proponents are careful in how they frame a user-
centered approach to design, maintaining a focus on designer’s intervening in
the process [Kujala, 2003]. Indeed, user-centered design presupposes designers
to enter into meaningful dialogical debate with users, in such a way that knowl-
edge is shared from both ends so that design decisions are rightfully informed in
a collaborative manner [Steen et al., 2007], never dictated exclusively by users’
preferences.
Game designers and producers, like other members of the design field, run the
risk of “reducing experience to the mere “pleasure due to the feel of the action”
[Hassenzahl, 2011], i.e., thinking of user satisfaction as opposed to user expe-
rience. For a number of reasons, players may not always like or feel something
positive in their experience, despite designers’ creative agenda being fulfilled.
By focusing procedural tools on, for example, creating frictionless artifacts that
can maximize ‘fun’, researchers seem to be establishing not only the means to an
end, but the very end itself of the creative process. Consumers can be superficial,
dismissive, indecisive and incapable of adequately expressing their thoughts and
emotions [Gladwell, 2005] – and as such to put EDPCG quality measurements
dependent only on their subjective evaluation may not be the best approach for
creators.
As such, this paper proposes an Author-centric approach to Experience-
Driven Procedural Content Generation for games, where the quality evaluation
is measured in reference to designer’s expectations of what gameplay behaviors
should be elicited. Artistic expression has, for the most part, been sustained by
authorial pursuit and vision, despite not always having been well accepted by
audiences or critics [Gombrich, 2009]. So we propose to put the focus back on
authors’ formal considerations instead of users’ own, while trying to maintain the
usefulness of using procedural generation tools. This view seeks to empower de-
signers and award them greater creative control, while permitting more effective
ways of tailoring their artifact to mediate specific gameplay patterns according
to their agenda.
In terms of how this process is carried out, figure 1 shows the proposed archi-
tecture from a conceptual point of view. To start using the approach, the design
team must provide 3 items: a game Archetype, a set of Target Indicators and
respective target values, and a definition of Parameters that can be to generate
new variations of the Archetype. These 3 elements are that which determines
the overall specification of the problem they wish to solve in what regards to
their game, and therefore the differentiating aspect of this approach. This is, in
essence, the definition of the design domain in which the actual EDPCG opti-
mization method will operate, and the notion of ‘Quality’ to be applied to the
generated content.
Content quality is measured as the inverse of the difference between a set of
Target Gameplay Indicator values chosen by Designers, and the values for those
Gameplay Indicators that generated artifacts mediate to real players. Parame-
ters are the set of design elements, implemented by designers, that they forfeit
direct control of, and which a platform that implements this architecture will
then use to generate new configurations. There is the assumption here that au-
thors choose these in a sensible manner, so that they affect the desired target
gameplay. The Archetype is the prototype they designed, deprived of the Param-
eter components, i.e. the base artifact of which all variations will evolve from.
In continuing with the designer-centric stance, it is assumed that this Archetype
is implemented by designers from the ground up. Naturally, all these elements
involve a configuration process carried through a platform user-interface (de-
termining, for example, the types of target indicators, their domain, etc.), and
direct integration by means of computational interfaces (methods for publishing
and subscribing metrics, for instance).
Once the 3 designer-defined elements are forwarded and integrated into a
platform, its actual functioning begins – that is that A-cycle in figure 1. At
that point, the platform runs without need of intervention from designers and
it is at this stage that actual procedural generation tools are used. First, there
is a phase of candidate generation, where new Parameter set values are found
by means of a search algorithm; in this case, a Genetic Algorithm (GA) with
its selection, recombination and mutation procedures. Secondly, by integrating
new Parameters into the Archetype, new candidate artifacts become ready for
playing, at which point players play these alternatives, and from their experience
with them, metrics are extracted, so as to characterize their gameplay accurately.
Thirdly, these are compiled into the Gameplay Indicators that designers defined,
and compared with their target values, by means of a distance metric. Finally,
if the best candidate’s quality is not sufficiently close to the values, then this
micro-cycle repeats with a new GA iteration, with each of the chromosomes being
awarded a fitness proportional to candidate’s quality values. Once the A-cycle
ends, a candidate that matches the target indicators, as well as all gameplay
indicators data, are given as output to designers, so they can continue their
iterative design process, by choosing whether or not the end-result is satisfactory.
If it is not, designers can fine-tune either of the elements provided, forwarding a
new design specification to the platform, in a new iteration of the B-cycle.
4 Experimental Trial
To put the approach into effect, a design case materialization was needed. The
main goal for this research was to test the feasibility of this approach by imple-
menting the architecture into a prototype software platform, and start to map
its practical constraints. The idea of this first trial was to take the ‘Infinite Mario
Bros’ level generator (also used in [Pedersen et al., 2009]) and impose a signifi-
cant variation on the original ‘Super Mario Bros.’ design. Imagine a hypothetical
author that wanted to make a hyped-up version of Mario, with high-speed and
high-stakes gameplay, where player experience would be akin to that of doing
a speed-run trial. So, this designer changed the core-physics of Mario accord-
ingly: Mario jumps double the height, can shoot more fireballs, and his default
state is Super Mario (though he is still turned to normal as he is hurt, as in
the original game). Now, these subtle changes in the game’s design affect the
experience – the game becomes a caricature of the original, both more frantic,
exaggerated and difficult to play. The core gameplay system is completed, even
though the experience is far from polished; the altered ‘Infinite Mario Bros.’ base
game thus serves as this trial’s Archetype. The designer wants users to play as
he envisioned: very fast, with a high rhythm of actions, all whilst maintaining it
playable enough so that players can conquer levels without much trying.
Now he must define target indicator values for gameplay metrics: Number
Tries for conquering a level, as a measure for player difficulty and challenge;
number of player triggered Actions per Second, as a measure of the experience’s
rhythm; and level play-through Time, as a measure of play-speed. He establishes
a target mean value for each of these metrics, as well as suitable maximum
range value that defines acceptable variation for these metrics; see table 1 for
the tested values. That is his experience ideal, materialized into target indicators
for gameplay behavior. As a measure of success for this first research trial, it was
established beforehand that the system would be deemed feasible if it was shown
to be capable of generating a best candidate that, at least, could statistically
guarantee indicator values between the target and a maximum of a quarter of
the accepted range for a population of players (see table 1).
Eight Parameters with a varied range of design objectives were chosen to be
permitted to be altered by the platform: game speed (24–48 cycles per second),
number of cannons (0–5), number of goomba enemies (0–30), number of koopa
Table 1. The three gameplay indicators that the algorithm meant to optimize, their
target value, and the maximum variation that was considered. Also, the minimum
condition for success for this trial – to have the best candidate’s indicator values nested
in the success range.
that is a vector of real values, each a gene corresponding to a given game design
parameter which can be varied. For each generation, the population size was of
12 individuals, plus one extra slot reserved for elitist maintenance of the best
individual. Uniform crossover was used as a recombination method, with a rate
of 90%. Mutation operated by replacing a gene to a new random value, and had a
starting probability of 5%, and that was increased 0.5% in each generation (these
seem large values, but one must consider that for this test, there were only 8
genes and 12 individuals, so lower values would practically exclude mutation
from ocurring). Selection was made by tournament for pairs of chromosome.
Evaluation of each candidate’s fitness is as follows: in each evaluation phase,
all the chromosome population is converted into full sets of design specifications
that are stored in the server. Players use a client application which automatically
downloads the next available set, which users then play and experience, after
which the metrics data from their play session is uploaded to the server. It is
from the metrics data that fitness is directly calculated, using the formula 1:
N
1
f itness = q(i) (1)
i
N
⎧
⎨ |tgt(i)−avg(i)|
(1 − ), tgt(i) − avg(i) ≤ max(i)
q(i) = max(i)
(2)
⎩0, tgt(i) − avg(i) > max(i)
where N is the total number of indicators, tgt(i) the target value for a given
indicator i, avg(i) the average value for indicator i measured from the available
sample of gameplay sessions, and max(i) is a value determined by designers on
what the maximum variation they are accepting in terms of the end sample,
before the usefulness of the candidate becomes 0.
The evaluation phase goes on until every population member has had at least
5 evaluations, i.e. 5 different play-throughs of the same level. Once this condition
is fulfilled, the GA will run its evolutionary mechanisms, and a new generation
will be evaluated. In between, if at any moment the best chromosome produces a
candidate whose fitness is above 0.9, an external validation phase is carried out,
where it is played at least 30 times, to validate its fitness level with greater levels
of credibility. Only after this process does the GA resume. Candidates in the form
of game parameter sets are forwarded to client-applications sequentially, which
means that once a player plays, he will likely experience a given generations’
entire population in sequence. However, the system is connected asynchronously,
with no control over who, how and when experiments a given set, so play-testers
were free to access the game at their own leisure, throughout the one week span
of the test.
Table 2. The table shows the total number gameplay sessions evaluated per generation.
Note that generation 7 was incomplete, and as such its data will not be further con-
templated (with the exception of new evaluations of the best individual that happened
in that final stage).
Generation 0 1 2 3 4 5 6 7*
Sessions 76 176 138 107 105 106 184 26*
asked to play the game as means of collecting data for research purposes. No
incentives were given, and no questionnaires were collected from the population.
There were a total of 696 gameplay sessions played on subjects’ own time and
convenience, giving 27.84 gameplay sessions per player on average. Data from
these sessions was used for 6 GA generations, plus a few more evaluations that
occurred afterwards that were not enough to complete a generation’s cycle. In
total, 68 different procedurally generated candidates were tested until the sixth
generation, in an equal number of fitness calculations. This means an average
of 9.85 gameplay sessions played per generated candidate (stdev = 5.52), dis-
tributed unevenly throughout generations and playtesters (see table 2).
4.3 Results
As can be seen from figure 2, the GA was able to improve solutions relatively
steadily until the final generation, assuring us of its overall convergence. Both
the population’s average fitness, as well as the best individual’s fitness increased
Fig. 2. Evolution of average population fitness, its standard deviation and the fitness
of the best individual in the population. Note there is folding in the y-axis.
moderately until the final generation. Fitness variance in the population shows a
tendency for decreasing, though it increases slightly from 4th generation onwards
(presumably from the increase in mutation rate, designed precisely to avoid
stagnation). While the best results seemed close to ideal, several things are
in need of improvement. For one, in this test, the initial setup seems overly
positive, with average fitness levels already in 0.8 levels (on a 0–1 scale). To a
certain extent this is to be expected, given that the platform’s aim is to iterate
on existing designs, and not necessarily to generate an entirely new design from
a completely broken one. That being said, the improvement that was measured,
while substantial in terms of player experience, did not allow a strong validation
of this method’s power to achieve the desired target indicators.
Also to take into account, when interpreting the graph, is that the fitness
landscape is, in essence, dynamic, as it depends on players behavioral patterns
during game sessions. This happens, for instance, when a chromosome passes
from one generation to another, and is then evaluated again, which can cause
significant fluctuations to its fitness level. One particular instance of this dynamic
component affecting fitness levels, is easily perceived in the second generation,
where a higher number of gameplay sessions seems to have led to a drift in values
for all individuals, with a significant decrease in both average and best fitness
(see chart 2 and table 2 side by side). This means that part of that initially
excessive positive assessment of the population was likely to be due to statistical
misrepresentation of the sample. Even so, overall, the test presented a good trend
for improvement of solutions, though from these results, it seems it presented a
low difficulty problem.
Figure 3 shows how parameters varied, on average, across generations. Given
the tendency for GA to improve the average fitness of the population (and there-
fore converge with target metrics), the major tendencies for parameters variation
mirror an improvement of the candidates. The seemingly more meaningful trends
in these parameter variations are: a steady value for the number of hills, a slow
decrease in cannons, holes, goombas and game speed, a subtle increase in item
blocks, and an initial sharp decrease in number of coins followed by sharp in-
crease. The lack of variation in number of hills is unsurprising, given how little
they impact gameplay other then providing players with a few extra platforms
for jumping. These could have improved the number of actions, but in this test
the effect proved negligible. The decrease in cannons, holes, goombas and game
speed, hand in hand with the small increment to item blocks, can be seen as
the algorithm seeking to decrease the game’s difficulty and its action rhythm,
both of which were initially higher than target levels (this trend is visible when
comparing this graph with figure 4 and target gameplay indicators absolute val-
ues). The initial decrease of coins followed by increase around generation 3, are
probably the result of trying to adjust the number of actions, as these are above
target average before generation 3, and below afterwards.
Looking at each of the quality vectors in the fitness variable in figure 4, one can
discriminate how the algorithm fared at improving each of them. While ‘Actions’
and ‘Tries’ seem to follow a somewhat stable convergence curve, tentatively
Fig. 4. Evolution of the three quality components, each for one of the indicators (‘Tries’,
’Actions’ and ‘Time’) used to calculate the fitness function. There is folding in the y-
axis between 0.3 and 0.6.
that we previously saw were probably varied by the GA to improve the number
of actions. Number of ‘Holes’ and ‘Cannons’ affect negatively the action rhythm,
though this is most likely only due to the increase in difficulty – more challenge
means a more cautious gameplay style which then means slower actions’ rhythm.
As for the ‘Tries’ indicator, difficulty seems to have been mostly dictated by
number of Holes, and on a smaller level, by Game Speed and number of Cannons,
and as expected, counter-weighed by number of item-blocks. Unexpectedly, there
is a weak, barely significant relationship between more coins and less ‘Tries’,
though it eludes us how more coins could improve survivability in a play-through.
The best Individual found by the GA had a final fitness value of 0.96, calculated
out of a total of 15 evaluations. Level retries: m = 1.13, std = 1.85; actions per
second: m = 2.14, std = 0.63; time to complete level: m = 45.26, std = 15.83.
Its parameter set was: 0 Cannons, 30 Coins, 15 Goombas, 1 Hill, 4 item blocks, 2
Holes, 9 Koopas and speed of 28 cycles per second. However, despite the closeness
of indicator results to their intended values, the distributions are uneven, thus
failing to provide a strong validation for the parameter set being able to general-
ize this behavioral pattern. To study the extent of the generalization possibilities
Table 3. Table showing statistical correlation between parameters and target indi-
cators. Significant values are highlighted in red and greater font size means greater
variance prediction.
for this candidate, one must turn to statistical indicators. The standard error for
skewness in a population of this small size is 0.580 and kurtosis is 1.121, so any
skewness and kurtosis that is greater than the double of these values is indica-
tive of a non-normal distribution. The ‘Number of Tries’ distribution of the best
candidate has skewness +1.739 and kurtosis 2.431, therefore being both skewed
and leptokurtic, i.e. significantly non-normal. Skewness and kurtosis values for Ac-
tions Per Second were −0.679 and −0.844, and for Time To Traverse Level were
−0.697 and 1.791 respectively, therefore not significantly non-normal in both these
aspects. Consequently, now it is possible to use the Shapiro-Wilk test of normal-
ity for both distributions, verifying that it is not possible to reject their normality
(sig = 0.203 and sig = 0.478 respectively). Given these facts, it can be asserted
to be somewhat likely that these two indicators are samples of a normal distribu-
tion. With this in mind, and assuming the playtester sample to be representative,
it is possible to calculate how close this game was to achieving the target metrics
for the population at large. The confidence interval, at 95% confidence levels, for
the“Actions Per Second” distribution is [1.79, 2.48]; for “Time To Traverse Level”,
[36.50, 54.03], at 14 degrees of freedom. Both these ranges are well within the con-
ditions of success for this trial, and while it is not possible to extend this calculation
for the ‘Tries’ distribution, at least its average did not veer off course (see table 1
for success ranges). Admittedly, the pre-established values for success turned out
to be underwhelming considering the results, in line with the higher than expected
fitness levels for candidates. To conclude, of the three target indicators, the best
candidate achieved appropriate mean levels for all, though only 2 of these pro-
duced sound distributions that hint at generalization of this success. Only these 2
can, with an adequate confidence level, be seen as statistically trust-worthy rep-
resentatives of the candidate’s indicators imprint. So, this method was a success
for the two indicators that the Parameters were capable of provoking variations
(as statistical correlations attested to).
Given the obtained results, it is with confidence one can conclude that in this
instance, it was possible to develop and implement a platform that subscribes to
the proposed Author-centric approach to EDPCG. It proved capable of achieving
sub-optimal values for gameplay indicators, though only as long as the design
space was well defined (meaning, when Parameters were shown to be capable of
significantly affecting said indicators). This entails that the experimental trial
achieved its goal: to find sufficient evidence to sustain the feasibility of this
approach. Moreover, this points towards the relevance of the approach herein
proposed, and specifically, towards the importance of incorporating forms of
design-space specification by designers in EDPCG methods.
In terms of the implementation details, results showed that some elements are
in need of revising. The simple GA that was used for generation purposes did
achieve moderate improvement to the game instances, though the fact remained
that convergence was achieved from an exceedingly positive starting point. Fur-
ther, both the high variability and dynamic nature of the fitness landscape im-
pacted significantly its optimization process. To solve this issue, in the future
the minimum number of gameplay sessions to evaluate candidates will need to
increase (therefore decreasing the likelihood of more drastic fitness variations),
and the selection process will be adapted to account for the varying nature of
the landscape. Coming trials must also see an improvement to the definition of
target gameplay indicators that provides designers with some form of control
over how much variability they intend to have in their candidates, as a measure
of quality introduced in the algorithm.
If future trials repeatedly show the approach’s capacity to find parameter sets
that can mediate specific gameplay metrics imprints to a playtester population,
it is our opinion that this will be a huge advantage for the videogame production
process. This test however, only showed the method’s plausibility. Coming exper-
iments shall study how to better interface this approach with designers while also
providing further evidence of the method’s capability to generate good candidates,
and the constraints in which such is possible. Only once its efficacy, specifications
and interface are fully studied can this approach be considered validated.
References
Drachen, A., Canossap, A.: Towards gameplay analysis via gameplay metrics. In: Pro-
ceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiq-
uitous Era, MindTrek 2009, pp. 202–209. ACM, New York (2009) ISBN 978-1-60558-
633-5, doi:http://doi.acm.org/10.1145/1621841.1621878
Gladwell, M.: Blink: The Power of Thinking Without Thinking. Little, Brown and
Company (2005)
Gombrich, E.H.: The story of art. Phaidon (2009)
Hassenzahl, M.: User experience and experience design. In: Soegaard, M., Dam, R.F.
(eds.) Encyclopedia of Human-Computer Interaction. Interaction-Design.org (2011)
Haupt, R.L., Haupt, S.E.: Practical genetic algorithms, 2nd edn. John Wiley & Sons,
Inc. (2004)
Kim, J.H., Gunn, D.V., Schuh, E., Phillips, B., Pagulayan, R.J., Wixon, D.R.:
Tracking real-time user experience (true): A comprehensive instrumentation solu-
tion for complex systems. In: Computer Human Interaction, pp. 443–452 (2008),
doi:10.1145/1357054.1357126
Kujala, S.: User involvement: A review of the benefits and challenges. Behaviour &
Information Technology 22(1), 1–16 (2003),
http://dx.doi.org/10.1080/01449290301782
Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience for content
creation. IEEE Transactions on Computational Intelligence and AI in Games 2(1),
54–67 (2010) ISSN 1943-068X, doi:10.1109/TCIAIG.2010.2043950
Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling player experience in super mario
bros. In: Proceedings of the 5th International Conference on Computational Intel-
ligence and Games, CIG 2009, pp. 132–139. IEEE Press, Piscataway (2009) ISBN
978-1-4244-4814-2, http://portal.acm.org/citation.cfm?id=1719293.1719323
Shaker, N., Yannakakis, G.N., Togelius, J.: Towards Automatic Personalized Content
Generation for Platform Games. In: Proceedings of the AAAI Conference on Artifi-
cial Intelligence and Interactive Digital Entertainment, AIIDE. AAAI Press (October
2010)
Steen, M., Kuijt-Evers, L., Klok, J.: Early user involvement in research and design
projects – A review of methods and practices. In: 23rd EGOS Colloquium (European
Group for Organizational Studies) (2007)
Yannakakis, G., Hallam, J.: A generic approach for generating interesting interactive
pac-man opponents. In: Proceedings of the IEEE Symposium on Computational
Intelligence and Games, pp. 94–101 (2005)
Yannakakis, G.N., Hallam, J.: Evolving opponents for interesting interactive computer
games. In: Simulation of Adaptive Behavior (2004)
Yannakakis, G.N., Hallam, J.: Towards optimizing entertainment in computer games.
Appl. Artif. Intell. 21, 933–971 (2007) ISSN 0883-9514,
http://portal.acm.org/citation.cfm?id=1392651.1392653,
doi:10.1080/08839510701527580
Yannakakis, G.N., Togelius, J.: Experience-driven procedural content generation.
IEEE Transactions on Affective Computing 99(PrePrints) (2011) ISSN 1949-3045,
http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.6
1 Introduction
Massively multiplayer online role-playing games (MMORPGs), such as World
of Warcraft1 (WoW), are providing increasing complex feature-rich game envi-
ronments to enhance their players’ experience. As this underlying complexity of
games continues to grow, there is the potential for that complexity to be exposed
to the player, especially when attempting to provide the player with important
information through the game interface.
Many games attempt to manage player exposure to complexity by limiting the
level of system detail which is shown to the player at early stages in the game,
and then gradually introducing the player to other concepts as they continue to
play the game. A common implementation of this approach, which is typically
seen in MMORPGs (e.g. RIFT2 , Age of Conan3 ), is to limit the number of
1
http://us.battle.net/wow/en/
2
http://www.riftgame.come/en/
3
http://www.ageofconan.com
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 29–44, 2013.
c Springer International Publishing Switzerland 2013
possible actions players can perform at early stages in the game, and as they
progress “reward” them by giving them access to larger number of actions.
Another method of managing interface-specific game complexity is to allow
players to customise their in-game interfaces. This allows individual players to
determine which information is displayed, and how it is displayed to them. In-
terface customisation can be achieved by installing packages or plug-ins which
alter the game interface in some way. A number of popular MMORPGs, includ-
ing WoW, provide support for such plug-ins. These plug-ins allow customisation
of game interface by modifying its visual aspects (e.g. colour, texture, position
of items), on-screen information (e.g. detailed game statistics), game-play as-
sistance (e.g. hints and alerts), etc. However, plug-ins often require advanced
configuration manually in order to provide significant benefits to the player.
An alternative, or perhaps complimentary, method of assisting players is for
the game interface to dynamically change and adapt to individual players based
on their in-game behaviour and past actions. In this paper we present one such
system, called WatchAndLearn, which includes a visual interface feedback mech-
anism for dynamically providing players with a list of predicted actions they are
likely to take, in order to simplify the game interface and improve players’ game
experience. We also describe a user study of WatchAndLearn that we have car-
ried out to evaluate its potential for improving player experience.
the player’s current position and speed. This approach to adaptive suggestion
does not require robust player modelling, since suggestions are based upon im-
mediately available factors (e.g. player position, direction, and speed) and an
optimal approach which can be calculated using the same approach used by the
game to calculate routes for AI drivers.
Having an understanding of the individual player and of their gameplay ex-
perience is insufficient if the goal is to improve the player’s game experience. In
order to impact the experience of different players, games must utilise this under-
standing and adapt the game accordingly. While previous research has explored
the benefits of the adaptation of general-purpose user interfaces, examples of
applications of adaptive interfaces in computer games are lacking. In particular,
there has been little research or commercial development that has focused on
applying adaptive interfaces for the purpose of improving player experience.
The WoW interface is driven entirely by events 5 , which are created by the game
client and sent to registered interface frames. These events are generated in
response to changes within the in-game world, or within the user interface itself.
Upon initial loading, the WatchAndLearn plug-in registers an event handler
for each relevant event type. These handlers capture events as they occur, and
5
The WoW API is documented at http://www.wowwiki.com/Event_API
Fig. 1. Overview of the data flow of the WatchAndLearn plug-in and its components
add them to a time-stamped player event log. This log persists between play
sessions, allowing for collection of data over an extended period of time, which is
useful for generating more robust behavioural models. Additionally, events are
added to a buffer and used for player modelling, as will be discussed below.
The modelling process runs continuously, monitoring the event buffer for
changes and updating Markov chains as necessary. When generating a model
using the entire event log, sub-sequences of length n + 1 (where n is the pre-
sequence chain length) within the event log are analysed individually, by mak-
ing a “sliding selection” of events from the event log, beginning with events E1
through En+1 , followed by events E2 through En+2 , and so forth. After this, the
standard process of identifying chains and outcomes is performed.
Players can elect to set a maximum time period between events that will be
considered for analysis. This allows for sequences of events which occur over
extended periods of time (with significant periods of inactivity in between) to
be ignored, preventing patterns from being incorrectly identified. In particular,
long periods of inaction (where the player may be taking a break, or otherwise
occupied) will also result in a break in terms of model generation. Where the
event buffer is found to contain two consecutive events with timestamps which
differ by a value exceeding the specified period, all events which occurred prior
to the latter event are flushed from the buffer. The result is that no chain pre-
sequences will ever be created where there is a delay of more than the allowed
time period between two consecutive events.
As more events are observed, the robustness of any generated model increases
as uncommon outliers are suppressed by more commonly occurring outcomes.
Because of this, models that are generated based upon a small number of actions
will generally be less accurate than models generated using a more substantial
event history. At early stages in model generation (with few source events), it
is impossible to establish which outcomes are potential outliers. Events which
occur rarely within limited event logs may in fact be common outcomes. Because
of this, total number of event observations should be taken into consideration
when estimating the potential accuracy of a generated model.
As chain pre-sequence length increases, the total number of unique pre-sequence
chains will increase, and the total number of predicted outcomes for each of those
chains will, in general, decrease. Because of this, it is expected that longer pre-
sequence chains will result in fewer but more accurate predictions. While increased
accuracy is desirable, for shorter event logs increasing the pre-sequence length may
reduce the number of predictions to the point that the model fails to predict any
outcome at all.
The trade-off in this case is between fewer but more accurate predictions
(with the risk of having no predictions made at all), and more but less accurate
predictions. In cases where a large number of events have been observed, longer
pre-sequence chains may be used to generate more accurate predictions without
running the risk of failing to predict any outcome. The ideal chain pre-sequence
length will ultimately be determined by the size of the source event log.
have changed (indicating that the player has recently performed a relevant ac-
tion). The contents of the event buffer are inspected, with the most recent n
events (where n is the Markov chain pre-sequence length) are selected. These
events are used to form a chain pre-sequence, which is then used as the key for
a look-up in the model dictionary, as described earlier. The retrieved value is a
list of all abilities that have been previously observed following the key chain
sequence, along with their observation counts.
In order to calculate the estimated likelihood for each prediction, the number
of observed occurrences (for all outcomes) is summed. The estimated likelihood
of each outcome is then calculated as the observation count for that outcome, as
a proportion of the sum of all observation counts. The result is a set of predicted
outcomes with predicted likelihoods ranging between 0.0 and 1.0, with the sum of
all likelihood estimations being 1.0. With predicted likelihood values calculated,
outcomes can also be ranked according to the order of estimated likelihood.
While Markov chains are used to identify likely outcomes and base-line estima-
tions for the likelihood of those outcomes, these estimations can be augmented
by considering characteristics of other collected data, such as currently active
buffs7 and state information. The goal is to utilise this additional information in
order to improve the accuracy of estimations of predicted likelihood. Historically
observed buff and state data are applied to likelihood estimations in a similar
manner.
– Estimation using buff history: using the name of the predicted ability
as a key, the buff history dictionary is queried in order to retrieve a record
containing all buffs that had been previously observed (and their observa-
tion counts) when the key ability was used. This list is compared with the
currently active player buffs, with the total observation counts for all buffs
which are also currently active being summed up. The number of observed
occurrences for all buffs is also summed. The raw outcome estimation is then
calculated as the sum of observation counts for currently active buffs as a
proportion of the sum of observation counts for all buffs.
– Estimation using state history: the name of an ability predicted accord-
ing to event history is used to query the state history dictionary, retrieving
a record which contains occurrence counts for pre-determined state factors.
The similarity between this retrieved record and the current player and tar-
get state is calculated, resulting in a value between 0.0 and 1.0, where 0.0
indicates no similarity between the current state and any historically ob-
served states for the key ability, and 1.0 indicates complete parity between
the current state and all historically observed states for the key ability.
(and not predict outcomes), the minimum weighting for event history is greater
than zero.
The following equation shows the process of calculating a predicted likelihood
P for an outcome O, where E is the initial estimation based upon event history,
B is estimation based upon buff history, S is estimation based upon state history,
and W is the weighting for each factor:
EO × WE + BO × WB + SO × WS
PO =
WE + WB + WS
short history visualisation, and allowing for identification of the patterns recog-
nised by the plug-in. As the icons of abilities used previously continue to float
further away from the visualisation, their opacity is gradually reduced until they
are eventually removed from the interface. In the case that the player uses an
ability which is not contained in the predicted list, all predictions temporarily
disappear (until familiar sequences begin to be recognised again).
While all historically used abilities (that were previously shown in the predic-
tion list) are shown for a short amount of time, only the most recent abilities
are used for prediction. The actual number of abilities used depends on the pre-
configured Markov chain pre-sequence size. Sequences of abilities which were
used as pre-sequences for Markov chain prediction are shown with lines drawn
between those abilities (see the left side of Figure 2).
4.1 Methodology
used, accuracy measurements for all predictions, and the feedback and answers
provided by the participants.
The evaluation itself consisted of a “warm-up” period, during which the sys-
tem was trained as players generated predictive sequences through play. Follow-
ing the warm-up, three trials were conducted. The only differing factor between
the three trials was the pre-sequence length of Markov chains used to generate
predictions, with this variable set at 2, 3, and 4 for trials 1, 2, and 3 respectively.
Each trial continued until the participant had performed 50 actions, with the
exception of the warm-up trial which consisted of 100 actions (more actions were
required during training in order to generate a more robust predictive model).
While the predictive model continued to develop during each trial as actions were
observed, it was reset at the end of each trial to the state that was recorded at
the end of the warm-up period. This was intended to ensure that later trials
were not favoured by more robust predictive models than earlier trials.
Participants were asked to play in their normal (typical) manner while per-
forming the evaluation.
4.3 Questionnaires
After logging into the game with the WatchAndLearn plug-in activated, using
an in-game dialogue box the participants were asked to answer a number of
questions providing basic demographic information, as well as some information
regarding their game-play habits in WoW.
After each trial (excluding the warm-up trial), participants were asked to rate
how accurate they felt the predictions were, on a 7-point Likert scale with the
anchors 1 being Not accurate and 7 being Very accurate.
After the three trials were completed, the participants were asked a series of
follow-up questions, all of which were rated on a 7-point Likert scale:
1. Overall, how accurate did you feel the predictions were (for all trials)?
2. Do you feel that the system correctly identified patterns of ability usage?
3. How clear was the visualisation in terms of communicating which abilities
were predicted to occur next (regardless of the accuracy of the prediction)?
4. Did you select your abilities based upon the predictions provided?
5. Did you check all predicted abilities before selecting an ability to use?
6. Did you feel that you had enough time to inspect all predicted abilities before
selecting an ability to use?
4.5 Results
Prediction Accuracy. The accuracy of the applied model was analysed in
three different ways: accuracy as perceived by the participants, accuracy of the
likelihood estimations, and accuracy of the ranked ordering of the predicted
abilities.
Perceived Accuracy: Table 1 shows the participant ratings (on a 7-point Likert
scale) of perceived accuracy for each of the three trials, as well as the perceived
overall accuracy. A one-way repeated measures analysis of variance was carried
out to test for differences between participant ratings of perceived accuracy for
each trial. No significant difference between trial ratings was found (F 2,57 =
1.53, p = 0.23).
Table 1. Rating counts of perceived accuracy for each trial, and overall
Rating
1: not accurate, 7: very accurate
1 2 3 4 5 6 7 Average (SD)
Trial 1 0 2 5 5 5 3 0 4.1 (1.3)
Trial 2 0 1 3 4 5 6 1 4.8 (1.3)
Trial 3 0 1 4 2 7 5 1 4.7 (1.3)
Overall 0 3 1 5 6 4 1 4.5 (1.4)
Figure 3 shows the perceived and measured accuracy across all three trials.
There was a moderate correlation between the ratings for perceived accuracy
and the measured accuracy over the three trials (r2 = 0.77).
7 100
90
Perceived Accuracy Rating
60 900
800
Fig. 4. Measured accuracy compared to the number of predictions made for each trial
Table 3. Rank of selected ability Vs. number of predicted abilities (%). Highest pro-
portion emphasized.
“[...] for the most part it recognized a pattern and displayed pretty
accurate results.”
“Its good, but unless it takes cool-downs into account [...] it won’t be
that accurate.”
4.6 Conclusions
There are a number of areas of potential future research that need to be inves-
tigated. These include: augmenting the player model with additional information
to improve its prediction accuracy, exploring alternate applications of model pre-
dictions (e.g. to create more visibly intelligent game agents), and applying this
approach to other massively multiplayer online games.
References
1. Bartle, R.: Hearts, clubs, diamonds, spades: Players who suit muds. The Journal
of Virtual Environments 1(1) (1996)
2. Bryant, B.D., Miikkulainen, R.: Acquiring visibly intelligent behavior with
example-guided neuroevolution. In: Proceedings of the National Conference on
Artificial Intelligence, vol. 1, pp. 801–808. AAAI Press (2007)
3. Charles, D., Kerr, A., McNeill, M., McAlister, M., Black, M., Kcklich, J., Moore,
A., Stringer, K.: Player-centred game design: Player modelling and adaptive digital
games. In: Proceedings of the Digital Games Research Conference, pp. 285–298
(2005)
4. Cowley, B., Charles, D., Black, M., Hickey, R.: Toward an understanding of flow
in video games. Computers in Entertainment (CIE) 6(2), Article No. 20 (2008)
5. Csikszentmihalyi, M., Csikszentmihalyi, I.: Optimal experience: Psychological
studies of flow in consciousness. Cambridge University Press (1992)
6. Deaker, C., Masoodian, M., Rogers, B.: A survey of players’ opinions on interface
customization in world of warcraft. In: Nijholt, A., Romão, T., Reidsma, D. (eds.)
ACE 2012. LNCS, vol. 7624, pp. 198–213. Springer, Heidelberg (2012)
7. Markov, A.: Extension of the limit theorems of probability theory to a sum of
variables connected in a chain. John Wiley & Sons, Inc. (1971)
8. Togelius, J., Nardi, R.D., Lucas, S.M.: Towards automatic personalised content
creation for racing games. In: IEEE Symposium on Computational Intelligence
and Games, pp. 252–259. IEEE (2007)
9. Hoorn, N.V., Togelius, J., Wierstra, D., Schmidhuber, J.: Robust player imitation
using multiobjective evolution. In: IEEE Congress on Evolutionary Computation,
pp. 652–659. IEEE (2009)
10. Yee, N.: Motivations of play in online games. Journal of CyberPsychology and
Behavior 9(6), 772–775 (2006)
Abstract. This paper presents an exploration into the effects of specific types of
persuasive technology in videogames according to a performance and a
physiological perspective. Persuasive mechanisms are often employed to
change the behavior of a determined person during a known time frame. In
videogames, these approaches are expected to produce results in a more limited
time window especially concerning the player’s performance. Literature
regarding how this type of persuasive mechanisms affects a user during a game
is scarce. We conducted a set of experiments with different games, on distinct
platforms and with thirty individuals per experimental period. Results suggest
that different persuasive techniques can effectively be used to improve or
decrease player performance as well as to regulate physiological state. We
provide a detailed analysis of these results along with a thorough discussion
regarding the design implications and opportunities of these findings and how
they are related with existing literature in the area.
1 Introduction
Videogames are currently one of the most important segments within the entertainment
industry. Annual ESA reports shows a growth tendency from 2007 to 2010, where
revenues have gone up from US$9.5 billion to US$25.1 billion, respectively. As of
2011, the videogame industry was valued at US$65 billion [9]. In recent years, we
witnessed a focus into using videogames as a driving force for behavior changes [2],
promoting a less sedentary life [29] and improving personal well-being [17][25].
Regardless of the game’s purpose and area of application (e.g. pure entertainment,
rehabilitation [11], therapy or sports [15]), players typically desire to achieve the best
performance possible according to their skill set. Different mechanics are often
employed to motivate users into achieving better performances [1] such as providing
rewards, attempting to gather the attention of the user by exploring emotionally en-
gaging gameplay sequences, etc. While some of these provide a positive experience to
users [3], others may detract them from having an enjoyable time [12], ultimately
leading them to forfeit playing the game. Independently of how they are delivered to
players, these mechanisms are, in their nature, persuasive technology (henceforth PT)
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 45–60, 2013.
© Springer International Publishing Switzerland 2013 2013
[19] – they are employed to effectively motivate the player within the game, promote
better performance displays and, in particular cases, attempt at changing the player’s
behavior or improving their well-being [2][22][27].
Persuasive technology gained momentum in recent years, particularly through the
interdisciplinary commitment to create applications which are augmented with
videogame-related features (via a technique called gamification [6][7][14]). The
aforementioned temporary rewards, scoreboards and motivational messages are some of
the approaches used to accomplish the desired results. Among these techniques emerged
what some researchers address as nudge interfaces – a strategy which capitalized on
subtle persuasive and motivational cues to drive end-users into improving their
performance in an application / game [1][8][16] or effectively changing their behavior
[18][21]. Nudge interfaces typically recur to well-known persuasive approaches (e.g.
motivating through natural language or through the employment of persistent / temporary
reinforcements such as achievements). Unfortunately, information concerning what kind
of immediate effect these mechanisms have on end-users is virtually non-existent
[4][20][28], leaving more questions unanswered. Furthermore, existing research often
relies on long persuasive intervention processes [25][29], failing to address the disruptive
effect of persuasive interfaces. Recent research trends in HCI also explore the possibility
of capitalizing on deceptive designs to motivate and drive users to change their behavior
[1]. However, empirical evidence on these effects is scarce.
We seek to research the effects of specific types of PT (which are not related with
task completeness or long motivational processes) in videogames and how a player
reacts from both a physiological and performance perspective. We also want to drive
the existing literature about PT forward, by providing empirical evidence whether the
knowledge about presence of persuasive rewards [13] is sufficient regardless of that
reward being delivered or not [1]. This article presents the main results of this
research, focusing on the physiological and performance shifts presented by players.
Testing was carried out with a total of 60 users in two different games. Results
show quite different effects and open the way to the definition of a set of PT design
guidelines according to the categorization of the empirical evidence hereby presented.
2 Related Work
Our review of existing literature tackles a few themes: PT and related models, how
these are applied in videogames and game studies regarding the influence of such
mechanisms on player experience.
3 Persuasive Technology
4 Research Questions
With this research we seek to enrich the HCI, persuasive and videogame research
communities, broadening existing knowledge and empirical evidence about the
effects of PT on players. Particularly we want to address if any changes are produced
on the player physiological state and on the performance within the game. To drive
current literature even further we question whether players are able to feel motivated
through reinforcements whether the latter is effectively delivered to them or not. The
following are the research questions for this work:
RQ1: can different types of PT effectively regulate (e.g. increase or
decrease) a player’s physiological signals while playing a videogame?
4
http://hub.guitarhero.com/
5
http://www.unrealtournament.com/
6
http://eu.battle.net/wow
7
http://mario.nintendo.com/
5 Experimental Games
Two games were developed to support our research: a casual game capitalizing on
“Whack-a-Mole” gameplay mechanics called Ctrl-Mole-Del and an arcade racing
game entitled Wrong Lane Chase.
5.1 Ctrl-Mole-Del
The first game developed within the context of this work was Ctrl-Mole-Del (Fig. 1)
for Windows Mobile platforms. This is a simplistic game which has Whack-a-Mole
(Aaron Fechter, Creating Engineering, Inc. 1971) as its main inspiration.
The goal of the game is to interrupt an invasion of moles in an open field – the
gaming area is typically populated by various holes from which the moles emerge for
a brief period of time. The players need to hit them using a plastic tool (arcade
version) or clicking / tapping over the mole (in recent versions).
Rules. Players are rewarded for hitting active moles and are penalized for missing to
hit the moles or idling (i.e. not taking any action for several seconds). The intent of
these design options is to keep the users interested in the game. Table 1 contains a
summary of the scoring rules for the game. Moles appear at random intervals, with a
1.5 to 3 seconds interval between each other. Once they spawn, the player has
approximately 1 second to hit them before they disappear. Each game lasts until the
player earns 500 points or for a maximum of 4 minutes. To interact with the game
players are able to tap the screen to hit the targets. Each time the game is executed,
the locations of the targets is randomly chosen, in order to avoid training bias.
to stop the robbers, having to shoot their vehicle until they pull over while dodging
spike strips spawned by the fugitives.
Rules. Wrong Lane Chase is a two phase game: during the first, the player needs to
avoid incoming traffic and collect gold coins; during the second phase, the player is
required to pull over the robbers’ vehicle by shooting it, while avoiding spike strips
spawned by the enemy. Table 2 presents the score policies for phase 1 and phase 2.
On both phases, obstacles are generated at random positions within the highway
and at random intervals. Players control their police vehicle with the keyboard’s
arrow keys during both phase 1 and phase 2. A new command is introduced in phase
2 – shooting a bullet – which is triggered by pressing the Z key. The full list of
controls is available in Table 3.
Command Action
Press ↑ key Moves police car up
Press ↓ key Moves police car down
Press ← key Moves police car to the left
Press → key Moves police car to the right
Press ‘Z’ key Shoots a bullet (phase 2 only)
6 Experiment
We designed two experimental periods which aimed at testing the PT features present
in both games. Each experimental period encompassed 30 subjects.
6.1 Users
A total of 60 individuals (50 male, 10 female; M=27.5; SD = 8.3) participated on both
experimental periods. 30 subjects were recruited for our first experimental period with
Ctrl-Mole-Del, while the remaining 30 participants were recruited for Wrong Lane
Chase’s experiment. No significant age differences were found between the two
groups. A quick profiling interview revealed that over 95% of the subjects played
videogames with some regularity and all of them were proficient and daily users of
computers and modern smart-phones.
6.2 Metrics
6.3 Procedure
Both experimental periods were comprised by a pre-task and a set of tasks pertaining
to each one of Ctrl-Mole-Del and Wrong Lane Chase prototypes. Each task lasted for
roughly 4 to 5 minutes. The following tasks are coincident for both experiments,
allowing users to interact with the basic version of each game and improved versions
encompassing the addition of the reinforcements described in previous sections:
Pre-Task – during this period, players had a one minute trial to get acquainted
with either Ctrl-Mole-Del’s or Wrong Lane Chase’s controls (using each
game’s first prototype). We also took this opportunity to obtain a quick profile
about the subject’s proficiency with technology and videogames.
Task 1 – participants interacted with the first prototype of either Ctrl-Mole-
Del or Wrong Lane Chase. We used this task to establish the baseline for the
users’ physiological signals.
Task 2 – for this task, participants were confronted with the second
prototype of the game they were assigned to. As a reminder, the second
Task order was randomly assigned to each subject. In our experimental design we
assured there was a balanced distribution in the task order to eliminate any task order
related bias. Participants were in stationary settings sitting in a chair on a well-lit room
interacting with the different games (even though Ctrl-Mole-Del was played on a
mobile device). Scoring policies were disclosed to players with one exception: in Wrong
Lane Chase they were not informed that avoiding a certain number of obstacles in a row
awarded more points. Subjects were only informed about the purpose of the experiment
at the end. For Ctrl-Mole-Del’s experimental period, subjects were handed a Windows
Mobile phone (HTC HD2), previously loaded with the 4 prototypes for that game. For
Wrong Lane Chase’s experiment, participants had access to a Sony VAIO VPCS13S9E
laptop model connected to a Dell 27’’ 2709W monitor. In both settings, an AliveTec
Heart Monitor8 sensor, previously prepared with electro-gel for better signal acquisition
was used to retrieve heartbeat rate data. Sensors were placed approximately 5cm apart
over the heart’s location on the subject’s chest.
6.4 Results
In our result analysis we employed a set of Friedman tests (data was not normally
distributed) accompanied by signed-rank Wilcoxon post-hoc tests to identify which
tasks yielded statistically significant results. The presented results account for a
Bonferroni correction to eliminate type-I errors.
Ctrl-Mole-Del. Table 4 and Fig. 3 present the results for Ctrl-Mole-Del’s trial. We
found statistically significant differences between tasks for the HBR metric (χ2 =
26.46; p < 0.001) and for the score metric (χ2 = 52.41; p < 0.001), leading us to assess
which ones yielded those results.
8
AliveTec: http://www.alivetec.com/products.htm
In what concerns the average HBR, players presented a significantly lower HBR
when comparing Task 2 (Z = -4.14; p < 0.001), Task 3 (Z = -4.23; p < 0.001) and
Task 4 (Z = -3.54; p < 0.001) to Task 1. No statistically significant differences were
found when comparing Task 2 to Task 3 (Z = -1.73; p = 0.082).
Fig. 3. Ctrl-Mole-Del’s box plots for HBR (left) and score (right)
Score results show that the only statistically significant results emerged from the
comparison between Task 1 and Task 2 (Z = -4.55; p < 0.001) and Task 1 and Task 3
(Z = -3.04; p = 0.002). The tasks which involved PT based on a real reward (Task 2)
and on a deceitful reward (Task 3) did not present any statistically significant
differences between them (Z = -1.36; p = 0.171).
Wrong Lane Chase. Table 5 and Fig. 4 present the results for the participants’
average HBR, score and obstacle avoidance streak for the 4 tasks of Wrong Lane
Chase, respectively. The Friedman test for the HBR metric indicated there were
statistically significant differences across tasks (χ2 = 36.92; p < 0.001). The same
happened to the score metric (χ2 = 50.04; p < 0.001) and the average number of
obstacles avoided in a row (χ2 = 42.19; p < 0.001).
A deeper analysis showed that the differences between Task 1 and Task 2 (Z = -
4.78; p < 0.001), Task 1 and Task 3 (Z = -4-43; p < 0.001) and Task 1 and Task 4 (Z
= -3.78; p < 0.001) yielded statistically significant differences. No statistically
significant differences were found between Task 2 and Task 3 (Z = -0.89; p = 0.371).
We also found statistical significance when addressing the score metric (χ2 =
50.04; p < 0.001). Players displayed significantly better performances when
confronted with a real reward when compared to Task 1 (Z = -4.20; p < 0.001);
similarly, they also reacted positively in terms of performance when confronted with
the deceitful reward mechanic (Z = -3.15; p = 0.002). Here we need to emphasize the
lack of statistical significance when comparing Task 2 to Task 3 (Z = -0.83; p =
0.405).
Fig. 4. Wrong Lane Chase’s box plots for HBR (left), score (middle) and average obstacle
avoidance streak (right)
The results for the average obstacle avoidance streak (χ2 = 42.19; p < 0.001) again
forced us to carry out a full comparative analysis between all tasks. Players presented
significantly higher avoidance streaks when comparing Task 2 (Z = -1.99; p = 0.046),
Task 3 (Z = -2.55; p = 0.011) or Task 4 to Task 1 (Z = -4.72; p < 0.001). Task 4 also
presented significantly higher streaks than any other task.
7 Discussion
The discussion of the obtained results tackles a few fronts: how different types of PT
affect players, the potential of deception as a persuasive mechanism and the
challenges behind this strategy.
most adequate and was ignored by most users; or the game genre was too fast paced
for them to momentarily peek at the feedback area. Judging by some of the user
feedback we obtained from the debriefing period, we have to abide that it was a mix
of the two which caused the players to completely ignore the language based PT used
for Ctrl-Mole-Del: “I was so focused on hitting the targets, I completely forgot about
my accuracy”; “I noticed how much time I had left in the beginning, but then I just
wanted to play the game and score the highest possible”. Can we already answer any
of our research questions? Not entirely, but these results partially support RQ1.
Existing literature shows some evidence that praising a user can effectively persuade
him / her into having a more positive posture [10]. We wanted to assess if such
approach was also valid in traditional videogames – to do so we recurred to Task 4 in
Wrong Lane Chase’s experimental period. The usage of praise messages which appear
to inform players about reaching certain feats provided us with unexpected results.
In this task, players capitalized on those persuasive cues to improve themselves and
attain higher scores. Some of our subjects stated: “I enjoyed the popping motivation
messages” or “This type of feats always leads me to want to improve more”.
Results for Task 4 are straightforward: not only players had a lower average
heartbeat rate throughout the test, they also had a significantly higher score and, more
importantly, presented longer obstacle avoidance streaks. The first remark can be linked
with a tenuous relaxation due to the praise effect. Taking into account existing literature
on flow in videogames and that “players should be warned about the outcome of their
actions” [5][28], it is natural that showing this type of information relieves the players of
some stress. The higher score, on the other hand, is a direct consequence of a behaviour
shift and compliance with the PT in play. Like previously mentioned, players were not
informed that avoiding a certain number of obstacles in a row awarded them with extra
points. As such, they did not feel the obligation to pursue this feat in other tasks due to
the lack of an obvious reward. However, we witnessed that players often preferred to
skip a gold coin to avoid an incoming obstacle than collecting all coins as fast as
possible. Such behavioural change resulted in them attaining multiple avoidance streaks
which ultimately awarded them with even more points. In summary, displaying these
praise messages was a catalyst to the creation of a sub-game or a secondary objective
which, in a vast number of cases, replaced the main goal of Wrong Lane Chase. Based
on these results, we argue that the display of praise sentences regarding feats is able to
not only prompt users to pursue different ways of playing the same game, but also to
excel and overcome their own limits as players. In light of these results and on the
previous ones (regarding persistent persuasive messages) we can point that RQ1 and
RQ2 have been answered positively.
achieve a certain milestone” [28]. Although we agree with this sentence and it holds
true for a diversity of games, there are numerous cases in which players are able to
benefit from temporary reinforcement mechanics which are not related with any
accomplishment at all. Developers populate games with multiple ephemeral rewards
which, although not game-breaking, can alter the player’s abilities in order to
introduce brief changes in gameplay. Compared to the stripped versions of Ctrl-Mole-
Del and Wrong Lane Chase, the addition of reinforcements produced less
physiological strain on our subjects. Furthermore, their scores were higher on both
games. Although this performance assessment may seem minimalistic, having into
account we are merely analysing one metric, we stay true to our arguments and
experimental design, which both capitalize on the usage of two different games, in
distinct genres. Also, the analyzed rewards are completely different from each other,
reinforcing the importance of the empirical evidence here discussed. Overall, we are
able to confirm King’s view on rewards as PT and conclude that their usage is capable
of providing players with a sense of reinforcement, actively aiding them in
performing better than with the absence of these mechanisms.
Our main conclusions point that PT can act as a physiological regulator, allowing
players to present more relaxed states depending on the type of PT employed,
answering our first research question. User performance can also be significantly
affected by the PT approach used in a videogame, thus providing the support to
answer our second research question. In this case, praise messages and reward
provision displayed different impacts on how they changed a player’s performance.
Nevertheless, this experiment’s breakthrough was the analysis of whether the
knowledge of the existence of rewards was a strong enough persuasive cue to
influence players regardless of that reward actually changing anything within the
game. Results showed no statistically significant differences regarding the players’
physiological reactions as well as their performance shifts. This introduces a new type
of PT which can be applied to videogames and this provides enough evidence to
answer our final research question.
In the future we want to explore scenarios of application for deceitful persuasive
mechanics. In particular, we want to address some of the questions we left
unanswered in this article such as whether the deceitful persuasive element derived
changes decay with experience and what is the best design strategy to seamlessly
transit from a deceitful reward mechanic towards a real one.
References
1. Adar, E., Tan, D.S., Teevan, J.: Benevolent deception in human computer interaction. In:
Proceedings of the 2013 ACM Annual Conference on Human Factors in Computing
systems (CHI 2013), pp. 1863–1872. ACM, New York (2013)
2. Chiu, M.-C., Chang, S.-P., Chang, Y.-C., Chu, H.-H., Chen, C.C.-H., Hsiao, F.-H., Ko, J.-
C.: Playful bottle: A mobile social persuasion system to motivate healthy water intake. In:
Proceedings of the 11th International Conference on Ubiquitous Computing (Ubicomp
2009), pp. 185–194. ACM, New York (2009)
3. Chumbley, J., et al.: Affect and the computer game player: the effect of gender,
personality, and game reinforcement structure on affective responses to computer game-
play. CyberPsychology & Behavior 9(3), 308–316 (2006)
4. Cowley, B., Charles, D., Black, M., Hickey, R.: Toward an understanding of flow in video
games. Comput. Entertain. 6(2), Article 20, 27 pages (2008)
5. Csíkszentmihályi, M.: Flow: The Psychology of Optimal Experience. Harper & Row
(1990)
6. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to gamefulness:
Defining “gamification”. In: Proceedings of the 15th International Academic MindTrek
Conference: Envisioning Future Media Environments (MindTrek 2011), pp. 9–15. ACM,
New York (2011)
7. Deterding, S., Sicart, M., Nacke, L., O’Hara, K., Dixon, D.: Gamification. Using game-
design elements in non-gaming contexts. In: CHI 2011 Extended Abstracts on Human
Factors in Computing Systems, pp. 2425–2428. ACM, New York (2011)
8. Eslambolchilar, P., Wilson, M.L., Komninos, A.: Nudge & influence through mobile
devices. In: Proceedings of the 12th International Conference on Human Computer
Interaction with Mobile Devices and Services, pp. 527–530. ACM, New York (2010)
9. Factbox: A look at the $65 billion video games industry. Reuters (June 6, 2011) (July 17,
2011) (retrieved )
10. Fogg, B.J.: Persuasive technology: Using computers to change what we think and do.
Ubiquity (December 2002)
11. Gouaïch, A., Hocine, N., Dokkum, L.V., Mottetp, D.: Digital-pheromone based difficulty
adaptation in post-stroke therapeutic games. In: Proceedings of the 2nd ACM SIGHIT
International Health Informatics Symposium (IHI 2012), pp. 5–12. ACM, New York
(2012)
12. Hackbarth, G., et al.: Computer playfulness and anxiety: Positive and negative mediators
of the system experience effect on perceived ease of use. Information & Management 40,
221–232 (2003)
13. Hopson, J.: Behavioral Game Design. GamaSutra Featured Articles,
http://www.gamasutra.com/view/feature/3085/
behavioral_game_design.php
14. Huotari, K., Hamari, J.: Defining gamification: A service marketing perspective. In:
Proceeding of the 16th International Academic MindTrek Conference (MindTrek 2012),
pp. 17–22. ACM, New York (2012)
15. Jensen, K.L., Krishnasamy, R., Selvadurai, V.: Studying PH. A. N. T. O. M. in the wild: A
pervasive persuasive game for daily physical activity. In: Proceedings of the 22nd
Conference of the Computer-Human Interaction Special Interest Group of Australia on
Computer-Human Interaction (OZCHI 2010), pp. 17–20. ACM, New York (2010)
16. Jianu, R., Laidlaw, D.: An evaluation of how small user interface changes can improve
scientists’ analytic strategies. In: Proceedings of the 2012 ACM Annual Conference on
Human Factors in Computing Systems (CHI 2012), pp. 2953–2962. ACM, NY (2012)
17. Jung, Y., Li, K.J., Janissa, N.S., Gladys, W.L.C., Lee, K.M.: Games for a better life:
Effects of playing Wii games on the well-being of seniors in a long-term care facility. In:
Proceedings of the Sixth Australasian Conference on Interactive Entertainment (IE 2009).
ACM, New York (2009)
18. Kalnikaite, V., Rogers, Y., Bird, J., Villar, N., Bachour, K., Payne, S., Todd, P.M.,
Schöning, J., Krüger, A., Kreitmayer, S.: How to nudge in Situ: Designing lambent devices
to deliver salient information in supermarkets. In: Proceedings of the 13th International
Conference on Ubiquitous Computing (UbiComp 2011), pp. 11–20. ACM, New York
(2011)
19. King, P., Tester, J.: The landscape of persuasive technologies. Communications of
ACM 42(5), 31–38 (1999)
20. Kultima, A., Stenros, J.: Designing games for everyone: The expanded game experience
model. In: Proceedings of the International Academic Conference on the Future of Game
Design and Technology (Futureplay 2010), pp. 66–73 (2010)
21. Loumidi, A.K., Mittag, S., Lathrop, W.B., Althoff, F.: Eco-driving incentives in the North
American market. In: Kranz, M., Weinberg, G., Meschtscherjakov, A., Murer, M.,
Wilfinger, D. (eds.) Proceedings of the 3rd International Conference on Automotive User
Interfaces and Interactive Vehicular Applications (AutomotiveUI 2011), pp. 185–192.
ACM, New York (2011)
22. Mahmud, A.A., Mubin, O., Shahid, S., Juola, J.F., de Ruyter, B.: EZ phone: Persuading
mobile users to conserve energy. In: Proceedings of the 22nd British HCI Group Annual
Conference on People and Computers: Culture, Creativity, Interaction (BCS-HCI 2008),
vol. 2, pp. 7–10. British Computer Society, Swinton (2008)
23. Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L.: Biofeedback game design: Using direct
and indirect physiological control to enhance game interaction. In: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pp. 103–112
(2011)
24. Oinas-Kukkonen, H., Harjumaa, M.: Persuasive Systems Design: Key Issues, Process
Model, and System Features. The Communications of the Association for Information
Systems (2009)
25. de Oliveira, R., Cherubini, M., Oliver, N.: MoviPill: Improving medication compliance for
elders using a mobile persuasive social game. In: Proceedings of the 12th ACM
International Conference on Ubiquitous Computing (Ubicomp 2010), pp. 251–260. ACM,
New York (2010)
26. Rowe, D., et al.: Heart Rate Variability: Indicator of user state as an aid to human-
computer interaction. In: Procs. of CHI 1998, pp. 480–487 (1998)
27. Soler, C., Zacarías, A., Lucero, A.: Molarcropolis: A mobile persuasive game to raise oral
health and dental hygiene awareness. In: Proceedings of the International Conference on
Advances in Computer Enterntainment Technology (ACE 2009), pp. 388–391. ACM,
New York (2009)
28. Sweetser, P., Johnson, D., Wyeth, P., Ozdowska, A.: GameFlow heuristics for designing
and evaluating real-time strategy games. In: Proceedings of the 8th Australasian
Conference on Interactive Entertainment: Playing the System (IE 2012), Article 1, 10
pages. ACM, New York (2012)
29. Yim, J., Nicholas Graham, T.C.: Using games to increase exercise motivation. In:
Proceedings of the 2007 Conference on Future Play (Future Play 2007), pp. 166–173
(2007)
1 Introduction
Video games, from the human players’ partners/opponents perspective, are di-
vided into two types: those where another human player controls the part-
ner/opponent, and where the video game agent (NPC) does so. For the latter,
game programmers have elaborately programmed the NPCs’ “behavior” and
“strategy” using the cut-and-trial approach, as those simulate human players’
approaches. Because of designing good degree of difficulty into video games,
programmers are obliged to engage in extremely complicated and difficult tasks.
Therefore, an automated method called “procedural technology” has received
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 61–76, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
“Unreal Tournament 2004”, and the judges tried to guess which opponents are
human. To win the prize, an NPC was required to be indistinguishable from
human players. Schrum et al. heuristically determined the human-like behavioral
patterns based on the database which stores the collected human players’ play-
logs, and they imposed these patterns as constraints to neural networks. Their
NPC’s behavioral patterns were judged more human-like than human players’
ones in October 2012.
The above approach can acquire human-like behavioral patterns using pro-
grammer’s heuristics, however the problem that programmers need to engage in
extremely complicated tasks has not been resolved.
3 Biological Constraints
human players lose their motivation. NPCs with biological constraints may ob-
tain consistent and irrational actions, e.g., hesitating to jump over an enemy,
unsteady key input and deliberating the most optimum action.
Cabrera et al. conducted an psychological experimentation which subjects
balance a vertical stick on a fingertip [5]. humans’ physical limitations occur
in this task, i.e., sensory error should occurs while they look at the tip of a
vertical stick with a fixed gaze, response time for moving their arm and finger
should be delayed and they should be tired from this task. Cabrera et al. verified
that subjects will be able to consider these physical limitations consciously or
unconsciously by training this task many times. Subjects will be able to handle
own arm and finger by imposing some noises on them. They eventually cope
with this task more safely and more efficiently.
Maslow proposed a theory in psychology “Maslow’s hierarchy of needs[7]” based
on his observations of humans’ innate curiosity. Maslow used the terms “Physio-
logical”, “Safety”, “Belongingness and Love”, “Esteem” and “Self-Actualization”
needs to describe the pattern that human motivations generally move through.
These needs should satisfy humans’ innate curiosity while human players play a
video game, e.g., a training to acquire knowledge and a challenge to reduce
tiresomeness.
1. “Sensory error”
As it is difficult for a human player to precisely recognize the position (coor-
dinates) of an object, a sensory error is imposed on the observable objects’
position. The video game agent can observe the position by applying Gaus-
sian noise to the correct coordinates of the object.
2. “Perceptual and motion delay”
“Perceptual and motion delay” means the time-lag between when a human
player observes and recognizes the game situation and when he/she inputs
keys. When the human player inputs keys, the observable game information
becomes the situation in a past game. The video game agent can observe a
game situation in several frames before the current frame.
3. “Physical fatigue”
Inputting keys successively within a long time, as well as inputting keys
repeatedly within a short time, makes human players tired. Therefore, the
video game agent should obtain better scores with fewer key controls.
4. “Balancing between repetition and novelty”
Human players often repeatedly train themselves with the same actions to
habituate themselves, and eventually they become skillful players. When
human players repeat failure with actions, they challenge themselves with
novel actions to fend off boredom and failure. The video game agent should
balance training and challenges like those engaged in by human players.
The three main advantages of our system for game programmers are that
they can reduce development costs because they do not have to do extremely
complicated and difficult tasks, they can provide these concepts intuitively in
various game genres, and they can easily introduce these concepts to various
techniques of strategy acquisition.
Fig. 1. Screen capture of “Infinite Mario Bros.” This game emulates the well-known
action game “Super Mario Bros” in a JAVA environment. Its game parameters are
public.
time from the starting node to the current node n, and h∗ (n) is given by sim-
ulating the time required to reach the right border of the screen. For further
details, refer to Baumgarten’s study [13].
5 Subjective Assessments
Fig. 2. Comparison of agent without biological constraints (top) and agent with them
(bottom): when agent avoids untouchable enemy (left), when there are five enemies on
hill (middle), and when agent jumps over a hole (right)
Participants compared two videos that were randomly displayed and answered
questions on a seven-scale relative evaluation. We analyzed the answers statis-
tically by using Scheffe’s method of paired comparisons[9]; first, we found the
significant differences in the analysis of variance, second, we plotted “relative
preferences” on the horizontal axis, and finally, we verified the confidence inter-
val for “relative preferences” of each video.
As shown in Figure 3 (a), the top line indicates the relative preferences for
human-like behaviors when comparing the Q-learning and a human, and the
bottom line indicates them when comparing the A* algorithm and a human.
First, we will present the results obtained by comparing NPCs. We demon-
strated in this experiment where [Q, Imp, Balance] and [A*, Imp] with the bio-
logical constraints were more human-like than [Q, None] and [A*, None] without
them. There were significant differences between [A*, Imp] (0.48) and [A*, None]
(−0.87): The difference in relative preferences of 1.35 (0.48 − (−0.87)) was wider
than the 99 percent confidence interval, 0.72. (After this, we will describe this
Fig. 3. Relative preferences for human-like and skillful behaviors plotted by comparing
Q-learning and Human (top) and comparing A* algorithm and Human (bottom)
as “1.35 > 0.72: 99%”). However, there were no significant differences between
[Q, Imp, Balance] and [Q, None] (0.37 < 0.48: 95%).
Next, we will present the results obtained by comparing NPCs and Humans.
We demonstrated that [Q, Imp, Balance] was more human-like than the humans:
[BEG], [INT], and [EXP]. Similarly, [A*, Imp] was more human-like than the
humans: [BEG] and [EXP]. There were significant differences between [Q, Imp,
Balance] and [BEG] (1.12 > 0.58: 99%), [Q, Imp, Balance] and [EXP] (1.33 >
0.58: 99%), and [A*, Imp] and [EXP] (0.71 > 0.59: 95%), but there were no
significant differences between [Q, Imp, Balance] and [INT] (0.44 < 0.48: 95%)
and [A*, Imp] and [BEG] (0.48 < 0.59: 95%).
As shown in Figure 3 (b), the top line indicates the relative preferences for
skillful behaviors when comparing the Q-learning and humans, and the bottom
line indicates them when comparing the A* algorithm and humans. The results
correspond to the order of video play times as expected (see Table 1 for video play
times), because the video play time was shorter if Mario was faster. The agents
with biological constraints, i.e., [Q, Imp, Balance], [Q, Imp, Challenge], and [A*,
Imp] were less skillful than agents without biological constraints, i.e., [Q, None]
and [A*, None]. There were significant differences between all combinations in
the 99% confidence interval ([Q, Imp, Balance]-[Q, None]: 0.54 > 0.46, [Q, Imp,
Challenge]-[Q, None]: 0.62 > 0.46, and [A*, Imp]-[A*, None]: 1.59 > 0.52).
collided with them.” The reasons for this might be that this agent was like a
novice player who had never previously played a Mario series. Participants who
had more advanced skills than beginners, tended to sense [Q, Imp, Challenge]
behaved mechanically. Three participants who had never played a Mario series
actually judged that “this Mario was controlled by a human.” Second, although
[BEG] was controlled by a human player, it was not as human-like as the oth-
ers. The reasons for this might be that the beginner was going to control Mario
slowly and safely by reducing the number of key inputs. The beginner contin-
ued pushing the RIGHT key throughout and sometimes pushed the JUMP key.
The participants’ answers for this video were “Mario jumped constantly and
smoothly.”
The experimental results in Figure 3 (b) indicate that introducing biological
constraints contributed to the design of a good degree of difficulty in video games.
There was a correlation between Mario’s speed (play time) in each video and
these results. It is remarkable that [BEG] was considered to be more skillful than
[Q, Imp, Balance] (0.47 > 0.46: 99%), although the play time for [BEG] was the
same as that for [Q, Imp, Balance]. The participants’ answers about this video
were that “although Mario did not dash, he jumped constantly and smoothly.”
Next, We investigated reasons into why there were no significant differences
between [Q, Imp, Balance]-[Q, None] and [Q, Imp, Balance]-[INT] in Figure
3 (a). We adopted a working hypothesis that the evaluation criteria to deter-
mine human-like behaviors were shifted by some kind of bias. We calculated the
correlation coefficients between video play times and evaluations of human-like
behaviors by participants, and we classified the 20 participants into three groups.
Participants for whom the assessment was correlated with a speedy Mario felt [Q,
None], [A*, None], [A*, Imp], [INT], and [EXP] were more human-like than the
other group. Conversely, participants for whom the assessment was correlated
with a slow Mario felt [Q, Imp, Balance], [Q, Imp, Challenge], and [BEG] were
more human-like than the other group. We called the former group the Speedy
Group, and the latter group the Slow Group.
Figure 4 indicates the relative preferences for human-like assessment by the 15
participants except for 5 in the Speedy Group. The top line indicates
the assessments for the Q-learning and humans, and the bottom line indicates
the assessments for the A* algorithm and humans. There were significant dif-
ferences between [Q, Imp, Balance]-[Q, None] (0.62 > 0.57: 95%) and [Q, Imp,
Balance]-[INT] (0.74 > 0.67: 99%). We demonstrated that agents with biolog-
ical constraints were more human-like than humans and agents without them
in analyzing these 15 particular participants. One plausible reason there were
no significant differences when we included the remaining five participants was
that the agents with biological constraints controlled Mario too slowly and safely.
The agents generated by the higher parameters of biological constraints obtained
slower and safer actions for Mario. If we had prepared the agents generated by
the intermediate parameters of biological constraints, the participants in the
Speedy Group could have assessed agents with biological constraints as being
more human-like.
We found that the evaluation criteria to determine human-like behaviors were
correlated with Mario’s speed in these experimental results where there were
significant differences for 15 participants except for 5 in the Speedy Group.
Mario’s speed that made participants feel he was human-like differed for the
participants. The higher the deviation from their own evaluation criteria, the
more participants could not feel Mario’s behavior to be human-like in the videos.
Because the average playthrough time for the Speedy Group was longer than it
was for the Slow Group, we expected that there was the possibility that the
evaluation criteria depended on the participants’ efficiency in the video game.
6 Discussion
We discuss the possibility of applying our framework to various techniques
of strategy acquisition and game genres. “Biological constraints” always oc-
cur when humans play video games. Our framework only introduced “biolog-
ical constraints” into techniques of strategy acquisition. It is unnecessary to
introduce them because game programmers analyze the most effective factors to
make NPCs human-like and they need to engage in extremely complicated and
difficult tasks to implement straightforward heuristics. Most video game titles
include rules that video game agents can observe in game situations and then
output the most effective actions. Therefore, our framework was neither oriented
toward techniques of strategy acquisition nor game genres.
We discuss that whether our agents will become alternatives to human players
or not from the viewpoint of psychology. Feeling characters are “human-like” has
not been established as explicit knowledge. We assume that humans acquire this
posteriori feeling by inferring it from physical laws and common knowledge. Min-
sky proposed a scheme in which human emotions were organized into six different
levels of processes [8] so that they could be handled by machine. Beginning with
simple instinctive reactions, each layer was built on the previous one, until they
extended to processes that involved our highest ideals. This scheme consisted
of 1) instinctive reactions, 2) learned reactions, 3) deliberative thinking, 4) re-
flective thinking, 5) self-reflective thinking, and 6) self-conscious reflection. The
strongest NPCs generated by reinforcement learning techniques and pathfinding
techniques consisted of only Nos. 2) and 3). Number 2) denotes learning about
advantages or disadvantages of game situations, and No. 3) denotes thinking
about the most effective action. Because No. 1) are not introduced to these
NPCs, their behavioral patterns are extraordinarily “mechanical.” However, as
our human-like NPCs introduce No. 1) as “biological constraints,” their behav-
ioral patterns are “human-like.” When participants assess NPCs as “human-like”
in subjective assessments, they should access higher levels of processes, i.e., Nos.
4), 5), and 6) acquired by inferring from their own past experiences. We aim
to achieve human-like NPCs in future work that can interpret what it is to be
“human-like” by implementing all six levels of Minsky’s scheme.
7 Conclusion
References
1. Fujii, N., Sato, Y., Wakama, H., Katayose, H.: Autonomously acquiring a video
game agent’s behavior: Letting players feel like playing with a human player. In: Ni-
jholt, A., Romão, T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 490–493.
Springer, Heidelberg (2012)
2. Fujita, H., Ishii, S.: Model-based reinforcement learning for partially observable
games with sampling-based state estimation. Neural Computation 19, 3051–3087
(2007)
3. Hoki, K.: Optimal control of minimax search results to learn positional evaluation.
In: Game Programming Workshop 2006, pp. 78–83 (2006)
4. Hoki, K., Kaneko, T.: The global landscape of objective functions for the optimiza-
tion of shogi piece values with a game-tree search. In: van den Herik, H.J., Plaat,
A. (eds.) ACG 2011. LNCS, vol. 7168, pp. 184–195. Springer, Heidelberg (2012)
5. Cabrera, J.L., Milton, J.G.: On-off intermittency in a human balancing task. Phys-
ical Review Letters 89(15) (September 2002)
6. Karakovskiy, S., Togelius, J.: The mario ai benchmark and competitions. IEEE
Transactions on Computational Intelligence and AI in Games 4, 55–67 (2012)
7. Maslow, A.H.: A theory of human motivation. Psychological Review 50, 370–396
(1943)
8. Minsky, M.: The Emotion Machine: Commonsense Thinking, Artificial Intelligence,
and the Future of the Human Mind, reprint edition. Simon and Schuster (2007)
9. Scheffe, H.: An analysis of variance for paired comparisons. Journal of the American
Statistical Association 47(259), 381–400 (1952)
10. Schrum, J., Karpov, I.V., Miikkulainen, R.: Human-like behavior via neuroevolu-
tion of combat behavior and replay of human traces. In: 2011 IEEE Conference,
CIG 2011, pp. 329–336 (2011)
11. Soni, B., Hingston, P.: Bots trained to play like a human are more fun. In: 2008
IEEE International Joint Conference on Neural Networks, pp. 363–369 (2008)
12. Sugiyama, T., Obata, T., Hoki, K., Ito, T.: Optimistic selection rule better than
majority voting system. In: van den Herik, H.J., Iida, H., Plaat, A. (eds.) CG 2010.
LNCS, vol. 6515, pp. 166–175. Springer, Heidelberg (2011)
13. Togelius, J., Karakovskiy, S., Baumgarten, R.: The 2009 mario AI competition. In:
2010 IEEE Evolutionary Computation (CEC), pp. 1–8 (2010)
14. Watkins, C.: Learning from delayed rewards. PhD thesis, Cambridge University,
Cambridge, England (1989)
15. Zajonc, R.B.: Attitudinal effects of mere exposure. Journal of Personality and
Social Psychology 9, 1–27 (1968)
Abstract. In this paper we compare the effects of using three game user
research methodologies to assist in shaping levels for a 2-D platformer
game, and illustrate how the use of such methodologies can help level
designers to make more informed decisions in an otherwise qualitative
oriented design process. Game user interviews, game metrics and psy-
chophysiology (biometrics) were combined in pairs to gauge usefulness in
small-scale commercial game development scenarios such as the casual
game industry. Based on the recommendations made by the methods,
three sample levels of a Super Mario clone were improved and the opin-
ions of a second sample of users indicated the success of these changes.
We conclude that user interviews provide the clearest indications for
improvement among the considered methodologies while metrics and
biometrics add different types of information that cannot be obtained
otherwise.
1 Introduction
In the late 1970s and early 1980s, when video games were still in their infancy,
developers and programmers produced very personal and sometimes low-quality
games as fast as they could [1]. This led to the North American video game crash
of 1983, which demonstrated what it means if low quality products saturate a
market [2]. In 1985 Nintendo started a strategy of far reaching quality testing
and became the most successful console system [3]. Since then, quality assurance
(QA) has become an essential phase of commercial video game releases. QA is
almost always part of an iterative production process, with test results being
reported back to the designers for evaluation. The objective of this process is to
ensure that the intentions and goals of the underlying game design are success-
fully conveyed to the player, that the players understand the metaphors and new
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 77–92, 2013.
c Springer International Publishing Switzerland 2013
concepts that the game introduces, and finally that the positive and negative
feedback is successful in motivating the player.
In this paper, we will concern ourselves with one particular type of QA which
is also called Game User Research (GUR). The term GUR is mainly used in
academic research, but industry practice also distinguishes between for example
fault-testing (“Is the product bug free?”) and user testing (“Do players like it?”)
and the usage of methods to provide feedback directly on the design [4].
Within GUR, there are three major types of information available: Data from
interviews (the user’s opinion); data from player metrics (the in-game behavior),
and data from psychophysiology (the bodily responses caused by the game).
There has been some previous work on the value of the different types of
information relative to each other. It has been suggested that biometric testing
is useful for adjusting level design and difficulty [5]. Comparing interviews and
psychophysiological data, it was found that both data sources made the game
experience more pleasant and satisfactory for the target audience. On a few
other dimensions, implementing the suggestions from psychophysiological data
increased the quality of the game by a small but significant amount, while imple-
menting the changes suggested by interview data did not raise the game above
a non-GUR method [6]. Mirza-Babaei and colleagues conclude that a study into
the combined effects of data sources would be prudent.
In this paper we look at three methodologies, using three different sources of
information, and compare which combinations are most productive in terms of
the quality of the changes and the user evaluation of these changes. Through
this comparison we want to illustrate how designers can gather and use GUR
data to make informed decisions in their games. To simplify matters, we focus
on 2-D level design: This is modular, fast and relatively easy to produce and
iterate, and provides a clear basis for comparison among level-sets. The choice
for a clone of a well-known 2-D platformer Super Mario Bros. meant that almost
all players know the objectives, mechanics and metaphors used in this type of
game, so we can look at the effect of level design while excluding other variables.
The methodologies tested were:
In our game improvement phase, data from two methodologies were combined to
create a new version of the levels. This was done three times to cover all possible
combinations.
The first phase of the research involved evaluation of the initial three levels by
a team of level designers. In the second phase we gathered GUR data on these
three levels. The third phase is the evaluation and the processing of the gathered
data by means of various (statistical) methods. The result of phase 2 and 3 was
a clear set of problems and recommendations that should be dealt with. Each
methodology rendered its own results. The fourth phase involved the qualitative
implementation of these recommendations in changes to the levels by a level
designer. There were three implementations, corresponding to the three possible
combinations of GUR methods (we did not make changes based on all three
recommendations combined). The fifth phase compared the different level-sets
created by the three combinations.
Why include metrics and psychophysiology in this comparison? The collection
of metrics data has gained an enormous popularity with the advent of web-
based games, mobile gaming and consoles that are connected to the internet
permanently [9]. In its wake, psychophysiological (biometric) testing has become
available to companies within the game industry and several development studios
have added the methodology to their QA efforts [10,11]. So far, there is little
actual research into just how useful biometrics can really be, what other parts
of game design it can be used for and how it compares to traditional testing
methods such as interviewing or observing players.
2 Related Work
Game user research (GUR) is a relatively recent field of research, which draws
upon theories and methodologies from Human Computer Interaction and Ex-
perimental Psychology to study digital games [12]. Research in this field may
also be called ‘player experience research’ or research into ‘user-centered game
design’. It involves studying the interaction between users and games with the
aim of understanding, and ultimately improving, the user experience. While the
body of research grows, there is currently no universally accepted methodology.
Many questions remain about validity and procedure, about data collection and
analysis methods [12].
Recent GUR studies have highlighted the need for research into a better under-
standing of the value of the different testing methodologies relative to each other.
A 2011 study [13] compared the data obtained from traditional observation-based
methods with that of biometric methods only (using input from galvanic skin
response (GSR) as a data collection measurement). The results showed that
different types of issues are revealed by the two approaches: Observation-based
methods mainly exposed issues related to usability and game mechanics, while
biometric research analysis was more suited to discovering issues related to game-
play and emotional immersion. Both methods uncovered unique issues that the
other method did not reveal. The study concludes that using a mixed-methods
approach allows for greater confidence and validation of issues. The approach
has received positive feedback from game developers and producers that the
researchers have collaborated with [13].
Mirza-Babaei et al. [6] performed another study in the same direction, which is
strongly in line with the aims of our research. Their experiment aimed to “iden-
tify the strengths, weaknesses and qualitative differences between the findings of
3 The Game
As basis for our game research we chose SuperTux, a side-scrolling 2-D platform-
ing game developed by the open source community (see Figure 1). The game
follows the design mechanics of the early Super Mario franchise on the Nintendo
Entertainment System. As is the case in Super Mario, the player has to maneu-
ver an avatar through a two-dimensional game environment (a level) by means
of running and jumping until the end is reached. In the course of the game, the
player has to avoid obstacles such as pits or enemies. The level typically features
not only ground surfaces to jump to and from, but also platforms in mid-air that
can be traversed. It is the occurrence of such platforms that give the genre its
name. Platform games that largely imitate the game mechanics of Super Mario
are often referred to as ‘Super Mario Clones’. SuperTux is one of such clones.
We chose SuperTux specifically since it is freely available and can be modified
by anyone due to its open source nature. Since we logged game states as part
of the metric and biometric data collection (described in later chapters), this
was a necessity in absence of a collaborating game development team. The game
comes with a tile-based level editor, allowing for easy and modular modification
of levels.
Fig. 1. A screenshot of SuperTux showing Tux - the protagonist - in its upgraded form
(with red helmet), three enemies, several bonus coins, and four platforms
4 Experiment
The comparative design of this study consisted out of 5 different phases and
entailed a comparison between levels that have been modified with a different
combination of GUR methodologies. The levels were part of a sequence (level-
set), which consisted of a tutorial (which was not changed) and three levels of
increasing difficulty.
A flowchart of the approach is shown in Figure 2.
Phase 3
Phase 1
Phase 4
Phase 2
Phase 5
modifications as a basic quality assurance. This level-set was then tested in phase
2, where the first round of GUR data was collected. In phase 3, the collected
data of the first testing round was evaluated and visualized according to the three
GUR methodologies that were used in this study. Phase 4 involves implementing
changes using three different combinations of GUR methodologies (including
modifications based on results derived from interviews + metrics data, interviews
+ biometrics data, and metrics + biometrics data). This resulted into three
different level-set versions. These level-sets formed the basis for Testing Round
2 in phase 5 and the modifications were then evaluated and compared with each
other. Each of these phases will be further described in the next section of this
paper.
General Level Ratings. After each level, participants were asked to rate the
level regarding fun, length, and difficulty. Ratings were given based on a 5-point
Likert scale with fun ranging from ‘Not at All’ to ‘Very Fun’, length ranging
from ‘Too Short’ to ‘Too Long’, and difficulty ranging from ‘Easy’ to ‘Difficult’.
The ratings of a level were used in all three GUR methodology combinations as
a basic reference point.
Interview Data. During the test sessions, the researchers monitored partici-
pants with two video cameras and a microphone through which they recorded
both the participant and the game screen. Observations were noted down and
peculiar situations were brought up during open-ended interviews.
Fig. 3. Example of a level sheet used to aid participants in recalling details about their
play experience. Levels are divided into four uniformly sized sections and are numbered
chronologically.
The interviews took place after each level and asked participants to answer
a set of semi-structured questions, which inquired about confusing, frustrating,
enjoyable, and surprising parts in the level. While answering these questions, the
participants could refer to a visual aid that divided each level into four equally
sized and numbered sections (see Figure 3) to allow locating the source of the
remark. Participants were asked to give any remaining comments or feedback
after all questions.
Game Metric Data. Due to the open source nature of SuperTux, the re-
searchers were able to add logging functionality to the game, which periodically
tracked the position of the player character as well as relevant game events,
such as defeating enemies, jumps, collecting of bonus items, etc. Game metrics
were stored in clear text and time-stamped to be in sync with audio and video
recordings.
Biometric GUR Data. All participants were monitored with several biomet-
ric sensors during the test sessions. Based on prior research in this field, we
used facial Electromyography (EMG) sensors to detect activity in the Corru-
gator Supercilii muscle group (associated with frowning), and the Zygomaticus
Major muscle group (associated with smiling). Both muscles are commonly used
to measure emotional valence [14]. Finger sensors were used to measure blood
volume pressure (BVP) and galvanic skin response (GSR), which have been
In the third phase of the research, we analyzed and processed the GUR data
that we acquired from test sessions in phase 2.
LEVEL 1
LEVEL 2
LEVEL 3
Fig. 4. The graphs show aggregated counts of instances in each of the three levels that
during interviews have been described as frustrating, enjoyful, surprising or confusing.
Note that the graphs are not in the same scale (specifically level 2)
frequency of themed instances (see Figure 4) for each level, data from the in-
terviews could be used to highlight the need for improvements within the four
sections of a level.
In order to determine which improvements to conduct, the filtered data was
divided into actionable changes and sorted based on demand for change. Here
we encountered situations that were mentioned positively by some participants
and negatively by others. In general, uncontested changes were prioritized when
looking for potential modifications.
Metrics Data. For the evaluation of metric GUR data we developed scripts
that analyzed logs from the test sessions to derive aggregated measures of play
statistics, such as amount of collected pick-ups, defeated enemies, etc. Where
necessary, measurements were normalized in terms of time that was spent in
the level, since the play duration had a direct affect on many play statistics.
In addition to deriving information from the individual measurements, we cal-
culated correlations of the acquired metric data. While these correlations could
have been useful to uncover possibilities for the improvement of a level, we did
not find actionable correlations. Apart from acquiring play statistics for each
participant, the logs were used to create heatmaps (see Figure 5), which tied the
position of the player as well as jumps, enemy kills, player deaths, and changes
in direction to locations in the level.
Fig. 5. Heatmap example showing part of a level overlayed with colored markers that
indicate where in the level players switched their movement direction. Marker colors
range from green (indicating a single event) to red (indicating the maximum amount of
events in a level). As the level geometry is tile-based, events were logged and illustrated
as heatmap marker per tile.
Biometrics Data. For the evaluation of biometric GUR data, each level was
divided into 12 sections of equal size in terms of quantity of horizontal modules.
Since biometric data works with averages, we chose this number to keep a bal-
ance between getting useful as well as localized data. We programmed scripts to
analyze the data and remove noise. For the visualization of biometric data, the
individual biometric measures were expressed in graphs and presented next to
the corresponding level sections as shown in Figure 6. Similar to [6], we analyze
biometrics data by investigating the signals and their relation to game events.
Usually, we investigate relative changes in biometrics signals compared to the
participant average. Although an initial classification of user mental states from
combined biometric signals has been shown in the research literature [16] we
do not think that this method is applicable to small samples of widely different
games that we analyze.
Fig. 6. Biometric data superimposed over map sections of level 3. The top row shows
the level graphics split into the four sections used during interviews and 12 sections
used for biometric GUR data. The red bars show the mean BVP of all participants in
each section. The height of the green triangles pointing upwards shows the mean of all
smiles for each section while the purple triangles pointing downwards show the mean
of all frowns for each section.
8 Modifications (Phase 4)
In this phase we used the evaluated data and their visualizations to facilitate im-
provements in the level design. The first author, who has an industry background
in game and level design, made all the level changes based on his perception on
the shortcomings and the sub-optimal level design choices made for the game
in relation to the target audience. This allowed for a consistent skill in terms of
implementing modifications in the level. Furthermore, all modifications had to
be connected to data that supports a change.
We looked at the three possible combinations of two GUR methodologies:
(1) Interviews and Metrics; (2) Interviews and Biometrics; and (3) Metrics and
Biometrics.
General level ratings (fun, length and difficulty of a level rated by participants)
were added to each of the three combinations to contextualize the gathered
data. For each of the three combinations we then decided to make the six most
important changes across the three levels. Where we made the changes depended
on where the data would support change.
We discuss the recommendations from the methodologies and the choice of
implementation in detail elsewhere [17]. In this paper we explain the process of
implementing the changes by one representative example. This particular change
was made in the first level and was based on the combination of interview and
biometric data:
Fig. 7. Example of changes in a level segment that have been performed in response
to interviews and biometric GUR data
In the final phase of the experiment, new participants, chosen through conve-
nience sampling in the environment of our University, were recruited to playtest
one of the three modified level-sets that had been created in previous phase. A
10 Results
The graph shown in Figure 8 illustrates the results of the GEQ. On the whole,
the differences between the combined methodologies were much smaller than we
expected. A consistent pattern can be seen across the variables Positive Affect,
Flow, Positive Experience, and Competence: The Interview & Metrics levels were
rated most positively, with the Interview & Biometrics levels in second place and
the Biometrics & Metrics levels last. This pattern is replicated for some of the
negative dimensions Tension/Annoyance, Challenge and Negative Affect.
"
#$* * #$*
0(-
/(3
/(2
/(0 /(/
/(/
/(- /(-
.(5
.(5 .(5
.(4
.(3
.(/
.(. .(/
.(-
-(6 -(6 -(6
-(5 -(5
-(3
& $ )
# # #
#
"
# & % %
&
Fig. 8. Graph showing the GEQ scores of the individual methodology testing groups
divided by the aspects that are scored by the GEQ
From the point of view of a level designer, each pair of GUR methodologies
was able to provide action-able indications regarding locations or situations that
11 Discussion
Given the insight gained in the five phases of the research and the GEQ results
of the individual methodologies, we feel that player interviews are essential to
the success of improving level design and should therefore always be involved.
The combination of interviews and metric methodologies puts both subjective
and objective information into context. While interviews are great to uncover
problems in a level, we found that metric data provides useful information re-
garding how to solve these problems, for example on the basis of heatmaps. We
feel that the biggest challenge for the use of metric data is the complexity (and
consequent time consumption) of its evaluation. Furthermore, it ideally requires
designers to establish rough goals that can be expressed in metric parameters.
The addition of biometric methodologies into QA processes remains a promis-
ing possibility, especially for the exploration of qualitative aspects in design that
are hard to evaluate through other means. As of the time of writing however, we
believe that further efforts need to go into making the addition of this method-
ology less intrusive, less time intensive and therefore less costly. Only then can
biometric methodologies be a viable addition to the QA processes of commercial
game development.
Whether the Interview & Biometric combination will be ranked second to
Interview & Metrics outside of the domain of level design for a 2-D platformer
is an open question. From our own experience, the strongest limitation of the
biometrics data was its lack of spatial precision. Because most participants com-
pleted a level in about 2 to 3 minutes, the amount of biometric data per game
tile is minimal. When aggregating over many tiles, the data becomes insightful,
however it is difficult to derive specific level design recommendations from this.
While it would have been interesting to add a control group to the second
testing round in form of a unmodified level-set, our research focused on the
comparison of methodologies. In our study the assumption was taken that the
implementation of GUR methodologies will raise player satisfaction. For future
comparisons of methodologies we do however advise to include such a control
group in order to better evaluate the magnitude of improvement.
As a final point of discussion, we would like to address the possibility of
replicating the processes of this study in 3-D games, which are arguably the
majority of game titles nowadays. While the addition of a third spatial dimension
raises the complexity in terms of visualizing data, there is no reason why the
approaches we have taken would not work in 3-D space. Heatmaps in 3-D games
are already part of metric evaluations and usually take an aerial perspective
for the visualization of level geometry. Likewise we can imagine the use of such
depictions of a level as visual aids during player interviews. In other words, while
11.1 Limitations
Lead Researcher As Level Designer and Participant Observer. The
author of this document was the lead researcher of this study as well as acting
game designer and was therefore involved in all steps of the research. In being so,
it becomes a challenge to remain objective over the course of the research. Also,
while prior experiences as game and level designer have given the researcher
insights into common design practices, it is ultimately difficult to prove a qual-
ification in terms of level design. We have been aware of these limitations from
the beginning of the study and attempted to mitigate these potential influences,
for instance by providing the research with external input in form of a focus
group and by requiring every level change to be based on research findings.
12 Conclusion
In sum, we can conclude that QA efforts regarding improvements in level design
benefit strongly from the involvement of player interviews and direct player ob-
servations. It stands to reason that having access to all three of the methodologies
discussed in this paper has strongest benefit for designers, as each methodology
offers unique insights that can often not be accessed by other means. However,
given the constraints of time and resources, studios may well be looking to add
only one additional method. From our research, in-game metrics seem to be
the most useful addition. This should be qualified by the observations that psy-
chophysiological data may be less applicable to (2-D platformer) level design
than to game design at large because of its relatively low spatial resolution. We
think that the most important take-away point is that we found complemen-
tary benefits when combining methodologies: each methodology offers unique
insights that can often not be accessed by other means. It is for this reason that
the addition of biometric GUR as design evaluation method remains promising,
despite the challenges in the evaluation and implementation.
References
1. Ernkvist, M.: Down many times, but still playing the game: Creative destruction
and industry crashes in the early video game industry 1971-1986. History of Insol-
vency and Bankruptcy, 161 (2008)
2. Wesley, D., Barczak, G.: Innovation and Marketing in the Video Game Industry:
Avoiding the Performance Trap. Gower Publishing, Ltd. (2012)
3. Sheff, D.: Game Over: How Nintendo Zapped an American Industry, Captured
Your Dollars, and Enslaved Your Children. Diane Publishing Company (1993)
4. Seif El-Nasr, M., Drachen, A., Canossa, A.: Introduction. In: Seif El-Nasr, M.,
Drachen, A., Canossa, A. (eds.) Game Analytics, pp. 3–13. Springer, London (2013)
5. Gualeni, S., Janssen, D., Calvi, L.: How psychophysiology can aid the design pro-
cess of casual games: A tale of stress, facial muscles, and paper beasts. In: Pro-
ceedings of the International Conference on the Foundations of Digital Games, pp.
149–155. ACM (2012)
6. Mirza-Babaei, P., Nacke, L.E., Gregory, J., Collins, N., Fitzpatrick, G.: How does
it play better? exploring user testing and biometric storyboards in games user
research (2013)
7. Drachen, A., Canossa, A.: Analyzing user behavior via gameplay metrics. In: Pro-
ceedings of the 2009 Conference on Future Play on GDC Canada, pp. 19–20. ACM
(2009)
8. Nacke, L.E.: An introduction to physiological player metrics for evaluating games.
In: Seif El-Nasr, M., Drachen, A., Canossa, A. (eds.) Game Analytics, pp. 585–621.
Springer, London (2013)
9. Canossa, A.: Interview with nicholas francis and thomas hagen from unity tech-
nologies. In: Seif El-Nasr, M., Drachen, A., Canossa, A. (eds.) Game Analytics, pp.
137–143. Springer, London (2013)
10. Ambinder, M.: Valves approach to playtesting: The application of empiricism. In:
Game Developer’s Conference (2009)
11. Zammitto, V., Seif El-Nasr, M.: User experience research for sports games. Pre-
sented at GDC Summit on Games User Research
12. Seif El-Nasr, M., Desurvire, H., Nacke, L., Drachen, A., Calvi, L., Isbister, K.,
Bernhaupt, R.: Game user research. In: Proceedings of the 2012 ACM Annual Con-
ference Extended Abstracts on Human Factors in Computing Systems Extended
Abstracts, pp. 2679–2682. ACM (2012)
13. Mirza-Babaei, P., Long, S., Foley, E., McAllister, G.: Understanding the contribu-
tion of biometrics to games user research. In: Proc. DIGRA (2011)
14. Cacioppo, J.T., Tassinary, L.G., Berntson, G. (eds.): Handbook of Psychophysiol-
ogy, 3rd edn. Cambridge University Press (2007)
15. Singleton, R.A., Straits, B.C.: Approaches to Social Research, vol. 4. Oxford Uni-
versity Press, New York (2005)
16. Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously mod-
eling emotion during interaction with play technologies. International Journal of
Human-Computer Studies 65(4), 329–347 (2007)
17. Maureira, M.G.: Supertux a song of ice and metrics: Comparing metrics, biomet-
rics and classic methodologies for improving level design. Master’s thesis, NHTV
University of Applied Sciences, Breda, the Netherlands (February 2013)
18. IJsselsteijn, W., de Kort, Y., Poels, K., Jurgelionis, A., Bellotti, F.: Characterising
and measuring user experiences in digital games. In: International Conference on
Advances in Computer Entertainment Technology, vol. 2, p. 27 (2007)
19. Nacke, L.: Affective ludology: Scientific measurement of user experience in inter-
active entertainment (2009)
1 Introduction
Interaction has always been a critical issue with mobile phones due to the small form
factor of these handheld devices. Since the introduction of the iPhone in 2007, touch
screens have become the predominant interaction mode on high-end smartphones.
The establishment of touch-based interaction as de facto standard for such devices can
be explained by two reasons. First, they enable designers to flexibly create interfaces
that are most suitable for the task at hand given a particular situation and context.
Icons can be made large enough to be easy to click. They can be arranged flexibly and
change in size, order, and location depending on the task. Swipe, flick, and multi
touch gestures can be used, if appropriate, to scroll various kinds of data, and so on.
Second, in a lot of situations, direct touch interaction is very intuitive. Clicking an
icon directly is normally considered more natural than using remote buttons. Most
users are able to handle touch gestures such as dragging and flicking or pinch and
zoom without any need for training when browsing lists of data and interacting with
maps, respectively. Yet, these arguments do not necessarily hold for navigation of 2D
and 3D environments on mobiles. For example, is it intuitive to “grab” a virtual cha-
racter and move it to another place, or is it better to control its movements remotely,
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 93–108, 2013.
© Springer International Publishing Switzerland 2013
for example via an on-screen joystick? Likewise, when you have a virtual environ-
ment, such as in a 3D game, an educational application, or the visualization of scien-
tific data, interface design is not as flexible but partly dependent on the actual content.
For example, large buttons might block too much of the environment and the actual
content might restrict their placement to some dedicated areas of the screen. Not sur-
prisingly, there is no real dominant interaction mode for such situations yet. Different
applications use different options or sometimes offer multiple ones and let users de-
cide which one to chose. The most common ones used right now are:
• Touch gestures, which have become popular especially for navigating 2D worlds,
where gestures such as swipe left or right can cause a virtual character to walk in
that direction.
• On-screen controllers, which are often used for 3D games, which in turn is not
surprising, since they usually resemble a software implementation of the actual
controllers used on game consoles.
• Tilting gestures, which are also common but are mostly used in games where cha-
racters are controlled from a third person view via steering actions (e.g. car racing
games) or for balancing tasks (e.g. marbles rolling through a 3D maze).
The goal of the work presented in this paper is to further investigate these three dif-
ferent interaction modalities for navigation of 2D and 3D virtual worlds on handheld
mobile devices such as smartphones in order to answer questions such as: What are
general characteristics for each individual modality? Are some better suited in certain
contexts than others? What are the differences between 2D and 3D environments? We
start by reviewing related work in section 2. Section 3 presents the setup of a compar-
ative study of these three interaction types for navigation in 2D and 3D virtual envi-
ronments and introduces our implementation. Results are analyzed and discussed in
section 4. Section 5 concludes the paper with a summary of the major contributions
and an outlook on potential future research.
2 Related Work
2D and 3D virtual environments are gaining increasing popularity on mobile devices
especially in the domain of mobile gaming. Yet, related work with respect to interac-
tion with and navigation in such mobile 2D and 3D virtual worlds is rather limited so
far. Duh et al. [3] present an interesting study on gameplay on mobile phones, but
focus more on different game types rather then interaction modes. Ponnada and Kan-
nan [9] analyze games with respect to certain playability heuristics but focus more on
overall gameplay experience rather than interaction issues. Lubitz and Krause [8]
present an interesting study on user inputs for mobile games but their work is limited
to a single jump and run game.
With respect to individual interaction modalities, hardly any scientific work can be
found about on-screen controller and joystick design. Related comments about im-
plementation options and related characteristics can mostly be found in related devel-
oper sites (see [4] for a good example) and are often restricted to important but rather
simple rules of thumb (see [11] for example).
The same goes for touch gestures which have been extensively studied with respect
to traditional interface design, evaluating for example optimum icon sizes, characte-
ristics of different touch gestures for 1D and 2D data scrolling, etc., but have not
gained similar attention from the scientific community with respect to 2D and 3D
virtual world navigation yet. Hürst and Helder [7] evaluate touchscreen interaction in
relation to virtual reality on smartphones and tablets, but focus on object interaction
and different implementations of the 3D visualization rather than navigation and ex-
ploration of 2D and 3D worlds.
Using device tilting as a means to navigate and scroll text and other static data has
been studied extensively in the past – see [10] for an example of one of the earlier
works. Newer research applies this technique to other media types such as video [6].
In fact, controlling video playback via tilting was already proposed in the earliest
work on handheld virtual reality [2]. Baillie et al. [1] use tilt to access additional in-
formation in a geographic information system scenario. Most relevant for the research
presented in this paper are the works done by Gilbertson et al. [5] who study the usa-
bility of tilt for control of a first person racing game (“tunnel run”). While all of these
related works demonstrate certain advantages of tilt for navigation of 3D data, none of
them provides a comparative evaluation of the basic characteristics with respect to
different interaction types.
3 Experimental Design
To compare the different interaction modes with each other, we implemented the 2D
environment shown in Figure 1 on a state of the art smartphone. It contains the
simplified illustration of a two-story building, with each of the two floors featuring
four doors. Users can move left and right on each floor, and switch between them by
using the staircase in the center. On the mobile phone, users only see parts of the
whole environment as illustrated in Figure 2. In the experiments, participants were
asked to go to certain dedicated doors and enter the rooms using the three interaction
modes described below. Entering a room is depicted by changing the texture of a
door.
Fig. 2. Example for the excerpt visible to a user on the device used in the experiments (left) and
visualization of the event “character has entered a room” (right)
Touch Gestures. There are many different ways to use gestures in order to control
the navigation of a virtual character on the screen. After reviewing common imple-
mentations in typical state of the art mobile apps, we decided to choose a rather
simple, yet powerful and flexible one for this experiment. As illustrated in Figure 3,
navigation is done by simply clicking on the dedicated target location, i.e. the charac-
ter automatically walks along the floor or up and down the stairs towards the latest
location in the virtual 2D world tapped by the user. Clicking on a door makes the
character enter the related room.
Fig. 3. 2D touch gesture-based navigation: the virtual character walks to the last screen location
tapped by the user.
along the floor. Up and down movements are used to climb or descend stairs and exit
or enter a room, depending if the character stands in front of a staircase or door,
respectively.
Fig. 4. 2D on-screen joypad: left/right drags of the virtual joystick evoke left/right movements
of the virtual character, up/down drags are used to climb/descend stairs and enter/exit rooms
Tilting Gestures. Using the accelerometer integrated in almost all modern smart-
phones we can map horizontal and vertical rotations of the device to related move-
ments of the virtual character. Naturally, we map horizontal rotations to left/right
movements and use vertical rotations to climb and descend stairs or enter and exit
rooms depending on the context, as illustrated in Figure 5. It should be noticed that
“natural” in this case depends on the context. For example, in flight simulations,
up/down movements might be interpreted reversely due to what kind of steering
wheel they aim to resemble.
Fig. 6. Histogram of the age distribution of the test subject sample used
Fig. 7. Histograms illustrating the subjects’ experience with (left to right) touch screens, tilting
virtual, and physical joypads, according to their own judgment about their usage, i.e. “1” for
“never”, 2 for “barely”, “3” for “often”, and “4” for “everyday” (= labels on x-axis)
We will discuss the results of our experiment with respect to these characteristics
in the following section. A statistical analysis of the presented data using a Chi-square
test revealed only one significant correlation: male subjects had much more
experience with physical controllers than females. It is further worth noticing that
experience with on-screen joypads was almost nonexistent among the test subjects.
Study Design and Tasks. Our study used a within-subjects design with the only in-
dependent variable being the interaction method. The order of interaction modality
was counterbalanced across the participants to avoid related influences on the results.
For each of these modes, users had to visit as many rooms as possible in a given order
within a three minutes time limit. In particular, users were provided with the image in
Figure 1 and asked to visit rooms 2.4, 1.3, 1.1, 2.1, 1.4, 2.2, 2.3, and 1.2 in exactly
this order. As dependent variable, we recorded the number of rooms visited within the
given time limit. In addition, the total distance travelled by the virtual character dur-
ing the test phase was recorded.
Procedure. All experiments were conducted individually with each subject in a class-
room of the school. Questionnaires were used to gather personal statistics (cf. Fig. 6
and 7), user feedback and ratings after the test (to be discussed in the next section),
and notes of the neutral observer. After a short introduction into the problem, the
otherwise neutral observer gave a short demonstration of each interaction mode. Be-
fore the actual tests, subjects were asked to perform a single subtask (visiting one
room) to familiarize them with the interaction metaphor. A whole session lasted ap-
proximately 15 minutes per participant with about five minutes of evaluation time per
interaction mode (training plus three minutes evaluation). Between the tests for each
interaction type, users took a short break to relax and concentrate on the next task.
For the test of navigation in 3D, we implemented the simple 3D model of the interior of
a house that is depicted in Figure 8. The number of floors and doors are comparable to
the 2D case (cf. Fig. 1), but the actual implementation depicted a first person view off the
related 3D world as shown in Figure 9. Again users could freely roam along the flooors,
enter rooms, and change beetween floors using one of the interaction modes descriibed
below. Because we are now w exploring a 3D world, motions are not just simple up/doown
and left/right movements lik ke in the 2D case, but can be described by a virtual cam mera
that represents the first persson view of the virtual character. In the following, we de-
scribe how the different intteraction modes are mapped to related modifications of the
virtual camera.
Touch Gestures. An inform mal analysis revealed a surprisingly low number of usaages
of touch screen gestures fo or navigation in currently available 3D games on mobiiles.
Most implementations are rather
r restrictive insofar as they limit the amount of sppace
that a user can explore and the locations one can visit. In order to achieve comparaabil-
ity with the preceding 2D experiment
e but most importantly the other two interacttion
modes of this evaluation, wew implemented the gestures illustrated in Figure 10: Drrag-
ging the finger left or rightt on the screen results in a rotation of the virtual cameraa in
the same direction (creatin ng a visual feedback resembling “looking around”). Baack-
ward motion is achieved by b a horizontal flick. Repeated taps on the screen evokke a
forward movement of the virtual camera (creating the visual feedback resemblling
walking backward and forw ward, respectively).
On-screen Joypad. Figurre 11 shows how movements of the virtual joypad are
mapped to the virtual cameera, which in turn defines the field of view shown on the
mobile device. Left and rigght movements of the joypad are mapped to a related roota-
tion similarly to the left an
nd right drag touch gestures above. Up and down moove-
ments evoke a virtual charaacter to walk forward and backward, respectively.
Tilting Gestures. Rotations of the device around its vertical axis are mapped to roota-
tions of the virtual camera, device rotations around its horizontal axis are mappedd on
backward and forward mov vements as illustrated in Figure 12.
Implementation. The sam me smartphone as in the 2D experiments described above
was used for the implemen ntation and in the experiments. Again, we made sure tthat
parameters such as velocityy of the character when moving are comparable acrosss all
three interaction modes in order
o to guarantee that the latter one are the only indeppen-
dent variable in our experim
ment.
Fig. 10. 3D touch gesture-based navigation: horizontal drags result in corresponding rotations
of the virtual camera, flicking and tapping evokes backward and forward motion, respectively
Fig. 11. 3D on-screen joypad: horizontal drags of the joypad result in corresponding rotations
of the virtual camera, up/down movements evokes forward and backward motion, respectively
Fig. 12. 3D tilting-based navigation: horizontal rotations of the device evoke corresponding
rotations of the virtual camera, vertical rotations are used to move forward and backward
otherwise the same criteria as described above applied for the selection process. Three
subjects were left-handed, all others right-handed. Distribution of ages and experience
are plotted in Figure 13 and 14.
Study Design and Tasks. Like in the 2D experiment, this test followed a within-
subject design with the interaction mode being the only independent variable, coun-
terbalanced across all participants. Again, users were provided with a map (the image
shown in Figure 8) and had to visit as many rooms as possible within three minutes
based on a given order (room 2.4, 1.3, 1.1, 2.1, 1.4, 2.2, 2.3, and 1.2). The number of
rooms visited and total distance travelled were recorded as independent variable.
Fig. 13. Histogram of the age distribution of the test subject sample used
Fig. 14. Histograms illustrating the experience of the test subjects in the 3D experiment with
touch screens, virtual joypads, physical joypads, and tilting (from left to right) according to
their own judgment on a scale representing their usage, i.e. “1” for “never”, 2 for “barely”, “3”
for “often”, and “4” for “everyday” (labels on the x-axis)
Procedure. Experiments were done individually on later dates, but under otherwise
similar conditions as the 2D experiment, with a short explanation and demonstration
of each interaction style at the beginning, followed by a training task for each mode
and the actual test. Sessions again took about 15 minutes with short breaks between
the three tests. Aside from a small modification of the questionnaire, which we will
describe when discussing the results, data was gathered in a similar way as in the 2D
case.
4 Results
In the statistic analysis of the following data, we measured correlations using Chi-
square tests (when dealing with categorical data), and Pearson Correlation tests (when
dealing with numerical data only). Statistical significance of the differences between
groups was tested using ANOVA, additionally performing Tukey’s post-hoc test in
order to cluster groups according to the significance of their differences. Tests were
carried out at a 95% significance level (p value < 0.05).
Table 1. Efficiency indicators (2D experiment; averages and standard deviation over all users)
If measured as number of rooms visited within the time limit, we cannot observe
any significant differences (p > 0.05 with ANOVA test; left column in Table 1). Per-
formance strongly varies across participants. Some subjects did very well with some
method whereas others performed very poorly with it.
If measured as distance travelled by the virtual character per number of rooms vi-
sited, touch gestures were significantly more efficient than the on-screen joypad (p<
0.05 with ANOVA test; center column in Table 1). A statistically significant differ-
ence between tilting and the other two approaches could not be observed however.
Looking at the implementation (cf. Section 3.2) this seems reasonable because for
touch interaction, people could click directly on a target position, thus avoiding situa-
tions in which a character misses a target and has to go back. The reason why this
significant difference is not reflected in the number of rooms visited is simply that the
distance between two rooms was too large to create a difference in absolute room
numbers.
If efficiency is measured as experiencing a lower amount of navigation mistakes
(such as missing a door, accidentally entering a wrong one, loosing control of the
virtual character, etc.), tilting proved to be significantly less efficient than the other
two interaction modes (p< 0.05 with ANOVA test; right column in Table 1).
User Feedback and Qualitative Data. In the closing questionnaire, users were asked
about efficiency and how much they enjoyed the interaction modes. In particular, they
were asked to rank them with respect to “efficiency” and “enjoyability”.
Although the analysis of the related performance indicators above did not reveal
any significant difference, subjects perceived touch screen as more efficient than tilt-
ing (significance verified with ANOVA test, p< 0.05). Yet, a significant difference
regarding the preference of the on-screen joypad could not be observed – neither
compared to touch nor to tilt.
Relating the background data provided at the beginning of Section 3.2 (cf. text and
Figures 6 and 7) only revealed one significant difference (Chi-Square, p< 0.05): Fe-
male participants considered touch gestures significantly more enjoyable than the on-
screen joypad, whereas for male subjects, it was just the other way around. This is
however most likely due to the larger level of experience of the males with physical
controllers which we already commented on. This is also reflected in the qualitative
user statements where typical comments by male subjects included: “The virtual joys-
tick was nice because it was kind of similar to the X-Box controller.” “With the vir-
tual joystick I felt like playing on the PSP.” And: “I guess (virtual joypad) was more
similar to how I am used to play games.” Females, on the other hand, commented
more positive about the touch gestures, with typical remarks being: “With touch
screen I felt way more in control of the character.” “I think I was faster using touch
screen.” And: “With touch screen you just got to tell the character where to go … and
he goes there … that’s nice.” Comments for the tilting approach had no clear relation
to a subject’s gender and included positive remarks such as “it’s innovative a chal-
lenging … I like it!” but also negative comments such as “I didn’t feel like I was in
control of the virtual character.”
Unlike in the 2D case, the Pearson correlation test showed that age highly corre-
lated with the efficiencies mentioned above. In particular, the older the subject, the
lower their performance (correlation coefficient: -0.740). Chi-Square tests could not
reveal a correlation between any of the other background factors and the efficiency
factors, except for gender and physical controller experience which again was signifi-
cantly higher for male subjects (Chi-Square test, p< 0.05).
Table 2. Efficiency indicators (3D experiment; averages and standard deviation over all users)
User Feedback and Qualitative Data. In the 2D experiment, most users rated the
most efficient system also as their most preferred one. In order to get a clearer idea of
their qualitative ratings, we therefore changed the questionnaire from asking for a
ranking to asking for a rating on a scale from -2 (worst) to 2 (best). Yet, again, no
statistically significant difference among the three interaction styles could be ob-
served. A Chi-Square test still showed a high correlation between the method users
perceived as more efficient and the one they considered more enjoyable. Similarly, a
preference for touch gestures among female subjects was observed again (Chi-Square
test, p< 0.05).
In order to gain more qualitative statements, we asked users to describe each of the
methods with one positive and one negative statement. Unfortunately, they had prob-
lems coming up with good words of their own, but most of them used one of the ex-
amples provided for each category. Touch gestures were positively described as
“easy” and “precise” by 33% and 50% of the subjects, respectively, and negatively
characterized as “boring” (33%). The most used words to describe positive aspects of
the on-screen joypad were “easy” (45%) and “precise” (41%). The mostly used nega-
tive words included “unstable” (45%), “prone-to-errors” (25%), and “difficult”
(16%). Tilting was described as “challenging” (in a positive, entertaining way, 75%)
and “innovative” (25%), yet negatively characterized as “unstable” (58%) “prone-to-
errors” (25%), and “difficult” (16%).
but give a clear indication that both pure performance measures as well as personal
preference seem to be rather based on very individual aspects with no clear characte-
ristics or user profiles that would prioritize one approach over another.
In the 2D case, we expected tilt to be the least favorable of the three options be-
cause it is also hardly used for this kind of data in current applications but mostly
applied in 3D environments. Apparently, many test subjects shared our view, because
we could observe a slight related trend in the qualitative user statements. Yet, our
analysis revealed no clear evidence that speaks against this interaction mode. In fact,
some users even enjoyed that it appeared to be more “challenging” and thus more
entertaining for them. We expected touch gestures to perform best because users
could directly select a target position instead of having to direct the virtual character
to it. Again, our intuitive assumptions were confirmed by some of the qualitative user
statements, but could not be verified with any of the data. Instead, personal preference
and individual performance again seems to dominate the outcome. What further sur-
prised us was the apparent influence of experience with comparable physical control-
lers on the preference of male users for the virtual joypad implementation. Yet, given
that there were no performance differences, and the fact that positive statements about
all three interaction types were equally distributed among subjects of both genders we
cannot conclude with a clear recommendation, i.e. we can neither suggest that expe-
rienced users should be trained to use other modes of interaction nor should inexpe-
rienced ones be forced to adapt to traditional modes, because none of the methods
proved to be superior.
In the 3D case, we actually expected tilt to perform better, not only because it is of-
ten used in current games and 3D apps, but also because the related interactions ap-
peared more natural for us in this case. Although the virtual character moves in 3D
now, the actual interactions on the virtual joypad are still in 2D – resulting in the situ-
ation that users have to move their fingers up and down in order to evoke forward and
backward movements of the virtual character. In case of tilting, the device is rotated
forward or backward in accordance to the resulting movement of the character, thus
suggesting a more intuitive and natural interaction. Yet, the results could not confirm
these assumptions. Neither performance nor the subjective user ratings could show
any significant difference between interaction modes. In fact, even the personal prefe-
rence did not reveal any significant trend in the 3D case, although subjects still re-
marked that touch interaction appears to be more accurate and easier to control than
tilting actions. Again, the overall conclusion is that picking the best interaction mode
is less a question of dedicated user characteristics but more a matter of personal prefe-
rence and individual habits.
role in user preference, there is no clear indication for either of the modalities. Even
people who expressed preference for one of the interaction types did not necessarily
perform well with it. In addition, experience depends on the past, and related user
preferences might change over time. In the 3D case, there was a trend that perfor-
mance correlates negatively with increasing age. Yet, the ratio of elderly within the
sample set was too small to draw a strong conclusion, but suggests interesting alleys
for future research. The concluding observation that personal preference and individ-
ual habits and abilities seem to be the major reasons for differences in the measured
quantitative and qualitative data, leads to the recommendation that developers should
not aim at providing one “best” interaction style, but rather offer a set of different
ones and let users decide which one to use. In fact, some of the current 3D games that
we used in the informal analysis when setting up our experiments are already doing
this. Yet, the different options are often hidden in the settings menus and therefore
frequently overlooked by users who then feel forced to use the default interaction
method, which might not be the most appropriate one for them.
Given the apparently strong influence of experience on user preference, we already
suggested that further, long term studies investigating how, for example, those
preferences change over time might reveal interesting observations. Based on our
experimental results, we assume that such a study would require users to utilize the
interaction mode several weeks, and thus is beyond the scope of this paper. Yet, it
provides an interesting area for further exploration in future works. Likewise, the
influence of age that was identified in an initial trend should be studied further, as
already suggested above. Considering other options for future work, we like to quote
one test subject who noted on the tilting interaction: “It was way too unstable. I
couldn’t center the character in front of the doors easily.” On the one hand, this state-
ment suggests that individual interaction modes might not be suitable for all kind of
tasks. For example, with tilting it is indeed harder to put a virtual character at a fixed
location due to the constant change in the accelerometer data because of minor
movements of the device – even if appropriate filtering techniques are applied to the
sensor data. Yet, in situations where this is not required, for example because the
virtual character is in constant motion, tilting might be favored by most users – a
speculation that seems to be confirmed by dominant usage of this modality for racing
and balancing games on mobile devices. On the other hand, this statement also indi-
cates room for optimization of the concrete implementation of the interaction modes.
For our initial study, we took particular care to set the parameters in a comparable
way in order to guarantee that interaction modality is the only independent variable in
our tests. Yet, individual optimizations, for example of the virtual character’s speed
when tilting a device might have an effect on user preference and performance and is
therefore worth further investigation. Likewise, there are different ways to implement
an on-screen controller, suggesting comparative studies of different designs.
References
1. Baillie, L., Kunczier, H., Anegg, H.: Rolling, rotating and imagining in a virtual mobile
world. In: MobileHCI 2005: Proceedings of the 7th International Conference on Human
Computer Interaction with Mobile Devices & Services, pp. 283–286. ACM, New York
(2005)
2. Buxton, B., Fitzmaurice, G.W.: HMDs, caves & chameleon: A human-centric analysis of
interaction in virtual space. SIGGRAPH Comput. Graph. 32(4), 69–74 (1998)
3. Duh, H.B.-L., Chen, V.H.H., Tan, C.B.: Playing different games on different phones: An
empirical study on mobile gaming. In: Proceedings of the 10th International Conference on
Human Computer Interaction with Mobile Devices and Services (MobileHCI 2008),
pp. 391–394. ACM, New York (2008)
4. Gamasutra.com: A Guide to iOS Twin Stick Shooter, cf,
http://www.gamasutra.com/view/feature/6323/
a_guide_to_ios_twin_stick_shooter_.php?print=1
5. Gilbertson, P., Coulton, P., Chehimi, F., Vajk, T.: Using “tilt” as an interface to control
“no-button” 3-d mobile games. Comput. Entertain. 6, 38:1–38:13 (2008)
6. Huber, J., Steimle, J., Mühlhäuser, M.: Toward more efficient user interfaces for mobile
video browsing: An in-depth exploration of the design space. In: Proceedings of the Inter-
national Conference on Multimedia (MM 2010), pp. 341–350. ACM, New York (2010)
7. Hürst, W., Helder, M.: Mobile 3D graphics and virtual reality interaction. In: Romão, T.,
Correia, N., Inami, M., Kato, H., Prada, R., Terada, T., Dias, E., Chambel, T. (eds.) Pro-
ceedings of the 8th International Conference on Advances in Computer Entertainment
Technology (ACE 2011), Article 28, 8 pages. ACM, New York (2011)
8. Lubitz, K., Krause, M.: Exploring user input metaphors for jump and run games on mobile
devices. In: Herrlich, M., Malaka, R., Masuch, M. (eds.) ICEC 2012. LNCS, vol. 7522, pp.
473–475. Springer, Heidelberg (2012)
9. Ponnada, A., Kannan, A.: Evaluation of mobile games using playability heuristics. In: Pro-
ceedings of the International Conference on Advances in Computing, Communications and
Informatics (ICACCI 2012), pp. 244–247. ACM, New York (2012)
10. Rekimoto, J.: Tilting operations for small screen interfaces. In: Proceedings of the 9th Annual
ACM Symposium on User Interface Software and Technology (UIST 1996), pp. 167–168.
ACM, New York (1996)
11. SimpleUsability.com: Mobile Gaming – A usability study, cf,
http://www.simpleusability.com/our-news/2012/01/
mobile-gaming-usability/
1 Introduction
While there are a number of industrial robots that support our daily lives, there are
also numerous fictional robots that have appeared in movies, comics and video
games. Many of us would be interested in understanding the experience of having a
tough iron body, hoping to become like these robotic heroes, if only for a short time.
The question naturally arises: what would it feel to be a robot? While we are seldom
conscious of the activities of our biological muscles or tendons, a robotic body would
have a definite robotic body sense that is different from that of humans.
In this paper, we focus on the body sense of robots and simulate robot-like feelings
on the human arm (Fig. 1). To create a realistic robot-like body sense, we provide
vibrotactile feedback based on vibration recording, modeling, and rendering of a real
robot’s actuation. Combined with conventional visual animation and sound effects,
our system allows the user to virtually robotize his or her body visually, aurally, and
haptically.
This paper mainly contributes to the field of computer entertainment technology by
presenting a new alternative for achieving an immersive experience in video games.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 109–122, 2013.
© Springer International Publishing Switzerland 2013
Gesture input devices, sometimes referred to as natural user interfaces (e.g., the Kinect
sensor from Microsoft, the Wii remote from Nintendo, and the Omni from Virtuix)
increase the player’s feeling of oneness with the game character by synchronizing the
character’s motion with the player’s body motion, resulting in an immersive game
experience. Also, some previous tactile entertainment systems have enhanced
the immersive experience by displaying vibrotactile feedback to the player’s body,
synchronized with characters being shot [1] or getting slashed [2].
However, playable characters in video games are not always human – sometimes
they are, for example, metallic robots. By creating a robot-like body sense and simu-
lating a situation in which the player becomes the robot, experiencing the game with a
robotic body could be made more immersive. Therefore, we envision that the tech-
nique of virtual robotization of the human body could enrich immersive video games
by offering the experience of being a fictional robotic hero.
2 Related Work
The alteration of human proprioception has also been studied. One method of altering
the sense of the body in space is called the kinesthetic illusion, which creates an illu-
sory arm motion [14-16]. The illusion can be produced by using a vibration of about
100 Hz to activate the muscle spindles. It can be extended to the elongation of parts of
the human body, which is known as the Pinocchio illusion [17].
An illusion of body-ownership called the rubber hand illusion [18-20] is provoked
by simultaneously tapping on a person's hidden real hand and a visible rubber hand
placed next to the real hand. The person feels as if the rubber hand has become their
real hand. This illusion can also be induced by the synchronous movement of the
person’s real hand and a virtual hand on a screen [20]. Additionally, the visual realism
of the virtual hand does not seem to contribute much to the enhancement of the body-
ownership illusion. In this study, we use this phenomenon to create the feeling of
ownership of a virtual robot arm using synchronous movements of the user’s real arm
and the virtual robot arm.
Our hypothesis is that presenting robot ego-vibrations to the user’s body in accordance
with his or her motion will make users believe that their body has become robotic.
Thus, we employed a data-driven approach using vibration recording, modeling, and
rendering, which has been reported to be a promising method in the creation of realistic
virtual textures [9], [21-22].
components attributed to the change of orientation of the robot’s forearm. Next, the
three acceleration channels were summed to a single wave. We normalized the dura-
tion of acceleration data captured at the various angular velocities by selecting clip-
ping one second of data around 45°, which is the center of the range of motion.
We employed Linear Predictive Cording (LPC) to approximate the spectral density
of the raw acceleration data (Fig. 3). LPC is known as one of the most powerful
speech processing techniques, and it is also used in haptic data modeling [9][22]. To
make a model that approximates the spectral density of the raw data, we applied a
tenth-order finite impulse response (FIR) filter to the acceleration data, and we calcu-
lated the coefficient vectors ) (k=1, 2 … 10) of the LPC as a function of angular
velocity, by minimizing the prediction error in the least squares sense. This calcula-
tion was performed using the lpc function in MATLAB (The MathWorks, Inc.).
The purpose of this modeling was to predict the next vibration value from a series
of past data samples. The predicted value can be written as:
(1)
where n is the number of steps (n=0 is substituted), is the value at the past k
steps, are the LPC coefficients, w is a sample of white Gaussian noise. While
the model contains a similar spectral density to the raw data, the model in the time
domain is not a representation of the same waves, because of the randomness of the
white Gaussian noise. Therefore, users can feel natural continuous vibration.
Fig. 3. Recorded vibration (left), example of LPC-modeled vibration (right), and overlaid
spectral density (center)
The purpose of this psychophysical experiment was to verify the contribution of vi-
brotactile feedback to the subjective robot-like feeling. Using our virtual robotization
system, we compared four sensory feedback conditions: visual only (V), visual +
auditory (V+A), visual + haptic (V+H), and visual + auditory + haptic (V+A+H) by
means of questionnaires.
noise-canceling headphones (QuietComfort 15, BOSE Corp.) to cancel out any sound
generated by the actuation of the transducer. The operating sound of the robot was
emitted from the right channel only because the position of the auditory and vibrotac-
tile feedback should be the same for a more realistic robot-like feeling. The experi-
menter confirmed with the participants that they could feel the vibrotactile stimuli
clearly.
The participants were asked to flex and extend their right elbow at various
velocities, looking at the robot arm animation in the monitor. Each trial was 15
seconds long. After each trial, the participants were asked to answer the following two
questions:
How Much Did You Feel the Robot-Like Feeling in Your Arm? The participants
evaluated their confidence about whether their right arm felt like the robot in the moni-
tor, on a visual analog scale (0: not robot at all, 100: totally robot). Note that we defined
the central point (50) as the robot-like feeling in the V+A condition, since the partici-
pants had never before experienced a robot-like body sense and the reference point
of the evaluation would be different between participants. In other conditions, the
participants evaluated the robot-like feeling by comparing with the V+A condition.
How Much Did You Feel a Reaction Force? The typical expectation of a robotic
body would be a friction-like force opposing the direction of body movement. There-
fore, if the participants felt a resistance force when the there was none, as in this sys-
tem, it might be a good quantitative measure of the perceived robot-like feeling. The
participants answered the amount of the perceived reaction force with the visual-
analog scale (0: completely smooth, 50: the same as usual, 100: felt strong force).
Scores less than 50 points meant that the arm movement felt smoother than usual.
4.3 Results
Fig. 7 shows the perceived amount of robot-like feeling and reaction force. Whiskers
indicate the standard deviation. The robot-like feeling was highest in the V+A+H
condition, followed by the V+A, V+H, and V conditions. We performed a one-way
analysis of variance (ANOVA) and found significant differences between the feed-
back conditions (F(3,24) = 3.35, p < 0.001). A post-hoc comparison using Tukey’s
HSD method between the feedback conditions showed a significant difference (p <
0.05) in all the pairs except V+A vs. V+H. The comparison between V+A and V+H
showed a marginally significant difference (p = 0.07 < 0.10).
***
Amount of reaction force
80 80
70 * 70
60 *** 60
50 50
40 40
30 30
20 20
10 10
0 0
V V+A V+H V+A+H V V+A V+H V+A+H
Feedback condition Feedback condition
Fig. 7. Mean values of the evaluation of robot-like feeling (left) and reaction force (right)
Participants felt that the reaction force was highest in the V+A+H condition, fol-
lowed by the V+H, V+A, and V conditions. A one-way ANOVA between feedback
conditions showed significant differences (F(3,24) = 3.34, p < 0.05). A post-hoc test
revealed significant differences only between the V and V+A+H conditions (p <
0.05).
5 Discussion
However, as shown in Fig. 7, there was a different tendency between the V+A and
V+H condition; the robot-like feeling in V+H condition was lower, while the reaction
force was higher. This inconsistency might be attributed to a higher contribution of
the auditory cue to the robot-like feeling, and a higher contribution of the vibrotactile
cue to the resultant illusory force cues, another haptic sensation.
100
Amount of robot-like feeling
90
80
70
60
50
40
30
20
y = 0.7238x + 8.4322
10
R² = 0.4254
0
0 10 20 30 40 50 60 70 80 90 100
Amount of reaction force
Fig. 8. Relationship between the robot-like feeling and the reaction force
Realism of Robot-Like Body Sense. Three participants commented that they felt
creaking in the conditions using haptic feedback (i.e., V+H and V+A+H). This com-
ment implies that the haptic feedback of robot vibration could produce a feeling of
creaking friction to some participants. Also, two participants reported that they felt as
if the robot arm model on the monitor became their right arm, because the robot mod-
el was synchronized with the movement of their real arm. As reported in [20],
synchronous movement of the virtual arm and the real arm can facilitate the body-
ownership illusion. However, we intend to improve the level of the body-ownership
illusion in the future studies. Completely hiding the participant’s real arm and over-
laying the virtual robot arm would be one promising approach in the facilitation of
this illusion.
According to these comments and the evaluation of the robot-like feeling and reac-
tion force, it was confirmed that the integration of robot vibration, creaking sound
effects, and the visual robot model synchronized with the user’s motion could cause
the participant to feel that their body had become robotic.
acceleration data, to which a 1.2 kHz low-pass filter was applied. The lack of high-
frequency components might cause an auditory mismatch between the generated
sound and the original noise.
To verify this effect, we performed an experiment that recorded robot sound using
a microphone (Gigaware 60139B, RadioShack Corp.) and the sound feedback at a
refresh rate of 22.05 kHz. However, the participants could not discriminate between
the acceleration-based sound and the sound-based sound. Thus, the lack of high-
frequency sound does not seem to play an important role in the auditory mismatch
feeling.
Another reason for the auditory mismatch feeling might be that we employed an
industrial robot to record vibration. The participants were unfamiliar with the sound
of an industrial robot; in fact, they had never seen this kind of robot before the expe-
riment, so they could not know how it should sound.
Matching the user’s image of the robot-like feeling would be an important future
study. One possibility is to show a movie of the PUMA 260 actuation to allow the
participants to experience a specific robot sound before the evaluation task. In con-
trast, the use of a representative robot sound that most people imagine is an alternative
idea in generating a convincing robot-like feeling. In science fiction movies, for ex-
ample, sound effects representing robot actuation are not at all like a real robot actua-
tion sound.
6 Conclusion
This paper presented a method to create a robot-like body sense, aiming for a new
entertainment experience as if the human user had actually become a robot. We
proposed the vibration recording of real robot actuation, data-driven modeling based
on spectral approximation, and vibrotactile rendering to the user’s elbow as a function
of the elbow angular velocity. We also developed a system that virtually robotized the
human arm visually, aurally and haptically by means of integrating a visual robot
model that tracks the user’s arm motion and produces a creaking sound and vibrotac-
tile feedback. Using this system, we compared four sensory conditions to evaluate the
participants’ subjective robot-like feeling and perceived reaction force. This experi-
ment revealed that the combination of visual, auditory, and haptic feedback was the
most effective in inducing a robot-like feeling. The pseudo reaction force, which
might also reflect a robot-like feeling, was generated most strongly with this combina-
tion. Additionally, some comments from the participants suggested that our approach
can simulate friction of the robot joint.
We intend to upgrade our system to an augmented reality (AR) system using a vid-
eo see-though head mounted display (HMD) so that the users can see their own body
visually changed into that of a robot (Fig. 9). A camera mounted on the HMD cap-
tures the user’s subjective view and tracks markers attached on the arms. The HMD
then superimposes virtual robot arms on the user’s arms. The AR system will provide
an even more immersive experience of the robotized body.
HMD view
References
1. TN Games, FPS Gaming g Vest, http://tngames.com/products (last access: Au-
gust 18, 2013)
2. Ooshima, S., Hashimoto, Y., Ando, H., Watanabe, J., Kajimoto, H.: Simultaneous Presen-
tation of Tactile and Aud ditory Motion to the Abdomen to Present the Feeling of Beeing
Slashed. In: Proceedings of
o the SICE Annual Conference, pp. 467–471 (2008)
3. McMahan, W., Kuchenbecker, K.J.: Spectral subtraction of robot motion noise for im-
proved event detection inn tactile acceleration signals. In: Isokoski, P., Springare, J. (eeds.)
EuroHaptics 2012, Part I. LNCS, vol. 7282, pp. 326–337. Springer, Heidelberg (2012)
4. Ince, G., Nakadai, K., Ro odemann, T., Hasegawa, Y., Tsujino, H., Imura, J.I.: Ego nnoise
suppression of a robot using template subtraction. In: Proceedings of the IEEE/RSJ Innter-
national Conference on Inntelligent Robots and Systems (IROS), pp. 199–204 (2009)
5. Visell, Y., Law, A., Coo operstock, J.R.: Touch is everywhere: Floor surfaces as ambbient
haptic interfaces. IEEE Trransactions on Haptics 2, 148–159 (2009)
6. Cirio, G., Marchal, M., L’ecuyer,
L A., Cooperstock, J.R.: Vibrotactile rendering of splaash-
ing fluids. IEEE Transactiions on Haptics 6, 117–122 (2012)
7. Okamura, A.M., Cutkosk ky, M.R., Dennerlein, J.T.: Reality-based models for vibraation
feedback in virtual enviro
onments. IEEE/ASME Transactions on Mechatronics 6, 245––252
(2001)
8. Hachisu, T., Sato, M., Fukushima, S., Kajimoto, H.: Augmentation of material property by
modulating vibration resulting from tapping. In: Isokoski, P., Springare, J. (eds.) EuroHap-
tics 2012, Part I. LNCS, vol. 7282, pp. 173–180. Springer, Heidelberg (2012)
9. Romano, J.M., Kuchenbecker, K.J.: Creating realistic virtual textures from contact accele-
ration data. IEEE Transactions on Haptics 5, 109–119 (2012)
10. Takeuchi, Y., Kamuro, S., Minamizawa, K., Tachi, S.: Haptic Duplicator. In: Proceedings
of the Virtual Reality International Conference, pp. 30:1–30:2 (2012)
11. Minamizawa, K., Kakehi, Y., Nakatani, M., Mihara, S., Tachi, S.: TECHTILE Toolkit: A
prototyping tool for designing haptic media. In: Proceedings of the ACM SIGGRAPH
2012 Emerging Technologies, p. 22 (2012)
12. Kurihara, Y., Hachisu, T., Sato, M., Fukushima, S., Kajimoto, H.: Virtual alteration of
body material by periodic vibrotactile feedback. In: Proceedings of the IEEE Virtual Reali-
ty Conference (2013)
13. Wellman, P., Howe, R.D.: Towards realistic vibrotactile display in virtual environments. In:
Proceedings of the ASME Dynamic Systems and Control Division, vol. 57, pp. 713–718
(1995)
14. Goodwin, G.M., McCloskey, D.I., Matthews, P.B.C.: The contribution of muscle afferents
to kinaesthesia shown by vibration induced illusions of movement and by the effects of pa-
ralysing joint afferents. Brain 95, 705–748 (1972)
15. Burke, D., Hagbarth, K.E., Löfstedt, L., Wallin, G.: The responses of human muscle spin-
dle endings to vibration of non-contracting muscles. J. Physiol. (Lond.) 261, 673–693
(1976)
16. Naito, E.: Sensing limb movements in the motor cortex: How humans sense limb move-
ment. Neuroscientist 10, 73–82 (2004)
17. Lackner, J.R.: Some proprioceptive influences on the perceptual representation of body
shape and orientation. Brain 111, 281–297 (1988)
18. Botvinick, M., Cohen, J.: Rubber hands “feel” touch that eyes see. Nature 391, 756 (1998)
19. Tsakiris, M.: My body in the brain: A neurocognitive model of body-ownership. Neurop-
sychologica 48, 703–712 (2010)
20. Slater, M., Perez-Marcos, D., Ehrsson, H.H., Sanchez-Vives, M.V.: Inducing illusory
ownership of virtual body. Frontiers in Neuroscience 3, 214–220 (2009)
21. Okamura, A.M., Webster, R.J., Nolin, J., Johnson, K.W., Jafry, H.: The haptic scissors:
Cutting in virtual environments. In: Proceedings of the IEEE International Conference on
Robotics and Automation (ICRA), pp. 828–833 (2003)
22. Romano, J.M., Yoshioka, T., Kuchenbecker, K.J.: Automatic filter design for synthesis of
haptic textures from recorded acceleration data. In: Proceedings of the IEEE International
Conference on Robotics and Automation (ICRA), pp. 1815–1821 (2010)
23. Dabrowski, J.R., Munsone, V.: Is 100 milliseconds too fast? In: Proceedings of the ACM
Human Factors in Computing Systems (CHI), pp. 317–318 (2001)
Abstract. We propose that pervasive games designed with mechanics that are
specifically in opposition with, or disruptive of, social rules of the environment
in which they are played, have unique potential to provide interesting, provoca-
tive experiences for players. We explore this concept through the design and
evaluation of an experimental game prototype, Shhh!, inspired by the juvenile
game Bollocks, and implemented on Android mobile devices, which challenges
players to make loud noises in libraries. Six participants played the game before
engaging in semi-structured interviews, explored through inductive thematic
analysis. Results suggest that the game provoked in players a heightened
awareness of social rules, as well as a complex social dilemma of whether or
not to act. We conclude by presenting a model for designing games that play
with the social, as well as physical, rules of the environments in which they
are set.
1 Introduction
Pervasive games [22,24], location-based games [5], and mixed reality games [7,9] are
all terms that describe entertainment computing applications in which the geography
of the real world functions as an essential component of game play. These games
typically use mobile computing technologies such as GSM [30], Bluetooth [8,27],
WiFi [4] GPS, or augmented reality [8] in order to incorporate the movement of play-
ers through the real world as part of the fantasy narrative of the game. The assumption
is that game tasks can make real world locations and activities more interesting or
meaningful [28]. However, there is a surprising lack of variety in the design of these
games and in the types of experiences they are designed to provoke. The majority take
the form of treasure hunts, where players must visit real-world locations in order to
tick off game-world tasks. Rarely do these games acknowledge or encourage players
to engage with interesting physical or social features of the environments in
which they are played. It appears that the motivation behind developing these games
is often to explore the capabilities of the enabling technologies, rather than to provide
interesting experiences for players.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 123–137, 2013.
© Springer International Publishing Switzerland 2013
Researchers have recently argued [20,21] that pervasive games have relatively un-
explored potential to provide engaging experiences through provoking players to
interact meaningfully with already interesting real-world environments. For example,
the game Blowtooth [20,21] is designed to explore the unique affordances of interna-
tional airports (see [16]). The game narrative requires players to smuggle virtual
goods through real airport security by planting them on fellow passengers, before later
tracking those passengers down and retrieving their goods. Blowtooth is extremely
simple both technologically and in terms of game design. Kirman, Linehan and Law-
son [20] argue that, despite its simplicity, the game provokes interesting experiences
for players, because, through the game mechanics, the player is confronted with as-
pects of the environment that already cause anxiety and exhilaration; in this case the
unparalleled security and surveillance of the airport environment. Conversely, the
game is not interesting or fun, or even a coherent game, if played anywhere other than
an airport.
The Blowtooth game provides an interesting starting point for the current paper.
Specifically, we were interested in exploring further the concept of designing simple
pervasive games that interact provocatively with the implicit features of built envi-
ronments (i.e., the human-made surroundings that provide the setting for human activ-
ity [26]). It must be noted that Blowtooth gained much of its engaging power from the
extraordinary nature of the airport environment. Here, we aim to investigate whether
similarly interesting experiences can be designed for more commonly experienced
environments. Specifically, we propose that these experiences can be achieved
through the design of game mechanics that are in opposition with, or disruptive of,
social rules that exist in everyday built environments. Indeed, the mischievous break-
ing of social rules is a type of play that many find engaging [19]. Of course, all games
are social experiences that take place in social spaces, as discussed at length in the
literature (i.e., [18,29]). Here we investigate specifically how pervasive games can
create engaging experiences through encouraging people to consider, explore, and
play with, existing social rules of their environment.
The following sections of the paper will explore the rationale for designing perva-
sive games that ask the player to engage in the mischievous breaking of social rules.
We first discuss how social rules effect the expression of behaviour, and identify the
built environment as a particular type of behavioural cue. We then discuss mischief
and naughtiness as a game play aesthetic, and consider the effects of mischievous
play on non-players. Subsequently, we present the design and evaluation of a game,
Shhh!, which challenges players to make loud noises in libraries.
2 Background
Social rules are essential to all game playing. For example, the ‘Magic Circle’ [15,29]
is a term that describes the unspoken agreement about acceptable behaviour in the
social context of game playing. The game-playing context provides players with cues
for, and gives permission for, many types of behaviour that would be unacceptable or
confusing in other contexts. Similarly, there are certain behaviours, such as cheating,
which are unacceptable in the game-playing context. Thus, games inherently function
as powerful social cues for appropriate behaviour and, indeed, all games are inher-
ently social experiences [18,29].
Mischievous play is a type of behaviour that serves to explore and test the bounda-
ries of social acceptability in game playing contexts [19]. Mischievous players enjoy
subverting and appropriating game rules and social expectations of the game-playing
context in order to produce surprising and entertaining experiences for themselves and
other players. For example, in farming games, players create elaborate pictures using
variety of crops [17], and in Spore [23], players used design tools to make humorous
creatures modelled after various parts of human anatomy [19].
Importantly, there is a distinction between mischievous play and genuinely anti-
social behaviour. The key to mischief is the apparent attitude of playfulness. The mis-
chievous player knows there are limits to their behaviour, and the intent is to do no
harm [13]. This is in stark contrast to the intent of griefing or trolling, which is pur-
posefully disruptive, often with negative and anti-social intentions. Researchers have
argued that mischievous play is a valid, interesting and valuable form of game play
behaviour, which designers should acknowledge and facilitate [19].
Kirman, Linehan and Lawson [19] primarily discuss mischievous play in terms of
playing with the social environment that surrounds game playing. However, there are
games that encourage players to mischievously explore and test the boundaries of
social acceptability in other environments (i.e., beyond the safety of the magic circle
of a game). For example, consider the game played by school children where a (typi-
cally obscene) word is chosen and players must take turns to speak the word in class
(generally called “Bollocks” in the UK and Ireland). Each successive player must say
the word louder than the previous, until somebody is caught by the teacher, or gives
in. The attraction of the game lies in the tension between the rules of the environment
(stay silent) and the rules of the game (make a noise), creating an exhilarating social
experience. Another example is that of juvenile “kissing” games (e.g. “Spin the Bot-
tle”), which give players permission to explore the social boundaries of intimacy.
We propose an extension to the conclusions of Kirman, Linehan and Lawson [19],
who insist that mischievous play in game playing contexts should be acknowledged and
supported. We suggest that mischievous playing with social rules can form the basis of
exhilarating, memorable experiences beyond traditional game playing contexts. We call
for the design of games that specifically encourage players to engage in behaviour that
is in opposition with, or disruptive of, the social rules of built environments. We suggest
that this approach may be particularly beneficial to the design of pervasive games. Since
buildings function as powerful signals for controlling the expression of behaviour, and
games do similarly, the playing of games in the built environment may present complex,
challenging social experiences for players, particularly in situations where the rules of
the game are in competition with the rules of the environment for control over player
behaviour. This could be seen as an example of a 'dark gameplay pattern' that intention-
ally causes emotional dilemmas for players [31].
just a measure of loudness. Thus, the game does not infringe upon non-player privacy,
according to the definition provided by Niemi et al. It could be argued, however, that
encouraging noise in a quiet environment is in some way an infringement on privacy.
We must remind the reader that the intention is not to provoke players into making
lots of noise, rather, to encourage in players a heightened awareness of how well de-
fined and understood the social rules are, as well as the implications of breaking those
social rules. We envision that few, if any, players will cause any serious disruption to
fellow library users. Further, any noise that players do create will be rare, brief and
more than likely contextually appropriate. This is something that we will investigate
in the user study.
Accountability refers to whether actions are traceable to the source, who can be
held accountable for any adverse effects caused [14,25]. In the current game, the po-
tentially invasive behaviour (the making of noise) is inseparably and observably
linked with the player (i.e., the person who makes the noise). Not only is the player
easily held accountable for their actions by non-players, but that accountability is
actually a core component that drives fun of the game. Indeed, if players do not feel
any compulsion to remain quiet, the game will not provide an interesting experience.
The experience that the game is designed to provoke is the complex dilemma of
whether or not to act. This dilemma serves as an analogue for some of the most chal-
lenging situations that we commonly experience, which often involve the strong urge
to act but reluctance to do so because of social norms; such as whether to intervene in
arguments between spouses, or the need to tolerate the extreme political beliefs of a
relative at a family gathering. We believe that pervasive games can provide fascinat-
ing, memorable experiences through allowing players to explore exactly these types
of complex social dilemmas.
3 Game Design
The experimental game prototype, called Shhh!, was inspired by the children’s play-
ground game, mentioned above, which dares players to make loud noises in inappro-
priate situations (usually the classroom). This game provides an interesting basis for
our exploration of social rules of built environments as game mechanics, due to the
combination of simple mechanics and the genuine excitement it encourages in play-
ers. We decided to set the game in libraries, as these are buildings that have obvious
and easily understood rules that are (primarily) enforced by social convention (i.e.,
disapproval by other library users is a much more likely consequence of play than
formal action by the library staff). Moreover, since we had a great deal of experience
with observing the social rules of the library, were confident that players would feel
compelled to engage in socially acceptable behaviour, thus creating for players the
desired social dilemma of whether or not to act.
The original playground game was altered through the development of an applica-
tion that runs on Android mobile devices. Specifically, the application uses the
phones’ audio input to measure sound levels. This allows players, upon making a
noise, to see a score that corresponds to the sound that they made. It also allows for
that score to be saved to a leader board, facilitating asynchronous play.
Upon reaching a library in which they wish to play, the player launches the game
application on their mobile device. The application initially verifies whether the
player is genuinely in a library (this feature was not implemented in the prototype
evaluated), before presenting the main input screen (figure 1). This screen is com-
posed primarily of one large button, which, when pressed, activates the device’s
microphone. The player is expected to make a noise at this point. The application
identifies the volume of sound in the environment at that moment (using an algorithm
that averages values returned by the ‘getMaxAmplitude’ function of the ‘Media Re-
corder’ utility within the Android OS). When they have finished making noise, the
player clicks the button again to stop recording. The application returns a score to the
player; louder noises produce higher scores. This score is then added to both the over-
all league table for the game, plus a league table for the specific library in which
they played. The local league table facilitates people to compete with their friends,
colleagues and classmates in an uncomplicated manner.
Fig. 1. Screenshots of prototype application. Left panel shows the input screen. Right panel
shows the leader board.
explanation of the game was presented. A smart phone, on which the game had been
installed, was then given to the participant and they were asked to carry out three
simple tasks using the application to ensure that they understood how it worked; view
the local high scores, view the global high scores, sign in. Participants were then
asked to enter the library, to try to “get a high score,” and to return when they were
finished. Subsequently, participants were taken to a more relaxed environment, where
a semi-structured interview took place to understand the player’s experience of play-
ing the game. The researcher began by asking participants how they felt generally
about the experience, before asking specifically about the library environment, how
they felt about making noise and playing games in that environment, and whether
other library users reacted in any way to their behaviour. Participant responses were
audio recorded and later transcribed. These transcripts form the basis of the analysis
presented below.
Interview transcripts were analysed through inductive thematic analysis [6], a form
of qualitative analysis particularly useful for investigating novel subjects in little-
understood domains. Data was first read carefully multiple times by the first author.
The structure of the data was then broken down to allow for analysis. Specifically,
each separate concept (often, but not necessarily corresponding to a sentence) ex-
pressed by a participant was assigned a separate row in a spreadsheet. A total of 24
relevant conceptual labels, formed of short sentences and quotes, were derived from
the corpus of interview data by the second author. These, together with a description
and examples of each code, were given to the first author, who analysed the data in-
dependently and examined the fit of the codes to the data. Results of both analyses
were compared and consensus reached on a code list of 13 categories.
Six first-order themes were identified, which formed two second-order themes; con-
siderations of the library environment, and considerations of game play. The coding
scheme is illustrated in (Figure 2). While the themes identified appear to represent
distinct concepts, it must be noted that they all discuss aspects of the tension between
playing the game and behaving appropriately in the library. Hence, participant’s utter-
ances could often be classified under a number of these codes. However, we are
confident that this coding structure aids the reader in understanding the subjective
experience of participants. The identified themes are expanded upon in detail below.
All quotes are presented unedited.
Own Behaviour. This theme refers to instances when participants expressed acute
awareness of their own behaviour while playing the game. This is interesting as it
provides some insight into their willingness (or reluctance) to engage with the game
task, as well as the impact of the social environment on their behaviour. Participants
often regretted making sounds and drawing attention to themselves;
“….I was stood in a stairwell where it was quite busy and I just yelled ‘mic check’
as if I was checking the microphone to get the levels right and I realised it was
quite loud…..I sort of looked at it and went ‘ I’m in a library I need to be quiet’”
(P6).
The majority of the participants reported feeling out of place and out of their comfort
zone by playing the game in the environment chosen. P1 suggested, “it was a bit
weird to be honest and very strange. The word most commonly used by participants
when describing how they felt when playing the game was “awkward”. They re-
ported feeling awkward while walking around the library devising strategies for mak-
ing sounds that wouldn’t draw attention to them. Another participant mentioned that
they spent a lot of the time sitting at a desk, speaking random words into the phone,
which led to them feeling “rather odd.”
Many participants tried to blend into the environment while playing the game, so
that non-participants wouldn’t think they were acting strangely or inappropriate. For
example, P5 reported,
“…the librarian walked passed me while I was sat at a desk and I turned the phone
off and just sort of opened up an email from my tutor and pretended to compose an
email…”
A similar experience was reported by P3, who sat at a desk surrounded by books
while playing the game, in order to give the appearance of doing work;
“…I also sat at a desk by myself with some books, so I made a look round to make
sure no one could see me…”- P3.
P2 was very uncomfortable with playing the game;“…erm I played it for literally
five minutes maybe a bit less and I felt like I didn’t want to play anymore.” Interest-
ingly, despite only staying in the library for a few minutes, the above participant re-
ported one instance of shouting directly into the microphone at the top of their voice.
“I was like, I cannot get any louder than what they have done …” In fact, it was after
making this very loud sound that the participant refused to play anymore due to peo-
ple staring at them.
The finding that the majority of participants felt very aware of their own behaviour,
often to the point of awkwardness, suggests that they were very aware of the social
environment while playing. However, it must be noted that only one participant (P1)
failed to engage enthusiastically with the task. This participant reported discomfort at
the task that the game asked them to carry out, “it had a whole stigma over the noise
levels." Thus, despite their awkwardness and reluctance, most players did seem to
engage (cautiously) with the game, taking steps to minimise social disapproval.
Rules of the Environment. This theme examines how players’ awareness of the so-
cial rules of the library environment affected their experience of playing the game.
Most participants reported initially respecting the rules of the environment, “When I
was first in there I was quite reserved ‘cus I didn’t want to make too much noise in
the library” (P1). Interestingly, while most participants felt like this initially, none of
them refused to engage with the game entirely.
In the particular library where this research was carried out, there are numerous
floors, only one of which has a strict rule on absolute silence. It seems that, out of
respect for the rules, participants largely avoided playing games on that floor, “…you
see I was on the first floor, you’re allowed to speak and do what you want, if I was on
the third floor the yes, obviously.” That participant was then asked if they had been on
the third floor would they have played the game or been too afraid to. They responded
with “I would have felt to embarrassed and scared to do it.” Thus, even while playing
an intentionally disruptive game, participants carefully adapted their behaviour to
subtle differences in social rules in different parts of the same building. This finding is
fascinating, as it illustrates the dilemma that players of Shhh! felt they were faced
with; the complex calculations involved in balancing the competitive, goal directed
behaviour of game playing versus the social rules of the environment in which it was
played. Players wanted to win, but they didn’t want to be embarrassed or to get
caught.
Since the third floor of the library in which we held the study has uniquely strict
rules on noise levels, we assumed that absolutely none of the players would attempt to
play there. However, this was not the case. Two participants saw the strict rules of
this floor as a challenge. Fascinatingly, players adapted the sounds they made due to
the less forgiving environment to something more “subtle and something you could
get away with” (P5). These participants felt the need to make sounds that seemed
natural in that context, for example, dropping books on the floor. P6 tried blowing
into the microphone as a way of making noise. However, instead of simply picking
the phone up blowing into the microphone the participant reported, "I'll pick up the
phone and look a bit confused into the phone and blow into the mic." (P6). The par-
ticipant felt the need to put on an act in order to avoid disapproval.
The playing of the game on this floor by a minority of players is interesting, but
not unproblematic, since it is not clear whether it crosses the line between mischief
and genuinely antisocial behaviour.
“…you could almost argue that competition in this game almost encourages ex-
treme behaviour or cheating because you can’t beat them without extreme behav-
iour.” (P5).
However, these players did not simply go into the quiet area and shout into their
phones, they attempted creative ways of gaining high scores that had less likelihood
of bringing about undesirable consequences, and thus were interacting thoughtfully
within subtle social rules of the social environment. In social terms, playing in this
location had a higher likelihood to cause embarrassment for the player, to cause
disruption to non-players, and, indeed, to bring about disciplinary action for the
player. In gaming terms, these players saw this floor as having a higher level of diffi-
culty and were intrigued and challenged to produce a high score in this less forgiving
environment.
Competition. Competition was facilitated through a simple leader board in the appli-
cation. Discussions of competition are interesting, as they demonstrate that the game
design, while incredibly simple, was engaging enough to present participants with the
dilemma of whether or not to break the social rules of the library environment.
Most participants reported competition as a motivator for their continued play, “It
said the world high score was 112. So I aimed for to beat that and I did beat that. I
didn’t smash that record but I beat it! Will you acknowledge that I beat your score?”
(P3). Other participants found that they just wanted to get on to the leader board and
not worry about being the best, “…I mean I saw the scores and I wasn’t aiming for
the top I was aiming to go mid table and I ended up score quite high on my first few
tries….” (P6). When participants saw that their scores were close to the record, it
made them want to play more, “…that’s when I thought, you know what if I’m going
to make my mark on the game, you got to set the bar for someone so I just raised the
bar an extra bit higher.” (P6).
It became apparent through analysis of the data that what was happening in the
game was being discussed amongst friends. For example, a number of participants
reported that, before they participated, they discussed the game with people who had
already played it. Rather than being upset at the apparent corruption of the naivety of
our participant pool, we saw this as evidence of the engaging nature of the game and
the inherent competition that if provoked. “…I heard from a previous person … that
he coughed also and I thought I would just try and beat him” (P1); “Well yeah, I
wanted to beat my house mate.” (P2); “it is more fun in a group” (P1). Participants
mentioned that their discussions of the game pushed them to get higher scores, as they
dared other players to beat their scores, “I mean I could twang a bass string or hit a
drum and be like ‘yeah go on, try and beat my score, I dare you” (P6); it is bragging
rights really……So you know it is good fun to see that I had taken over all the leader
board, absolutely” (P6).
Strategies for Gaining High Scores. The only action available in the game was to
make and record a noise. However, this action was in direct contradiction of the social
rules of the library environment. Thus, it is interesting to understand the strategies
adopted by players in making those noises, as they give some indication of the di-
lemmas faced in playing this game. It seems that generally the strategy was to use
sounds that are naturally loud, but also not uncommon in library environments. Figure
3 presents a summary of the different strategies reported by players. As seen in figure
3 many participants chose to play by coughing into the microphone. This strategy was
chosen in an effort to not draw attention to themselves. When asked whether they
tried anything else, many replied in the negative. Participant 1’s response was this:
“No, as I didn’t want people looking at me in a funny way”.
Fig. 3. Number of players wh ho reported using each of a number of strategies for generaating
sounds. Note: some players used multiple strategies.
A number of participantss did have creative ideas to make sounds, often choosingg to
t create loud noises without incurring the disapprovall of
stage ‘accidents’ in order to
fellow library users,
“…as you know they ha ave those little step stool kind of things to get to the top
shelves. I sort of consideered walking down the aisle with a book and not notice aand
just kick it by “mistake”” and maybe try and pretend I sort of fell over. But yeaah I
didn’t do that in the end” (P5).
This combination of creaativity and reluctance demonstrates the tension between the
social rules and game ruless, and the complex dilemma experienced by game playyers.
As shown in figure 3 the most
m popular ‘accident’ was the dropping of books on the
floor. Indeed, this strategy often led to high scores for the participants. Other playyers
slammed doors, and tapped d the device on objects such as desks. Participants preefer-
ence to be seen as clumsy rather than intentionally loud demonstrates the powerful
need for social acceptance ini this environment.
Many of the participantss felt that they couldn’t achieve a high score by simply us-
ing their own voice or by making
m noises with books and door. This led to them too, in
their own words, cheating, through using the hand dryer in the toilets to gain a scoore.
Specifically, participants walked in to the toilets and put their hands under the dryeer to
set it off, put the phone underneath
u and pressed ‘Go!’ This produced a very hhigh
score. In discussion, these participants
p mentioned they did this because they coulddn’t
find any other way of ach hieving the high score without shouting “…I was kindd of
coughing really loudly….to o beat him, but I didn’t so I thought I will cheat because I
want that score!” (P5). Partticipants were apparently more willing to break the rules of
the game than the rules of thhe environment.
In summary, due to the competing
c demands of the game rules and the social ennvi-
ronment, players demonstrrated great creativity in adapting their behaviour to social
expectations. This emphasiises the social dilemma presented by the game as welll as
the power of the social env vironment over player behaviour. Players were reluctannt to
ever cause significant disrupption to non-players.
6 Conclusions
This paper proposes the harnessing of existing social rules in built environments as a
basis for designing provocative and engaging pervasive games. An experimental
game, Shhh!, was designed, which challenged players to make noise in an environ-
ment (a library) where that is acknowledged as inappropriate behaviour. Six partici-
pants played the game before engaging in semi-structured interviews. Participants
reported a keen awareness of the rules of the social environment in which they played.
While they demonstrated a willingness to play the game, and enjoyment at doing so,
they also demonstrated a commitment to cause as little disruption as possible to non-
players, taking steps to minimise social disapproval. Players showed remarkable sen-
sitivity and creativity in adapting their behaviour to social expectations, and indeed,
demonstrated great complexity of behaviour in playing with the rules of the social
environment. These findings suggest that pervasive games that play with social rules
of built environments have great potential to provide interesting, challenging and fun
experiences for players.
Since participants in this study found their playing with social rules a challenging
and engaging activity, it may be worth exploring this design strategy further. Specifi-
cally, few pervasive games have explored how the social rules inherent in the built
environment can function as part of pervasive game play mechanics. We propose a
model for designing pervasive games that acknowledges the engaging potential of the
social, as well as physical, characteristics of a built environment (see Table 1). This
model describes how, when designing the mechanics of a pervasive game, we should
consider devices and mobility as means for players to navigate the physical environ-
ment, and the subjective experience and behaviour of the player as means to navigate
the social environment.
Table 1. Description of the features that must be considered in the design of pervasive games
In designing games that play with existing social rules it is important to consider
the line between mischievous and antisocial behaviour. Mischievous behaviour is
playful and serves to explore and test the boundaries of social acceptability [19]. This
is contrast to the intent of griefing or trolling, which is purposefully disruptive, often
with negative and anti-social intentions. As pervasive game designers, we must con-
sider our own behaviour in these terms, ensuring that while our games allow for an
exploration and questioning of social rules, they don’t simply give permission for
players to behave in an antisocial manner (for example, ‘dark play’ [31]). We do not
want to lower the barriers to behaviour that could cause real harm to the experienced
quality of life of non-players. In this respect, we were guided by Niemi, Sawano and
Waern [25], in understanding gameplay in terms of anonymity and accountability. In
the design of the current game we speculated, based on a great deal of experience
with the environment in question, that players were unlikely to ever cause significant
disruption to the library. However, we could not be certain of this without carrying
out a user study. Findings suggested that players were extremely reluctant to engage
in any majorly disruptive behaviour, and indeed, went to great lengths to only gener-
ate context-appropriate noises. We suggest that an observational user study is particu-
larly important, for ethical reasons, when designing a game that plays with social
expectations in order to ensure that the game does not unintentionally provoke or
facilitate genuinely antisocial behaviour.
The approach to pervasive game design outlined here may serve useful for the de-
sign of games for purposes other than entertainment (see [11]). Indeed, the approach
seems ideal for provoking in players critical reflection on the values underlying the
design of built environments, the social rules inherent in those environments, and the
ways in which our behaviour is controlled by those rules. This seems to align well
with the goals of critical design [10], which “provides a critique of the prevailing
situation through designs that embody social, cultural or technical values” (p.58).
Indeed, the approach outlined here may represent a uniquely powerful type of critical
design, since it allows the designer to provoke users to reflection in the very environ-
ment the artefact is intended to criticise.
References
1. Bandura, A., McClelland, D.C.: Social learning theory. Prentice Hall, NJ (1977)
2. Bandura, A.: Social cognitive theory: An agentic perspective. Annual Review of Psychol-
ogy 52, 1–26 (2001)
3. Baron, R.A., Byrne, D.E., Branscombe, N.R.: Social psychology, 9th edn. Holt, Rinehart
and Winston (1999)
4. Bell, M., Chalmers, M., Barkhuus, L., Hall, M., Sherwood, S., et al.: Interweaving mobile
games with everyday life. In: Proceedings of ACM CHI, pp. 417–426. ACM (2006)
5. Benford, S., Anastasi, R., Flintham, M., Greenhalgh, C., Tandavanitj, N., et al.: Coping
with uncertainty in a location-based game. Pervasive Computing 2, 34–41 (2003)
6. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Research in
Psychology 3, 77–101 (2006)
7. Cheok, A.D., Yang, X., Ying, Z.Z., Billinghurst, M., Kato, H.: Touch-space: Mixed reality
game space based on ubiquitous, tangible, and social computing. Personal and Ubiquitous
Computing 6, 430–442 (2002)
8. Cheok, A.D., Goh, K.H., Liu, W., Farbiz, F., Fong, S.W., Teo, S.L., et al.: Human pacman:
A mobile, wide-area entertainment system based on physical, social, and ubiquitous com-
puting. Personal Ubiquitous Computing 8, 71–81 (2004)
9. Crabtree, A., Benford, S., Rodden, T., Greenhalgh, C., Flintham, M., et al.: Orchestrating a
mixed reality game’on the ground’. In: Proc. ACM CHI, pp. 391–398. ACM (2004)
10. Dunne, A., Raby, F.: Design noir: The secret life of electronic objects. Springer (2001)
11. Flanagan, M.: Critical Play: Radical Game Design. MIT press (2009)
12. Gentile, A.P.: Reinventing Airspace: Spectatorship, fluidity, intimacy at PEK T3. J of Ar-
chitecture, City & Environment 4, 9–19 (2009)
13. Foo, C.Y., Koivisto, E.M.I.: Defining grief play in MMORPGs: Player and developer per-
ceptions. In: Proc. of ACE 2004, pp. 245–250. ACM (2004)
14. Friedman, B., Kahn Jr., P.H., Borning, A.: Value sensitive design and information systems.
In: Human-Computer Interaction in Management Information Systems: Foundations,
vol. 4 (2006)
15. Huizinga, J.: Homo ludens. Rowohlt, Hamburg (1956)
16. Kellerman, A.: International airports: Passengers in an environment of ‘authorities’. Mo-
bilities 3, 161–178 (2008)
17. Kirman, B.: Emergence and playfulness in social games. In: Proceedings of Mindtrek, pp.
71–77. ACM (2010)
18. Kirman, B.: Playful networks: Measuring, Analysing and Understanding the Social Effects
of Game Design. PhD Thesis, University of Lincoln (2011)
19. Kirman, B., Linehan, C., Lawson, S.: Exploring mischief and mayhem in social computing
or: How we learned to stop worrying and love the trolls. In: Proceedings of ACM CHI, pp.
121–130. ACM (2012)
20. Kirman, B., Linehan, C., Lawson, S.: Blowtooth: A Provocative Pervasive Game for
Smuggling Virtual Drugs through Real Airport Security. Personal and Ubiquitous Comput-
ing 16, 767–775 (2012)
21. Linehan, C., Kirman, B., Lawson, S., Doughty, M.: Blowtooth: Pervasive gaming in unique
and challenging environments. In: Proc. ACM CHI Extended Abstracts, pp. 2695–2704.
ACM (2010)
22. Magerkurth, C., Cheok, A.D., Mandryk, R.L., Nilsen, T.: Pervasive games: Bringing com-
puter entertainment back to the real world. Computers in Entertainment 3, 4 (2005)
23. Maxis. Spore. PC: Electronic Arts (2008)
24. Montola, M., Stenros, J., Waern, A.: Pervasive Games: Theory and Design. Morgan
Kaufmann (2009)
25. Niemi, J., Sawano, S., Waern, A.: Involving non-players in pervasive games. In: Proc. 4th
Conference on Critical Computing (2005)
26. Rapoport, A.: The meaning of the built environment: A nonverbal communication ap-
proach. University of Arizona Press (1982)
27. Rashid, O., Mullins, I., Coulton, P., Edwards, R.: Extending cyberspace: Location based
games using cellular phones. Computers in Entertainment 4, 4 (2006)
28. Reid, J.: Design for Coincidence: Incorporating real world artifacts in location based
games. In: Proceedings of the 3rd International Conference on Digital Inter-Active Media
in Entertainment and Arts, pp. 18–25. ACM (September 2008)
29. Salen, K., Zimmerman, E.: Rules of Play: Game Design Fundamentals. MIT Press (2003)
30. Sotamaa, O.: All the World’s a Botfighter Stage: Notes on Location-Based Multi-User
Gaming. In: CGDC Conference Proceedings, pp. 35–45 (2002)
31. Zagal, J.P., Björk, S., Lewis, C.: Dark Patterns in the Design of Games. In: Proceedings of
Foundations of Digital Games (2013)
Abstract. Cuddly is a mobile phone application that will enchant soft objects to
enhance human’s interaction with the objects. Cuddly utilizes the mobile
phone’s camera and flash light (LED) to detect the surrounding brightness value
captured by the camera. When one integrate Cuddly with a soft object and
compresses the object, the brightness level captured by the camera will
decrease. Utilizing the measurement change in brightness values, we can
implement diverse entertainment applications using the different functions a
mobile phone is embedded with, such as animation, sound, Bluetooth
communication etc. For example, we created a boxing game by connecting two
devices through Bluetooth; with one device inserted into a soft object and the
other acting as a screen.
1 Introduction
We are surrounded by soft objects such as plush toys, cushions, mattress, sofas and
others in our daily lives. Soft objects act as a buffer between people and hard objects
such as the floor or furniture. People often hug soft objects when they are feeling
emotional such as while watching movies, or they tend to punch or throw soft objects
when they are feeling frustrated. These actions show that there is a close emotional,
physical interaction between soft objects and oneself. From the psychological point of
view, soft objects can highly influence people’s lives and behavior [1]. Research
shows that it can help reduce stress as well as increase effectiveness [18]. Because of
that, soft objects are often used in different fields such as medical therapy [25],
communication [12], gaming purposes [19] etc. Our research group has also created
FuwaFuwa, a sensing module that uses soft object as an interface in the past.
However, in order to develop and commercialize these soft interfaces and robots to
the society, developers require users to purchase the device. When choosing a pet
from a wide variety of animals or just choosing, for example, a pet dog based on its
breed, people make their choices based on their own likes and dislikes. Similarly,
when choosing soft objects, the same rules apply. Taking this into account, it can be
seen that for the creations of a well-accepted and loved soft object, one needs to
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 138–151, 2013.
© Springer International Publishing Switzerland 2013
provide either many varieties, as with the case of ‘Furby’[6], or re-design repeatedly
until universally accepted, as with the case of ‘Paro’[25]. This shows what huge leap
there would be between the initial development stage and final manufacture for sales.
Currently, majority of our society is not aware of the usefulness of soft interfaces.
Therefore, by providing a system which users can easily obtain and experience at low
cost, we can spread the benefits of soft interfaces. We believe that this is important to
discover new ways and playing methods to fulfill human needs. Based upon which,
this research, entitle Cuddly, aims to utilize a device, which a majority of us are
equipped with, to create interaction with soft objects which we already have in our
home.
This research utilizes a smart mobile phone, which has had very high sales in the
recent years, to create an application to enhance user’s interaction with their soft
objects (Fig. 1). This is done by embedding the mobile phone into a soft object.
Cuddly then utilizes the mobile phone’s camera and flashlight (LED) to detect the
surrounding brightness values captured by the camera. When user inserts the mobile
phone into a soft object and presses the object, the density of the material surrounding
the mobile phone increases. This increase in density decreases the brightness of the
material surrounding the camera. The camera captures its surroundings in the form of
average RGB values and converts these values into brightness values. With this data
of brightness change, Cuddly can return feedback by utilizing functions a mobile
phone has, such as sounds, lights, or even the animation on the display. In addition, as
a mobile phone is a tool for communication, we can connect two or more mobile
phones together to create a multi-user interaction.
Fig. 1. Mobile phone can be placed in many soft objects to convert them to soft interfaces
2 Related Work
New interactions in mobile phones tend to leverage the device’s built-in sensors or
attached external sensors. Miyaki et al. proposed “GraspZoom”, an input model using
pressure sensing, sensed by a force sensitive resistor attached to the backside of a
mobile phone [14]. Similarly, Goel et al. introduced GripSense, to detect hand
postures by sensing the pressure exerted by the user, through observing gyroscope
readings, and using it to facilitate interactions [7]. Iwasaki et al developed
AffectPhone, a system that detects the user’s emotional state using galvanic skin
response electrode attached to the side of the handset and covert the data into warmth
or coolness in a Peltier module attached the back panel of another device [10]. Our
proposal leverages the device in-built proximity sensor, camera and flash light for
interaction. Other sensors can be taken into account as well.
Many gadgets in the market can enhance the ability of a smart phone or protect the
phone. For example, Fisher Price introduced Apptivity Case for iPhone and iPod
devices, a sturdy case accommodated for babies, protecting the iPhone from dribbles,
drool and unwanted call making. These allow babies to play and experience advance
devices from a young age [5]. Cube-works released CocoloBear, an interactive teddy
bear whereby users place their iPhone on the bear’s stomach and the bear’s mouth
will move according to the frequency received [4]. This shows that innovations
leveraging soft interfaces for mobile devices are gradually expanding in the market.
This enhances the motivation to create soft user interface to increase hands-on
experiences for younger generations with computing devices.
We are surrounded by objects that we use on a daily basis. Our gestures towards these
objects are quite similar to our interaction with smart devices. Therefore, there is
much research in enhancing the functions of existing daily objects by combining them
with smart devices.
Cheng explored the approach of utilizing everyday objects as tabletop controllers,
taking into account common computer gestures such as rotate, drag, click etc. [3].
Masui et al. proposed MouseField, a device that allows user to control various
information appliances. This device consists of an ID recognizer and motion sensors,
which sense the movement of objects placed onto it, and interprets it as a command to
control the flow of information [13]. Sugiura et al. developed a skin-like user
interface consisting of an elastic fabric, photoreflectors and phototransistors, to
measure tangential force by pinching and dragging interactions on the interface [20].
His development can be used in daily wearable such as stockings. Sugiura et al. also
developed a ring-like device that can be attached to any plush toys to animate the
plush toys, converting it into a soft robot [22].
Seeing how important daily objects are to our lives, we are more attached to them
than other random objects. Therefore, our proposal is to use our plush toys or pillow,
an item that we have hold for a long time or have stronger memory, as the interface
for the interaction.
3 Principle
Soft objects are usually encapsulated with soft, fluffy materials such as cotton, wool,
feather etc. as their padding. These materials have many spaces in between the atoms
allowing a flow of light to spread out.
Previously, our research group has created FuwaFuwa [21], a sensor module that
uses photoreflectors sensors to detect the changes in brightness when integrated with
soft objects. These sensors consist of an infrared emitter and a transistor side-by-side
whereby the transistor detects the infrared light reflected back to it. From this concept,
we discover that this measurement is possible even when the two components are at
varied distance apart, until a maximum distance. As a note, the measurement method
is the opposite of FuwaFuwa: the brightness sensed increases with compression for
FuwaFuwa, while when the two components are apart for Cuddly, the brightness
sensed decreases with compression.
Fig. 2. Surrounding detected by camera is brighter at low density (left) than at high density
(right)
We then implemented our test with a mobile phone by making use of its camera
and flashlight. We integrate the mobile phone into soft objects and utilize the camera
as a sensor to detect the brightness of the surrounding area; thereby distinguishing the
change in density of the padding. In order to do so, the flashlight will act as the source
of light and the camera will detect its reflection. In a mobile phone, the flashlight and
camera are at a distance apart. When there is no pressure applied to the soft material,
the surrounding is bright and numerous external light sources can enter the camera
(Fig. 2 left). However, when the soft object is compressed, the density of the material
surrounding the light source increases, reducing the light reflection – thus, reducing
the surrounding brightness (Fig. 2 right).
The camera will capture the video of the surroundings and convert the pixels into
average RGB values. These values will then be converted into brightness value using
the equation below:-
(1)
There are quite a number of equations for brightness that can be found. We chose
to use the equation which best corresponds to the human perception [15]. The data of
this brightness value is used to create different feedbacks such as sounds, lights,
animation etc.
4 Experimentation
when density increases by pushing, and increases when density decreases by pulling.
However, there is a certain level of hysteresis observed. Most soft materials have
many holes in between their atoms. After undergoing push-pull activity, some does
not return to its original state. Instead, they still remain in their compressed form.
Therefore, the camera detects less brightness when it undergoes pulling activity than
pushing activity. This is what causes the hysteresis. However, all the graphs show a
significant decrease in brightness value when compressed. This proves that our
proposal is capable to detect the change in brightness values.
Fig. 5. (continued)
4.3 Implementation
For this proposal, we tested the application on Galaxy Nexus (API 4.1.1), HTC J (API
4.0) and Galaxy S SC-02b (API 2.3.6); making use of Android SDK and Eclipse.
5 Application
Kids like to play as if their plush toy is alive, by giving them voices or moving their
limbs [18]. We have designed a few applications by giving voices to different soft
toys when the phone is inserted into the toy. For example, dog barking sound for a toy
dog, a character voice for a cute character and playback of different recorded voices
for a puppet. Besides that, using this application, we can animate a soft object by
creating visual faces on the screen itself (Fig. 6). For this application, when the soft
object is not pressed, the range of brightness value is about 100. When the object is
pressed, it reduces to slightly below 50. Therefore, the brightness value chosen is 50.
When the soft object is pressed and the brightness value reduces to 50 and below, the
animated character will close its eyes.
Fig. 6. Character
5.2 Game
We have also designed a gaming application that utilizes the screen while integrating
part of the phone with a soft object. For example, depending on the brightness value,
the character on the screen will jump at a different height (Fig. 7). For this game, as
the brightness value decreases, the height increases and vice versa.
Fig. 7. Game
6 Discussion
From both the experiment and test for application creation, a few points were brought
up regarding its influence on the accuracy of the application.
• Hysteresis:
Hysteresis differs by materials. However, many daily soft objects are filled with
cotton, which has low hysteresis. From our test with different objects, many of the
data shows that the readings do return back close to its original values before and
after compression. Therefore, we can use optimum values for each interaction.
• Ambient light from the surroundings:
We conducted the experiment in both a dark and bright environment. From
observations, the results were similar as the light’s reflection was too bright, that
the ambient light does not highly affect its performance.
• Movement of the phone in the object:
The phone is constraint by material in the object and thus, small movement may
cause some glitches, but will not highly affect the application. However, Cuddly
can detect harsh movements as the pressure does change and can generate
versatile interactions. For example, when one pulls the soft object at both ends,
there would be pressure change in the center of the object. Through this, other
interactions such as pulling a part of the soft object can also trigger interactions.
• Color of material:
In the experiment, we have tested with three different colored materials as well as
each application presented uses a different colored soft objects. As the brightness
value taken is an average, the colors of the material do not highly effect the
application.
In this section, we will explain some of the limitations and possible future works to
reduce these limitations. One of the limitations is the size of the mobile phone.
Currently most mobile phones are rather long in size. Therefore, Cuddly can only be
used in soft objects that are bigger than the size of the mobile phone. In addition, the
hardness of the phone may be a challenge if the soft object does not have much
stuffing in it. In this case, users may detect the phone when they press the soft object.
The compensation level and hysteresis are also some of the limitations as different
materials have different compensation level and hysteresis level. As a future work, it
may be possible to calibrate both accordingly using different algorithms.
Currently, Cuddly is only programmed to take an average value of the all the pixels
captured by the camera – taking only readings in one dimension. However, by sing
better image processing techniques, it may be possible to detect the areas of pressure
– allowing to obtain readings in two dimension. Besides that, for an extra large
interface, one may insert multiple phones and communicate the phone’s data with
each other to detect the brightness value at different spots. Besides that, another
suggestion is to insert a few mobile devices into a big soft object (e.g. a bean bag) to
detect the position where the interaction is taken place. This allows the object to act
like a controller for different functions such as for gaming purposes. Currently, only
the proximity sensor is being utilized – other sensors will be utilize as well in the
future to create a more sensitive detection.
8 Conclusion
Cuddly is a phone application that allows users to integrate their mobile phone with
soft objects found in the surrounding environment. This application works like a light
sensor; making use of the mobile phone’s camera and flash light. When the mobile
phone is inserted into a soft object, the light from the flashlight will be reflected into
the camera. This camera will capture RGB values of the surroundings and convert it
into brightness value. When a soft object is compressed, the material’s density
surrounding the mobile phone will increase and reduces the light reflection. This
causes the brightness value detected to decrease; thus, causing a range of value
changes. Using these values, Cuddly can create feedback interactions with the soft
objects. For example, sound feedback, light feedback, making a call, etc. Our
experiment shows that Cuddly is capable of detecting the change in brightness.
In conclusion, this allows users to make use of their most common device, a mobile
phone with soft objects from their surrounding environment. This may enhance their
user experience with their objects.
References
1. Ackerman, J.M., Nocera, C.C., Bargh, J.A.: Incidental haptic sensations influence social
judgments and decisions. Science 328(5986), 1712–1715 (2010)
2. Bianchi, M., Serio, A., Bicchi, A., Scilingo, E.P.: A new fabric-based softness display. In:
Proc. Haptics Symposium 2010, pp. 105–112. IEEE (2010)
3. Cheng, K.Y., Liang, R.H., Chen, B.Y., Liang, R.H., Kuoy, S.Y.: iCon: Utilizing Everyday
Objects as Additional, Auxiliary and Instant Tabletop Controllers. In: Proc. CHI 2010,
pp. 1155–1164. ACM (2010)
4. Cube Works official website, http://www.cube-works.co.jp/works/
index_sub.html?/works/cocolobear/index.html
5. Fisher Price official website, http://www.fisher-price.com/en_US/
products/64176
6. Furby official website, http://www.furby.com
7. Goel, M., Wobbrock, J.O., Patel, S.N.: GripSense: Using Built-In Sensors to Detect Hand
Posture and Pressure on Commodity Mobile Phones. In: Proc. UIST 2012, pp. 545–554.
ACM (2012)
8. Harrison, C., Hudson, S.E.: Providing Dynamically Changeable Physical Buttons on a
Visual Display. In: Proc. CHI 2009, pp. 299–308. ACM (2009)
9. Hiramatsu, R.: Puyo-con. In: ACM SIGGRAPH ASIA 2009 Art Gallery & Emerging
Technologies, p. 81. ACM (2009)
10. Iwasaki, K., Miyaki, T., Rekimoto, J.: AffectPhone: A Handset Device to Present User’s
Emotional State with Warmth/Coolness. In: BIOSTEC 2010, Workshop on B-Interface
(2010)
11. Kadowaki, A., Yoshikai, T., Hayashi, M., Inaba, M.: Development of Soft Sensor Exterior
Embedded with Multiaxis Deformable Tactile Sensor System. In: Proc. Ro-Man 2009, pp.
1093–1098. IEEE (2009)
12. Marti, S., Schmandt, C.: Physical Embodiments for Mobile Communication Agents. In:
Proc. UIST 2005, pp. 231–240. ACM (2005)
13. Masui, T., Tsukada, K., Siio, I.: MouseField: A Simple and Versatile Input Device for
Ubiquitous Computing. In: Mynatt, E.D., Siio, I. (eds.) UbiComp 2004. LNCS, vol. 3205,
pp. 319–328. Springer, Heidelberg (2004)
14. Miyaki, T., Rekimoto, J.: GraspZoom: zooming and scrolling control model for single-
handed mobile interaction. In: Proc. MobileHCI, Article No. 11 (2009)
15. Relating HSV to RGB, http://www.dig.cs.gc.cuny.edu/manuals/
Gimp2/Grokking-the-GIMP-v1.0/node52.html
16. Rossiter, J., Mukai, T.: A Novel Tactile Sensor Using a Matrix of LEDs Operating in Both
Photoemitter and Photodetector Modes. In: Proc. Sensors. IEEE (2005)
17. Sato, T., Mamiya, H., Koike, H., Fukuchi, K.: PhotoelasticTouch: Transparent Rubbery
Tangible Interface using an LCD and Photoelasticity. In: Proc. UIST 2009, pp. 43–50
(2009)
18. Strommen, E., Alexander, K.: Emotional Interfaces for Interactive Aardvarks: Designing
affect into social interfaces for children. In: Proceedings of ACM CHI 1999, pp. 528–535
(1999)
19. Juul Sørensen, A.S.: Pillow Fight 2.0: A Creative Use of Technology for Physical
Interaction. In: Nijholt, A., Romão, T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624,
pp. 506–512. Springer, Heidelberg (2012)
20. Sugiura, Y., Inami, M., Igarashi, T.: A Thin Stretchable Interface for Tangential Force
Measurement. In: Proc. UIST 2012, pp. 529–536. ACM (2012)
21. Sugiura, Y., Kakehi, G., Withana, A., Lee, C., Sakamoto, D., Sugimoto, M., Inami, M.,
Igarashi, T.: Detecting Shape Deformation of Soft Objects Using Directional
Photoreflectivity Measurement. In: Proc. UIST 2011, pp. 509–516. ACM (2011)
22. Sugiura, Y., Lee, C., Ogata, M., Withana, A., Makino, Y., Sakamoto, D., Inami, M.,
Igarashi, T.: PINOKY: A Ring That Animates Your Plush Toys. In: Proc. CHI 2012,
pp. 725–734. ACM (2012)
23. Ueki, A., Kamata, M., Inakage, M.: Tabby: Designing of Coexisting Entertainment
Content in Everyday Life by Expanding the Design of Furniture. In: Proc. ACE 2007,
pp. 72–78. ACM (2007)
24. Väänänen-Vainio-Mattila, K., Haustola, T., Häkkilä, J., Karukka, M., Kytökorpi, K.:
Exploring Non-verbal Communication of Presence between Young Childrenand Their
Parents through the Embodied Teddy Bear. In: Paternò, F., de Ruyter, B., Markopoulos, P.,
Santoro, C., van Loenen, E., Luyten, K. (eds.) AmI 2012. LNCS, vol. 7683, pp. 81–96.
Springer, Heidelberg (2012)
25. Wada, K., Shibata, T., Saito, T., Sakamoto, K., Tanie, K.: Psychological and Social Effects of
One YearRobotAssisted Activity on Elderly Peopleat a Health Service Facility for the Aged.
In: Proc. International Conference on Robotics and Automation 2005, pp. 2785–2790. IEEE
(2005)
26. Yagi, I., Kobayashi, S., Kashiwagi, R., Uriu, D., Okude, N.: Media cushion: soft interface
to control living environment using human natural behavior. In: ACM SIGGRAPH 2011
Posters, Article 46, 1 page. ACM (2011)
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 152–167, 2013.
c Springer International Publishing Switzerland 2013
2 Background
User manuals are a part of the technical documentation of a product. The Secure-
Doc guideline [18] states, “Products are not complete without documentation.”
The guideline interprets, an important standard in technical documentation, the
IEC 62079:2001 on “Preparation of instructions. Structuring, content and pre-
sentation” [10] and helps designing the technical documentation of a product. It
outlines the requirements that are induced by European law, such as to enable
customers to use all features of a product and to protect customers of potential
hazards. The guideline relates to factors that are often ignored when a product
is accompanied by a user manual of poor quality. For instance, “High quality
documentation helps reduce customer support costs,” because with the right in-
formation at hand it enables customers to solve many problems without further
assistance. Furthermore, “High quality documentation enhances customer sat-
isfaction,” because a poorly designed user manual can prevent customers from
exploring the full potential of a product. Another factor is that customers re-
late the quality of the product to the quality of its documentation. Therefore,
the need for high quality user manuals is given and the exploration of emerging
technologies, such as augmented reality, to improve their usability is worthwhile.
Although other formats are available, the standard format for user manuals is
still the printed handbook. Typically it offers an index that lists all functionalities
or use cases of a product. Each one of them is then described in a step-by-step
manner. Technical illustrations improve understandability by “expressive images
that effectively convey certain information via the visual channel to the human
observer” [20]. Mainly for cost reasons comprehensive user manuals are often
included as digital PDF documents on CD or for download. Although the content
is normally taken one-to-one from the printed version, it improves the aspects
of information retrieval concerning finding the user manual itself (if available for
download) and finding the relevant information in it by searching the document.
were also perceived as a tool that can manage contextual information. They have
pervaded our everyday life, and the number of mobile applications is growing.
As computational things become everyday things, what we design for can
not be restricted to how to enable people to become more productive.
Thus, there is a need for complementary design philosophies. [14]
Redström herby, refers to how HCI researchers used to focus on performance
and supporting people in accomplishing tasks. In the last decade many re-
searchers started to focus on creating new experiences enabled by mobile de-
vices. De Sa and Churchill [16] argue that the new affordances offered by mobile
augmented reality have potential to enhance users‘ experiences. More specifically
they discuss how this can be achieved through the provision of digital informa-
tion which is relevant for the user context. With mobile devices information that
is digitally stored can be processed anywhere anytime. Consequently, more and
more information is presented in digital form as and through mobile media.
One experiences the world through the technology, and the technology
inevitably becomes part of the way one relates to the world. While many
technologies appear in between the user and the world, not all are em-
bodied. For a technology to hold an embodiment relation it must be
technically transparent, it must allow its user to “see through” it. [3]
Based on philosopher Don Ihde‘s non-neutrality of technology-mediated ex-
perience, Fallman hereby points out that for technology to be embodied it needs
to recede into the background of experience. One could argue that this is true
for mobile devices and mobile augmented reality applications. Similar to how
one does not feel the weight of glasses after carrying them for a while, a mobile
device, although heavier, recedes into the background of experience during in-
teraction with the real world. However, this real world becomes augmented with
digital information.
Different lines of research within HCI have recognized the increasing blur be-
tween the digital and the physical in today’s designs. Efforts to improve the
understanding of those kinds of designs exist. For example, Vallgårda et al. dis-
cuss how knowledge of materials has been essential to design practice and how
computers share important characteristics with other materials that are used in,
for example, industrial design or architecture. They introduced computational
composites as a new type of composite material [19]. They argued that compu-
tational properties of a computer are difficult to exploit, but through combining
it with other material can come to use.
In mobile augmented reality applications, mobile devices are used to augment
real-world objects and materials. One could regard mobile augmented reality as
a computational property of a mobile device. Through augmenting a specific real
world object (e.g. an oven); i.e. combining a mobile device with the real world
object, augmented reality as a computational property comes to use.
While printed manuals are still the standard media for manuals, there is a
need for digital manuals and interaction that is more timely and uses features
of mobile devices that people have already become familiar with.
However the mobile device itself is a medium that supports through its prop-
erties (e.g. being mobile, lightweight, having a camera, having a screen etc.)
interaction that is fundamentally different from paper. Finding out how to best
interact with new media is in general a difficult task and requires exploration.
This is particularly true for digital counter parts of physical designs. For exam-
ple, Grasset et al. [6] try to answer the question if a mixed-reality book is still
a book and explore the design and development process of visually augmented
books. They argue that exploring design issues becomes more important as tech-
nology gets more mature; and that the development of prototypes requires time
and very specialized experts.
The intention of this section was to reflect on what it means to exploit mobile
devices for interaction in a broader sense and to remind the reader of current
design notions and problems. A mobile device is a new medium and mobile
augmented reality can provide rich sensory effects. The key challenge for AR-
based user manuals lies in building on the aforementioned technological advances
in AR and to connect the device and the required information in a manner that
is intuitive to the user and if possible even fun to use. It is unclear how much
of the knowledge on user manual design can be applied to mobile phones and
which new challenges arise when user manuals are adapted to the mobile device.
In order to provide some insights we now move to a concrete implementation
of a digital manual that is inspired by properties of a mobile device.
4 GuideMe Prototype
The GuideMe system identifies home appliances by using the camera and re-
trieves an interactive manual from a server. The interactive manual is specified
in the User Manual Markup Language (UMML). The following sections out-
line the method used to recognize the device, the UMML specification and the
resulting user interface.
Appliances differ in size, form and functionality. In result each user manual has
to be customized. GuideMe aims at providing the best possible user manual
for a specific appliance and a specific mobile device. Therefore, we decided to
put the design of the user interface into the hands of the experts by providing
them a simple format to author user manuals. In consequence designers can
adapt existing manuals to a specific device and experiment with different layouts
and structures. We developed a new format to define manuals and refer to it
as User Manual Markup Language (UMML). UMML defines the layout of the
recognized user interface. In comparison to existing formats like APRIL [12],
UMML benefits from a clear focus on user manuals. APRIL aims at structuring
narrative content and defines cast, interactions and behavior. This flexibility
leads to complex definitions that require advanced knowledge.
The resulting schema builds on a smaller set of XML statements to define
the user manual. The user interface was split into the basic elements like text,
arrows or any other graphical element. Custom elements can be defined and used
as well. The grouping of these elements and the final structure of UMML was
inspired by the design of current printed user manuals.
An UMML file consists of two parts: the menu description and the list of func-
tionalities. Each functionality definition has a unique ID. The menu description
organizes the available functionalities into a hierarchical structure of menus that
can contain functionalities or submenus. Functionalities can be linked into a
menu several times at several levels, if desired. Functionalities contain a title tag
and consist of a number of steps. The steps are identified and linked by a unique
ID. Each step definition contains a list of virtual objects and their position in
relation to the frame marker, e.g. an arrow pointing to a button. Furthermore,
images, videos and audio files can be linked into the manual.
(a) Oven with Vuforia (b) GuideMe highlights (c) GuideMe screenshot with
frame marker and enclosed the corresponding operat- highlighted operating elements
QR code ing elements
objects defined in the UMML file can be classified by their positioning on the
screen. The user has to hold the tablet in front of the user interface to recognize
the marker. GuideMe uses the size and position of the marker to adapt the
coordinate plane. Texture components, symbols, arrows, frames and user defined
graphics are placed relative to the recognized marker position. In general these
components are placed in the same plane as the marker but a 3-dimensional
placement is possible as well.
In some cases superimposed objects might not be sufficient to indicate the
correct action. For example, if one button has to be held while pressing other
buttons, an image of two arrows pointing to both buttons will not be enough.
The instructions that are printed in a manual have to be conveyed to the user as
well. Using UMML and GuideMe, they can be printed on the screen or played as
an audio or video clip. All three methods are possible with the existing UMML
elements. Videos, audio clips, Android webviews, buttons and option menus
are placed at fixed positions on the screen. Android webviews can load HTML
context into the Android application from a file or the Internet. Video and
audio clips start automatically when the corresponding step is rendered. Textual
descriptions are placed at the top of the screen. All components can be freely
combined, although a full screen video overlaps other components.
Navigation between user manual steps is possible by pressing buttons or by
selecting from an options menu. Options menus and buttons can be defined in
the UMML file. Users can always return to the last step using the Android back
button or return to the menu by using the navigation bar at the top.
5 Evaluation
We conducted two subsequent studies to obtain first insights on the user experi-
ence and task performance using GuideMe. The first study compared GuideMe
based manuals against excerpts from the original printed user manual and the
second against video tutorials.
We selected two state-of-the-art ovens, the Stoves SEB900MFSe and the
Bosch HBL78B7.1. Both provide sufficient complexity to require a manual for
first usage of advanced functionality and provide several user interface elements,
such as knobs, buttons and displays that could be augmented. They are simi-
lar enough to allow a comparison but have different user interfaces to mitigate
learning effects during the study.
The GuideMe manual was structured according to the numbered steps in the
manual. For each step, the corresponding explanation is shown at the top of
the screen and the step is visualized on the screen as shown in figure 2. Arrows
images and further markers are superimposed on the camera image to indicate
relevant buttons and displays. Navigation between steps was realized by buttons.
10.1 tablet with Android 3.1. We measured task completion times and used the
NASA TLX [8] to measure the subjective task load of participants. Participants
were asked to think aloud while completing the given tasks. Observers noted
down their statements, the perceived body language and the operation of the
appliance. A concluding questionnaire captured the subjective preferences of the
user. The subsequent second study replicates the design of the first study, but
compares (a) a video-based tutorial on a tablet and (b) GuideMe on a tablet.
The participants in the second study did not overlap with participants from the
first study.
During the user study, each participant performed overall 4 measured tasks,
two on each oven: (a) 1 simple and 1 complex task using GuideMe for one oven
and (b) 1 simple and 1 complex task using the printed original manual for the
other oven. Switching between devices and reversing the order of technologies
and devices aimed at mitigating learning effects of the oven’s user interface. Par-
ticipants could use as much time as necessary and should mention when they
deemed the task to be completed or if they want to give up on it. There was no
additional help offered except the manual to be tested. After each task partic-
ipants completed the NASA TLX questionnaire. The concluding questionnaire
collected overall feedback on both used technologies and the perceived complex-
ity of the used ovens.
This data was evaluated in a within subjects design to mitigate the individual
differences in performance and NASA TLX usage. The significance of quantita-
tive data (NASA TLX and task completion times) was analyzed using a paired
T-test and Cohen’s d.
2 groups of 10 young technology-savvy participants (22-31, 4 female, 16 male)
took part in the studies. All 20 participants used GuideMe and all of them
operated both ovens. The majority of the users already knew similar ovens before
the test (55 % Stoves, 70 % Bosch). 53 % agreed or strongly agreed when asked to
state their familiarity with Android device on a 5 point Likert scale. 74% agreed
or strongly agreed regarding their familiarity using tablet PCs. Only 10% had
any experience with augmented reality applications. Albeit this user group does
not represent the overall population, it represents the potential early adopters
of mobile media based user manuals.
Each of the 20 participants performed 4 tasks, resulting in a total number of
80 measurements. 5 of these measurements were classified as outliers, because
participants required exceptionally long to complete a given task. Participants
were stuck because they entered unforeseen menus and got lost in the mismatch
between instructions and available options. As we use a pairwise T-test the cor-
responding measurement using the second device was excluded from the analysis
as well.
6 Results
All participants in both studies reported that the two provided user manuals
were useful to complete the given tasks. Nevertheless, when asked if the oven
was difficult to use the majority of users disagreed or strongly disagreed (65%
Stoves, 85% Bosch). According to this question, the Stoves oven was experienced
as more difficult.
#
"
!
Fig. 3. GuideMe (black, left columns) compared to the original printed manual (blue,
right columns)
#
"
!
Fig. 4. GuideMe (black, left columns) compared to a video tutorial on a tablet (red,
right columns)
7 Discussion
The two studies and the design of the GuideMe prototype provide insights into
the usage of different user manual formats, the developed prototype and the
design challenges when creating user manuals for and with mobile devices that
combine physical user interface elements and digital information.
Three different formats of user manuals were used in the two studies. The average
task completion time for the printed user manuals is the fastest and the number
of errors was the lowest among all technologies. Nevertheless, our technology-
savvy participants preferred the two technology based solutions. On the one
hand, this might be due to the nature of our user group. It appears that video
as well as GuideMe provide a special positive experience to this group. While
the paper was treated like a tool, the participants did not put the tablet away
during the study. The used platform, the interactive nature of the presentation
or the ongoing animation on the screen, contribute to this effect. On the other
hand, participants felt, although wrongly, more efficient when using video or
GuideMe as reflected in the performance scale of the NASA TLX. Video and
GuideMe allow users to simply imitate actions or follow detailed advice. The
paper manual had to be understood and mapped to the ovens’ user interface.
The participants have to decide more in the mapping process and each small
decision adds to their uncertainty.
Although GuideMe learned from the printed manual, some design aspects
became only clear by comparing both media in the study. For instance, the
step structure of GuideMe was copied from the original manual but not the
arrangement of steps on a single page. Each augmented image requires the whole
screen of the mobile device. Hence, content that is printed on one page in the
manual is split into a sequence of screens. This split is a critical point as users can
no longer scan through all steps on a page to recognize the current state of the
device. Future implementations should aim at supporting this type of scanning,
e.g. by employing similar methods as in mobile image browsing. Furthermore,
GuideMe could combine benefits of printed paper and digital augmented objects
by combining both media as explored by Grasset et al. [6].
A user manual does not only contain a series of steps, but aims at teaching
the user interface of the device. Hence, task descriptions contain additional in-
formation such as warnings on possible errors, the impact on other functionality
and a method to revert or abort the current task. Video tutorials often select
a simple subset of steps and neglect this additional information for the sake of
a shorter and clearer presentation. However, this background information is es-
sential to actually create an understanding of a new device. Albeit we included
all this information in the video, the video was praised for the clarity of the
explained steps. However, participants criticized the perceived “time pressure”.
Participants did not use the available controls to stop the video. Splitting the
instructions into a linked set of small videos may eliminate this behavior. Such
manuals can be realized using the current functionality of GuideMe.
Both studies were influenced by the weight and size of the used tablet. The
weight increased the physical demand. The size of the tablet was perceived as a
barrier between user and the oven. Further evaluations using smaller devices can
research the resulting trade-off between screen size and weight for this application
domain.
is located on the top of the tablet thus creating a shift of the image that can be
confusing. The camera is expected to be in the middle of the tablet. Completely
different devices like wearable displays5 or personal projection devices [11] would
better align movements of the device and the created image.
The ovens’ user interface in our study was too wide for the used tablet camera.
Ovens were identified by a single marker in the middle of the user interface that
had to be within the cameras’ field of sight all the time. On the Stoves oven for
example, the knob to change the temperature is more than 20 cm away from the
marker. Participants often held the tablet too close to the user interface to see
marker and knob in the magic lens. Multiple markers or a new kind of distributed
markers could provide a solution. Another option would be to decorate the user
interface in a manner to enable marker-less object recognition. The participants
themselves came up with a third simple solution. They missed a button to freeze
the current state of the screen. This approach would combine augmented reality
and image based user manuals.
UMML enables GuideMe to be flexible and abstracts from the required pro-
gramming. Technical documentation experts can design augmented reality user
manuals by writing UMML. However, we did not evaluate the modeling capa-
bilities of UMML in this paper.
Future user manuals should emphasize the playful exploration of the function-
ality provided by mobile devices. The two conducted studies have provided first
evidence that video and AR-based manuals can support this change. Users felt
more secure using these manuals. Moreover, their body language indicated a
positive attitude towards this new medium. This preference was confirmed in
the concluding questionnaire.
However, the initial positive attitude may change, if the design of such user
manuals does not account for the specific properties of the mobile devices, e.g.
the position of the camera. Designers and technical documentation experts are
needed to facilitate this change. Until now technology has dominated the de-
velopment. UMML is a first approach to simplify development and include the
experts. The notion of computational composites as suggested by Vallgårda [19]
can help to bridge the gap between designers and technology driven developers.
In AR-based manuals the appliance becomes part of the user interface. For
instance, the color of superimposed graphics has to to blend into the camera
picture of the device. The design of the appliance is linked to the design of the
manual. Properties of both components influence each other and create the final
design. Therefore, GuideMe aims at a maximizing the flexibility of the design.
A further integration on the technical level was proposed by Liu et al. [13].
They compared AR manual implementations and concluded that “the conven-
tional AR approach performs almost as well as pictures” but feedback from the
device would “significantly improve task performance and user experience”. A
5
Project Glass http://plus.google.com/+projectglass/
direct feedback from the home appliance to GuideMe would prevent possible
discrepancies between the actual state of a device and the state that GuideMe
is assuming and thus significantly lowering error rates and frustration. Although
standardized methods for device feedback would be beneficial to applications
like GuideMe such a standardization of feedback mechanisms across appliances
and manufacturers is not foreseeable.
8 Conclusion
We described GuideMe a mobile design for AR-based user manuals, discussed
the underlying combination of mobile media and physical user interfaces and
presented the results of two small scale studies. Both studies compared GuideMe
manuals to current alternatives, the printed manual and mobile video tutorials,
with young users and real home appliances. The printed manual resulted in the
fastest task completion times and the lowest error rates. However, these classic
usability metrics were not decisive arguments for the participants; they preferred
video tutorials and GuideMe on the tablet. Both are interactive formats that
allow users to imitate actions. In result participants enjoyed using these manuals
on the tablet.
Augmented Reality and video provide starting points to new user manuals
that are fun to use. Both manual types can be designed, combined and tested
using UMML and GuideMe. A mobile application for consumers to adapt and
create simple UMML manuals by drag and drop is currently in development.
Applications like GuideMe and a simple editor would open the explored design
space for a broader audience. The current progress in mobile media has the
potential to transform the user manual but requires more research on the manual
design.
References
1. Bier, E., Stone, M., Pier, K., Buxton, W., DeRose, T.: Toolglass and magic lenses:
the see-through interface. In: SIGGRAPH 1993, pp. 73–80. ACM (1993)
2. Bødker, S.: When second wave hci meets third wave challenges. In: Proceedings of
the 4th Nordic Conference on Human-computer Interaction: Changing Roles, pp.
1–8. ACM (2006)
3. Fallman, D.: The new good: exploring the potential of philosophy of technology
to contribute to human-computer interaction. In: Proceedings of the 2011 Annual
Conference on Human Factors in Computing Systems, pp. 1051–1060. ACM (2011)
4. Feiner, S., Macintyre, B., Seligmann, D.: Knowledge-based augmented reality.
Communications of the ACM 36(7), 53–62 (1993)
5. Geiger, C., Kleinnjohann, B., Reimann, C., Stichling, D.: Mobile AR4ALL. In:
ISAR 2001, pp. 181–182. IEEE Comput. Soc. (2001)
6. Grasset, R., Dunser, A., Billinghurst, M.: The design of a mixed-reality book: Is
it still a real book? In: 7th IEEE/ACM International Symposium on Mixed and
Augmented Reality, ISMAR 2008, pp. 99–102. IEEE (2008)
7. Hakkarainen, M., Woodward, C., Billinghurst, M.: Augmented assembly using a
mobile phone. In: Proceedings of the 7th IEEE/ACM International Symposium
on Mixed and Augmented Reality, ISMAR 2008, pp. 167–168. IEEE Computer
Society, Washington, DC (2008)
8. Hart, S., Staveland, L.: Development of nasa-tlx (task load index): Results of em-
pirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human
Mental Workload, pp. 139–183. Elsevier, Amsterdam (1988)
9. Henderson, S., Feiner, S.: Exploring the benefits of augmented reality documen-
tation for maintenance and repair. IEEE Transactions on Visualization and Com-
puter Graphics 17(10), 1355–1368 (2011)
10. IEC International Electrotechnical Commission: IEC 62079:2001 - Preparation of
instructions. Structuring, content and presentation (July 2001)
11. Kawsar, F., Rukzio, E., Kortuem, G.: An explorative comparison of magic lens and
personal projection for interacting with smart objects. In: Mobile HCI 2010, pp.
157–160. ACM (2010)
12. Ledermann, F., Schmalstieg, D.: April: a high-level framework for creating aug-
mented reality presentations. In: Proceedings of the Virtual Reality, VR 2005, pp.
187–194. IEEE (March 2005)
13. Liu, C., Huot, S., Diehl, J., Mackay, W.E., Beaudouin-lafon, M.: Evaluating the
Benefits of Real-time Feedback in Mobile Augmented Reality with Hand-held De-
vices. In: CHI 2012, vol. 2012, pp. 2973–2976 (2012)
14. Redström, J.: Designing Everyday Computational Things. Ph.D. thesis, Göteborg
(2001)
15. Rohs, M.: Marker-based embodied interaction for handheld augmented reality
games. Journal of Virtual Reality and Broadcasting 4(5) (2007)
16. de Sa, M., Churchill, E., Isbister, K.: Mobile augmented reality: design issues and
opportunities. In: Proceedings of the 13th International Conference on Human
Computer Interaction with Mobile Devices and Services, pp. 749–752. ACM (2011)
17. Takacs, G., Chandrasekhar, V., Gelfand, N., Xiong, Y., Chen, W.C., Bismpigiannis,
T., Grzeszczuk, R., Pulli, K., Girod, B.: Outdoors augmented reality on mobile
phone using loxel-based visual feature organization. In: Proceedings of the 1st
ACM International Conference on Multimedia Information Retrieval, MIR 2008,
pp. 427–434. ACM, New York (2008)
18. TCeurope (Organisation) and European Commission: SecureDoc Guideline for Us-
able and Safe Operating Manuals for Consumer Goods: Fourth Brussels Colloquium
for User-friendly Product Information. TCeurope (March 29, 2004)
19. Vallgårda, A., Redström, J.: Computational composites. In: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, CHI 2007, pp.
513–522. ACM, New York (2007)
20. Viola, I., Meister, E.G.: Smart visibility in visualization. In: Computational Aes-
thetics in Graphics, Visualization, pp. 209–216 (2005)
Abstract. Most people will have experienced squishing clay and making
it flat. The action of changing the shapes of real objects induces pleasant
feelings or excitement. In this research, we propose a system, named
Petanko Roller, which enables users to experience the sensation of rolling
out any object in the real world with a rolling pin virtually. This system,
by detecting the shapes of physical objects with a range camera, can
represent haptic sensations of unevenness or friction of the objects, using
modules for clunk mechanisms and brakes of a rolling-pin-based interface.
Furthermore, by projecting images of the objects being squished on a
tabletop display, it can also give optical feedback to users. In this paper,
we discuss the system design, implementation, and behavior of users in
exhibitions.
1 Introduction
When playing with clay, mixing cookie dough, or crushing cans, most people will
have experienced squishing something and making it flat. Changing the shapes
of real objects induces pleasant feelings or excitement. In cartoons or comics, we
are often shown characters or physical objects being flattened by large objects.
In contrast, in the real world, there are many objects that cannot be flattened
because of their hardness, costliness, or uniqueness.
Under such backgrounds, we propose an entertainment system that can pro-
vide a virtual experience of flattening any physical objects including human’s
body by giving haptic and visual feedbacks. In this research, to realize such sys-
tem, we mainly aimed to develop a novel interface that can represent textures
and shapes of digital objects.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 168–181, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
In resent years, in the field of HCI or VR, many systems have been proposed to
provide rich and realistic experience to users by giving multi modal feedbacks
other than visual or audio. Especially, interfaces which provide haptic sensa-
tion have been popular these days[10][12][5]. Such interfaces can dynamically
represent shapes or textures of digital objects.
Among such researches, there are approaches that encourage intuitive inter-
action by using common tools or objects as metaphors. As for a major example
of such interface in HCI, the metaDESK[11] utilizes physical objects as a com-
puter interface under the concept of Tangible Bits[4][3]. There are also haptic
interfaces proposed under such approaches. Virtual Chanbara[6] is a VR system
that uses the inertial force of a spinning weight to enable users to feel Chanbara
feedback. It can represent the reaction force when a player’s sword hits virtual
enemies. Funbrella[13] can record and replay rain with an umbrella-like interface
that is composed of a vibration-giving mechanism based on a speaker and a mi-
crophone. Ikegami et al. proposed handy haptic feedback devices imitating daily
tools such as a tweezer or a syringe named Tool Device[2]. Using the metaphor
of everyday tools, they enable users to manipulate informations intuitively. The
advantages of taking such approaches are that users can figure out how to use
the device easily and also that it is easy to install various actuators or sensors
in devices because they can have a certain size according to the tools used as
metaphors. As basing on such approaches and focusing on a “rolling pin”, we
aim to develop an interface which represents the shape and resistance of digital
objects with haptic and visual feedbacks.
One of the major feature of Petanko Roller is that it can represent the three
dimensional shape of objects. Pin-displays [1][8] are similar purposed system
for representing shapes. They physically represents the shape information by
actuating arrayed pins up and down. Although pins are arrayed on tabletop
device in such system, in our research it is the interface that includes linear
actuated modules. It is able to represent the haptic sensation of rolling over
objects with rolling pin by regulating the modules up and down according to the
motion of the device. In the pin-display system, the size of the object which can
be represented depends on the size of the display, but in our system, the size of
the interface doesn’t limit it, because of its unique system design.
Also there are related work on expressing hardness or texture of virtual ob-
jects. Claytric Surface[9] is a tangible display which can change the hardness of
its surface by regulating internal particle density. The Haptic Touch Toolkit[7]
can express the height and friction of virtual objects at single point, with a de-
vice which has an actuating rod and a brake. In contrast with these system, we
propose a handheld sized interface which can represent 3D physical shape with
clunk and brake modules placed in parallel.
3 Petanko Roller
3.1 Concept
As focusing on the pleasant feelings or excitement induced when changing the
shapes of physical objects, we propose an entertainment system that user can
virtually squish any object in real world as if they were clay. We developed a
VR haptic system, named Petanko Roller, which user can experience flattening
any physical object in the real world.
We describe the process of the user’s experience in the system of Petanko
Roller below (Fig. 2).
First, a user chooses an object in the real world that he/she wants to make
flat. The system then captures the 3D shape of the object.
Secondly, by moving a rolling-pin-based device on the visual image on a table-
top display, the user can feel the unevenness and friction of the object. Putting
pressure on the virtual object through the device stretches the object according
to the force of the pressure and the motion of the device. With time, the object
will be flat.
Thirdly, at the end, the user can keep a printout of the flat object that he/she
rolled out as a souvenir.
4 System Design
4.1 Detection of the Object
We implemented Kinect which is an infrared depth sensor for measuring the
physical object’s shape. Kinect is fixed approximately 100 cm above the tabletop
display, in a downward direction (Fig. 4). On the basis of the range data acquired
from Kinect, shape data are generated and saved for PC processing. In the shape
data, Each pixel’s height datum is allocated from 0 to 255 by setting 0 as the
top of the tabletop display and 255 as the highest part of the object. Kinect is
preliminarily calibrated.
In addition, color image data clipped according to the height data are also
saved at the same time(Fig. 5).
Fig. 4. Kinect detecting a physical Fig. 5. Generated image data and shape data
object using Kinect
on the cross, a brake plate, and two tires (Fig. 9). A gear cut is attached to the
servo motor, so that it pushes the plate when the servo motor actuates. With
this mechanism, users feel the friction between the tires and the plate as the
friction of the virtual object. Also, as the tension of brake is adjustable, it can
represent various kinds of material texture or changes in resistance according to
the shape.
We implemented the device, which is 580 mm in length and 1.7 kg in weight,
and has a handle of length 90 mm at each end (Fig. 6). Clunk modules are
installed every 35 mm on the device. The device diameter, excluding the clunk
modules, is 104 mm, and it can represent a maximum thickness range of 30 mm.
We used servomotors as following: S11H/2BBMG/JR manufactured by GWS
for the clunk module, and ZS-F135 by ZEBRA for the brakes. Each motor has
torques of 7.6 [kgcm] and 1.6 [kgcm]. We used FSR402 by Interlink Electronics
Inc. for the pressure sensors. The interface is connected to PC by wired from
one end of the interface.
Pressure Sensor
Also, the infrared camera detects two IR LED modules installed at the bottom
of the device, so it is able to regulate the clunk modules according to the position
and the rotation of the device in real-time. Fig. 11 shows an example of the
detected data and the calculated position of each module.
The tabletop display is 75 cm wide by 75 cm deep by 90 cm high.
4.4 Software
Fig. 12. Processing of the object data (Left: image data, Right: shape data)
At the stage of checking the reality of haptic sensations, many users were
surprised when the device actuated at the moment it was placed over the im-
age. Moreover, many users commented, The sensation of rolling out objects was
pleasant. In particular, when body parts such as faces were chosen, we could
see users enjoying the visual changes, as if their facial expressions were chang-
ing as they squashed them. Also, some people commented that they felt special
sensation or emotion when squishing themselves or accompanying person.
However, some users commented, “It was hard to feel the haptic sensations,”
when the detected objects were too small, or when the object was not very
uneven. One reason for this could be that the system is unable to detect small
unevennesses with the distance resolution of Kinect. In addition, there were some
cases of poor object measurement because the infrared rays emitted from Kinect
were reflected by, or went through, objects with a reflecting surface or some
degree of transparency. Also, there were difficulties for children such as less power
to squish or short arms to reach at the back of tabletop display, especially seen
at some exhibitions which has many children for visitors. However, we could see
many children enjoying squishing object together with the help of their parents
(Fig. 14).
– Q1 How much did you felt squishing objects in this system? (1=Not at all,
2=Little, 3=N/A, 4=Somewhat well, 5=Well.)
– Q2 What did cause you to feel the sensation of squishing objects? (Visual
Feedback, Haptic Feedback, Unknown, Others) [multiple answers allowed]
6 Conclusions
In this paper, we proposed an entertainment system for flattening any physical
object in real world using rolling-pin-based interface. The interface we developed
can represent both shapes and textures of digital objects by clunk modules and
brake for the tires. Through exhibiting the system, we found that the experience
of squishing objects provided by this system is realistic and enjoyable to some
extent.
Future research could consider using advanced physical simulations in the al-
gorithm, to obtain greater realism. Representing various kinds of materials other
than clay, by adjusting the algorithm and device design could be another subject.
To enhance the haptic sensations, several ways of upgrading the device design
could be considered: installing vibration speakers, increasing the maximum value
of the thickness range for the clunk mechanism, or downsizing the device.
In addition, the previously noted problem of errors in object measurement
using Kinect need to be solved. We are thinking of increasing the precision
References
1. Blackshaw, M., Lkatos, D., Devincenzi, A., Leithinger, D., Ishii, H.: Recompose:
Direct and Gestural Interaction With an Actuated Surface. In: Proceedings of
ACM SIGGRAPH 2011, Emerging Technologies (2011)
2. Ikeda, Y., Sato, K., Kimura, A.: Tool Device: handy haptic feedback devices imi-
tating every day tools. In: Proceedings of HCI Int. 2003 (2003)
3. Ishii, H.: Tangible Bits: Tangible bits: beyond pixels. In: Second International Con-
ference on Tangible, Embedded, and Embodied Interaction, TEI 2008 (2008)
4. Ishii, H., Ullmer, B.: Tangible Bits: Towards Seamless Interfaces between People,
Bits, and Atoms. In: Proceedings of ACM CHI 1997 (1997)
5. Iwamoto, T., Takezono, M., Hoshi, T., Shinoda, H.: Airborne ultrasound tactile
display. In: Proceedings of ACM SIGGRAPH 2008 New Tech Demos (2008)
6. Koga, D., Itagaki, T.: Virtual Chanbara. In: Proceedings of ACM SIGGRAPH
2002, Emerging Technologies (2002)
7. Ledo, D., Nacenta, M., Marquardt, N., Boring, S., Greenberg, S.: The Haptic Touch
Toolkit Enabling Exploration of Haptic Interactions. In: Sixth International Con-
ference on Tangible, Embedded, and Embodied Interaction, TEI 2012 (2012)
8. Leithinger, D., Ishii, H.: Relief: a scalable actuated shape display. In: Fourth In-
ternational Conference on Tangible, Embedded, and Embodied Interaction, TEI
2010 (2010)
9. Matoba, Y., Sato, T., Takahashi, N., Koike, H.: ClaytrcSufrace: an interactive sur-
face with dynamic softness control capability. In: Proceedings of ACM SIGGRAPH
2012, Emerging Technologies (2012)
10. Sato, M.: Development of String-based Force Display: SPIDAR. In: Proceedings of
VSMM 2002 (The Eighth International Conference on Virtual Systems and Multi
Media) (2002)
11. Ullmer, B., Ishii, H.: The metaDESK: Models and Prototypes for Tangible User
Interfaces. In: Proceedings of the 10th Annual ACM Symposium on User Interface
Software and Technology, UIST 1997 (1997)
12. Minamizawa, K., Fukamachi, S., Kajimoto, H., Kawakami, N., Tachi, S.: Gravi-
tyGrabber: Wearable Haptic Display to present Virtual Mass Sensation. In: Pro-
ceedings of ACM SIGGRAPH 2007 Emerging Technologies (2007)
13. Yoshida, A., Itoh, Y., Fujita, K., Ozaki, M., Kikukawa, T., Fukazawa, R., Kitamura,
Y., Kishino, F.: Funbrella: making rain fun. In: Proceedings of ACM SIGGRAPH
2009, Emerging Technologies (2009)
Abstract. People often communicate with others using social touch interactions
including hugging, rubbing, and punching. We propose a soft social-touchable
interface called “Emoballoon” that can recognize the types of social touch inte-
ractions. The proposed interface consists of a balloon and some sensors includ-
ing a barometric pressure sensor inside of a balloon, and has a soft surface and
ability to detect the force of the touch input. We construct the prototype of
Emoballoon using a simple configuration based on the features of a balloon,
and evaluate the implemented prototype. The evaluation indicates that our
implementation can distinguish seven types of touch interactions with 83.5%
accuracy. Finally, we discuss possibilities and future applications of the
balloon-made interface.
1 Introduction
When people communicate with others, they often make various forms of physical
contact. For example, they shake hands or hug in amiable greeting, and also punch or
slap to express anger. Since these physical expressions called “social touch interac-
tions” can be used as natural interactions with surroundings, many studies attempt to
detect these physical expressions to expand human-computer interaction [1, 19]. Us-
ing these social touch interactions, users can input their intentions and emotions. For
example, in human-robot interaction, some robots recognize rubbing and hitting as
communication with users [14]. Previously, for therapy applications, stuffed toys with
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 182–197, 2013.
© Springer International Publishing Switzerland 2013
touch sensors that can distinguish some types of social touch interactions have been
developed [19]. Haptic telecommunication [1] is one of the novel applications that
expand the channels of social touch interactions. The usage of social touch interaction
is an intuitive and natural way to express intention.
Interfaces ready for social touch interactions require two features. The first is a sur-
face soft enough to accept hard contact like hugging, grasping, and slapping. The
other is the ability to detect the force of the touch. Soft surfaces are more appropriate
than conventional rigid ones regarding safety. In addition, soft interfaces have another
advantage in that they can provide passive haptic feedback in response to user’s touch
interactions. Some soft interfaces have been proposed to accept such strong actions as
hugging and grasping [10, 19]. They employ many sensors inside their components to
detect the force of touch interactions.
To satisfy these conditions to implement an interactive surface for recognizing so-
cial touch interaction, we use such a well-sealed container as a balloon. The balloon is
soft enough to accept hard contact. As well, barometric pressure inside such a well-
sealed container as a balloon reflects the force of a user’s touch. This feature of a
balloon means that a barometric pressure sensor inside can detect the force of the
touch. Since the force is one of the important factors for recognizing kinds of social
touch interactions, sealed containers are suitable materials for constructing devices to
recognize social touch interactions. Furthermore, these features of a balloon would
not be lost when changing the size or the shape of a balloon, which means the bal-
loon-shaped interface could have many possibilities of device-design and its applica-
tion. Naturally, equipment other than pressure sensors is necessary to detect the
touched area. However, the sealed containers and the barometric pressure sensor are
still effective for distinguishing the types of social touch interactions.
In this paper, we propose a soft social-touchable interface called “Emoballoon”.
Figure 1 shows the conceptual image of Emoballoon. It can recognize various social
touch interactions such as rubbing, hugging, and so on with a balloon. The soft inter-
active surface accepts social touch interaction from users with no harm to them. The
Emoballoon has a handheld elastic sphere and simple configuration that contains sen-
sors to recognize social touch interactions. A balloon is one of the soft objects and has
such large space inside that we could enclose some electronic devices. The Emobal-
loon can present some visual feedback on its surface as well using an internal LED or
external display device. Emoballoon employs a thin elastic skin as a soft material. The
elastic skin is convenient material not only to receive strong actions but also to project
some visual feedback through its thin surface [5, 18]. We implement Emoballoon
using a balloon, and evaluate recognition accuracy for various actions acquired with
the investigation of people’s interaction with it.
We try to recognize basic touch interactions with a balloon. This attempt should
contribute to solving more complex recognition issues such as mixed actions. Our
challenge expands the input technique with inflatable soft interfaces. Our balloon-
shaped interface provides users with a wider range of social touch interactions in a
simpler configuration using a barometric pressure sensor than with existing handheld
soft interfaces. The proposed interface can contribute to creating not only handheld
interfaces detecting social touch interactions but also a large interactive surface by
arraying them. Thus, Emoballoon has a possibility to change and extend the interac-
tions between human and surfaces.
2 Related Work
In this paper, we employ a balloon as a soft inflatable material. Previously, soft inter-
faces provided the users with novel interaction techniques [5, 18]. Some interfaces
utilizing thin elastic can present visual feedback on their surfaces with projectors or a
sort of light source. Some soft interfaces have utilized elastic to detect the force of the
touch [8, 17]. These silicon-made physical widgets detect direction and strength of the
force of a user’s touch based on the deformation of widgets. While the soft physical
widgets have no electronic equipment to detect touch interactions, the system can
detect them accurately and robustly in the special environment containing a camera
and some optics.
On the other hand, some elastic interfaces detect the force of touch based on the
changes of barometric pressure. Harrison et al. have developed a shape-changing
touch screen for fusion of flexibility of a touch screen and physical cues of a physical
button [5]. The surface of the shape-changing touch screen consists of an elastic sheet.
Since this elastic sheet seals a space under the surface, barometric pressure under the
sheet reflects the shape changing caused by touch interactions. The system can
present uneven shapes on the surface by controlling the barometric pressure, and
detect the force of a user’s pressing by detecting it. Stevenson et al. also have pro-
posed an elastic interactive surface [18]. In these soft surfaces, inflatable materials
play an important role in novel input and output techniques.
Inflatable materials have also been used for handheld devices. Volflex [7] is one of
the volumetric displays using an array of small balloons. The display shows various
shapes and elasticity by controlling barometric pressure inside each balloon. Interac-
tive Balloon [16] can record and play a sound using a piezo film on a balloon. The
vision for this balloon-shaped interface is geared towards sensing and actuation in
everyday objects. The implementation is more focused on acoustic interaction rather
3 Emoballoon
Sealed objects like a balloon containing a pressure sensor have the potential to be
utilized as a soft and low-cost force sensor module. The force sensing would be help-
ful for recognizing social touch interactions like hugging, grasping, and so on. We
measure the change of the pressure in a balloon pressed with various force intensities.
In this measurement, we used a balloon (Qualatex 16 inch, white, Pioneer Balloon
Company), and BMP085 (Bosch Sensortec) as a barometric pressure sensor. The
sensor was inserted in a balloon, and the hole of the balloon was sealed with a clay-
made cylinder. We inflated the balloon until its longest diameter was 1.5 times the
size of the non-inflated one. The size was determined by a preliminary study that
indicated that the size made the balloon spherical and its surface thick enough to
guard against the balloon popping. Figure 2 shows the result of the measurement. The
result indicates that the pressure inside the balloon clearly reflects the force of press-
ing on its surface. From the result, we can find that the force of pressing or touching
can be detected by the barometric pressure of a soft-sealed container like a balloon.
A balloon has appropriate physical properties for a body of social touch device be-
cause it has softness and ability to sense the force of the touch. In addition to these
physical properties, we investigate how natural a balloon is as a form of an input de-
vice for users to input social touch interactions. We observed what actions partici-
pants perform with a balloon. If the participants perform the same touch interactions
1
Balloon Messenger, http://projects.kumpf.cc/
Fig. 2. Barometric pressure vs. force in the Fig. 3. Average scores of satisfaction with
balloon and ease of thinking of performed actions
on a balloon as they do with other people, the balloon can be considered a natural
object for social touch interactions. While the shape of the balloon is an important
factor for the naturalness, we used a simple spherical shape to evaluate the primitive
property of a balloon. In this observation, we also focused on the variation of partici-
pants’ actions to construct the list of natural touch interactions with a balloon.
In the observation, we asked participants to perform actions with an ordinary bal-
loon as if they experiencing certain emotions (happiness, love, relief, anger, sadness,
disgust, excitement, and surprise). These emotions contain Ekman’s list of basic emo-
tions (anger, disgust, fear, happiness, sadness, surprise) and some additional emotions
(relief, love, excitement) to balance the number of positive emotions and negative
ones. Participants could try to express instructed emotion in one minute per emotion.
In this trial period, the participants were left alone in a room so as not to be ashamed
to perform various actions. After the one-minute trial, we asked them to select the
most appropriate actions to express the instructed emotion, and perform it again.
Finally, they answered two following questionnaires on a seven-point scale.
• How satisfied are you with your selected actions as an expression of the instructed
emotion? (1: not at all – 7: very well)
• How easily did you think of the appropriate actions? (1: not at all – 7: very well)
The experiment from trial to questionnaire was repeated for each emotion. The
longest diameter of the balloon was approximately 25 cm. A female and seven males
(average age 23.0, from 21 to 26) participated in this study. The whole of the experi-
ments including the one-minute trial were recorded on video. We observed their per-
formance by watching the video, and picked up touch interactions with the balloon.
Figure 3 shows results of the questionnaires. The actions performed by participants
have much diversity, and participants responded to the questionnaires referring to
their own performance. The satisfaction scores of “anger” and “love” are quite high.
These high scores indicate these emotions have a strong relationship to performed
actions, and the participants smoothly think of physical expression of these emotions.
Seven of eight participants performed a “hug” as an expression for “love”, and most
of them performed “punch” as “anger”. These results indicate that “hug” and “punch”
are natural interactions with a balloon. On the other hand, there is great diversity in
actions of the other emotions. In the emotions other than “love” and “anger”, we
Table 1. Observed touch interactions whose satisfaction score is higher than four
could not find common actions among all participants. Table 1 shows the list of ac-
tions that acquired a 5 or higher satisfaction score. While these actions were not
commonly found among participants, the actions acquired a high satisfaction score
and included slapping or rubbing. In expressing some emotions, participants accepted
and felt it natural to interact with a balloon using touch interactions.
From the observational result, participants seemed to think of the same touch inte-
ractions as they did in human-human communication even when the shape of the
balloon was just spherical. They performed hugging and rubbing as if they considered
the balloon as a representation of a human. The result indicates that participants did
not really hesitate to interact with a balloon using social touch interactions, and a
balloon is one of the convenient objects with which to construct a physical interface
to input social touch interactions.
Next, from the observational study, we extracted touch interactions to be recog-
nized. We also consider the performances listed in Table 1 as natural interactions with
a balloon. While the list includes minor actions not to be found among most partici-
pants, we selected all listed actions because even minor actions might be performed
commonly with many people in other contexts. To construct a set of actions to be
recognized, we extract basic touch interactions with a balloon from the listed actions
as follows: “hug”, “punch”, “kiss”, “rub”, “slap”, “grasp”, and “press”.
From this observational study, we define six basic interactions with the exception
of “kiss” as target interactions to be recognized by a balloon-shaped interface. We
eliminated “kiss” from the targets due to hygienic issues.
A pressure sensor
A microphone
Arduino UNO
Actually, participants performed these basic actions with various force intensities
and some did multiple actions at the same time. For example, one of participants
rubbed a balloon while hugging it for expressing “love”. Another participant per-
formed a “hug” action both for “love” and “relief” with different force. These ob-
served behaviors indicate that it is necessary not only to recognize basic interactions
but also to detect the force of touch and to understand mixed actions. In this paper, we
try to recognize basic interactions with a balloon. This challenge should contribute to
solving more complex recognition problems such as mixed interactions.
inside. Rubbing the balloon’s elastic surface produces loud sounds. In many cases,
sounds caused by touch interactions have been effective cues for distinguishing user
actions [6, 13]. The system recognizes a user’s touch interactions with the balloon by
Support Vector Machine (SVM) based on the sensed data. The prototype of Emobal-
loon also contains a full-color LED inside of it to display a sort of visual feedback.
The Emoballoon can respond to touch interactions itself. Though the current configu-
ration is quite simple, we try to recognize various interactions based on the features
specific to a balloon that produce the change of barometric pressure and the unique
sound caused by a user’s touch interactions.
In addition to recognizing types of touch interactions, it is also important to detect
touched areas. Thus far, some interactive surfaces using elastic material have utilized
a basic touch detection technique based on infrared light and an infrared camera put
behind the elastic surface [5, 18]. On the other hand, a flexible capacitive sensor is
becoming more familiar in the research field to detect a touched area. In the future,
flexible sensors should be available for such soft elastic interfaces as mentioned
above. Naturally, they will also be applied to elastic handheld interfaces including our
Emoballoon, and should enable a remarkable input recognition technique in combina-
tion with the other sensors. We believe that, in the future, force sensing will be consi-
dered as a more important touch input technique in combination with detecting the
touched areas, and inflatable objects with a barometric pressure sensor will be reason-
able equipment for sensing the force of touch. Our challenge contributes to expanding
possibilities of input recognition using the force of touch.
3.4 Implementation
(kHz)
20
10
Fig. 6. Frequency spectrums of audio signals in touch interactions with loud sound. The
spectrum is calculated per 1024 samples (or 23.2 msec).
(a) Pressures in touch interactions with loud sound (b) Pressures in silent touch interactions.
Fig. 7. Barometric pressures in basic touch interactions
elastic is more resistant to bursting. The diameter of the selected white balloon when
fully inflated is approximately 40 cm. In consideration of the safety of the following
user study, we inflate the white balloon 2.0 times from the initial one in diameter, and
the diameter is approximately 20 cm, which is not fully inflated but quite safe against
hard physical contact like punching and slapping.
When the system calculates the feature values using the barometric pressure, it
uses difference values between measured pressure and the initial one (or the pressure
when doing nothing) instead of measured pressure values. This preprocess for pres-
sure values prevents the initial pressure of a balloon from affecting the recognition
process. Finally, the system separates the sensed data in 150 msec with 33.3% over-
lap, and calculates feature values from each separated data. The data length and the
overlap rate for separation are decided by a preliminary evaluation. According to
these differences in sounds and pressures, we apply the following values to feature
values for the recognition process:
These values form a feature value vector, and then the SVM classifies the vector
into basic social touch interactions.
4 Evaluation
Table 2. Average recognition rates within participants (green color density of cells means
degree of recognition rate)
Recognition rates (%)
nothing grasp hug punch press rub slap
nothing 94 0 1 0 3 1 0
grasp 0 85 1 0 13 0 0
hug 1 1 97 0 1 0 0
In-
punch 0 0 0 74 0 8 18
put
press 3 12 0 0 83 1 0
rub 2 0 1 9 1 83 4
slap 0 0 0 21 0 6 72
5 Discussion
Our implementation can distinguish seven basic actions with 83.5% accuracy on aver-
age even though it has a quite simple configuration with a microphone and a barometric
pressure sensor. From the result shown in Table 2, there are many incorrect recognitions
between “grasp” and “press”, and between “punch” and “slap”. It is still difficult to
distinguish exactly these pairs of touch interactions. However, additional evaluation
indicates the system can recognize five actions with the exception of “press” and “slap”
with 88.1% accuracy among all participants, and with approximately 93.7% within
participants on average. The system can recognize five actions including “do nothing”,
“hug”, “grasp”, “punch”, “rub” with high accuracy. To improve the accuracy in the
future, it would be useful to consider the duration of each interaction. The “punch” and
“slap” are finished in a short time while some actions including “hug” and “rub” can be
continued for a relatively long time. Thus, the difference in duration among touch inte-
ractions would be a good cue to distinguish users’ touch input.
In our current implementation, the barometric pressure is utilized to recognize
touch interactions with a balloon. Furthermore, it is also useful to detect the force of
touch as well. The balloon-shaped interfaces with a barometric pressure sensor poten-
tially can recognize both the types and the force of the touch simultaneously. If it is
possible to detect both, for example, the system can distinguish a tight hug from a
gentle hug, and a soft punch from a hard punch. Thus far, Huggable [19] has tried to
distinguish between hard and gentle touch interactions by using QTC sensors, tem-
perature sensors, and accelerometers, and it did not work well. However, employing
inflatable objects with a barometric pressure sensor will be a simple but effective way
to sense the force of touch interactions. One of our future steps is recognizing both the
types and the force of touch interactions simultaneously.
As far as we observed people in various ages interacted with our implementation,
the system does not tend to make more false recognition to the elderly than the other.
The age seems not to be a serious error factor. However, since the difference of mus-
cular strength might have a bigger impact to recognition accuracy, the evaluation with
more diverse participants has to be conducted in the future.
Another important aspect of our recognition technique is softness of the sensor mod-
ule itself. The balloon-shaped interfaces can make surrounding soft objects interactive
by being inserted into them. In this case, the balloon-shaped one would work well
while maintaining the original softness. We received such comments from the partici-
pant of the evaluation as “It is comfortable for me to input strong contact like punch-
ing and slapping because of the softness of the balloon”. The comment also indicates
that the softness of the module is an important factor for social touch input technique.
To construct the soft interface recognizing touch interactions, we used a general
balloon as a sealed container. The balloon has a risk of popping. Actually, some par-
ticipants sometimes seemed to be afraid of the risk when they were asked to punch or
slap the balloon. (Naturally, for risk avoidance, we used a thick balloon not fully in-
flated in all experiments. There was no popping in the entire observational study and
evaluation). However, the proposed recognition technique can be applied to other
sealed objects. The soft plastic-made balls used as kids’ toys are appropriate sealed
objects that have enough durability to input hard touch interactions. Such durable
objects are also useful when inserting the sensor modules into soft artifacts.
6 Future Application
Our current implementation can be applied for an emotive game controller even while
it only shows ambiguous feedback such as changing its color. Using this controller,
the users can communicate with the avatar in video games by social touch interactions
and determine the internal state of the avatar by the color emission from the control-
ler. Such a soft game controller enables more emotive and intuitive interactions with
the avatar than a touch pen by which user pokes the avatar on touch screen or gesture
input in the air like pantomime.
In the future, the social touch recognition could allow computers to understand us-
ers’ emotional expression. When the input device is connected to a computer, it
enables the users to easily tell the computer about their satisfaction with the output or
the performance of the computer not by computer language but by such physical con-
tacts as rubbing, hugging, and slapping. For simplified example, hugging or rubbing
the device would mean user’s very satisfaction, and punching or slapping would do
user’s dissatisfaction. The measured user’s satisfaction is useful to make the computer
smarter since the measured satisfaction could supervise the computer’s various calcu-
lations including the autocomplete function in text input, the recommendations in
shopping websites, and so on.
The inflatable interfaces can also enrich telecommunication by providing users
with a technique to input various physical interactions. While InTouch [1] has
achieved one of the most fundamental haptic communications using rollers as a force
input device and a haptic display, the social touch recognition is necessary to realize
more complex haptic communication. The proposed inflatable interfaces would ena-
ble the users to communicate with others using such physical expressions as hugging,
shaking hands, and slapping. These physical expressions reflect a user’s emotions
including familiarity, anger, and so on. Figure 8 (a) shows a conceptual image of hap-
tic telecommunication using inflatable interfaces. In this application, when the user
performs the social touch with the balloon, the type and strength of touch are trans-
mitted to the other user in a remote location over a network, and she or he receives
haptic feedback by a sort of haptic displays. The proposed recognition technique for
an inflatable object can enrich communication between the users in remote location.
We believe that a balloon is a familiar and appropriate object as an avatar. Floating
Avatar [22] is one of the representative examples of applications using inflatable in-
terfaces. The floating balloon of the avatar’s body provides the avatar with tangible
existence. We can regard our balloon-shaped interface as a physical avatar that can
accept such user touch interaction as rubbing and slapping. In the distant future, the
users and the operator of the avatar will be able to communicate their haptic sensation
over the balloon-made avatar.
The balloon-shaped interfaces are also useful to construct interactive stuffed toys.
Figure 8 (b) shows a conceptual image of a stuffed toy capable of responding to social
touch interactions. The stuffed toys that contain balloon-shaped sensor modules can
detect social touch interactions from users. When the stuffed toy contains small sensor
modules in some parts of their bodies, it can recognize not only types of touch interac-
tions, but also the touched area of the toy’s body. The stuffed toys will be able to actuate
their arms and legs in response to a user’s social touch. Hopefully, the interactive
stuffed toys will contribute to therapy applications as well as entertainment.
In addition to haptic communication, Emoballoon might be able to estimate a us-
er’s emotion. In the observational study, we found that almost all participants per-
formed the same actions to express some of the instructed emotions. For example,
they performed “punch” for anger, and they did “hug” for love. These common beha-
viors imply a strong relationship between emotions and physical expressions with a
balloon. To estimate a user’s emotions, it will be necessary not only to greatly im-
prove the recognition process for touch interaction with a balloon, but also to further
study the relationships between users’ emotions and behaviors.
References
1. Brave, S., Dahley, A.: inTouch: a medium for haptic interpersonal communication. Ex-
tended abstracts on CHI 1997, pp. 363–364 (1997)
2. Chang, C., Lin, C.: LIBSVM: a library for support vector machines. ACM Transactions on
Intelligent Systems and Technology 2, 27:1–27:27 (2011)
3. Fitzmaurice, G.W., Buxton, W.: An empirical evaluation of graspable user interfaces: to-
wards specialized, space-multiplexed input. In: Proc. CHI 1997, pp. 43–50 (1997)
4. Fitzmaurice, G.W., Ishii, H., Buxton, W.A.S.: Bricks: laying the foundations for graspable
user interfaces. In: Proc. CHI 1995, pp. 442–449 (1995)
5. Harrison, C., Hudson, S.E.: Providing dynamically changeable physical buttons on a visual
display. In: Proc. CHI 2009, pp. 299–308 (2009)
6. Harrison, C., Hudson, S.E.: Scratch input: creating large, inexpensive, unpowered and mo-
bile finger input surfaces. In: Proc. of UIST 2008, pp. 205–208 (2008)
7. Iwata, H., Yano, H., Ono, N.: Volflex. In: ACM SIGGRAPH 2005 Emerging Technolo-
gies, Article No. 31 (2005)
8. Kakehi, Y., Jo, K., Sato, K., Minamizawa, K., Nii, H., Kawakami, N., Naemura, T., Tachi,
S.: ForceTile: tabletop tangible interface with vision-based force distribution sensing.
ACM SIGGRAPH 2008 New Tech. Demos, 17:1–17:1 (2008)
9. Kim, K.-E., Chang, W., Cho, S.-J., Shim, J., Lee, H., Park, J., Lee, Y., Kim, S.: Hand grip
pattern recognition for mobile user interfaces. In: Proc. IAAI 2006, vol. 2, pp. 1789–1794
(2006)
10. Kim, S., Kim, H., Lee, B., Nam, T.-J., Lee, W.: Inflatable mouse: volume-adjustable mouse
with air-pressure-sensitive input and haptic feedback. In: Proc. CHI 2008, pp. 211–224
(2008)
11. Kitamura, Y., Sakurai, S., Yamaguchi, T., Fukazawa, R., Itoh, Y., Kishino, F.: Multi-
modal Interface in Multi-Display Environment for Multi-users. In: Jacko, J.A. (ed.) Hu-
man-Computer Interaction, Part II, HCII 2009. LNCS, vol. 5611, pp. 66–74. Springer,
Heidelberg (2009)
12. Knight, H., Toscano, R., Stiehl, W.D., Chang, A., Wang, Y., Breazeal, C.: Real-time social
touch gesture recognition for sensate robots. In: Proc. IROS 2009, pp. 3715–3720 (2009)
13. Lopes, P., Jota, R., Jorge, J.A.: Augmenting touch interaction through acoustic sensing. In:
Proc. ITS 2011, pp. 53–56 (2011)
14. Mitsunaga, N., Miyashita, T., Yoshikawa, Y., Ishiguro, H., Kogure, K., Hagita, N.: Robo-
vie-IV: a robot enhances co-experience. In: Proc. the Workshop on Ubiquitous Experience
Media at ISWC 2005, pp. 17–23 (2005)
15. Pai, D., VanDerLoo, E., Sadhukhan, S., Kry, P.: The Tango: a tangible tangoreceptive
whole-hand human interface. In: Proc. World Haptics (Joint Eurohaptics Conference and
IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Sys-
tems), pp. 141–147 (2005)
16. Paradiso, J.A.: The interactive balloon: Sensing, actuation and behavior in a common ob-
ject. IBM Systems Journal 4, 473–487 (1996)
17. Sato, T., Mamiya, H., Koike, H., Fukuchi, K.: PhotoelasticTouch: transparent rubbery
tangible interface using an LCD and photoelasticity. In: Proc. UIST 2009, pp. 43–50
(2009)
18. Stevenson, A., Perez, C., Vertegaal, R.: An inflatable hemispherical multi-touch display.
In: Proc. TEI 2011, pp. 289–292 (2011)
19. Stiehl, W.D., Breazeal, C.: Affective Touch for Robotic Companions. In: Tao, J., Tan, T.,
Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 747–754. Springer, Heidelberg
(2005)
20. Sugiura, Y., Kakehi, G., Withana, A., Lee, C., Sakamoto, D., Sugimoto, M., Inami, M.,
Igarashi, T.: Detecting shape deformation of soft objects using directional photoreflectivity
measurement. In: Proc. UIST 2011, pp. 509–516 (2011)
21. Taylor, B.T., Bove Jr., V.M.: Graspables: grasp-recognition as a user interface. In: Proc.
CHI 2009, pp. 917–926 (2009)
22. Tobita, H., Maruyama, S., Kuji, T.: Floating avatar: blimp-based telepresence system for
communication and entertainment. ACM SIGGRAPH 2011 Emerging Technologies, 4:1–4:1
(2011)
23. Wimmer, R., Boring, S.: HandSense: discriminating different ways of grasping and hold-
ing a tangible user interface. In: Proc. TEI 2009, pp. 359–362 (2009)
1 Introduction
All over the world, screens are used to display various digital contents, such as
movies, presentations, and shows, and are essential in the field of entertainment. The
fundamental process utilized to display content on a screen is as follows: digital
content is created, the content is rendered, and the content is shown on a screen via a
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 198–214, 2013.
© Springer International Publishing Switzerland 2013
Image View
Vibration
projection angle
Colloidal
screen
Fig. 1. The image is projected (black arrow with line) from the top. The ultrasonic waves
(black arrow) hit the membrane to reflect the image (light dotted arrow). Note that there are
two types of effects related to the intensity of ultrasound: A weak ultrasound mainly changes
the viewing angle while a stronger ultrasound additionally changes the shape of the screen.
2 Related Work
In our study, we dynamically change the shape and texture of the screen (Figure 1).
Therefore, in this section, we cite researches that are relevant to active screens, and
which either change the spatial position or the texture of an object. We then look at
how our research relates to these relevant research efforts.
constructs 3D forms on the surface of the screen using an actuator array set under the
screen. In addition, LUMEN, proposed by Poupyrev et al. [10], comprised actuated
dot matrix LEDs—physical pixels shown in RGB and H (height). Leithinger et al.
[11] also proposed an interactive deformable screen called Recompose.
Other researches dealing with control of the spatial positions of displays also exist.
There are image projection technologies that use fog as a screen, such as the systems
proposed by Rakkolainen et al. [12] and Lee et al. [13]. These technologies display
images in the air using a fog screen and a camera. This is projected in the air as a
result of fog's diffusibility characteristics.
Research on displays using water is also being conducted. Sugihara et al. [14] used
fountain as a deformable water screen. Barnum et al. [15] developed a screen that uses
the water drops in the air. Water drops have lens-like characteristics. By using these
characteristics, they were able to project an image onto the water drops. They made a
water drop sequence in the air and projected the image corresponding to the spatial
position by synchronizing the projector with the water bulbs, and applied their
technology to create a multilayer screen. Other interesting artworks include those
presented by Suzuki et al. [16], which uses underwater air bubbles, and Kodama et al.
[17], which uses magnetic liquid.
Displays have also been made using soap bubbles. Bubble Cosmos [18] is a
technology that constructs a screen in the air by confining fog in a bubble; and Shaboned
Display [19] turns a bubble into a pixel. Bubbles have also been used as a musical
instrument [20].
Size of transformation
10 m
water fall display
[14]
-1
plain bending screen with
10 m actuator array
screens
[8][9][10][11]
this research
-2
10 m soap bubbles
display [18][19]
magnetic fluid
[17] texture with
-3 screen with
10 m grid actuators fur display
water drops [15] [4]
[2][3]
Fig. 2. Position of our research corresponding to other relevant researches. Vertical axis represents
the amount of transformation dealt to the object while the horizontal axis represents the actuator’s
resolution. Notice how our research (red) covers the areas that have not been explored yet.
Our research is positioned as shown in Figure 2. The size of the changes on the
vertical and horizontal axes represents the resolution of actuation. For example, since
a dynamic BRDF [5] changes the surface texture by detailed vibration, the resolution
of actuation is high and the size of change is small. Researches on dynamic textures
with small actuators are not high-resolution and the size of spatial change is limited.
Using this rationale, we positioned our research in the area colored in red. The
actuator resolution of our research is high when the spatial change is small and low
when the spatial change is large. No research equivalent to this domain exists. Our
research contribution is high because we use the same hardware settings to
accomplish many of the features achieved in related research.
3 Theory
In this section, we describe the theory underpinning the colloidal screen technology.
First, we describe the capillary waves that are induced on the soap film, then
introduce the reflection model of the colloidal display, and finally, describe the
ultrasounds.
(1)
where σ is the surface tension, ρ is the density of the colloidal solution and f is the
excitation frequency. Wavelength λ is estimated from Equation (1). The surface
tension of colloidal liquid σ is 0.07275 N/m (20 deg C). Suppose the surface tension
of colloidal liquid to be 1/2 of water and density to be the same as water 1000kg/m3.
In this situation, with 40kHz ultrasounds, the wavelength λ is 2π/k=83μm.
These minute waves (Figure 3) occur on the ultrasound-activated colloidal film and
diffuse the light on the surface.
t = 0s t = 2s
200x
t = 1s 500fps
Fig. 3. Capillary waves on the colloidal film. ( 200x, 500 fps high speed camera capture
image).
BRDF model
on the surface irradiance Li normal
vector irradiance Lr
illuminance Ei
n
incoming light θi
ωi outgoing light
ωo
surface(colloidal membrane)
eye
light source
perspective
o1
o2
o1 o2
i1
o1
i2 o2
point
image
the BRDF model. The expansion of the existing range of reflection is shown as ω '' o1
and ω ' for ω in Figure 5. The perspective light source covering a certain range
o1 o1
makes the image appear on n the film. We project the image on the colloidal screenn in
this manner.
(3)
(4)
In our research, we can control the reflectance distribution to some extent, but we
cannot control the radiationn and absorption of the material. The relationship betwween
radiance L and radiance L follows two formulas. It does not depend on the
r i
distribution ratio of ω . This indicates that a very bright image can be presented whhen
o
ω is a narrow distribution.
o
The focal point of the ultrasound is generated by setting adequate phase delays in
the multiple transducers. In addition, the focal point can be moved to an arbitrary
position by controlling the phase delays. A trade-off exists between the spatial
resolution and the array size. Theoretically, the spatial distribution of ultrasound
generated from a rectangular transducer array approximates the shape of a sinc
function [23]. The width of the main lobe (w [m]) is parallel to the side of the
rectangular and is written as
2λ R
w= (6)
D
where λ [m] is the wavelength, R [m] is the focal length, and D [m] is the side length
of the rectangular array (Figure 8). Our system can control the spatial position of
focus and it contributes to activate the films partially.
4 Design
In this section, we look at the design requirements for the colloidal display and give
an overview of the system.
waterfall based
colloidal screen to viewer
reflection
ultrasound image
servo
projection
image
ultrasonic
device liquid
projector
tank
Fig. 7. Colloidal display system components: focused ultrasonic devise set on the opposite side
of projector. The image on the screen can only be seen from the projector side.
Ultrasonic waves directly affect the reflection of the film, so if we switch the
ultrasonic waves on/off, to human eyes, it appears as if a switch is being made from
transparent to opaque.
Frequency 40 kHz
Figure 7 shows one of the configuration that uses these components. Projector's
light goes to the film and film's frame. Ultrasound waves are produced from the
speaker simultaneously and hit the film, vibrating it. When the film is broken, it is
replaced by servo-motors. The specifications of each part are described in table 1.
5 Evaluation
from zero to 90 degrees. Because the screen is vibrated intensely, the laser’s
reflection is not consistent. This is shown in Figure 8 (right), where t is time. There
are slight differences in each image but the shape of the laser reflection is similar.
The graph of the brightness on each angle is shown in Figure 9. It shows that the
narrow reflection angle (with low ultrasonic intensity) is bright in the center of
the reflection and the brightness diminishes in the neighborhood of the center of the
reflection. In contrast, the wide reflection angle (with high ultrasonic intensity) results
in a gentle decrease in brightness.
This result shows that we can control the view angle and distribution of the
brightness by changing the intensity of the ultrasonic wave. It indicates that the
colloidal screen solves the problem of screen selection and enlarges the application of
the projector screen.
Fig. 8. Laser experiment example of wave form modulation. Different types of waves create
different view angles. White noises create 40 degrees, sine waves create 14 degrees, and square
waves create 2 degrees.
Fig. 9. This graph shows how view angle affects brightness. The red line has a small viewing
angle but it is brighter than the blue line, while the blue line has a wider viewing angle but
brightness is low.
0 min 2 min
Fig. 10. From left to right the image quality decreases in relation to time due to water evaporation
5.2 Stability
In this subsection, we describe the retention time along with the stability of the colloid
film. In our experiment, the colloidal film kept its membrane stable for three minutes
on average when the ultrasonic waves were applied. The key component is water; the
less water there is in the membrane, the more noisy the image. Figure 10 shows how
the image quality changes with time. Observing the circle on the four images, it can be
seen that the images on the left devolve into the images on the right over time. This is
because as time passes the water evaporates, causing the transmission characteristics to
be reduced, which results in the image being disturbed. The main reason for this
disturbance of the display is failure to control the reflection characteristic.
6 Applications
left-view right-view
of frames image of frames
image
mirror
front-view
of frames
image
Fig. 11. (Perspective Screen) Three different images from different perspectives are shown.
The top image viewed from the left, the middle from the center, and the bottom from the right.
Fig. 12. (Plane based 3D screen) The transparency alternates within the 3 screens
We set up multiple sets of colloidal screen and ultrasonic speakers with ultrasonic
oscillations synchronized with the projector. The installation position is shown in Figure
12. Each colloidal screen had an ultrasonic speaker that set its transparency. Toggling
the transparency of each screen was achieved by simply turning the speakers on and off.
The system was able to show three different images on each colloidal screen by
controlling the transparency of each screen synchronized with the projector's images.
This is effectively a 20 Hz time division plane-based 3D screen with a single projector.
projector as projector
= =
light source light source image source
texture
texture + reflection
reflection
of surface texture
reflection θ
BRDF of light source
mirror diffuse
surface texture in real world
x% y%
Fig. 13. Shows how we can mimic a texture in real world controlling the percentage between
mirror (x%) and diffuse (y %) in time division
Fig. 14. Sample results for four reflective states from five different view angle. Note that red
circle shows the reflective spot from the mirror and it is consistent with the bottom images.
Colloidal film is a unique material whose reflection range can be controlled. This
characteristic, coupled with high frequency control of ultrasounds, facilitates this
application. The results depicted in Figure 14 shows that our system successfully
operated as a reflective display. It follows the reflection of the viewing point.
By controlling the focus and the spatial position of the force field, it was possible
to change the size of the deformation point on the membrane from 2 mm to 20 mm
(Figure 15). In addition, when two phased arrays were used we were able to control
both the projection and the deformation.
7 Discussions
By turning soap film into a projector screen, we gained controllable properties such as
flexibility, transparency, and durability. However, we acknowledge that a soap film
does not last forever and there are limitations on the view angles. In this section, we
first look at the limitations, then discuss the possibilities of our system for use in
entertainment computing.
The maximum view angle is 180 deg because this display is based on reflection. We
achieved a maximum view angle of 40 deg. The view angle changes according to the
intensity, the waveform of amplitude modulation, and the frequency of the ultrasound.
If the view angle is too small, only the light is shown and not the content of the
projected image because it is too acute for human eyes. However, we used this to our
advantage and displayed three different images from different angles (we applied it to
the perspective screen).
References
1. Ochiai, Y., Oyama, A., Toyoshima, K.: A colloidal display: Membrane screen that
combines transparency, BRDF and 3D volume. In: ACM SIGGRAPH 2012 Emerging
Technologies (SIGGRAPH 2012), Article 2, 1 page. ACM, New York (2012)
2. Raffle, H., Joachim, W.M., Tichenor, J.: Super cilia skin: An interactive membrane. In:
CHI 2003 Extended Abstracts on Human Factors in Computing Systems (CHI EA 2003),
pp. 808–809 (2003)
3. Coelho, M., Maes, P.: Sprout I/O: A texturally rich interface. In: Proceedings of the 2nd
International Conference on Tangible and Embedded Interaction (TEI 2008), pp. 221–222
(2008)
4. Furukawa, M., Uema, Y., Sugimoto, M., Inami, M.: Fur interface with bristling effect
induced by vibration. In: Proceedings of the 1st Augmented Human International
Conference (AH 2010), Article 17, 6 page (2010)
5. Hullin, B.M., Hendrik, P., Lensch, A., Rasker, R., Seidel, H., Ihrke, I.: A dynamic BRDF
display. In: ACM SIGGRAPH 2011 Emerging Technologies (SIGGRAPH 2011), Article
1, 1 page (2011)
6. Koike, T., Naemura, T.: BRDF display. In: ACM SIGGRAPH 2007 Posters, SIGGRAPH
2007, Article 116 (2007)
7. Poupyrev, I., Nashida, T., Okabe, M.: Actuation and tangible user interfaces: The
Vaucanson duck, robots, and shape displays. In: Proceedings of the 1st International
Conference on Tangible and Embedded Interaction (TEI 2007), pp. 205–212 (2007)
8. Cholewiak, R., Sherrick, C.: A computer-controlled matrix system for presentation to skin
of complex spatiotemporal pattern. Behavior Research Methods and
Instrumentation 13(5), 667–673 (1981)
9. Iwata, H., Yano, H., Nakaizumi, F., Kawamura, R.: Project FEELEX: Adding haptic
surface to graphics. In: Proceedings of the 28th Annual Conference on Computer Graphics
and Interactive Techniques, pp. 469–476 (2001)
10. Poupyrev, I., Nashida, T., Maruyama, S., Rekimoto, J., Yamaji, Y.: Lumen: Interactive
visual and shape display for calm computing. ACM SIGGRAPH 2004 Emerging
Technologies (2004)
11. Leithinger, D., Lakatos, D., Devincezi, A., Blackshaw, M., Ishii, H.: Direct and gestural
interaction with relief: A 2.5D shape display. In: Proceedings of the 24th Annual ACM
Symposium on User Interface Software and Technology (UIST 2011), pp. 541–548 (2011)
12. Rakkolainen, I., Diverdi, S., Olwal, A., Candussi, N., Hoellerer, T., Laitinen, M., Piirto,
M., Palovouri, K.: The Interactive FogScreen. ACM SIGGRAPH 2005, Emerging
Technologies (2005)
13. Lee, C., Diverdi, S., Hoellerer, T.: Depth-fused 3D imagery on an immaterial display.
IEEE Transactions on Visualization and Computer Graphics 15(1), 20–33 (2009)
14. Sugihara, Y., Hashimoto, H., Yamamoto, K.: A proposal of water display. In: Proceedings
of the Seventh International Conference on Artificial Reality and Tele-Existence, pp. 63–70
(1997)
15. Barnum, P.C., Narasimhan, S.G., Kanade, T.: A multi-layered display with water drops.
ACM Trans. Graph. 29(4), Article 76, 7 page (2010)
16. Suzuki, T.: Water Canvas 2004-2013
17. Kodama, S., Takeno, M.: Protrude, Flow. In: SIGGRAPH 2001 Electronic Arts and
Animation Catalogue, p. 138 (2001)
18. Nakamura, M., Inaba, G., Tamaoki, J., Shiratori, K., Hoshino, J.: Bubble cosmos. In: ACM
SIGGRAPH 2006 Emerging Technologies, SIGGRAPH 2006, Article 3 (2006)
19. Hirayama, S., Kakehi, Y.: Shaboned display: An interactive substantial display using soap
bubbles. In: ACM SIGGRAPH 2010 Emerging Technologies (SIGGRAPH 2010), Article
21, 1 page (2010)
20. Suzuki, R., Suzuki, T., Ariga, S., Iida, M., Arakawa, C.: “Ephemeral melody”: Music
played with wind and bubbles. In: ACM SIGGRAPH 2008 Posters (SIGGRAPH 2008),
Article 80, 1 page (2008)
21. Lamb, H.: Hydrodynamics, 6th edn. Cambridge University Press (1994) ISBN 978-0-521-
45868-9
22. Hoshi, T.: Compact ultrasound device for noncontact interaction. In: Nijholt, A., Romão,
T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 502–505. Springer, Heidelberg
(2012)
23. Hoshi, T., Takahashi, M., Iwamoto, T., Shinoda, H.: Noncontact tactile display based on
radiation pressure of airborne ultrasound. IEEE Transactions on Haptics 3(3), 155–165
(2010)
Abstract. This paper presents the design and evaluation of “the man-machine
interface” a punchable interface designed to criticise and react against the val-
ues inherent in modern systems that tacitly favour one type of user (linguisti-
cally and technically gifted) and alienate another (physically gifted). We report
a user study, where participants used the device to express their opinions before
engaging in a group discussion about the implications of strength-based interac-
tions. We draw connections between our own work and that of evolutionary bi-
ologists whose recent findings indicate the shape of the human hand is likely to
have been partly evolved for the purpose of punching, and conclude by examin-
ing violent force as an appropriate means for expressing thoughts and feelings.
1 Introduction
This paper presents a project designed to explore, criticise and provoke consideration
of values and biases inherent in modern interface design. We propose that communica-
tion technologies have become increasingly advantageous to linguistically talented
people, to the detriment of those whose strengths lie more in gross motor movements
and physical prowess. Further, due to the increasing technological mediation of many
aspects of modern life such as work, education, health care, play, socializing, and gov-
ernance, we are concerned over the potential wide scale disenfranchisement and disillu-
sionment of people whose talents are primarily physical. As a case study, we examine
inherent biases in contemporary methods for expressing preferences and opinions. We
challenge these biases through the design of a provocative artefact, the “man-machine
interface,” which intentionally privileges physically gifted people at the expense of
those who are less physically talented. The approach is inspired by the methods of
feminism, which are designed to identify and criticise systematic inequalities.
Below, we first make the argument that contemporary technology, designed by
‘nerds’, serves the interests of nerds far better than their traditional enemy, the ‘jock’
[1]. We introduce critical design as a methodology for both understanding and
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 215–229, 2013.
© Springer International Publishing Switzerland 2013
1.1 Background
Since the first proto-human picked up a bone club to cave in the skull of his oppressor
[2], tools have often been used by the physically weak to subjugate the strong. Just as
David slew Goliath through his covert use of technology, modern technology disem-
powers honest strength in favour of (wordy) obscurantism. Further, it is apparent that
today’s systems of symbolic manipulation have allowed a disconnect to develop be-
tween work and value, allowing the subjugation of the most physically able. Stagger-
ing sums of money (c.f. work) can be manipulated by a few clicks or the effortless
swipe over a tablet’s surface. There is no feedback that relates to the physical conse-
quences of such casual gesture.
"Written words destroy memory and weaken the mind, relieving it of work that
makes it strong… They are an inhuman thing.” Socrates (in Plato’s Phaedrus)
In general, a system develops to benefit those who develop the system [3]. Modern
society functions through mechanisms put in place by successive generations of bu-
reaucrats whose chief skills are in the manipulation of symbolic information. Those
best able to engage with these semiotic mechanisms will naturally flourish and be
promoted to positions whereby they can strengthen the same system by which they
were valued in the first place. A runaway process of natural selection has occurred
with systems of governance becoming ever more bloated and the majority of populace
ever more distant from the policy makers [4].
Computing technology is an essential component of the systems through which the
modern world is governed. There is fundamental digital mediation in almost all aspects
of our experience, from work, to education, health care, play, socializing, and govern-
ment. Indeed, much discussion has taken place recently on the consequences, both in-
tended and unintended, and both potential and already realised, of delegating decisions
and responsibility to technology [5]. It is beyond question that computing technology
serves to perpetuate and promote societal inequalities. However, the idea of technological
determinism; that a society is solely a product of its enabling technology [6], is a fallacy.
Digital computers have the capacity to deploy any number of systems, and have no re-
quirement to benefit any particular group. So, the values designed into computing tech-
nology - either deliberately, or by default - do not necessarily need to conform to any
present set of agreed upon social norms. Instead, technology can, and should, react
against the status quo and provoke the development of new values and morals.
As feminism has provided a lens on the gender inequalities in society, and the often
tacit discrimination that occurs through the unquestioning implementation of systems
within a specific set of norms [7]; we draw inspiration from these ideas to develop
work which can benefit those currently disenfranchised by the disempowerment of
strength [8,9,10].
We believe that it is important for society to recognise, accept and learn to deal with,
the inherently aggressive and violent nature of the human species. Indeed, recent
work in evolutionary biology emphasises that fighting, specifically the use of the hand
as a fist, has markedly shaped its evolution [18]. Prof David Carrier discusses this in a
recent interview,
"I think there is a lot of resistance - maybe more so among academics than people
in general - resistance to the idea that, at some level humans are by nature aggres-
sive animals. I actually think that attitude, and the people who have tried to make
the case that we don't have a nature - those people have not served us well….. I
think we would be better off if we faced the reality that we have these strong emo-
tions and sometimes they prime us to behave in violent ways. I think if we acknowl-
edged that we'd be better able to prevent violence in future." [19].
There are many methods used in the social sciences that allow participants to express
internal values; opinions, thoughts and feelings (e.g. questionnaires, interviews, forms
with Likert scales and so on). These methods are all inherently verbal; they require the
participant to undertake complex linguistic, or relational, reasoning in order to produce a
response that can be used by others when making decisions. The response produced by a
participant is typically accepted as a valid measure of that person’s opinions, thoughts
and feelings, regardless of their relative ease or difficulty in formulating that response.
Crucially, those who have greater skill at matching their internal experience with the
specific response produced, have greater chance of being understood, and of having their
interests acknowledged and represented. It seems interesting to investigate methods
through which physically talented people can express opinions with similar finesse.
2 Design
2.1 Conceptual Design
In addition to affording a novel and physically active method of soliciting feedback
that may reveal baser truths about a participants’ internal state, the design of the sys-
tem highlights the inherent inequalities in existing methods of feedback and decision-
making. The system deliberately allows individuals who are able to punch more
strongly to have a greater impact on results. This is in contrast to traditional systems
of voting in which each individual’s vote is valued the same; however, the ability to
engage in verbal debate is not equitable and the feelings of the rhetorically weak are
often left underrepresented. The interface is designed to facilitate public display of
physical prowess (with some targets able to be punched powerfully merely for show).
This contrasts with the ‘secret-ballot’ in which individuals are forced to cloak their
representations in the plebiscite veil of the voting booth.
The design of the man machine interface is conceptually simple. A martial arts
training aid is used as input for a computer application that allows participants to
express opinions. A screen presents participants with questions. Paddles to the left
and right of the device (see Figure 5) are used to answer yes/no (or agree/disagree
etc.) questions. The paddle at the top is used for expressing strength of feeling. All
three paddles measure the strength with which participants punched it.
There are some pragmatic considerations in relation to the design. Some people are
left-handed some are right-handed. An interface based on punching power therefore
needs to be symmetric so as to favour neither. Hence, while binary left/no triggers are
suitable for asymmetric distribution (ignoring any cognitive bias), a power measure-
ment is not. Thus the central head target was used to detect power; with the left/right
targets used for selection. The character used to ask questions was a bland, non-
threatening automated character (see Figure 6) that would generate little response in
and of itself (and early prototypes showed this to be the case). Practically, the auto-
mated creation of the animation (based on a text script) allowed the interface dialog to
be altered very quickly, which allowed rapid iteration of the design.
2.2 Technology
The system has been adapted from a standard martial arts training aid (a Body Oppo-
nent Bag, or BOB). The device selected was the MATT (the Mixed Martial Arts Tar-
get Trainer)1 and this was selected as it was advertised as being the only home based
fitness product capable of simulating the feeling “punching a man’s head” (although
there are now more products offering this feature of debatable benefit).
1
http://www.bayls.co.uk/products/fitness/
mmatt-mixed-martial-arts-target-trainer/
Three of the four targets were used (the left and right ‘hands’ and the ‘head’), with
the central (‘body’) target being replaced with a flat screen monitor to provide instruc-
tion and feedback to the participants.
USB (Singstartm) microphones were loosely embedded on the back of each target
and these were connected to a Raspberry Pi computer for processing.
The Raspberry Pi ran a standard Raspbian version of the Linux operating system.
A Python script was developed to detect punches (via the RMS of the microphone
responses). Punches were recorded to a text file (with volume deemed to approximate
the strength of each punch) and used to trigger sound effects (via an external USB
speaker) and to start and pause a full screen HD video rendering (oxplayer) of a cute
character reading the following instructions.
3 Evaluation
Thirty potential participants - the entire (all male) undergraduate game design class -
were asked at the end of the semesters final practical session if they would consider
participating in a study relating to interface design. Nine participants, aged 20-23,
were recruited and were fully informed as to the purpose of the study and what was
required from them. All those who volunteered initially gave written consent to par-
ticipate and were then asked to fill in a paper-based module evaluation form. This
form allows students to anonymously express opinions regarding the quality of that
module and is a task routinely undertaken at the end of each module studied. Next,
participants were asked to use the interface situated in the Games Computing Lab,
one at a time, following the instructions outlined above. Afterwards, two focus groups
were conducted in a semi-structured manner, led by the authors. Discussion initially
focused on the subjective experience of users engaging with the device, before explor-
ing the concept of expressing “strength of feeling” through “physical strength” and
the effects that such a means of expression may have on users, and use, of technology.
and enthusiastic than other groups that we could have sampled. However, participants
were also third- and fourth-year computer science students. As such, they are people
who have already benefitted from the shift in cultural values away from physical to-
wards linguistic and technological skills, and may expect to continue to do so in future.
Due to these potentially conflicting opinions, we believe that they are a particularly
interesting group with which to discuss the concept of strength-based interfaces.
Two separate focus groups were convened to discuss the man-machine interface.
Group one consisted of five participants and group two consisted of four. Both ses-
sions were audio recorded and those recordings were transcribed for analysis. Group
one discussion lasted for approximately one hour; group two lasted for 30 minutes.
someone is following you, you could just punch something over ten, it would either
turn on or turn off,”
“Definitely off.”
though the actions were against martial arts training aids specifically designed for this
purpose, this particular group seemed unusually averse to acts of physical expression.
This is perhaps to be expected given those activities that would have likely received
positive reinforcement through the participants’ formative development and training,
and, a different group of participants in a different context may well not exhibit such
extremes.
Performance. As the session wherein participants interacted with the device took
place in public, and classmates were allowed to watch if they wished, there was an
element of performance to the behaviour of participants. Participants discussed how
they were very conscious of being watched while punching the device;
“It was a little bit weird getting in front of everybody while they was looking at
you, punching something,”
and how being watched affected their subjective experience and their actions;
“I mean, you don’t want to like, hit really slightly if everyone else is like, smacking
it,”
“you don’t want to be like the worst person,”
“How am I supposed to react to this properly … ?… in a way that won’t make me
look like an idiot,” “You didn’t want it to look like we were weak.”
These comments are fascinating in the context of this paper as they underline the
disruptive influence of the device. Specifically, this group of people rarely if ever
interact with each other in a way that values physical strength. Rather, being an
undergraduate computer science cohort, the social hierarchy of most influential or
valued members of that group is determined more by their technical skills. The device
provided a disruptive influence on the established pecking order and provided the
opportunity for physicality-based sexual display.
“a lot of people would do like a typical movie punch and just swing from the side,
[…] where as I do […] a straight sort of punch, which generally gets more power.
So, […] I think from that I might feel strongly about something more so than some-
one else,”
or confidence,
“Not even the physical size, just like more confident in yourself. You get up and
don’t care if those people watch you, then you’re going to do what you’re going to
do,” “if you get people that aren’t confident standing in front of other people, […]
even if they are very strong, they’re still going to be really, really reserved.”
Participants also suggested means for remedying the unreliability of the interface.
Specifically, they suggested taking baseline readings for every user,
“You’d have to look at baseline,” “You have like a frame of reference. Everyone’s
working then on the same kind of relevant level,” “Unless, of course, like, the sec-
ond part was like I saw, a mini-baseline hits, because people are bound to hit ei-
ther “Yes,” and punch it really hard,” “Or hit “X” amount of between zero and
ten times.”
Indeed, the researchers had to repeatedly steer discussion away from implementation
of baselines and how they could be implemented. This insistence on fairness across
participants is very interesting.
Unfairness of the Interface. Most participants expressed some concerns over general
“unfairness” inherent in a device that uses physical strength as an input technique.
These concerns came in two forms; the first addressed reliability of measurement
across individuals, the second focused on the potentially discriminatory effects of this
type of input mechanism on those people who use it. Disappointingly none of the
groups seemed willing to accept that these biases were a deliberate aspect of the de-
sign, and were largely blind to the inherent inequalities in existing systems of societal
operation. It seems that participants had difficulty considering the meaning of the
design. Given time and opportunity for greater reflection and discussion it is possible
that deeper insights may have emerged (though again, results from this idiosyncratic
participant pool may not generalise well).
Again, while participants identified and discussed the inherent discrimination in the
punch interface system, there was little evidence of further thought around existing ine-
qualities perpetuated by other types of systems, whether technological, political or social.
4 Conclusion
The expression of physical force is a natural aspect of human behaviour. This study
provides an initial exploration of violent force as an input mechanic to interactive
technology (tapping into emotions and providing an outlet for pent up aggression). It
seems that punching is an interesting interaction technique, enabling responses to be
collected based on momentary expression of explosive power. The system reliably
detected punches of various strengths, and although more sophisticated measuring
devices could make the measurement of force more accurate, in terms of this initial
enquiry the correlation with volume proved to be accurate enough.
Participants in the current study demonstrated an ability to effectively control the
strength of their punches. They also demonstrated, and reported, significant individual
differences in base level of strength, skill in punching, and reaction to the public test
situation. The design of the system sought to make use of these differences to provoke
discussion relating to the inequalities in other systems. There was clearly a good deal
of cognitive dissonance in the group caused by this request and the activity of behav-
ing in a ‘violent’ manner in a computer lab, there are several aspects to this.
Firstly, we found indications that social conditioning undermines tendencies towards
violence in social situations. Participants reported being taught by parents from a young
age that violence is not appropriate, particularly as a means of expression. Secondly, the
group was highly sensitive to disturbances of the status quo. This included threats to
their position within the group hierarchy (through embarrassment or lack of ability).
Thirdly, participants were concerned with notions of fairness. Many of the objections to
the interface sought to rectify, or reinstate existing bias (e.g. by calculating power rela-
tive to a base line). The findings that participants naturally (and unknowingly) wish to
reinstate bias that benefits themselves is broadly as predicted, and is coherent in terms
of the initial assertions regarding the development of societal structures.
In summary, this study constitutes an initial exploration of violent force as an in-
teraction style for interactive technology. It also examines violent force as a means for
expressing thoughts and feelings. While participants reported enjoyment at interacting
with the device, they also expressed reservations about the social acceptability of
behaving in violent ways. In addition, while participants recognised that the interface
was not fairly designed, there was very little evidence that it provoked critical insight
on the inherent inequalities in existing systems of societal operation. We intend, in
future studies, to explore further the critical potential of violent interaction.
References
1. Gilsdorf, E.: Geek Pride: Jocks vs. Nerds, Brawn vs. Brain, Hunks vs. Dweebs. Psycholo-
gy Today (September 25, 2010), http://www.psychologytoday.com/
blog/geek-pride/201009/jocks-vs-nerds-brawn-vs-brain-hunks-
vs-dweebs (retrieved January 7, 2013)
Abstract. Comics are a unique and classical form of storytelling. The advent of
interactive narratives brings the possibility of interaction to the traditional comic
books. In this paper we present a non-branching interactive comics system capa-
ble of generating dynamic interactive narratives in the format of comic books.
The system allows users to interact with certain objects, and then observe the
consequences of their actions in the unfolding story. We validate the proposed
system with a user study conducted with 18 participants. The results indicate that
such systems may indeed provide an attractive form of entertainment.
1 Introduction
Comics are a popular form of visual storytelling, wherein juxtaposed still images are
combined with text. Modern comic books emerged at the turn of the 19th century and
evolved in different ways in Europe, America and Japan. In the early 19th century, the
Swiss artist Rodolphe Töpffer published the book Histoire de M. Vieux Bois, which
some consider the first "comic book" [17]. In America, the genre of superheroes has
dominated the mainstream for decades, and its popularity has varied widely since the
first period of popularity in the 1940s, known as the American Golden Age of comic
books. The appreciation for the storytelling abilities of comics, however, was even
more remarkable in Japan, where manga (Japanese term for comics) gained reputation
both for profitable sales and diversity of genres. Nowadays, classical printed books
are sharing space with digital forms of comics, such as Marvel’s The Avengers: Iron
Man – Mark VII [34]. The developers of Webcomics, which are comics published on a
website [31], are now exploring new forms of comics, such as narrative branching
structures (also known as hypercomics) and animated panels with sounds (also known
as motion comics). An example of hypercomic is Meanwhile [30], which is a branch-
ing interactive comics where one navigates making choices and solving puzzles.
Through the last ten years, many researchers have attempted to transform the classical
form of comics into a new form of digital interactive content. The applications
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 230–245, 2013.
© Springer International Publishing Switzerland 2013
2 Related Work
The automatic generation of comics has been an active topic of research since Kur-
lander et al. proposed their famous Comic Chat system [1] in the nineties. Comic
Chat is a system capable of automatically generating comics from chat sessions. It
determines the correct poses for each character and situation through a semantic anal-
ysis of the participants’ messages. Following a similar approach, Sumi et al. present
ComicDiary [2], a system that automatically creates personal diaries in a comic style.
In the gaming context, Shamir et al. [3] and Shuda and Thawonmas [4] propose a
system to automatically generate comics from game sessions summarizing the game
events. Chan et al. [5] adopt a similar approach and present a system that automatically
summarizes players' actions and interactions in the game World of Warcraft through
comics. Pizzi et al. [6] use comic-like storyboards to represent game level solutions as a
game design tool. Their system generates all possible solutions to a given game level
using the player character as the main agent. Then, they attach elements of storyboards
to the planning operators so that a complete solution generates a comic strip
representing the storyboard.
Alves et al. [7] describe an XML-based language that semantically describes com-
ics. They also present Comics2D, a system able to interpret the XML language and
generate comic strips. In a more recent work, Alves et al. [8] present another system
able to create comic-like summaries of agent-based stories generated by the interac-
tive storytelling system FearNot! [9]. The system analyses the story logs, characters'
emotional information to understand their actions and their importance in the story,
and then selects the most important events to create comic strips.
There are some interactive experiences based on comics designed for mobile de-
vices, like Nawlz [22], which is an adventure comic book designed for iPad that com-
bines animation, interactivity, music, and text in a panoramic comic format. However,
the story is entirely linear and user interactions don’t have any effect in the story out-
come. Some major comic publishers have tried to create interactive comic books. A
recent example is the already cited Marvel's The Avengers: Iron Man - Mark VII [34],
which is an interactive comic book designed for mobile devices that allow users to
play with some interactive elements of the scenarios, but without affecting the story.
The possibility of choosing between different story paths is explored by Goodbrey
[19][20]. The author presents several web applications that combine the concept of
branching narratives with the idea of “infinite canvas” proposed by McCloud [21].
The commercial product Meanwhile for iPad and iPhone [30] is another example of
branching technique. Taking a different approach, Andrews et al. [23] explore the
application of interactive comics in a projected multi-touch interface. Their system
projects on a sheet of paper predefined lower-level narrative elements, such as charac-
ters and dialogues, allowing users to enrich the story by adding “top level” objects.
Our system differs from the aforementioned works because we integrate three
important features: plot generation; interaction affecting the plot generation; and au-
tomatic layout generation based on the plot. Furthermore, the story unfolds from a
logical framework rather than from a predefined branching structure. Such an inte-
grated and flexible approach is possible because the system is supported by the fol-
lowing components: (1) a planning module that automatically generates coherent and
diversified stories according to the user interventions; (2) intelligent algorithms capa-
ble of generating comics in real-time. To the best of our knowledge, this is the first
work on non-branching interactive comics in the literature.
3 Comics
Comics are a narrative form that uses a sequence of panels containing signs and im-
ages combined with texts, where the reader perceives time spatially. The texts are in
the form of speech balloons, captions, and onomatopoeic words for sound effects
(SFX). Our treatment of comics relies on the understanding that comics consist of:
“sequential art”, as pointed by Eisner [18]; semiotic entities, as claimed by O’Neil
[29]; and juxtaposed images in deliberate sequences, where time and space are one
and the same, as observed by McCloud [17]. The question of time is clearly explained
by McCloud using a comparison with movies in his book. He points out that each
successive movie frame is projected on exactly the same space, while each comic
frame occupies a different space – and concludes that space does for comics what
time does for film.
An easy way to understand the language of comics is by looking at the structure of
a comic book. A comic book usually comprises one or more pages, each page consist-
ing of panels, whose varying size and location influence the reader’s progress. These
panels convey space and time and behave as picture planes through which the reader
views the action of the story. Inside the panels, story events are represented through
three types of visual elements: figurative, iconic, and textual [16]. Figurative elements
are the characters themselves, their actions, facial expressions, and gestures,
representing what they are doing and what they are feeling. Iconic elements include
speech balloons, thought balloons, SFX balloons, and narration boxes. These icons
inform the reader which words are being thought, said, or narrated. In addition, the
shapes of these icons can be altered to add nuance to the words found within them.
Finally, the textual elements represent the text found within the speech balloons,
thought balloons and narration boxes. Figure 1 indicates the basic elements of comic
books. A more complete and detailed analysis of the visual language of comics is
presented by McCloud [17] and Eisner [18].
caption SFX
gutter
panel
speech
balloon
Fig. 1. Elements of comic books (panel, gutter, caption, speech balloon, SFX). Copyright ma-
terial under “fair use” policy.
Although comics are sometimes seen as a childish form of storytelling, with im-
ages that are often deliberately simplified, they are still capable of evoking strong
emotional reactions from readers, creating identification, and effectively conveying a
story in an very appealing manner [3].
4 Interactive Com
mics
Our method to create a neew form of interactive content for comics has three m main
tasks: (1) the automatic gen neration of interactive narratives using a story planner; (2)
the definition and compositting of panels in the format of comic books; (3) the interrac-
tion with the user. The prroposed system is embedded in a tablet computer, whhere
users are able to visualize narratives through comic panels and interact with certtain
objects that can affect the unfolding
u stories (Figure 2a).
Initially, the system gennerates an entire story plot and users can read it as a traadi-
tional comic book. Additio onally, some scenarios include interactive objects that can
be activated by users by tap pping on the tablet. When this happens, the logical conttext
of the story is modified at that
t specific point of the narrative according to the effeects
of the activated object. Thee intervention propagates to the next story events, and the
comic panels are updated to represent the new storyline. Users thus have a wayy to
interact with the story and observe
o the consequences of their actions in real time.
Tablet computers are th he most suitable platform to support the visualizationn of
comics in a digital format, because
b they come closer to real comic books than deskktop
computers. However, the liimited processing power of tablets may not be capablee of
running a complex plannin ng algorithm in real time. Consequently, the proposed ssys-
tem is based on a client-seerver architecture (Figure 2b), where the server hosts the
planner responsible for gen nerating the stories, and the client contains the visualizattion
interface that presents the narrative
n in the format of comics.
Plot Manager
Story Planner
Comics Manager
Actor
Actor Actor
Actor
Actors Locations
In the proposed architeccture, the Story Planner consists of a heuristic search pllan-
ner that performs a forward d search in the space of world states to find a path from
ma
given state to a goal state based
b on the Story Context. The Story Context defines the
o authorial goals, and a set of planning operators using the
characters, locations, a set of
STRIPS formalism [24]. On O the client side, the Plot Manager is responsible for re-
questing new plans for thee remote Story Planner through a TCP/IP connection. T The
Plot Manager sends the cu urrent state of the story to the Story Planner, and receiives
back the sequence of story events to achieve an authorial goal. Then the Plot Manaager
sends the story events to the Comics Manager, which uses its graphical resources
(Actors and Locations) to generate comic panels to represent the events in the format
of a comic book and interacts with the user.
5 Story Planner
Interactive storytelling systems can follow three basic approaches: plot-based [10],
character-based [11], or a hybrid approach [12]. In our system we adopted a plot-
based approach, where the story plot is automatically built and updated in real time by
a planning algorithm. The Story Planner is based on a standard Heuristic Search Plan-
ning (HSP) [13] algorithm that performs a forward search in the space of world states
using a weighted A* algorithm [14] with inadmissible heuristic values. The planner
solves STRIPS-like planning problems with ground operators. The Story Context
contains the definition of the planning problem, which includes a set of propositions P
used to describe the states and operators, a set of planning operators O, the initial state
I ⊆ P, and a hierarchy of authorial goals G ⊆ P. Each operator o ∈ O has a list of
preconditions and a list of effects.
The initial state is represented by a set of propositions describing the starting point
of the story. It includes the definition of the characters, their current situation, and a
description of the world. Examples of propositions for the initial state could be: cha-
racter(emma) for “Emma is a character”; at(emma, house) for “Emma is at House”;
healthy(emma) for “Emma is healthy”; and path(house, hospital) for “There is a path
connecting the House to the Hospital”.
Goal states are also represented by sets of propositions, and are used by the comic
writer to guide the development of the story towards an admissible outcome. The
planner adopts a hierarchy of authorial goals, in the sense that if a higher goal cannot
be achieved, the planner tries its successor. The planner can fail to achieve a desired
goal either if there is no valid sequence of actions that leads from the initial state to
the goal state; or if the prescribed time limit for searching for a solution is exceeded.
In both cases, the planner tries to achieve the next successor goal from the authorial
goal hierarchy.
The planning operators correspond to the events that may occur in narratives. They
are represented using a STRIPS-like formalism. Each operator has: (1) a list of argu-
ments, indicating the roles played by the characters involved in the event, and other
features of the corresponding actions; (2) a list of preconditions, specifying facts that
must hold prior to the execution of the event; and (3) a list of effects, specifying facts
that hold immediately after the execution of the event. An example of event is “a
zombie attacking a victim” (where ∧ means AND and ¬ means NOT):
In this case, the operator has 3 arguments: two characters (CH1 and CH2) and a
place PL. The preconditions for the execution of this event indicate that CH1 must be
a zombie, CH2 cannot be a zombie, and both CH1 and CH2 must be at the place PL.
The effects of this event indicate that CH2 is not healthy and is infected.
Each planning operator is also associated with an action template that describes the
story event in terms of actions. For example, if Get(Emma, Toy, House) is an instance
of the planning operator Get(CH, OB, PL), the event will be decomposed into 5 ac-
tions: (1) Emma looks at the object; (2) Emma says “Wow! It’s a Toy!”; (3) Emma is
happy; (4) Emma crouches; and (5) Emma gets the Toy. The use of the action tem-
plates allows the planner to produce more detailed stories without sacrificing its per-
formance. However, it requires an addition authorial effort, since it is up to the author
to specify a detailed vision of how the event occurs.
The client-server architecture of the system allows several clients to be connected
with the planning server simultaneously. However, a large number of clients request-
ing new story plots and the server having to run several instances of the planning
algorithm may compromise the real-time performance of the system. To remedy this
problem, the story planner stores all computed story plots in a database of plans in-
dexed by the initial state and the goal state. When a client requests a new story plot,
the server checks the database of plans. If the requested initial state and goal state are
found, the server returns the corresponding precomputed plan; only if they are not
found, the server runs a new instance of the planning algorithm.
The proposed process for the visualization of interactive comics is divided in two
phases. The panel definition phase comprises the process of assigning the story events
to their corresponding panels, computing the size required for each panel, and defin-
ing the layout of each page. The panel compositing phase includes the process
of rendering the content of the panels (background, characters, objects, and word
balloons).
Panels are used to present a single moment frozen in time. Letting i refer to time, a
panel Pi represents a discrete instant ti (Figure 3). Besides the association with time,
the specification of panel Pi comprises a specific location Li and a set of events Ei:
, , , , , , , , , , , . An event e is an instance of a planning
operator (e.g. Get(Emma, Toy, House)) or of a simple action (e.g. Talk(Emma, “Wow!
It’s a Toy”), which corresponds to a speech balloon). Events are always sequential in
time (i.e. the story planner does not generate parallel events), but this sequence is
compressed in the discrete instant of time ti represented by the panel Pi. The story
planner generates a long sequence of events to be assigned to panels. Furthermore, the
nature and emotional intensity of the events attributed to a panel determine its size
and shape.
We establish the following rules to decide whether or not a new event en can be
grouped with its preceding event ep in a panel P, without breaking the continuity of
time and space:
1. If en and ep are both speeches of the same character, or different characters that are
at the same place, and the number of speeches already added to the panel P is
smaller than α (maximum number of speeches supported by a single panel), then
the event en can be assigned to the panel P.
2. If en is a speech and ep contains an action performed by the speaking character, then
the event en can be assigned to the panel P.
3. If ep and en are the same event (which is not a speech) performed by two different
characters at the same place, then the event en can be assigned to the panel P.
4. Otherwise, en is assigned to a new panel.
The next step is the process of creating the pages that will support the panels. A
page is composed of a sequence of panels with varying size and location, which
present the story events to the reader. The size of a panel is generally proportional to
the amount of narrative content presented, and its position is relative to the chronolog-
ical order of the events. In order to dynamically calculate the size of the panels, we
propose a method to estimate the importance of a panel based on weights associated
with the events and the location where the events take place.
Firstly, each class of event (e.g. Go, Talk, LookAt) and the locations where the
events can happen are associated with a weight based on their importance to the
narrative. For example, a Go event (where a character goes from one place to another)
may have less importance to the narrative than an Attack event (where a character is
attacked and infected by a zombie). The same idea is valid to the locations where the
events happen, some places are more important than others. The story author assigns
weights to the events and locations. These assignments can be done by a single nu-
merical value or a conditional expression (e.g. a Go event may have its weight in-
creased if certain specific events occur at time ti represented by panel Pi). Therefore
the weights are calculated by a function that depends on Pi. Weights are also calcu-
lated for each row and, finally, for the whole page. We propose the following equa-
tions to calculate the weights of a panel Pi, a row Rk, and a page Fj:
,
the number of rows in a page; β is the maximum value allowed for a row; and γ is the
maximum number of rows allowed in a page.
The algorithm that calculates the size of the panels and the layout of the pages
starts by iterating through the panels and assigning them to a page and a row accord-
ing to their chronological order. When the weight of a row (sum of the row panels’
weights) reaches β (maximum weight allowed to a row), the panels begin to be as-
signed to next row. When the number of rows reaches the maximum number of rows
per page (γ), the panels begin to be assigned to the first row of the next page. The
algorithm ends when all panels are assigned to a page and a row. We must notice that
the parameters β and γ determine the general aspect of the comic book page. In our
prototype we assume β=6, and γ=3.
In the next step, the actual size of each panel is calculated according to its weight
and position in the page. The width di of a panel Pi in a row Rk and the height hi of a
row Rk in a page are given by:
, ,
where D is the horizontal size of the page and H is the vertical size of the page in
pixels, and gutter is the space between panels (Figure 3a). Figure 3b illustrates the
process of the panel definition phase. First, the story events {A,B,C,G,I,J,M,N,Q} are
grouped in panels {P1, P2, …, P6} and their respective weights are calculated. Then,
the panels are chronologically assigned to their corresponding rows {R1, R2, R3} and
pages {F1}, considering the maximum weight supported by a row β=6. The weights
of the rows and pages are also calculated. Finally, the actual size of the panels is cal-
culated and the layout of the page is defined. In this example, the page size is D = 800
and H = 1000 pixels, the width of the panel P1 is d1(P1, R1) = 800x3/5 = 480 and its
height is h1(R1, F1) = 1000x5/16 = 312 without discounting the gutters.,
480
D
P1w=3 P2w=2
312
R1w=5
gutter A,B C
di
Rk P3w=5 R2w=5
Pi hk H G,I,J
P 4w P5w P6w
=2 =2 =2 R3w=6
Panels M N Q
P1 P2 Pi
discrete F1w=16
t1 t2 ti time
Fig. 3. (a) Elements of a comic book page of size D x H, where the panel Pi is within row Rk
and has a size of di x hk. (b) Example where the story events {A,B,C,G,I,J,M,N,Q} are grouped
in panels {P1,…,P6}and row weights Rkw are calculated. The subscript w stands for weight.
front of the speaking character have a higher weight associated with them. If a region
overlaps an occupied candidate region of another character that speaks first, the re-
gion is marked as unavailable (C2R1). The same happens when the candidate region
overlaps another character or interactive object. In the second step, the selected re-
gions are expanded (EC1R2, EC2R3) to allow the arrangement of the word balloons
according to the reading order of comics, where word balloons must be placed in a
left-right and top-down order. Finally, in step 3, the word balloons are generated and
placed in their corresponding regions.
C1 C2
EC1R2
C1R2
C1 C2
C2R3
EC2R3
After placing the characters, interactive objects and word balloons, the size of the
image frame being generated is adjusted to match the actual size of its respective
panel. Panels that don’t include changes of the emotional state of the characters are
framed so that all the characters, interactive objects and word balloons stay visible
and centered in the panel. Panels that include modification of the emotional state of
the characters are framed starting from the character’s hips, which emphasizes the
emotional expressions of the character’s face.
7 User Interaction
The proposed method for user interaction is based on interactive objects. Through this
device users are allowed to interact with the narrative by changing the content of the
panels, and, consequently, altering the logical context of the story. Such interventions
may force the characters to take other actions and change the course of the narrative.
Each interactive object includes a list of effects consisting of facts to be added or re-
moved from the current state of the world when the object is activated. The story events
generated by the planner are associated with a description of the current state of the
world when the event occurred during the planning process. When an interactive object
is activated by the user, the state of the world at that specific point of the narrative is
8 Application and
d Evaluation
In the main storyline, one of the characters is attacked and infected by a zombie and
the family tries to get an antidote to save the victim’s life. The story takes place in
three main locations: the family house, a dangerous park, and a hospital. Users can
interact with the story through the following interactive objects: (1) house door -
opening the door of the family house allows a zombie to enter in the house and possi-
bly attack someone; (2) old trunk - opening an old trunk in the family house reveals a
toy that can entertain the children; (3) zombie at the park - pushing a zombie in the
park makes the zombie fall and die; (4) hospital main door - closing the main door of
the hospital prevents the entrance/exit of the characters in/from the hospital through
the main door, while opening the door allows their entrance/exit; (5) hospital back
door - closing the main back door of the hospital prevents the entrance/exit of the
characters in/from the hospital through the back door, while opening the door allows
the entrance/exit; and (6) antidote bottle - pushing the antidote bottle makes it fall and
spill the antidote. The interactive objects can have different effects in the story de-
pending on which part of the narrative is occurring when they are activated.
The prototype application is able to generate a considerable number of diversified
stories. In the more conventional stories, Emma is attacked by a zombie in the park
and then saved by her father who finds an antidote in the hospital; in stories with a not
so happy ending, John does not find the antidote and decides to kill his daughter in
order to protect the others; and in stories with an even darker outcome, John gets
stuck in the hospital and cannot return home with the antidote, the daughter turns into
a zombie, kills the rest of the family, and John commits suicide in the hospital.
In order to evaluate our system, we have conducted a user evaluation with 18 high
school students, 15 male and 3 female, aged 16-17. Six of them had previous expe-
riences with interactive storytelling systems. Twelve of them play video games at least
weekly. Sixteen of them like comic books, of which four read comics at least weekly.
We asked participants to use both our interactive storytelling system (S) and the
mobile interactive comic book The Avengers: Iron Man - Mark VII [34] (M) for An-
droid (Figure 6). We aimed to compare the proposed system with the “state of the art”
of interactivity in comics. In order to avoid the focus on the appealing art quality of
Marvel’s system, we asked participants to concentrate their attention mainly on the
application flow, user interaction, and the way the story is presented. And to reduce
learning effects, half of the participants used S first, and the other half used M first.
On average, each session of S lasted 12.3 minutes (σ=2.4) and included 5.8 interac-
tions (σ=2.2). In M, each session lasted 7.5 minutes (σ=1.1).
After using each version, the participants filled out a questionnaire with 25 questions
derived from the IRIS Evaluation Toolkit [27][28]. We evaluated the system usability,
the correspondence of system capabilities with user expectations (user satisfaction), the
interaction effectiveness and the users’ experience (curiosity, flow and enjoyment).
Each statement was followed by a seven-point Likert scale ranging from “strongly dis-
agree” (-3) through “neutral” (0) to “strongly agree” (+3). After having interacted with
both systems, the participants were interviewed about their experience.
(a) Our Interactive Comics System (b) The Avengers: Iron Man - Mark VII
Fig. 6. Visual comparison of the
t evaluated systems. Our system presents several panels onn the
screen (a), while Marvel’s systtem presents only one panel per screen (b).
Figure 7 summarizes thee results of the questionnaires. Our interactive comics ssys-
tem clearly obtained betterr results in the evaluated topics (system usability, interrac-
tion effectiveness, user sattisfaction, and user experience). Marvel's The Avengeers:
Iron Man - Mark VII obtaiined very low interaction effectiveness, mainly becausse it
has a linear storyline and user interactions don’t have any real effect in the story pplot.
As far as the interviews are
a concerned, sixteen of the participants stated that tthey
preferred our interactive co
omics system. They declared that the layout of our systtem
is more similar to real commic books, and the possibility of changing the story is vvery
attractive and innovative. According
A to them, Marvel’s system is more similar tto a
game than to a comic boo ok. Following are some quotes from users regarding thheir
experience while using our system:
“The experience was amazzing! It was really cool to see the panels and the sttory
changing according to my actions.”
a (P1)
“The ability to interfere in the
t story was very exciting. I felt very curious and interrest-
ed in finding out everything g that my actions could do, and what else might happenn in
the story.” (P2)
“I really enjoyed the possib bility of changing the story by touching the objects in the
panels.” (P3)
Interaactive Comics (S) The Avengers: Iron Man - Mark VII (M)
4,00
3,00
2,00
1,00
0,00
Usability Effectiveness Satisfaction Experience
-1,00
-2,00
-3,00
Fig. 7. Average and standard deviation of questionnaire topics for our Interactive Comics sys-
tem and Marvel's The Avengerrs: Iron Man - Mark VII system
9 Conclusion
References
1. Kurlander, D., Skelly, T., Salesin, D.: Comic Chat. In: Proceedings of the 23rd Annual
Conference on Computer Graphics and Interactive Techniques, pp. 225–236 (1996)
2. Sumi, Y., Sakamoto, R., Nakao, K., Mase, K.: ComicDiary: Representing individual expe-
riences in a comics style. In: Borriello, G., Holmquist, L.E. (eds.) UbiComp 2002. LNCS,
vol. 2498, pp. 16–32. Springer, Heidelberg (2002)
3. Shamir, A., Rubinstein, M., Levinboim, T.: Generating Comics from 3D Interactive Com-
puter Graphics. IEEE Computer Graphics and Applications 26(3), 53–61 (2006)
4. Shuda, T., Thawonmas, R.: Frame Selection for Automatic Comic Generation from Game
Log. In: Stevens, S.M., Saldamarco, S.J. (eds.) ICEC 2008. LNCS, vol. 5309, pp. 179–184.
Springer, Heidelberg (2008)
5. Chan, C.J., Thawonmas, R., Chen, K.T.: Automatic Storytelling in Comics: A Case Study
on World of Warcraft. In: Proceedings of the 27th International Conference on Human fac-
tors in Computing Systems (CHI 2009), pp. 3589–3594 (2009)
6. Pizzi, D., Lugrin, J., Whittaker, A., Cavazza, M.: Automatic Generation of Game Level
Solutions as Storyboards. IEEE Transactions on Computational Intelligence and AI in
Games 2(3), 149–161 (2010)
7. Alves, T., Mcmichael, A., Simões, A., Vala, M., Paiva, A., Aylett, R.: Comics2D: Describ-
ing and Creating Comics from Story-Based Applications with Autonomous Characters. In:
Proceedings of the International Conference on Computer Animation and Social Agents,
Belgium (2007)
8. Alves, T., Simões, A., Figueiredo, R., Vala, M., Paiva, A., Aylett, R.: So tell me what hap-
pened: Turning agent-based interactive drama into comics. In: Proceedings of the 7th In-
ternational Conference on Autonomous Agents and Multiagent Systems, Portugal,
pp. 1269–1272 (2008)
9. Aylett, R.S., Louchart, S., Dias, J., Paiva, A., Vala, M.: FearNot! - an experiment in emer-
gent narrative. In: Panayiotopoulos, T., Gratch, J., Aylett, R.S., Ballin, D., Olivier, P., Rist,
T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 305–316. Springer, Heidelberg (2005)
10. Spierling, U., Braun, N., Iurgel, I., Grasbon, D.: Setting the scene: playing digital director
in interactive storytelling and creation. Computers & Graphics 26(1), 31–44 (2002)
11. Cavazza, M., Charles, F., Mead, S.: Character-based interactive storytelling. IEEE Intelli-
gent Systems, Special issue AI in Interactive Entertainment 17(4), 17–24 (2002)
12. Mateas, M.: Interactive Drama, Art, and Artificial Intelligence. Ph.D. Thesis. School of
Computer Science, Carnegie Mellon University, Pittsburgh, USA (2002)
13. Bonet, B., Geffner, H.: Planning as Heuristic Search. Artificial Intelligence Special Issue
on Heuristic Search 129(1), 5–33 (2001)
14. Pearl, J.: Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison-
Wesley (1985)
15. Ghallab, M., Nau, D., Traverso, P.: Automated Planning: Theory and Practice, 1st edn.
Morgan Kaufmann Publishers, Amsterdam (2004)
16. Witty, S.: Illustrative Text: Transformed Words and the Language of Comics. In: Interna-
tional Conference of Arts and Humanities, Honolulu, Hawaii (2007)
17. McCloud, S.: Understanding Comics. Kitchen Sink Press, Northampton (1993)
18. Eisner, W.: Comics & Sequential Art. Poorhouse Press, Tamarac (1985)
19. Goodbrey, D.M.: PoCom-UK-001 (2003), http://e-merl.com/pocom.htm
20. Goodbrey, D. M.: The Archivist. Available at: http://e-merl.com/archivist/, 2010
21. McCloud, S.: Reinventing Comics: How Imagination and Technology are Revolutionizing
an Art Form. William Morrow Paperbacks (2000)
22. Campbell, S.: Nawlz – Interactive Comic, http://www.nawlz.com/hq/about/
23. Andrews, D., Baber, C., Efremov, S., Komarov, M.: Creating and Using Interactive Narra-
tives: Reading and Writing Branching Comics. In: Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems (CHI 2012), pp. 1703–1712 (2012)
24. Fikes, R., Nilsson, N.: STRIPS: A new approach to the application of theorem proving to
problem solving. Artificial Intellligence 2, 189–208 (1971)
25. Chun, B.-K., Ryu, D.-S., Hwang, W.-I., Cho, H.-G.: An Automated Procedure for Word
Balloon Placement in Cinema Comics. In: Bebis, G., et al. (eds.) ISVC 2006. LNCS,
vol. 4292, pp. 576–585. Springer, Heidelberg (2006)
26. Riedl, M.O., Young, R.M.: From Linear Story Generation to Branching Story Graphs.
IEEE Computer Graphics and Applications 26(3) (2006)
27. Klimmt, C., Roth, C., Vermeulen, I., Vorderer, P.: The Empirical Assessment of The User
Experienc. In: Interactive Storytelling: Construct Validation Of Candidate Evaluation
Measures. Technical Report, Integrating Research in Interactive Storytelling - IRIS (2010)
28. Roth, C., Vorderer, P., Klimmt, C.: The Motivational Appeal of Interactive Storytelling:
Towards a Dimensional Model of the User Experience. In: Iurgel, I.A., Zagalo, N., Petta,
P. (eds.) ICIDS 2009. LNCS, vol. 5915, pp. 38–43. Springer, Heidelberg (2009)
29. O’Neil, D.: The DC Comics Guide to Writing Comics, 1st edn. Watson-Guptill (2001)
30. Shiga, J., Plotkin, A.: Meanwhile, http://zarfhome.com/meanwhile
31. Fenty, S., Houp, T., Taylor, L.: Webcomics: The Influence and Continuation of the Comix
Revolution. ImageTexT: Interdisciplinary Comics Studies, vol. 1. Univ. of Florida (2004)
32. Lima, E.S., Pozzer, C.T., Feijo, B., Ciarlini, A.E.M., Furtado, A.L.: Director of Photogra-
phy and Music Director for Interactive Storytelling. In: IX Brazilian Symposium on
Games and Digital Entertainment - SBGames 2010, Brazil, pp. 122–131 (2010)
33. Lima, E.S., Feijo, B., Furtado, A.L., Pozzer, C.T., Ciarlini, A.E.M.: Automatic Video Edit-
ing For Video-Based Interactive Storytelling. In: Proceedings of the 2012 IEEE Interna-
tional Conference on Multimedia and Expo (ICME 2012), Melbourne, pp. 806–811 (2012)
34. Loudcrow Interactive. Marvel’s The Avengers: Iron Man – Mark VII,
http://loudcrow.com/marvels-the-avengers-iron-man-mark-vii
1 Introduction
The well-known game of tug of war, where two teams, each holding one end of
a rope, attempt to pull the other team over a predesignated point, is a prime
example of a game where players are physically engaged with their whole body.
What is more, force feedback plays a crucial role. Here, the rope serves as a
medium that delivers the force feedback generated by each of the teams over a
short distance. In essence, we can speak of a spatially distributed game, where
players are in physical contact with each other through the shared medium
of the rope. Though the spatial distribution of a classic rope pulling game is
limited by the length of the rope, efforts have been made to enlarge the spatial
distribution of the rope pulling game through the use of Internet communication
[1][2]. Harfield et al. [1] describe a distributed rope pulling system that was
designed to enable children in distant locations to play a game of tug of war
over the Internet. Similarly Christian et al. [2] present a tug of war game in
which the player plays against a virtual character. The strength of the virtual
character’s pull is determined by the player’s own strength.
In distributed rope pulling systems, remote touch [3], in the form of active
force feedback about the actions of the other player, could play an important
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 246–257, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
2.1 Distributed Exertion Interfaces
In distributed entertainment, players play with each other at a distance with
the use of interactive technologies, giving the feeling as if the players are co-
located. These entertainment technologies can vary in their goal, style of play,
and the used technology [8]. Work by Mueller et al. [9] [10][11] provides a number
of examples of systems that enable players to engage in physically demanding
exertion games with other players at a distance. Mueller et al. [9] created a com-
petitive break-out-style game where players had to kick or throw a ball against
a wall with a video projection of another player. Compared to an alternative
played with a standard computer keyboard, the physical game showed an in-
crease in social bonding and a perceived increase in quality of the sound and
video. An example of a similar game is Mueller et al.’s [10] “Airhockey over a
distance”. Here, players play on an augmented airhockey table that incorporates
a projection screen of a player in another location, and “puck-launchers” that
can launch physical pucks, creating the illusion that the physical object crossed
the space between the players. An informal evaluation indicated the potential
for enhancing players’ feelings of connectedness. In Mueller et al.’s [11] shadow
boxing, two physically separated players each stand in front of a screen showing
a projection of the other player’s silhouette. Players can punch, kick or use their
whole body to hit the projection of the other player and score points. The aim
of the system was to demonstrate the possibilities of creating sports-like social
games that can be played at a distance. In a similar fashion, Harfield et al. [1]
developed a tug-of-war exertion game that can be played by physically sepa-
rated players. The system is capable of measuring forces of up to 250 kg. Two
teams of physically separated players are tasked with generating as much force
as possible. The position of the rope is visualized on a screen and adjusted in
accordance with the amount of force generated by each team. Though no formal
evaluation was conducted the authors conjectured that the system helped young
students collaborate with students from another continent. Finally, Yao et al.
[12] describe a number of rope-based exertion games. For example, Multi-jump
is a distributed rope jumping system where one player twirls a real rope. The
motion of the rope is measured, and is displayed on a projection screen in front
of another player in a distant location, who has to time his/her jumps in ac-
cordance with the visualization. Other games include a collaborative kite flying
game, a competitive horse riding game, and a wood cutting game. In informal
evaluations, the authors found that the rope-based games show the potential to
stimulate social experiences, by providing users with the feeling of being engaged
in a co-located activity.
Each of the systems described above incorporate some tangible element which
allows players to physically interact with each other at a distance. However, none
of these systems incorporate any haptic feedback technology. Research into hap-
tics and telepresence shows benefits of using haptic feedback in interactions be-
tween users at a distance. The next section provides a brief overview of literature
on haptics and remote touch.
was found to increase feelings of social presence in the virtual environment. Fi-
nally, Sallnäs [7] conducted a study in which participants passed objects to each
other in a virtual environment, using two force feedback joysticks. Compared
to the condition where no force feedback was provided, force feedback improved
perceived virtual presence, perceived social presence and perceived performance.
It is worthwhile to note here that in the studies described above, all of the
tasks are collaborative. Research into social touch between co-located individuals
suggests that this may be a very important contextual factor. Camps et al. [20]
found that touch in competitive settings reduces helping behavior, whereas touch
in collaborative settings enhances helping behavior. Therefore, beneficial effects
of remote touch in a rope pulling game are most likely to occur in collaborative
settings. This was considered in the design of our distributed rope pulling system
described in the next section.
Game. We designed a game in which players have to catch eggs falling from the
top of the screen with a basket. The game was intentionally designed to be very
minimalistic and only shows the basket, falling eggs and score. The basket also
slants slightly when it is pulled to one side as to provide players with additional
visual feedback about the movement of the basket.
4 User Studies
The final prototype was used in two studies. First, a pilot study that served
to assess the level of enjoyment of the egg-catching game played with the rope
pulling system. Second, the main study was conducted to investigate the effect
of remote touch on feelings of social presence and enjoyment within the collab-
orative egg-catching game.
large scale distributed rope pulling installation (see also [1]). All children played
the game once. Two rope pulling devices were placed next to each other on a
table, and a computer monitor displayed the game.
After playing the game we ask each individual child three questions: 1)“What
did you like about the installation?”, 2)“What did you not like about the instal-
lation?” and based on the again-again table method [21]: 3) “If there was more
time, would you like to play this game again?” with three possible answers: yes,
maybe or no. The children were also asked whether they had any comments on
the game and system in general.
From this informal evaluation we gathered that the children generally liked
the rope pulling system and egg-catching game. Eleven children had no nega-
tive comments, and all except one child liked at least one aspect of the game.
The most frequent positive comment was the aspect of working together (men-
tioned 5 times). The social aspect of trust in the other player and arguing with
the other player were also mentioned as positive. The act of physically pulling
the rope was mentioned as a positive aspect of the system (4 times). Finally, the
majority of children stated that they would have liked to play the game again
(14 yes, 1 maybe, 1 not). Other observations included that some children tried
to take the rope from the other player and play the game by themselves. Others
tested the limits of the system and most were very curious about how the system
worked.
The pilot study indicated that the system definitely has the potential to be
a fun and interesting way for children to interact with each other in a physical
game environment. After observing the children playing with the rope pulling
system and hearing their comments we were confident that the system would be
suitable for use in the main study.
The main study was conducted in order to study the effect of remote touch, in
the form of active force feedback about another player’s actions, on feelings of
social presence and enjoyment of a collaborative game. To this end we formulated
two hypotheses, namely:
H1: Remote touch in the form of active force feedback about another
player’s actions will increase feelings of social presence in a collaborative
game;
H2: Remote touch in the form of active force feedback about another
player’s actions will increase enjoyment of a collaborative game.
Fig. 3. Screenshot of the two different conditions of the game. The test condition is
shown on the left, the control condition is shown on the right.
free to communicate as they liked using the headset and video call. Next, par-
ticipants were asked to play the game for two minutes. After the play session,
participants completed the social presence questionnaire, additional questions,
and demographic questions on the computer that previously displayed the video
call. Finally, participants were debriefed about the goals of the study.
Results. After correcting for the reversed questions, the items on the social
presence scale showed acceptable to good internal-consistency (Cronbach’s α),
as can be seen in Table 1. Note, that two items were removed from the Perceived
Emotional Interdependence scale.
To test the hypotheses, we used a one-tailed independent samples t-test.
Scores on the six social presence items and the scores on the additional questions
were compared between the control group and the test group. The results for
the social presence items are shown in Table 1. As can be observed from Table
1 four of the six items of the social presence questionnaire showed a significant
difference (p < .05) between the control group and the test group, and two
items showed a marginally significant difference. The test group showed signifi-
cantly higher scores than the control group for co-presence, perceived affective
understanding, perceived emotional interdependence and perceived behavioral
interdependence (all at p < .05). In addition, the test group showed higher
scores for perceived message understanding and attention allocation, although
the difference was only marginally significant (at .05 < p < .1). These findings
support H1.
Table 1. Social presence questionnaire items. 5-point Likert scale, 1 = strongly dis-
agree, 5 = strongly agree.
Table 2 shows the seven additional questions about game experience. Four
of the questions show a significant difference between the groups. For questions
3, 5 and 6 the test group showed significantly higher scores than the control
group. Again, these findings support H1. Question 4, which was about enjoyment,
showed a significant difference between the groups. Participants in the test group
indicated more strongly than participants in the control group, that they enjoyed
playing the game with the rope pulling interface more than they would have
enjoyed playing the game with a more traditional controller. For the other three
questions (questions 1, 2, and 7) about enjoyment, scores in the test group were
higher than in the control group, but the difference was not significant. With
this H2 is not supported.
The findings from the main study support H1. Indeed, results from the social
presence questionnaire showed that the test group who received remote touch
force feedback, had stronger feelings of social presence towards their game part-
ner than participants in the control group who received the elastic band force
feedback. This shows that the social aspect of the remote touch force feedback
added to the feelings of social presence, more than did the general type of haptic
feedback. This statement is supported by the additional questions in which par-
ticipants in the test group indicated feeling a stronger connection to their game
partner than did participants in the control group. In addition, participants in
the test group indicated strongly that they had the feeling that the force feed-
back was generated by their partner. This indicates that the remote touch force
feedback was indeed perceived as a form of physical contact between the two
players. Furthermore, participants in the test group found the game to be more
like real life rope pulling than participants in the control group. However, it is
possible that the slight difference between both versions of the egg-catching game
(i.e. participants in the test group actually pulled ‘against’ each other), explains
this difference. This can be considered a limitation of the approach of our study.
In our current approach we can not infer a difference between the role and effect
of the remote touch force feedback and the effect of sharing control over a single
entity in the game (i.e. the basket) versus only sharing an overall goal but with
each player controlling an individual entity in the game (i.e. separate baskets,
but a cumulative score). However, research into telepresence [4][5][6][7] shows
clear effects of remote touch force feedback on feelings of social presence in dif-
ferent contexts with and without manipulation of a common object. Therefore it
seems most likely that the effects found in this study are due to the differences
in haptic feedback.
Overall the rope pulling system was considered to be very enjoyable by players
in both groups. However, no significant differences were found for any of the ad-
ditional questions dealing with player enjoyment. These findings do not support
H2. Though for all the enjoyment questions (see Table 2 questions 1, 2, 4, and
7) the test group did have higher mean scores. A potential explanation is that,
although the remote touch did make the game more enjoyable, the base level of
enjoyability was already high for both variations of the game, reducing potential
differences on the 5-point Likert scale. It also has to be noted here that this is
only a first time use measure. It would be interesting for future research to look
at long term effects and different types of games.
The findings from our study have a number of implications for research, as
well as the entertainment industry. Our findings support the notion that adding
remote touch haptic feedback, in our case force feedback, can add to a player’s
feelings of social presence of another player more so than general haptic feedback.
The inclusion of remote touch haptic feedback into games and game controllers
might be a fruitful direction for providing entertaining, social experiences. Exer-
tion type games, played at a distance, might benefit from remote touch in that
it could provide additional realism (i.e. the physical aspect of playing sports to-
gether). Moreover, a full-scale installation, that would ideally feature high torque
motors that could handle multiple children pulling at a larger rope, would seem
a viable approach to connecting children with their peers across cities, countries
and continents.
6 Conclusions
In this paper we investigated whether remote touch in the form of force feed-
back from another player’s actions could enhance feelings of social presence and
enjoyment of a collaborative distributed rope pulling game. To this end, in an
iterative fashion, we designed, created and tested a desktop-sized distributed
rope pulling game. We created two variations of a collaborative egg-catching
game where players had to catch eggs falling from the top of the screen to in-
crease their cumulative score. In one version of the game players each had an
individual basket to catch eggs and received elastic band force feedback, while in
the other version players controlled a shared basket and received remote touch
force feedback of the other player’s actions. The main study with 40 participants
playing the game in dyads showed that players in the remote touch condition,
had significantly stronger feelings of social presence towards the other player,
than did players in the control condition. Players in both groups found the game
to be very entertaining, but this was slightly more so the case for players in the
remote touch group. However, ratings for enjoyment did not differ significantly
between the remote touch and control groups.
Overall, our results point to the importance of remote touch haptic feedback
for enhancing feelings of social presence in collaborative games.
References
1. Harfield, A., Jormanainen, I., Shujau, H.: First steps in distributed tangible tech-
nologies: a virtual tug of war. In: Proceedings of the 8th International Conference
on Interaction Design and Children, IDC 2009, pp. 178–181. ACM (2009)
2. Christian, V., Smetschka, J., Pötzelberger, W., Lindinger, C., Praxmarer, R.,
Stadler, W.: Ars Electronica Futurelab Tug of War,
http://www.aec.at/futurelab/en/referenzen/alle-jahre/2000/tug-of-war/
(June 6, 2013) (retrieved)
3. Haans, A., IJsselsteijn, W.: Mediated social touch: a review of current research and
future directions. Virtual Reality 9(2-3), 149–159 (2006)
4. Chellali, A., Dumas, C., Milleville-Pennel, I.: Influences of haptic communication
on a shared manual task. Interacting with Computers 23(4), 317–328 (2011)
5. Sallnäs, E.L., Rassmus-Gröhn, K., Sjöström, C.: Supporting presence in collabo-
rative environments by haptic force feedback. ACM Trans. Comput.-Hum. Inter-
act. 7(4), 461–476 (2000)
6. Giannopoulos, E., Eslava, V., Oyarzabal, M., Hierro, T., González, L., Ferre, M.,
Slater, M.: The effect of haptic feedback on basic social interaction within shared
virtual environments. In: Ferre, M. (ed.) EuroHaptics 2008. LNCS, vol. 5024, pp.
301–307. Springer, Heidelberg (2008)
7. Sallnäs, E.-L.: Haptic feedback increases perceived social presence. In: Kappers,
A.M.L., van Erp, J.B.F., Bergmann Tiest, W.M., van der Helm, F.C.T. (eds.)
EuroHaptics 2010, Part II. LNCS, vol. 6192, pp. 178–185. Springer, Heidelberg
(2010)
8. Moreno, A., van Delden, R., Poppe, R., Reidsma, D.: Socially Aware Interactive
Playgrounds. IEEE Pervasive Computing 12(3), 40–47 (2013)
9. Mueller, F.F., Agamanolis, S., Picard, R.: Exertion interfaces: sports over a dis-
tance for social bonding and fun. In: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, CHI 2003, pp. 561–568. ACM (2003)
10. Mueller, F.F., Cole, L., O’Brien, S., Walmink, W.: Airhockey over a distance: a
networked physical game to support social interactions. In: Proceedings of the 2006
ACM SIGCHI International Conference on Advances in Computer Entertainment
Technology, ACE 2006. ACM (2006)
11. Mueller, F.F., Agamanolis, S., Gibbs, M.R., Vetere, F.: Remote impact: shadow-
boxing over a distance. In: CHI 2008 Extended Abstracts on Human Factors in
Computing Systems, CHI EA 2008, pp. 2291–2296. ACM (2008)
12. Yao, L., Dasgupta, S., Cheng, N., Spingarn-Koff, J., Rudakevych, O., Ishii, H.:
Rope revolution: tangible and gestural rope interface for collaborative play. In:
Proceedings of the 8th International Conference on Advances in Computer Enter-
tainment Technology, ACE 2011, pp. 11:1–11:8. ACM (2011)
13. Huisman, G., Darriba Frederiks, A., Van Dijk, E., Heylen, D., Kröse, B.: The
TaSST: Tactile Sleeve for Social Touch. In: Proceedings of the IEEE World Haptics
Conference 2013, WHC 2013, pp. 211–216. IEEE (2013)
14. Park, Y.W., Nam, T.J.: Poke: a new way of sharing emotional touches during phone
conversations. In: CHI 2013 Extended Abstracts on Human Factors in Computing
Systems, CHI EA 2013, pp. 2859–2860. ACM (2013)
15. Brave, S., Dahley, A.: intouch: a medium for haptic interpersonal communication.
In: CHI 1997 Extended Abstracts on Human Factors in Computing Systems , CHI
EA 1997, pp. 363–364. ACM (1997)
16. Haans, A., de Nood, C., IJsselsteijn, W.A.: Investigating response similarities be-
tween real and mediated social touch: a first test. In: CHI 2007 Extended Abstracts
on Human Factors in Computing Systems , EA 2007, pp. 2405–2410. ACM (2007)
17. Haans, A., IJsselsteijn, W.A.: The Virtual Midas Touch: Helping Behavior After a
Mediated Social Touch. IEEE Transactions on Haptics 2(3), 136–140 (2009)
18. Bailenson, J., Yee, N., Brave, S., Merget, D., Koslow, D.: Virtual interpersonal
touch: Expressing and recognizing emotions through haptic devices. Human-
Computer Interaction 22(3), 325–353 (2007)
19. Huisman, G., Darriba Frederiks, A.: Towards tactile expressions of emotion through
mediated touch. In: CHI 2013 Extended Abstracts on Human Factors in Computing
Systems , CHI EA 2013, pp. 1575–1580. ACM (2013)
20. Camps, J., Tuteleers, C., Stouten, J., Nelissen, J.: A situational touch: How touch
affects people’s decision behavior. Social Influence 8(4), 237–250 (2013)
21. Read, J.C., MacFarlane, S.: Using the fun toolkit and other survey methods to
gather opinions in child computer interaction. In: Proceedings of the 2006 Confer-
ence on Interaction Design and Children, IDC 2006, pp. 81–88. ACM (2006)
22. Harms, C., Biocca, F.: Internal consistency and reliability of the networked minds
measure of social presence. In: Alcaniz, M., Rey, B. (eds.) Proceedings of the 7th
Annual International Workshop on Presence (2004)
1 Introduction
The purpose of this project is to develop a system that allows users to act
like a tenor by moving their arms and lips but without real singing. Shaping
vowels with her mouth and getting corresponding multimodal feedback creates
a believable user experience. It is difficult to synthesize the human voice based
on purely visual analysis of the mouth’s shape. A possible solution is to restrict
the recognition and synthesis to trackable vowels and use additional cues like
arm gestures for other vowels and to generate pitch and volume of the singing
voice.
The origin of music, and thus singing as part of it, has been researched for
a long time. Charles Darwin advanced the theory that the human antecessors
learned musical notes and rhythms to charm the other sex. According to evolu-
tionary biologist Geoffrey Miller, music and dancing emerged from rituals sym-
bolizing combat and hunting [6]. A different approach arose from the observation
that every human culture has lullabies used by mothers to calm their children.
Due to anthropologist Dean Falk singing provided sort of a remote maintenance
for the helpless baby, as long as the mother stayed within hearing distance [3].
Singing apparently had an important impact in human evolution and cultural
development. Nevertheless, nowadays there is a strong tendency towards con-
sumption of externally generated music and few people are able to really expe-
rience the positive feeling of their own successful singing voice production. This
is the motivation of the project presented in this paper: to provide users with
a believable user experience of performing an aria. Although we know that it is
not possible to exactly simulate the performance of a professional opera singer
we aim to create an entertaining user experience with our work.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 258–269, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
Several projects studied the synthesis of sound with the mouth or with gestures.
De Silva et al present a face tracking mouth controller [11]. The application
example focused on a bioacoustics model of an avian syrinx that is simulated
by a computer and controlled with this interface. The image-based recognition
tracks the user’s nostrils and mouth shape and maps this to parameters of a
syrinx’s model. This model is used to generate sound. At NIME 2003 Lyons
et al presented a vision-based mouth interface that used facial action to control
musical sound. A head-worn camera tracks mouth height, width and aspect ratio.
These parameters are processed and used to control guitar effects, a keyboard
and look sequences [5]. Closely related to our approach is the ”Artificial Singing”
project, a device that controls a singing synthesizer with mouth movements
that are tracked by a web camera. The user’s mouth is tracked and recognized
mouth parameters like width, height, opening, rounding, etc. are mapped to a
synthesizers parameters like pitch, loudness, vibrato, etc [4]. Recently, Cheng
and Huang published an advanced mouth tracking approach that combines real-
time mouth tracking and 3D reconstruction [2]. The synthesis of singing is an
ambitious area of research with a long tradition and the human singing voice is a
most complex instrument to be synthesized. A good overview is presented in [10],
[12]. The creation and control of a 3D avatar has also been discussed in a number
of projects. FaceGen (www.facegen.com) is a prominent base technology used in
many AAA game productions. Many musical interfaces apply RGB-D cameras
like Kinect for controlling sound synthesis (e.g., [7]) and we also chose this device
for our purposes. While most projects focus on either body tracking or facial
tracking, we combine both tracking methods to give the user a better overall
experience. The body tracking used to control the movement of a virtual tenor
character and aims at giving the user the impression of ”being” that character.
Thus, body movement is an essential part of our performance and also used for
sound synthesis by controlling pitch and volume. Mouth gestures are used to
control the sound synthesis by simply shaping the desired vowel with the user’s
mouth.
3 System Overview
The installation consists of a 3x3 video wall with 46” monitors, one Microsoft
Kinect for full body skeleton tracking and a Primesense Carmine (or another
Kinect) sensor for facial tracking. Figure 2 provides an overview of the system.
The processing is done by two computers connected via Ethernet. One is respon-
sible for operating the hardware system for facial tracking and the synthesizer
modules while the other runs the main application with a software-based skele-
ton tracking system and the hardware sensors. The sensor for skeleton tracking
is placed in about two to three meters distance to the user. The facial sensor
is positioned and fixed hanging in the air in front of and above the user, look-
ing at the user in about a 20◦ C angle. This minimizes the interference of the
two depth cameras by reducing the overlapping region of both cameras. One
computer processes all tracking data and uses this data also to fully animate a
virtual 3D character on a theater stage. The character’s head can be adapted
towards the user’s face by optionally using an integrated face generation module.
To use this feature the user takes a picture of him/herself and the system creates
a corresponding textured 3D face model. The user can also choose to modify sev-
eral face model parameters to modify the final look of the 3D character. After
this face generation step the user starts the performance mode and positions
him/herself in front go the two depth cams (see fig 2). By moving the arms und
shaping vowels with the mouth, the user can not only control the tenor’s arms
and facial expression, but also produce a singing voice and thus feel as being in
an opera performance. The user’s goal during the performance is to reproduce
the original singing voice of an aria for a given background music as accurately
as possible. This is similar to the well-known Karaoke music game but in our
installation the user does not really sing with her own voice. With an increasing
positive rating of the user’s performance the 3D character’s face morphs from
the user’s face to a virtual Enrico Caruso, a famous tenor from the 20th century.
5 User Tracking
Once the user decided that she likes her avatar look she switches to ”performance
mode”. In this mode she is being tracked by two depth sensors (e.g. Microsoft
Kinect) for full body skeleton tracking for facial motion tracking. The tracking
quality of two sensors decreases significantly if the sensors are positioned with
a larger the overlap of the camera’s structured infra-red pattern. This problem
was solved by shielding the field of view of both sensors from each other as well
as possible by having the facial tracker hang in the air and only aim at the face
while the full body tracker is further away capturing the full picture. In addition
to solve the last bit of interference the ”Shake’n’Sense” [1] approach was applied.
We attached a small vibrating servo motor to one Kinect sensor and introduces
artificial motion blur through vibration and eliminates crosstalk.
The current prototype of our facial motion tracking system utilizes faceshift
(http://www.faceshift.com), a software originally used for marker-less facial mo-
tion capturing and 3D character face animation. Previous prototypes used a 2D
face tracking using OpenCV and the face tracking provided by the Microsofts
Kinect SDK 1.5+ but were too inaccurate for our purposes. The integration of
faceshift resulted in a much better performance. To get reliable data from the
faceshift tracking module the user has to create a profile that is efficiently used
during the performance. This profile has to be created in the software’s training
mode by capturing a few default facial expressions in order to create a suitable
3D representation of the user’s facial structure. Although an existing profile can
be used for new user’s the tracking performance may be less accurate without
an individual profile.
Fig. 4. Training of a user-specific profile for face tracking using face shift
After the calibration step is completed the user proceeds to ”performance mode”.
If the user’s face is recognized in this mode the camera image, the correspond-
ing 3D representation and all the facial parameters are captured and tracked.
During this mode a network server provides a streaming service for clients that
can connect via the network. The sound synthesis module described in section
6 connects to this service, streams the tracking data from a custom protocol
consisting of head pose information, blend shapes (also called coefficients), eye
gaze and additional manually set virtual markers. The coefficients and marker
positions are sent directly to the audio synthesizer where they are processed for
sound generation. Furthermore the head pose, blend shapes and eye gaze are
used to animate the avatar’s facial features in real time. The head pose is used
to rotate the neck bone and the eye gaze to rotate the specific eyes while the
blend shapes are used to change the look of the avatar’s face using the morph
capabilities of the FaceGen SDK. To make this work correctly with our avatar
customization we adjusted our tenor model to work well with both the facial
tracking data from faceshift and the fitting and morphing system of FaceGen.
This was done by adjusting the basic face shape to the base model used by
FaceGen, creating custom blend shapes that mimic the blend shapes of faceshift
and finally converting this into a suitable model base. To give the user feedback
on the quality of her performance we’ve also created functionality to morph be-
tween the user’s customized 3D head and a 3D head created from a real opera
singer’s photograph (e.g. Enrico Caruso). The better she performs singing an
pre-selected song like ”Ave Maria” the more she will turn into a virtual tenor
character like Caruso. This is done using the same base structure for both heads
and simply interpolating between the 3D mesh data of the two models.
To make it easier to calibrate the user’s body and face gestures we developed a
small Android app that functions as a control panel for the different calibration
modules. This makes it possible to send networked commands to start or stop
tracking, calibrate the neutral position and angle and whether to treat the head
position relative or absolute simply from or nearby the user’s place on the stage
and not directly in front of the laptops.
6 Sound Synthesis
The sound synthesis of this work is based on formants which we introduce in
this section.
6.1 Preface
The analysis technique used for transforming signals from the time domain to
the frequency domain to generate the illustrations for this paper is somewhat
inaccurate. This is due to the fact, that most illustrations are snapshots taken
while performing with the installation, and thus the transformations had to be
operated in real time. A sample is shown in Fig. 7 b): only the peaks of the ”hills”
represent contained sine waves, the remainder is to be seen as ”overshoot”. In Fig.
7 a) the corresponding correct analysis is shown schematically. Only contained
frequencies display a bar in the graph. Both figures represent an analysis of the
same signal and are to been seen equal in this paper.
6.2 Formants
A formant is a concentration of acoustical energy in a specific frequency range,
that means these frequency ranges are emphasized in relation to other frequency
ranges. The middle of this frequency range contains the most energy, the fre-
quencies to both sides have gradually decreasing energy levels. Talking about a
formant at 730 Hz refers to the formant with the mid-frequency of 730 Hz. Fig.
8 displays a signal in the frequency domain that features three formants, 8 a)
serves for better understanding of the allocation of the frequencies. The middle
of these formants is marked with the blue arrows (in this case at 730 Hz, 1090
Hz and 2440 Hz). The signal is based on the signal shown in Fig. 7. The relevant
filters used to generate the formants are outlined by the colored curves.
Formants are an essential part of the characteristic sound of an instrument
(there are other factors as well). Even though two sounds might have the same
fundamental frequency, they can sound differently. If a piano and a guitar play
the same note, they are still clearly distinguishable. In this sense, the human
voice is an instrument as well, and its different vowels can be distinguished be-
cause of their different formants.
6.3 Synthesis
The tenor’s voice is synthesized by using a vowel synthesis via formants. Max/MSP
is used for audio processing and calculations, while data is sent as OSC packages
from one program to another. The user determines the target fundamental pitch
of the singing with her arms (see chapter Mapping). Based on the target pitch of
the tenor’s voice, multiple sine waves are added up. These sine waves have pitches
that are integer multiples of the target pitch, that means if the target pitch is 110
Hz, the added sine waves are at 220 Hz, 330 Hz etc. In this synthesis, the upper
limit was defined at 12 kHz, as higher frequencies are not relevant for the charac-
teristics of the human voice. Next, the signal is modulated by several signals that
also depend on the fundamental pitch, but are randomized to a certain amount
to create a preferably non-artificial impression of the end result. This modulation
adds a vibrato effect, that means a periodical small change of the target pitch, and
slightly manipulates the amplitude of the signal to simulate a human singing voice,
assuming that a singer can’t produce a perfectly constant tone. Three band pass
filters are used to create the target vowel’s characteristic formants. These formants
have the property of being more or less independent of the generated fundamen-
tal pitch. Fig. 7 shows the unfiltered signal with a fundamental frequency of 110
Hz and it’s harmonic frequencies at 220 Hz, 330 Hz etc. In Fig. 8 this signal was
filtered by three bandpass filters (outlined by the colored curves) to ”cut out” the
characteristic formants and thus create the apparent envelope curve with three
summits (in this example at 730 Hz, 1090 Hz and 2440 Hz marked by the blue
arrows to form the vowel ”A”). The example in Fig. 10 a) shows the result of the
same process with a fundamental frequency of 220 Hz and it’s harmonic frequen-
cies at 440 Hz, 660 Hz etc. The resulting formants are the same as the one at a
fundamental frequency of 110 Hz, so the resulting vowel is still ”A”. This shows
that formants are more or less independent from the fundamental frequency. As an
evaluation of our synthesized voice, we separated the vowel ”A” from a recorded
aria in Fig. 10 b). The real singing has three apparent formants at the beginning of
the frequency spectrum that have similar middle frequencies to those that are cre-
ated in our system. The difference in sound level and the other amplified regions
form the singer’s personal timbre. The selected formant frequencies were taken
from [8]. Most other sources give only two formants, as two of them are sufficient
for the perception of a vowel. However, a third formant turned out to be useful to
make the result more human. The synthesis actually uses an additional fourth for-
mant at 2,7 kHz. That formant is both independent of the fundamental frequency
and the target vowel. It’s called ”singers’ formant” and is only visible in the fre-
quency spectrum of trained singers. This formant is essential when singing with
an orchestra and allows professional singers to be heard without further amplifi-
cation, as the peak frequency range of an orchestra is much lower than 2,7 kHz.
After the formant filtering process, several additional filters are used to achieve a
20th century gramophone like sound. In a final step, vowel-dependent amplitude
variations are reduced by normalizing and compressing the signal. Additionally,
an optional reverb effect can be added to simulate the acoustic of an opera.
7 Mapping
The user controls pitch and volume of the synthesized singing voice with her
arms. The volume is determined by the user’s arm stretch and is directly mapped
to the system’s gain control. The arm-height is used to calculate the target
pitch by using the change of arm-height over time. This method was selected
to make it easier and more enjoyable to use the system (previous methods used
fixed positions for determining the pitch). The simplest implementation uses
only three commands: ”no change”, ”change upwards” and ”change downwards”
(finer steps can be chosen to make playing more precisely, for example ”big
change upwards”, ”small change upwards” etc). If the melody’s next note is
above the current pitch, the user has to move her arms upward, if the next note
is below, she has to move her arms downwards. The system then automatically
chooses the right note to play the correct melody. If the user makes a ”wrong”
move, the system either stays at the currently sung pitch (”no change”), or
uses a mapping-table to determine the next higher respectively next lower pitch.
There are a total of 25 MIDI-pitches that can be sung, from MIDI-pitch 41
(≈ 87 Hz) to MIDI-pitch 65 (≈ 349 Hz). The actual singable pitches are limited
by the song’s current key (all keys need to be predefined for the whole song).
The program triggers events in accordance to the song’s beat to change the
currently used mapping table. The basis of these mapping tables has always the
keynote ”C”, that is transposed later, with a variety of scales: C-Major, C-Minor,
C diminished and C augmented, as well with optionally added seventh, major
seventh, sixth or diminished fifth. In addition, the system distinguishes whether
the song’s measure is in beat 1 or 4, or in beat 2 or 3. In case one, only pitch
values from the current key’s chord can be played, in case two the whole scale
can be selected. These limitations help the user to create a melody that sounds
always more or less suitable for the currently chosen song’s background music,
even if she is not performing the correct arm movements for singing the song’s
melody. This allows for more or less harmonic improvisation if the user does not
want to sing the original song. The desired vowel is given via the user’s mouth
Fig. 10. Vowel ”A” at approximately 220Hz a) synthesized, b) sung by real tenor
shape. 23 of the 46 parameters provided by the face tracking module are relevant
for the area around the user’s mouth. Therefore they are used as input nodes
for a neural network trained to identify vowels by a given mouth shape. The
neural network uses resilient back-propagation. This is a fast back-propagation
algorithm that allows to train new data to the network in a matter of seconds [9].
The output of the neural network consists of 4 probability parameters, ”Closed
Mouth”, ”A” [a:], ”E” [e:] and ”O” [o:]. These four values are monitored, and
the highest determines the current vowel, sending the frequency values to the
representing formant filters. To give the user the opportunity of choosing another
set of vowels ”Ä [ε:]”, ”I” [i:] and ”U” [u:], the option to switch between the two
sets is activated by moving the eyebrows up and down or by means of a dedicated
hand / finger gesture.
8 Conclusion
The current prototype is functional, i.e., the individual components work but are
still a bit limited concerning the tracking quality and the range of expressively
synthesized vocals. The integration into a final system prototype is complete but
we are still working on an expressive performance mode. Currently we use an old
gramophone induced by an exciter to produce an enjoyable auditive experience
for both the user and the audience. We also evaluate the combination of vibra-
tional feedback and bone conduction headphones to enhance the user experience
and support the immersion of the user during the performance. This should fur-
ther increase the intended believability and enjoyment of being an opera tenor
during a performance.
References
1. Butler, A., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., Kim, D.:
Shake’n’Sense: Reducing Interference for Overlapping Structured Light Depth
Cameras. In: ACM SIGCHI Conference on Human Factors in Computing Systems
(2012)
2. Cheng, J., Huang, P.: Real-Time Mouth Tracking and 3D reconstruction. In: Int.
Conf. on Image and Signal Processing (2010)
3. Falk, D.: Finding our Tongues: Mothers, Infants, and the Origins of Language.
Basic Books (2009)
4. Hapipis, A., Miranda, E.R.: Artificial Singing with a webcam mouth-controller. In:
Int Conf. on Sound and Music Computing (2005)
5. Lyons, M., Haehnel, M., Tetsutani, N.: Designing, Playing, and Performing with a
Vision-based Mouth Interface. In: Proc. of NIME 2003 (2003)
6. Miller, G.: The mating mind, how sexual choice shaped the evolution of human
nature. Anchor (2001)
7. Odowichuk, G., Trail, S., Driessen, P., Nie, W., Page, W.: Sensor fusion: Towards
a fully expressive 3D music control interface. In: Pacific Rim Conference on Com-
munications, Computers and Signal Processing (2011)
8. Peterson, G.E., Barney, H.L.: Control methods used in a study of the vowels. J. of
the Acoustical Society of America 24, 183 (1952)
9. Riedmiller, M., Braun, H.: Rprop - A Fast Adaptive Learning Algorithm. In: Proc.
of the International Symposium on Computer and Information Science VII (1992)
10. Rodet, X.: Synthesis and Processing of the Singing Voice. In: 1st IEEE Workshop
on Model Based Processing and Coding of Audio (2002)
11. de Silva, C., Smyth, T., Lyons, M.J.: A novel face-tracking mouth controller and
its application to interacting with bioacoustic models. In: Proc. of NIME (2004)
12. Sundberg, J.: The KTH Synthesis of Singing. J. on Advances in Cognitive Psy-
chology 2(2-3) (2006)
1 Introduction
One of the keys to the recent and continued success of the game industry has
been the ability to expand to new demographics of game players outside the
normal user groups [2]. A large portion of the recent success can be attributed
to a developmental shift in the way players interact with gaming systems, such as
Nintendo Wii, PlayStation Move and Microsoft Kinect. As the demographics of
game players expands, so too will the range of players abilities and entertainment
needs. Players have varying skill levels in terms of characteristics such as reaction
times, hand eye coordination, and tolerance for failure. The variation in players’
abilities will increase the difficulty for game designers to sort players into the
usual static and preset difficulty settings of easy, medium and hard. When the
game’s difficulty is not correctly matched to the player’s ability, the player could
become bored or frustrated with the game [2], which results in reduced play time
or abandonment of the game. To overcome these problems, there is a growing
need to more thoroughly understand game difficulty, especially the impact of
various factors and design decisions on difficulty and outcomes.
To form this understanding, experimental methods are required to collect
and analyze the necessary data in a rigorous fashion. This, unfortunately, is a
daunting task for any game of reasonable size and complexity. Consequently,
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 270–283, 2013.
c Springer International Publishing Switzerland 2013
the goal of our current work is to formalize an approach to studying game diffi-
culty, in particular identifying prominent factors and determining their impact
on player experience. This approach has applications to offline analysis during
production to support game balancing, level tuning, and issues with playability
and usability. Online applications for adjusting adaptive game systems include
determining suitable factors and the granularity of the adjustment to optimize
the player’s experience. In this paper, we present our approach, based on a full
factorial analysis methodology, and demonstrate its usefulness through applying
an experimental assessment to a variant of the classic game Pac-Man.
2 Related Work
Our evaluation method for identifying factors with statistical significance effects
on the response variables involves the use of a full 2k factorial design. A full 2k
factorial analysis, contains k factors, each with 2 values or levels. A full factorial
analysis calculates the effect sizes and statistical significance of each factor and
for all possible combinations of the set of factors for each response variable. This
information provides the magnitude and direction of the effect on a particular
response variable. Although, this creates a large volume of terms and data to
evaluate, in practice the game significance of many of these terms will be minimal
and thus the amount of information stored in the adaptive game system can be
greatly reduced.
moves Pac-Man towards the centroid of the remaining items. These heuristics are
modified versions of heuristics shown to be successful for improving Pac-Man’s
score in the work of other researchers [10,13].
5 Conducting Experimentation
In each simulated game, Pac-Man has 3 lives and is unable to gain additional
lives via points or bonus items. Due to the fact that our experiment continuously
runs using the same level, if the player exceeded 350 steps, that particular life
would be halted. This restriction helped normalize situations where Pac-Man was
significantly better than the opponents or where Pac-Man is unable to complete
the level but remains alive. Our experiment consists of four cases; one for the
simulation of each of the algorithm pairs: SSS-AB* and flocking (SSS_FLOCK),
SSS-AB* and GW (SSS_GW), PW and flocking (PW_FLOCK) and PW and
GW algorithms (PW_GW). In our largest case, the PW_GW algorithm pair has
20 factors. The 2k factor analysis of this case results in over a million simulations,
which is then multiplied by 3 repetitions for over 3 million simulations. One of
the key benefits of the full factorial analysis is that it is orthogonal, meaning
the analysis can be divided and performed in a distributed method. Thus, we
further separated the cases for the algorithm pairs into groups containing 10
factors, thus 1024 minimal runs. The decision to review groups of 10 factors was
partially made because of the total number of factors for each algorithm, but
it also minimized the intermediate file sizes and was computationally efficient.
Furthermore, if cases in the analysis use the same number of factors, index
information for the factorial analysis only needs to be calculated once.
6 Experimental Results
Presenting the massive volume of data collected and organized from the simula-
tions would be verbose and would detract from the experiment goals of evaluat-
ing the application of our methodology to the domain of gaming and dynamic
difficulty. Instead, we focus the discussion of our results on the main effects of
selected cases from the experiments; in-depth results can be reviewed in [3]. For
SSS_FLOCK, we selected the fruit frequency at a low level, as the two cases
have nearly identical results. For the SSS_GW algorithm, we selected the fruit
time at a high level value. For the PW_FLOCK and PW_GW algorithm, we
selected the factor for avoiding the ghosts at a high level value, as it had the
most positive influence on the response variables for PW algorithm.
The model evaluation indicates the percent of variation that our model was ca-
pable of explaining with the 127 highest sum of square terms. The 127 terms
limitation comes from the commercial software Minitab; this evaluation method
could indicate whether additional terms are required to improve the model’s per-
formance. Calculating the R-Squared (R-Sq) values provides two useful pieces of
the information: it indicates the percentage of variance that our model explains
and the model’s accuracy in predicting other data points. This information can
be used in adaptive systems to distinguish between response variables that are
potentially less predictable or difficult to adapt for a particular player. The sec-
ond part of the model evaluation investigates the consequences of the commercial
term limitation, as not all 1024 terms could be included in the model; we cal-
culated the lack-of-fit values for the 897 terms not included and whether their
exclusion played a significant role in the results of the experiment.
The results in Table 1 identified that the most difficult and inconsistent re-
sponse variables to predict and adapt for are collecting the fruit and the number
of levels completed. The models for the number of close calls, steps and score
showed higher levels of predictability among the response variable models and
are potentially good candidates for the adjustment process. The lack-of-fit F-
values ranges for the models are: SSS_FLOCK ( 0.16 - 0.38 ), PW_FLOCK (
Table 1. Model evaluation using the R-Sq (Adj) for each algorithm and all response
variables
SSS_ PW_
Response SSS_GW PW_GW
FLOCK FLOCK
Variables R-Sq% R-Sq%
R-Sq% R-Sq%
CC 83.1 70.6 63.7 69.5
FC 69.6 70.0 58.2 61.0
GE 80.4 71.0 69.3 72.8
LVL 79.5 65.1 60.4 58.1
PP 80.3 66.0 64.4 65.7
RSq 78.5 74.0 67.0 76.5
Sc 80.2 68.4 59.3 67.17
STP 79.7 74.0 67.2 77.62
TOK 80.52 67.4 65.0 68.8
0.29 - 0.39 ), SSS_GW ( 0.34 - 0.47 ) and PW_GW (0.33 - 0.41). The lack-of-fit
testing proved the commercial limit of 127 terms did not play a significant role
in limiting the results of our separated case models, although the term limit did
force splitting the models into separate cases due to the large number of factors
included in the design phase. The lack-of-fit results indicate that many terms
were not statistically significant to the game. This highlights one of the bene-
fits of this methodology; we identified a large number of factors game designers
would have experimented with but had no impact on the results of game play.
additional time the bonus items are available. The perceived value of the fruit
not being a statistically significant main effect for more response variables could
be due to the bonus item always being worth more than tokens or power-pellets.
As such, it represents a goal the SSS-AB* algorithm will always attempt to com-
plete, while minimizing uncertainty and unnecessary risks.
Table 2. SSS_FLOCK statistically significant main effects are listed with: P for pos-
itive or N for negative effect. Non-statistically significant main effects are indicated by
the - symbol.
Table 3. SSS_GW statistically significant main effects are listed with: P for positive
or N for negative effect. Non-statistically significant main effects are indicated by the
- symbol..
have a statistically significant effect for only a few response variables. For these
three cases, we observe a similar pattern in which Pac-Man successfully collects
additional points from fruits and ghosts, at the expense of a shorter lifetime.
So while these factors had minimal impact on the statistical significance of the
response variables, these factors seemed to have altered the risk and reward ra-
tio tempting the player to gain more points at the risk of a shorter life span.
The PW algorithm takes a greater number of chances to collect the fruit. This
behaviour is unlike the SSS-AB* algorithm, which uses the extra flee and death
time to complete level tasks. The PW_FLOCK session results indicate that
nearly all factors had a statistically significant impact on the fruit collected.
The flock struggled to capture Pac-Man, especially when becoming overly ag-
gressive, however, they were effective in stopping Pac-Man from engaging in too
many bonus tasks. This suggests small modifications to the flock‘s behaviour
focusing on cohesion and hunger would provide a more balanced challenge.
Table 4. PW_FLOCK statistically significant main effects are listed with: P for pos-
itive or N for negative effect. Non-statistically significant main effects are indicated by
the - symbol..
Table 5. PW_GW statistically significant main effects are listed with: P for positive
or N for negative effect. Non-statistically significant main effects are indicated by the
- symbol..
of Pac-Man’s path. Interestingly, chasing the end point of Pac-Man’s path pro-
duced less direct challenge but ultimately produced comparable overall results
to chasing Pac-Man’s current position. The overall result of these cases is nearly
identical for most response variables, but gameplay would provide less conflict
and likely a less stressful game session to the players.
6.3 Limitations
One of the largest caveats to examining game difficulty using a full factorial
analysis methodology is the exponential growth of calculations that occurs for
the inclusion of each additional factor. Fortunately, one of the strengths of the
full factorial analysis is that it is orthogonal, which allows the calculations to be
separated and calculated independently, reducing some of the strain of the data
size issues. The complexity of the full factorial statistical analysis scales well
performance rather than additional points scored by the player. Given this result,
game designers could strategically place bonus items or tasks in areas with lower
levels of interest or challenge. The placement of bonus items could be gradually
adjusted into more challenging areas, which could be used to gauge the level of
risk the player is comfortable competing against.
Commercially, this methodology has the potential to be effective for testing
new features and tuning final factor settings for release. An advantage of this
methodology is that the factorial analysis is completely independent of the factor
level values or game values selected, so integration of this subsystem requires
only identifying response variables to collect information about and thus this
subsystem can easily be injected into a wide variety of games. It can be easily
integrated into commercial products during the testing phase and due to the
independence and modularity of these methodologies as a sub-system, it can
be easily removed before the product is released. The ability of the analysis to
evaluate any modifications to the player attributes, NPCs or level factors offers
the advantage of a wider variety of adaptive decisions.
References
1. Andrade, G., Ramalho, G., Santana, H., Corruble, V.: Challenge-sensitive action
selection: An application to game balancing. Intelligent Agent Technology, 194–200
(2005)
2. Bateman, C., Boon, R.: 21st Century Game Design. Charles River Media (2006)
3. Fraser, I.J.: Game challenge: A factorial analysis approach. Master’s thesis, Uni-
versity Of Western Ontario (2012), http://ir.lib.uwo.ca/etd/563/
4. Hunicke, R., Chapman, V.: Ai for dynamic difficulty adjustment in games. In:
Challenges in Game Artificial Intelligence AAAI Workshop, pp. 91–96 (2004)
5. Miller, S.: Auto-Dynamic Difficulty. Published in Scott Miller Game Matters Blog
(2004), http://dukenukem.typepad.com/game-matters/2004/
01/autoadjusting.html
6. Plaat, A., Schaeffer, J., Pijls, W., Bruin, A.: SSS*= AB+ TT (1995)
7. Reynolds, C.: Flocks, Herds and Schools: A Distributed Behavioral Model. ACM
SIGGRAPH Computer Graphics, 25–34 (1987)
8. Spronck, P., Sprinkhuizen-Kuyper, I., Postma, E.: Difficulty scaling of game ai.
Intelligent Games and Simulation, 33–37 (2004)
9. Spronck, P.H.M.: Adaptive game ai. Ph.D. thesis (2005)
10. Szita, I., Lorincz, A.: Learning to Play Using Low-Complexity Rule-Based Policies:
Illustrations through Ms. Pac-Man. Artificial Intelligence Research pp. 659–684
(2007)
11. Togelius, J., De Nardi, R., Lucas, S.: Towards automatic personalised content cre-
ation for racing games. Computational Intelligence and Games, 252–259 (2007)
12. Togelius, J., Lucas, S.: Evolving controllers for simulated car racing. Evolutionary
Computation, 1906–1913 (2005)
13. Yannakakis, G., Hallam, J.: Evolving opponents for interesting interactive com-
puter games. In: From Animals to Animats, pp. 499–508 (2004)
14. Yannakakis, G., Hallam, J.: A Generic Approach for Generating Interesting Inter-
active Pac-Man Opponents. In: IEEE Symposium on Computational Intelligence
and Games, pp. 94–101 (2005)
Christian Geiger1 , Simon Thiele1 , Laurid Meyer1 , Stefan Meyer1 , Lutz Hören1 ,
and Daniel Drochtert1
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 284–295, 2013.
c Springer International Publishing Switzerland 2013
design of a appropriate archery user interface, using a real bow that is extended
with electronics and a damping system [1]. This allowed realistic and safe indoor
practicing of the archery sport in reality-based virtual outdoor scenes. Based
on positive feedback from this work we extended the interaction methods and
the previous realistic content of the virtual environment for entertaining show-
cases at a large public fairs. With the goal of making the archery experience
most entertaining for every type of visitor in mind, an adventure game called
”Goin’ Goblins” (GG) was developed in a design process that extends techniques
from user centered design, software engineering and digital media production. In
addition to the interaction with the real bow, a set of advanced interaction tech-
niques extends the regular way of using the archery simulator towards a more
fun-oriented application. This includes choosing different projectiles, switching
to an rookie mode or cooperation of multiple users, by featuring a wizard that
helps the main player through the game. However, we learnt during the design
and evaluation of this project that not every technique that is technically feasi-
ble and looks like an exciting idea on its own is also suitable to be used in a fair
scenario where users usually have only a limited time frame and attention span.
2 Related Work
Apart from archery simulation, the design of interfaces and simulations of sport
activities has been proposed in several works in the past. One example for this
are exertion interfaces by Floyd Mueller et al. [2]. Their projects, like Table
Tennis for Three, Jogging over a Distance or Shadow Boxing over distance,
aim at creating social bonding through physical activity between user that are
spatially separated. The proposed conceptional design framework consists of a
design vocabulary that is suitable for usage in the design process of exertion
games. Most of the exertion interfaces, however, focus the design on the spatial
separation and social bonding aspects and thereby neglect simulating a real-life
activity and providing entertaining experiences. Other sports interfaces focus on
being beneficial for training purposes. Examples for these interfaces, focussing
of realism, are proposed by Multon et al. [3], who created a training system for
gymnasts or Zhang and Liu [4] for training in ball sports.
Archery simulations with focus on entertaining experiences are available with
the commercial games on the PlayStation and Nintendo Wii, using the move con-
troller, as well as the Wiimote. Both games feature a virtual environment that
resembles a real archery range, with varying visual realism, where the players’
goal is to achieve highscores by shooting at static and moving targets in various
distances from a static position. While the archery game on the PlayStation
offers a more realistic simulation than the Wii, by adapting the posture of the
bow to the players’ movements, the realism of both games however, comes short
because of the lack of haptic feedback, the triggering of the shot through press-
ing a button and the accuracy constraints of the hardware of the controllers.
The archery simulator by TechnoHunt [5], aimed at expert archers, features the
realistic feeling of handling a real bow, however, the visual simulation is achieved
through playing video sequences that lack an immersive feeling.
The work described in this paper is based on the development of the bow as
the user interface and the design process of the archery simulation that has begun
with previous prototypes that first used infrared tracking for all movements of
the bow [6], serving as a proof of concept, and that was subsequently further
developed by enhancing the bow with various sensors [1] that greatly increased
the accuracy of the tracking and shot detection and thereby creating a realistic
experience of the activity. The content of the virtual environment described
in [1] is based on a realistic simulation of an outdoor archery setting. This paper
describes the adaption of the realistic sensation of shooting with a bow to an
entertaining game environment with further advanced user interaction methods
to make the experience attractive to a broad audience.
Advanced user interfaces (AUI) include natural user interfaces and reality-based
interaction that try to mimic the users’ everyday experience and provide a easy-
to-use and expressive way to interact with digital content. These interfaces are
useful for short-time use of applications and a large, heterogenous user base
as it is the case when developing attractive interactive installations for trade /
science fairs or exhibitions. The target group is able to instantly interact and
attain their goals with only a very short learning curve. The development of
these advanced UIs is rather complex as the field of expertise is more demand-
ing in relation to traditional computer and GUI based applications. Few design
experience exists and there is no structured design approach available of design
guidelines that guide designers how to create a compelling AUI. We provide a
pragmatic approach that uses state of the art concepts in user centered design
(UCD) and specified a structured workflow that combines an extended Scrum
model with techniques from UCD, SE and Usability Engineering (UE). The ap-
proach is based on Paelke’s and Nebe’s work [7] and additionally applies methods
and techniques from UCD and digital media production (see fig 1). A detailed
description of this process is beyond the scope of this work and we will focus
only on selected methods we used to design of our archery experience.
The process described in figure 1 only suggests suitable methods in different
iteration cycles. It depends on the project and the design team which methods
should be applied and to what extent. For our project we applied:
Creativity techniques We applied Forced Analogy, Speed Inventing and 3-12-
3 Brainstorming in this project. Suitable techniques could be found in [8].
Persona Personas are well known in HCI and help to identify user types based
on interviews and other means to gather user relevant data, see [9].
Task/Goals and Scenarios Goals of a persona are accomplished by more
technical tasks. This activity is described narratively in a scenario, see [9].
Scribble and Storyboard Scenarios could be visualized using sketches and
storyboards. This approach is borrowed from film making and animation.
4 Requirement Specification
Before the design process was initiated, a number of professional archers were
interviewed to identify the most important factors for a believable user expe-
rience. Furthermore, we participated in a 2-day archery course to understand
the different steps of an optimal shot and the special constraints of traditional
archery. Additionally, we also visited the cooperating archery school at a hunt-
ing fair, interviewed fair visitors and identified a number of requirements for a
believable and entertaining archery experience.
– We identified two types of fair visitors. Visitors like archers that are inter-
ested in a reality-based user experience and visitors like gamers that want a
new player experience. The former user type is interested to experience the
simulation as close to reality as possible. Outdoor scenes including targets
should be modeled in a realistic way and different ways of handling the bow
(e.g. different draw lengths) should result in different shooting results. The
latter user type is interested in gaming concepts and want to solve tasks to
proceed along an exciting story line.
– The system should be easily set up and should work robust during the exhibi-
tion. This includes to consider diverse demonstrator set-ups, e.g. considering
individual light conditions and to allow multimodal ways to interact with
the system
– Spatial cues like 3D stereo and viewpoint tracking to increase the user’s
immersive feeling and advanced audio-visual representation of all scene ob-
jects to maximize hedonic qualities. This includes realistic animation and
expressive sound design.
Based on these requirements and the client’s primary goal to attract novice users
at trade fairs to start archery courses we decided to focus on providing a believ-
able and entertaining archery experience. Virtual reality techniques like spatial
user tracking, stereo viewing, haptic feedback, user adaptive viewpoint control
on a large projection screen and advanced 3D graphics should be implemented to
create a semi-realistic application scenario of traditional archery. Based on the
user interviews we first created personas with task/goals and a small scenario [9].
The following description is a brief summary of the results.
Fred B. and his daughter Sophie visit a science fair and want to ex-
perience the archery simulator featuring the ”Goin’ Goblins” level. The
demonstrator’s goal its to provide an entertaining archery experience for
two people. Fred is a passionated traditional archer practicing his hobby
2-3 times a month. Sophie is interested in sporting games with full-body
interaction. She has practiced archery in Wii Sports Resort and PS3
Sports Champion Archery. They both are interested in fantasy films and
like adventure games which they play together sometimes. An assistant
introduces them on how to handle the bow and the application. A number
of quests have to be solved in cooperation to proceed through the story-
line. The first player acts as an archer and the second player acts as a
magician that assists to solve the quests.
Design Experience: In this phase we successfully used the design methods
user interviews, Personas, task / goals and scenarios. For illustration we also
created scribbles and a story board. Figure 2 shows a scribble of the described
scenario that was also used as part of a story board.
children that are not able to handle the heavy bow. By using a dedicated laser
tracking system, touches on large vertical planar surfaces in front of the pro-
jection can be detected. If a plush toy hits the projection screen and thus the
invisible tracking plane, the center of the touch area is used as the target point
for the arrow simulation (see fig. 3).
The two-player mode requires interaction of the archer player using the bow
and the wizard player using two gestures for healing and magic. We selected a
wipe gesture with the right hand to heal from damage and a wand to create
magic. The healing spells are casted with the sliding left hand gestures that is
detected by the Kinect sensor. The magic spell is casted using a magic wand with
an integrated Wiimote controller. Due to connection problems with the Wiimote
during a 8-hour presentation day at fairs, we also provided an alternate wave
gesture with the other (right) hand. See the video for further explanations or
visit www.virtual-arrow.net.
Design Experience: To illustrate and discuss the interaction techniques we
created a video prototype of the system before we implemented a first system
prototype. This helped us to discuss the interaction design with experts. Video
prototypes are a powerful means to quickly create first impressions of intended
systems [10]. It served also a very valuable communication tools to discuss design
ideas and prototypes.
6 System Design
The user handling the bow can move freely inside the interaction area in front of
the projection screen. The tracking itself is accomplished by two camera systems
facing in opposite directions. One, installed underneath the projection screen,
facing the interaction area, is used for providing data on the movements and
gestures of the users inside the interaction area and the other one, facing the
projection screen, for tracking the shot direction of the bow into the virtual
scene. The visualization is then calculated based on the data of the sensors of
the bow and the two camera systems and rendered in full HD quality on a large
screen.
The bow is equipped with a custom damping system based on a archery shot
trainer (see fig. 4), that allows to dry-fire a real bow without damaging it. It uses
a pneumatical approach similar to an air pump to safely release and stop an arrow
shaft. The correct draw length, aim and the release is measured using magnetic
switches, also know as reed sensors. These sensors are cheap, light-weighted, and
rugged and have very little interference liability. If a magnetic field appears close
to the sensor then a pair of ferrous metal reeds of an electronic switch connect
and the switch closes. If the attractive force of the magnetic field is not strong
enough anymore the reeds are disconnected and the electronic switch is opened.
We attached a strong neodymium magnet to the compression ring of the arrow
damper and attached a dozen of reed sensors outside of the tube (see fig. 4). This
way we can measure the movement of the arrow. These sensors are connected to
an Arduino Nano, which calculates the position, speed and acceleration of the
arrow shaft and transmits them to the host computer via Bluetooth or USB.
To detect the position in the virtual scene that the real bow is aiming at we
attached an infrared (IR) laser to the bow (see fig. 4). The laser invisibly points
to the screen position the user is aiming at. The system uses an IR sensitive
camera and can thereby easily detect the IR spot and calculate the orientation
and position of a virtual arrow in the 3D environment. Tracking of the user/bow
is only used for viewpoint animation. We selected a pragmatic and inexpensive
solution with a Kinect that detects the nearest person in front of the camera.
The viewpoint (the virtual camera in the 3D world) is moved accordingly to
provide an immersive depth cue.
Different shooting modes can be switched using RFID objects that are de-
tected using a RFID sensor at the Arduino Nano board. The physical objects
used for mode triggering are equipped with corresponding RFID tags. If an ob-
ject is attached to the bow, the RFID reader detects the tag of the object and
the corresponding mode is triggered until another object is detected. Although
this feature worked quite well, we decided not to use it in the Goin’ Goblins level
described in this paper because the switching of arrows slowed down the game
flow as we experienced during some walk-through sessions. For smaller children
that are not able to handle the bow we provide an alternative shooting method
by using a dedicated laser tracking system (see fig. 3) with custom developed
software for recognising, filtering and processing touch events and transmitting
the data to a 3D game engine via OSC (Open Sound Control protocol). A more
detailed description of the hardware system was presented in [1]. Based on ex-
pert reviews we conducted at the client archery school we decided to provide
wind feedback in 2013. We used a DMX interface of the wind machine which
also communicates with our archery simulation via OSC . Based on wind events
in the 3D simulation this interface triggers the wind machine. We successfully
tested the effectiveness of wind output using a ”Wizard of Oz”-Test, i.e. the fan
was operated by a team member. After successful tests we integrated this haptic
feedback into the game level at selected places.
Design Experience: The design of the project’s hardware parts was not trivial
because we had to test a number of alternative ways to track the draw length
and release of the bow damper, the orientation of the bow and the position of
the players. For selected interaction techniques the ”Wizard of Oz” served useful
to rule-out unsuitable interaction techniques and get first impressions without
investing too many resources. Detailed hardware test in the lab and in ”real-life”
scenarios were necessary to find the best available option.
Fig. 5. Level Design and corresponding 3D scenes of the final game level
from the Unity 3D asset store and adapted to the design of the scenario. Shooting
targets were imported or modeled, rigged, animated and integrated into the
application logic. We reused most of the content from other projects due to the
limited time frame (6 weeks) we had to create this game level.
Design Experience: Due to timing constraints our design approach was to
design the game level after(!) the assets have been selected. This proved to be
rather useful for the given short project period for the game development. We
managed to create the level from scratch within the given time frame, but heavily
reused 3D assets from other projects and the Unity asset store. Due to the large
variety of users we needed to easily configure the system on-the-fly (e.g changing
the virtual draw weight, recalibrating the system and interactively adjusting off-
sets for precise tracking). To easily help unexperienced players that could not
easily hit the targets to proceed in the story we developed key shortcuts that
were mapped to a Ring Mouse that one of our team members wears. S/he could
”solve” the current quest if it took too long for the users to complete the task.
This goal is also supported by a number of novel interaction techniques that ap-
ply different advanced interaction techniques during the game experience. The
system was created by a student group of 12 project members (3D programmers,
3D artists, graphics and sound designers and electronic specialists) by applying
an iterative ’design & test’ process. In our view, the contribution of this paper
is to describe a best practice to create an entertaining interactive 3D experience
that is suitable to attract visitors at fairs. However, we learnt that although we
performed different evaluations during the iterative design process and we could
rule out many wrong decisions at early stages that some design approaches could
only be validated in a real life setting.
We identified the following best-practice hints and design guidelines during
this work for an entertaining user experience at trade fairs:
References
1. Geiger, C., Thiele, S., Meyer, L., Drochtert, D., Stöcklein, J., Wöldecke, B.: Virtual
archery with tangible interaction. In: IEEE 3DUI (2013)
2. Mueller, F.F., Edge, D., Vetere, F., Gibbs, M.R., Agamanolis, S., Bongers, B.,
Sheridan, J.G.: Designing sports: a framework for exertion games. In: Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2011,
pp. 2651–2660. ACM, New York (2011)
3. Multon, F., Hoyet, L., Komura, T., Kulpa, R.: Interactive control of physically-
valid aerial motion: application to vr training system for gymnasts. In: Proceedings
of the 2007 ACM Symposium on Virtual Reality Software and Technology, VRST
2007, pp. 77–80. ACM, New York (2007)
4. Zhang, L., Liu, Q.: Application of simulation and virtual reality to physical ed-
ucation and athletic training. In: Pan, Z., Cheok, A.D., Müller, W., Chang, M.,
Zhang, M. (eds.) Transactions on Edutainment VII. LNCS, vol. 7145, pp. 24–33.
Springer, Heidelberg (2012)
5. ArcheryInteractive: Technohunt bow simulator, http://www.technohunt.com
6. Geiger, C., Herder, J., Göbel, S., Heinze, C., Marinos, D.: Design and virtual studio
presentation of a traditional archery simulator. In: Entertainment Interfaces Track
2010 at Interaktive Kulturen (2010)
7. Paelke, V., Nebe, K.: Integrating agile methods for mixed reality design space
exploration. In: Proceedings of the 7th ACM Conference on Designing Interactive
Systems, pp. 240–249. ACM (2008)
8. Gray, D., Brown, S., Macanufo, J.: Gamestorming - A Playbook for Innovators,
Rulebreakers and CHangemakers. O’Reilly (2010)
9. Cooper, A.: About Face 3 - The Essentials of Interaction Design. Wiley (2007)
10. Zimmermann, J.: Video sketches: Exploring pervasive computing video sketches: Ex-
ploring pervasive computing interaction designs. Pervasive Computing 4(4), 91–94
(2005)
11. Höysniemi, J., Hämäläinen, P., Turkki, L.: Wizard of oz prototyping of computer
vision based action games for children. In: Proceedings of the 2004 Conference on
Interaction Design and Children: Building a Community, IDC 2004, pp. 27–34.
ACM, New York (2004)
12. Ijsselsteijn, W.A., Poels, K., de Kort, Y.A.W.: Game experience questionnaire (fuga
the fun of gaming: Measuring the human experience of media enjoyment. Technical
report Deliverable 3.3 (2008)
13. Hassenzahl, M., Platz, A., Burmester, M., Lerner, K.: Hedonic and ergonomic
quality aspects determine a software’s appeal. CHI Letters 2(1), 201–208 (2000)
14. Schubert, T.W.: The sense of presence in virtual environments: A three-component
scale measuring spatial presence, involvement, and realness. Zeitschrift für Medi-
enpsychologie 15(2), 69–71 (2003)
1 Introduction
Movies and video are amongst the biggest sources of entertainment, in individual
and social contexts. The evolution of technology has enabled the fast expansion
of media and social networks over the internet, giving rise to huge and increasing
collections of videos and movies accessible over the internet and through video
on demand services on iTV. These multimedia collections are tremendously vast
and demand for new and more powerful search mechanisms that may benefit
from video and audio content based analysis and classification techniques. Some
researchers [2,6] pointed the importance for the development of methods to ex-
tract interesting and meaningful features in video to effectively summarize and
index them at the level of subtitles, audio and video image. The approach de-
scribed in this paper was designed for the VIRUS1 project [9] to help towards
1
Video Information Retrieval Using Subtitles.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 296–307, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
The approach to create entertainment applications to collect data from the user is
not a novelty, being currently identified with the terms “Game With A Purpose”
(GWAP) and “Human Computation”.
One of the first applications to apply such ideas was the ESP application [13]
where players label the same image in pairs without any kind of communica-
tion between them, and are rewarded when the labels used are the same. This
approach is named “output-agreement” because a reward is received when the
output used by the players matches, being used also by other games such as Peek-
aboom [14], ListenGame [12], TagaTune [10], MajorMiner [11], and Herdit [1].
Herdit [1] is a GWAP dedicated to the gathering of audio data (from mu-
sic) and provides a multi-player experience (more than the common two player
games) over a social network. Another GWAP for music is TagaTune [10], propos-
ing a different approach. This two player game is based on input-agreement. The
two players, that cannot communicate, listen to the same sound clip and propose
labels that describe it. Each player sees the labels proposed by the other player
and the round finishes when players indicate if they are listening to the same
piece of music. The agreement is therefore on the input and not on the output
as in the previous ones. The ListenGame [12] is also a game for labelling music.
In this game, players choose among a predefined list of labels those that fit best
and worse to a given music piece. Based on the answers of several players, a
confidence level is associated to each tag.
Concerning the aspect of user-interfaces, the authors of the games previously
mentioned opted for minimalistic features. Several aspects can be identified: 1)
the item (music clip) has often no graphical representation (HerdIt, ListenGame,
MajorMiner, MoodSwing), in this case the music clip is automatically played and
the users cannot interfere (stop/pause/continue) nor can they see the progress
of the listening. In the case of TagATune, more traditional media player controls
are presented. 2) Labels are represented by simple words (no decoration) in
MajorMiner and TagATune or by buttons (ListenGame) or in the case of HerdIt,
by floating bubbles. When labels are chosen by the user, they are entered using
a text box and buttons (or bubbles) are used in case of predefined labels. 3) The
use of colours differ between interfaces. In TagATune only two colours are used
(purple and white) with gradients in order to produce an aesthetic effect. The
bubbles in HerdIt are coloured but the colouring scheme seems to be random.
In the ListenGame, the colours are used to identify the choices made (labels)
by the two players. Finally, the MajorMiner game uses very few colours, for
differentiating levels of confidence of each label (italicised font is also used for
this purpose). 4) Scores are represented either by a numerical value (ListenGame,
MajorMiner, and TagATune) or using a thermometer metaphor (HerdIt and ESP
game).
3 Our Aproach
Fig. 1. MovieClouds Movie View navigation[9]: a) unique tagcloud for 5 content tracks;
b) 5 separated tag clouds for each track. Some tags selected and highlighted in the
movies where they appear (top).
in five tracks (subtitles, emotions in subtitles, audio events, audio mood (based
on music), and felt emotions), for the power, flexibility, engagement and fun
usually associated with them.
After selecting the Back to the Future movie in Figure 1b) (top right), it plays in
the Movie View in Figure 2a), with five timelines for the content tracks showed
below the movie, and a selected overview tag cloud of the content presented on
the right (audio events in the example), synchronized with the movie that is
playing and thus with the video timeline, to emphasize when the events occur
along the movie. Users may choose to see, on the right, any of the five content
tracks, and compare in the timelines where the selected tags occur along time
(e.g. what is happening in the other tracks when the audio has a gunshot event?).
From the timelines, users may select which time to watch in the video. If the user
is registered, a small head-up display is always displayed in the top-right corner
with the name, accumulated points and rank level (based on points: “rookie”
in the example) to the left of the SoundsLike star logo, reminding the user to
access it.
excerpts and possibly watch them in the context of the movie, SoundsLike was
designed to support the identification of the current audio excerpts to be labelled.
Three timelines are presented below the video: the top timeline represents an
overview of the entire video soundtrack; the middle one presents a zoomed-in
timeline view with the level of detail chosen by the user, by dragging a marker
on the soundtrack timeline. Here, audio excerpts are smaller fragments from
the movie’s soundtrack represented as coloured rectangles; the bottom timeline
displays a close-up on a chosen excerpt as an audio spectrogram. We designed
the audio timelines to be similar to the timelines already used in Movieclouds
for content tracks (figure 2a)). The relation between each of the timelines is
reinforced by colour matching of selection and frames.
The audio similarity graph is provided as a helping tool for the classifica-
tion of audio excerpts by displaying similar sounds to the current excerpt. The
visualization is achieved through a dynamic force-directed graph representing
the similarity relations to most similar audio excerpts in the movie (the shorter
connection, the similar). The similarity value is the Euclidean distance between
audio histograms computed from extracted features (more details in [4]), and it
is translated to the graph metaphor as the screen distance between nodes. The
graph defines minimum and maximum distances for its edges and normalizes all
similarity values between those distances. The nodes repel each other and the
graph may be dragged around with an elastic behaviour. This is particularly
useful, if the users identify this audio better than the one they were trying to
identify, and had not classified it before, allowing to earn points faster; and to
select similar audios after they have finished the current classification.
Similar to MovieClouds’ navigation, the views are synchronized in SoundsLike
besides the synchronization between the timelines. This is adopted by using the
same colours mappings and similar behaviours in all views. In all the timelines
and the graph, the current excerpt to be classified is highlighted in blue, segments
which are yet to be classified are grey, excerpts classified before or skipped are
marked with green and yellow colours. In a similar way, all representations of
sound excerpts have the same interaction: hovering over an excerpt will play the
sound associated and display a bright red coloured frame to temporarily highlight
the location of the excerpts everywhere, while selecting for play (simple click an
audio excerpt) will display the same frame in a brownish tone of red, also around
the video.
Below the graph lies the labelling area. Here users will choose one or more
textual labels (tags) to describe the current sound segment by choosing from a
list of suggested labels or by adding new custom labels. Choosing a label either
to select, reject or ignore, is done through clicks until the desired option is on
(green v marker to accept, red x marker to reject, and no marker to ignore).
In addition, labels that were introduced by the users have an additional frame
in the same colour of the marker (green, red) or grey to ignore. The choices
are saved when the save button is pressed. At any time, the user can skip an
excerpt and choose another one, or simply leave the game. The score will remain
registered. When choices are submitted, the current excerpt changes colour to
green in the similarity graph and the timelines, displaying the earned score at
the central node of the graph that is also enlarged. Now, the user may use the
Next Sound button to randomly navigate to another segment in the same movie,
or may choose a specific excerpt from the similarity graph or from the timeline.
Excerpts already labelled (green) cannot be labelled again.
In this section, we present the main design options to persuade users to con-
tribute for the labelling task of movies soundtracks through a gamification ap-
proach.
Before users are allowed to freely label audio pieces, we need to assess the
user level of expertise to assign a confidence level to their labelling. When users
play for the first time, the system will present audio excerpts for which the
confidence level by the model is very high (i.e. Audio excerpts labelled by experts
or with a considerable amount of aggregated labels and users consensus). These
excerpts are introduced seamlessly in the game flow, being chosen frequently in
the beginning and fading over time. This approach provide us with information
that can be used to better step up or down the difficulty of the queries presented
to the user and the consequent rewards. Our models give us a confidence level
for each label, indicating how “difficult” the query is; and an estimate of the
similarity between two audio excerpts (independently of labels).
Gamification typically involves a kind of reward to the users. Points are earned
by the users as consequence of the following achievements: 1) When their corre-
sponds to a consensus by partial or entire textual matching, taking into account
their confidence level; 2) When the consensual answer corresponds to a difficult
query. An audio excerpt is considered difficult when we observe a high level of
disagreement in the users’ labels; and 3) When a label proposed by a user gets
confirmed by others, the former is rewarded. This way, users can see their score
increase even when off-line, through in-game and email notifications that can
act as an incentive to return to the game later.
6 User Evaluation
We performed an evaluation to assess SoundsLike and detect usability problems
and obtain feedback directly from the users about future improvements in the
application. 10 users participated in the assessment of the application interface
and features, with ages ranging from 21 to 44 years (24 mean value), computer
experience and frequent use of internet. The evaluation focused into the user
perceived usefulness, satisfaction and ease of use.
On an interview fashion evaluation, a task-oriented approach was conducted
to guide the user interactions with the different features, while the evaluator was
observing and taking notes of every relevant reaction or commentary. Every task
involved a set of features and its evaluation was focused on the user experience,
emphasising perceived Usefulness, Satisfaction and Ease of Use (based on the
USE questionnaire [8] and using a 1-5 Lickert scale).
In the end, users were asked to rate the application globally, refer to the
aspects or features they liked the most and the least, and classify it with a
group of terms from a table representing the perceived ergonomic, hedonic and
appeal quality aspects [5].
In the second task we asked the users to play the sound (T2.1) and video (T2.2)
of some excerpts from the timeline and graph. Most users identified a relation-
ship between the graph nodes and the timeline elements and appreciated the
interaction between every component during the video playback.
The third task is similar to the previous, but focused on the timeline fea-
tures, such as the overview of the film provided by the video timeline, scrolling
and manipulation of the dynamic zoom. Users were told to play the video audio
excerpts in the zoom timeline in a fashion that would require them to manipu-
late the timeline to attain the task’s objectives (T3.1). All users used the scroll
buttons without problems, and the majority quickly perceived the zoom manip-
ulation in the video timeline (T3.2). But most users did not find utility about the
audio spectrogram (T3.3), which they reported was due to a lack of knowledge
in the area of sound analysis, but recognized its importance to users that would
have that knowledge.
The fourth task was meant to introduce the user to the similarity graph and
navigation by changing (selecting) the current audio excerpt. We simply asked
the user to listen to the 3 most similar sounds, select one of them and observe the
transition to become the current excerpt (T4.1). We observed that most users
got the distance metaphor correctly, but they did not move the graph to verify
ambiguous cases (when every node stands almost at the same distance), until
they got instructed to do so (T4.2) and since the distances did not differ that
much, in this case the usefulness was 3.5. We also evaluated the user’s perception
about each audio element and their colouring (T4.3).
In the fifth task, we introduced labelling (tagging) to the users, where they
could accept suggested tags (T5.1) and add some labels (T5.2) to audio excerpts
and submit the changes to the database. We prepared three cases: one audio
excerpt without any kind of labels associated, and two with suggested labels:
one case where labels were presented related with the current audio excerpt,
the other one with unrelated labels. We observed that the users were able to
introduce and accept suggestions without significant problems, but some failed to
perceive the label rejection feature (T5.3) without a proper explanation from the
evaluator. Despite the usefulness of the fast labelling feature (comma separated
text), without a proper tooltip, users were unable to find and use it without help
(T5.4 - U:5,S:4.4,E:2.4), but it is a typical feature for more experienced users as
a shortcut, and we observed that it was very appreciated as soon as they became
aware of it.
With every user interface section covered, tasks 6, 7 and 8 were meant to
immerse the user inside the labelling experience in the application by labelling
a considerable amount of excerpts. Users were asked about the utility of au-
dio (T6.1), video (T6.2) playing and the similarity graph (T6.3) in this con-
text. After this, we asked the user about what was the excerpt he would label
next (T7). Here users found the similarity graph more useful than other compo-
nents due to their highly similarity and the propagation of previous used labels as
suggestions. We also inquired users about the scoring mechanism (T8), and they
found it interesting and a great way to stimulate users to participate, although
they did not have a full game experience in this short time. We also observed
a great user’s engagement and immersion within the application, and received
compliments and suggestions to be discussed and applied in future developments.
# Terms # Terms
6 Controllable H 4 Innovative H
6 Original H 4 Inviting A
5 Comprehensible E 4 Motivating A
5 Simple E 3 Supporting E
5 Clear E 3 Complex E
5 Pleasant A 3 Confusing E
4 Interesting H 3 Aesthetic A
and 8 negative terms for each category, 48 in total), as many as they would feel
appropriate (most frequent in Table 2). Almost all the terms were positive, the
most for Ergonomic qualities which are related with traditional usability prac-
tices and ease of use. The most frequent negative terms were “Complex” and
“Confusing”, although the first is correlated with interesting, powerful or chal-
lenging applications (a frequent trade-off, we also recognize in our application),
and both terms are also opposite to the terms “Simple” and “Clear”, that were
selected more often.
7 Conclusions
This paper describes and evaluates SoundsLike, a new Game With A Purpose
whose objective is to collect labels that characterize short audio excerpt taken
from movies. The interface is designed to entertain the user while pursuing the
data collection task. The proposed interface innovates with respect to previous
GWAPs with similar objectives by providing a rich context both in terms of
temporal aspects through three timelines with different time scales, and in terms
of similarities between items by displaying a dynamic force-directed graph where
the neighbourhood of the current item is represented. This graph allows the
exploration of nearby items in terms of similarity, but it is also used to navigate
through the set of audio samples while always keeping track of the corresponding
scene in the video document. The application does not restrict the user to a
closed set of labels but, by making suggestions, tends to overcome the never-
ending vocabulary problem found in other labelling games. The interface lets
the user play audio excerpts in several ways: by hovering with the pointer on
the representation of the excerpt in the graph, or by clicking it to play the
corresponding video in the context of the movie. The user evaluation results were
quite good and encouraging. We observed that users had a pleasant experience
and found it original, controllable, clear and pleasant. They particularly liked the
similarity graph and the timelines. This also provided us with useful feedback
and we were able to identify some usability aspects to improve.
Future work includes: 1) the refinement of the interface based on the feedback
received and the perceived issues; 2) the refinement of the scoring mechanisms; 3)
to remove the constrain of navigation inside a movie at a time, and expand it to
References
1. Barrington, L., O’Malley, D., Turnbull, D., Lanckriet, G.: User-centered design of
a social game to tag music. In: Proceedings of the ACM SIGKDD Workshop on
Human Computation, p. 7. ACM Press (2009)
2. Daniel, G., Chen, M.: Video visualization. IEEE Transactions on Ultrasonics, Fer-
roelectrics and Frequency Control, 409–416 (2003)
3. Deterding, S., Dixon, D., Sicart, M., Nacke, L., Cultures, D., O’Hara, K.: Gamifi-
cation: Using Game Design Elements in Non-Gaming Contexts. In: CHI EA 2011
Workshop, pp. 2425–2428. ACM, New York (2011)
4. Gil, N., Silva, N., Dias, E., Martins, P., Langlois, T., Chambel, T.: Going Through
the Clouds: Search Overviews and Browsing of Movies. In: Proceeding of the 16th
International Academic MindTrek Conference, pp. 158–165. ACM, Tampere (2012)
5. Hassenzahl, M., Platz, A., Burmester, M., Lehner, K.: Hedonic and ergonomic
quality aspects determine a software’s appeal. In: Proceedings of the SIGCHI Con-
ference on Human Factors in Computing Systems, vol. 2, pp. 201–208 (2000)
6. Hauptmann, A.G.: Lessons for the future from a decade of informedia video analysis
research. In: Leow, W.-K., Lew, M., Chua, T.-S., Ma, W.-Y., Chaisorn, L., Bakker,
E.M. (eds.) CIVR 2005. LNCS, vol. 3568, pp. 1–10. Springer, Heidelberg (2005)
7. Khaled, R.: Its Not Just Whether You Win or Lose: Thoughts on Gamification
and Culture. In: Gamification Workshop at ACM CHI 2011, pp. 1–4 (2011)
8. Lund, A.M.: Measuring Usability with the USE Questionnaire. In: Usability and
User Experience, vol. 8(2), Usability SIG (2001)
9. Langlois, T., Chambel, T., Oliveira, E., Carvalho, P., Marques, G., Falcão, A.:
VIRUS: Video information retrieval using subtitles. In: 14th International Aca-
demic MindTrek Conference, p. 197. ACM Press (2010)
10. Law, E.L.M., Dannenberg, R.B., Crawford, M.: Tagatune: A game for music and
sound annotation. In: ISMIR (2007)
11. Mandel, M., Ellis, D.: A Web-Based Game for Collecting Music Metadata. Journal
of New Music Research 37, 15 (2008)
12. Turnbull, D., Liu, R., Barrington, L., Lanckriet, G.: A Game-Based Approach for
Collecting Semantic Annotations of Music. In: ISMIR (2007)
13. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: CHI 2004,
vol. 6, pp. 319–326. ACM, New York (2004)
14. Von Ahn, L., Kedia, M., Blum, M.: Verbosity: A game for collecting common-sense
facts. In: Knowledge Creation Diffusion Utilization, vol. 1, pp. 75–78. ACM Press
(2006)
Abstract. We have created story-based exercise games utilizing light and sound
to encourage children to participate in physical exercise in schools. Our reason-
ably priced technological setup provides practical and expressive means for
creating immersive and rich experiences to support physical exercise education
in schools. Studies conducted in schools showed that the story and drama ele-
ments draw children into the world of the exercise game. Moreover, children
who do not like traditional games and exercises engaged in these activities. Our
experiences also suggest that children’s imagination plays a great role in the de-
sign and engagement into exercise games, which makes co-creation with child-
ren a viable and exciting approach to creating new games.
1 Introduction
There is great need to encourage children and adolescents to engage in physical ac-
tivity. Childhood obesity is a serious and increasing challenge to public health [1] and
regular physical activity in childhood and adolescence is shown to improve health and
quality of life [2]. Physical Education (PE) classes in schools play an important role in
guiding children to lead a life with healthy amount of exercise, but the actual time
available for physical activity can be low [3]. Furthermore, there are children who
find physical exercise and sports unpleasant or uninteresting. Supporting the im-
provement of physical abilities through games could potentially increase the chances
of engaging in and benefitting from the positive outcomes of physical activities [4].
One way to foster health-related behavioral change is to use video games designed for
this purpose [5]. Especially exertion-based games have been shown to stimulate phys-
ical activity in inactive children [3], and to increase energy expenditure over sedenta-
ry activities [6].
We have devised a game-based approach to physical exercises, where storytelling
and dramatic elements, such as interactive lighting, guide and motivate children. The
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 308–319, 2013.
© Springer International Publishing Switzerland 2013
aim is to make physical activities more pleasant and motivating for children who find
current forms of exercise uninteresting or even intimidating. Our prototype is targeted
for 7–12-year-old schoolchildren and is played during a PE class under the supervi-
sion of a teacher.
We aimed at a solution that would be economically viable for schools. It consists
of a laptop computer, audio speakers, a wireless gaming controller, and a small set of
computer controlled lighting fixtures in a mobile trolley. The whole solution can be
assembled for about €1000. The setup, augmented with additional hardware like mo-
tion sensors and a projector, can be used for many other applications in schools, for
example for teaching mathematics or physics in a more immersive way.
The prototype has been studied in situ in PE classes and at an interactive science
fair, where several new games were co-designed with children. Our results indicate
that it is possible to create immersive and engaging, story-driven exercise games us-
ing a small set of lighting hardware and audio. From these simple elements, children’s
imagination can create rich experiences, which engage even the children who do not
enjoy usual PE class activities.
2 Related Work
With the introduction of the Nintendo Wii controller and the Microsoft Kinect, health
and activity related games have become popular. Brox et al. [7] differentiate between
genres of such games: educational games, persuasive games, and exergames. Educa-
tional games are primarily designed for improving health literacy. Persuasive games,
on the other hand, attempt to modify players’ behavior, be it increases in exercise or
adjustments of dietary habits. Finally, exergames are video games that are used in an
exercise activity [8]. Definitions of exergames state the games either encourage phys-
ical exercise [6] or their outcome depends on the physical activity [9]. Exergames can
be categorized according to dimensions such as the nature of the gaming aspects,
technological enablers, the type of physical activity, and engagement.
Exergame user interfaces range from free motion interfaces, i.e., games where the
players can freely move their body, via exercise equipment to traditional electronic
interfaces, and game worlds can be categorized as virtual, like in traditional video
games, augmented reality, or reality based [10]. Most of the existing exergames re-
viewed by Yim and Graham [10] are based on a virtual world or augmented reality,
and are either free motion interfaces or utilize some form of equipment.
To be successful, exergames must both make the players exercise effectively and
attract them to play long and often enough [8]. Sinclair et al. [8] provide dual flow
model, based on the concept of flow state [11], which combines attractiveness and
effectiveness, and focuses on challenge level and skill level both from the psychologi-
cal and physiological sides. Baranowski et al. [5] list features that make games
appealing and potentially efficient as behavior change tools, interactivity and persona-
lized, interactive goal setting and tailored feedback being the key aspects. However,
they place more focus on the immersive and attention-maintaining nature of games
and identify stories and fantasy as tools for reaching this goal.
Many researchers raise the issue of social aspects and feedback in exercise motiva-
tion. Bekker et al. [12] raise motivating feedback as their first design value. Allowing
players to create their own game goals can be greatly beneficial for the longevity of
the exergame, and supporting social interaction patterns is important. Park et al. [13]
supported interpersonal synchrony to leverage the positive social effects of “improv-
ing rapport and entitativity” and also make exercises more enjoyable. Social aspects
are also noted as important for improving motivation by Yim and Graham [10]. They
instruct to avoid systemic barriers to grouping and to actively assist players in form-
ing groups. They also suggest that music should be used, and that leadership should
be provided for novice players. Players should also be provided achievable short and
long-term goals, but at the same time the design should hide players’ fitness level to
minimize demotivating messages.
The guidelines mentioned above help make the games attractive to players, but to
ensure effectiveness, also the physical activity should be appropriately designed.
The design should consider physical exercise parameters (intensity); the game must
be playable for the required time period (duration); and the game should provide
structure, where proper warm-up and cool down take place in addition to the actual
exercise [8]. The exercise level can be adjusted per player either by using appropriate
sensors, e.g., by a heart rate monitor, or by collecting explicit user input. This way,
exercise levels can be balanced according to user’s fitness, motivation, and goals.
The proposed game concept is designed for physical education classes in grade school
where the entire class can play the game together. The game uses lighting hardware
that can project light on different parts of the room. The lighting and audio create an
immersive story environment. In the games we have built, stories have a central role,
and the games ask players to cooperate and work towards a common goal instead of
competing with one another. The games do not have direct response from players’
actions to game events. Instead, the teacher acts as the intermediary and controls the
progress of the game using a wireless controller. Children were often unaware of the
human in the loop in our evaluations. The use of human supervisor minimizes tech-
nical challenges while keeping the game interactive. This also improves safety since
the teacher can stop the game in case of any hazards.
Our main goal is to address the children who do not like the usual activities that
take place during the PE classes. The challenge has been addressed before by using
exergames by Fogel et al. [3]. Many children can feel that they are not performing
well enough in the environment where other children are watching their activities and
tend to react by minimizing their participation. This low self-efficacy aspect has been
identified as an important factor in demotivating people [14]. Our game design has no
explicit goal of behavioral change, but activating inactive children and providing
positive exercise experiences will hopefully help them towards a more active lifestyle.
To encourage participation, the game asks all players work together towards a shared
goal. A single player is never in focus and actively observed by the others. The game
is designed to capture the focus and most of the time the players follow the moving
lights. The room is dark during the game, except for the lit areas, and the darkness
provides some comfort to those shying from attention. Finally, the immersive story
draws the players in so that they want to participate.
4 Technical Setup
The system consists of a laptop PC, audio speakers, a Playstation Move® wireless
gaming controller, a moving head light fixture, and optionally fixed lights and a
projector. The hardware is mounted on a 1 by 1 by 1.8 meters (w, d, h) sized trolley
(Fig. 1). The trolley enables easy transportation of the hardware and acts as a physical
extension of the virtual characters in the story.
The moving head light (Stairville RoboHead X-3 LED) on top of the trolley has
three independent 540-degree joints, and can rotate 360 degrees in about 2 seconds. It
creates a sharp, round spot with a diameter of 40 centimeters when pointing down on
the floor next to the trolley. Light output is about 15,000 lux at one meter. Color fil-
ters alter the spot color while brightness can be adjusted gradually. Optional fixed
lights include four RGB LED Par lights on the lowest level of the trolley that illumi-
nate the floor on each side, and an RGB LED flood light on the second highest shelf
that points upwards to an inverted pyramid shaped reflector.
(Fig. 2). Games are modeled as state machines using XML markup. Entry to each
stage can produce sequences of lighting adjustments and audio. Moves to new stages
can be triggered with timers and by pressing buttons on the controller. Scripting can
be used for more complex logic.
Audio consists of speech generated with a speech synthesizer and a set of audio
files to provide music, sound effects and soundscape. In our initial game, a Finnish
language synthesizer by BitLips was utilized. The speed and pitch of the voice were
modified to create two different voices. In an extended version, audio files pre-
generated with BitLips, Acapela and Loquendo synthesizers were used, different syn-
thesizers playing the different roles. Distortion and echo was added to the BitLips
files to make it sound more menacing. The use of speech synthesis, as opposed to
prerecorded audio with voice actors, enables fast prototyping and quick changes to the
game. This was seen more beneficial than the improved dramatic effect of recorded
actors’ voices. In particular, the use of a synthesizer enabled us to prototype games
during workshops, as they were co-created with children (see Section 7).
In addition, background music tracks loop during the different parts of the story,
the story characters have identifying soundscapes that play when they enter and there
are some sound effects for story and game events.
Playstation Move controller is used to control the progress of the game. At the end
of each activity, the operator presses a button when the task is finished. Two buttons
are used so that the operator can mark the task success quality (“okay” or “excel-
lent”). Additional button can be used to replay the last instruction in case the players
had trouble understanding instructions or there was some mishap, which interrupted
the game. This type of controller was chosen because it provides a wireless method of
control and does not draw unnecessary attention to technology, but can rather be inte-
grated into stories, for example, as a magic wand.
In this version the game starts with an introduction by a narrator. When the actual
game starts, the Light character guides the players and the story goes through four
stages: awakening, empowerment, calling, and battle. The awakening part consists of
four slow stretching and warm-up exercises. Empowerment contains four more active
and physical exercises. Calling part is even faster, containing lot of movement around
the space in its four activities. Finally, the battle is the most physical part, consisting
of four tasks with fast movements in various ways. Once the story part finishes in the
end of the battle, the narrator returns and takes the players through a cool-down phase
consisting of three slow, relaxing activities. Each part has its own music, and music
style and tempo match the level of activity aimed for the part.
The second version incorporates the five fixed RGB lights, four to illuminate areas
on the floor and one to provide overall illumination to the room. Use of a reflector
pyramid also created a strong visual point in the tower. We also added a projector and
projection screen to display a signature image for both the Shadow and the Light cha-
racters, stars to signify players’ success, and images of animals and elements that are
awakened and called to help the players during the story.
In this version, the teacher rates activities on binary scale, each activity rated to be
either acceptable or great performance. The feedback is given immediately by speech
output and after each exercise set, one, two or three stars are displayed with the pro-
jector based on the performance.
6 Evaluations
Method. Subjective questionnaire and interview data was gathered. Almost all partic-
ipants, 61 in total, filled in a questionnaire and most also participated in interviews,
which were conducted in small groups (about 5 persons). The questionnaire included
21 statements which were answered on a scale consisting of three smiley faces, i.e.,
happy, neutral and sad face. These statements concerned the overall thoughts of the
system (e.g., would they like to play again, was it boring), the fluency of the game
(e.g., did they understand the instructions, was it too slow), the physical strain of the
game (e.g., did they get winded, were the tasks too easy), and the atmosphere of the
game (e.g., was the atmosphere good, did they feel outsiders during the game). In
addition to the statements, an overall grade between 1 and 10 for the game was in-
quired, and open-ended questions were asked: the participants were able to tell what
was best and worst in the game, and how could it be made more interesting.
Results. Our findings indicated that the basic concept works well: the system received
a favorable average overall grade of 6.84 (SD=2.199), and although some of the
children testing the system were a bit older than our target age of 10 years, they were
still observed to get into the game and enthusiastically participate in the exercises.
Both interviews and observation showed that the current design had too much waiting
and too little exercise. Addition of feedback on performance was the most common
request in the interview feedback: the static structure of the game provided no feed-
back on players’ activities; they could not fail or affect the outcome in any way.
Other opportunities for improvement were also identified. Use of speech synthe-
sizer made the speech somewhat hard to understand, the two characters sounded too
similar which reduced the emotional effect of the story. This was found both from
player feedback and observations. Also, the group size of twenty was too big for the
combination of the activities and the very small gym where the tests took place. Con-
gestion resulted when groups were supposed to move into the same area and especial-
ly when all players chased the light spot as one group.
Overall, while many ways to improve the experience were found, our observation
of the tests showed that the fundamental concepts, i.e., the use of audio and lighting
created a very powerful effect, immersing the players the very moment the game
started. Based on the results, we continued the development.
The second version was evaluated in a different school with 110 participants (56 girls,
54 boys) over the course of a week. The ages ranged from 1st graders all the way to 6th
graders, i.e., the participants were 6–11 years old with a mean age of 9.1 years
(SD=1.1). The participant groups were classes either as one or two groups. Almost all
participants (97%) liked physical exercise and 77% reported to exercise in their free
time, while 42% practiced some team sport. These background variables were not
affected by gender. However, boys were more active players of videogames than girls
out of the participants who reported playing videogames (74% of respondents).
The teacher was present in most cases but the game was introduced and controlled
by a researcher. Additional researchers were present and sessions were videotaped.
After the introduction, the researcher remained silent, unless there were significant
troubles, which occurred only in a few cases. The fact that the researcher was rating
the performances was not told to the children. The game was again updated slightly
after the first day, shortening or splitting some of the longest instructions. Some were
so long, that the players could not remember all the relevant information and got
bored. We split such instructions into two parts and included first part of the activity
in between, where possible.
The overall grade for the game was reported with a five-step smiley face scale rang-
ing from extremely sad to extremely happy as an answer to the question “How much
did you like of the game as a whole?” Background questions included age and gender,
do they play videogames, do they like physical exercise, do they do exercise in their
free-time and do they practice some team sport (e.g., football, ice hockey).
Results. The median overall grade for the game was 5 out of 5, and almost 60% of the
participants gave the system the extremely happy face. Neutral or a little sad face was
selected only by 16% and none of the participants selected the extremely sad face.
The participants would like to move this way again (76%) and they felt that exercis-
ing was now more pleasant than usually on gym classes (72%). The majority (66%)
experienced the music and the voices to be compelling, and even more (78%) saw the
lights of the game to be compelling. Only 6% of the participants reported they felt
like an outsider in the game and only 5% stated playing the game was hard.
There were several statistically significant (p < 0.05) interactions between the ex-
perience variables (according to Pearson Chi-Square test, “I don’t know” responses
set as missing values). For example, whether the participants would like to move this
way again had an effect on almost all the other responses as well: only difficulty of
playing (statement 1), understanding the speech (5) and feeling an outsider (13) did
not interact with willingness to move this way again. Similarly, the overall rating
affected almost all other experiences while only difficulty of the game (1), ability to
move with own style (12) and feeling an outsider (13) had no interaction with the
overall rating of the system.
Remarkably, all 17 participants who did not exercise in their free-time felt that ex-
ercising now was more pleasant than usually on gym classes (3). We also observed
gender related differences in experiences. Girls felt more often (in fact, all girls) that
exercising was now more pleasant than usually on gym classes (X2=17.848, df=1,
p=0.000) and they also liked the game as a whole clearly more than the boys. Boys,
on the other hand, experienced the speech sound less frequently pleasant (X2=4.262,
df=1, p=0.039) and the music and voices less compelling than the girls (X2=5.643,
df=1, p=0.018).
Age seems to have had slight effect on almost all of the statements: older partici-
pants received the system a bit more negatively. The differences in other statements
are not that surprising as the story approach of the game may feel a little childish for
the oldest children. The only statements that were not significantly affected by age
were difficulty of playing the game (statement 1), easiness of the exercise tasks (11),
ability to move with own style (12), and feeling an outsider (13).
fixture (Martic Mac 300) and four fixed lights RGBW LED fixtures. A pair of active
speakers was used for audio and the games were controlled by a researcher operating the
laptop where the games were created and run.
In each workshop, a group of 10 to 15 schoolchildren participated and a new game
was created. Each workshop lasted 70 to 85 minutes. Participants were free to leave at
any time, and in some cases about a third of the participants left after half an hour.
Each workshop started with an introduction where the functionality of the lighting
hardware was demonstrated. Next, an example game (a 10-minute shortened version
of our extended game) was played and after this the actual game design started. The
game was designed and implemented during the workshop so that different parts of
the game were tested during the development and the full game was played at least
once in the end. The created games were finalized after the workshop by adding miss-
ing parts like instructions, which were unnecessary for the participants themselves,
and fixing other remaining issues. The resulting games were then tested on the last
day of the event when interested event visitors were allowed to play the games.
Landry et al. [16] note that it is a challenge to make children aware of the physical
and virtual potential of interactive technology. Like them, we made children try out
the system to foster understanding of the system, and in addition added an introduc-
tion of the technology. However, we went mostly story first and this seemed natural
to children. Our process was different from that of Landry et al. in that we worked all
the time as one group. This was problematic in larger groups since not everybody
could provide their input, and splitting into smaller groups could have worked better.
In the end, the exercise and gameplay ideas were less imaginative and followed more
on what was in the example game, which suggest the children were biased by their
initial exposure to the game environment. However, when testing the games on the
last day of the event with interested children, we found that the created games worked
well. In particular, the very simple description of the game world (e.g., “You are in
the jungle, you are small monkeys.”), together with lights, was enough to create im-
mersive environments as children’s imagination did the rest. The players also did the
necessary interpretations to figure out ambiguous instructions.
We built light and sound based exercise games for children. Our evaluations showed
that it is possible to create strong, immersive experiences with this technology, capa-
ble of pulling children into the world of a story. The use of simple graphics did not
seem to significantly improve the effect, at times the opposite. Speech was found to
be a powerful way to tell a story while sound effects are very important in building
the atmosphere. The story does not need to be told in detail; very simple descriptions
of stereotypic scenarios are enough as players’ imagination takes care of the rest. This
means that creating new games does not require particular skills in storytelling. The
most important part in writing games is to keep the length of the instructions short and
keep players active. The story should incorporate activity and the instructions on what
to do should be given at the time the players are supposed to do something.
The basic concepts seemed to work exactly as envisioned, with children participat-
ing in the physical activities even if they did not usually like sports and such children
reported liking the game more than the usual PE class activities. The facts that the
games did not contain strong competitive elements and that the focus is on lighting
and the story seemed to help achieve this goal.
The age range of children to which the same game seemed to appeal to was a posi-
tive surprise. The original target group was children around 10 years of age. However,
even 6-year-olds could easily follow most of the instructions and were very interested
in the story. Among children approaching the age of 12, the number of persons consi-
dering the game too childish did increase, but many of those who “misbehaved” still
did so within the fiction of the game. Only a couple of the most mature children did
their best to remain uninterested in the world of the story.
Following the suggestions by Yim and Graham [10] was helpful to the overall suc-
cess: the application of music clearly made the experience more engaging and guided
players toward the tempo we aimed for in the different exercises. The game also gave
direct instructions to players, thus providing leadership. Early on, the users did not yet
follow the instructions from the system without hesitation. This changed when they
noticed that the game progressed when they did what was asked.
The current version of the game has the temporal extent of one exercise session,
i.e., less than an hour. This means the goals we provide to users are only short and
medium term goals. We plan to introduce long-term goals, for example by creating
alternative endings to the game depending on players’ performance. This could also
provide implicit exercise goals, while still hiding players’ fitness levels, which is
relevant in helping children with low self-efficacy. In our current solution, we sup-
ported players’ self-efficacy during group activity by keeping the story at the center of
attention and not putting any individuals in focus during the game. Together, the
above aspects created an atmosphere where everybody could enjoy the game without
worrying about their performance. The overall design focused on the entire group
working together to accomplish a common goal. In this sense the game facilitated
grouping and group support. In the second version, there were no elements separating
the players. This can also be considered a limitation, since the players could not get
the kind of support they could get by forming smaller groups.
Above all, exercise games must also be fun in order to be efficient [8, 10]. We feel
the game reached this goal. The lighting and audio provide an immersive and novel
experience and the spatial nature of the moving spot light naturally encourages physi-
cal activity. We also feel that the game matched reasonably well the players’ abilities.
Acknowledgements. This work is part of the “Active Learning Spaces” project,
funded by the Finnish Funding Agency for Technology and Innovation.
References
1. World Health Organization. Global Strategy on Diet, Physical Activity and Health,
http://www.who.int/dietphysicalactivity/childhood/en/
2. U.S. Department of Health and Human Services. Physical Activity Guidelines Advisory
Committee report. Washington, DC: U.S. Department of Health and Human Services
(2008)
3. Fogel, V.A., Miltenberger, R.G., Graves, R., Koehler, S.: The Effects of Exergaming
On Physical Activity Among Inactive Children In A Physical Education Classroom. Appl.
Behav. Anal. 43(4), 591–600 (2010)
4. Peer, F., Friedlander, A., Mazalek, A., Mueller, F.: Evaluating technology that makes
physical games for children more engaging. In: 10th International Conference on Interac-
tion Design and Children (IDC 2011), pp. 193–196. ACM, New York (2011)
5. Baranowski, T., Buday, R., Thompson, D.I., Baranowski, J.: Playing for real: Video games
and stories for health-related behavior change. Am. J. Prev. Med. 34(1), 74–82 (2008)
6. Whitehead, A., Johnston, H., Nixon, N., Welch, J.: Exergame effectiveness: What the
numbers can tell us. In: Spencer, S.N. (ed.) 5th ACM SIGGRAPH Symposium on Video
Games (Sandbox 2010), pp. 55–62. ACM, New York (2010)
7. Brox, E., Fernandez-Luque, L., Tøllefsen, T.: Healthy Gaming – Video Game Design to
Promote Health. Appl. Clin. Inform. 2(2), 128–142 (2011)
8. Sinclair, J., Hingston, P., Masek, M.: Considerations for the design of exergames. In: 5th
International Conference on Computer Graphics and Interactive Techniques in Australia
and Southeast Asia (GRAPHITE 2007), pp. 289–295. ACM, New York (2007)
9. Mueller, F., Edge, D., Vetere, F., Gibbs, M.R., Agamanolis, S., Bongers, B., Sheridan,
J.G.: Designing sports: A framework for exertion games. In: SIGCHI Conference on Hu-
man Factors in Computing Systems (CHI 2011), pp. 2651–2660. ACM, New York (2011)
10. Yim, J., Graham, T.C.N.: Using games to increase exercise motivation. In: 2007 Confe-
rence on Future Play (Future Play 2007), pp. 166–173. ACM, New York (2007)
11. Csikszentmihalyi, M., Csikszentmihalyi, I.S. (eds.): Optimal experience: Psychological
studies of flow in consciousness. Cambridge University Press (1992)
12. Bekker, T., Sturm, J., Eggen, B.: Designing playful interactions for social interaction and
physical play. Personal Ubiquitous Comput. 14(5), 385–396 (2010)
13. Park, T., Lee, U., Lee, B., Lee, H., Son, S., Song, S., Song, J.: ExerSync: Facilitating inter-
personal synchrony in social exergames. In: 16th ACM Conference on Computer Sup-
ported Cooperative Work and Social Computing (CSCW 2013), pp. 409–422. ACM, New
York (2013)
14. Hagger, M.S., Chatzisarantis, N.L., Biddle, S.J.: A meta-analytic review of the theories of
reasoned action and planned behavior in physical activity: Predictive validity and the con-
tribution of additional variables. J. Sport and Exercise Psychol. 24(1), 3–32 (2002)
15. Hakulinen, J., Turunen, M., Heimonen, T.: Light Control Architecture for Multimodal In-
teraction in Physical and Augmented Environments. In: DIS 2012 Workshop on Designing
Interactive Lighting (2012)
16. Landry, P., Parés, N., Minsky, J., Parés, R.: Participatory design for exertion interfaces for
children. In: 11th International Conference on Interaction Design and Children (IDC
2012), pp. 256–259. ACM, New York (2012)
1 Introduction
The conclusion of a game session derives from a series of decisions and actions made
throughout the game. In many situations, analyzing and understanding the events,
mistakes, and fluxes of a concrete gameplay experience may be useful for understand-
ing the achieved results. A game flux analysis may also be fundamental for detecting
symptoms of problems that occurred due to wrong decision-making or even bad ga-
meplay design. Besides that, without any formalized process, this type of analysis
may be subjective and, depending on the game dynamics and its complexity, it would
require playing the game successively, making the same decisions, to intuitively guess
which ones were responsible for generating the observed effects. Thus, reproducing
the same state can be unviable, making it difficult to replay and identify, in a trial and
error approach, the source of the problem. In addition, examining the game flux might
allow for the identification of good and bad attitudes made players. This knowledge
can be used in future game sessions to avoid making the same mistakes or even to
adjust gameplay features.
The analysis process for detecting gameplay issues is done nowadays in an artisan-
al way by using a popular beta testing [1] approach. The beta test phase is an indis-
pensable source of data for the developers about technical issues or bugs found in
the game. Normally, beta testers are volunteers who were recruited to play the game
in an early, pre-release, build of the game where they can provide information about
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 320–331, 2013.
© Springer International Publishing Switzerland 2013
technical issues and provide feedback about the gameplay mechanics. Thus, beta test-
ing is a crucial part of the development to identify important issues in the game.
However, developers have little control over the beta testers’ gameplay experience or
the environment because they can play at home.
The goal of this paper is to improve the game designer’s understanding of the game
flux, providing insights on how the gameplay progressed and influences in the out-
come. In order to improve understanding, we provide the means to analyze the game
flux by using provenance. The provenance analysis requires processing the collected
gameplay data and generating a provenance graph, which relates the actions and
events that occurred during the game session. This provenance graph allows the user
to identify critical actions that influenced the game outcome and helps to understand
how events were generated and which decisions influenced them. This analysis could
be used in conjunction with the beta testing in order to aid in the identification of
gameplay or technical issues, allowing the designer to analyze the tester’s feedback
report and the gameplay data from the game session.
In our previous work [2], we introduced the usage of digital provenance [3] in
games. The main goal of the previous work was to propose a framework that collects
information during a game session and maps it to provenance terms, providing the
means for a post-game analysis. This was the first time that the provenance concept
and formalization was used in the representation of game flux. The present paper is
based on the conceptual framework definition introduced in the previous paper. How-
ever, while in the previous work focused on the provenance gathering, this work fo-
cuses on the provenance graph construction and manipulation to support analysis.
Even though the scenario used in this paper is over a serious game, we believe that the
concepts discussed here are applicable to any kind of game and are useful to support
advanced game flux analysis, such as gameplay design and balancing, data mining,
and even for storytelling.
This paper is organized as follows: Section 2 provides related work in the area of
game flux analysis. Section 3 provides a background on provenance, and Section 4
introduces our framework for provenance gathering. Section 5 presents our approach
for provenance visualization through graphs. Section 1 presents the adoption of prov-
enance visualization in a software engineering game, with visualization examples.
Finally, Section 7 concludes this work and points out some future work.
2 Related Work
In the digital game domain, Warren [4] proposes an informal method to analyze the
game flux using a flux graph, mapping game actions and resources into vertices. By
his definition, resources are dimensions of the game state that are quantifiable, while
actions are rules of the game that allowed the conversion of one resource into another.
Consalvo [5] presents a more formal approach based on metrics collected during the
game session, creating a gameplay log to identify events caused by player choices.
Playtracer [6] allows the user to visually analyze play steps, providing detailed visual
representation of the actions taken by the player through the game.
These three approaches are developer-oriented, meaning that they aim to improve
the quality of the game by providing feedback to the development team. However,
Consalvo [5] presents a template for analysis, acting as guidelines to how the analysis
should be done. Meanwhile, Playtracer [6] focuses on identifying the player’s strate-
gies by visually analyzing play traces instead of using queries.
Lastly, the Game Analytics [7] from Unity3D [8] allows visualizing game data as
heat maps directly on the scene. This identifies bottlenecks and hotspots, and unde-
rused and overused areas of the game. It also measures the game retention rate, where
players are stopping playing, and how the game progression develops.
3 Provenance
Provenance is well understood in the context of art or digital libraries, where it re-
spectively refers to the documented history of an art object, or the documentation of
processes in a digital object's life cycle [9]. In 2006, at the International Provenance
and Annotation Workshop (IPAW) [10], the participants were interested in the issues
of data provenance, documentation, derivation, and annotation. As a result, the Open
Provenance Model (OPM) [11] was created during the Provenance Challenge [12],
which is a collocated event of IPAW. Recently, another provenance model was de-
veloped, named PROV [13], which can be viewed as the successor of OPM. Both
models aim at bringing provenance concepts to digital data.
Both provenance models assume that provenance of objects is represented by an
annotated causality graph, which is a directed acyclic graph enriched with annota-
tions. These annotations capture further information pertaining to execution. Accord-
ing to [11], a provenance graph is the record of a past or current execution, and not a
description of something that could happen in the future. The provenance graph cap-
tures causal dependencies between elements and can be summarized by means of
transitive rules. Because of this, sets of completion rules and inferences can be used in
the graph in order to summarize the information.
In order to adopt provenance for the context of games, it is necessary to map each
type of vertices of the provenance graph into elements that can be represented in
games. As mentioned in section 3, OPM and PROV use three types of vertex: Arti-
facts/Entities, Process/Activities, and Agents. In order to use these vertex types, it is
first necessary to define their counterparts in the game context. To avoid misunders-
tanding, we adopt throughout this paper the terms used in PROV (entities, activities,
and agents).
In the context of provenance, entities are defined as physical or digital objects. Tri-
vially, in our approach they are mapped into objects present in the game, such as
weapons and potions. In provenance, an agent corresponds to a person, an organiza-
tion, or anything with responsibilities. In the game context, agents are mapped into
characters present in the game, such as non-playable characters (NPCs), monsters,
and players. It can also be used to map event controllers, plot triggers, or the game’s
artificial intelligence overseer that manages the plot. Thus, agents represent beings
capable of making decisions, while entities represent inanimate objects. Lastly, activi-
ties are defined as actions taken by agents or interactions with other agents or entities.
In the game context, activities are defined as actions or events executed throughout
the game, such as attacking, dodging, and jumping.
With all three types of vertex mapped into the game context, it is also necessary to
map their causal relations to create the provenance graph. The PROV model defines
some causal relations that can be used similarly to their original context. However, it
also provides rules to extend these relationships or to create new ones. For instance, it
is possible to create relationships to express the damage done to a character or rela-
tionships that affect specific core mechanics from the game, like attack rolls, healing,
and interactions with NPCs or objects. Also, the PROV model deals well with the
aspect of time, which can be heavily explored in games, especially on games focused
on storytelling.
Each NPC in the game should explicitly model its behavior in order to generate
and control its actions, providing an array of behavior possibilities. For example, de-
cision trees [14] can be used to model the NPC’s behaviors. With this explicit model,
a behavior controller can register information about the action when it is executed.
Actions can be represented by a series of attributes that provide a description and the
context of the action, allowing the creation of a provenance graph. As illustrated by
Fig. 1, every action needs some information: a reason for its existence, why the action
was performed, what triggered it, and who performed the action. In addition, the time
of its occurrence can be important depending of the reason of using provenance. The
main reason of using provenance is to produce a graph containing details that can be
tracked to determine why something occurred the way it did. Therefore, with this
assumption, the time of the action, the person who did it, and the effects of the action
can be recorded for future analysis.
Fig. 1. Data model diagram. Gray classes represent generic provenance classes.
Events also work in a similar way as actions, with the difference in who triggered
them, since events are not necessary tied to characters. For objects, its name, type,
location, importance, and the events that are generated by it can also be stored to aid
in the construction of the graph. Lastly, agents can have their names, attributes, goals,
and current location recorded. The information collected during the game is used for
the generation of the game flux log, which in turn is used for generating the prove-
nance graph. In other words, the information collected throughout the game session is
the information displayed by the provenance graph for analysis. Thus, all relevant
data should be registered, preferentially at fine grain. The way of measuring relevance
varies from game to game, but ideally it is any information that can be used to aid the
analysis process.
5 Provenance Visualization
Different formats can also be used for edges, as well as colors. The thickness
shows as how strong the relationship is. A thin edge represents a low influence on the
activity. On the other hand, the higher the influence is, the ticker the edge. This fea-
ture can be used to quickly identify strong influences in the graph just by looking at
the edge’s thickness. The edge’s color is used to represent the type of relationship,
which can be any of these three types: positive, which indicates a beneficial relation;
negative, which is a prejudicial relation; and neutral, which is neither beneficial nor
prejudicial. For each type of relationship (positive, negative, and neutral), a different
color is used. Green is used for positive influences, red for negative, and black for
neutral. Lastly, dashed edges represent edges without values, which are association
edges such as the edges binding activities to an agent. These edge types are also illu-
strated in Fig. 2, where neutral edges are dashed to emphasize their lack of impor-
tance.
In order to better analyze graph data, the vertex filter feature is also available.
Since the graph is generated from collected game data, not all collected information is
relevant for every type of analysis. Thus, the provenance graph might contain actions
that did not provoke any significant change. These elements act as noise and can be
omitted during analysis. To do so, it is possible to collapse vertices in order to reduce
the graph size by changing the information display scale, grouping nearby vertices
together and thus changing the graph granularity. Another usage of collapse is to
group activities from the same agent, improving visibility of all influences and
changes that the agent did throughout the game. Similar edges that have the same
target are also grouped together when collapsing vertices, as shown in Fig. 3. The
collapsed edge’s information is calculated by the sum or average (depending on the
edge type) of the values from the collapsed edges. For example, edges representing
expenses uses sum, while edges representing aid modifiers (in percentage) uses aver-
age. Another type of filter present in Prov Viewer is the edge filter, which filters
edges by context and by the type of relationship.
The last feature present is the attribute status display. When selecting the desired
attribute, all vertices with the specified status have their colors changed according to
their respective values. It uses the traffic light scale [16], which indicates the status of
the variable using three colors: red, yellow, and green. As an example, imagine that
Fig. 3. Collapsing vertices. The original state showing four activities and two entities with
edges of the same type (a), the collapse of both entities into one (b), and the collapse of two
activities, and their respective edges since they were from the same type (c). The size of the
resulting edge is bigger than the original ones as a resulting from summing each edge’s values.
we desire to analyze the player’s financial situation throughout the game. When fil-
tered by this status, all vertices that contain a player financial value have their colors
changed according to their values. Activating this type of feature allows the developer
to see the player’s finances throughout the game, making it easier to identify situa-
tions where he might have had financial problems (red color). Section 1 provides
more examples of those features.
Using these features for graph manipulation and visualization, the developer is able
to interact with the provenance graph, identifying relevant actions that had an impact
in the story or in the desired type of analysis. It can also be used to analyze player’s
behavior, detecting situations that the player had difficulties or didn’t behave accord-
ing to the developer’s plan. The next subsection describes alternatives to deal with
problems related to graph size and visualization.
5.1 Granularity
Depending on the game style, a game session might take several hours or days to be
completed. This makes the size of the provenance graph overwhelming, even when
making pre-filtering during the generation of the game flux log. One way to avoid
such situations is to show the provenance graph with some filters selected instead of
its full extension. For example, before showing the graph to the user, it is possible to
collapses less relevant vertices to reduce the graph’s size. For instance, combats stag-
es can be identified and collapsed into a single vertex for each instance. Places visited
in the game can also be collapsed into a single vertex, containing all interactions made
in that location. It is also possible to collapse collapsed vertices. In this case, a col-
lapsed combat inside a collapsed area visited by the player may contain other actions
aside from the combat, such as interactions with the ambient. This gives an impres-
sion of a map from the player’s journey, showing vertices for each location visited by
the player, while allowing the developer to expand only the situations he desires to
analyze.
Instead of collapsing all combats and locations, filters can be used to decide which
combats or locations were not relevant to the story, or had no noticeable impact in the
player’s journey, while keeping important events visible to the developer. This is
possible because provenance is analyzed from the present to the past. This way, com-
bats outcomes are known and can be used to decide if they are relevant or not. If the
player was victorious with minor challenge, did not suffer severe wounds, or barely
used any resources at his disposal, then the entire combat can be simplified into just
one vertex representing the combat with the enemy. However, if the combat was chal-
lenging or the player lost, it may be interesting to display all actions for a correct
analysis, allowing the player to identify important facts that influenced the combat
outcome. Note that this type of filter is heavily dependable of the game context, so a
specific set of filters should be implemented for each individual game.
6 Case Study
Fig. 4. An example of credits status filter (a) and the non-collapsed provenance graph (b) using
filter: Morale. Brown activity vertices represent employees leaving the player’s staff.
Fig. 4 was already subject of an attribute status display and a filter to show the
player’s credits status, both in the edges and in the vertices. In vertex 1, the project
had a substantial financial income and a new employee was hired, as marked by
the thick green edge to an agent and thick red edge to a gray dotted activities. The
player’s credits are also in a green zone as marked by the project’s vertex color. How-
ever, due to the hiring fee paid in vertex 1 and the resources used by the staff in vertex
2, the player’s credits changed to a yellow zone, even with the minor income from
agent Ag.
In vertex 3, the player’s credits changed to red zone due to payments, meaning that
his resources are almost empty and he will not have enough credits to keep paying his
employees. When that happens, employees’ morale is lowered due to the lack of
payment and if it reaches red zone, they can resign, as shown by brown activities.
Observing Fig. 4, we can see the employees’ morale getting lower by lack of pay-
ment. This helps us to understand why they resigned. Without credits to hire new
employees and without a staff, the player loses the game.
This analysis can be used to detect player’s behaviors and the reasons of why they
lost the game. In the example, the cause was the lack of resources due to hiring a new
employee. If it was necessary to hire a new employee, then there is a problem that
requires immediate attention. However, hiring an employee causes the player to lose
1
In order to reduce graph size and provide a quicker understanding for the presented exam-
ples, some in game parameters were configured to allow faster state transitions.
the game, leading to the conclusion that if hiring is optional, then some changes might
also be required because the penalty is too severe and causes the player to lose, in-
stead of giving only a small setback.
Another example of analysis is checking the employee productivity and under-
standing why variations occurred by using multiple filters to test theories. In SDM,
productivity is defined by the executed task, the amount of outside help, the em-
ployee’s job (junior, mid-level, and senior), the working hours, and the stamina and
morale stats. Fig. 5 illustrates an example scenario. To simplify the graph visualiza-
tion due to size limits, we focus only on two agents and the entity known as “project”.
Those agents’ roles are programmer and manager, with the manager acting as a sup-
porting role for the programmer.
Analyzing the picture we can see that the programmer’s productivity fluctuated
throughout vertices 1 to 7. We can also see that the manager did not cause this fluctu-
ation, since his aid bonus did not have much variation. In vertex 2, the programmer
maximizes his productivity at the cost of quality. This information, as well as other
details about the vertex, is displayed in the vertex’s tooltip. The change in vertex 3
can be identified by observing his working hours, which can be done by looking at
each individual vertex or by adding a filter, as shown in Fig. 5.
Fig. 5. Example of a provenance graph analysis. The entity is project’s stages of the develop-
ment. Agents are employees from the development staff, with the programmer being the upper
agent and the manager the lower one. Graph using different display features: default mode
(gray activity vertices) displaying edge values (a), working hours (b), stamina (c), and morale
visualization modes (d). Red vertices represent low, yellow represent average, and green
represent high values.
In Fig. 5 one can see via the change from yellow to red that the programmer’s
working hours per day increased. Since the activity in vertex 3 is red, it means the
employee is doing extra hours, which increases his productivity. From vertices 3 to 7,
his working hours remained unaltered. Therefore, the change from vertices 2 to 3 was
mainly due the change on his daily working time. However, if we look at vertex 4, we
can see a drop in his productivity.
By changing the filter again to show stamina levels, we can see that in vertex 3 the
programmer’s stamina dropped to yellow because of the extra hours and in vertex 4 it
reached red due to exhaustion. Another side effect of his exhaustion was the change
on the programmer’s morale, which also reached the red zone in vertex 5. Lastly, the
small variation from vertices 5 to 7 comes from a random range modifier during
productivity computation, since the programmer is already working at minimal levels
at the current configuration. With both the morale and stamina at lowest levels, the
extra hours were not compensating his productivity loss. As previously shown, if his
morale levels do not increase, the programmer might resign. This example of analysis
covered all possibilities that affect a programmer’s behavior and can be used to fur-
ther refine game modifiers or state transitions, as well as identifying odd behaviors
caused by game modifiers.
7 Conclusion
This paper introduces new perspectives on gameplay modeling and analysis, leverag-
ing the current state of the art, based on gameplay, to a level where the game prove-
nance can aid the detection of gameplay issues. This knowledge can help on (1)
confirming the hypotheses formulated by the beta tester, (2) supporting developers
for a better gameplay design, (3) identifying issues not reported by testers, and (4)
data-mining behavior patterns from individual sessions or groups of sessions.
The provenance visualization can happen on either on-the-fly or post-mortem sessions.
It allows the discovery of issues that contributed to specific game fluxes and results
achieved throughout the gaming session. This analysis can be used on games to improve
understanding of the game flux and identifying actions that influenced the outcome, aiding
developers to understand why events happened the way they did. It can also be used to
analyze a game story development, how it was generated, and which events affected it in
the following game genres: role-playing, strategy, and simulation.
Currently, we do not make inferences to the user, but let the user or developers de-
cide what needs to be inferred. However, we provide the necessary tools to create
inference rules, like filters and collapses (both for vertices and edges). Studies in this
area can be made in order to identify information that can be omitted from the user
without affecting the overall analysis. Another interesting research is to automatically
identify patterns in the game flux. Lastly, we are working on different graph visuali-
zation layouts and also studying the possibility of using game provenance in educa-
tional digital games to aid in the understanding of the concepts taught in the game.
Acknowledgments. We would like to thank CNPq, FAPERJ, and CAPES for the
financial support.
References
1. Davis, J., Steury, K., Pagulayan, R.: A survey method for assessing perceptions of a game:
The consumer playtest in game design. Game Studies 5 (2005)
2. Kohwalter, T., Clua, E., Murta, L.: Provenance in Games. In: Brazilian Symposium on
Games and Digital Entertainment (SBGAMES) (2012)
3. Freire, J., Koop, D., Santos, E., Silva, C.T.: Provenance for Computational Tasks: A Sur-
vey. Computing in Science Engineering 10, 11–21 (2008)
4. Warren, C.: Game Analysis Using Resource-Infrastructure-Action Flow,
http://ficial.wordpress.com/2011/10/23/game-analysis-using-
resource-infrastructure-action-flow/
5. Consalvo, M., Dutton, N.: Game analysis: Developing a methodological toolkit for the qu-
alitative study of games. Game Studies 6 (2006)
6. Andersen, E., Liu, Y.-E., Apter, E., Boucher-Genesse, F., Popović, Z.: Gameplay analysis
through state projection. In: Foundations of Digital Games (FDG), pp. 1–8 (2010)
7. Wulff, M., Hansen, M., Thurau, C.: GameAnalytics For Game Developers Know the facts
Improve and Monetize, http://www.gameanalytics.com/
8. Higgins, T.: Unity - 3D Game Engine, http://unity3d.com/
9. PREMIS Working Group: Data Dictionary for Preservation Metadata. Implementation
Strategies (PREMIS), OCLC Online Computer Library Center & Research Libraries
Group (2005)
10. Moreau, L., Foster, I., Freire, J., Frew, J., Groth, P., McGuiness, D.: IPAW,
http://www.ipaw.info/
11. Moreau, L., Clifford, B., Freire, J., Futrelle, J., Gil, Y., Groth, P., Kwasnikowska, N.,
Miles, S., Missier, P., Myers, J., Plale, B., Simmhan, Y., Stephan, E., den Bussche, J.V.:
The Open Provenance Model core specification (v1.1). Future Generation Computer Sys-
tems 27, 743–756 (2007)
12. Miles, S., Heasley, J., Szalay, A., Moreau, L., Groth, P.: Provenance Challenge WIKI,
http://twiki.ipaw.info/bin/view/Challenge/
13. Moreau, L., Missier, P.: PROV-DM: The PROV Data Model,
http://www.w3.org/TR/prov-dm/
14. Moret, B.: Decision Trees and Diagrams. ACM Computing Surveys (CSUR) 14, 593–623
(1982)
15. O’Madadhain, J., Fisher, D., Nelson, T.: JUNG: Java Universal Network/Graph Frame-
work, http://jung.sourceforge.net/
16. Diehl, S.: Software Visualization: Visualizing the Structure, Behaviour, and Evolution of
Software. Springer (2007)
17. Kohwalter, T., Clua, E., Murta, L.: SDM – An Educational Game for Software Engineering.
In: Brazilian Symposium on Games and Digital Entertainment (SBGAMES), pp. 222–231
(2011)
1 Introduction
The way how narratives and videogames are connected is still a topic that is open to
debate. Nevertheless, it is undeniable that over the years several videogames have
been used to tell stories. In fact, this topic started to catch attention in later years with
the appearance of textual adventure videogames, some kind of Interactive Fiction [1].
The term ludology has been proposed by Frasca [2] as a movement based on narra-
tology having different features with distinct merits. Therefore, the main question is
not if games are effectively narratives or not but how they relate to narratives. An
interesting recent approach on this question is a model presented by Aarseth [3].
Briefly, the author confirms videogames as an interactive fiction medium, whose
elements can be characterized in a ludo-narrative dimension. The different elements
can have a more meaningful (narrative) effect or a more challenging and amusing
(ludic) importance.
In the context of automatic level generation, existing algorithms focus the lower
abstraction concepts to provide adequate actions (Section 2), yet higher abstraction
concepts must be considered to create more intricate scenarios. We present our studies
in how to relate stories and story related concepts with automatic level generation to
produce more meaningful content, which include: the definition of a set of abstraction
layers in hero-centred stories (Section 3.1); a study on the platformers’ storytelling
capabilities (Section 3.2); the definition of higher level game design patterns for plat-
formers (Section 3.3); and a final reasoning on ways to promote more contextualized
automatic level generation in platform videogames applied to a prototype (Section 4).
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 332–343, 2013.
© Springer International Publishing Switzerland 2013
2 Related Work
Game design patterns have been used to understand the rules behind the mind of game
creators. This concept was introduced by Björk et al. [19] as commonly recurring parts
of the design of a game regarding gameplay. Naturally, each genre has particular de-
sign patterns, which have been studied lately as this topic is becoming popular. For
instance, First-Person Shooters have been examined by Hullet and Whitehead [20] and
‘Ville Games such as Farmville has been studied by Lewis et al. [21].
In the context of platform videogames, besides the first overview that is provided
in the previously referred studies by Compton and Mateas, the question of game
design patterns was brought by Dahlskog and Togelius [22] in their recent work fo-
cusing Infinite Mario Bros.. However, the main focus goes once again to the lower
abstraction layers, as the authors identify how multiple entities are combined in a
sequence. As lower level abstractions are very precise in describing a particular game,
they are not likely to be generalized to other titles of the same genre.
An interesting alternative study but yet related to the topic of game design patterns
was presented by Sorenson and Pasquier [23]. The authors defined a system based on
a genetic algorithm implementation that creates levels by encoding design patterns
into the fitness function. Some possible design patterns are given as an example for
the videogame Infinite Mario Bros. to show how it could be applied.
An adventure platformer that allows exploration uses principles from an RPG,
where higher abstractions patterns have been defined, and still use action based me-
chanics. To generate game scenarios that allow this combination, it is important to
understand how those higher abstraction patterns can connect to the lower abstraction
set of actions.
In order to bring higher abstraction aspects to level generation and provide meaning to
the avatar actions representing somehow a story, within a certain context, it is impor-
tant to understand the key concepts of plot representation. Initial requirements can be
identified in Ryan’s [24] proposal for the conditions of narrativity. The author
refers spatial and temporal dimensions as the essential requirements for a broad con-
cept of story. A mental dimension is defined as the next step to include the human
experience in stories, where intelligence and emotions are described with agents,
goals, actions and events. A final formal and pragmatic dimension expands the previ-
ous semantic concepts to significance, namely regarding the relation among events
and their meaning.
Tosca [25] identified the notion of quest as “a way of structuring events in games,
and (…) incarnate causality at two levels: a semantic one (how/why actions are con-
nected); and a structural one (plan of actions, interaction of objects and events)”. In
fact, this concept is natural in videogames mainly because they tend to represent user
incarnations on a certain character, the avatar. This hero centred approach that we find
in videogames makes them suitable to be represented as quest-based games.
Sullivan et al. [26] created a quest browser recalling Howard’s definition of quest
as “a goal-oriented search for something of value” [27] and stated that quests “give
meaning to the player’s actions”. In addition, authors gathered previous studies based
on World of Warcraft to identify a set of quest types.
Another perspective to look at quests in videogames was proposed by Aarseth [28],
based in a division of the concept into three main axis: place, time and objective.
A practical approach is presented in the playable quest-based story game Mis-
manor [29], where the authors applied their theories about playable stories. The main
aspect to refer regards the proposed quest structure, which contains a certain intent
and different sets of pre-conditions, starting and completion states.
Finally, a motivating study about quests was presented by Smith et al. [30]. Their
work regards RPG’s, namely the patterns used in that genre of games, which were
divided into level patterns and quest patterns. The second type is particularly interest-
ing in the scope of the present work because, as stated, adventure platformers can
share several principles with RPG’s. Authors grouped quest patterns into five main
groups: actions, objectives, structure, superstructure and purpose.
In the remaining of this document we will explore how the presented different ap-
proaches to describe quests can be merged in the specific genre of platform video-
games and how we envision these principles as a way to improve automatic level
generation.
3 Platform Videogames
• Actions. As a rule of thumb, one can consider an action as something that can be
performed by the avatar and that the player can assign to an input button. Actions
do not mean anything in particular and can only be interpreted together with other
actions in the context of a goal. Examples: walking, attacking, jumping or running.
Several approaches have been considered for automatic generation on the most
concrete layer of the model. On the top layers the common efforts are focused on the
narrative and are not suitable for an action game. So, the current goal for automatic
level generation is to provide an appropriate bridge between the bottom and the top
layers.
Static
Fully Plotted
Deep & rich
pole LBP
PP
BB Tr RD
Br Ca So
BB Sp
RD Br SM
Usable
PP PP LBP
Lin-
Dynamic Satellites
SM RD
ear
Tr So
Flat characters
LBP
SM PP Ca
Labyrinth
So
Modifiable
LBP Tr
Ca Br Br
Sp
Dynamic Kernels
Ca Tr So
Hub-shaped
Creatable
Sp
Sp
No kernels
Inventible
SM
Ludic BB
Open
Bots
pole RD
As we stated, in this quest type of stories, the hero is the centre of all the action and
cannot be considered as a simple agent. Considering the hero as a fifth element on the
presented model, we propose the following descriptions to define the features from
the narrative to the ludic pole:
• A character with personality that is presented and explored throughout the game,
as the presented case of the videogame Trine.
• An expressive character that shows his/her current feelings, opinions or sugges-
tions, even though without any particular knowledge about the inner self, such as
the Sackboy character introduced in the videogame Little Big Planet.
• A living character that suggests possible feelings but without manifesting them in
any way through the game, such as the initial versions of the character Mario.
• Checkpoint. This is the most common type of quest in platform games. It consists
on reaching a certain part of the scenario with a certain goal. Those goals might be
decomposed on the following types:
─ Talk to someone (e.g. in Little Big Planet the character breaks one of Zola’s
creation and is told that should talk to him);
─ Grab a certain item (e.g. in Prince of Persia the character has to find a sword);
─ Achieve a milestone on the path (e.g. in Trine, characters explicitly state land
regions as milestones on the main path);
─ Deliver an item to someone or someplace (e.g. in The Cave, each possible char-
acter has a certain item of desire to be collected within a chapter);
─ Escort something/someone (e.g. in the Bride Reception level in Little Big Planet
the character finds the groom injured in a cave and is asked to transport him
back);
─ Escape a certain enemy or object (e.g. in Rick Dangerous, the first level starts
with the player escaping a rolling stone ball);
─ Activate (or deactivate) a certain entity (e.g. in the end of the last level of Rick
Dangerous, the character has to disable a missile);
• Hostile Zone. This pattern represents a stop to the main movement mechanics in a
particular part of the level. It may be represented by one of the following goals:
─ Defeat n opponents (e.g. in Trine 2 the player encounter hordes of Goblins);
─ Defeat a horde of enemies until a certain condition is reached (e.g. in Trine 2 the
player encounters some situations where enemies are spawned infinitely while
he/she tries to accomplish another goal, such as opening a door);
─ Defeat one particular opponent, in the typical form of final boss (e.g. in the end
of each act, Sonic has an encounter with Dr. Robotnik);
• Multiple Checkpoints. Refers to quests that represent multiple checkpoints that
should be performed in a specific order, such as the following:
─ Key/Lock situations, where the character has to take a detour to get a certain
item that will allow passage (e.g. in Braid the character has to grab keys that
open certain doors);
─ Switch/Door, which is a similar situation to Key/Lock but in this case passage is
allowed because of a certain trigger. This may lead to a timed challenge, as the
trigger might open passage for a short time window (e.g. in Prince of Persia, the
character frequently presses buttons that open certain gates);
─ Composed Key/Lock, which are the situations where there is a need to gather
multiple items to get passage (e.g. in some community levels of Little Big
Planet we observed doors that open with a set of different keys);
─ Composed Switch/Door situations represent the need to activate several triggers
to get passage (e.g. In Trine, some contraptions that are controlled with multiple
levers that should be pushed in an appropriate order to open some passages);
─ Character modifier situations represent a first stop on a certain entity that will
somehow change the characters’ features, allowing him/her to reach another
checkpoint that otherwise would not be accessible (e.g. in Spelunky, the player
can get a rope to access additional areas);
• Puzzle. A non-trivial combination of game elements to overcome a small part of
the level. Puzzle nature, content and difficulty are particularly subjective and we
will consider those situations as a quest without a decomposition in different goals.
In addition, we have identified a set of game patterns to promote additional game-
play, which we have previously referred as side-quests, as follows:
• Grab collectible points. Represent the inclusion of scoring elements without any
gameplay related reward or with a simple impact on the number of tries (e.g. in
Blues Brothers, player earns an extra life for every 100 gathered records).
• Reach a secret zone. Represent the inclusion of areas that are not directly identifi-
able by the player, and that may contain the following type of reward:
─ Collectibles data (game art or videos) that do not have any effect on the game
and are mere trophies (e.g. in Trine, there are secret arcs that unlock game art);
─ Collectible powers, that enhance the avatar features (e.g. in Super Mario, some
blocks hide flowers that upgrade the character).
The previous structure of quests and goals allows a designer to sketch the content
of a level that represents those concepts. In the next section we will observe how
these ideas could be materialized into an automated generation tool.
In this section we briefly present a prototype that is currently being developed to generate
platform levels based on the previously described patterns. Although the focus of this
article is the previous study, this level editor unveils the potential of the referred ideas.
Figure 2 shows the current UI in the creation of an example for the videogame Prince of
Persia. It consists on a semi-automatic process, where the designer manually defines the
sequence of quests with their respective goals and a set of side-quests, as observable on the
left part of the figure. The automatic generation process takes those quests as input and
generates the geometry using a set of algorithms that differ according to the used patterns.
The output is presented on the right part of the window, where the generated segments
have been highlighted for comprehension aid.
It is possible to rebuild a whole level from scratch or only certain quests, allowing
a designer to regenerate some parts while keeping others intact. Figure 3(a) shows a
sample of multiple variants that were generated for the pattern Checkpoint in different
full generation runs. For the generation of this specific type of quest in Prince of Per-
sia, a composition algorithm was implemented in association with an automatic chunk
detection process. Checkpoint segments were thus created by retrieving common
linear segments from the original set of levels in this videogame.
When the system is rebuilding a specific quest, the algorithm receives the previously
existing segment and uses the entry and exit points of that same section as restrictions
for the generation process. This gives the algorithm the necessary information to create
an alternative geometry that can replace the previous one.
Also, most patterns can be set according certain parameters. As observable back in
Figure 2, the side quest Secret Zone allows the definition of an item to include in that zone.
Moreover, some patterns allow the definition of difficulty based variables. An example is
the Hostile Zone pattern, which we have implemented to map user skills to the frequency
of opponents and respective strength. Figure 3(b) shows different generated alternatives
for the same segment with distinct difficulty profiles, in this case using the videogame
Infinite Tux. A simple terrain displacement generator was used followed by a random
inclusion of opponents regulated by the difficulty estimation constraints (roughly, a certain
difficulty profile has to match a respective frequency of opponents).
Fig. 3. (a) Alternative geometries generated for a certain patterns and (b) alternative configura-
tion of a certain level part configured for different difficulty settings
As we are decomposing the structure in quests and goals, generation algorithms are
responsible for small level parts at each time, reducing the computational complexity.
For instance, evolving complete level structures with genetic algorithms as a global
process is a time consuming task but generating a small portion to match a certain
quest can be achieved promptly. Also, this decomposition into smaller regions poten-
tiate simpler algorithms to be used integrated with more complex techniques. As an
example, the referred Checkpoint pattern typically consists on a systematic sequence
of jumps and/or traps, which can be generated with ease using an ad-hoc algorithm.
Still, these parts can be merged with other segments representing quests that require
more complex algorithms, such as the Multiple Checkpoint pattern where paths and
routes must be analysed to create those situations.
final result is a set of level design patterns grouped in two abstraction layers, quests
and goals, which can lead automatic generation processes to a more contextualized
output. Lower abstraction patterns presented in some other works can be integrated to
generate level parts where a certain quest or goal is being represented.
As stated, it is important to bridge the higher and the lower abstraction layers of level
design patterns in order to improve existing automated generation. With that in mind, a
prototype for level construction using the identified patterns in multiple layers has been
designed. Our first tests with our primal prototype showed us that this can be a fast way of
semi-automatic level generation, having a human to define the main level content structure
with the identified patterns and having lower level generation algorithms in the bottom to
concretize those structures. At the moment, a few simple generation algorithms were asso-
ciated with the quest creation mechanism. The main goal at the moment in this aspect is to
complete the prototype by including different generation algorithms, considering that
some will be more suitable for certain patterns.
Finally, multi-player gameplay is one aspect that has been evidenced in recent plat-
form videogames. Generating a level that is suitable for multiplayer requires to take
into account a few more features. For that, one of our current goals is also to map the
multiplayer design patterns proposed Rocha et al. [31] and El-Nasr et al. [32] in our
prototype, allowing the designers to create levels with specific multi-player situations.
References
1. Niesz, A., Holland, N.: Interactive fiction. Critical Inquiry 11, 110–129 (1984)
2. Frasca, G.: Ludology meets Narratology: Similitude and differences between (video)
games and narrative (1999)
3. Aarseth, E.: A narrative theory of games. In: Foundations of Digital Games, pp. 129–133
(2012)
4. Shaker, N., Togelius, J., Yannakakis, G.N., Weber, B., Shimizu, T., Hashiyama, T., Soren-
son, N., Pasquier, P., Mawhorter, P., Takahashi, G., Smith, G., Baumgarten, R.: The 2010
Mario AI Championship: Level Generation Track. IEEE Transactions on Computational
Intelligence and AI in Games 3, 332–347 (2011)
5. Smith, G., Whitehead, J., Mateas, M.: Tanagra: A Mixed-Initiative Level Design Tool. In:
Foundations of Digital Games, FDG 2010, pp. 209–216. ACM Press, New York (2010)
6. Smith, G., Whitehead, J., Mateas, M.: Tanagra: Reactive planning and constraint solving
for mixed-initiative level design. IEEE Transactions on Computational Intelligence and AI
in Games 3, 201–215 (2011)
7. Smith, G., Whitehead, J., Mateas, M.: Tanagra: An Intelligent Level Design Assistant for
2D Platformers. Artificial Intelligence and Interactive Digital Entertainment (2010)
8. Compton, K., Mateas, M.: Procedural Level Design for Platform Games. In: Artificial In-
telligence and Interactive Digital Entertainment, AIIDE, pp. 109–111 (2006)
9. Smith, G., Cha, M., Whitehead, J.: A framework for analysis of 2D platformer levels. In:
ACM SIGGRAPH symposium on Videogames, Sandbox 2008, p. 75. ACM Press, New
York (2008)
10. Smith, G., Treanor, M., Whitehead, J., Mateas, M.: Rhythm-based level generation for 2D
platformers. In: Foundations of Digital Games, FDG 2009, p. 175 (2009)
11. Smith, G., Whitehead, J.: Launchpad: A Rhythm-Based Level Generator for 2-D Platfor-
mers. In: Foundations of Digital Games, vol. 3, pp. 1–16 (2011)
12. Smith, G., Gan, E., Othenin-Girard, A., Whitehead, J.: PCG-based game design: Enabling
new play experiences through procedural content generation. In: Workshop on Procedural
Content Generation in Games, pp. 5–8 (2011)
13. Smith, G., Othenin-Girard, A.: PCG-based game design: Creating Endless Web. In: Foun-
dations of Digital Games 2012, FDG 2012 (2012)
14. Mawhorter, P., Mateas, M.: Procedural level generation using occupancy-regulated exten-
sion. In: Computational Intelligence and Games, pp. 351–358 (2010)
15. Pedersen, C., Togelius, J.: Yannakakis: Modeling Player Experience in Super Mario Bros.
In: Computational Intelligence and Games, CIG 2009. IEEE Press (2009)
16. Mourato, F., Próspero dos Santos, M., Birra, F.: Automatic level generation for platform
videogames using genetic algorithms. In: Advances in Computer Entertainment Technolo-
gy, ACE 2011, pp. 8:1–8:8. ACM, New York (2011)
17. Mourato, F., Birra, F., Próspero dos Santos, M.: Enhancing level difficulty and additional
content in platform videogames through graph analysis. In: Nijholt, A., Romão, T., Reids-
ma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 70–84. Springer, Heidelberg (2012)
18. Mourato, F., dos Santos, M.P., Birra, F.: Integrated System for Automatic Platform Game
Level Creation with Difficulty and Content Adaptation. In: Herrlich, M., Malaka, R., Ma-
such, M. (eds.) ICEC 2012. LNCS, vol. 7522, pp. 409–412. Springer, Heidelberg (2012)
19. Björk, S., Lundgren, S., Holopainen, J.: Game Design Patterns. Digital Games Research (2003)
20. Hullett, K., Whitehead, J.: Design patterns in FPS levels. In: Foundations of Digital
Games, FDG 2010, pp. 78–85 (2010)
21. Lewis, C., Wardrip-Fruin, N., Whitehead, J.: Motivational game design patterns of ’ville
games. In: Foundations of Digital Games. 172–179 (2012).
22. Dahlskog, S., Togelius, J.: Patterns and Procedural Content Generation: Revisiting Mario
in World 1 Level 1. In: FDG - Workshop on Design Patterns in Games (2012)
23. Sorenson, N., Pasquier, P.: Towards a generic framework for automated video game level
creation. Applications of Evolutionary Computation (2010)
24. Ryan, M.: Avatars of story (2006)
25. Tosca, S.: The quest problem in computer games. In: Technologies for Interactive Digital
Storytelling and Entertainment, TIDSE (2003)
26. Sullivan, A., Mateas, M., Wardrip-fruin, N.: QuestBrowser: Making Quests Playable with
Computer- Assisted Design (2009)
27. Howard, J.: Quests: Design, Theory, and History in Games and Narratives. A K Peters Ltd.
(2008)
28. Aarseth, E.: From Hunt the Wumpus to EverQuest: Introduction to Quest Theory. In: Ki-
shino, F., Kitamura, Y., Kato, H., Nagata, N. (eds.) ICEC 2005. LNCS, vol. 3711, pp.
496–506. Springer, Heidelberg (2005)
29. Sullivan, A., Grow, A., Mateas, M., Wardrip-fruin, N.: The design of Mismanor: Creating
a playable quest-based story game. In: Foundations of Digital Games, pp. 180–187 (2012)
30. Smith, G., Anderson, R., Kopleck, B., Lindblad, Z., Scott, L., Wardell, A., Whitehead, J., Ma-
teas, M.: Situating Quests: Design Patterns for Quest and Level Design in Role-Playing Games.
In: André, E. (ed.) ICIDS 2011. LNCS, vol. 7069, pp. 326–329. Springer, Heidelberg (2011)
31. Rocha, J., Mascarenhas, S., Prada, R.: Game mechanics for cooperative games. ZON Digi-
tal Games 2008, 72–80 (2008)
32. El-Nasr, M.S., Aghabeigi, B., Milam, D.: Understanding and evaluating cooperative
games. Human Factors in Computing Systems, 253 (2010)
Abstract. Mobile photo taking and sharing has become a frequent leisure-time
activity for smartphone users. This paper presents a field study of a mobile
application called Social Camera, which enables instant sharing of photos
within small groups. The application enables collaborative creation of photo
collections with shared folders in the cloud and instant connection through the
folders and photos. Social Camera was evaluated using four groups (altogether
17 users) in a field trial. The results reveal six enablers of instant photo-sharing
experiences within small groups: sense of connectedness and social awareness,
presentation and expression of self, lightweight and surprising interaction,
collective photography, documentation of experiences, and finally, privacy and
user control. This work gives design implications of these enablers for photo-
sharing applications.
1 Introduction
Media and entertainment online relies ever more on user-generated content. Social
networking services such as YouTube1, Facebook2, and Flickr3 are built on video and
photo content from the users [1]. The “interestingness” of content depends on the
freshness of the content, the person’s relation to the content, personal nature of the
content, and whether the content is actually targeted to the receiver [2]. Users
currently have solutions for sharing their photos publicly in social networking
services (SNS) but lack dedicated solutions for instant sharing within small groups.
To this end, novel ideas to support pleasurable user experiences are needed. The
private group setting and instant sharing are the key differentiators from conventional
photo-sharing tools such as Facebook and Flickr. In our current setup, user’s photos
are directly uploaded to a shared album, which is accessible to relevant users [3, 4].
This study examines social user experiences enabled by design solutions for small-
group photo sharing with mobile devices. A prototype implementation of Social
Camera aims to combine the experiences of remote sharing and collocated sharing
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 344–355, 2013.
© Springer International Publishing Switzerland 2013
into a new way to share photos instantly. The goals of this study are 1) to evaluate the
implemented Social Camera prototype, 2) to identify habits of photo sharing within
the intimate group, 3) to observe users’ interactions with each other during the trial
period using the prototype and not using it 4) to identify the needs and wants of the
groups that the implementation actually has to fulfill. The results are significant in the
area of human-computer interaction since the online sharing of photos in small groups
remains a less-studied area, and dedicated solutions for instant sharing with a small
group do not exist. This work contributes to the area of social user experience design
by introducing enablers of social experience in small-group sharing.
2 Related Work
Development in photo-capturing devices and channels for sharing the content has
created a new culture of instant photo sharing. The new culture of digital photo
capturing and sharing has also gained interest and sparked rich research in the human-
computer interaction community. Kirk et al. has named the whole process of photo
taking “photowork” [5]. Frohlich [6] has introduced a model of four categories in the
photowork process: co-present sharing, remote sharing, archiving, and sending. The
most essential phase needing support in the current online world is the sharing of
photos. Data transfer with broader bandwidths and development of cloud computing
technology allows new solutions for archiving and sharing photos outside the hard
disk and physical drives [7]. The cloud technology has become available for the
public audience only recently, by services such as Dropbox1. An emerging design
trend in mobile photo sharing is automatic upload of photos to a cloud server. Studies
by Lucero et al. [8] explore the Image Space application, which allows automatic
upload of photos with location tags and sharing the photos within a limited
community. A similar cloud-based application called Image Exchange is introduced
in work by Vartiainen [9].
As a basis for the design of the small-group–photo-sharing application, several
studies were utilized to give background. Previous work has identified a problem in
small-group sharing especially after the events a group has attended or trips a group
has taken together [10]. Users do not have dedicated services and habits of
distributing the photos. Therefore, distribution takes time (and the photos have the
greatest value right after the event). Additionally, previous studies suggest that people
are willing to share personal content in private circles such as family or close friends
[10, 11]. Close-knit groups have needs for demonstrating the group identity and for
collectively managing content [10, 12]. A study by Miller and Edwards [13]
introduces a culture of “snaprs,” photographers that base their photography on sharing
with the online community, in the mentioned case Flickr. Snaprs described habits of
downloading pictures of others as a part of their own photo collection, where “kodak
culture” wanted to have a solution to share to the limited group only [13], which
suggests that there is a need for shared or collaborative folders amongst the small
groups. Photo sharing using camera phones as the capturing device and the sharing
platform is a widely researched area. Frohlich et al. [6] handle photowork on two
dimensions: temporal and locational. Photo sharing is handled either instantly or after
some time, collocatedly (on the spot) or remotely. Traditional photography requires a
certain time gap between the photo capture and the sharing. Digital photography has
made the time gap narrower. According to the Kindberg et al. taxonomy [14], there
are affective and functional dimensions in photo sharing: mutual task and experiences
with the collocated users and remote tasks and shared experiences with those who are
absent. Kindberg et al.’s [14] study of camera phone users introduces a taxonomy of
six reasons for image capture on a camera phone: individual personal reflection,
individual personal task, social mutual experience, social absent friend or family,
social mutual task, and social remote task.
The definition of storytelling includes sharing of multiple photos with a textual or
oral commentary attached. Collocated sharing and storytelling are the focus of Van
House et al.’s work [15, 16, 17]. Van House et al. [16] introduced three motivations
for social use of personal photographs: constructing personal and group memory,
creating and maintaining social relationships, and self-expression/self-presentation..
Balabanovic et al. [18] have identified patterns of photo-driven and story-driven
photo sharing. Sharing with the people who were present in the capturing moment is
referred to as “reminiscing” [14, 12], and telling about the event to those who were
not present, “storytelling” [6]. Content consumption and creation are often not in
balance in communities that are built on user-generated content [19, 20]. Most of the
users are eager consumers of content and less willing to produce or especially share
the content. However, content production can be motivated by giving users a sense of
social interaction and connectedness.
Research on social user experience focuses on the social context of the products,
where the presence and actions of other users creates the user experience. Interaction,
collaborative creation of content, and reciprocity are essential for the social user
experience. Social dimension in the interactive products as a factor of user experience
remains a less-studied area. The social user experience is built on the social context in
which other users and their presence define the actual interaction. In their study of
socially used web services, Väänänen-Vainio-Mattila et al. [21] explored social user
experience as UX with services that support social activity through their functionality.
The research identifies curiosity, learning, self-expression, suitability of content and
functionalities, completeness of networks, and competition as the motivational drivers
for social user experience. The study findings presented in this paper contribute to the
understanding of social UX with photo sharing in small groups.
The design process of Social Camera started with two user studies in 2011 [3, 4, 10].
Relying on the findings from these two studies, the implementation of Social Camera
application was completed. Prototype implementation Social Camera aims to combine
the experiences of remote and co-located sharing into a new instant way to share
photos. The research involved three aspects of interaction design: understanding the
users, prototype design, and evaluation [22]. The initial studies related to small group
photo sharing raised design framework for the system. It was found out that: 1) small
groups have problems in sharing the picture content from many devices between the
group, 2) people have suspicions over sharing the data in social media or social
networking services, because they are not willing to risk losing their control over their
photos 3) people use web folders and cloud services such as Dropbox, although they
might not be implemented for sharing and storing photos taken with smart phones 4)
the event pictures tend to be most interesting right after the event they are shot into 5)
people would like to have dedicated “co-located workspace” for the group photos. In
addition to support photo management and sharing, the software aims to support sense
of belonging and connectedness and social user experience [2, 21, 12] within the
group by as the experience-based design ideology suggests [23].
The Social Camera prototype enables users to create shared photo albums and
automatically distribute photographs within a selected group of people. The actual
prototype application is divided into four logical sections: shared albums, own photos,
notifications, and the camera. Shared albums are the core of the prototype application. The
users are able to create and name new albums (Figure 1). The shared album content can be
browsed similarly as the local device photo gallery. Users are able to browse by viewing
thumbnails under different albums and touching a thumbnail to enlarge the photograph to
a full screen size. Under the shared album name a list of users are displayed who share the
same folder. Other people can be invited to join the shared albums using NFC i.e.
selecting the invite option and bringing the devices in close proximity (Figure 1). The
invitation to the shared album is sent automatically to the recipient for acceptance. Under
the shared albums, the user is able to select the corresponding album to be default for the
camera to assign new photographs to. The assignment can also be changed from the
camera section while taking photographs. The captured photographs are distributed
automatically to the members of the shared album.
Fig. 1. a) Inviting people to shared albums, b) Shared albums, c) Own photos and d) sharing to
albums views in the Social Camera prototype
Alternatively the user is able to access her own photos view from the prototype and
share photographs to the shared albums (Figure 1). The own photos view shows a grid
view of available photographs in the device as the original photo gallery of the device.
The notifications center shows a feed of recent activity to the users, including
notifications on new photos in shared albums, new pictures of the user, and group
invitations. The aim of the organized field study was to identify the needs for sharing
photos instantly within small groups, and to understand the social user experience in
such activity. In order to collect actual use experiences of the application, a field trial
was organized for four participating small groups. The participant groups included
both non-technical photographers who do not use professional equipment and more
advanced photographers with semi-pro or pro equipment. The groups were recruited
through multiple mailing lists for students of Tampere universities and also through
forums of photographers. Fig. 2 shows the events in which the groups participated in.
Fig. 2. Photo taking and sharing activity of the groups during the trial
Background data of the participants was collected by using a web form before the
interviews. The participant selection was made on basis of their equipment usage, age
and activity of photographing and sharing habits in social networking services. The
aim was to have four groups with different backgrounds. Group 1: Skiing family (2
fathers, 3 boys, ages 13-48) took the cameras to the skiing trip to Ylläs in Lapland,
where they spend the holiday week, taking and sharing photographs with Social
Camera. Group 2: Electronic music event group (1 female, 3 males, ages 18-35) went
together to electronic music event, which lasted for twodays in Lahti. The group
attended on various smaller events during the trial. Group 3: DJ club (2 female, 2
male, ages 24-27) had a ten days of trial period, where they attended a album release
party, a student party cruise and smaller events events together and separately. Group
4: Group of friends (3 female, 1 male, ages 24-25) used the application in a evening
get-together, a housewarming party and mostly individually during the week.
The data collection consisted of four different methods: the trial period of 1-2
weeks with the diaries, an individual interview of the trial experiences, a group
interview concerning the small group sharing and future feature ideation and a user
experience questionnaire with 19 statements. Each group trial started with the session
where the test devices were handed out to the participants. In the session, participants
were also introduced with the service in detail. Participants were given N9
smartphones with the Social Camera application software installed in the starting
sessions, before the trial period. In the starting sessions a brief introduction for the
application use was given. Participants were instructed to take photos with the given
device throughout the trial period. Findings from the group interviews and diaries
were treated as relative notes in the analysis. Comments in the interviews and diaries
were transcribed and organized under themes.
4 Results
Creating and managing the collective group folders together was the peak experience of
the trial for the most users. Users appreciated seeing the different viewpoints, getting
pictures of themself from other photographers, and getting an idea of how others
experienced the event. Users saw value in sharing instantly with close individuals and then
expanding the audience if desired. There were major differences in the amount of
photographing and sharing rates of the photos between the groups during the trial periods.
The amount of all pictures captured ranged from 133 pictures in Group 1 to 742 pictures in
Group 2. The different natures of the events the groups participated in explain some of the
differences. Table 1 shows the photo taking and folder creation activity throughout the
trial. For example, on the skiing trip, the wintery weather and poor network connections
prevented photo sharing. Sharing rate also varied between the groups, as Table 1 shows.
Table 1. Photo taking and sharing activity of the groups during the trial
Group 2 had the lowest sharing rate, which also was referred to in the interviews,
in which the group participants said that they selected the best photos only to be
shared. Group 4 was active in creating subgroups and folders that were only shared to
some of the users in the group. After the analysis of the diary and interview data,
findings were categorized under the following main enablers: 1) connectedness and
social awareness; 2) presentation and expression of self: sharing everyday life
activities; 3) lightweight communication and surprises; 4) collective photography; 5)
documentation of exceptional events; and 6) privacy, user control, and utility. Finding
categories include the most promising use cases and motives for small-group sharing
in the six categories. The finding categories can be divided into two types of photo
experiences: long-term and short-term. The first three categories are short-term
enablers, which are related to an instant and spontaneous photo sharing, and the latter
three categories are related to long-term experiences and photo storage and archiving.
follow their photos appearing in the group folder if they decided to carry the device
and shoot photos. Since commenting, giving ratings or likes, and similar interaction
features were left out of the implementation, people were forming ways to
communicate with the photos only The peak experience was seeing the latest photos
from others appearing in the system in real time when people were in different
locations, it was fun to see what others were doing at the same time. The application
created a channel for instant communication throughout the trial, and evidently
enriched the social interaction of the participating groups. “This gave me an excuse to
see my friends!” Female, Group 4. Photos appearing in the application in a real time
motivated picture taking and sharing. “I waited for T. to share his photos from a
‘famous people cruise’ but unfortunately he didn’t have the connection and did not
share them instantly onboard!” Female, Group 3.
Group 3 had an album release event, which they attended together. Before the
event, they were getting ready for the event and shared photos each on their own. Two
males from the group were having dinner together and watched the photos sent by the
female participant from her home, where she was doing programming homework.
Afterwards at the album release events the whole group was photographing and
speaking about the photos from before the event when they were preparing from the
event. “It was interesting to see the timeline from the members of our group, all on
their way to the album release party in the evening. Me and M. were at the wings
restaurant and the girls and others on their own route.” Male, Group 4.
The final wrap-up of the event happened at home, where participants watched the
collection of photos form the evening. In the user experience questionnaire
participants gave an average of 5.77 of 1-7 Likert scale in ratings, in which the
standard deviation (SD.) was 1.71, to the statement “It was fun to share the photos
with my friends in the service; whereas, “The service offered me a novel way to share
photos got an average of 5.16 (SD. 1.87), and “This way to share photos was suitable
for me,” slightly lower with 4.88 (SD. 1.80).
In order to support a sense of connectedness with the application, the application
should offer fresh and real-time content easily and should facilitate visibility for
the newest updates. During the trial, participants followed the application, but in the
longer term use of the application should give notifications to the user without the
user having to actively to seek the newest content.
photos when they shot photos of their face gestures, telling others about their mood.
One of the female participants in the group 3 said that all of the members of their
group did not know about her musical hobby, so she wanted to tell about it. ”I wanted
to send the others photos of my cembalo lessons in Riihimäki. All in our group wasn’t
aware that I even have such a hobby” Female, Group 3. Picture blogging or creating
a photo diary of everyday activities was also seen as interesting option to utilize the
application. Some of the participants shared snapshots of what they are at home. ”I
see value if this would be a shared photo diary for our group, so everybody could
easily share” Male, Group 4.
In order to support self-expression the applications should offer clear control of
privacy and effortless sharing to the targeted audience. Dedicated folder to one’s own
pictures in which others can react by adding their responses with photos but still
realizing the actual author of the folder offers support for self-expression.
Figure 3 shows examples of the photo communication the participants made in the
DJ group. Participants used the application, for example, negotiating places and for
inviting others to certain places. Since the classical means to communicate through
comments, for example, was removed, users made up creative ways to interact
through sending the photos. In some of the cases during the trial, users wanted to
tease others by showing what they are up to. “I was waiting for M. and O. to come to
the Sunday chill music club. I shared few photos from there to hurry them up!” Male,
Group 2. In the questionnaire users gave averagely 4,06 (SD. 1,65) to the statement
“The service offered positively surprising experiences”.
A folder that is dedicated for the light interactions and spontaneous photos in
addition to event-related, person-related or theme-related folders can support
lightweight interactions within the group. Positively surprising experiences can be
supported by features that offer targeted photos to the users automatically.
The participants agreed that the application is most suitable in situations in which the
group would spend a certain period of time together but still be apart at times. For
example, trips abroad, seminars or music events were said to be ideal for using the
application. Participants saw the system most suitable for event photos that have a
relatively short life span. Physical location of the people and how the group is
scattered around define the way to interact through photos. During the trial the social
and physical context of the group varied, and five different group location formations
were faced: 1) Whole group at the same event physically together, 2) people scattered
around at a bigger event, physically nearby, 3) one at home, seeing photos from others
in the event, most of the group at the event, 4) one attending and sharing with others
and 5) everyone in different locations. Browsing the photos was not as interesting
when they all were at the same event, however, seeing the collection of photos from
the whole group instantly after the events was an appreciated experience. “It was nice
to see the products of the whole group right after the event! It was nice to see the
things that interest others. I would like to share video also though.” Male, Group 2.
The electronic music group was documenting the music event at Lahti together,
whereas the boys from the Group 1 were shooting photos of their tricks while skiing
at Ylläs and instantly sharing them. Storytelling for those who were not present by
using the application was seen as an interesting idea. “If we had this when we were
together with the club in Berlin and tried to get in to different clubs. There were 18 of
us and only a few got in so they could share the experience and tease others when
they get in.” Male, Group 3.
Users had to implicitly select the photos for sharing after the capture. Most of them
would like to have an easy sharing mode, where all the captured photos could be
automatically transferred to the event folder in special occasions. Event-based folders
must be created instantly on the spot of the event or, alternatively, before the event.
The access should be granted to the attendees or content contributors. Also features
for expanding the group with the acceptance of the group should be included.
The results of the field study with Social Camera reveal the following main enablers
of photo-sharing experience in small groups: 1) instant interactions and social
presence to support a sense of connectedness, 2) expression and presentation of self
Acknowledgements. Authors thank Sanna Malinen, Guido Grassel, Jari Kangas, and
Yanqing Cui for their valuable contributions to the work.
References
1. Lehikoinen, J., Aaltonen, A., Huuskonen, P., Salminen, I.: Personal Content Experience.
Wiley (2007)
2. Malinen, S., Ojala, J.: Maintaining the Instant Connection—Social Media Practices of
Smartphone Users. In: International Conference on the Design of Cooperative Systems,
pp. 197–211. Springer, London (2012)
3. Vyas, D., Cui, Y., Ojala, J., Grassel, G.: Producing while Consuming: Social Interaction
around Photos Shared within Private Group. In: Nijholt, A., Romão, T., Reidsma, D. (eds.)
ACE 2012. LNCS, vol. 7624, pp. 133–150. Springer, Heidelberg (2012)
4. Ojala, J.A.O., Vyas, D., Lehtiniemi, A.J.: Technique for evaluating photo sharing interfaces
with the early prototypes - group simulation. In: Holzinger, A., Ziefle, M., Hitz, M., Debevc,
M. (eds.) SouthCHI 2013. LNCS, vol. 7946, pp. 36–53. Springer, Heidelberg (2013)
5. Kirk, D., Sellen, A., Rother, C., Wood, K.: Understanding photowork. In: CHI 2006,
pp. 761–770. ACM, New York (2006)
6. Frohlich, D., Kuchinsky, A., Pering, C., Don, A., Ariss, S.: Requirements for photoware.
In: Proceedings of CSCW 2002, pp. 166–175. ACM, New York (2002)
7. Odom, W., Sellen, A., Harper, R., Thereska, E.: Lost in translation: Understanding the
posession of digital things in the cloud. In: Proc. CHI, pp. 781–790. ACM (2012)
8. Lucero, A., Boberg, M., Uusitalo, S.: Image Space: Capturing, sharing and contextualizing
personal pictures in a simple and playful way. In: Proc. ACE 2009, pp. 215–222. ACM (2009)
9. Vartiainen, E., Väänänen-Vainio-Mattila: User experience of mobile photo sharing in the
cloud. In: MUM 2010. ACM (2010), Article No. 4
10. Ojala, J., Malinen, S.: Photo sharing in small groups - identifying design drivers for
desired user experiences. In: MindTrek 2012, pp. 69–76. ACM (2012)
11. Kairam, S., Brzozowski, M.J., Huffaker, D., Chi, E.H.: Talking in circles: Selective
sharing in Google+. In: Proc. CHI 2012, pp. 1065–1974. ACM (2012)
12. Olsson, T.: Understanding Collective Content: Purposes, Characteristics and Collaborative
Practises. In: Communities and Technologies, C&T 2009, pp. 21–30. ACM (2009)
13. Miller, A., Edwards, W.K.: Give and take: A study of consumer photo sharing culture and
practice. In: CHI 2007, vol. 20, pp. 347–356. ACM (2007)
14. Kindberg, T., Spasojevic, M., Fleck, R., Sellen, A.: The Ubiquitous Camera: An In-Depth
Study of Camera Phone Use. IEEE Pervasive Computing 4(2), 42–50 (2005)
15. Patel, N., Clawson, J., Voida, A., Lyons, K.: Mobiphos: A study of user engagement with
mobile collocated-synchronous photo sharing application. International Journal of Human-
Computer Studies, IJHCS 67(12), 1048–1059 (2009)
16. Van House, N.: Collocated photo sharing, story-telling and the performance of self.
International Journal of Human-Computer Studies, IJHCS 67(12), 1073–1086 (2009)
17. Van House, N.A.: Personal photography, digital technologies and the uses of the visual.
Visual Studies 26(2) (2011)
18. Balabanoviç, M., Chu, L.L., Wolff, G.J.: Storytelling with digital photographs. In:
Proceedings of CHI 2000, pp. 564–571. ACM, New York (2000)
19. Lehtiniemi, A., Ojala, J.: MyTerritory: Evaluation of outdoor gaming prototype for music
discovery. In: MUM 2012. ACM (2012), Article No. 35
20. Ojala, J.: Personal Content in Online Sports Communities: Motivations to Capture and
Share Personal Exercise Data. International Journal of Social and Humanistic
Computing 2(1-2), 68–85 (2013), doi:10.1504/IJSHC.2013.053267
21. Väänänen-Vainio-Mattila, K., Wäljas, M., Ojala, J., Segerståhl, K.: Identifying Drivers and
Hindrances of Social User Experience in Web Services. In: CHI 2010, pp. 2499–2502.
ACM (2010)
22. Jones, M., Marsden, G.: Mobile interaction design, 1st edn. John Wiley & Sons Ltd. (2006)
23. Hassenzahl, M.: Experience Design; Technology for all the right reasons. Morgan &
Claypool Publishers (2010)
24. Ahern, S., Eckles, D., Good, N.S., King, S., Naaman, M., Nair, R.: Over-exposed?:
Privacy patterns and considerations in online and mobile photo sharing. In: Proceedings of
CHI 2007, pp. 357–366. ACM (2007)
25. Cui, Y., Kangas, J., Holm, J., Grassel, G.: Front-camera video recordings as emotion
responses to mobile photos shared within close-knit groups. In: CHI 2013, pp. 981–990.
ACM (2013)
1 Introduction
Procedural crowd generation techniques are increasingly common in the video games
industry to simulate large groups of agents. Hordes of zombies have been rendered in
games such as the Left 4 Dead series (Valve, 2008 and 2009) and Dead Rising (Capcom,
2006). Games such as Hitman Absolution (IO Interactive, 2012) and Assassins Creed 2
(Ubisoft, 2009) also rely on the simulation of "realistic" and "believable" multi-agent
crowds for the player to interact with.
When simulating crowds in a virtual environment such as a game, it is often hard
to produce a great deal of variety in characters. Usually due to the complexity of
manual character modelling, it is very time-consuming for a 3D modeller to manually
create a great deal of heterogeneous agents [10]. It is common to use a fixed number
of template characters to instance into an environment; this, however, may lead to the
problem of visually similar 'appearance clones' in the environment [6]. To combat
this, procedural generation of characters is becoming a more popular solution [10].
This paper explores techniques in which crowd generation algorithms can be adjusted
to better support greater diversity, in order to support more believable play environ-
ments. Firstly, a case study of an approach used in a commercial game series is ex-
plored, and then we introduce a new, multi-faceted approach to crowd generation, and
evaluate it in terms of computational overhead and effect on perception of diversity.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 356–367, 2013.
© Springer International Publishing Switzerland 2013
Colour Tinting on Clothes: The clothes on common infected were tinted using a
random multiplication of RGB values. This multiplication allowed for the clothing
texture to maintain its detailing, as contrast was never lost. Dark parts of the clothes
would remain dark no matter what was generated.
Grimes [3] states that fine details and low-contrast areas on textures did not work
well to create visual variety amongst members of the horde, as these were difficult to
change using colour multiplication techniques and lead to poor-quality textures on
models. She declares that geometric variation on the models warped facial features
and therefore helped to hide re-used textures. The image below shows two different
models with the same facial texture and how they both look visually different from
one another:
Decals such as blood flecks and dirt were also very effective in decreasing the
amount of times that playtesters noticed clones of recognisable - the decals obscured
recognisable features and allowed one texture to be re-used multiple times without
players noticing during play.
Left 4 Dead 2 (Valve, 2009) is the sequel to Left 4 Dead. As with the original, it
was a first person shooter with an emphasis on four-player teamwork in order to com-
plete missions set in a zombie apocalypse scenario. Focus was put on expanding the
game experience and making it more complex than the predecessor. This included the
development of an improved system for generating varied common infected [3]. A
new shader was developed in order to create even more variance in the common in-
fected models. According to Vlachos [11], the simplest infected has over 24,000
variations. This large amount of visual variation were achieved by a combination of
the following methods:
Gradient Mapping in the Luminosity channel was used to tint the textures on the
common infected models. It was noted that one single gradient map could not be used
for the whole model as having more than one colour in the gradient map caused prob-
lem with inappropriate colouration. Grimes [3] describes that the problem was over-
come by exclusively masking certain areas of the texture, allowing the tints to be
applied to exclusive areas of the final texture.
Blood and Grime Decals were added to the models in a similar technique to that
of Left 4 Dead. Blood masks were created and were then added on top of the charac-
ters tinted texture. Grime was then added on top in the same fashion; since grime did
not need to be as defined as blood, this was added using transparencies and a high
detail texture [3].
less about the game itself. They were then asked about what they noticed about the
crowds. Gameplay was recorded from Hitman Absolution (IO Interactive, 2012) and
Assassins Creed 2 (Ubisoft, 2009). Levels used were specifically chosen to have large
crowds of non-player characters.
Fig. 3. Non-Player Crowds in Hitman: Absolution (left) and Assassins Creed 2 (right)
task of creating variation. Based upon the suggestions from the focus group, the ap-
proaches from the games industry and proposed variety features in the works of
Thalmann and Musse [10],.a list of several visual variety-changing factors were col-
lated for inclusion in a proposed multi-dimensional solution. All of these changes are
focussed on addressing the variety of appearance of non-player characters in the
crowd. Each approach focuses on altering the virtual representation of the game char-
acter by altering either the shape of the character model or the post-processing effects
used on each character.
4.1 Colouring
One of the simplest approaches to increase visual diversity is through the use of ran-
dom colouring to different aspects of the character model (i.e. clothing, skin tone,
hair). RGB values can be randomly generated and applied to various parts of the char-
acter, with the use of a Colour Look Up Table (CLUT) in order to limit potential col-
ours to realistic subsets and distributions. However, Thalmann and Musse [10]. point
out that even with techniques of colour change, it does not overcome the feeling of
watching the same model - thus physical model changes must be considered as more
important than colour changes.
4.3 Accessories
The addition of possible items such as hats and glasses to characters can also
potentially increase variety. Depending on gender, beards, moustaches and other
(generally) gender-associated accessories could be added to character models. These
accessories would be modelled separately and then instantiated and rendered using the
same world transformation as the character that they are assigned to.
5 Evaluation
The multi-faceted nature of the proposed approach to generating diversity creates the
opportunity to evaluate the different approaches to introducing variety based on their
effect on believability. In addition, it is also important to look at the processing cost of
the various approaches, as this will help to evaluate whether or not the changes to a
complexity of procedural character generation system are cost-effective in terms of
increasing perceived diversity whilst not impacting too heavily on overhead.
Given the argument that lack of variety is associated with negative believability (and
therefore engagement/immersion in a game world), we built the evaluation to test
player perception of variety in crowds.
Crucially, it is the player’s perception of variety that is important, instead of
some automated measure of diversity based on combinations of generated models.
Although a real crowd has vast diversity based on the millions of differences between
individuals, the combination of factors that make a virtual crowd believable may be a
much smaller subset. For example, many of the more subtle differences between indi-
viduals may have different value in the perception of diversity.
Indeed, it is also exceedingly important that false positives (i.e. player incorrectly
identifies a false lack of diversity) are taken into account since it is user perception
that is most important (more so than realism, for example). Increasingly subtle
changes to the procedural generation of crowds may have no meaningful effect on
increasing perceived diversity (i.e. believability).
McDonnell et al. [6] conducted an experiment into the perceptual impact of ap-
pearance and motion “clones” in simulated crowds. In their experiment participants
were asked to watch a crowd simulation running and then click on clones of charac-
ters as and when they noticed them - the time between the start of the simulation and
the click was then recorded.
Using elements from the technique described from above, we performed an evalua-
tion to investigate the effectiveness of the variety of factors in the proposed crowd
generation algorithms. This does not look at the speed at which participants find
clones - instead it looks at the total clones that the player notices within a certain time
period as a measure of player perception of variety.
5.2 Procedure
Fifty naive individuals between the ages of eighteen and thirty (26 Male, 24 Female)
were asked to take part in the evaluation. They had mixed educational backgrounds
and not all had an interest in computer gaming. Each participant was required to pro-
vide informed consent and retained anonymity in the study.
Participants were then asked to sit down and watch one of five pre-recorded video
sets. These sets of videos had the same six fifteen-second videos in them, but in a
randomised order to avoid unintended variables.
Each video featured an animation of a crowd of non-player characters in a virtual
test environment (described in [8]), where the generation of character models in each
condition used a different combination of approaches. The videos were screen-
captures of the crowd simulation in action, but with differences to the way in which
the character models were generated.
All videos contain the crowd moving around the environment at an average walk-
ing pace as this better recreates a situation in which the crowd would actually be
viewed (instead of having all the characters just standing still). The camera was also
positioned at eye-level with the other characters in order to simulate the height at
which the crowd would generally be viewed. A fixed seed was used to ensure crowds
behaved identically between participants.
Participants were individually shown the six videos in random order. They were
asked to take a tally whenever they saw two agents in the video that they considered
to look the same. In this way, perception of variety is recorded based on the number
of clones identified, across conditions to allow comparison of the facets of the crowd
generation technique.
6 Results
Data collected represent the inverse of the user's perception of diversity in the crowd
simulation. The more clones that were spotted suggest less perceived diversity in the
crowd regardless of how many "real" clones there were. False positives still lower the
perceived diversity in the same way true appearance clones. The population mean
calculated from the results suggests that as the models became more varied, it became
harder for participants to pick out characters that looked the same within the scene.
This supports the case that the improvements made to the basic character generation
system (i.e. fixed set of model skins) have been effective; reducing the average of
6.48 clones spotted per session to 3.1 clones.
Figure 7 shows the frequency of total clones spotted in each condition. As the
model's variation increases, the graph peaks tend towards to lower totals. This shift
towards the left implies that it was harder for participants to notice appearance clones
as variation increased in the scene.
It is also interesting to note that there is a flatter distribution in the 'original' sys-
tem's results compared with that of the more complex systems. This increased spread
may be due to the inability of some participants to be totally sure that two figures
looked entirely the same. As the variation increased it may have been easier for test-
subjects to define similar-looking characters due to their more obvious colour changes
and similar accessories.
It also appears that basic scaling on the original model (the yellow line on Figure
7) had a dramatic difference to that of its predecessor. This suggests that merely add-
ing the random scaling to each character could provide the greatest benefit on the
crowd whilst also being the lowest cost on processing power (compared with the other
changes to the model).
It was also noticed that participants spotted many false-positives in the final model
version. This may be due to the addition of the accessory hats, which may have made
the models look more visually similar to one another than without them (i.e. two men
with top hats may have been considered as appearance clones even when they had
totally different heads or bodies.) This is an interesting result since it shows that the
use of limited accessories such as hats or bags in procedurally generated crowds
should be done with great caution since they act as a signifier of lack of diversity.
Unpaired t-Tests were conducted to compare the data collected from the original model
test with that of the data collected from the other models. This was done to see whether
there was a chance that the models did not actually affect the perceived diversity in the
scene. The comparison between the basic (random skin) model and others showed a sig-
nificant difference in the number of clones identified (therefore perceived diversity)
(p<0.001) in each condition, except the basic scaling version where p <0.05.
In order to see how the extra complexity of additions to the new model (colours
and accessories) affected the perceived diversity, t-tests were conducted on the new
model and its more complex variations (colour palette alterations and accessories).
The results show that while variation of colours was not significant in increasing per-
ceived diversity (p=0.26), the addition of random accessories did significantly in-
crease diversity (p<0.05).
An experiment was run in which the FPS of the software was recorded as the total
number of agents in the scene increased. An agent was instantiated once per frame
until reaching a maximum of 1,500 agents at once. Data was recorded once every
three frames to reduce the load on the system (see figure 9).
Four tests were carried out based on four conditions: The basic model shape, the
“best case” of the new model (lowest possible polygon count), “average case” (nor-
mal probability of additional accessories/polygons), and worst cast (all agents have
maximum polygons allowed by accessories).
In order to ensure a fair test, the four simulations were run on the same machine
with a minimised number of other programs active at run-time. The simulations were
carried out using a locked random seed, ensuring that the movement and character
generation would be the same each time (the same as with the previous experiment).
Comparing the average case of the new model with the original model's frame rate
shows an average difference of 1.568132 frames per second which at high frame rates
is negligible, however the frame rate for the new model drops below twenty-five
frames per second at around 376 total agents whereas the frame rate for the original
drops at 413, which is a noticeable difference of 37 extra agents.
There is also a dramatic difference between frame rate for the original model and
the worst-case scenario (in which every member of the crowd simulation is wearing a
hat) however the probability of this case occurring during the simulation is incredibly
slim (P< 0.001 based approximation using Normal Distribution).
Although the new model does have a slightly lower frame rate than that of the
original model, the increased processing overhead is worth the trade-off considering
its noticeable effect on the viewer's perception of the crowd (52.16% improvement on
the original in terms of participants perception of appearance clones).
7 Conclusion
This paper has investigated the player perception of diversity in video game crowds.
In particular, where crowds are procedurally generated, there is a potential gameplay
issue where players experience loss of immersion when exposed to game crowds
featuring frequently repeated character models. This was confirmed through a
pilot qualitative focus group activity where participants watched game videos and
commented on lack of diversity in crowds.
Taking inspiration from Sheldon's body somatotype theory a new character genera-
tion algorithm was designed with a character model mesh that would deform to repre-
sent different body shapes [7], and also vary based on colour, height, weight and
accessories (e.g. hats). In experimental conditions, the different facets of the model
generation process were tested to understand user perception of diversity in virtual
crowds. This relative measure is much more important than the absolute measure of
diversity since it is based on the personal experience of the player within the game.
With all character generation aspects operating, the new model significantly im-
proved upon the basic version (P < 0.0001) in terms of reducing the user's perception of
crowd diversity. However, we identified that when the new model was colour-tinted
References
1. Booth, M.: The AI Systems of Left 4 Dead (pdf) (2009),
http://www.valvesoftware.com/publications/2009/
ai_systems_of_l4d_mike_booth.pdf
2. Claypool, M., Claypool, K., Damaa, F.: The effects of frame rate and resolution on users
playing First Person Shooter games. In: Electronic Imaging 2006 (2006)
3. Grimes, B.: Shading a Bigger, Better Sequel (pdf) (2010),
http://www.valvesoftware.com/publications/2010/
GDC10_ShaderTechniquesL4D2.pdf (accessed January 2, 2013)
4. Kirman, B.: Emergence and Playfulness in Social Games. In: Proceedings 14th Academic
Mindtrek Conference, Tampere. ACM (2010)
5. Lewis, M.: Initial Models created by Lewis’s Maya Plugin (2000),
http://citeseerx.ist.psu.edu/viewdoc/
download?doi=10.1.1.23.3410 (January 3, 2013)
6. McDonnell, R., Larkin, M., Dobbyn, S., Collins, O.C.: Clone attack! Perception of crowd
variety. In: SIGGRAPH 2008, Los Angeles (August 2008), Article No. 26
7. Sheldon, W.: Atlas of Men: A Guide for Somatotyping the Adult Image of All Ages. New
Edition. Macmillan Pub. Co. (1970)
8. Szymanezyk, O., Dickinson, P., Duckett, T.: Towards agent-based crowd simulation in
airports using games technology. In: O’Shea, J., Nguyen, N.T., Crockett, K., Howlett, R.J.,
Jain, L.C. (eds.) KES-AMSTA 2011. LNCS, vol. 6682, pp. 524–533. Springer, Heidelberg
(2011)
9. Thalmann, D., Musse, S.R.: Somatotype Deformation on a Simple Human (2000)
10. Thalmann, D., Musse, S.R.: Crowd Simulation, 1st edn. Springer, London (2007)
11. Vlachos, A.: Rendering Wounds in Left 4 Dead 2 (pdf) (2010),
http://www.valvesoftware.com/publications/2010/
gdc2010_vlachos_l4d2wounds.pdf (accessed January 2, 2013)
Valentina Rao
Playful Pandas
Galgenstraat 11, 1013LT Amsterdam, Netherlands
v@playfulpandas.org
1 Introduction
A growing number of online and mobile software products aim at influencing user
behavior through interactions that mimic the dynamics or the aesthetics of (video)
games; the diffusion of such phenomenon calls for an increase in attention on what
strategies are available to persuade in a gaming context and their efficacy.
In the last years, aside from the already well established practice of video games
for persuasive purposes called "persuasive games" or "games for change" (or
sometimes by the even more general term of "serious games"), that usually offer a full
game experience that is separate from everyday experience, we have witnessed the
emergence of other practices, called “gamification” [1] or “gameful design” [2], that
instead employ only selected elements from game mechanics, dynamics and
aesthetics, often with the final goal of influencing behavior towards a desired goal.
While the boundaries of the above mentioned definitions are subject to ongoing
debate, the actual distinctions between these different game forms in practice is quite
blurry, and the initial differentiation between persuasive games as systems that use a
fictional, immersive environments with rules, goals and a game universe [3] and
gamification or gameful design that instead employs only game elements or game
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 368–379, 2013.
© Springer International Publishing Switzerland 2013
“atoms” [4] doesn’t hold anymore as in different databases often the same titles are
listed alternatively as gamification or as serious/persuasive games.
In spite of this, persuasive games and persuasive gamification and gameful design
continue to develop separately both as academic fields and as design fields.
Theoretically and methodologically, the difference between the two is still vast: the
definition of persuasive games has been created in conjunction with the definition of
procedural rhetoric, a game-specific kind of rhetoric that achieves persuasion by
representing ideas and experiences through the computational procedures that recreate
the virtual processes portrayed by the game [5].
Maybe because of the theoretical nature of procedural rhetoric, which makes it
difficult to empirically measure its results, existing works on procedural rhetoric tend
to ignore the design issues that relate to persuasive effectiveness.
On the other side, gamification as a new phenomenon born out of marketing
strategies has still to develop as a research field, and while persuasive strategies are an
over-mentioned topic, scientific interest is superficial and instrumental to developing
some ready made standards to arouse engagement derived from behavioral science,
like the popular usage of points, badges and leaderboards, and the strategies
themselves are often accused of being manipulative and coercive [6]. The theoretical
approach called gameful design tries to give scientific and ethical perspective to
gamification principles, supporting designs that produce positive change in the world
and that stimulate the better instincts of the players, instead of the worst [7] and
expanding the disciplinary boundaries to include officially studies of psychology of
persuasion [8].
The approach suggested in this paper is to temporarily set aside definition issues
and try to focus on the theme of persuasive goals that is shared by all, considering that
a large majority of gamification by definition wants to motivate users towards
achieving defined goals [1], and persuasive games use rhetoric for defending a point
or opinion and changing the user’s attitudes and ultimately behaviors.
In this perspective, both kinds of artifacts can be considered as persuasive
“interventions”, to borrow the term employed in social sciences to describe initiatives
aimed at behavior change [9], and different sorts of game forms can be considered as
declinations of the generic category of “game systems”, information systems that use
the interaction modes of games to achieve a persuasive goal. This angle follows the
assumption that video games are first of all HCI principles characterized by shared
values (the playful interaction) that reframe the experience in a certain direction [10].
Lockton highlights how persuasive design is explicitly intended (my italics) to
influence users towards particular behavior [11] to distinguish from other design
artifacts that can be susceptible to interpretations that find them persuasive, but in
which the implementation of persuasion is not an issue. To the purpose of studying
the specificities of such intentional design, Lockton renames persuasive design
“design with intent”. This explicit intent is what sets aside persuasive games and
gamification systems from entertainment games, and what connects them beyond all
theoretical differences, together with the frame of reference mentioned above.
For this reason, while the association between the two fields of persuasive games
and gamification might still need some theoretical refinement, it is nevertheless
proposed here not just for the sake of argument but to offer an interdisciplinary
perspective that revolves around the problem of effectiveness, and initiate a research
direction in the hope of arousing further discussion about implementation practices
that draw equally from game design and persuasive design.
This paper is conceptual in nature, the following pages will offer an overview of
the different angles on the topic of persuasion through games, and tentatively propose
the notions of game systems as behavior change support systems [12] and of game
persuasion through human computer interaction as opposed to procedural rhetoric as
the first steps in the development of a theoretical framework.
Not accidentally, the second thread consists in works that instead use methodolo
gies from social sciences and focus on the evaluation of efficacy in specific cases.
Evaluation in the majority of cases considers the general efficacy of the game medium
with respect to other mediums or with respect to the general efficacy of the game
experience [17] [18] [19] without further investigations into which design elements
contribute to the results. Bang et al. consider persuasion from a learning perspective,
and view the persuasive process as a kind of cognitive and behavioral learning [20],
while others consider persuasion from the perspective of social cognitive theory and
self-efficacy, narrative persuasion and usage of stories [17]. The main problem in this
approach is that games developed with very different purposes such as teaching,
training, and persuasion are analyzed and assessed with the same tools, leaving re-
searchers to denounce the necessity of specific methodologies for different goals [21].
With very rare exceptions [20] [22], available psychological theories of persuasion
and evaluation procedures of effectiveness are not employed for game systems, and a
tentative agenda for future work on the issue of efficacy evaluation of such systems.
A third thread can be found in those works that instead adopt the persuasive
technology framework. Persuasive technology theory [23] maintains that technology
can be designed to change attitudes and behaviors, and can employ a large array of
channels, including digital games.
One sentence on persuasive tech
Works employing the persuasive technology framework either use games as a
platform for the exploration of persuasive strategies in relation to different issues such
as for instance culture and persuasive technology, either they focus on a specific
subtopic, such as cultural differences in persuasion [24], and a systematic framework
in that direction is yet to be defined.
To these three main threads a more recent fourth one can be added, consisting in
recent works that consider game-like interactions such as gamification and gameful
design; although the focus of most publications is rather on motivational strategies
and engagement than on persuasive architecture [25] [1], the persuasive intent and
consequent design is recognized and analyzed in single cases (for example, [26]).
A common weakness in otherwise diverse angles can be found in the lack of proper
evaluation methodologies, as more often than not the few existing ones are
incomplete in relation to both design heuristics and users response. The main obstacle
in developing effective methodologies for evaluation seems to lie in the absence of a
general framework to conceptualize persuasion in games beyond the more developed
model based upon the notion of procedural rhetoric, in which the focus is on artistic
expression and interpretation, both difficult to measure.
products. It is not possible here to go in depth into the arguments surrounding serious
games classifications; to summarize, persuasive games suffer from the same
classification issues visible in classifications of serious games, in which games are
categorized according to market areas, distribution areas, or reception areas [27], but
rarely according to their designed intent, to reiterate Lockton’s expression.
A more pragmatic classification categorizes games according to their kind of
gameplay, the purpose of the design, and the scope of the user contexts in which the
game is received (G/P/S model) [28], although this model reduces most serious games
to the purposes of message broadcasting, data exchange, and training.
As maintained earlier, a first step in developing a research focus on persuasive
design in games is to restrict the field only to games with a stated persuasive intent,
and that means creating a very clear distinction between those and other kinds of
games such as those for education or training. To this goal, an analysis has been
conducted on some of the largest online databases that include persuasive games: the
online database of Games for Change, association that focuses on games for social
impact, the database of Health Games Research, a US program that supports research
and development of health-related digital games, and the database of Gamification.co,
official website for gamification and design for engagement (for gameful design
unfortunately existing databases consider mostly academic publications and not
games).
First, the whole databases have been categorized according to purpose, and
subsequently all games with a specific persuasive purpose have been extrapolated.
At first the categories employed to isolate game forms with persuasive purposes
have been borrowed from the categories defined by the rhetoric scholar Kinneavy to
describe the different communicative purposes of rhetorical discourse, categories that
in precedence have been suggested as a useful tool to categorize games according to
their intent, rather than by other elements such as context or gameplay. [29]
Such categories that in discourse theory are articulated as referential (educational,
scientific), to present, inform of, educate on, or discuss about an existing reality,
persuasive (with a persuasive purpose), such as oratory, propaganda, advertising and
safety warnings, literary or artistic, when the primary goal is to “be beautiful”, such as
in literature and theatre, and expressive, when the goal is to express a self or a social
unit (diaries, manifestos) (Kinneavy, in [29]).
After applying these categories to the corpus of games examined, the shortcomings
of this model were immediately clear in that a percentage of games and especially of
gamification projects, outside of wanting to be educational/scientific, rhetorically
persuasive, artistic and expressive, were in fact not just supporting a discourse but
they were the place of the action, like for instance in games that help children bear
painful therapies or gamification systems that help players achieve their house chores
(Chore Wars).
By focusing just on those games which intent was overtly persuasive, it was possible
to recognize five different strategies to achieve persuasion1:
1
A full study reporting all the statistics is to be published soon, with extensive examples.
Interestingly enough, this empirical distinction made during the analysis of game
examples was more precisely explained in a pre existing text by the persuasive design
scholar Oinas Kukkonen, asserting that online persuasive design systems basically
utilize two modalities to persuade, either computer-human persuasion, or computer
mediated persuasion [30].
Oinas Kukkonen elaborates further, suggesting that persuasive technologies can be
categorized by whether they affect attitude and behaviors through human computer
interaction, which often uses patterns similar to social communication and features
the computer system as an actor and interlocutor, or by performing a mediating role
and allowing computer mediated communication like discussion forums, blogs, chats
and similar [30] (my italics) (also in Cassell et al. [31]).
After ulterior analysis, it was clear that the persuasive strategies enacted through
gamification systems and persuasive games can be understood through the two
modalities of mediated communication, in which the game is a medium to transmit a
message (procedural rhetoric, product placement, aesthetic experience) and of human
computer interaction, in which the game environment is used to facilitate an action or
to influence behavior during a process (facilitation, behavior change support).
By following this logic, then procedural rhetoric qualifies as computer-mediated
communication, in which the game system is used to transmit meaning from the de-
signer/author to the player, and when the game experience is over the player ideally
takes away an idea, a message that is conceptual in nature, and possibly affects in
greater degree attitudes rather than behaviors.
In this view, video games that produce rhetorical discourse on different topics, for
instance children’s labor [32], or sex change (Dys4ia [33]), perform the role of the
medium, one of the three roles that computers can play as persuasive technology,
together with tools and social actors [23].
Fig. 1. The game Participatory Chinatown lets citizens provide their real input about city planning in
a fictional environment
On the other hand, systems that aim to persuade through direct interaction during
the performance of a task (such as breathing correctly with cystic fibrosis [34] or
dealing productively with fellow citizens in urban management [35]) or systems that
facilitate an action by adding engagement, like many gamified systems do, can be
seen at the same time as tools, as they help perform an action, and also as social actors
and interlocutors, as the systems engages in dialogue with the player, through
feedback, rewards or other tools [23]. To note that such dialogue is not “about” a
topic external to the action to be performed, in which case the whole interaction takes
the modes of a representation or, as Bogost defines it, “representative simulation” [6]
to convey a message, but rather its communication regards the “how” an action is
performed during the process itself.
In this second case, the gaming part is instrumental to reaching some goal, giving a
new depth to the notion of instrumental play, which happens when hardcore gamers
play compulsively to gain points or status in multiplayer games (MMOs) and the act
of playing is instrumental to external purposes [36], purposes that can also be not
related to the game.
Existing models for persuasive games don’t provide the conceptual tools to
describe this other approach to persuasion in games, different from procedural
rhetoric. To that purpose, it is suggested here to utilize the notion, originally
conceived to describe information systems developed for persuasive purposes, of
Behavior Change Support Systems.
provide a model also for the understanding of ethical issues involved in the use of
game systems.
Current works on behavior change support systems seem to favor case studies that
deal with human-computer persuasion, but in fact the model can be used to consider
both computer-mediated persuasive communication and human-computer persuasion.
4.1 Elaboration Likelihood Model and the Many Shades of Game Persuasion
BCSSs offer comprehensive tools to understanding the different modalities of
persuasion because they interpret such modalities in the light of the Elaboration
Likelihood Model (ELM), in which a general theory of attitude change is developed,
by identifying two main processes that underlie the effectiveness of persuasive
communications. The first process is called “direct route”, and takes place when the
message presents a direct argument that takes a long time to be processed. The second
process, the “indirect route”, employs emotional, almost subconscious heuristics that
affect the user in a less conscious (and less effective) way. The term “elaboration”
signifies the extent to which a person scrutinizes the persuasive message and its
arguments and “the likelihood of elaboration will be determined by a person’s
motivation and ability to evaluate the communication presented.” [40]
An advantage of employing the ELM model as a main model of persuasion (also
used to describe and understand behavior change supports systems) is that it
comprehends all kinds of persuasive communications, both arguments and facilitation
of tasks: “In the ELM, arguments are viewed as bits of information contained in a
communication that are relevant to a person’s subjective determination of the true
merits of an advocated position”. [40]
A definition of certain game systems as behavior change support systems doesn’t
contradict the main theory that sees persuasive games as vehicles for procedural
rhetoric, but simply puts that particular kind of game systems on a different point in
the spectrum of all the possible persuasive communications that can take place
through gaming.
Another advantage of using such model is that it provides a first integration of
various classic theories of persuasion, other than the ELM model and puts them to use
for the structuring and evaluation of information systems and software development
[30].
With regards to game systems, ELM offers the advantage of providing a
visualization of how the strategies used in procedural rhetoric and in behavior change
support systems can be placed on the two opposite sides of one spectrum:
representational procedural rhetoric offers procedural argument that address the
central route, by being based on logic and because they embed a message in a game
experience and not a game experience in a process, like behavior change support
systems do.
Further development of the correlation between game elements and routes of
persuasion is in the agenda, and there are indications that such research angle can be
useful in finding methodologies for the evaluation of persuasive effectiveness,
analogue to the Persuasive Systems Design (PSD) model [41], created to evaluate
5 Conclusions
Acknowledgments. Many thanks to professor Oinas Kukkonen for his long distance
support and availability, to Sitwat Langrial and the other colleagues at Oulu research
group whose discussions made these theoretical developments possible.
References
1. Zichermann, G.: Game Based Marketing. John Wiley & Sons, New Jersey (2010);
Baldonado, M., Chang, C.-C.K., Gravano, L., Paepcke, A.: The Stanford Digital Library
Metadata Architecture. Int. J. Digit. Libr. 1, 108–121 (1997)
2. Deterding, S., Dixon, D., Nacke, L.E., O’Hara, K., Sicart, M.: Gamification: Using Game
Design Elements in Non-Gaming Contexts. In: Proceedings of the 2011 Annual
Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA 2011),
Vancouver, BC, Canada. ACM Press (2011)
23. Fogg, B.J., Cuellar, G., Danielson, D.: Motivating, Influencing, and Persuading Users. In:
Jacko, J.A., Sears, T. (eds.) The Human- Computer Interaction Handbook: Fundamentals,
Evolving Technologies and Emerging Applications. Lawrence Elbaum Associates,
Taylor& Francis, New York (2007)
24. Khaled, R., Biddle, R., Noble, J., Barr, P., Fischer, R.: Persuasive Interaction for
Collectivist Cultures. In: Seventh Australasian User Interface Conference (AUIC 2006),
Hobart, Australia (2006)
25. Deterding et al: CHI 2011 Workshop Gamification: Using Game Design Elements in Non-
Game Contexts. In: CHI 2011 Proceedings, May 7-12, Vancouver, BC, Canada (2011)
26. Inbar, O., Tractinsky, N., Tsimhoni, O., Seder, T.: Driving the Scoreboard: Motivating
Eco-Driving Through In-Car Gaming. In: CHI 2011 Workshop Gamification: Using
GameDesign Elements in Non-Game Contexts, CHI Conference Vancouver (2011)
27. Klimmt, C.: Serious games and social change. In: Ritterfeld, U., Cody, M., Vorderer, P.
(eds.) Serious Games: Mechanisms and Effects. Routledge, USA (2009)
28. Djaouti, D., Alvarez, J., Jessel, J.-P.: Classifying serious games: The G/P/S model. In:
Felicia, P. (ed.) Handbook of Research on Improving Learning and Motivation Through
Educational Games: Multidisciplinary Approaches. IGI global, Hershey (2011)
29. Rao, V.: How to say things with actions: A theory of discourse for video games for
change. In: Digra Conference, Hilversum (2011),
http://www.digra.org/dl/db/11312.32336.pdf
30. Oinas-Kukkonen, H., Harjumaa, M.: Towards deeper understanding of persuasion in
software and information systems. In: Proceedings of the First International Conference on
Advances in Human-Computer Interaction (ACHI), pp. 200–205 (2008)
31. Cassell, M.M., Jackson, C., Cheuvront, B.: Health communication on the Internet: An
effective channel for health behavior change? Journal of Health Communication 3 (1998)
32. Sweatshop, game, http://www.playsweatshop.com
33. Dys4ia, game, http://www.newgrounds.com/portal/view/591565
34. Ludicross, game, http://www.youtube.com/watch?v=s99Iwe_p3nc
35. Participatory Chinatown, game, http://www.participatorychinatown.org
36. Taylor, T.L.: Power gamers just want to have fun?: Instrumental play in a MMOG. In:
Digra 2003 Proceedings Utrecht University, The Netherlands (2003)
37. Salen, K., Zimmerman, E.: Rules of Play, Game Design Fundamentals. MIT Press (2004)
38. Räisänen, T., Lehto, T., Oinas-Kukkonen, H.: Practical findings from applying the PSD
model for evaluating software design specifications. In: Ploug, T., Hasle, P., Oinas-
Kukkonen, H. (eds.) PERSUASIVE 2010. LNCS, vol. 6137, pp. 185–192. Springer,
Heidelberg (2010)
39. Petty, R.E., Cacioppo, J.T.: The elaboration likelihood model of persuasion. In: Berkowitz,
L. (ed.) Advances in Experimental Social Psychology, pp. 123–205. Academic Press, New
York (1980)
40. Räisänen, T., Lehto, T., Oinas-Kukkonen, H.: Practical findings from applying the PSD
model for evaluating software design specifications. In: Ploug, T., Hasle, P., Oinas-
Kukkonen, H. (eds.) PERSUASIVE 2010. LNCS, vol. 6137, pp. 185–192. Springer,
Heidelberg (2010)
41. Isbister: Game Usability. Elsevier, Burlington (2008)
42. Bernhaupt, R. (ed.): Evaluating User Experience in Games Concepts and Methods.
Springer, London (2010)
1 Introduction
The modern society has evolved in many aspects in the recent years, but people still
lack routines and behaviours that support and help the environment. In our everyday
life, we are confronted with situations that compromise our planet's sustainability,
sometimes without even noticing them. Making people aware of this kind of
situations is crucial to change their perspective, concerns and attitudes. With that
objective in mind, we conceived eVision, an application which tries to persuade users
to have a pro-environment behaviour. eVision uses mobile devices and augmented
reality technology to provide users with information regarding the environmental
threats in their surroundings and to offer a game-based activity where users are
rewarded when completing tasks of interest. To engage users, Snowkin, a
customizable virtual character, interacts with the users encouraging them to proceed.
It is unrealistic to think that everyone will change his or her behaviours day over
night. However, making people aware of the problems and their possible solutions
helps to engage them in activities that contribute for these solutions. Moreover, social
influence and peer pressure help to disseminate and reproduce desirable behaviours
[1]. Therefore, users activities and achievements while using eVision can also be
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 380–391, 2013.
© Springer International Publishing Switzerland 2013
2 Related Work
3 eVision
Fig. 1. User Interaction: (1) Detection, (2) Cleaning, (3) Overlaying of the Environmental
Threat
After the user and task analysis, eVision development followed an iterative design
process that started with the design of a paper prototype, which was informally
evaluated by several users in 3 iterations. According to the results achieved during
this evaluation, the original design was improved and a second computational
prototype was implemented as described below in this paper.
During the second phase, the most important detected issue was related to the
interface used to customize the Snowkin. A slot machine interface element was
included to allow users to select Snowkin customizing items (e.g. sweatshirts, hats).
The third phase demonstrated that the system development could proceed to the
implementation of the computational prototype.
Regarding the informal questions asked in the end of each test, most feedback was
positive and, generally, the participants felt that eVision was an interesting and
innovative application.
3.2 Implementation
eVision was developed for iPhone 4 using iOS 5.1. All the implementation code was
written using Apple’s Xcode IDE in Objective-C. The eVision’s interface elements were
made on Xcode Interface Builder, with some exceptions generated through specific code.
The image processing module was developed using the OpenCV library compiled for iOS
5.1, using C++ to write the module responsible to detect the dynamic environmental
threats present on the snapshots. Facebook mobile API for iOS was used for Facebook
integration, with minor refinements. Figure 2 illustrates the system’s architecture.
eVision current implementation allows users to detect different pollution sources:
cars, airplanes and factories. Knowing the static location of factories, GPS was used
to identify factories in the users surroundings. Although it was not the main focus of
this work, image detection methods had to be investigated in order to detect the
dynamic position of cars and airplanes during eVision’s game activity. In conjunction
with the OpenCV library, one airplane detection method was designed and for car
detection tests were conducted using two different methods.
The airplane detection method is based on contour detection and analysis. It
worked quite well, since tests were conducted in a controlled environment (always on
a clear sky). Limiting the sky appearance we could achieve successful airplane
detection based on contours.
Regarding vehicle detection, two different methods were tested, being the first one
based on Cascade Classifiers and Haar-like features. This method was considered
since it has been demonstrated to be successful detecting cars from various angles [2].
Unfortunately, when testing this method in eVision two problems arose: cars’ side
views would always produce unsatisfactory results and the method was extremely
slow on iPhone 4 hardware.
A contour based method for cars’ detection was then designed. Several problems
were immediately identified: different cars had different contour shapes and the
background did not have a uniform colour, introducing a lot of noise when processing
the image. To overcome these problems, we decided to detect cars’ registration plates
instead. This would guarantee that the application would always search for something
with a rectangular shape independently from the car type and background.
The mobile device accelerometer was used to detect if the users were aiming at an
airplane on the sky or a car, in order to activate the appropriate image detection
process. After some optimization procedures, the image detection accuracy was
sufficient to ensure the validity of the conducted tests It is worth mentioning that
image detection is not the main focus of this work and any improved methods can
easily replace the currently used by eVision.
When the user accepts the invitation to participate on the game activity, a
countdown will appear and the activity starts. A score gauge appears on the upper
right corner and a “capture image” button with the eVision’s logo shows up on the
bottom left corner, allowing users to take a snapshot (Fig. 4a). Snowkin assumes his
motivational role and pushes the user to find environmental threats.
While the user scans his surroundings with the mobile device, static environmental
threats (currently factories) are represented on the screen, in real time, in the correct
direction of their location. Information about factories is given on a smoke cloud to
represent the pollution effect caused by them.
When a threat is detected on a snapshot taken by the user, Snowkin warns the user so
he can clean it (Fig. 4b). After the detected threat is cleaned (the users just need to use
their finger to rub the mobile phone display over the detected threat, as shown in Fig. 1), it
disappears or it is automatically overlaid with a pro-environmental object (e.g.: overlaying
a car with a bicycle) and Snowkin immediately congratulates the user and motivate him to
keep going. The score gauge is also filled depending on the value of the threat cleaned (a
greater area will be filled according to the score value of the eliminated threat).
It is important to notice that Snowkin’s face expression changes according to the
game contextual situation (Fig. 5). After cleaning threats, Snowkin’s expression
changes to a happy face.
After the score gauge is fully filled, the current game level ends and Snowkin
praises the user. A score table is shown, discriminating the values that were factored
to calculate the final score in green leaves. This moment, when the user is focused on
the screen, is exploited to persuade the user by providing him with information
regarding the effects of his actions on the environment.
The eShop interface allows the user to spend the green leaves earned playing
eVision’s game activity. A list of items to customize Snowkin is presented in an
appealing way using a small icon to represent each item, showing the item’s name and
slot, as well as its price in green leaves (Fig. 6a). The user’s green leaves balance is
shown on the upper right corner of the screen and the items that were already
purchased have a Snowkin icon.
Fig. 6. Customization: (a) eShop and (b) Snowkin Menu (c) Facebook
After buying items the user can visit the Snowkin interface and customize its
appearance. The commitment and time the user puts on building Snowkin’s appearance
makes him more likely to follow the character’s advices and recommendations regarding
the environment, thus being persuaded by the application. In figure 6b every column
represents an item slot (head, torso, legs, feet and hands). The slot machine style selection
is very intuitive and avoids the danger of simultaneous choosing two different items of the
same slot. The Statistics menu option allows the users to keep track of their game
progression and to access data related to the application, such as time spent cleaning
threats or the number of each kind of threat cleaned.
Finally, the Facebook interface (Fig. 6c) is responsible for handling the Facebook
features available on eVision, allowing users to connect and share their activity. After
verifying the user’s credentials, the button to publish statistics and the “Like” button
for Project DEAP’s Facebook page become available. A user’s profile picture also
appears in the frame held by Snowkin.
4 Evaluation
User tests were performed in order to evaluate eVision’s usability, persuasive
potential and impact on users environmental awareness and attitudes.
were not enough to proper evaluate how people would use and be affected by the
application. To overcome these shortcomings and evaluate how people were affected
by eVision, usability tests based on a live setting were carried out with a diverse
group of twenty participants (50% male) aged 18-56 years old (mean = 30.9). 10% of
the participants had MSc degrees, 40% had BSc degrees and 50% had high school
education. 30% of the participants were students, 20% were unemployed and 50%
were full time workers. All participants were familiar with new technologies. Most of
them use mobile phone, computer and Internet on a daily basis.
The tests took place, in the outdoors, in a location where some road and air traffic
were expected, in the afternoon during a clear sky day. These circumstances allowed
us to control the experience, providing the users with appropriate conditions to use
eVision during the period of the test, and to focus on the main objectives of the user
tests (we were not testing the performance of the image detection algorithms).
Before starting to use eVision, each participant was briefed on the context and
functionalities of the mobile application. The participants had twenty minutes to use
the application at their convenience. In the end of each test, a questionnaire was given
to each participant to evaluate the eVision’s usability and potential persuasive effect.
Questionnaire
Results
Participants agreed that eVision responded quickly to the various actions and that it
required just a few steps to perform each task. They also found very easy to detect a
vehicle, while the detection of a polluting infrastructure or a plane was more difficult,
because the former ones were more frequent to find in the area where the application
was tested. Anyway, the participants considered easy to obtain information relating to
the environment, as well as to clean any detected threats. Buying items and
customizing Snowkin, as well as consulting statistics regarding users’ performance
were also perceived as being easy to execute.
Most of the participants (45%) did not sign in to Facebook, mainly because they
didn’t feel comfortable entering their credentials on a mobile device which was not
their own. This made the experience less customized for these users and this also
might have decreased the persuasive capacity of eVision due to a minor tailoring
effect (e.g. Snowkin could not address the user by his name). Those who signed in
found it easy to publish statistics on their “wall”.
Users enjoyed the experience when using the eVision (mean = 2.25, in a scale 1-5
where 1 represent excellent). The fact that 55% of participants considered that the use
of the application contributed to increase their awareness regarding environmental
issues demonstrates the eVision’s persuasive potential.
From the analysis of the question based on the Microsoft “Product Reaction Cards”
we concluded that all users held positive feelings when classifying their experience
using eVision. The most selected word was “interesting” (80%), followed by the
words “curiosity” (70%), “fun” (55%), “friendliness” (35%), “empathy” (25%) and
“joy” (15%). Emotions such as friendliness and empathy show a relational dimension
that demonstrates the social role assigned to Snowkin, consequently ensuring him a
greater persuasive power. Fun and joy are also relevant characteristics for a game
application. Negative emotions such as boredom, frustration and disappointment were
also experienced but only by a very small number of participants (5%).
We also wanted to study the Snowkin design and role to understand how his
appearance, face expressions, and dialogs could influence users feelings and
perception. In general, the feedback from users was positive and congruent with the
image that we wanted to create for the virtual character. Users agreed that Snowkin
cares about the environment (mean = 1.1 (male), 1.3 (female)), it is a friendly
character (mean = 1.2 (male), 1.2 (female)) and they felt good when the Snowkin
praised their progression (mean = 1.3 (male), 1.2 (female)). Users also agreed that
Snowkin is expressive (mean = 1.8 (male), 1.7 (female)), the information given by
Snowkin was credible (mean = 1.8 (male), 1.7 (female)) and that they will take into
account the information provided by Snowkin (mean = 2.0 (male), 1.9 (female)). The
participants’ opinion about the Snowkin was not significantly different between men
and women (ρ > 0,05 for all the statements, using Mann-Whitney U test).
These results showed the considerable potential of persuasion present in eVision
through Snowkin, which can put into practice several persuasion principles, such as
tailoring and conditioning. As a social actor, Snowkin is also very effective: it has not
only the right physical cues (e.g. expressiveness), but also the right psychological
(e.g. cares about the environment) and social traits (e.g. friendly). The Snowkin was
considered a teacher or advisor for the majority of women (60%). Women considered
it also a friend (20%), a teammate (10%) and "a friendly character" (10%). On the
other hand, men considered it a teammate (50%) and a teacher or advisor (40%) –
10% of men have not answered this question. Anyway, it is important to note that in
all cases, it was clear that the users always established a positive relationship with the
virtual character.
Most participants felt motivated to go on and wanted more time to continue
experiencing the eVision. The Snowkin character had a particular strong impact,
driving the participants to keep scoring, so they could buy more items to customize
the virtual character. Some participants became sad when the test was over and they
had to say goodbye to Snowkin.
Finally, the participants made suggestions and recommendations to improve the
application, such as; the boxes used to highlight the detected threats should be more
appealing; and the icons that appear after the cleaning of the detected environmental
threats should be more diverse. Other users expressed their satisfaction with eVision:
"I really enjoyed playing this game, because it gave me more information about the
environment", "the idea is quite original and well developed", "I really enjoyed the
experience".
To better evaluate the persuasive effect of eVision and to check if it still holds
sometime after the eVision experience, an impact test was performed. Roughly one
month after the usability tests were conducted, participants were asked to answer a
new questionnaire, expressing their opinions regarding four statements. They had to
indicate their degrees of agreement using a five-point Likert scale ranging from 1
(totally agree) to 5 (totally disagree).
Participants agreed that eVision played an important role in alerting them to
Earth’s sustainability problems (mean = 2.25) and they also reported that their
experience with eVision influenced their daily life decisions, such as what type of car
they should buy (mean = 2.3). They felt that Snowkin’s dialogues and information
were still relevant to them a month after the experience (mean = 2.35). Curiously,
they also reveal that they missed Snowkin after interacting with it (mean = 2.05),
showing the importance of the virtual character for the goal of this mobile application.
Although the results were fairly positive it is necessary to perform further tests
with a larger number of users and allowing them to use the application for a longer
period of time to better evaluate the persuasive power of the application.
which interacts with users encouraging them to play, helping to persuade them to
adopt pro-environmental behaviours and making them aware of environmental threats
in their surroundings. The paper also presents the user studies performed to evaluate
the application’s usability and impact on users environmental awareness and attitudes.
eVision is making a contribution to increase people environmental awareness
through an engaging experience. The results from the user tests were very positive,
regarding both the gameplay and eVision’s persuasive ability. Although it is difficult
to evaluate behaviours since these are complex and non-linear, we believe, from the
feedback we got, that eVision can help to shape people’s attitudes and behaviours
towards a better environmental conscience.
Despite the positive and encouraging results there is room for improvements.
Future work includes the detection of additional environmental threats, as well as the
optimization of existing image detection methods and algorithms. To better evaluate
the persuasive effect of the application, further user tests that include the use of the
application by the participants during a longer period of time (e.g. a few days or
weeks) will be performed.
Acknowledgments. This work is funded by Fundação para a Ciência e Tecnologia
(FCT/MEC), Portugal, in the scope of project DEAP (PTDC/AAC-AMB/104834/2008)
and by CITI/DI/FCT/UNL (PEst-OE/EEI/UI0527/2011). The authors thank Bárbara
Teixeira for her contribution on the graphic design.
References
1. Cialdini, R.: Influence: Science and Practice. Allyn & Bacon, Boston (2001)
2. Horta, R., Jonna, K., Krishna, P.: On-road vehicle detection by cascade classifiers. In: Proc.
of the Third Annual ACM Bangalore, Compute 2010, Bangalore, India, January 22-23.
ACM Press (2010)
3. Fogg, B.J.: Persuasive Technology: Using Computers to Change What We Think and Do.
Morgan Kaufmann, San Francisco (2003)
4. Fogg, B.J.: Captology: The Study of Computers as Persuasive Technologies. In: Abstracts
CHI 1998, Los Angeles, USA, April 18-23, p. 129. ACM Press, NY (1998)
5. Benedek, J., Miner, T.: Measuring Desirability: New Methods for Evaluating Desirability in
a Usability Lab Setting. In: Proc. of Usability Professionals’ Association 2002 Conference,
UPA 2002, Orlando, USA, July 8-12 (2002), http://www.microsoft.com/
usability/UEPostings/DesirabilityToolkit.doc
(accessed on June 8, 2013)
6. Eckles, D., Fogg, B.J.: The Future of Persuasion is Mobile. In: Mobile Persuasion: 20
Perspectives on the Future of Behavior Change, pp. 5–11. Stanford Captology Media,
Stanford (2007)
7. Fogg, B.J.: Mass Interpersonal Persuasion: An Early View of a New Phenomenon. In:
Oinas-Kukkonen, H., Hasle, P., Harjumaa, M., Segerståhl, K., Øhrstrøm, P. (eds.)
PERSUASIVE 2008. LNCS, vol. 5033, pp. 23–34. Springer, Heidelberg (2008)
8. Bogost, I.: Persuasive Games: The Expressive Power of Videogames. MIT Press,
Cambridge (2007)
9. White, S.: Augmented Reality: Using Mobile Visualization to Persuade. In: Mobile
Persuasion: 20 Perspectives on the Future of Behavior Change, pp. 55–62. Stanford Captology
Media, Stanford (2007)
1 Introduction
We live in a world surrounded by technological equipment. The technical pos-
sibilities have developed massively during the last decades and changed our ev-
eryday lives and communication patterns. In the field of so-called collaborative
virtual environments there are many possibilities of application ranging from
gaming, or educational purposes to health care. There are several promising at-
tempts to integrate virtual environments effectively into therapy [13]. Virtual
environments are increasingly used as emotion-inducing stimuli for exposure
therapy. Yet, women and men seem to differ regarding their perception of vir-
tual environments [6], and their behavior in virtual environments [28]. Thus
far however, there is lack of research combining both, environmental factors of
Mareike Schmidt and Johanna Xenia Kafka contributed equally to this paper.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 392–402, 2013.
c Springer International Publishing Switzerland 2013
2 Related Work
The technology acceptance model [3] which is based on the Theory of Reasoned
Action by Fishbein and Ajzen [8] is one of the most commonly used models to
evaluate computer technology systems like communication and entertainment
applications for educational and/or therapy purposes (cf. [5,13,17,24,27]). It has
been shown that perceived usefulness and ease of use strongly influence the be-
havioral intention to actually use virtual environments. Interestingly, Fetscherin
and Lattermann [7] found that technology-related anxiety has no significant
impact on the behavioral intention to use the technology; instead, a strong re-
lationship between the possibility to interact or collaborate with others and a
person’s intention to use the technology was reported.
Immersion: Considering this, other factors possibly influencing the acceptance of
virtual environments are the constructs of immersion and the sense of presence.
While the sense of presence refers to the experience of actually being there in a
virtual environment which is based on a perceptional illusion of non-mediation
[15], immersion can be regarded a characteristic of a technology [22]. As this,
immersion describes the extent to which a computer interface is capable of trans-
porting a vivid, surrounding and inclusive experience to the user, who feels en-
veloped by the presented environment. This approach has been criticized as it
implies that every user should experience the same level of immersion when us-
ing the same technology. Studies however have shown that users seem to differ in
their extent of experienced immersion even when using the very same interface
(c.f. [12,29]). Here, immersion seems to depend on one’s ability to fully focus on
and engage in the virtual environment, thus “forgetting” the outside world.
Environmental factors: When watching a movie, the scenery can become so vivid
and compelling that we start feeling for the characters and we place ourselves into
those artificial settings. It can be assumed that a true life depiction of environmen-
tal factors plays an important role on whether we perceive something as realistic
or lifelike. There is some evidence that environmental factors, such as lighting or
weather conditions, in virtual reality would be effective as mood induction proce-
dures with anxiety-inducing and relaxing stimuli. It seems that a medium must
be able to induce a feeling of presence in order to provoke affective responses [19].
Yet, some other studies showed that the lighting level of the virtual environment
alone has no effect on the affective state of the users [25].
Gender: Gender differences are a widely researched field. Unsurprisingly, the
inclusion of the gender perspective when studying usability factors or the ex-
perience of immersion is gaining more importance in research and the game
industry [28]. It is estimated that 48% of video game purchasers are women [4].
Thus, computer games can no longer be seen as an exclusively masculine domain,
yet some studies still show an advantage of male over female gamers regarding
reported game experience [9]. However, individual differences such as gender and
Following the presented literature above the study focuses on the two following
research questions:
– Research question 1: Are there differences between women and men regarding
technology usage factors (ease of use and perceived usefulness), immersion
or technology-related anxiety?
– Research question 2: Are there differences between two different emotionally
charged context conditions (i.e., a sunny outdoor scene vs. a rainy outdoor
scene, see Section 4.4) regarding technology usage factors (ease of use and
perceived usefulness), immersion or technology-related anxiety?
4 Experiments
4.1 Methods
4.2 Participants
range 20 − 43) years, in the rainy condition the mean age was 23.71 (SD = 4.340;
range 19 − 34) years. All participants had normal or corrected to normal vision
and were fluent German speakers. Table 1 and Figure 1 show the participants
computer and computer gaming experience for women and men separately.
Table 1. Means (M) and standard deviations (SD) for computer gaming hours per
week for males and females seperately
Male Female
M Range SD M Range SD
4.3 Procedure
The study was described to the participants as an experiment about the usage
of virtual realities and was conducted on weekdays between 9 a.m. and 2 p.m.
Upon their arrival, the participants signed the informed consent form and then
filled out a short demographic survey. Subsequently, they were guided to the lab-
oratory and after resting five minutes an experimenter explained to them how
to navigate through the virtual environment using a smart phone device (HTC
Desire SV, Taoyuan) and later set up the head-mounted-display (HMD, Sony
HMZ-T1 3D Visor, Tokyo, Japan). Head movement in the virtual simulation
was tracked with an externally applied head tracking system (TrackIR 5, Nat-
uralPoint, Corvallis, USA). The dwell time in the simulation was five minutes
during which the participants were free to explore the park without instructions
or tasks. Participants were only asked to memorize self-selected objects in the
virtual environment. After the experimental phase the psychological question-
naires were administered (see Section 4.5).
The non-playing characters (NPCs) strolling through the park were mod-
eled on the photographs of the research staff involved in this study. GIMP and
Blender 3D were used as modeling tools, as well as the render engine OGRE3D
for real time rendering. To control the avatar, participants used an Android
smart phone as input device. Subjects could steer their avatar, seen from the
first person perspective, by tilting the phone.
Fig. 3. Pond with ducks and resting NPC in the sunny condition
4.5 Measures
To measure the participant’s evaluation of the virtual environment the self-report
questionnaire Technology Usage Inventory (TUI, [14]) was used. The TUI com-
prises the following four factors: (1) Ease of Use (three items referring to the
perceived user-friendliness, e.g. “The application of this technology is easy to
understand”), (2) Immersion (four items assessing the absorption in a virtual
environment, e.g. “I could forgot my real problems in the virtual situation”), (3)
Perceived Usefulness (four items related to the perceived benefit of the technol-
ogy, e.g. “This technology would help me cope better with my daily duties”) and
(4) Technology-related Anxiety (four items regarding mistrust towards the use
of a specific technology, e.g. “I think that the use of this technology is always
associated with a certain risk”). After the experimental phase in the virtual
environment participants were asked to respond to the 15 items, which were
scored on a seven-point Likert scale (does apply – does not apply). The reliabil-
ity (Cronbach’s alpha) for the scales ranged between α = .70 (Technology-related
Anxiety) and α = .84 (Immersion).
5 Results
5.1 Manipulation Check
To validate the emotion-inducing content of the virtual environments, a short
manipulation check including a single item on a 4-point Likert scale (“I feel
happy in this environment” – “I feel sad in this environment”) was implemented.
The results of the manipulation check indicate that the participants reported a
significantly higher level of happiness in the sunny environment than in the rainy
environment (T = 2.212; p = 0.036).
Table 2. Means (M) and standard deviations (SD) for the scales of the TUI for the
sunny and rainy condition and for gender
Gender Condition
Perceived usefulness 10.79 5.63 11.36 6.29 9.00 5.05 13.14 5.65
Ease of Use 15.93 3.71 17.00 3.08 16.64 4.08 16.29 2.67
Immersion 12.43 4.03 18.64 5.12 13.86 4.94 17.21 5.72
Technology-related Anxiety 12.57 3.43 15.86 4.33 13.79 4.40 14.64 4.06
To assess the influences of gender and group on technology usage factors, im-
mersion and technology-related anxiety a two-way ANOVA was performed. The
four TUI scales Perceived Usefulness, Ease of Use, Immersion and Technology-
related Anxiety constituted the dependent variables. The condition participants
were assigned to and their gender appeared as the between-group factors. A
main effect of the factor gender was found for Immersion (F (1, 24) = 13.87;
p = 0.001) and for Technology-related Anxiety (F (1, 24) = 4.737; p = 0.040). The
only main effect regarding the rainy condition was found for Perceived Useful-
ness (F (1, 24) = 6.029; p = 0.022). Additionally there was an interaction effect
of group and gender for Perceived Usefulness (F (1, 24) = 16.516; p < 0.001).
There were no significant effects or interactions found for the other scales. Over-
all, women scored higher in both, Immersion and Technology-related Anxiety.
Moreover participants in the rainy condition rated the Perceived Usefulness of
the technology higher than those in the sunny condition. And finally, there was
an interaction effect on the judgement of the usefulness of the technology. Men
perceived the technology as being more useful when they were assigned to the
sunny condition, whereas women rated the technology as more useful when they
were part of the rainy condition.
6 Discussion
Given the fact, that virtual environments are increasingly adopted in all areas
of life for all kinds of purposes – ranging from entertainment applications to
References
1. Agarwal, R., Karahanna, E.: Time Flies When You’re Having Fun: Cognitive Ab-
sorption and Beliefs about Information Technology Usage. MIS Quarterly 24(4),
665–694 (2000)
2. Braak, J.: Domains and determinants of university students’ self-perceived com-
puter competence. Computers & Education 43(3), 299–312 (2004)
3. Davis, F., Bagozzi, P., Warshaw, P.: User acceptance of computer technology - a
comparison of two theoretical models. Management Science 35(8), 982–1003 (1989)
4. ESA. Essential Facts about the Computer and Video Game Industry 2012: Sales,
Demographic and Usage Data (2012),
http://www.theesa.com/facts/pdfs/esa$_$ef$_$2012.pdf (May 29, 2013)
5. Fallows, D.: How women and men use the Internet (2005),
http://www.pewinternet.org/Reports/2005/
How-Women-and-Men-Use-the-Internet.aspx (May 29, 2013)
6. Felnhofer, A., Kothgassner, O.D., Beutl, L., Hlavacs, H., Kryspin-Exner, I.: Is vir-
tual reality made for men only? Exploring gender differences in the sense of pres-
ence. In: Annual Proceedings of the International Society on Presence Research.
Philadelphia, USA (2012)
7. Fetscherin, M., Lattemann, C.: User acceptance of virtual worlds. Journal of Elec-
tronic Commerce Research 9(3), 231–242 (2008)
8. Fishbein, M.A., Ajzen, I.: Belief, Attitude, Intention, and Behavior: An Introduc-
tion to Theory and Research. Addison-Wesley (1975)
9. Hartmann, T., Klimmt, C.: Gender and computer games: Exploring females’ dis-
likes. Journal of Computer-Mediated Communication 11(4), article 2 (2006)
10. Hsu, C.L., Lu, H.P.: Why do people play on-line games? An extended TAM with
social influences and flow experience. Information & Management 41(7), 853–868
(2004)
11. IJsselsteijn, W.A.: Presence in Depth. Eindhoven: Technische Universiteit Eind-
hoven (2004)
12. Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., Walton, A.:
Measuring and defining the experience of immersion in games. International Jour-
nal of Human-Computer Studies 66, 641–661 (2008)
13. Kothgassner, O.D., Felnhofer, A., Beutl, L., Hlavacs, H., Lehenbauer, M., Stetina,
B.: A virtual training tool for giving talks. In: Herrlich, M., Malaka, R., Masuch,
M. (eds.) ICEC 2012. LNCS, vol. 7522, pp. 53–66. Springer, Heidelberg (2012)
14. Kothgassner, O.D., Felnhofer, A., Hauk, N., Kastenhofer, E., Gomm, J., Kryspin-
Exner, I.: TUI. Technology Usage Inventory. FFG, Wien (2013),
http://www.ffg.at/sites/default/files/allgemeine downloads/
thematische%20programme/programmdokumente/tui manual.pdf
15. Lombard, M., Ditton, T.: At the heart of it all: the concept of presence. Journal
of Computer-Mediated Communication 3(2) (1997)
16. Nicovich, S.G., Boller, G.W., Cornwell, T.B.: Experienced Presence within
Computer-Mediated Communications: Initial Explorations on the Effects of Gender
with Respect to Empathy and Immersion. Journal of Computer-Mediated Com-
munication 10(2) (2005)
17. Ong, C.S., Lai, J.Y.: Gender differences in perceptions and relationships among
dominants of e-learning acceptance. Computers in Human Behavior 22(5), 816–829
(2006)
18. Papastergiou, M., Solomonidou, C.: Gender issues in internet access and favourite
internet activities among Greek high school pupils inside and outside school. Com-
puters & Education 44(4), 377–393 (2005)
19. Riva, G., Mantovani, F., Capideville, C.S., Preziosa, A., Morganti, F., Villani, D.,
Alcaiz, M.: Affective interactions using virtual reality: The link between presence
and emotions. CyberPsychology & Behavior 10(1), 45–56 (2007)
20. Sanchez-Franco, M.J.: Exploring the influence of gender on the web usage via
partial least squares. Behaviour & Information Technology 25(1), 19–36 (2006)
21. Schumacher, P., Morahan-Martin, J.: Gender, internet and computer attitudes and
experiences. Computers in Human Behavior 17(1), 95–110 (2001)
22. Slater, M., Usoh, M.: Body centred interaction in immersive virtual environments.
Artificial life and Virtual Reality 1, 125–148 (1994)
23. Szell, M., Thurner, S.: How women organize social networks different from men.
Scientific Reports 3 (2013)
24. Terzis, V., Economides, A.A.: Computer based assessment: Gender differences
in perceptions and acceptance. Computers in Human Behavior 27(6), 2108–2122
(2011)
25. Toet, A., van Welie, M., Houtkamp, J.: Is a dark virtual environment scary? Cy-
berPsychology & Behavior 12(4), 363–371 (2009)
26. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of infor-
mation technology: Toward a unified view. MIS Quarterly Archive 27(3), 425–478
(2003)
27. Whitley, B.E.: Gender differences in computer-related attitudes and behavior: A
meta-analysis. Computers in Human Behavior 13(1), 1–22 (1997)
28. Williams, D., Consalvo, M., Caplan, S., Yee, N.: Looking for gender: Gender roles
and behaviors among online gamers. Journal of Communication 59(4), 700–725
(2009)
29. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: A pres-
ence questionnaire. Presence: Teleoperators and Virtual Environments 7, 225–240
(1998)
1 Introduction
Successful entertainment technology is all about achieving the right user ex-
perience. When we think about new computer entertainment, we explore new
experiences that we can offer to the user through the invention of new technol-
ogy and interfaces, or through the development of interesting new application
concepts. However, we should also not forget to think about when and how these
experiences are worthwhile. Not just because the interaction targets useful sec-
ondary goals (e.g. edutainment applications) but also because the interactive
experiences themselves fulfil certain needs.
Designing for user experience is a challenging endeavour. As Hassenzahl says,
experiences are elusive and ephemeral, and resist engineering. His solution is
to similarly look below the surface of the experiences: underlying fundamental
human needs engender and determine to a large extent the experience resulting
from interaction with technology. He suggests that by properly understanding
these fundamental needs, and understanding how they relate to user experi-
ences, we can more easily systematically design for experience by designing for
fundamental needs [1].
In the current paper we mean to introduce into the field of entertainment
computing an overview of insights concerning fundamental human needs. Re-
searchers such as Hassenzahl, Desmet, and others, discuss design approaches
based on psychological insights from various and varied sources [1,2]. We collect
these and expand them with a focus on meaning in life as seen in humanistic phi-
losophy, following the suggestion by Cockton [3]. We summarise the various roles
that these insights can play in our research on new technology, and illustrate the
discussion with examples from the field of computer entertainment.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 403–414, 2013.
c Springer International Publishing Switzerland 2013
Although many acknowledge the value of SDT, there are of course some dis-
cussion points on the SDT in the psychological field. Some point to the aspect of
the subjective-perceived need fulfilment instead of actual fulfilment which seems
to be addressed in SDT [17]. Others suggest a more central social comparative
aspect of need fulfilment, people tend to see their fulfilment with respect to
others [18]. Furthermore, according to Andersen et al., relatedness plays a more
important role and they acknowledge autonomy and compentence as important
needs [19]. However, they also point to an explicit need for meaning and security
and safety -a notion of stability and perceived future psychological and physio-
logical states [19]. Also, Andersen suggests it would be valuable to look at the
coherence with social-cognitive theory of Baumeister and others. Therefore, in
the next section we look at such a theory of Baumeister on the meaning of life,
and the humanistic view of van Houten.
3.1 Baumeister
Baumeister wrote extensively about meaning in life and proposed a set of fun-
damental universal human needs for meaning in life [21]. His theory is based on
a meta-analyses over several fields, including history, sociology and anthropol-
ogy. We choose to incorporate this meta-study as it is based on a broad range
of disciplines, has not yet been mentioned in HCI context, and in our opinion
gives an insightful addition to both the theoretical framework and the list of
fundamental needs that can be addressed in our work.
Baumeister states that human need for meaning in life can be divided into
four basic needs: a need for purpose, a need for efficacy, a need for self-worth and
a need for (moral) values in life. If someone does not satisfy one or more of the
needs to satisfaction, it will lead to negative connotations such as depression or
sadness. In general, if people are faced with a reduction in fulfilment of one (or
several of) the need(s), they will cope by intensifying other activities in order
to fill the gap. Besides this they might be more open at that point to engage in
new activities that might satisfy one or more needs. Below, we discuss the four
needs in some more detail.
3.2 Purpose
Purpose concerns the interpretation of one’s activities and how these seem to
influence future states. Directly linked to this are goals and fulfilment. Fulfilment
Sheldonetal.
7 selfͲactualization(Ͳmeaning2) x x x
1 selfͲesteem x x10 x 10 x10
10 10
=42 relatedness x x x10 x x10
Epstein(1990) pleasure(vs.pain)
2
Asused,mentionedandinterpretedinSheldonetal.'slist
6 ng
ds 0) 1) mi 7) )
ee 01 2 2 00 12
n 01 yga
( 06) (20
al l (2 l.( od 0 l
gic ah ta Ͳb & um (2 & ah
l lo nz e le an b a n th
e c ho e k to wor m
et enz
s X
od y sse UX
a r
th who hID
y r t nag sen P4
s c r s s U
M Ps H fo W fo w i F la Ni V A Co fo De Ha for
Existence(physiologicalandsafety)orIdentity x10 x
Alderfer7 Relatedness(socialandexternalesteem)orBelonging x10 x10 x10 x x10
Growthneeds(selfͲactualizationandinternalesteem) x
FriedmanandKahn
(2003)5 x9 x
Purpose
Values x8,10 x8,10
Baumeister(1991)
SelfͲworth x10 x10
Efficacy x10 x10 x10 x10
R. van Delden and D. Reidsma
Otherhumanisticviews
10
selfͲconsciousness(reflectingonone'slife) x x10
3
Rokeachproposedtwosetsofuniversalvalues(althoughdifferinginstrength),(1)terminalvalues, thosetobefulfilledbeforedeath: acomfortablelife,anexcitinglife,asenseof
accomplishment,aworldatpeace,aworldofbeauty,equality,familysecurity,freedom,happiness,innerharmony,maturelove,nationalsecurity,pleasure,salvation,socialrecognition,
truefriendship,wisdom,and(2)instrumentalvalues, thosemodesofbehaviourtoattainendͲstateofexistence: ambitious,broadminded,capable,cheerful,clean,courageous,forgiving,
helpful,honest,imaginative,independent,intellectual,logical,loving,obedient,politeandselfͲcontrolled
deals with positive affect and achieving the goals. Goals are more related to
extrinsic motivations. An activity might be unpleasurable in itself but is done
to reach or to come closer to a goal. When the entertaining value or fulfilment
results from performing the activity itself it is an intrinsic motivation.
3.3 Self-worth
The need for Self-Worth concerns the fact that people need to feel that they have
something to contribute. This often results in people having the tendency to feel
better than others. This can be done by showing-off, engaging in competition or
scraping together small details to form a superior identity. It has some overlap
with value and especially having positive value. Nonetheless, it is a distinct need
and the two can even conflict.3
3.4 Efficacy
Efficacy, control and competence are very important feelings about being capable
and being strong in life. Remarkably, it is especially about the perception hereof.
An illustrative example is given from Rodin and Langer [22]. In this experiment
elderly were made aware of their responsibilities for themselves and were given
the task to care for a plant. It gave them a sense of efficacy and being needed.
Results showed that this group felt better and lived longer.
3.5 Values
Having values, a belief in what is good and bad, can be a source of a sense
of meaning in one’s life – independent of whether one actually manages to live
according to the values. Also, people want to justify their behaviour even if it is
not a key factor of a choice. Morals are generally stated in negative ways, like
“Thy shall not kill”. Positive morals or values such as sharing, helping others
and defending the group, exist as well in most societies. A specific type of value
is a “value base”. This kind of value, according to Baumeister, are values that
need no further justification. Typical examples are doing something “for God’s
sake” or “for the children’s sake”.
3
When the self-worth of a person is raised by things that actually conflict with their
own values, this discrepancy may negatively influence the self-consistency discussed
by Epstein.
4 Needs in Entertainment
In this section we briefly explain the identified needs of the several models that
can be targeted and fulfilled with entertainment. We address the needs from Fig-
ure 1 and 2. The mentioned examples are not exhaustive but are intended to show
that entertainment on itself can fulfil a wide array of human needs. Although the
several models are not easily fitted into a universal model of everything, turning
to these needs in the design and development process for new (entertainment)
technology still yields worthwhile and interestingly new applications.
Efficacy. Efficacy, control and competence focus on the need of feeling capa-
ble, and being strong in life. As stated in Baumeister, merely understanding
something helps to fulfil this need of efficacy. The whole edutainment movement
signifies the importance of this.
Safety/Security. The need for safety and security is about perceived threats
for future need fulfilment. Related to the need for efficacy mentioned earlier,
learning to deal with such threats might increase security as well as efficacy.
There are several serious games targeting security with the use of virtual reality
to train police officers and the military, and to help people with a variety of
phobias [28].
Self-esteem. To feel worthy, good and in some cases better compared to others,
are essential in the need for self-esteem. Again this need is related to efficacy and
competence but has its own specifics. In practice, in games this aspect is often
limited to the use of high-scores, bulletin boards, levels and solving puzzles.
Values. Having certain values to live by, moral standards and things to measure
up to, the ‘basics’ that can give guidance to one’s life, those are central in the
need for values. Traditional board games, take into account values in a indirect
way. Many allow and depend on a set of house rules, additional rules to make
the game fair, age appropriate or enhancing the aspect of personal causality
by reducing randomness. These rules make such values explicit. Values could
also be targeted through contemplation. For instance, the game series of GTA
and Molleindustria contain a set of explicit values (although odd and perhaps
morally crooked) that could encourage the gamer to contemplate on their own
set of values.
5 Conclusion
In this paper we looked at fundamental human needs in an HCI perspective,
looking at existing work in this direction, and adding work from humanistic
philosophy. This provided us with an overview of fundamental needs that might
be addressed through new technology as well as some general insights from the
various underlying theoretical frameworks. By linking entertainment examples
to the list of identified needs, based on a variety of theories, we showed that
entertainment can be more than mere fun. We conclude that entertainment
can be a welcome addition for intrinsic motivation, mental well-being, and for
increasing meaning in life. Turning to needs in the design and development
References
1. Hassenzahl, M., Diefenbach, S., Göritz, A.: Needs, affect, and interactive products
- facets of user experience. Interacting with Computers 22(5), 353–362 (2010)
2. Desmet, P., Hassenzahl, M.: Towards happiness: Possibility-driven design. In:
Zacarias, M., de Oliveira, J.V. (eds.) Human-Computer Interaction. SCI, vol. 396,
pp. 3–28. Springer, Heidelberg (2012)
3. Cockton, G.: Designing worth is worth designing. In: Proceedings of the 4th Nordic
Conference on Human-Computer Interaction: Changing Roles, NordiCHI 2006, pp.
165–174. ACM, New York (2006)
4. Muller, M.J., Wharton, C.: Workshop on HCI research and practice agenda based
on human needs and social responsibility. In: Proceedings of CHI 1997, CHI EA
1997. ACM, New York (1997)
5. Wyeth, P., Johnson, D., Sweetser, P.: Motivating Whole Body Gaming for Peo-
ple with Intellectual Disability. In: Proceedings of 4th International Workshop on
Whole Body Interaction at ACE 2011 (2011)
6. Baumeister, R.F., Leary, M.R.: The need to belong: Desire for interpersonal attach-
ments as a fundamental human motivation. Psychological Bulletin 117(3), 497–529
(1995)
7. Sheldon, K.M., Elliot, A.J., Kim, Y., Kasser, T.: What Is Satisfying About Sat-
isfying Events? Testing 10 Candidate Psychological Needs. Journal of Personality
and Social Psychology 80(2), 325–339 (2001)
8. Epstein, S.: Cognitive-experiential self-theory of personality. In: Milton, T., Lerner,
M.J. (eds.) Comprehensive Handbook of Pscycholgy, pp. 159–184. Wiley & sons,
Hoboken (2003)
9. Friedman, B.: Value-sensitive design. Interactions 3(6), 16–23 (1996)
10. Friedman, B., Kahn Jr., P.H.: Human values, ethics, and design. In: Jacko, J.A.,
Sears, A. (eds.) The Human-Computer Interaction Handbook, pp. 1177–1201. L.
Erlbaum Associates Inc., Hillsdale (2003)
11. Flanagan, M., Nissenbaum, H.: A game design methodology to incorporate social
activist themes. In: Proceedings of CHI 2007, pp. 181–190. ACM Press, New York
(2007)
12. Alderfer, C.P.: Existence, relatedness, and growth: human needs in organizational
settings. The Free Press, New York (1972)
13. Deci, E.L., Ryan, R.M.: A motivational approach to self: Integration in personality.
In: Nebraska Symposium on Motivation, vol. 38, pp. 237–288 (1991)
14. Ryan, R.M., Deci, E.L.: Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American Psychologist 55(1), 67–78
(2000)
15. Deci, E.L., Ryan, R.M.: The “what” and “why” of goal pursuits: Human needs and
the self-determination of behavior. Psychological Inquiry 11(4), 227–268 (2000)
16. Coleman, G.C.: Aging and satisfaction of psychological needs. Psychological In-
quiry 11(4), 291–295 (2000)
17. Pyszczynski, T., Greenberg, J., Solomon, S.: Toward a dialectical analysis of growth
and defensive motives. Psychological Inquiry 11(4), 301–305 (2000)
18. Buunk, B.P., Nauta, A.: Why intraindividual needs are not enough: Human moti-
vation is primarily social. Psychological Inquiry 11(4), 279–283 (2000)
19. Andersen, S.M., Chen, S., Carter, C.: Fundamental human needs: Making social
cognition relevant. Psychological Inquiry 11(4), 269–275 (2000)
20. Rokeach, M.: The nature of human values. The Free Press (1973)
21. Baumeister, R.F.: Meanings of Life, 1st edn. The Guilford Press (1992)
22. Rodin, J., Langer, E.J.: Long-term effects of a control-relevant intervention with
the institutionalized aged. Journal of Personality and Social Psychology 35(12),
897–902 (1977)
23. van Houten, D.: De standaardmens voorbij. Elsevier/De Tijdstroom (1999)
24. Maslow, A.: A Theory of Human Motivation. Psychological Review 50, 370–396
(1943)
25. Bekker, T., Sturm, J.: Stimulating physical and social activity through open-ended
play. In: Proceedings of the 8th International Conference on Interaction Design and
Children, IDC 2009, pp. 309–312. ACM, New York (2009)
26. Moreno, A., van Delden, R., Poppe, R., Reidsma, D.: Socially aware interac-
tive playgrounds: Sensing and inducing social behavior. IEEE Pervasive Comput-
ing 12(3), 40–47 (2013)
27. Wei, J., Nakatsu, R.: Leisure food: Derive social and cultural entertainment through
physical interaction with food. In: Herrlich, M., Malaka, R., Masuch, M. (eds.)
ICEC 2012. LNCS, vol. 7522, pp. 256–269. Springer, Heidelberg (2012)
28. Gamito, P., Oliveira, J., Morais, D., Rosa, P., Saraiva, T.: Serious games for serious
problems: From ludicus to therapeuticus. In: Kim, J.J. (ed.) Virtual Reality. InTech
(2010)
29. Schell, J.: The art of game design: A book of lenses. Morgan Kaufmann Publishers
(2008)
1 Introduction
Pervasiveness of digital cameras has led to large collections of digital photos that
users often browse on computer displays, by rearranging them to gather similar ones
based on specific features/ meta-data. While several techniques to do this efficiently
exist, most of them are somewhat systematic or goal-driven in terms of applying prin-
ciples for displaying photos. These methods are useful in systematically organizing
and finding photos but previous studies suggest that users often browse their photo
collections without a specific search goal (e.g. [1]) but a more general purpose such as
looking back at previous memories. Moreover, users often browse photos with actions
such as displaying/enlarging photos randomly or starting a slideshow for personal
gratification and pleasure. To support these behaviors, the presentation of photos
should be flexibly and dynamically adapted with visual effects based on user’s input.
Consequently, we propose a novel method to flexibly display a set of photos by
showing each of them in a dynamic and continuous motion like a living object. It
allows users to replace or add displaying principles interactively and flexibly. In
order to achieve such flexibility, we introduce an approach based on emergent compu-
tation. Geometric parameters (i.e. location, size, and photo angle) are considered to be
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 415–427, 2013.
© Springer International Publishing Switzerland 2013
functions of time. Photos are dynamically moved toward the directions determined by
local relationships with adjacent photos at each time instance. As a result, the global
layout of all photos varies automatically and converges gradually with time. These
dynamic behaviors provide users enjoyable interactions with less effort to recall good
story from the photos. This will enhance one of the most enjoyable parts of personal
photos, which is to share memories and reminisce with friends or relatives.
We illustrate example behaviors of photos and then do a user study to evaluate D-
FLIP against Windows Explorer, a photo managing program familiar to Windows
users. The evaluation involved two participants, a narrator and a listener to prepare
and share a story. We measured both participants EEG to quantitatively measure us-
ers’ metal effort/ task engagement. In addition, NASA-TLX forms were also collected
from the narrators and listeners after each task.
The contributions of this paper are: (1) a proposed method to dynamically and flex-
ibly display photos; and (2) an evaluation method using EEG which can be used to
evaluate interactive applications.
2 Related Work
2.1 Browsing Digital Photos and Photo Collages
Many efforts were proposed to arrange photos effectively. For example, a browser
that arranges multiple photos in folders by grouping them with different magnification
levels [2], or by categories with different hierarchy depths [3]. Other examples are
arranging photos calendar by using their shooting dates [4], displaying them on a
digital geographical map at their shoot locations using meta-data [5], grouping photos
with shoot locations and persons [6], and browsing large image datasets using Voro-
noi diagrams [7]. A technique for browsing large image collections was presented by
[8] using the rectangle‐packing algorithm, and by [9] using hierarchical tree struc-
tured organization of images with level of details. However, most of these methods
lack flexibility in displaying with mixtures of requirements based on user’s input.
Digital photo collages, which summarize meaningful events or memorabilia, are
widely used to display photos. This is efficient because users can view multiple pho-
tos at once. However, it requires two types of treatment: (1) a geometric treatment
concerning about arranging multiple photos in a pre-determined area but avoids over-
lapping and empty regions, and (2) a semantic treatment concerning about content of
the photos. Several authoring tools have been proposed to create photo collages easily
(i.e. AutoCollage [10], Picture Collage [11], and Digital Tapestry [12]).
Each photo has three parameters: its position, size, and rotational angle. They are
considered as functions of time and are controlled to arrange multiple photos simulta-
neously on a display. The photo movement is shown by gradually changing the values
of these parameters at every time instant. The algorithm is explained by Eq. (1):
dx / dt = f ( x) + η
(1)
Here, x is a set of the three parameters above and its variation dx/dt is derived by
f ( x) , the principle to achieve the photo arrangement, and noise term η . Larger ampli-
tude noise increases the fluctuation and is useful for escaping local optimum. Fur-
thermore, Eq (1) can be re-written in another form with the weight coefficients:
(2)
= { wi fi ( x )+ηi }
dx
dt
, a variety of principles, is used to achieve the photos arrangement or
i
In here, f ( x)
layout. Let P represents the data of a photo, I represents the information of certain
input or output devices, P is all the photos in the environment, Position(P) is the
photo position, Size(P) is its size, and Rotation(P) is its rotational angle. Assuming
that the number of principles related to position, size, and rotational angle are l, m,
and n, respectively. Eq. (3) is obtained by modifying Eq. (2). It controls the parame-
ters of photo P and is calculated from all photos. Here, f Pi (x ) , f Si (x ) and f Ri (x )
are functions that represent the changes of position, size, and rotation, respectively:
d
dt
n
{ dt
m
{ d
dt
} l
{
Position( P) = i f Pi ( I , P)+ ηi }; Scale( P) = i f Si ( I , P) + ηi ; Rotation( P) = i f Ri ( I , P) + ηi
d
} (3)
Al ⋅ {L − L( Pi )} if L( Pi ) < L
N A ⋅ {B − B( P )}
if B( Pi ) < B
f molld (I , P) = b i
(5)
i Ar ⋅ {R − R( Pi )} if R(Pi ) > R
At ⋅ {T − T (Pi )} if T (Pi ) > T
N
f rotation ( I , P ) = å Ang ( P , Pi )
(8)
i
Eq. (10) shows how a focused photo attracts other ones with similar attributes.
Here, Similarity(Pi, Pj) is the similarity between photos Pi and Pj, and if this value is
larger than a threshold, Pj moves toward Pi, and away otherwise. The similarities are
assumed to be calculated by feature values obtained by image processing or from tags
of photos. Other related principles of semantic mapping can be obtained similarly.
N
Aa 2 {Similarity ( P, Pi ) − Threshold }
i
⋅ {Position ( Pi ) − Position ( P)} (10)
if Similarity ( P, Pi ) ≥ Threshold
f attraction ( I , P) = N
Aa 2 {Similarity ( P, Pi ) − Threshold }
i
/{Position( P ) − Position( P)}
i
Viewer’s Interactions: Even after the system reaches the balanced condition, the
photograph behaviors can be observed when parameters of the display environment
vary (e.g. when new photos are added or the size of the displaying window
is changed). Also if a cursor (operated by a mouse, for example) is used, the photo
overlaid by the cursor becomes larger using Eq. (8) with certain weight coefficients.
Users can observe the displayed photos and interact simultaneously with separate
input devices (i.e. touch or gaze input devices). In addition, the display resolution, the
number of displays, their positions, orientations, and sizes are variables of the output.
Such information about input and output is treated as I in previous equations.
a)) b) c)
Fig. 3. Geometric packing: a) original layout; b) final layout without rotation, c) with rotation
Photograph Behaviors with Geometric Packing: Fig. 3 (a, b) shows how photos
avoid overlaps when Eqs (4)(5)(6)(7) are applied. Fig. 3a shows the initial state where
75 photos are located randomly and overlapped. However, they gradually move to
avoid overlaps and occupy the empty regions (using Eqs. (4)(5)), as shown from (a) to
(b). At the same time, the photos’ sizes are varied (using Eqs. (6)(7)), then soon be-
come almost equal (Fig. 3b). Fig. 3 (a, c) is an example of photos avoiding overlaps
by rotating in addition to the equations used in Fig. 3 (a, b). In Fig. 3a is the original
layout, and Fig. 3c is the layout with collision-free arrangement with rotation. This is
useful when photos are shown on tabletop surface displays shared by several users.
Fig. 4 shows an example of photograph behaviors when one of the environmental
parameters, the window size, is changed. Once the window is enlarged (left figure),
the contained photos steadily move to the empty space (middle figure), according to
Eq. (5). This is gradually convergent with time so that the sizes of all photos become
almost equal but avoid overlapping (right figure).
Photograph Behaviors with Semantic Mapping: Fig. 5a shows an example where one
photo is focused by overlaying a cursor (at the bottom center of the photo). Fig. 5b shows
photos arranged by color and user interests using the principles of geometric packing (i.e.,
Eqs. (4)(5)(6)(7)). Here, two cursors (magenta and green) point at two photos (bottom-left
night scene and upper-right daylight scene). Soon photos with similar colors are moved
toward the focused ones. The final layout is achieved by Eq. (9) with semantic mapping
principles. Similarly, other feature values calculated by image processing can be used to a
group of photos by applying this principle.
(a) A photo is focused by overlaying a pointer (b) Photos are arranged by color and interest
Fig. 5. Examples of geometric packing (a) and semantic mapping (b)
Fig. 1a shows an example of photos arranged using Geotags. In this example, pho-
tos arranged without overlapping (using the Eqs. (4)(5)(6)(7)) are attracted by their
geographical identification metadata (latitude and longitude coordinates) based on Eq.
(9) and moved to their corresponding positions on a world map.
Fig. 1b shows an example of finding someone's photos. Given a photo set of a so-
cial relationship, when a human face in a photo is selected, the size of that photo be-
comes larger by Eq. (8). Also, all photos containing the selected person are attracted
and gathered around the focused photo dynamically by Eq. (9). Here, a face recogni-
tion function is assumed to be working and tags for the faces are adequately given in
advance. Similarly, Fig. 1c displays examples of grouping photos with closed curves
drawn by a mouse using meta-data given to each of the photos in advance. In this
figure, photos having meta-data of Mr. A and Mr. B are gathered in the red closed
curve in the left and the green closed curve in the right, respectively. In the overlap-
ping area of these closed curves there are photos belonging to both Mr. A and Mr. B.
4.2 Discussions
Our proposed method has each photo arrangement generated at every time instant
which may not always optimally satisfy all principles. However, these arrangements
provide viewers a dynamic photo-viewing environment where they can observe the
smooth transitions of photos layout and the behaviors caused by their interactions.
Theoretically, the displayed photos by our algorithm are slightly vibrated because
the function inherently includes a noise term in Eq. (1) that causes the system to be
constantly in motion. In the above examples, a fixed small amplitude was used for the
noise. Larger amplitude noise increases the fluctuation of the environmental parame-
ters, and such fluctuation is often useful for escaping local optimum. However, the
vibrations can be removed using a filter before rendering if not suitable. Moreover,
the amplitude can be varied during photo arrangement to obtain a better performance.
A performance evaluation confirms that the proposed algorithm provides about 60
FPS to show 700 photos with 512x320 pixels on a Windows 7 64-bit PC with Intel
Core i7 CPU (3.20 GHZ), 12.0 GB memory, NVIDIA GeForce GTX 285. The system
performance drops quickly when the number of photos exceeds 700. However, this is
a reasonable number for photo browsing because of: (1) screen resolution (all photos
are displayed on a screen and should be large enough to view individually); and (2)
common number of photos belonging to an event. However, this number can be in-
creased with different implementation (parallel run, multi-core CPU usage, etc.).
one narrator and one listener who sat beside each other in front of a 27-inch monitor
(2560x1440 resolutions) displaying the narrator’s personal photos. We measured task
engagement from both narrator and listener to investigate the effect of the interactive
application on the person who actually interacted with the program (the narrator) and
on the person who only observed the interaction and listened to the story (the listen-
er). Participants wore Emotiv EEG headset during the study.
There was a storyboard in D-FLIP to support the storytelling mode. The area had
ten equal boxes which held ten selected photos (Fig. 6, left). In case of Explorer, we
used two Explorer windows with one window above another. The bottom window
was used to store 200 photos and the top window was used to store the selected 10
photos (Fig. 6, middle). In Windows 7, filenames, all bars and additional panes (e.g.
Library, Details, and Navigation) were hidden to make Explorer comparable to D-
FLIP in term of visualization and functions. Features of D-FLIP in this experiment
included dynamic arrangement when overlaying a pointer, attraction for photos with
similar colors. These features were chosen as they are comparable with Explorer.
5.2 Method
14 participants (9 males) between the ages of 19 and 32 volunteered for the study and
were arranged into 7 pairs. The narrators brought to the experiment their 400 photos
divided into 2 sets of 200 each. Photos were resized to 640x480, rotated if necessary
by the experimenter before beginning the study.
Fig. 6. Storytelling mode with D-FLIP (left) and Windows Explorer (middle); An example of a
narrator’s task engagement changes in time in one session (right)
First, the narrator had adequate time to practice with D-FLIP and Explorer using a
sample photoset (Fig. 6, left & middle). After this both users wore the Emotiv head-
sets. Each experiment session had two blocks each with either D-FLIP or Explorer. In
each block, the narrator prepared a story in 5 minutes by selecting 10 photos from her
first photoset. Then she told the story to the listener within 5 minutes. Both partici-
pants had a 2-minute break before the same procedure was repeated with the next
program. NASA-TLX forms were given to both participants after each block.
In the storytelling step, we only recorded EEG signals from the listener. This was
because the narrator needed to speak freely with facial, hand, and body movements
which would contaminate the EEG signals. Narrators were instructed to use the soft-
ware as normally as possible. This includes free using any desired features (including
sorting, grouping, etc.). They were also encouraged to use the non-selected photos in
the collection to enrich their story. In addition, all users were asked to sit comfortably.
Tasks S
Story Preparation Story Telling
Programs plorer
Exp D-FLIP Explorer D-FLIP
Narrator 0.603 0.554
Listener 0.480 0.436 0.512 0.440
For the NASA-TLX, only mental demand (for narrator) presented with a signifi-
cant difference (p < 0.05) between two tasks: Explorer (mean: 9.14) and D-FLIP
(mean: 5.00). For the listener, there was no significant difference (p > 0.05) for any of
the NASA-TLX questions. These results showed that the narrators had higher en-
gagement when using the Explorer compared to D-FLIP. This implies that narrators
need to put in more effort with Explorer than D-FLIP to complete the same task in the
same amount of time. Consequently, besides the benefit of having more interest, D-
FLIP makes the task easier to perform. The results of NASA-TLX (for both partici-
pant types) are consistent with the task engagement results.
Users had adequate practice time with D-FLIP. However, the familiarity with Ex-
plorer may result in a more skilful performance. Hence, the actual difference between
measured task engagements of two programs might be larger if a less famous program
(e.g. PhotoMesa) was used instead of Explorer. Our results show that even if there are
effects of familiarity, D-FLIP still requires less task engagement and mental demands.
7 Discussion
An interesting finding from our user evaluation is that D-FLIP requires less mental
effort and task engagement compared to Explorer in narrators but not listeners.
A probable cause is that only the narrators physically actually interact with the
programs. Hence, with less task engagement or workload in performing the task,
interacting with D-FLIP is easier and more pleasurable.
Several factors of D-FLIP may contribute to this result viz.: high visibility with va-
riety of layout, dynamic and smooth motions of photos, gathering photos based on
similar attributes. Consequently, users can focus on interactions with other users and
the story contents with less effort in performing the task. Additionally, D-FLIP keeps
motivation and interest in users due to the dynamic, flexible, and interactive motions
of photos produced by the proposed principles, thus making D-FLIP a particularly
powerful system for visualizing large collections of images.
Our next step is to improve the algorithm categorizing the parameters and optimiz-
ing it for different types of interactive applications. Besides existing content types (i.e.
text, broadcasts, movies, etc.), our algorithm can adapt to work with new and emerg-
ing content types such as dreams visualization [27] where values of parameters from
fMRI patterns for an image. This can also help to build an ecosystem of photos which
includes various promising features such as pigeonholing, printing, automatic acquisi-
tion of meta-data, evolving into photo sharing sites, and coordinating with social
network services. We can also explore other interaction devices such as multi-touch
digital tables, voice recognition, brain-machine interface, and other sensing devices to
further enhance the fluidity of interaction in different application contexts.
Although the focus of this paper is not on a novel evaluation methodology, we be-
lieve that our way of measuring task engagement using EEG offers greater insights
into workings of an application. Our measured task engagement is consistent with the
NASA-TLX results; providing a source of external validity to our measurement me-
chanism. It can also be improved to capture emotions (e.g. relaxation, meditation) and
other users’ inner states (e.g. error awareness).
8 Conclusion
References
1. Kirk, D., Sellen, A., Rother, C., Wood, K.: Understanding photowork. In: CHI (2006)
2. Bederson, B.B.: PhotoMesa: a zoomable image browser using quantum treemaps and bub-
blemaps. In: UIST (2001)
3. Dachselt, R., Frisch, M., Weiland, M.: FacetZoom: a continuous multi-scale widget for
navigating hierarchical metadata. In: CHI (2008)
4. Graham, A., Garcia-Molina, H., Paepcke, A., Winograd, T.: Time as essence for photo
browsing through personal digital libraries. In: JCDL (2002)
5. Ahern, S., Naaman, M., Nair, R., Yang, J.H.-I.: World explorer: visualizing aggregate data
from unstructured text in geo-referenced collections. In: JCDL (2007)
6. Gomi, A., Itoh, T.: A personal photograph browser for life log analysis based on location,
time, and person. In: SAC (2011)
7. Brivio, P., Tarini, M., Cignoni, P.: Browsing Large Image Datasets through Voronoi Dia-
grams. IEEE Trans. Vis. Comput. Graph. 16(6) (2010)
8. Itoh, T., Yamaguchi, Y., Ikehata, Y., Kajinaga, Y.: Hierarchical data visualization using a
fast rectangle-packing algorithm. IEEE TVCG 10(3) (2004)
9. Gomi, A., Miyazaki, R., Itoh, T., Jia, L.: CAT: A Hierarchical Image Browser Using a
Rectangle Packing Technique. In: Book CAT: A Hierarchical Image Browser Using a Rec-
tangle Packing Technique (2008)
10. Rother, C., Bordeaux, L., Hamadi, Y., Blake, A.: AutoCollage. SIGGRAPH (2006)
11. Wang, J., Quan, L., Sun, J., Tang, X., Shum, H.-Y.: Picture Collage. In: CVPR (2006)
12. Rother, C., Kumar, S., Kolmogorov, V., Blake, A.: Digital Tapestry. In: CVPR (2005)
13. Kim, S., Yoon, M., Whang, S.M., Tversky, B., Morrison, J.B.: The effect of animation on
comprehension and interest. JCAL 23(3) (2007)
14. Rieber, L.P.: Animation, incidental learning, and continuing motivation. Journal of Educa-
tional Psychology 83(3) (1991)
15. Kintsch, W.: Learning from text, levels of comprehension, or: Why anyone would read a
story anyway. Poetics 9 (1980)
16. Vi, C., Subramanian, S.: Detecting error-related negativity for interaction design. In: CHI
(2012)
17. Hagemann, D., Hewig, J., Walter, C., Schankin, A., Danner, D., Naumann, E.: Positive
evidence for Eysenck’s arousal hypothesis: A combined EEG and MRI study with multiple
measurement occasions. J. of Personality and Individual Differences (2009)
18. Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A., Davis, G., Zivkovic, V.T.,
Olmstead, R.E., Tremoulet, P.D., Craven, P.L.: EEG Correlates of Task Engagement and
Mental Workload in Vigilance, Learning, and Memory Tasks. ACEM 78(5) (2007)
19. Matthews, G., Campbell, S.E., Falconer, S., Joyner, L.A., Huggins, J., Gilliland, K., Grier,
R., Warm, J.S.: ‘Fundamental Dimensions of Subjective State in Performance Settings:
Task Engagement, Distress, and Worry’. Emotion 2(4) (2002)
20. Berka, C., Levendowski, D.J., Olmstead, R.E., Popovic, M.V., Cvetinovic, M., Petrovic,
M.M., Davis, G., Lumicao, M.N., Westbrook, P.: Real-time Analysis of EEG Indices of
Alertness, Cognition, and Memory with a Wireless EEG Headset. IJHCI 17(2) (2004)
21. Pope, A.T., Bogart, E.H., Bartolome, D.S.: Biocybernetic system evaluates indices of
operator engagement in automated task. Biological Psychology 40 (1995)
22. Rodden, K., Wood, K.R.: How do people manage their digital photographs? In: CHI 2003
(2003)
23. Anderson, E.W., Potter, K.C., Matzen, L.E., Shepherd, J.F., Preston, G.A., Silva, C.T.: A
user study of visualization effectiveness using EEG and cognitive load. In: EuroVis (2011)
24. Ames, M.G., Manguy, L.: PhotoArcs: ludic tools for sharing photographs. In: MM (2006)
25. Ekandem, J.I., Davis, T.A., Alvarez, I., James, M.T., Gilbert, J.E.: Evaluating the ergo-
nomics of BCI devices for research and experimentation. Ergonomics 55(5) (2012)
26. Lee, J.C., Tan, D.S.: Using a low-cost electroencephalograph for task classification in HCI
research. In: UIST (2006)
27. Horikawa, T., Tamaki, M., Miyawaki, Y., Kamitani, Y.: Neural Decoding of Visual Im-
agery During Sleep. Science (2013)
1 Introduction
People goes through many daily experiences which invokes numerous emotions –
from happy, sad, angry etc. One way to express themselves is through logging their
daily events; either through a blog, diary or other archives. As technology advances,
there exist many networking services that help people connect with each other
through these loggings such as Twitter and Facebook.
Sumi et al. mentioned that one of the meanings to recording experience is to share
the records of memories [7]. Yamashita et al. mentioned that the popularization of the
web log or web diary is to allow one to learn more and to understand oneself -- to
express their satisfaction or emotion. They analyzed that feedbacks from the readers
can result in satisfaction psychologically [10]. In recent times, contents through SNS
such as daily recordings or pictures have been developing at great speed as it holds
great values and high demands from the society. Especially interesting contents or
experiences such as travelling, visits to the museum, or even outing at an amusement
park. Through these activities, we can observe new sceneries and meet new people
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 428–439, 2013.
© Springer International Publishing Switzerland 2013
from different places. From m these experiences, we build up new emotions, reactiions
and expressions. When we lookl back at those times, we can recall the feelings we ffelt.
hese experiences as a “Journey”. Normally, when one loook
In this project, we call th
back at their albums, mosst pictures found have similar sceneries, poses or faacial
expressions. Many of thesee pictures do not reflect the interaction that took placee at
that time. Therefore, we propose PukaPuCam, a camera to capture picturess of
different moments, from diifferent angle and direction, to give the user a better reecap
of those moments when theey view the pictures. This camera would capture the useer’s
pictures from a third-person n view without the user being conscious about them. As the
camera is placed on a balloo on, it enhances an exciting journey experience (Fig.1).
Fig. 1. PukaPuCam is a servicce to capture users’ photo continuously from a third-person viiew
2 Related Work
There are several attempts to record user’s activity from a third-person view. Flyying
Eyes is a system that utillizes AR.Drone to follow and record user’s activity by
recognizing their cloth’s coolor [2]. However, as the system requires lots of poweer to
keep flying, it faces challen
nges to follow the user for a long period. Hasegawa ett al.
developed a system whereb by a camera is mounted at the end of a pole that is attached
to skydiver’s waist, to capture the user and project it real-time in a HMD [1]. T This
way skydiver can see and adjust
a his body movement accordingly. Pfeil et al. creaated
ball camera, a throw-able sphere device embedded with 36 small cameras to shhoot
from a panorama view wh hen the user throws the ball [5]. Sugimoto et al. desiggned
Time Follower’s Vision, a robot control interface based on third-person view whhich
combines past images taken n by a camera mounted on a robot with CG [6]. Dog-Leeash
Interface is a robot that follows
f its user to capture the user’s image. Similarr to
walking a dog, user will pulll its string to control its route [11].
NHK Balloon Camera iss an anchored floating body system with a camera mounnted
on a balloon ship to take pictures
p from a birds-eye view without the need of a crrane
[4]. Our approach differs frrom these researches, as our system is not proposed to act
together with users in their everyday environment. Floating Eye is known as a meedia
art, equipped with a spherical dome screen to project images in the sky, taken from a
camera attached to an airship in real-time [3]. Although these systems are similar to
our study, our aim is to give user a new travel logging experience. Therefore, our
system differs in terms of user experience awareness.
In terms of life logging, Vicon Revue is a wearable camera system equipped with
different sensors (temperature, luminance, accelerator etc.), which automatically
captures the user’s pictures [9]. This system is suspended on the neck and captures
from a first-person view at every 30 seconds intervals or from the influences of the
surroundings it detects.
While most experience recording system’s approach is to take videos and images
from a first-person view, Sumi et al. proposed a combination of wearable sensors and
environmental sensors to provide multiple camera view system: first-person view
(mounted on the user), second-person view (mounted on the person the user is
interacting with) and third-person view (placed in the surrounding environment) [8].
They observed that pictures from first-person view lack many things for users to learn
more about their behavior and expression. Second and third-person view are equally
important to remind users of their feelings and actions at a particular situation.
However, most of the proposed system faces challenges to be used frequently
when user goes for a walk. Or it may be hard to blend in with the surroundings.
Among the researches in life logging, many can be mounted on the neck or blend into
daily wearable. However, most of these systems captures from a first-person view and
thus, one’s own figure is hardly reflected in the shots. Our thought of an ideal life
logging system is where users are projected in the pictures while is unconscious about
being photographed. This way, the records reveal the user’s natural expression. To
achieve this, we propose the design of a camera system that captures in a third-person
view and can blend easily into the surroundings.
3 Concept
This project aims to create a “Camera that continuously records your journey”.
PukaPuCam is a design service to create a personal photographer that will capture the
users’ journey experience and fun interactions between the users and their best
friends. Here, we emphasize on three main design guidelines.
(1) Users can take pictures that differs from the common first-person view
camera – pictures from a bird's-eye view
(2) Users can easily look back at their journey and explain to others.
(3) Users can enjoy a fun journey experience.
From these aim, we aim to accomplish (1) with a system utilizing a camera attached
to balloons, (2) with a viewer where users can glance through back at their recorded
pictures to recall their surrounding environment as well as the route they have taken
and lastly (3), by using balloon to increase the fun experience.
3.2 Balloon
・
place.
Majority has held a balloon
b during their childhood period. Therefore, its ussage
allows easy prediction
n of behavior.
PukaPuCam will fit into o places where balloons are commonly used such as theeme
park or tourist attractions. One
O advantage is that a balloon has high resistance agaainst
air resistance as it is filleed with helium gases. This is beneficial to record the
surrounding environment and the users’ behavior at that moment, such as thheir
interest or interactions. In addition, it can also capture the surrounding conditioons,
such as the wind’s direction n or weather. Besides that, people have different curiossity,
actions or expression which h many would not take notice of normally. These unussual
moments would too be a great part of the users’ collection.
An example of an interaction with the PukaPuCam system is shown in Fig. 2 wherreby
when the user stops, the balloon will be on top of the user (Fig.2 left), or when the uuser
moves, the balloon incline backwards
b and record sceneries from the user’s back (FFig.2
right). The angle will chang ge according to the user’s walking speed. In order for the
balloon to automatically recoord form a third-person view at any place, there are 2 dessign
points needed to be taken intto account: the mooring method for the camera to capturee the
user’s field of view continuou
usly and the length of the string .
4 Implementation
n of PukaPuCam
Balloon is highly influenced by air resistance. Due to this, the balloon may waver and
cause blurriness effects to the
t shootings. If the recordings are in video form, the uuser
may feel sick when they glance back at the records. In addition, many may sskip
forward bits and parts of thhe video. Therefore, we decided to test with short interrval
shootings while placing imp portance on the journey’s time. To find an optimum tim ming
c feel the journey’s flow when they look at the recorrds,
for each shot so that user can
we decided to experiment with 5 seconds intervals. For example, if users look aat 4
continuous pictures taken at a similar scene, it reflects that they were there forr 20
seconds or so.
We want to implement a camera that captures at a wide angle and is less affeccted
by vibration. Our selection n for the experimentation is Digital HD Video Cam mera
Recorder Action Cam "HDR-AS15" -- weighing about 109g in addition of the Eyee-fi.
The balloons are two circu ular aluminum evaporated balloons, each filled with 80l of
Helium gas. The buoyantt force of the balloons is approximately 172g at 10℃
temperature and 1 atm. Thee camera is fixed at 2 points to make its lens parallel to the
string (Fig.4). This allows it to capture the users regardless of the balloon’s positionn.
volunteers in their 20’s. Du uring this experiment, the angle was fixed at 120o. Here, the
users will adjust the string’s length accordingly and from these records, we w will
decide the maximum length h by calculation.
Then, from the results, we w developed a hand-held device whereby user can adj djust
the string’s length up till 21
10 [cm] (between the camera and the spool). If the threaad is
extended to its limit, and thhe height of the hand from the ground is 1[m], it is possiible
to shoot approximately 532 2 [cm] (310 [cm] × √3 = 532 [cm]) radius around the ussers,
assuming that the balloon and the camera are right above the users.
Fig. 5. PukaPuCam Viewer: Scroll View (above), Map View (left), Enlarge View (right))
5 User Experiencee
We conducted a user stud dy, to observe how PukaPuCam can enhance the useer’s
journey experience. In thiss study, participant will use PukaPuCam and the viewwer
during their journey. We will
w observe their interaction and receive their feedbackk of
the system.
5.1 Participants
H: Female in her 20s. She loves
l travelling around by train.
B: Female in her 20s. She hardly
h takes a walk but she loves places with lots of natuure.
M: Male in his 20s, H’s best friend. He learns about the journey from H’s user study .
On 12th December 2012 (sunny), H visited the Yume Migasaki Zoo. During her visit,
she used both PukaPuCam and her own digital camera to take pictures (Fig. 6). After
walking for 10 minutes, she met up with B and they walk together. We observed the
whole situation. Four days later, H explained her journey using pictures from both her
own digital camera and the viewer app “PukaPuCam Viewer”. We observed the
situation again and lastly we interviewed her for her feedback.
6 Discussion
(1) Users can take pictures which differs from the common first-person view
camera – pictures from a bird's-eye view.
Air resistance and wind
As balloon receives air resistance, it changes direction depending on the wind and
the user’s walking speed. However, thanks to this, when the balloon moves to the
front, H was able to get a shot of a pose with her friend as shown in (Fig. 7). H was
really fond of this picture when she looked through the viewer. H also mentioned that
the balloon was really easy to use. Even situations where the wind was strong, she can
intuitively aim to record different scenes.
Due to the wind, some of the pictures taken were blurred. However, this blurriness
gave a touch of difference to the pictures such as in Fig. 8 where both users
introduced the picture to M as “interesting pictures”. They mentioned happily, “It
doesn’t feel look like a picture taken when it is windy right?”
Natural Expression
As PukaPuCam captures from a third-person view, it can capture pictures of oneself
naturally – to capture the expressions of the users, giving great pictures where the
users can learn more about themselves. For example, “the type of expressions the user
usually have” or “how the user smiles in daily conversations” etc. From this, users
can learn the type of atmosphere that can lift up their own happiness.
When H met up with B, many of the pictures captured reveals their smile and
happy faces. H mentioned, “I am really glad to be able to capture my natural smile,
as I am quite camera shy and I hardly smile in most pictures.” She also added, “If the
picture was taken from my own camera, most of the time I will not be part of the
picture. However, when taken from a third-person view, both our pictures and the
surroundings are taken as well. We can really feel the atmosphere of the journey.”
Besides, some of the other interesting pictures which were captured unexpectedly
by the camera are such as, user looking up at the balloon when the wind blows
slightly, or when there was a strong wind, user was pulling and looking at the
direction of the balloon. Therefore, PukaPuCam allows capturing these interesting
expressions caused by the balloon.
Interest
PukaPuCam can capture scenes which the users themselves hardly notice, such as
what type of objects which the user are interested in or when user sees object which
do not catch their interest, they would just walk by them quickly. Some examples are
B’s figure who was engross in taking pictures or H’s figures peeping at the animals.
Cute
This is attracts a lot of attention from females and children as by using balloons,
PukaPuCam can capture pictures which can increase the cuteness values. Balloon can
capture pictures such as H’s face looking up (Fig. 9) or H posing with her friend B
(Fig.7). The two participants really like these two pictures. Other than that, as the
camera follows and captures a lot of picture from the back of the participant, it was
able to capture the “cute unprepared moments”. Fig. 10 shows the picture taken from
the front by the user’s camera where the user posed for it together with a picture of
back captured by PukaPuCam.
Fig. 10. Photo by the user’s camera (Left), Photo by PukaPuCam at same place (Right)
(2) Users can easily look back at their journey and explain to others.
Video sequence
The pictures captured by the camera shows the subtleties of emotion, movement
and behavior of the users with the surrounding. While browsing through the viewer,
M said, “As PukaPuCam captures the pictures at short intervals, we can feel the flow
of the journey recorded.” He added that, “I can really understand their interaction
and relationship such as how did they feel when they were walking or what interests
them the most.” H commented, “It was really fun to look at pictures taken at different
moments as there are many interesting things I had missed or had never known.”
Participants reviewed that the Viewer was really smooth and interesting to use.
They also commented that this system really guides them to explain their journey to
people. Therefore, from our observation, having interval shot recorded from a third-
person view supports the story telling of the users’ behavior in their journey.
Indexing the Location
The MapView on the PukaPuCam Viewer record the location and each picture taken
will be added to the photographing point using GPS information. By having an index
of the location, it can easily help recollect one’s memories. We questioned H on her
opinion on whether is it a useful guide for explanation. H replied, “It was a grreat
assistance. The viewer show ws the whole map of Japan and when you click on the zooo’s
location, it will zoom in and
d show the route that you have taken in your journey.”
Feedback due to Limited Field F Vision
However, there were feedbacks that participants had challenges in explanation duee to
limited field of vision captuured. At places where people tend to stop often, the ballooon
captures pictures from righ ht above the head. These pictures do not reflect the reaason
the user stopped as the sccene is outside the camera’s field of view. Althoughh, it
advantages over first-perso on view pictures, as the recordings were interesting and
users are always reflected in i the pictures. Therefore, a combination of both will bbe a
great complementary to thee journey’s story telling. For example, (Fig. 11 left) shoows
the user pointing her fingerr, taken at a third-person view and (Fig. 11 right) which is
taken a while later by the usser. From the combination of both pictures, we can tell tthat
the user is really interested in flamingo and was telling her friend about it.
Fig. 12.
1 Reactions from children and animals
Heart-shape balloon
We also observed that wheen the participants noticed that the shadow of the ballooon
was a heart shape, they jum
mped with excitement and took a picture together with thheir
shadow (Fig.13). This lifted
d up the atmosphere of the journey.
Fig. 13.
1 Balloon makes a heart-shaped shadow
Usability
H gave some feedbacks reegarding the length of the string whereby she mentionned,
“As there is wind and lotss of trees at the surroundings, it was hard to handle the
string as it may get entanglled between the trees. However, it was easy to get used tto it
after a while.” We question ned H and M that from their short experience, would tthey
be interested to use PukaPu uCam service in the future or would they recommend the
service to others. Both excitedly replied “Yes, we would love to.” “Especially, iff we
go to theme parks such as Disney
D Land, it would be really cool to take recordings of
when we meet and shake handsh with the characters. We would like many peoplee to
use it as well.”
Summary
To summarize the above, participants
p were really happy with the pictures and w when
others look at the pictures, they get to learn about their journey, their expression, the
surroundings etc. without asking the participants beforehand. From our user stuudy,
walking with a balloon can n produces a fun experience for users as they can have nnew
interaction with children annd animals. Therefore, by using PukaPuCam, it is possiible
to record a new journey ex xperience that differs from recordings from a first-perrson
view.
7 Future Work
First, many pictures do nott reflect the surrounding environment as it was taken frrom
straight up or when the useers walked by too fast. Therefore, the combination of bboth
first and third-person view images will be a great tool for journey logging. A secoond
possible future works is to share the pictures with friends. This way, it may lead tto a
new perspective or findings. Finally, one of the participant commented that it woould
be a great motivation for travelling if users can save and store the picture on a
national map, rather than just the route they have taken.
8 Conclusion
In this project, we propose a balloon camera system PukaPuCam to give users a new
journey experience that differs from the ordinary way of travelling. As a result of our
user study from both interview and observation, PukaPuCam has exceeded our
expectation. Participants had a fun journey and PukaPuCam was very useful for
participants to glance back through their journey. Furthermore, this system can record
the atmosphere and can bring out the “cuteness” of the users.
Acknowledgment. This work was supported by the Strategic Information and
Communication R&D Promotion Programme (SCOPE) of the Ministry of Internal
Affairs and Communications, Japan.
References
1. Augmented Ski, http://haselab.net/~hase/ski/ski.php
2. Higuchi, K., Ishiguro, Y., Rekimoto, J.: Flying eyes: free-space content creation using
autonomous aerial vehicles. In: Proc. CHI EA 2011, pp. 561–570. ACM (2011)
3. Iwata, H.: Floating Eye, ARS ELECTRONICA (2001),
http://90.146.8.18/en/archives/prix_archive/
prix_projekt.asp?iProjectID=11055
4. NHK Balloon Camera Shoots Stable Aerial Footage,
http://www.diginfo.tv/v/12-0020-r-en.php
5. Pfeil, J., Hildebrand, K., Gremzow, C., Bickel, B., Alexa, M.: Throwable panoramic ball
camera. In: Proc. SIGGRAPH Asia 2011 Emerging Technologies, Article 4, 1 page. ACM
(2011)
6. Sugimoto, M., Kagotani, G., Nii, H., Shiroma, N., Inami, M., Matsuno, F.: Time
follower’s vision. In: Elliott-Famularo, H. (ed.) ACM SIGGRAPH 2004 Emerging
Technologies, p. 29. ACM, New York (2004)
7. Sumi, Y.: Tatsuyuki Kawamura. Towards Experience Medium: Technical Survey for
Record and Utilization of Experiences. In: The 20th Annual Conference of the Japanese
Society for Artificial Intelligence (2006) (in Japanese)
8. Sumi, Y., Ito, S., Matsuguchi, T., Fels, S., Mase, K.: Collaborative capturing and
interpretation of interactions. In: Pervasive 2004 Workshop on Memory and Sharing of
Experiences, pp. 1–7 (April 2004)
9. Vicon Revue, http://viconrevue.com
10. Yamashita, K., Kawaura, Y., Kakami, Y., Asako Miura, A.: The psychology of the
weblog. In: NTT Shuppan (2005)
11. Young, J.E., Kamiyama, Y., Reichenbach, J., Igarashi, T., Sharlin, E.: How to Walk a
Robot: A Dog-Leash Human-Robot Interface. In: Proc. Ro-Man 2011, pp. 376–382 (2011)
Abstract. Immigration imposes a range of challenges with the risk of social ex-
clusion. As part of a comprehensive suite of services for immigrants, the
MASELTOV game seeks to provide both practical tools and innovative learn-
ing services via mobile devices, providing a readily usable resource for recent
immigrants. We introduce advanced results, such as the game-based learning
aspect in the frame of recommender services, and present the rationale behind
its interaction design. Benefits and implications of mobile platforms and emer-
gent data capture techniques for game-based learning are discussed, as are me-
thods for putting engaging gameplay at the forefront of the experience whilst
relying on rich data capture and analysis to provide effective learning solutions.
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 440–455, 2013.
© Springer International Publishing Switzerland 2013
Fig. 1. Using the MASELTOV app (MAPP) for mixed reality gaming in the streets
1
http://www.maseltov.eu
The European MASELTOV project recognizes the major risks for social exclusion
of immigrants from the local information society and identifies the considerable
potential of mobile services for promoting integration and cultural diversity:
When adopting user-centered design (UCD), the involvement of target users in the
design and development process of a new service is crucial for its success (Vreden-
burg et al. 2001). Based on frequent user feedback addressing ideas, scenarios and
prototypes the service concept is iterated and refined in order to match and satisfy
users´ needs. However, former research has shown that cultural differences (e.g. per-
ceptual and cognitive processing) matter in interface design and affect the quality of
user generated feedback within a UCD process (Callahan 2005). A precondition to
any user involving activity is to know who the users of the service will be.
MASELTOV and the serious game aim at supporting immigrants to approach socio-
cultural conditions of the host country and successfully integrate themselves (and
their relatives) into the foreign societal system.
Immigrants moving to countries within the European Union come from all over the
world and form a very heterogeneous group (Eurostat 2013). Due to great variety of
user characteristics depending on maternal language, cultural background, motivation,
education, profession, religion and duration of stay, immigrants´ needs (such as indi-
vidual, social as well as security related) differ widely. Hence, approaching all immi-
grants in Europe at once seems to be unreasonable, especially when supporting social
integration. Keeping the cultural diversity in mind, we explicitly target three large
immigrant groups within the European Union (Eurostat 2013): Arabic-speaking im-
migrants from North Africa, Turkish-speaking Turks and Spanish-speaking Latin
Americans. These groups are meant to have a perceptibly different cultural back-
ground than Western cultures, especially in the cultural dimensions of power-distance
and collectivism/ individualism (Hofstede 2001) which should be addressed within
the serious game.
Working with immigrant users requires the consideration of some special chal-
lenges for the process of UCD: (i) in general the impact of the cultural orientation of
the immigrants toward the home and the host country on the design process is unclear,
(ii) designers might not be familiar with relevant particularities within the cultural
orientation of the home country (e.g. basic values, attitudes), (iii) recruitment
processes for the successful acquisition of participating volunteers from a vulnerable
user group according to pre-defined criteria are complex and laborious (Aykin et al.
2006), and (iv) mistrust towards involved experts from design and research might lead
to a lack of frankness on the part of the immigrants (Hynes 2003). These issues and
challenges emphasize the importance of user involvement as otherwise no accurate
predictions can be made whether immigrants of the defined target group would be
able and willing to use the developed services. To reduce the mentioned issues we
created a framework for the requirements analysis of the UCD process (Bobeth et al.
2013) and collaborated with three nongovernmental organizations (NGOs) in Graz
(Austria), London (UK) and Madrid (Spain) who provide support services for immi-
grants and organized the recruitment of the participants for user involving activities in
MASELTOV.
sensors that can make sense of the environment and trigger recommendations. For
example, in a typical MASELTOV scenario, an inbuilt GPS sensor may register that
the user is in the locality of a train station, and may offer to provide a language learn-
ing lesson based around travelling. Knowledge gained from incidental learning has
been argued to develop self-confidence and increase self-knowledge in learning (Oga-
ta & Yano, 2004) and is hence particularly suitable for recent immigrants who may
not have had previous positive experiences with formal education.
Flexibility in educational provision and delivery is crucial to support these learners,
who may also struggle to attend regular classroom based learning due to constraints of
work and family lives. Mobile ICT based learning may therefore provide a very suita-
ble form of education for immigrants (Kluzer et al. 2011, pp.9-10). Mobile phones are
known and trusted by many people, and ‘domesticated’ into their everyday practices.
This integration with everyday life is beneficial for language learning. Second lan-
guage acquisition is perceived by adult migrants as well as host governments, “as a
crucial factor for socio-economic and cultural integration”, with language acquisition
and social integration closely intertwined (Van Avermaet & Gysen, 2009). For exam-
ple, knowledge of specific terminology is required to sit a driving licence exam,
which then opens up opportunities for employment (Farinati et al., 2012).
However, a challenge with incidental learning is that, since it occurs in everyday
life during other activities, it may occur in “fragments” or small learning episodes that
are weakly structured and disconnected from each other. It is therefore necessary to
understand how such fragmented learning episodes can be reconceived by users as
elements of a more coherent, longer term learning journey towards social inclusion.
We argue that timely and appropriate feedback and progress indicators may encour-
age this shift towards a more reflective learning process.
We propose that feedback and progress indicators (FPIs) may play an instrumental
role in helping learners reflect upon the individual learning episodes and conceive
them as constituting elements of a longer learning journey (Gaved et al., 2013). Fur-
thermore, educational research suggests that timely and appropriate feedback and
indicators of progress can motivate learners (Nix and Wyllie, 2009). We define ‘feed-
back’ as responses to a learner’s performance against criteria of quality and as a
means of directing and encouraging the learner; and ‘progress indicators’ as responses
indicating the current position of a learner within a larger activity or. Well-presented
feedback can help learners to take responsibility for their own learning. In addition,
learners’ activity using a computer mediated set of learning tools (such as the
MASELTOV serious game and other apps) can be captured and held within the sys-
tem to act as feedback to the software and educational developers, who can act on
patterns of use to iteratively improve their services, for example recognizing that
learners struggle with particular tasks, either because the user interface requires fur-
ther design or because the content itself is at an inappropriate level.
FPIs provided to the learners can be instantiated in a range of ways and can be
cognitive (such as knowledge achieved, assessment results), affective (praise, emo-
tional reflection) and social (peer ratings of quality of participation, support). FPIs are
recognized as an important aspect of learning, and have been well described for both
formal and informal learning. We have identified previously (Gaved et al., 2013) that
there is a scarcity of FPIs present in incidental learning, notably with respect to goal
setting, planning, structured feedback from peers, and reflection on improving per-
formance for specific activities. Encouraging these specific types of feedback and
progress indicators may enable the transition from isolated, sporadic learning on a
problem by problem basis towards a more reflective learning process and enable the
recent immigrant to consider their broader social inclusion goals. However, feedback
has to be managed sensitively since the effects of feedback on performance are highly
variable: “under some conditions, feedback may improve performance, and under
other conditions, feedback may reduce performance (Kluger and DeNisi, 1996)” (in
Garris, 2002). Gibbs’ model of reflection (see Figure 2) indicates cognitive, affective
and social aspects of the process, and we can see that these could form the basis for
feedback prompts to the MASELTOV user to encourage them to both look back on
what they have done, as well as forward to plan what they might do next.
Summaries of a MASELTOV learner’s progress and feedback to and from the
learner will be stored and displayed in a personal space to enable the learner to reflect
and monitor their own progress. Many services offer a “progress dashboard” which
shows users a report of their progress across a number of tasks, often as a summary of
all their activities. The information in this dashboard is presented as a list, or as a
graphical representation; it may be presented in an initial ‘home page’ for the soft-
ware or alternatively held on a personal profile page. Such dashboards enable learners
to ‘own’ their user profile and use it as a tool for self-reflection and modifying future
goal planning. This approach is being considered for implementation in MASELTOV.
The first generation of serious games focused more on high priority learning envi-
ronments such as first response training, levee patrolling and medical simulation
(Knight et al., 2010). But more recently, games have been regarded not only as a basis
for learning through simulation, but also as a tool for the development of a wider
range of skills such as communication, through more diverse and abstract forms of
play (Zielke et al., 2009; de Freitas & Routledge, 2013). The main reason that games
can be so effective in cognitive-based areas of training arises from the way in which
games allow for negotiated meanings to be developed within individuals; as part of
group activities, shared realities can be negotiated and derived. This characteristic is
mainly affected through game design mechanics that allow for individual tasks and
practices to be blended with social elements and services. One way in which games
can offer this capability is to have the learner assume the role of an avatar, embodying
them within the environment and thus allowing for subjective relations to be asso-
ciated with the behaviors of their own and other avatars. This individual connection
between gameplay and the player or learner then consequently placed at the heart of
the relationships set up in such a game, making it an effective tool for both replicating
real, and projecting imagined environments. Both support higher cognitive learning
and can be a valuable tool in how games as learning tools can be deployed to support
formal and informal learning and training.
Serious games have demonstrated capabilities for supporting a diverse range of pe-
dagogical models. The cultural learning space is a complex and intricate environment,
which must take into account many different aspects such as language learning skills,
interpersonal skills, team skills and problem-solving. Within the MASELTOV
project, the design seeks to encompass both the notions of 'playful' cultural learning,
whereby learning is supported in an engaging and intrinsically motivating form, and
the concept of using a 'freemium' model which provides premium content to players
on the basis of positive actions and learning activities, using players' time and activi-
ties as a currency rather than a financial model. In previous work, we have reviewed
the use of games in cultural heritage for example (Anderson et al., 2010), a key find-
ing being that the method of game design and development needs to find a balance
between fun and learning, utilize immediate feedback, and requires research and de-
velopment based upon four dimensions: the learner, their context, the representation
of the game, and the learning theories and approaches deployed. Otherwise, it may be
either too close to an entertainment game, or too close to existing formal educational
materials, limiting efficacy.
To this end, development seeks to focus first on establishing an engaging and 'fun'
gameplay model, attracting a player base. In the context of MASELTOV, the game
will be deployed on a mobile (Android) platform. We seek to exploit these benefits by
focusing first on the entertainment aspects, working with an existing cultural model to
ensure pedagogical aspects (Hofstede, 2005). Research will then seek to work with
both the attracted audience of 'real' players of the game, as well as with an invited
audience of immigrants in form of usability and user experiences studies supported by
non-government organizations who provide support to immigrants on a daily basis.
Working with this target group to evolve the game and ensure pedagogical require-
ments are met aims to address the careful balance between engagement and education,
allowing the various methods of conveying learning content and encouraging beha-
vioral changes to be evaluated, refined, and validated. This will be achieved through
both direct metrics obtained from the game engine, for example time spent with cer-
tain content elements and the extent to which external resources within the
MASELTOV platform are used, as well as through qualitative feedback and surveys.
An image taken from the prototype game is shown in Figure 3. Whilst the envi-
ronment seeks to immerse the player in a pseudo-realistic context, a game-based narr-
ative and 'dimension flip' mechanic, coupled with playable platform levels, it wraps
cultural learning through dialogues with an immersive storyline and platform gamep-
lay which seeks to engage the player and to generate a perceived value for character
upgrades and game progression. The game then intends to use this perceived value to
explore how the currency for these upgrades might be earned through positive actions
using other MASELTOV services: for example rewarding use of the social network,
posting on the forums, using information sources, or engaging with mixed-reality
challenges using the context awareness services or text lens. The dimension flip also
plays a central role in the game, allowing the player to observe two distinct, fictional
cultures at opposite poles informed by Hofstede's model of cultural differences, and
via scaffolding provided by other learning services within MASELTOV, to learn
incidentally how, where, and why differences in cultures can cause issues.
Fig. 3. Image from the MASELTOV game, currently in development for Android
The User Profile is a central component of the MASELTOV game platform that
maintains information about personal data and user preferences, as well as collected
knowledge on user usage behavior, progress and user context recognitions (Dimakis
et al., 2010) that enable the personalization of used services, for example triggering of
personalized recommendations on MASELTOV functionalities, targeted services and
assistance. The feedback and progress indicators that are maintained with the User
Profile will facilitate the overall learning journey, allowing the monitoring of the user
satisfaction for the offered services and the offering of advanced personalized targeted
services.
(a) (b)
Fig. 4. (a) Architecture of the User Profile and the Recommender System. (b) Modes of
geo-contextual awareness applied in MASELTOV.
Recommendations can feed into serious game components, such as, recommending
to play a specific component of the mobile serious game, e.g., buying a ticket at the
virtual bus stop. The objective of the recommender system is to issue useful persona-
lized recommendations to its user based on the events and notifications it receives
from other MASELTOV app (MAPP) applications. The information that is carried
through the events and notifications allow the recommender system to formulate the
required level of context awareness, and, in combination with a set of rules that speci-
fy the preconditions under which actions should be taken, allow it to issue targeted
recommendations. The functioning of the recommender system depends on (a) events
received from MAPP applications, which are application dependent and reflect the
current user activity and context, (b) notifications received from MAPP platform ser-
vices, like the current user location, (c) the information contained in the user profile,
like user preferences and progress indicators, and (d) a set of defined rules, which
may also be specialized to specific user groups.
The architecture that is presented in Figure 4a shows the two platforms, namely the
smartphone and the backend, and the components that run on each of the two. The
user profile makes use of a backend database to store user preferences data and in-
formation that is communicated to it, like events coming from MAPP applications and
notifications coming from MAPP services. The stored information can also be used
by more advanced recommender systems, either rule based ones or based on statistical
processing of data. The architecture shows a lightweight rule based recommender
system that runs on the smartphone and a fully-fledged rule based recommender sys-
tem that runs on the backend server.
The recommender makes use of a set of rules, which specify actions (recommenda-
tions in this case) to be taken when certain conditions are satisfied. A general form of
a rule is Predicate → Action [expiration specification]. The interpretation of this rule
is that when the predicate is satisfied the rule fires and the action is taken. A typical
recommendation for language learning, for example, would take status features from a
geo-contextual event (‘at the bus station’) to trigger an appropriate recommendation
action ‘propose language lesson on vocabulary/expressions used at a bus station’.
(a) (b)
Fig. 5. Progress indicator underlying the mobile game based language learning. Dialogues are
evaluated using voice analysis and its relation to eye contact.
References
1. Aykin, N., Honold Quaet-Faslem, P., Milewski, A.E.: Cultural Ergonomics. In: Salvendy,
G. (ed.) Handbook of Human Factors and Ergonomics, pp. 1418–1458 (2006)
2. Bobeth, J., Schreitter, S., Schmehl, S., Deutsch, S., Tscheligi, M.: User-Centered Design
between Cultures: Designing for and with Immigrants. In: Kotzé, P., Marsden, G.,
Lindgaard, G., Wesson, J., Winckler, M. (eds.) INTERACT 2013, Part IV. LNCS,
vol. 8120, pp. 713–720. Springer, Heidelberg (2013)
3. Callahan, E.: Interface design and culture. Annual Review of Information Science and
Technology 39(1), 255–310 (2006)
4. de Freitas, S., Routledge, H.: Designing leadership and soft skills in educational games:
The e-leadership and soft skills educational games design model (ELESS). British Journal
of Educational Technology (2013)
5. Dimakis, N., Soldatos, J., Polymenakos, L., Bürkle, A., Pfirrmann, U., Sutschet, G.:
Agent-based architectural framework enhancing configurability, autonomy and scalability
of context-aware pervasive services. Autonomous Agents and Multi-Agent Systems 21(1),
36–68 (2010)
6. Dunwell, I., Lameras, P., Stewart, C., Petridis, P., Arnab, S., Hendrix, M., de Freitas, S.,
Gaved, M., Schuller, B., Paletta, L.: Developing a Digital Game to Support Cultural Learn-
ing amongst Immigrants. In: Proc. IDGEI 2013, SASDG (2013)
7. Eurostat Migration & Migrant Population Statistics,
http://epp.eurostat.ec.europa.eu/statistics_explained/
index.php/Migration_and_migrant_population_statistics
(last access: July 11, 2013)
8. Farinati, L., Masseroni, M., Vimercati, M.: Parliamoci Chiaro: an online opportunity to
learn Italian for immigrants. In: Pixel (ed.) Proc. ICT for Language Learning, Padova,
Italy, pp. 65–68 (2012)
9. Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, and learning: a research and prac-
tice model. Simulation and Learning 33(4), 441–467 (2002)
10. Gaved, M., Kukulska-Hulme, A., Jones, A., Scanlon, E., Dunwell, I., Lameras, P., Akiki,
O.: Creating coherent incidental learning journeys on mobile devices through feedback and
progress indicators. In: Proc. mLearn (2013)
11. Gibbs, G.: Learning by doing: a guide to teaching and learning methods (1988)
12. Halfman, J.: Citizenship Universalism, Migration and the Risks of Exclusion. The British
Journal of Sociology 49(4), 513–533 (1998)
13. Hofstede, G., Hofstede, G.J.: Cultures and Organisations: Software of the Mind (2005)
14. Hofstede, G., Hofstede, G.H.: Culture’s consequences: Comparing values, behaviours, in-
stitutions, and organizations across nations. Sage, Beverly Hills (2001)
15. Kluger, A.N., DeNisi, A.: The effects of feedback interventions on performance: A histori-
cal review, a meta-analysis, and a preliminary feedback intervention theory. Psychological
Bulletin 119, 254–284 (1996)
16. Kluzer, S., Ferrari, A., Centeno, C.: Language learning by adult migrants: policy
challenges and ICT responses. JRC-IPTS, Seville (2011)
17. Knight, J., Carly, S., Tregunna, B., Jarvis, S., Smithies, R., de Freitas, S., Mackway-Jones,
K., Dunwell, I.: Serious gaming technology in major incident triage training: A pragmatic
controlled trial. Resuscitation Journal 81(9), 1174–1179 (2010)
18. LeClus, M.A.: Informal learning in the workplace: a review of the literature. Australian
Journal of Adult Learning 51(2), 355–373 (2011)
19. Muller, M.J.: Participatory design: the third space in HCI. In: Sears, A., Jacko, J.A. (eds.)
The Human-computer Interaction Handbook, pp. 1061–1081 (2009)
20. Muratet, M., Delozanne, E., Torguet, P., Viallet, F.: Serious game and students’ learning mo-
tivation: effect of context using prog&play. In: Cerri, S.A., Clancey, W.J., Papadourakis, G.,
Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 123–128. Springer, Heidelberg (2012)
21. Nix, I., Wyllie, A.: Exploring design features to enhance computer-based assessment:
learners’ views on using a confidence-indicator tool and computer-based feedback. British
Journal of Educational Technology 42(1), 101–112 (2009)
22. Ogata, H., Yano, Y.: Context-aware support for computer-supported ubiquitous learning.
In: Roschelle, J., Chan, K.T., Yang, S. (eds.) Proc. WMTE 2004, pp. 27–34 (2004)
23. Paletta, L., Santner, K., Fritz, G., Mayer, H., Schrammel, J.: 3D Attention: Measurement
of Visual Saliency Using Eye Tracking Glasses. In: Proc. CHI 2013, pp. 199–204 (2013)
24. Schuller, B., Dunwell, I., Weninger, F., Paletta, L.: Pervasive Serious Gaming for Beha-
vior Change – The State of Play. IEEE Pervasive Computing 3(12), 48–55 (2013)
25. Silva, P.M.: Epistemology of Incidental Learning. PhD, Virginia Polytechnic and State
University (2007), http://scholar.lib.vt.edu/theses/available/
etd-10162007-224008/unrestricted/silva.pdf (retrieved)
26. Song, Y., Fox, R.: Uses of the PDA for undergraduate students’ incidental vocabulary
learning of English. ReCALL 20(3), 290–314 (2008)
27. Van Avermaet, P., Gysen, S.: Language policies for citizenship and integration: the
Belgian case. In: Extra, G., Spotti, M., Van Avermaet, P. (eds.) Language Testing,
Migration and Citizenship: Cross-national Perspectives on Integration Regimes (2009)
28. Vredenburg, K., Isensee, S., Righi, C.: User-Centered Design: An Integrated Approach (2001)
29. Vygotsky, L.: Mind in Society. Routledge, London (1970)
30. Wildman, D.M.: Getting the most from paired-user testing. Interactions 2(3), 21–27 (1995)
31. Zielke, M.A., Evans, M.J., Dufour, F., Christopher, T.V., Donahue, J.K., Johnson, P.,
Flores, R.: Serious games for immersive cultural training: Creating a living world. IEEE
Computer Graphics and Applications 29(2), 49–60 (2009)
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 456–475, 2013.
c Springer International Publishing Switzerland 2013
2 Background
2.1 Serious Games for Autism
Serious games are increasingly employed as tools for supporting a wide range of
activities, from therapy, e.g. for pain management and rehabilitation, to training
of specific skills, such as literacy and numeracy [6]. Educational games based on
virtual reality are believed to offer particular benefits for children with ASCs
[7], because they can help alleviate children’s stress associated with real social
interactions and because they allow the child to repeatedly rehearse behaviours
in different scenarios, from simple and structured situations to increasingly more
complex and unpredictable ones. Studies show that autistic children who were
taught by a virtual human experience higher levels of retention than those in
traditional classroom settings [8] and that virtual agents may promote generali-
sation [9] - a holy grail of any autism intervention. There is also some evidence
that both role-play and practice of behaviours across different contexts can con-
tribute to increasing the chances of transferring the learned skill from the virtual
to the real world [10].
The majority of serious games for autistic children have focused mainly on
the following domains of social communication:
Language Skills. Games targeting these skills range from systems that aim to
increase the fluency of speech [11] to systems that improve the intelligibility of
speech [12]. Since understanding child speech is still technologically challenging,
some of the more successful applications involve a “Wizard of Oz” approach,
where a human experimenter manipulates the system [9]. Recent improvements
in speech recognition allow for technologically more sophisticated support (e.g.
TouchStory [13]), and open up the possibility of focusing on literacy skills and
expanding children’s vocabulary [14,15].
Although in the last ten years there has been a growing interest in the po-
tential of artificial agents in the area of autism intervention, both virtual and
physically embodied, the efforts have focused primarily on agents with little or
no autonomy. Typically, in such contexts, virtual agents are either authored a
priori or controlled by a practitioner through a control panel, while robots are
tele-operated. The approaches which form the exceptions in this respect include
the Thinking Head project [25], which focuses on developing a talking head
that teaches social skills, through realistically portraying facial expressions, and
the virtual peers, Baldi and Timo, [14] – 3D computer-animated talking heads
for language and speech training. The limited effort in the area of autonomous
agents for autism intervention can be in part explained by the fact that, in line
with some behaviourist clinical intervention frameworks, most technologies in
this context focus on training children with respect to specific skills, e.g. recog-
nising a predefined set of facial expressions, rather than on creating believable
social interaction experiences, which can be freely explored by the child.
Planning techniques are crucial to enriching virtual characters with autonomy.
Path planning, for example, is extensively used in action games to avoid collisions
between multiple characters and to allow characters to inhabit and move around
architecturally complex environments [26,27]. Task planning is used to enhance
the quality of individual moves in puzzle games [28] and to help coordinate
actions by multiple characters [29]. Narrative and storytelling planning are of
particular relevance to serious games for supporting social interaction, because
their aim is to support the generation of evolving contexts and stories within
which characters are able to manifest intentionality and emotions [30]. Such
emergent narratives are useful in contextualising the behaviours of characters
and the training of social skills as reviewed in Section 2.1
User Modelling. In general, user modelling consists of: (1) static profile of the
user, containing information such as gender, age, preferences etc., and (2) a diag-
nostic system responsible for inferring the user’s hidden psychological states from
behaviours observable online. Specifically to games, recent reviews discuss mod-
elling players’ behaviours in terms of their actions vis á vis the games’ objectives,
their tactics (short-term behaviours) and strategies (long-term action planning)
and player profiling in terms of their psychological predispositions based on the
detected actions, tactics and strategies [31]. In this context, the player modelling
applications tend to focus on adult players in shoot-them-up or strategy games.
Relatively little effort is available in relation to modelling children as players,
with the notable exception of work by [32], who create and evaluate user models
that predict 8-11 year old children’s entertainment of games.
The recent recognition of serious games as a genre with substantial potential
in complimenting traditional education motivates the need for user modelling in
the context. Much work that is relevant to modelling players of serious games
comes from the related fields of Intelligent Learning Environments and Affec-
tive Computing. Many accounts exist of the nature of mental states that relate
to user motivation and learning. Malone [33] highlights factors such as challenge,
curiosity and fantasy that impact on users’ enjoyment of and learning with dig-
ital environments and these factors are also increasingly investigated in relation
to users of serious games. Csikszentmihalyi [34] discusses flow as crucial to users’
ability to immerse themselves in virtual worlds, with the specific dimensions of
flow in relation to children game playing, being identified as users’ endurabil-
ity, engagement and expectations [35]. Flow refers to users’ sustained focus of
attention which psychologists link to children’s improved memory, learning and
increased ability to self-regulate emotionally [36]. Burleson [37] discusses the
state of stuck as a factor potentially detrimental to users’ enjoyment and learn-
ing. User achievement of the goals within a specific environment and the user’s
personal goals that emerge from a combination of their mental states and per-
sonality traits was examined by Ortony et al. [38] and most notably implemented
in a learner model of 6th and 7th grade students (in the US) by Conati [39].
Research in Artificial Intelligence in Education provides further insights into the
older children’s mental states that are important for learning with technology
(e.g. [40]), how these may relate to the specific forms of digital environments
and contexts of use (e.g. [41]) and types of help provided to the child (e.g. [42]).
Research in affective computing and use of sensors for detecting users’ be-
haviours in games and education, is thriving. Most existing approaches rely on a
combination of cues, such as touch, eye-gaze, facial expression tracking and voice
recognition [43]. A recent example of user modelling that relies on cue detection
in the context of serious games for social interaction include the work by the
TARDIS project, which focuses on developing a technology for training young
adults at risk of social exclusion to self-regulate emotionally and to self present
to potential employers [44].
Fig. 1. A child interacting with the Fig. 2. Andy, the SHARE-IT agent
ECHOES system through multi-touch inter- demonstrates pot stacking to the child.
face. In SHARE-IT, we reuse the types of the The idea is that the child will imitate
activities and most of the skin (appearance) Andy’s action and take turns with Andy
of the objects and the environment. in subsequent moves.
unexpected, e.g. the user not responding, not taking turns, or interacting with
the environment in ways unrelated to the activity. Unlike in many games, the
difficulty levels in SHARE-IT do not correspond to the different activities, but
rather they are organised around the different types of pointing, from relatively
ambiguous distal pointing with gaze (i.e. looking at an object), to proximal
pointing by touching an object of interest. A combination of several types of
pointing, such as distal pointing through gaze and finger point, as well as the
agent’s verbal request, e.g. “Pass me that pot” is considered the least difficult in
SHARE-IT. Prompting and verbal guidance, such as Andy reminding the child of
their turn is also used to regulate the difficulty level within activities. Children
who respect turn-taking will get fewer prompts, whereas more scaffolding is
needed for children who have difficulties in this area.
Child's
Eye Data Engagement
User Pedagogic
Face Data Model Component
Agent Scene
User Input
Architecture Change
FAtiMA Planner
Scene
Touch/ Contents Scene
Mouse UNITY 3D Games Manager
Input Engine
Scene
Change
support. However there are major drawbacks with this approach, which can hin-
der long-term extendibility of the system, its computational efficiency and conse-
quently its real world robustness. Specifically, developing a system that relies on
a distributed architecture is very resource heavy, because a separate process is
needed for each component (and possibly Java Virtual Machine). Furthermore,
debugging such an architecture is difficult, because a separate debugger is needed
for each component. Based on our experience and evaluation of the ECHOES
system, we opine that a distributed architecture in this context increases system
complexity without adding any significant gain.
The idea behind SHARE-IT is to deliver a platform for authorable game
development and this, coupled with the fact that there is already significant
complexity involved in implementing an autonomous agent, leads to our adopting
the philosophy to keep the SHARE-IT system architecture and code design as
simple as possible at every level, with additional layers of abstraction being
avoided wherever possible. To this end, the whole system is designed around
a single process, using just one language. All system resources are kept in one
location avoiding the problem of contention as all threads in the system are
managed from the one process. Additionally, using a single language allows us to
maintain a high degree of code consistency making the project easier to maintain
and extend.
Another decision relates to the use of an existing games engine to generate
the SHARE-IT game. While the system could have been written from scratch, as
indeed it was for ECHOES, a lot of unnecessary work and complexity is added
to the project for no appreciable gain. There are many commercial and freeware
games engines that can handle all the rendering, audio file handling, character
animation, 3D effects and physics that are required for projects like SHARE-IT.
Any games engine of this type will automatically handle user input from mouse
and touch screen. However, eye-tracking and facial expression recognition need
to be handled separately. Since the User Model is the primary consumer of these
data, which it uses to make judgements about the child’s level of engagement
with the system and the frequency and consistency of the desired behaviours,
these inputs are routed directly to it. The output from the User Model is then
routed to the Planner to allow it to make more informed decisions about how
the agent should collaborate with the user in the task at hand. In addition to
the User Model, the Pedagogic Component also communicates with the Planner
providing a top level control over what forms of support should the agent provide
to the individual children along with the frequency and the kind of prompting,
thereby being responsible also for setting the difficulty levels for each individual
child. The Pedagogic Component is essentially a rule-base expressing SCERTS
recommendations, which it executes based on information about the child’s en-
gagement supplied by the User Model.
ground 3D objects. Although the objects may be animate, they are so in very
simple ways compared to an agent. An agent itself comprises a physical embod-
iment: a number of manifest behaviours – generally animated movement and
speech, and rules to control these behaviours.
The choice of artwork to be used for the agent and the environment is critical
for the success of a game. However life-like the agent’s behaviour or enticing the
activities, the child will not be drawn in unless the physical environment of the
game is comfortable and attractive. Ideally, the appropriate art design should
involve a participatory effort between children, practitioners and designers to
ensure that the design is attractive to children, while lending itself to support-
ing the pedagogical aspects of the game. For example, in activities focused on
improving children’s attentional control, it is important that the number, the
type and the interactive nature of objects acting as distractors are in line with
the pedagogic requirements of the task. This paper does not address the creation
of artwork, since its focus is on game development. In SHARE-IT, we re-use the
artwork that has been inspired by the children and teachers themselves, and
which proved very successful with most of almost a hundred children who used
the ECHOES environment over the years.
However, it is important to consider that in the context of serious games for
social interaction, there is an increased need for precision in the execution of
certain gestures of the agent and facial expressions, especially if the expecta-
tion is for the child to recognise emotions in such expressions, follow pointing
by the agent or imitate the behaviours modelled by the agent. Large libraries
of resources are available for choosing artwork, for example TurboSquid1 and
RocketBox2 and both provide rigged characters that can be animated for use as
agents. Motion capture offers the best results, especially for fine-grained hand
gestures such as one-finger pointing needed by an agent to unambiguously point
to objects, e.g. in initiating bids for interaction with the child, as well as Maka-
ton sign gestures, often used to support instructions and communication with
children with autism (see Fig. 4).
1
http://www.turbosquid.com/
2
http://www.rocketbox-libraries.com/
Fig. 5. SHARE-IT authoring tool for morphing the agent’s emotional expressions
activities ought to focus and (ii) the specific narrative content of the activities
itself. These will be incorporated in the Pedagogic Component of the system
and will be used to inform the goals for the agent to actively pursue along with
appropriate action strategies for each learning activity.
User Modelling. SHARE-IT also reuses ECHOES’ user model, which esti-
mates the child’s cognitive and affective states in real-time and feeds back this
information to the Pedagogic Component and the Planner whenever a change in
a given state is detected. In SHARE-IT, the model receives real-time information
from the touch and eye-gaze systems and, based on this data, it interprets the
current level of child’s engagement as well as the level, consistency and desirabil-
ity of the child’s attention vis á vis the goals of a given activity or specific agent
actions. The cognitive assessment is facilitated by a rule-based engine which es-
timates the extent to which the child has achieved the goals associated with
the session, e.g. based on the number of the child’s responses to the agents bids
for interaction in a pre-specified time frame. The rules are based on SCERTS
guidelines and precise timing constraints for establishing the child’s mastery of
joint attention and symbolic use skills. For example, the behaviour “shifts gaze
between people and objects” is satisfied if the child shifts gaze spontaneously
between a person and an object at least three times and the entire sequence oc-
curs within two seconds. Detailed real-time eye-gaze data is essential to enable
the interpretation outlined.
In ECHOES, the assessment of the child’s engagement was based on a com-
bination of supervised and unsupervised learning techniques [49]. Engagement
is defined in terms of: very engaged, engaged and not engaged. The children are
considered “very engaged” when they are fully absorbed by the learning activity;
“engaged” when they are interested in the current learning activity, but are not
immersed in it; “not engaged” when they do not interact with the system at all,
either via gaze or touch.
5 Authoring Tools
The authoring environment for SHARE-IT is intended for use by teachers and
parents, so that they can tailor the game according to the needs of the individual
children in their care. The authoring environment is designed to allow existing
activities to be chosen and new ones to be created from a palette of components.
A great deal of effort in constructing the Unity game for the project was spent
creating a range of objects and agent animations and speech to provide the
building blocks for this tool. Ease of use is a primary consideration given the
non-specialist nature of the end users and is a continuing focus of participatory
design workshops with teachers and parents.
The authoring application consists of three components: (1) A scene selector
tool (Fig. 6) that allows the scenes and their order to be selected. It also controls
display of a scene selection menu in the game which permits the user free choice
of scenes. Furthermore, basic object properties, such as object transformations
can be set from this too; (2) A scene creator/editor window (Fig. 7), which
is used to setup or alter the physical components of a scene - how many of
each object and where they are placed, their colour and size, and the scene
background; (3) A rule creation dialogue (Fig. 8), which is used after the
scene creator to set up the planner rules which will govern the agent’s behaviour
in the scene.
The first two of these components write the XML initialisation file for Unity
to read on startup. This allows Unity to create all the required scenes at runtime.
Fig. 6. SHARE-IT authoring tool for se- Fig. 7. SHARE-IT authoring tool for
lecting and ordering learning activities choosing the objects, their properties and
number
The rule creation component is somewhat more involved, as it has to map the
elements selected by the scene author into rules that the FAtiMA planner can
consume. As shown in Fig. 8 this might be an activity to throw bouncy balls
through the cloud to turn them different colours and in which the agent and user
take turns with each ball. Each action type has an XML template that forms the
basis of the rule in Fig. 9, which then gets filled in with the details of the objects
Fig. 8. SHARE-IT authoring tool for Fig. 9. XML rule relating to the rule created
creating planner rules. in the editor shown in Fig. 8
individual activities as specified by the user in the Rule Creation tool (Fig. 8).
In this way any combination of objects and actions on objects can be specified
to create an activity.
In this paper, we introduced the SHARE-IT project, which aims to design and
implement an infrastructure for parents and teachers to become co-creators of
their children’s technology-enhanced learning experiences. We outlined a number
of practical considerations of importance to building intelligent and autonomous
serious games for children with autism and for their carers. The work reported
is very much in progress, albeit it is based on substantial previous work re-
lated to the ECHOES project, which it extends. The considerations are there-
fore grounded in our extensive experience of developing an intelligent serious
game for social interaction. Much work still remains to be done, particularly
in relation to enriching our virtual character with more interesting and varied
behaviours, extending the range of characters available to include females and
those representing different ethnicities. A major limitation in the agent’s intelli-
gence is that the agent cannot currently deal with inappropriate or unexpected
behaviour from the user, such as not responding to the agent, not honouring
turn taking, being occupied with objects in the scene not related to the activ-
ity, complete disengagement, etc. We are currently adding further rules to the
planner’s actions to cover the most obvious cases, although in the longer term it
would be desirable to make these behaviours part of the agent’s personality, so
that they would vary with the affective system parameters. We are also focusing
on improving the accuracy of the SHARE-IT modelling tools. Incorporating a
robust eye-gaze tracking system is of special importance in this context because
the ability to follow gaze by social partners provides the foundations of suc-
cessful social communication. While the Tobii technologies currently employed
are still unaffordable by most, they are set to enter the mainstream market as
part of ordinary PCs in the next 12 months. We are currently working with
parents and teachers on refining the design and the variation of the SHARE-IT
learning activities to make them more flexible to extend and we are testing the
authoring tools to assess their usability and robustness. In the next 3 months
studies with children, parents and teachers are planned, where the SHARE-IT
technologies will be piloted simultaneously in school and home contexts and the
authoring tools will be tested in-the-wild, with a special attention being paid
to the relevance and feasibility of the activities across the two contexts. While
children are the main beneficiaries of the game developed, providing parents
and teachers with authoring powers is also expected to enhance parents-school
communication and continuity, consistency and coherence of the experience pro-
vided to children. The results of the planned studies will be reported in due
course.
References
1. Medical Research Council: Autism - Research Review (2001) (accessed October 20,
2012)
2. Knapp, M., Romeo, R., Beecham, J.: The economic consequences of autism in the
uk. Foundation for People with Learning Disabilities Report (2007)
3. Prizant, B., Wetherby, A., Rubin, E., Laurent, A., Rydell, P.: The SCERTS R
Model: A Comprehensive Educational Approach for Children with Autism Spec-
trum Disorders. Brookes (2006)
4. Wass, S., Porayska-Pomsta, K.: The uses of cognitive training technologies in the
treatment of autism spectrum disorders. Autism: International Journal of Research
and Practice (in press)
5. Bernardini, S., Porayska-Pomsta, K., Smith, T.J., Avramides, K.: Building au-
tonomous social partners for autistic children. In: Nakano, Y., Neff, M., Paiva,
A., Walker, M. (eds.) IVA 2012. LNCS (LNAI), vol. 7502, pp. 46–52. Springer,
Heidelberg (2012)
6. de Freitas, S.: Learning in immersive worlds. Technical report, Joint Information
Systems Committee, Bristol (2006) (accessed October 20, 2012)
7. Parsons, S., Cobb, S.: State-of-the-art of virtual reality technologies for children on
the autism spectrum. European Journal of Special Needs Education 26(3), 355–366
(2011)
8. Grynszpan, O., Martin, J.C., Nadel, J.: Multimedia interfaces for users with high
functioning autism: An empirical investigation. International Journal on Human-
Computer Studies 66(8), 628–639 (2008)
9. Tartaro, A., Cassell, J.: Playing with virtual peers: bootstrapping contingent dis-
course in children with autism. In: Proceedings of the ICLS 2008, pp. 382–389 (2008)
10. Parsons, S., Mitchell, P.: The potential of virtual reality in social skills training
for people with autistic spectrum disorders. Journal of Intellectual Disability Re-
search 46(5), 430–443 (2002)
11. Anwar, A., Rahman, M., Ferdous, S., Anik, S., Ahmed, S.: A computer game based
approach for increasing fluency in the speech of the autistic children. In: Proceed-
ings of the11th IEEE International Conference on Advanced Learning Technologies
(ICALT), pp. 17–18 (2011)
12. Rahman, M.M., Ferdous, S., Ahmed, S.I., Anwar, A.: Speech development of autis-
tic children by interactive computer games. Interactive Technology and Smart Ed-
ucation 8(4), 208–223 (2011)
13. Davis, M., Otero, N., Dautenhahn, K., Nehaniv, C., Powell, S.: Creating a software
to promote understanding about narrative in children with autism: Reflecting on
the design of feedback and opportunities to reason. In: IEEE 6th International
Conference on Development and Learning, Proceedings of the Development and
Learning, ICDL 2007, pp. 64–69 (July 2007)
14. Bosseler, A., Massaro, D.: Development and evaluation of a computer-animated
tutor for vocabulary and language learning in children with autism. Journal of
Autism and Developmental Disorders 33(6), 653–672 (2003)
15. Massaro, D.W.: Embodied agents in language learning for children with language
challenges. In: Miesenberger, K., Klaus, J., Zagler, W.L., Karshmer, A.I. (eds.)
ICCHP 2006. LNCS, vol. 4061, pp. 809–816. Springer, Heidelberg (2006)
16. Finkelstein, S.L., Nickel, A., Harrison, L., Suma, E.A., Barnes, T.: cMotion: A new
game design to teach emotion recognition and programming logic to children using
virtual humans. In: Proceedings of the 2009 IEEE Virtual Reality Conference, pp.
249–250 (2009)
17. Abirached, B., Zhang, Y., Aggarwal, J., Tamersoy, B., Fernandes, T., Miranda,
J., Orvalho, V.: Improving communication skills of children with asds through
interaction with virtual characters. In: 2011 IEEE 1st International Conference on
Serious Games and Applications for Health (SeGAH), pp. 1–4 (2011)
18. Schuller, B.: ASC-Inclusion Project (accessed October 20, 2012)
19. Hopkins, I.M., Gower, M.W., Perez, T.A., Smith, D.S., Amthor, F.R., et al.: Avatar
assistant: Improving social skills in students with an asd through a computer-based
intervention. Journal of Autism and Developmental Disorders 41(11), 1543–1555
(2011)
20. Beaumont, R., Sofronoff, K.: A multi-component social skills intervention for chil-
dren with asperger syndrome: The junior detective training program. Journal of
Child Psychology and Psychiatry 49, 743–753 (2008)
21. Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P.,
Ben Sasson, A., Gal, E., Weiss, P.L.: Collaborative puzzle game: a tabletop inter-
active game for fostering collaboration in children with autism spectrum disorders
(asd). In: Proceedings of the ACM International Conference on Interactive Table-
tops and Surfaces, ITS 2009, pp. 197–204. ACM, New York (2009)
22. Barakova, E., van Wanrooij, G., van Limpt, R., Menting, M.: Using an emergent
system concept in designing interactive games for autistic children. In: Proceedings
of the 6th International Conference on Interaction Design and Children, IDC 2007,
pp. 73–76. ACM, New York (2007)
23. Kozima, H., Michalowski, M., Nakagawa, C.: Keepon: A playful robot for research,
therapy, and entertainment. International Journal of Social Robotics 1(1), 3–18
(2009)
24. Dautenhahn, K., Werry, I.: Towards interactive robots in autism therapy: Back-
ground, motivation and challenges. Pragmatics and Cognition 12(1), 1–35 (2004)
25. Milne, M., Luerssen, M., Lewis, T., Leibbrandt, R., Powers, D.: Development of a
virtual agent based social tutor for children with autism spectrum disorders. In:
Proceedings of the International Joint Conference on Neural Networks, pp. 1–9
(2010)
26. Snape, J., Guy, S., Lin, M., Manocha, D.: Local and global planning for collision-
free navigation in video games. In: Proceedings of the 3rd International Planning
in Games Workshop, ICAPS 2013 (2013)
27. Jaklin, N., van Toll, W., Roland, G.: Way to go - a framework for multi-level
planning in games. In: Proceedings of the 3rd International Planning in Games
Workshop, ICAPS 2013 (2013)
28. Do, M., Tran, M.: Blocksworld: An ipad puzzle game. In: Proceedings of the 3rd
International Planning in Games Workshop, ICAPS 2013 (2013)
29. Menif, A., Guettier, C., Cazenave, C.: Planning and execution control architecture
for infantry serious gaming. In: Proceedings of the 3rd International Planning in
Games Workshop, ICAPS 2013 (2013)
30. Riedl, M., Young, M.: Narrative planning: Balancing plot and character. Journal
of AI Research 29, 217–268 (2010)
31. Bakkes, S., Tan, C.T., Pisan, Y.: Personalised gaming. Creative Technologies 3
(2012)
32. Yannakakis, G., Hallam, J.: Modeling and augmenting game entertainment through
challenge and curiosity. International Journal on Artificial Intelligence Tools 16(6),
981–999 (2007)
33. Malone, T.W.: Guidelines for designing educational computer games. Childhood
Education 59, 241–247 (1983)
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 476–491, 2013.
c Springer International Publishing Switzerland 2013
The use of serious games for job interview simulations has two advantages: 1) re-
peatable experience can be modulated to suit the individual needs, without the risk of
real-life failure; 2) technologies are often intrinsically motivating to the young [MSS04]
and may be used to remove the many barriers that real-life situations may pose, in par-
ticular the stress associated with engaging in unfamiliar interactions with others. In
this context, the originality of the TARDIS platform is two-fold. First, TARDIS is able
to detect in real-time its users’ emotions and social attitudes through voice and facial
expression recognition, and to adapt the game progress and the virtual interlocutor’s be-
haviour to the individual users. Second, it provides field practitioners with unique tools
for 1) designing appropriate interview scenarios without reliance on researchers and
2) measuring individuals’ emotion regulation and social skill acquisition (via the user
modelling tools), thus enabling a more flexible and personalized coaching for young
people at risk of social exclusion.
This paper presents the TARDIS architecture and game design. The next section
briefly presents some related work in serious games and affective computing. Section 3
presents the TARDIS architecture. Section 4 presents the TARDIS game itself.
2 Related Work
Addressing social exclusion of marginalized young people is a key issue that must be
tackled on a wide range of different levels. The INCLUSO project [EDVDB10] outlined
two main directions in supporting social integration: (1) to encourage and support per-
sonal development and (2) to encourage and support social participation. The TARDIS
project focuses on the first challenge, i.e. on improving communication skills in one-on-
one interactions. Based on the analysis that computer games as well as serious games
are intrinsically motivating to users (e.g. see [MC05]), we have developed a virtual so-
cial coaching platform that follows a serious games paradigm, in which the youngster
can train and evaluate his/her social interaction skills with a virtual agent.
The use of virtual agents in social coaching has increased rapidly in the last decade.
Projects such as those by Tartaro and Cassell [TC08] or e-Circus [APD+ 09] provide
evidence that virtual agents can help humans improve their social skills and, more gen-
erally, their emotional intelligence [Gol06]. In job interview situations, emotional intel-
ligence of both the applicant and the recruiter plays a crucial role: in human to human
job interviews, the personality of the applicant is inferred by the recruiter according to
the mood, the emotions and the social attitudes expressed by the youngster [HH10].
Furthermore, visual and vocal perceptions of the interviewer have been shown to affect
their judgments of the applicants during job interviews [DM99]. Youngsters’ emotional
intelligence, in particular their ability to self-regulate during the interview will affect
the outcome of the interview.
In the TARDIS project, we address two challenges. First, in line with the objectives
of the field of Affective Computing [Pic97], we aim to build virtual agents that react in a
coherent manner (see also [MGR03, PDS+ 04, PSS09]). Based on the non-verbal inputs
(smiles, emotion expressions, body movements) and their goals (making the applicant
at ease or, on the contrary, trying to put him/her under pressure), the agent must select
relevant verbal and non-verbal responses. Second, the model presented in this paper
seeks to take into account all the different dimensions of the socio-affective interaction,
in the context of a job interview situation, including indepth real-time understanding of
the individual youngsters emotional states while they are interacting with the system.
A number of serious games exist that relate to different aspects of TARDIS. First,
the utilization of Microsoft Kinect has become popular as an affordable markerless
motion capture system, and has been utilized successfully in many games with seri-
ous applications. Mostly, however, these games tend to be applied in the medical field,
with applications for stroke rehabilitation patients [JYZ12] and people suffering with
Parkinsons [SS12]. However, when it comes to reading more complex parameters, such
as emotions, regular cameras are also frequently used.
Some examples of serious games that infer complex paradigms about the state of
mind of the user from visual and audio inputs are LifeIsGame [Orv10] and ASC Inclu-
sion [SMBC+ 13]. Both are ongoing projects within the EC framework and focus on
helping children on the Autism spectrum to recognize and express emotions through
facial expressions, tone of voice and body gestures. While these games access the facial
expressions and behaviour of the user in a similar manner to TARDIS, virtual humans
will interact based on low-level signal analysis as opposed to the virtual recruiter in
TARDIS who will respond to high-level socio-emotional behaviours and mental states’
representations.
Some serious games also focus on employment and the work place. A serious game
with some similar objectives to TARDIS is interview training game Mon Entretien
D’embauche [dR13]. Released by telecoms company SFR, the user is represented by an
avatar and progresses through a number of events before the interview itself by select-
ing appropriate responses to interactions with other avatars. While the goal to relieve
stress or anxiety is similar to that of TARDIS, the approach is quite different. While
Mon Entretien D’embauche assesses progress based on the verbal responses of the
player, TARDIS focuses on whether the player is completely engaged and displaying
appropriate non-verbal social behaviours. Another workplace focused game is iSpec-
trum [Ins13], which aims to help people on the Autism Spectrum to prepare themselves
for working in a new environment and increase their employability. It can also educate
employers, increasing their knowledge about integrating someone with Autism within
their company.
My Automated Conversation coacH, MACH [HCM+ 13] utilizes a virtual human
to coach the user’s social communication skills. While one of the applications of this
game is for interview situations, the focus here is on the facial expressions, speech and
prosody. The intelligent agent here will mimic the users behaviour and respond ver-
bally and non-verbally, as was done in Semaine [BDSHP12]. However, while TARDIS
focuses on the underlying emotion or state-of-mind, MACH provides feedback on phys-
ical behaviours such as explicit facial expressions and tone and volume of voice.
practitioner from the social inclusion association. In this context, the TARDIS system
architecture includes four main components:
– The Scenario module controls the discourse during the interview. It tells the vir-
tual recruiter the expectation in terms of emotions and attitudes expressed by the
youngster, depending on the stage of the interview (Scenario Execution in Fig. 1,
Section 3.2).
– The Social Signal modules provide the affective model with information about the
youngster’s emotions and social attitudes (we refer to them as mental states) that
are detected by the system (Social Cues Recognition and Online User Model in
Fig. 1, Section 3.3 and 3.4).
– The Affective Module is responsible for building a model of beliefs and intentions
(for the virtual recruiter) about the mental states of the youngster (in terms of af-
fects) and about the course of actions in the ongoing interview (Affects Dynamics
in Fig. 1, Section 3.5).
– The Animation module is responsible for expressing the virtual recruiter’s affec-
tive state built by the affective model through verbal and non-verbal behaviour, both
in terms of facial animation and body movements (Affects Expressions in Fig. 1,
Section 3.6).
The interaction is recorded and post-game debriefing sessions can be organized between
the practitioner and the youngster. Fig. 1 gives an overview of this architecture. The
following subsections briefly present the different components of the platform. Section
4 presents the TARDIS game paradigm.
The multimodal dialog content is specified with a number of scenes that are orga-
nized in a scenescript (see Fig. 2, panel 2). The scene structure, which can be compared
to those in TV or theatre playbooks, consists of utterances and stage directions for the
actors. In our scenes, directions are animation commands for the non-verbal behaviour
of the virtual recruiter (e.g. gestures, facial expressions, or postures). The (narrative)
structure of our interactive recruitment simulation and the interactive behaviour of our
virtual recruiter is controlled by parallel hierarchical finite state machines (HFSM) spec-
ifying the logic of the scenes played and the commands executed according to a user
reactions and a given state of the interactive performance, see Fig. 2, panel 1.
In contrast with a linear theatre scene arrangement, an interactive presentation comes
along with another degree of freedom, i.e. the reactions of the system to user’s detected
affects or to the virtual recruiter’s mental state. Such reactions have to be considered
in the scenario control. In order to realize this, we have enhanced a linear five stage
recruitment story (see Fig. 2, panel 1.) with reactions to user input. During the five
stages, depending on the attitude that the virtual recruiter wants to convey (see Section
3.5), the virtual recruiter may mimic specific behaviours of the user. For this task, we
use two HFSMs (modelled within each recruitment stage) that allow the character to
react in a similar way as the user’s behaviours. The state machines react to the detected
social cues and trigger an overlay behaviour. This behaviour is blended with any on-
going animation that the virtual recruiter is performing at any time (Section 3.6). For
example, if a user smiles while the virtual recruiter asks a question, the recruiter will
also smile. Similar behavioural reactions are modelled in the case of head movements.
In order to detect and interpret the social cues expressed by the youngster during
the interview, we have integrated social signal modules. In the next section, we first
present the module for the recognition of the youngster’s social signals. In section 3.4,
the module for the interpretation of the recognized social signals are introduced.
The social signal recognition module is able to record and analyze social and psycho-
logical signals from users and to recognize predefined social cues, i.e. behaviours of the
interviewee, conscious or unconscious, that have a specific meaning in a social context
such as a job interview, in real-time. Examples of such signals are smiles, head nods
and body movements. These social cues will allow us to determine the mental state of
the user (Section 3.4) in real-time and will act as a basis for debriefing sessions between
youngsters and practitioners after the interviews.
In TARDIS the social cues were selected based on two criteria: pertinence to the
job interview context and automatic recognition viability from a technological point of
view. In order to determine what social cues are pertinent to the desired context, we
evaluated the data from multiple user studies involving real NEETs and practitioners.
The technological viability was determined using a literature review as well as in-depth
analysis of available state-of-the-art sensing hardware. This is an ongoing process and
we are continuously looking to extend the list of social cues.
The system uses a combination of sensors and software algorithms which offer good
results in terms of accuracy and low intrusion. High accuracy ensures that a youngster’s
social cues are correctly recognized and allows the virtual recruiter to react to them cor-
rectly. It is equally important that the approach has a low intrusion factor. For example,
biological signal sensors are not feasible in this scenario because attaching various sen-
sors to the skin of the users will most likely result in an increase in stress which might
have a negative effect on the user’s job interview performance, but may not be actually
indicative of the user’s actual abilities. Therefore, in the context studied, remote sensors
are preferred.
For recording and pre-processing human behaviour data, our system relies on the SSI
framework5 which was developed as part of our previous work [WLB+ 13]. It provides
a variety of tools for the real-time analysis and recognition of such data. Using SSI
and the Microsoft Kinect sensor we are able to recognize various social cues [DBA13],
such as hand to face, looking away, postures (arms crossed, arms open, hands behind
5
http://openssi.net
Fig. 4. Video playback facility, with the annotation data and Bayesian network output in the
bottom-right corner of the screen
head), leaning back, leaning forward, gesticulation, voice activity, smiles and laughter.
In addition to these, our system is also able to compute the expressivity [CRKK06] of
the user’s movements (i.e. energy, overall activation, spatial extent and fluidity of the
movement). Fig. 3 exemplifies some of the recognized social cues.
Given the recognized social signals, the next step consists in interpreting these sig-
nals to determine the socio-affective state of the youngster. In the next section, we
present the process to interpret the youngster’s social cues.
Net Populator, which infers the probability tables of the Bayesian Networks (BNs),
representing the mental states; Net Demonstrator, which allows to playback a video of
an interaction along with the annotations used to train the networks and the real-time
output of the network for a given mental state; Net Tester – a facility that feeds the
annotations through the Bayesian Networks and produces statistics about the network’s
performance in terms of correct and incorrect classifications.
The BNs and the Social Cue Recognition component (Section 3.3) are integrated , so
that the inferences about mental states can occur in real-time. While the accuracy of the
BNs themselves is relatively high when tested with video data, real-time recognition of
social cues that would provide a sufficient and necessary evidence for the BNs remains
a challenge for TARDIS and for the field of affective computing as a whole. Presently,
we are testing several solutions which leverage and combine continuous stream of in-
formation from the Social Cue recognition module along with information about con-
textual information, in particular the type of question being asked by the recruiter and
the expectations that relate to appropriate response on the part of the youngster.
Based on the inferred youngster’s mental states and based on the course of the inter-
action, the virtual recruiter should determine its affective behaviour. For this purpose,
we have developed an Affective Module presented in the next section.
The Affective Module for the virtual recruiter has two main computational functions:
– it will periodically compute the new affective states of the virtual recruiter, based
on the perceptions, expectations from the scenario and its current affective states.
– it will select actions (i.e. different forms of utterances and specific branching in the
scenario) according to its intentions following the youngster’s mental state.
Figure 5 illustrates the different elements of the Affective module as well as the links
to the other modules.
The Affective model. This module is detailed in [JS13]. It provides a reactive model
based on expectations from the youngster and perceptions (by Social Signal Interpreta-
tion) on the youngster. It allows a computation of the virtual recruiter emotions, moods
and social attitudes. Emotion computation relies on the difference between expectations
and perceptions. Emotions are modelled with OCC categories [OCC88]. Moods evolve
on a middle term dynamic and are directly influenced by emotions following Mehra-
bian Pleasure, Arousal, Dominance framework [Meh96]. One of the originality of this
model lies in the proposal for a computational model of the virtual recruiter social atti-
tudes that rely on its personality and its current mood.
The Decision module. The goal of this module is to build a representation of the young-
ster’s beliefs (a.k.a. Theory of Mind or ToM [Les94]). The virtual recruiter considers
the affective dimensions of the youngster’s answers (computed by the Social Signal
Modules, see section 3.3 and 3.4) in a particular context (the question that has just been
asked by the recruiter) to derive the positive or negative attitude the youngster has with
respect to the considered topic. For instance, if the youngster reacts with too high a
detachment to a question about the technical requirements for the job, then the agent
might deduce that, on the topic skill, the user is not very confident. It will lower the
value of BY oung (skill). This model will, in turn, influence the next utterances of the
virtual agents in function of the recruiter high-level intentions. The agent will select in
the scenario subjects where the youngster is at ease (helpful/friendly) or not (confronta-
tional/provocative).
A person’s affective state is mainly expressed through their non-verbal behaviour (e.g.
facial expressions, body postures, audio cues, and gestures). In order to give the ca-
pability to the virtual recruiter to display its emotions and attitudes computed by the
Affective Module (Section 3.5), we use the Greta system [BMNP07] to compute the
animation parameters for the multimodal expressions of emotions [NHP11]. Moreover,
the virtual recruiter will be animated using a combination of hand-animated gesture
outputs (based on the animation parameters) and posed facial expressions, along with
motion-captured animation techniques. This allows us to exploit the both methods: the
essential flexibility relates to the character’s gestures and movements when reacting to
the input of the youngster, and the innate naturalness and subtleties of human motion
that can be displayed through motion-captured data.
In order to ensure that the emotions conveyed by the virtual character are interpreted
as intended, a number of studies on the perception of emotion through body motion
Fig. 6. Stimuli from one perceptual study showing, from left to right synchronized facial and
body motion, body motion alone and facial motion only for the basic emotion fear
have been carried out, with conversing virtual characters as the stimuli. We have found
that some complex and subtle body motions are difficult to convey through body motion
alone, and so have highlighted the importance of the appropriate combination of facial
and body motions [EE12]. The literature suggests that the combination of facial and
body motions does increase the perception of expressiveness of such emotions on video
game characters (Figure 6). To this end, one major research challenge for the animation
of the virtual character is the integration between the gestures, facial and body motions
of the recruiter to ensure the maximum effectiveness of expressing emotions, while
ensuring the character maintains a sense of naturalness about his/her motions. One way
of making our virtual recruiter appear as natural as possible is through the inclusion of
inherent human behaviours such as idle movements and interaction with objects and the
environment around them.
The attitude of the virtual recruiter may be also expressed through non-verbal signals
(such as smiles) but also through the dynamics of the behaviour. The global behaviour
tendency (e.g. the body energy), the combination and sequencing of behaviours (e.g. or-
der of display of smile and head aversion) as well as the mimicry of the the youngster’s
behaviour may convey different attitudes [KH10]. In order to identify the non-verbal
behaviours that the virtual recruiter may express to display different attitudes, we are
analyzing the videos that were recorded at Mission Locale (Section 3.4). For this pur-
pose, annotations have been added to consider the interaction context (e.g. turn-taking
and conversation topic), the non-verbal behaviour of the recruiter (e.g. gestures, facial
expressions, posture) and his expressed attitudes (friendliness of dominance using a
bi-dimensional scale based on Argyle’s attitude dimensions [Arg88]). In order to give
the capability to the virtual recruiter to display attitudes, we are extracting patterns
and characteristics of behaviours associated with different levels of friendliness and
dominance.
Fig. 7. Our two different game settings: (top) flipchart/presentation scene with our male character
and (bottom) our office desk environment with female recruiter
glasses, clothing and age. This is important as the practitioners will have some con-
trol over the virtual recruiter and they will ensure that it is appropriate the individual
scenarios. Examples of the TARDIS game assets can be seen in Figure 7.
– Menu mode: Here, the user is presented with a menu and can select whether to mod-
ify settings or practice or play the game. There is also an option for the practitioner
to enter edit mode.
– Settings/Options mode: Here, the user can select the level of difficulty at which
they wish to play the game in, as well as set sound/visual options and check that
their equipment is working adequately.
– Practice mode: Here, the user can practice a virtual interview, which does not count
towards their progress. This mode will include a virtual interview as in the Play
mode, but will allow for a “Pause” function, and will also include visual feedback
to inform if their actions have been read by the system.
– Play mode: Here, they will be able to select which position they wish to interview
for and take part in the virtual interview.
– Edit mode: This opens the editor for the practitioners, where they can modify in-
terview scenarios to tailor individual needs of the user or job descriptions. They
will also have access to change some dialog of the interaction, such as small-talk.
This will result in a more varied experience for the youngster, but will not affect
the expectations of the scenario.
as well as a final score when the interview comes to an end, we will provide the player
with a breakdown of how well they performed over a number of relevant factors. Phys-
ical parameters such as body or hand movement, head gaze or the evenness of the vocal
intonation can be obtained from our SSI and reported to the user. So, the youngster can
identify which aspects of their social interaction are satisfactory and they can acknowl-
edge other areas where they may need to focus on improving. We also plan to translate
the physical parameters calculated from the SSI in non-technical terms, such as ges-
tures, posture, vocal, head gaze, or even more abstract personality traits like openness
or professionalism. Then the user can be provided with further details about their per-
formance along with suggestions on how to improve. For example, “you have scored
xx% on comfort, maybe next time try to ensure that you keep your voice even and do
not fidget too much to appear more at ease”. Any such feedback will be provided from
a source outside of the virtual recruiter.
After each Game Session, the users receive feedback regarding their performance using
our NOnVerbal behaviour Analysis tool (NovA) [BDL+ 13]. It enables the playback of
the audiovisual recording of the interaction and visualizes recognized social cues, as
well as continuous data streams in several modalities, such as graphs, time line tracks,
heat maps, pie- and bar charts and others. Typically, different kinds of behaviours are
coded on different parallel tracks so that their temporal relationships are clearly visible.
Additionally, a series of statistics regarding the user’s performance is automatically
computed which is meant to help users track their improvements over time.
The analysis tool is designed to help youngsters reflect on their behaviour as well as
to support the practitioners in identifying problems in the youngsters behaviour. The de-
sign of the tool has been informed during a workshop conducted in association with the
practitioners at Mission Locale Goussainville. During the workshop, several key design
points have been discussed. For instance, the practitioners pointed out that the ability of
reviewing the performance of the user in parallel to the performance of the virtual agent
is critical to interpreting the user’s behaviours correctly. The discussions also yielded
that it is equally important that the tool gives the practitioners information regarding
when a certain behaviour of the user happened relative to the scenario’s progress.
5 Conclusion
The originality of the TARDIS platform is threefold. First, we consider a participatory
design for both the game and the simulator architecture. Workshops with practition-
ers and mock interviews with youngsters facing either human beings or virtual agents
was used to define the expectations of the system. Second, the system is able to detect
in real-time user’s emotions and social attitudes through voice and facial expression
recognition, and to adapt the progress of the game and the virtual interlocutor’s be-
haviour to the individual users. Third, TARDIS gives practitioners new instruments to
measure individuals’ progress in being able to self-regulate emotionally and their social
skill acquisition, thus facilitating reflection on their own practice and enabling a more
flexible and personalized coaching for young people at risk of social exclusion.
We are currently evaluating the first prototype of the project and working on the next
iteration of our opensource platform, implemented in C++ and Java using the Semaine6
framework with EmotionML7 messages.
Acknowledgement. This research was funded by the European Union Information So-
ciety and Media Seventh Framework Programme FP7-ICT-2011-7 under grant agree-
ment 288578.
References
[APD+ 09] Aylett, R., Paiva, A., Dias, J., Hall, L., Woods, S.: Affective agents for education
against bullying. In: Affective Information Processing, pp. 75–90. Springer, Hei-
delberg (2009)
[Arg88] Argyle, M.: Bodily Communication. University paperbacks, Methuen (1988)
[BDL+ 13] Baur, T., Damian, I., Lingenfelser, F., Wagner, J., André, E.: Nova: Automated
analysis of nonverbal signals in social interactions. In: Human Behavior Under-
standing (HBU) Workshop at 21st ACM International Conference on Multimedia
(2013)
[BDSHP12] Bevacqua, E., De Sevin, E., Hyniewska, S.J., Pelachaud, C.: A listener model: in-
troducing personality traits. Journal on Multimodal User Interfaces 6(1-2), 27–38
(2012)
6
http://www.semaine-project.eu
7
http://www.w3.org/TR/2010/WD-emotionml-20100729/
[BMNP07] Bevacqua, E., Mancini, M., Niewiadomski, R., Pelachaud, C.: An expressive eca
showing complex emotions. In: Proceedings of the AISB Annual Convention,
Newcastle, UK, pp. 208–216 (2007)
[BP02] Bynner, J., Parsons, S.: Social exclusion and the transition from school to work:
The case of young people not in education, employment, or training (neet). Journal
of Vocational Behavior 60(2), 289–309 (2002)
[CRKK06] Caridakis, G., Raouzaiou, A., Karapouzis, K., Kollias, S.: Synthesizing gesture ex-
pressivity based on real sequences. In: Workshop on Multimodal Corpora: from
Multimodal Behaviour Theories to Usable Models, LREC Conf., Genoa, Italy
(May 2006)
[DBA13] Damian, I., Baur, T., André, E.: Investigating social cue-based interaction in digital
learning games. In: Proceedings of the 8th International Conference on the Foun-
dations of Digital Games, SASDG (2013)
[DM99] DeGroot, T., Motowidlo, S.J.: Why visual and vocal interview cues can affect in-
terviewers’ judgments and predict job performance. Journal of Applied Psychol-
ogy 84(6), 986–993 (1999)
[dR13] Société Française de Radiotéléphonie. Mon entretien d’embauche game (June
2013)
[EDVDB10] Engelen, J., Dekelver, J., Van Den Bosch, W.: Incluso: Social software for the social
inclusion of marginalised youth. Social Media for Social Inclusion of Youth at
Risk 1(1), 11–20 (2010)
[EE12] Ennis, C., Egges, A.: Perception of complex emotional body language of a virtual
character. In: Kallmann, M., Bekris, K. (eds.) MIG 2012. LNCS, vol. 7660, pp.
112–121. Springer, Heidelberg (2012)
[GMK12] Gebhard, P., Mehlmann, G., Kippo, M.: Visual scenemaker - a tool for authoring
interactive virtual characters. Journal on Multimodal User Interfaces 6(1-2), 3–11
(2012)
[Gol06] Goleman, D.: Social intelligence: The new science of human relationships. Bantam
(2006)
[HCM+ 13] Hoque, M.E., Courgeon, M., Martin, J., Mutlu, B., Picard, R.W.: Mach: My auto-
mated conversation coach. In: Proc. ACM International Joint Conference on Per-
vasive and Ubiquitous Computing, UbiComp 2013 (to appear, 2013)
[HH10] Hareli, S., Hess, U.: What emotional reactions can tell us about the nature of oth-
ers: An appraisal perspective on person perception. Cognition & Emotion 24(1),
128–140 (2010)
[Ins13] Serious Games Insitute. ispectrum game (June 2013)
[JS13] Jones, H., Sabouret, N.: TARDIS - A simulation platform with an affective virtual
recruiter for job interviews. In: IDGEI (Intelligent Digital Games for Empower-
ment and Inclusion) (2013)
[JYZ12] Jiao, Y., Yang, X., Zhang, J.J.: A serious game prototype for post-stroke rehabili-
tation using kinect. Journal of Game Amusement Society 4(1) (2012)
[KH10] Knapp, M.L., Hall, J.A.: Nonverbal Communication in Human Interaction. Cen-
gage Learning (2010)
[Les94] Leslie, A.M.: ToMM, ToBy, and agency: Core architecture and domain specificity.
In: Hirschfeld, L.A., Gelman, S.A. (eds.) Mapping the Mind Domain Specificity in
Cognition and Culture, ch. 5, pp. 119–148. Cambridge University Press (1994)
[MC05] Manske, M., Conati, C.: Modelling learning in an educational game. In: Proceeding
of the 2005 Conference on Artificial Intelligence in Education: Supporting Learn-
ing through Intelligent and Socially Informed Technology, pp. 411–418 (2005)
1 Introduction
*
Corresponding author.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 492–496, 2013.
© Springer International Publishing Switzerland 2013
cards, with their aim being to have the most advanced piece. In our game, a piece
corresponds to a plant and grids, to the succession phase of the plant. In other words,
the children play the role of plants in the simulation. We conducted an experimental
evaluation that revealed that our game effectively stimulated the interest of students
and supported their learning.
However, one drawback is that it is digital and is therefore played on a computer
screen; we found that the virtual world did not well approximate the real world. The
experimental evaluation suggested that students were not enough immersive in the
virtual world and making the virtual world more immersive would not only further
motivate the students but also enhance their further understanding. To immerse stu-
dents in the digital game, we focused on realizing operations via body movements.
Accordingly, we developed a new learning support system called “Human
SUGOROKU.” To make this game immersive, a full-body interaction interface was
realized by combining a human detector interface to measure a person’s movement
and the digital SUGOROKU game. In Human SUGOROKU, the students themselves
move on the board as pieces and play the digital game.
2 Human SUGOROKU
2.1 Structure
In “Human SUGOROKU”, people operate the pieces of the digital game by moving
on the grids. To realize this operation, technology to measure a people’s position and
to identify them in the room is necessary. A technology to measure people’s 3D posi-
tion through attached ultrasonic sensors has already been proposed [3]. Accordingly,
we used ultrasonic sensors as the human detector interface.
This system is composed of ultrasonic sensors, two computers, and a projector
(Figure 1). Receivers are placed on the ceiling and transmitters are attached to people
moving on the grids. A server computer is connected to the ultrasonic sensor. Another
client computer runs the digital game that is projected by the projector. Because the
transmitters have unique identifiers, the ultrasonic sensors can identify them. The 3D
position and ID measured by the ultrasonic sensors is sent over the network (Figure
2). Therefore, by setting transmitters that correspond to a type of plant piece in ad-
vance, we can understand the position of a learner and type of plant.
This game simulates the real forest area of Mt. Rokko in Japan. Figure 3 shows a
screenshot of the game. The plant pieces represent the characteristic plants that grow
on Mt. Rokko. The surrounding part is the grid area. There are event cards that disturb
or promote plant growth. Pieces move on the grids accordingly. The window visualiz-
es vegetation succession according to the progress in the game.
In the digital game, there are 3 types of plant pieces: vertebral, red pine, and Rubus
microphyllus. These plants have different growth rates and size. There are 6 types of
event cards: sunny, rainy, wild boar, insects, landslides, and felling. The extent of
disturbance depends on the characteristics of each event card, and they affect plant
breeding. Vegetation succession is expressed by the relative progress of each plant
piece. Players can visually understand the state of vegetation succession from the
visualization.
3 Experiment
No. of
Category
students
I had a real sense of being a plant because my own body was a piece on 16
the game board.
I could feel that I was in the game because the movements of my body 14
were linked to the movements of the plant on the computer screen.
It was good that I could learn various things about vegetation succession. 11
N = 35, includes multiple answers.
Table 3. Statement relating to the movement linking the students‘ body and the plant on the
computer screen
I: Please tell us what you liked about the ‘Human SUGOROKU’.
S2: Because I have a sensor on my head and the piece displayed on the screen
moves as I move, it was as if I were inside the game.
I: Interviewer, S2: Subject 2
4 Conclusion
The results of the evaluation indicated that many students were immersed in the vir-
tual world of vegetation succession by becoming a piece on the game board and hav-
ing their own movement linked to the movement of the piece displayed on the screen.
On the basis of these results, we can conclude that ‘Human SUGOROKU’ is effective
for facilitating students’ immersion in the virtual world of vegetation succession.
Acknowledgments. This research was partly supported by the Grants-in-Aid for
Scientific Research (B) (No. 24300290).
References
1. Deguchi, A., Inagaki, S., Kusunoki, F., Yamaguchi, E., Takeda, Y., Sugimoto, M.: Vegeta-
tion interaction game: Digital SUGOROKU of vegetation succession for children. In: Yang,
H.S., Malaka, R., Hoshino, J., Han, J.H. (eds.) ICEC 2010. LNCS, vol. 6243, pp. 493–495.
Springer, Heidelberg (2010)
2. Deguchi, A., Inagaki, S., Kusunoki, F., Yamaguchi, E., Takeda, Y., Sugimoto, M.: Devel-
opment and evaluation of a digital vegetation interaction game for children. In: Natkin, S.,
Dupire, J. (eds.) ICEC 2009. LNCS, vol. 5709, pp. 288–289. Springer, Heidelberg (2009)
3. Nishida, Y., Aizawa, H., Hori, T., Hoffman, N.H., Kanade, T., Kakikura, M.: 3D ultrasonic
tagging system for observing human activity. In: IEEE International Conference on Intelli-
gent Robots and Systems (IROS 2003), pp. 785–791 (2003)
1 Introduction
Among the many discussions within the realm of video games, one of the most widely
debated relates to the experience of playing games. Commonly used concepts to de-
scribe game experiences are immersion [1–3] and flow [4], for example. Other studies
[2, 5, 6] have developed specific models that characterize the nature of the gameplay
experience. We consider these examples do not fully grasp the extent of the gameplay
experience and our interpretation of its key actors: the video game, the player and the
multiple relations that result from their interplay and work towards creating, ideally, a
satisfying gameplay experience. Furthermore, there lacks a model that equally bal-
ances and characterizes the dynamic interaction described. The work presented here
seeks to fill the identified gap, by presenting a conceptual model that characterizes
our interpretation of the gameplay experience – a twofold experience, considering it
both the process and the outcome – and which attributes equal importance to the vid-
eo game and the player. Furthermore, we present results from a study steered towards
partially validating the Motivations and Expectations dimension of the model within a
specific context.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 497–500, 2013.
© Springer International Publishing Switzerland 2013
The model proposed here identifies the key characteristics that help construct the
gameplay experience. The model looks to equally balance the two elements we be-
lieve are essential in the construction of the gameplay experience – the video game
and the player – and is sustained on the initial premise that gameplay experiences
result from the interaction between a game and the player [2]. Fig. 1 represents the
proposed Gameplay Experience Model, including the video game and player ele-
ments, and respective supporting dimensions and characteristics.
Developed from a literature review analysis and focus group results, the model
considers the gameplay experience as a twofold experience: it is both an interactive
experience (process) and an emotional experience (outcome). During game play,
these experiences can influence each other and are shaped by the multiple characteris-
tics of the model, defining the outcome of the experience. Initially, the model consid-
ers a Gameplay Situation, characterized by an ‘ambient setting’ (e.g. time of day) and
a ‘platform setting’ (e.g. PC or console). The Video Game element incorporates four
dimensions: Mechanics (Goals, Rules, Rewards), Interface (Visuals, Audio, Input,
Feedback), and Narrative (is itself a dimension and characteristic). Connected to the
video game element is Consistency, a characteristic that refers to the balance estab-
lished between the three dimensions. The Player element incorporates three dimen-
sions: Motivations (Motivation, Attention, Effort, Time, Connection), Background
(Preferences, Ability, Knowledge) and Expectations (Expectations, Action, Control).
A study was carried out to demonstrate possible connections among dimensions and
characteristics of the model, focusing mainly on Motivations and Expectations. We
hypothesized that (i) changes in rules and visuals influence player expectations; and
(ii) changes in rules and visuals influence player motivations. Data was collected with
a Pre-Questionnaire (PQ) and a Gameplay Experience Questionnaire (GExQ), in-
cluding 27 items (measured on a 5-point Likert scale) and developed specifically to
measure the presence of each characteristic in participants’ interpretation of the expe-
rience. The game – ‘ReCylce’, a multi-player First-Person Shooter developed by
Moura [7] – consisted in three game maps with differences among them. Each map
M1 (base map), M2 (more energy loss) and M3 (smaller map) was played twice. Af-
ter the first three rounds (M11, M22, M33), players responded to the GExQ.
4 Preliminary Results
Data from the PQ and GExQ was processed from 40 participants with different play-
ing experiences. In order to measure the possible evolution of the Motivations and
Expectations dimension, multiple statistical tests were applied using SPSS.
Three Expectations dimension (ED) variables (EDM1, EDM2, and EDM3) were com-
puted based on four questionnaire items. Based on the mean scores results, there was
a small increase in Expectations from the first round M1 (M=3.556, SD=0.627) to the
second M2 (M=3.594, SD=0.627) and the third M3 (M=3.656, SD=0.757), suggest-
ing changes in rules (M2) and visuals (M3) reflected on players’ expectations. Three
Paired-Samples t-tests were applied to statistically compare the means for three va-
riables (EDM1, EDM2, EDM3). In the first test (M1/M2), t=-0.422, p=0.675, which is
not statistically significant. For the second test (M1/M3), t=-0.75, p=0.458, indicating
the differences in means for MDM1 and MDM3 are not statistically significant. In the
third test (M2/M3), t=-0.527, p=0.601, which is also not statistically significant. Con-
sidering our hypothesis (Hyp. 1) and study sample, there is no statistical evidence that
alterations in game rules (M2) or visuals (M3) have an effect on player expectations.
Three Motivations dimension (MD) variables were computed based on 10 question-
naire items. Based on the mean scores results, there was an evident increase in the
mean value of Motivations from the first round map M1 (M=3.633, SD=0.494) to the
second M2 (M=3.715, SD=0.540) and third M3 (M=4.010, SD=0.705), also suggest-
ing that the changes in game rules and visuals increased player motivations. Three
Paired-Samples t-tests were applied to compare the means for the three variables
(MDM1, MDM2, MDM3). In the first test (M1/M2), t=-1.159, p=0.253; which is not
statistically significant. In the second test (M1/M3), t=-3.798, p=0.000; indicating the
differences in the means for MDM1 and MDM3 are statistically significant. In the third
test (M2/M3), t=-2.899, p=0.006, which is also statistically significant. Considering
our hypothesis, there is statistical evidence to show that game visuals (M3) alone or
combined with rules (M2) influence player motivations. However, rules (M2) alone
do not influence motivations.
5 Conclusion
The work explored here presents a gameplay experience model proposal that equally
balances two key elements in our interpretation of the gameplay experience: the video
game and player, each considering several dimensions and respective characteristics.
In order to demonstrate the relation between multiple dimensions of the model, a
study was carried out with a video game and 40 participants. Based on collected data,
despite a visible increase in both players’ Expectations and Motivations across all
three game maps, these results do not have statistical significance with Expectations,
but are significant with Motivations in two statistical tests. Further studies showed no
statistically significant differences between genders in terms of Motivations, while
Expectations did significantly differ according to gender.
With this study we have looked to initially validate multiple dimensions of the
model. While initial results show some relation among the dimensions, further statis-
tical studies are recommended to continue looking into these relations. Also, the mod-
el can be further explored and its potential studied, namely looking into the ‘Interac-
tive Experience’ based on game metrics. Analyzing metrics can possibly provide
further insight into the questionnaire results and how they reflect in the multiple di-
mension variables and the emotional experience of the game.
Acknowledgments. Thanks to Labs.SAPO, Arnaldo Moura and Rui Rodrigues; thank
you to those that participated in the study. Our acknowledgment to the institution
‘FCT’ for funding this project PhD grant no. SFRH/BD/66527/2009.
References
1. Brown, E., Cairns, P.: A grounded investigation of game immersion, pp. 1297–1300 (2004),
doi:http://doi.acm.org/10.1145/985921.986048
2. Ermi, L., Mäyrä, F.: Fundamental Components of the Gameplay Experience: Analysing
Immersion (2005)
3. Jennett, C., Cox, A.L., Cairns, P., et al.: Measuring and defining the experience of immer-
sion in games. Int J. Hum. -Comput. Stud. 66, 641–661 (2008),
doi: http://dx.doi.org/10.1016/j.ijhcs.2008.04.004
4. Csíkszentmihályi, M.: Flow: The Psychology of Optimal Experience (1990)
5. Gámez, E.H.C., Cairns, P., Cox, A.L.: Assessing the Core Elements of the Gaming Expe-
rience 288 (2010)
6. Fernandez, A.: Fun Experience with Digital Games: a Model Proposition. In: Leino, O.,
Wirman, H., Fernandez, A. (eds.) Extending Experiences: Structure, Analysis and Design of
Computer Game Player Experience, pp. 181–190. Lapland University Press, Rovaniemi
(2008)
7. Moura, A., Roque, L.: ReCycle Report. Universidade de Coimbra (2011)
1 Introduction
Mobile technologies and interactive digital media allow to collaborate in a large
number of application domains and apply digital content in real life scenarios.
Technology-mediated games have greatly increased and many of them claim to
build on everyday experiences. However, many commercially available games
simply add a virtual dimension to an existing game concept. On the other hand,
advanced game prototypes often focus on the technological issues of the digital
augmentation of games. The design of handheld augmented reality game inter-
faces is a complex endeavor and effective solutions should significantly expand
the “magic circle of game play” [3]. This concept is often used as a short-hand
for a dedicated place in space and time where a game takes place. Players enter
the magic circle of a game and enter a new virtual world that separates them
from reality. It is not necessary to create this new world with digital means and
traditional games often use physical artifacts to create these worlds using cards,
tokens, boards, etc. Digital games are well known and in recent years the com-
bination of real and virtual elements, also known as mixed reality games, gained
much attention as they try to increase the game experience by using digital
technologies. Many of these approaches focus on pervasive gaming to provide a
ubiquitous gaming experience and thus offer a game play logic that is suitable
for the special requirements of pervasive gaming. Considering game play, such
games are more simple than traditional games (e.g. board games) taking into
account the location-based properties and the very special game concepts that
pervasive mixed reality games require [3]. In this work we developed an aug-
mented reality board game that is played with real tokens on a physical board.
To enhance the user experience of the “magic circle of the game” we apply a
smartphone as “magic window” to show individual information to each player.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 501–504, 2013.
c Springer International Publishing Switzerland 2013
Fig. 1. OUTLIVE real game play (left) and smartphone view (right)
The key objective of this project is the useful combination of a traditional board
game and a digital computer game and to combine the benefits of both gaming
experiences. At the beginning of each game, players find themselves stranded on
a tropical island. An active volcano in the centre of the island and a tribe of
natives threaten the players and lead them to try to escape the island as fast
as possible. To achieve this they have to collect several raw materials to build
a raft. Whoever escapes the island first wins the game. The game consists of
the following physical objects: a game board, a volcano mounted on a wheel
of fortune, and small tiles each representing a virtual player. Virtual objects
include animated players, raw material, the raft shown in different stages of its
building progress, and some animals moving around the island. A central game
design concept was to employ the mobile device as a “magic window”. The main
advantages of combining the board game concept with a mobile AR device is
that only the active player is able to see his/her resources and discover fields.
Other players do not know which raw materials the player has gathered. This
encourages players to explore the whole island and think of an efficient strategy
how to get their required items first.
The prototype runs on an Android smartphone, we currently use a Samsung
Note II running on Android 4.1. The required physical elements include a board
game and tokens. The virtual scene is realized in Unity3D V4 and the tracking
is implemented using the Unity plug-in Vuforia SDK v2.5. It uses the built-
in camera of the smartphone and takes physical objects such as game board
and tokens as tracking reference for virtual objects. Once the game board is
tracked the software sets its coordinate system in the centre of the volcano and
player movements are calculated and validated with regard to this global co-
ordinate system. The physical volcano placed in the centre of the game board
is also used as 3D tracking reference to trigger game events after the user has
spun it. The wheel consists of two small wooden hexagon plates. The top one
is mounted on the bottom plate via ball bearing, which allows spinning the top
plate independently from the bottom one. In the upper plate there are twelve
magnetic bolts screwed in which will react to a magnet attached under the bot-
tom plate. This results in the wheel stopping shortly after it is spun. After it
has stopped, the volcano will reveal one symbol representing an event occur-
ring in the game. While playing, the user holds the camera in his hands and
points the camera towards the game board. Looking at the display of the smart-
phone enables the player to look through a magic window. S/he is able to turn
the camera around the game board and to see the virtual objects such as the
parrot in a correct angle. The virtual object’s size is scaled according to the
camera’s distance to the tracking references, as it would be if the user moved
closer to real objects. Besides graphics, sound is arranged spatially in the virtual
environment. So while holding the camera close to the game board the ambi-
ent sounds of the wood become louder than while the camera is in the sky. If
the user holds the device close to the water image of the physical game board,
s/he can hear the sound of the waves. During a turn the user sees an interface
presenting all of his possible actions, his/her health and resources in addition to
the augmented camera image. The user is able to choose between the different
options. After s/he finishes her/his turn the device is handed to the next user.
4 Conclusion
We showed that it is possible to integrate a smartphone as integral part of a game
play experience. The prototype is fully functional and preliminary internal tests
indicate an interesting game play. However, in future work we need to evaluate
the OUTLIVE game using game evaluation questionnaires like GEQ.
References
1. Geiger, C., Stöcklein, J., Klompmaker, F., Fritze, R.: Development of an Augmented
Reality Game by Extending a 3D Authoring System. In: Proc. of the Interna-
tional Conference on Advances in Computer Entertainment Technology, ACE 2007,
Salzburg, Austria, June 13-15 (2007)
2. Huynh, D.-N.T., Raveendran, K., Xu, Y., Spreen, K., MacIntyre, B.: Art of defense:
a Collaborative Handheld Augmented Reality Board Game. In: Spencer, S.N. (ed.)
Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games (Sandbox
2009). ACM, New York (2009)
3. Montola, M.: A Ludological View on the Pervasive Mixed Reality Game Research
Paradigm. Personal Ubiquitous Computing 15(1) (January 2011)
4. Xu, Y., Mendenhall, S., Ha, V., Radu, I., MacIntyre, B.: Trade-Offs for Designing
Handheld Augmented Reality Game Interfaces. In: Workshop on Mixed Reality
Games at CSCW 2012, Seattle, Washington, February 11-15 (2012)
1 Introduction
When musicians play guitar or other instruments, they often come up with rhyth-
mic ideas. In this situation, they would like to write down their ideas quickly,
before they forget them. This can be difficult, however, especially when the ideas
involve complicated rhythm patterns. These can be made up of many different
sounds, such as electronic glitch sounds or ambient sounds, rather than con-
ventional drum set sounds. They can also include frequent changes of musical
parameters such as volume and pan.
Musicians often record their musical ideas using Digital Audio Workstations
(DAW). However, the use of conventional DAWs for recording such complicated
rhythm patterns often takes a lot of time. Recording with DAW needs 1) time
for searching and selecting sounds with the tone that they want to use, 2) time
for large inputs with a mouse or some other MIDI (Musical Instrument Digital
Interface) input devices, and 3) time for adjusting parameters for each sound. If
the rhythm pattern recording process takes too long, the original musical ideas
can be forgotten. This becomes more serious as the complexity of the ideas
increase.
There are some systems that allow users to record musical ideas quickly. Voice
Drummer[1] allows users to input rhythm patterns by voice percussion. In this
system, however, users cannot use more than two sounds, such as a bass drum
and a snare drum sounds, and they cannot input multiple sounds at the same
time. Therefore, it does not suit recording complicated rhythmic ideas. The
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 505–508, 2013.
c Springer International Publishing Switzerland 2013
Music Notepad[2] allows users to write down their musical ideas quickly by
entering music notation on the interface as they sketch music with pen and
paper. However, in order to use the system, users must be able to understand
and write music notation.
In this paper, we propose a system named Onomatrack, in which users can use
onomatopoeias to record their complicated rhythmic ideas quickly and easily.
2 Onomatrack
We propose Onomatrack which is a system to allow a user to record complicated
rhythmic ideas and to create the rhythm track by inputing onomatopoeias1 .
Onomatrack includes features to reduce the three time problems related to using
a DAW as identified in Introduction.
1) With Onomatrack, users can easily choose a specific tone by representing
it with an onomatopoeia. This feature allows users to save the time of searching
a certain sound with their desired tone in a database. Musicians often express a
rhythm pattern using onomatopoeias like “don-tan-don-don-tan....” In this case,
a musician would imitate a base drum sound with the onomatopoeia “don” and
a snare drum sound with “tan.” The use of an onomatopoeia to express a specific
sound is more natural and quicker than a conventional DAW.
Another value of using onomatopoeias is the ease of re-choosing or changing
of the tone. It is generally known that the vowel of an onomatopoeia has a
relationship with the pitch[3]. That is, a sound pitch changes from high to low
according to the change of the vowel of the onomatopoeia as follows: /i/ →
/e/ → /a/ → /u/ → /o/. Using this property of onomatopoeias, users can
easily re-choose or change the tone by changing the vowel of the onomatopoeia.
For example, if the user inputs a sound with a tone of an onomatopoeia
“dang,” but after the playback wants to change the tone to one with a lower
pitch, he or she can change the “dang” to “dong,” which is similar to the former
onomatopoeia, but has a different vowel making the pitch lower.
2) Users can input the sound by writing down the onomatopoeia of the tone
directly on the screen. This feature allows users to input their desired sound more
intuitively compared to conventional inputs using a mouse or some other MIDI
devices. They can record their rhythmic idea as quickly as drawing a “rough
sketch.” When people come up with an idea, they often take a note or draw a
rough sketch to remember it. With Onomatrack, users can draw such a “rough
sketch” as a rhythm track that can be played back.
3) Users can adjust the volume of the sound by changing the size of an ono-
matopoeia and adjust the panning of the sound by changing the vertical position
of an onomatopoeia on the screen. This feature allows users to adjust these pa-
rameters more quickly than with conventional DAWs.
We implemented a prototype of Onomatrack on an Apple iPad. R It allows
users to input sounds by drag-and-drop from a list of onomatopoeias. We chose
1
The use of words which sound like the noise they refer to. Japanese usually use a lot
of onomatopoeias.
this method so that we could test the effectiveness of point 1) described previ-
ously in this section for making complicated rhythm patterns. In this prototype,
users can select from 40 onomatopoeias. These onomatopoeias include electronic
glitch sounds, ambient sounds, and conventional drum set sounds.
Fig. 1 shows the prototype interface. (A) is an example of an onomatopoeia
icons that represents a sound with a specific tone. Users select their desired
onomatopoeia icon from (B) and drag-and-drop it to the field (C). The vertical
position of an onomatopoeia is related to the sound pan, which is volume balance
between left and right speakers. The field (C) is divided into five horizontal
regions. These regions represent the position of the sound in the stereo sound
field. If an onomatopoeia icon is placed in the top region, the sound for the
onomatopoeia appears in only the right channel. The second region represents a
45 rightward angle. If an onomatopoeia icon is placed here, the sound appears
at an angle of 45 to the listener. Other regions are similar.
Users can adjust the sound volume by changing the size of the onomatopoeia
icon. The onomatopoeia icon (D) shows an example of adjusting volume. The
size of an onomatopoeia icon is changed by dragging the adjustment handle (E).
When users want to check a rhythm track they have created, they can listen
to it by pushing the play back button (F), at which point the play head (G)
begins to move. By default, the rhythm track is played repeatedly. Users can
add new onomatopoeia icons while listening to the rhythm track.
3 Discussion
An empirical evaluation of our prototype was conducted to confirm the effective-
ness of inputting sounds with onomatopoeias for recording complicated rhythm
patterns. Participants were asked to make at least three rhythm tracks with the
prototype during a one week period. Then they were interviewed. All four par-
ticipants were skilled musical performers with experience using a DAW to make
sound track.
From the result of the interviews, it was found that the prototype did allow
users who wanted to make complicated rhythm patterns to express their ideas
more easily than when using conventional systems. In addition, it was found
that all participants were able to find their desired sound data easily from the
database with onomatopoeia.
On the other hand, some participants said that they were sometimes unable
to recognize which onomatopoeia they wanted to use among those that have a
similar sound image, such as “dan,” “den,” and “don.” This suggests that our
system should have the functionality to allow users to customize the relationships
between an onomatopoeia and a sound.
4 Conclusion
References
1. Nakano, T., Goto, M., Ogata, J., Hiraga, Y.: Voice Drummer: A Music Notation
Interface of Drum Sounds Using Voice Percussion Input. In: Proceedings of the ACM
Symposium on User Interface Software and Technology, pp. 49–50 (2005)
2. Forsberg, A.S., Dieterich, M., Zeleznik, R.C.: The Music Notepad. In: Proceedings
of the ACM Symposium on User Interface Software and Technology, pp. 203–210
(1998)
3. Hiyane, K., Sawabe, N., Iio, J.: Study of Spectrum Structure of Short-Time Sounds
and its Onomatopoeia Expression. Technical Report, IEICE, no. SP97-125, pp. 65–72
(1998) (in Japanese)
1 Introduction
We are interested in developing tools that support the enjoyable education and
performing music. Since so many digital musical instruments have been devel-
oped, The interface of electronic musical instrument has been evolved to new
types, such as gestural, wearable, computer vision, etc. From the imitation of
the classic musical instrument to a totally new one, these kinds of digital in-
struments provide new possibility for performance and education of music. To
learn and performing some piece of music, we generally use staff based score that
consists of five horizontal lines and notes. The staff originated from musically
annotated text, though the Gregorian Chants around the 12th to 13th centuries.
It is a basic and important literacy that not only professional musicians but also
beginners can use staff based score for learning, playing and composing.
We have developed an interactive system that enables users to write and
play their handwritten musical score[1](see Fig.1). In this study, we applied our
interactive technique to digital pen interaction and record disc, Gocen Record
which is a kind of interactive artwork.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 509–512, 2013.
c Springer International Publishing Switzerland 2013
2 Related Works
In the field of media art or tangible UI, many researchers reported kinds of
new musical instruments. Especially several research or works are focused on
enhancing comprehensibility for music performing and learning as a motif of the
music score.
Bottelo proposed Tangible user interface for music learning to help children
learning the music theory. He developed the software system that enables users
to create music by putting objects labeled optical marker on the table[2]. A user
can not only put objects but also rotate or extend objects to make change pitch
or length of the note.
Noteput by Jonas Friedemann Heuer[3] is an interactive music table based on
five line staff, which combines all three senses of hearing sight and touch to make
learning the classical notation of music for children and pupils easier and more
interesting. These kind of timeline based musical interface is already well known
by Golan Levin[4] , Toshio Iwai[5] and et al. Drawn by Liberman[6] presents a
whimsical scenario in which painted ink forms appear to come to life, rising off
the page and interacting with the very hands that drew them. Inspired by early
filmic lightning sketches, in which stop-motion animation techniques were used
to create the illusion of drawings escaping the page, drawn presents a modern
update: custom developed software alters a video signal in real-time, creating a
seamless, organic and even magical world of spontaneous and improvised per-
formance of hand and ink. Calligraphy is one of art form and it is the supreme
art form in China. So we could not separate writing and paper. This kind of
primitive and intrinsic interaction between human and paper is important for
our creative activity.
Tsandilas and et al. [7] focus on the creative use of paper in the music composi-
tion process and proposed Musink to provide composers with a smooth transition
between paper drawings and OpenMusic as a flexible music composition tool by
using the Anoto pen.
Our goal is to develop music performing and learning system that do not re-
quire mouses, keyboards and displays on a computer and design user interaction
between users and a paper or real object.
3 Gocen
Gocen is our project name which is derived from a Japanese word of musical
staffs. We have already developed scanning device type system shown[1]. Fur-
thermore, we created a digital pen based system and an interactive art work.
Fig. 2. Snapshot of Gocen Digital Pen which is consist of a digital pen system with a
computer
Fig. 3. Snapshot of Gocen Disc which is consist of musical notation engraved on record
disc, computer vision system and a turn table
References
1. Baba, T., Kikukawa, Y., Yoshiike, T., Suzuki, T., Shoji, R., Kushiyama, K., Aoki,
M.: Gocen: a handwritten notational interface for musical performance and learn-
ing music. In: ACM SIGGRAPH 2012 Emerging Technologies (SIGGRAPH 2012),
Article 9, 1 page. ACM, New York (2012),
http://doi.acm.org/10.1145/2343456.2343465, doi:10.1145/2343456.2343465
2. Bottello, L.: Tangible user interface for music learning (2010),
http://lorisbottello.it/ (last accessed, June 2013)
3. Heuer, J.F.: Noteput is an interactive music table with tangible notes, that combines
all three senses of hearing, sight and touch to make learning the classical notation
of music for children and pupils more easy and interesting,
http://www.jonasheuer.de/index.php/noteput/ (last accessed, June 2012)
4. Levin, G., Lieberman, Z.: Sounds from Shapes: Audiovisual Performance with Hand
Silhouette Contours in “The Manual Input Sessions”. In: Proceedings of NIME 2005,
Vancouver, BC, Canada, May 26-28 (2005)
5. Toshio, I.: Chess on the table (1997)
6. Zachary, L.D.: http://thesystemis.com/projects/drawn/ (last accessed, June
2013)
7. Tsandilas, T., Letondal, C., Mackay, W.E.: Musink: composing music through aug-
mented drawing. In: Proceedings of the 27th International Conference on Human
Factors in Computing Systems, CHI 2009, pp. 819–828. ACM, New York (2009)
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 513–515, 2013.
© Springer International Publishing Switzerland 2013
2 Expected Outcomes
The workshop will be an excellent opportunity of gathering experiences from both the
designing and the implementation perspectives, making community and discussing
current challenges and future perspectives.
Acknowledgements. This work has been partly financed by the Spanish Government
through the DGICYT contract TIN2011-24660.
References
1. Al Mahmud, A., Mubin, O., Shahid, S., Martens, J.B.: Designing and evaluating the table-
top game experience for senior citizens. In: 5th Nordic Conference on Human-Computer
Interaction (NordiCHI 2008), pp. 403–406. ACM, New York (2008)
2. Hinske, S., Langheinrich, M.: W41K: digitally augmenting traditional game environments.
In: Proceedings of the 3rd International Conference on Tangible and Embedded Interac-
tion, TEI 2009, pp. 99–106 (2009)
3. Iwata, T., Yamabe, T., Poloj, M., Nakajima, T.: Traditional games meet ICT: a case study
on go game augmentation. In: Proceedings of the Fourth International Conference on
Tangible, Embedded, and Embodied Interaction (TEI 2010), pp. 237–240. ACM, New
York (2010)
4. Kaltenbrunner, M.: reacTIVision and TUIO: a tangible tabletop toolkit. In: Proceedings of
the ACM International Conference on Interactive Tabletops and Surfaces, 2009, pp. 9–16.
ACM (November 2009)
5. Li, Y., Fontijn, W., Markopoulos, P.: A Tangible Tabletop Game Supporting Therapy of
Children with Cerebral Palsy. In: Markopoulos, P., de Ruyter, B., IJsselsteijn, W.A., Row-
land, D. (eds.) Fun and Games 2008. LNCS, vol. 5294, pp. 182–193. Springer, Heidelberg
(2008)
6. Marco, J., Cerezo, E., Baldassarri, S.: ToyVision: a toolkit for prototyping tabletop tangi-
ble games. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interac-
tive Computing Systems, pp. 71–80. ACM (June 2012)
7. Marco, J., Cerezo, E., Baldassarri, S., Mazzone, E., Read, J.: Bringing Tabletop Technolo-
gies to Kindergarten Children. In: 23rd BCS Conference on Human Computer Interaction,
Cambridge University, September 1-4, pp. 103–111. British Computer Society, Swinton
(2009) ISBN:978-1-60558-395-2
8. Microsoft surface, http://www.microsoft.com/surface/en/
us/default.aspx
9. NUI Group web, http://nuigroup.com
10. Schöning, J., Hook, J., Motamedi, N., Olivier, P., Echtler, F., Brandl, P., Muller, L., Dai-
ber, F., Hilliges, O., Löchtefeld, M., Roth, T., Schmidt, D., von Zadow, U.: Building Inter-
active Multi-touch Surfaces. JGT: Journal of Graphics Tools (2009)
11. Rogers, Y., Rodden, T.: Configuring spaces and surfaces to support collaborative interac-
tions. In: O’Hara, K., Perry, M., Churchill, E., Russell, D. (eds.) Public and Situated Dis-
plays, pp. 45–79. Kluwer Publishers (2004)
1 Introduction
This workshop’s purpose is to gain awareness of using all five of our senses for
creating multimedia artifacts such as in entertainment. We are interested in the
user’s emotional feedback from using the addition of touch, smell, and taste in
entertaining activities. The workshop organizers will present a background review
for each of the three senses and their key projects for each: RingU, a wearable
fashion interface for sending virtual hugs, “Digital Taste Interface”, a device that
transmits a sense of taste without chemicals, and ChatPerf, an accessory that ac-
tuates scent from a user’s mobile phone (Fig 1). We also welcome attendees to
discuss work of their own that exhibits one or a combination of these three senses
in a fun and novel way.
2 Touch
2.1 Background
Touch effects our emotions and social relationships with one another. Not only to
help us complete motor tasks, the sensation of touch also provides the scaffold upon
which we build our judgments and decisions of others [1]. While touch expressions
can be seen in game controllers for haptic feedback, few works explore the use of
touch for communicating emotions like love to the user. Some examples of this are
“Hug Over A Distance” [2], “Hug Shirt” [3], and “Huggy Pajama” [4].
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 516–518, 2013.
© Springer International Publishing Switzerland 2013
2.2 RingU
Touch expressions can be utilized in small artifacts that we can wear like fashion
accessories. RingU is our ring-shaped, wearable fashion system for sending hug mes-
sages tactilely via internet communication [5]. When paired with a user’s smartphone,
the ring can send a small squeeze to their loved one anywhere in the world, and the
recipient can feel the squeeze with a color coded light accompanying it on their ring.
By touching the top of their ring or by pressing a button within our app, users can
send the message to anyone in their friend list. With the system, two people can
communicate more intimately remotely than with plain text messaging.
3 Taste
3.1 Background
While the sense originally evolved from a primary need for survival to differentiate
edible food, taste is now reserved for pleasure in dining. Taste is vastly underexplored
in the paradigm of entertainment from one person to another. Taste will be important
as internet bandwidth increases to extend the internet to all of our senses. We explore
the research carried out to gauge response from chemically stimulating users using a
variety of tasting displays. “Food Simulator” [6], “Taste Screen” [7] and “Meta Coo-
kie” [8] will be reviewed. We also review Nakamura’s work [9] which closely paral-
lels our own with regards to electric taste.
4 Smell
4.1 Background
Smell is perhaps the most fundamental sense to all living organisms and was probably
the first sense developed. Smell evolved from finding food and mates, deciding
whether to flight or fight, and helped establish social hierarchies [12]. Like taste, it is
now reserved for pleasure such as perfuming or adding aroma to food. We present
some background on instruments of scent delivery such as early work with the “Smell
O Vision” [13] and “iSmell” [14], and experimental work such as “Virtual Cocoon”
[15] and “olfaction printer” [16].
4.2 ChatPerf
Koki Tsubouchi and his company in Tokyo have created Chatperf [17], a self con-
tained accessory that can be attached to a mobile phone device. Together with the
SDK, it can allow app developers to bring a new dimension of sensory enhancement
to their apps. The development cycle of ChatPerf will be explored as well as the
business model of product delivery to the consumers.
References
1. Morton, A.H., Schiff, W.: The Psychology of Touch. Lawrence Erlbaum Associates,
Hillsdale (1991)
2. Mueller, F., et al.: Hug over a distance. In: CHI 2005 (2005)
3. Hug shirt, http://www.cutecircuit.com/hug-shirt/
4. Teh, J.K.S., Cheok, A.D., Choi, Y., Fernando, C.L., Peris, R.L., Fernando, O.N.N.: Huggy
pajama: a parent and child hugging communication system. In: Proc. IDC 2009, pp. 290–291
(2009)
5. RingU, http://ringu.mixedrealitylab.org/
6. Kortum, P.: HCI Beyond the GUI: Design for Haptic, Speech, Olfactory and Other Non-
Traditional Interfaces, pp. 291–306. Morgan Kaufmann, Burlington (2008)
7. Maynes-aminzade, D.: Edible bits: seamless interfaces between people, data, and food. In:
Proceedings of the 2005 ACM Conference on Human Factors in Computing Systems, CHI
2005, Portland, OR, April 2-7, pp. 2207–2210 (2005)
8. Narumi, T., Kajinami, T., Tanikawa, T., Hirose, M.: Meta cookie. In: ACM Siggraph 2010
Emerging Technologies, Siggraph 2010, Los Angeles, CA, July 25-29, p. 143 (2010)
9. Nakamura, H., Miyashita, H.: Augmented gustation using electricity. In: Fourth Aug-
mented Human International Conference, AH 2011, Tokyo, Japan, March 12-14, p. 34
(2011)
10. Nimesha, R.: Digitally stimulating the sensation of taste through electrical and thermal
stimulation. Ph.D. dissertation, National University of Singapore, Singapore (2012)
11. Volta, A.: On the electricity excited by the mere contact of conducting substances of dif-
ference kinds. Abstracts of the Papers Printed in the Philosophical Transactions of the
Royal Society of London 1, 27–29 (1800)
12. Herz, R.: The Scent of Desire. HarperCollins, New York (2007)
13. A Sock in the Nose. Review, Behind the Great Wall. Time Magazine, 57 (December 21,
1959)
14. How internet odors will work, HowStuffWorks (January 5, 2012),
http://computer.howstuffworks.com (accessed November 20, 2012)
15. Derbyshire, D.: Revealed: the headset that will mimic all five senses and make the virtual
world as convincing as real life. Mail Online (March 5, 2009),
http://www.dailymail.co.uk (accessed November 19, 2012)
16. Bannai, Y., Noguchi, D., Okada, K., Sugimoto, S.: Ink jet olfactory display enabling in-
stantaneous switches of scents. In: Proceedings of the International Conference on Multi-
media, MM 2010, New York, pp. 301–310 (October 2010)
17. ChatPerf, http://chatperf.com/
1 Introduction
The current investigation explores a space between games and music with intent to
generate alternative play experience leveraging a common simulation. The method is
focused on rendering interactive media events or content dynamically generated or
transformed by data from a simulation, controlling both audible and visible attributes
of the media.
To design a system between games and music there are two criteria, one for
computation and one for sound design. Computational criteria are consulted for an
interaction method with sound, and then consulted again in relation to sound design to
elucidate a simulation’s salient features. This project compares the two criteria and
analyzes their relationship. What makes sound interactive, and what makes it relevant
to engagement? The structural model for this investigation uses a common simulation
as the sole data source coordinating sound and visual media.
The experimental system minimizes peripherals concerns such game rules or narrative
and focuses on implementing two requirements in sound design. The first requirement is
to establish a relationship between sound and simulation such that the temporal model of
sound synthesis is compatible to the temporal model of the simulation dynamics. The
second requirement is that the sound synthesis model corresponds to visual rendering of
the simulation. This correspondence requires understanding how the simulation
properties are visualized in the graphical display. Here the graphics and sounds are
rendered in parallel as both graphics and sounds receive data from the simulation. This
means visual media are not a source for sound control. The system is neither a musical
instrument nor a game, yet is playable and performable [1].
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 519–523, 2013.
© Springer International Publishing Switzerland 2013
to designate sound synthesis signal pathways and control data routing and gating. A
development and application environment is required to connect procedural sound
programming with simulation environments, graphics computation, interactive control
devices, and related processes.
their continuous movements. The result is a dense layer of simple and mostly uniform
sounds that change individually without regular tempo or rhythm.
“Literal” means there is a baseline for correspondence with visual features: each
agent produces an individual sound at a rate of ~1 Hz with duration of ~3ms.
Persistence in sound by repetition of small multiples is aligned with agent persistence
in the visual field. Give 10 or more agents the outcome of this granular approach is a
layered sound texture. The experiment in this approach is to observe whether audible
clusters emerge as features in the sound texture. Audible ranges in frequency (pitch)
and horizontal position (stereo panning) were tuned to optimize for this model.
Result: The Literal interpretation produces mixed results in terms of
correspondence with visual information. Agents’ density and distribution are echoed
in the granular sound texture, which intuitively reflects the swarm visualization. But
clusters formed clearly on the screen do not emerge distinctively in sonic clusters,
except when a cluster’s vertical distribution generates a narrow frequency range, such
that a pitch center can be heard. As the spatial area of a cluster increases the
corresponding sound is less differentiated from the swarm’s overall audible texture.
So while the sound texture provides suitable audible “wallpaper” for the swarm
experience, the engagement is not particular to many of the cluster transformations
that a player can generate in the visualization.
4 Future Work
Future work begins with user validation for generalization of the engagement models
developed for swarm simulation. In many applications playability with sound could
be greatly facilitated by libraries and interfaces for iterative design workflow coupled
a capacity to select from and develop interactive sound synthesis palettes. To extend
the engagement model beyond single users, currently under development is a peer-to-
peer mechanism for passing simulation control and sounds between mobile devices.
References
1. Choi, I., Bargar, R.: A Playable Evolutionary Interface for Performance and Social
Engagement. In: Camurri, A., Costa, C., Volpe, G. (eds.) INTETAIN 2011. LNICST,
vol. 78, pp. 170–182. Springer, Heidelberg (2012)
2. Sayama, H.: Decentralized Control and Interactive Design Methods for Large-Scale
Heterogeneous Self-organizing Swarms. In: Almeida e Costa, F., Rocha, L.M., Costa, E.,
Harvey, I., Coutinho, A. (eds.) ECAL 2007. LNCS (LNAI), vol. 4648, pp. 675–684.
Springer, Heidelberg (2007)
3. Reynolds, C.: Flocks, herds and schools: A distributed behavioral model. In: SIGGRAPH
1987: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive
Techniques, pp. 25–34. Association for Computing Machinery (1987)
4. Frassinetti, F., Bolognini, N., Làdavas, E.: Enhancement of visual perception by crossmodal
visuo-auditory interaction. Experimental Brain Research 147(3), 332–343 (2002)
Kim Dung Dang, Phuong Thao Pham, Ronan Champagnat, and Mourad Rabah
Abstract. The games are typical interactive applications where the system has
to react to user actions and behavior with respect to some predefined rules
established by the designer. The storytelling allows the interactive system to
unfold the scenario of the game story according to these inputs and constraints.
In order to improve system’s behavior, the scenario should be structured and the
system’s control should be validated. In this paper, we deal with these two
issues. We first show how to validate Interactive Storytelling (IS) control using
Linear Logic (LL). Then we present “situation-based” hierarchical scenario
structuring which allows the state space reduction.
1 Introduction
Video games, nowadays, are considered as one of the most popular media that has
important contribution to the edutainment domain. Indeed, [6] demonstrated that,
thanks to the effects of interactivity, video games brought a learning support that was
more efficient than the traditional supports (non-interactive media formats such as
text, oral presentation, video). However, the unfolding of a game (the unfolding of the
story corresponding to the game) and its level of interactivity are commonly thought
to be opposite [2]. The first relates to a designer’s control on the game s/he has
created as the second relates to a player's control on the game s/he is playing. In order
to deal with this opposition, we have proposed an approach based on a LL model (an
executable formal model) [1], which allows balancing these two controls.
The weakness of this approach is the explosion of state space for complex
scenarios. Furthermore, a game designer cannot plan all the possible actions that a
player can realize. To handle this issue, we propose to contextually structure the
application execution into interaction sequences, called “situations”. Each situation
corresponds to a contextual resource-centered sequence of activities/events, and it is
characterized by pre-conditions and post-conditions. That enables the system to
control the execution and to establish casual links between the situations. Thus, the
model confines actors’ interactions according to shared contexts allowing hierarchical
view of the scenario.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 524–527, 2013.
© Springer International Publishing Switzerland 2013
3 Situation-Based Scenario
In general, in an interactive system, the execution is composed of a succession of
activities and actions performed by the actors or a sequence of events affecting their
behavior. Hence, a story is defined as a set of partially ordered events that solve the
storytelling problem. That leads us to organize a scenario into a set of interaction
sequences called “situations” [5].
We have built a video game on the LRC story to which we have added multiple
options for the non-player and player characters to increase the unpredictability as
well as the interactivity of the game. The game scenario is modeled and validated by
the LL approach to guarantee the objectives that the game has to reach. In order to
reduce the state space explosion, we have also applied our situation-based structuring
on the created scenario by reorganizing it with “situation” blocks. The starting point
of this transformation comes from the fact that these two approaches are based on
common notions such as events/actions, states, state transitions... Our idea is to
reorganize the scenario graph by regrouping the events/actions, happening in a same
context and relating to each other as a chain of different interactions, into one same
situation. Therefore, we can eliminate the interaction sequences repeated in the graph
obtained after the initial modeling for the LL validation. Thus, we are able to avoid
the redundancy in the scenario graph. As a result, we have obtained a situation graph
with 20 nodes instead of the initial graph with 56 nodes in the LL approach.
5 Conclusion
In this paper, we addressed the IS control from the validation and structuring points of
view. We have tried to combine the user choices with the game designer’s intentions.
To this purpose, we use a LL model to express the causality of the story and the
distinction between player's choices and game controller's choices. LL is also well
suited to prove the correct scenario execution and properties such as the reachability
of predefined system states and the absence of deadlock.
However, the drawback of this approach is the state space explosion. In order to
deal with this problem, we propose a hierarchical model based on the notion of
“situation”. A “situation” encloses a set of interactions related to a shared context of
several actors. In addition, each situation contains a set of management components
that ensure a correct termination of the enclosed sequence. Compared to the state
diagram, the situation can be seen as a higher level block representing several initial
states and transitions. This approach allows us to reduce the state space by eliminating
the redundancy in the interaction sequences in the modeled scenario.
References
1. Dang, K.D., Champagnat, R., Augeraud, M.: Modeling of Interactive Storytelling and
Validation of Scenario by Means of Linear Logic. In: Aylett, R., Lim, M.Y., Louchart, S.,
Petta, P., Riedl, M. (eds.) ICIDS 2010. LNCS, vol. 6432, pp. 153–164. Springer, Heidelberg
(2010)
2. Juul, J.: A Clash Between Game and Narrative. In: Digital Arts and Culture Conference,
Bergen (1998)
3. Magerko, B.: Story Representation and Interactive Drama. In: 1st Artificial Intelligence and
Interactive Digital Entertainment Conference, Los Angeles, California (2005)
4. Mateas, M.: Interactive Drama, Art, and Artificial Intelligence. PhD Thesis, School of
Computer Science, Carnegie Mellon University (2002)
5. Trillaud, F., Pham, P.T., Rabah, M., Estraillier, P., Malki, J.: Online Distant Learning Using
Situation-based Scenario. In: CSEDU 2012 (2012)
6. Wong, W.L., Cuihua, S., Nocera, L., Carriazo, E., Tang, F., Bugga, S., Narayanan, H.,
Wang, H., Ritterfield, U.: Serious video game effectiveness. In: Proceedings of the ACE
2007 Conference, Salzburg (2007)
1 Introduction
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 528–531, 2013.
© Springer International Publishing Switzerland 2013
2 GlowSteps
In this section we present our design called GlowSteps. We describe the design and
the different interaction behaviors. Next, we discuss several recent user evaluations.
GlowSteps is a set of interactive tiles that responds with light output on input
measured by pressure sensors. If someone stands on a tile or touches it with their
hands, the lights react depending on the current interaction behavior. The tiles are
autonomous objects that together form a decentralized play environment. They can be
picked up and moved around. In this way, children can create their own play spaces.
GlowSteps is an improved version of an earlier prototype called FlowSteps [6].
The following scenario describes how children might play with GlowSteps:
Peter and Jessica are playing with GlowSteps. They place the tiles in a path and jump
on them one after another. The tiles light up in different colors. “We have to make
them all red!” shouts Jessica. She continues to jump on one tile until it is red. “No,”
says Peter, “I make a rainbow and you have to repeat it.” Suddenly, one tile lights up
in blue. “Did you see that?” asks Jessica. Peter nods. “I think we have to catch it!”
Together they run towards the tile with the blue light.
So far, three interaction behaviors have been designed for GlowSteps: Catch,
Create and Toggle. We will now describe these behaviors in more detail. Catch is
designed to encourage physical active play. One tile lights up in a certain color for a
short amount of time. In this time, players must step on that tile to ‘catch’ the light.
All tiles then flash to communicate that the light has been caught. If the players are
not quick enough, the light disappears and then appears again at another tile. Players
can prevent the light to come to a certain tile by stepping on that tile. In this way, they
‘block’ the tile. Create is focused on expression and creativity. This interaction
behavior is more reactive than active, i.e. the tiles themselves do not light up but they
react on the players stepping on them. When a player steps on a tile, this tile starts to
cycle through a range of colors (e.g. from red to blue to green to yellow to purple and
so on) and a player can ‘stop’ this by stepping down the tile again. This tile then
‘freezes’ in the last color. After a while, it fades out. In this way, players can ‘draw’
and create patterns, or they can come up with a sequence of colors that other players
need to repeat as in the scenario above. Lastly, Toggle is developed to stimulate both
competitive and cooperative play. All tiles light up in red, green or blue. By stepping
on a tile, this tile moves to the next color in the sequence (e.g. red, then green, then
blue and then red again). Players can decide to turn all tiles into one color; but if one
player is aiming for red and the other for blue, they have to compete against each
other. All three interaction behaviors are rather simple but have the potential to lead to
diverse forms of play.
with other children for as long as they wanted. During this day, all three different
interaction behaviors were tried out. Throughout the day, many children (and adults)
played enthusiastically with GlowSteps. The different interaction behaviors resulted
in various forms of play as stepping on the tiles, catching the light or playing
hopscotch.
We plan to do more, mostly formal, studies in the near future, in which we want to
evaluate the interaction behaviors in more detail and further improve them. Exploring
how children interact with our design gives us important insights that are extremely
valuable in our iterative design process.
The proposed installation for ACE 2013 will include GlowSteps with various
interaction behaviors. We aim on presenting the three interaction behaviors described
in this paper, although they might be improved depending on user evaluations
performed in the meantime. During the conference, participants can try out the
different versions, compare their experiences and give us feedback. Furthermore, we
hope to discuss our design approach focusing on decentralized systems and open-
ended play in more detail.
References
1. Acuff, D.S., Reiher, R.H.: What Kids Buy and Why; the Psychology of Marketing to Kids.
Free Press (1997)
2. Bekker, T., Sturm, J., Eggen, B.: Designing Playful Interactions for Social Interaction and
Physical Play. Personal and Ubiquitous Computing 14(5), 385–396 (2010)
3. van Essen, H., Rijnbout, P., de Graaf, M.: A Design Approach to Decentralized Interactive
Environments. In: Nijholt, A., Reidsma, D., Hondorp, H. (eds.) INTETAIN 2009. LNICST,
vol. 9, pp. 56–67. Springer, Heidelberg (2009)
4. Huizinga, J.: Homo Ludens: A Study of the Play Element in Culture. Beacon Press, Boston
(1955)
5. Rijnbout, P., de Valk, L., Vermeeren, A., Bekker, T., de Graaf, M., Schouten, B., Eggen, B.:
About Experience and Emergence – A Framework for Decentralized Interactive Play
Environments. In: 5th International Conference on Intelligent Technologies for Interactive
Entertainment (2013)
6. de Valk, L., Rijnbout, P., Bekker, T., Eggen, B., de Graaf, M., Schouten, B.: Designing for
Playful Experiences in Open-ended Intelligent Play Environments. In: IADIS International
Conference Games and Entertainment Technologies, pp. 3–10 (2012)
7. de Valk, L., Bekker, T., Eggen, B.: Leaving Room for Improvisation: Towards a Design
Approach for Open-ended Play. In: 12th International Conference on Interaction Design and
Children, pp. 92–101 (2013)
Mara Dionísio1,2, Duarte Teixeira1,2, Poan Shen1,2, Mario Dinis3, Monchu Chen1,
Nuno Nunes1, Valentina Nisi1, and José Paiva3
1
Madeira Interactive Technologies Institute, University of Madeira, Funchal, Portugal
{monchu,njn,valentina}@uma.pt
2
Entertainment Technology Center, Carnegie Mellon University, PA, USA
{mdionisi,dteixeira,poans}@andrew.cmu.edu
3
Eat&Travel, Lisboa, Portugal
{mario.dinis,jose.paiva}@eatandtravel.pt
Abstract. Modern society is moving towards a busier lifestyle, people have less
time to enjoy themselves despite the increasing leisure options. Also, many
countries around the world are facing a financial crisis, making people less wil-
ing to travel abroad. In this paper we describe Eat&Travel, an interactive solu-
tion that ties dining and traveling together, offering restaurants the possibility to
offer their customers an opportunity to enjoy a unique regional meal combined
with a virtual trip in a river, where they can appreciate the landscape and learn
more about the country and the culture, using an immersive CAVE environ-
ment, without the need to travel abroad.
1 Introduction
Today's society is moving towards a busier lifestyle despite of the available leisure
options. Hence, people look for options to recover the energy spent with their busy
lives, get rest and enjoy meaningful moments. One of the ways that people can do this
is by traveling. In order to travel, people need to be in good health, financially fit,
available time wise, and ultimately willing to explore unknown places. Having all
these factors working together is very hard, especially for families with children.
So people in order to fulfill this search for meaningful moments turn to their meal
breaks. With this in focus, we see a relevant opportunity to bring together the meal-
time with the desire of fleeing from stressful routine in seeking engaging activities
and unique environments, such as touristic destinations. Along goes the customer’s
desire to try out new things, novel experiences or new sources of stimulation.
In the current competitive economy, the restaurant industry seeks to provide me-
morable experiences, constant novelties, aiming for customers’ loyalty and satisfac-
tion. Studies indicate that the service’s atmosphere is an essential aspect to customer
satisfaction as well as determining repeated patronage [1, 2]. The physical set-up of a
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 532–535, 2013.
© Springer International Publishing Switzerland 2013
CAVE enables the development of immersive environments in which users can en-
gage with each other as well as with the images projected within the CAVE [3]. We
identified that little attention has been paid to the potential of CAVEs in creating a
new and memorable experience in traditional places such as restaurants.
Eat&Travel creates a solution that ties dining and traveling to offer restaurants, the
ability to provide their customers with the novelty of memorable experiences by al-
lowing them to travel the world while enjoying a unique menu.
2 Eat&Travel
The setting of the Eat&Travel restaurant consists of a “cruise ship” like dining space
divided in two different spaces: the waiting area and the dining area. The Waiting Area
is where the customers arrive and wait until the “trip” starts. The Dining Area is a space
enclosed by screens that replicates exactly an 180º view of the landscape through video
projections thus creating an immersive environment. It enables customers to visit off-
location destinations in an immersive journey through the rivers around the world. A
cruise ship in a river is the perfect setting since it provides a calm journey that can cross
many famous cities. Therefore becoming the ideal setting to enjoy a meal and sightsee-
ing. The menu is specifically designed with reference to the destination’s gastronomy
for a whole experience. Consequently, customers are led in a multi-sensory experience
where taste plays an important role in driving the experience into the visited location.
Additionally, this experience will enable customers to interact with their journey
by self-indulged curiosity with venue-specific information. The outcome is a perva-
sive learning process.
It's a known fact that every cruise ship must have a captain and that a captain uses a
log to record his traveling’s. Therefore, we designed the whole experience around
Captain Jack Jones and his Journal. Usually cruise ship captains are experience sai-
lors, who traveled around the world many times and certainly, in most of the cases,
have many experiences to share. The first contact that customers will have is in the
waiting area where the captain will welcome the guests and arouse the customer’s
curiosity about the trip. Once the captain is ready to embark, one of his assistants, a
waiter, leads the customers to the dining area. Along the trip Captain Jones will ap-
pear several times to highlight places of interests appearing on the panoramic screen.
Customers could also learn more about the surrounding area using a copy of his jour-
nal, a digital tablet, placed on every table in the restaurant.
In the journal, Captain Jones wrote interesting facts and stories about the places,
people, fauna, flora and food. His deep passion for food is well documented through
the journal in his meticulous notes and recipes. To support various needs from differ-
ent customers, the journal will provide customers with information they can relate to
(for a celiac customer it is important to know what contains gluten and the available
gluten-free options) and information they seek for (cooking process, food origin, his-
toric info about dishes, curiosities).
3 Implementation
For our prototyping purpose, we used the CAVE room in our lab for implementation
and testing. The prototyping platform consisted of a CAVE with four 135-inch
screens four projectors, a surround sound system, two LCD screens, a webcam and a
tablet. The set up enabled us to create a fully unified and immersive experience. It is
planned to implement the system in a real restaurant in the near future.
Each table will have a tablet displaying the Captain Jones' journal where the custom-
ers can learn more information related to the food and the journey by flipping through
the pages of the journal. Customers can also order the food and access the order and
to change at any time. Finally when done with their selections they can send the order
to kitchen and enjoy the trip while their food is being prepared.
In order to make the Captain visible to all customers for the entire dining area, four
LCD screens are to be placed in the four different corners of the room. Whenever the
Captain feels there something worth mentioning to the customers, he will appear on
the LCD screens and talk to the customers. All the systems are interconnected, in that
way to synchronize the projections with the captain appearing on the LCD screens
highlighting the points of interest so that the customers can see it at that precise mo-
ment in the projections.
Our findings highlight the success of our prototype. The Eat&Travel restaurant pro-
vided an immersive experience that engaged visitors to interact and explore. Visitors
felt immersed with our CAVE like environment highlighting the feeling on being on a
cruise ship. Customers also felt tempted to learn more about the food, reading and
sharing trivia information as well as the recipe. Having the captain appearing at pre-
determined moments in the trip to point out places of interest, such as castles and
monuments as well a fauna and flora helped to guide the visitors along the trip and to
built a meaningful connection between customers and the captain. We aim to continue
our work to test the Eat&Travel experience tested in a real restaurant, and to have a
menu specifically designed with reference to the destination gastronomy for a whole
experience. With this prototype, we achieved the goal to deliver an immersive and
unified experience. We believe that this platform can provide a significant breath of
fresh air to the current competitive restaurant industry.
References
1. Bitner, M.J.: Evaluating service encounters: the effects of physical surroundings and em-
ployee responses. Journal of Marketing 54 (1990)
2. Wakefield, K.L., Blodgett, J.G.: The importance of servicescapes in leisure service settings.
Journal of Services Marketing (1994)
3. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A.: SurroundScreen Projection-Based Virtual Real-
ity: The Design and Implementation of the CAVE. In: Proceedings of the ACM
SIGGRAPH 1993 Conference, pp. 135–142 (1993)
1 Introduction
It is difficult for deaf children to enjoy the contents of a puppet show as much as
children with normal hearing do. The reason for this is that some of the contents, the
characters’ dialogue, the sound effects, and the background music consist of aural
information. To help these children, we developed Interactive Puppet Theater (IPT)
[1], is a puppet-show system designed use by deaf children.
IPT has two features. One is a function regarding the presentation of dialogue
information, which supplements the characters’ dialogue as text. The other is a
function in which the audience participates in the theater, using physical movement.
In IPT, the story progresses as the actors perform the puppetry while the children
watch the show. What is different, though, is that the children can actually participate
in the puppet-show by acting as facilitators of the story. This interactive experience
amplifies their appreciation and promotes their understanding of the story’s content
and characters, thus allowing them to have an enjoyable viewing experience.
*
Corresponding author.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 536–539, 2013.
© Springer International Publishing Switzerland 2013
2 Design
and a cursor appear on the screen. Operating the cursor, the children choose one icon
they like that corresponds to the story’s development. The operation of the cursor is
effected via the measurement of physical movement by a range image sensor. We
selected Kinect as a range image sensor for two reasons: first, it is compact and easy
to carry and set up, and second, the depth sensor installed in the device is compatible
with IPT. This depth sensor is able to accurately sense human bodies in a dimly lit
room. Thus, this type of function is effective during the performance of IPT, because
the lights in the room are dimmed in order for the animation to be projected clearly.
When a person’s hand is moved toward Kinect, it detects his/her hand, making it
possible to trace the hand movements. Kinect traces these hand movements and
measures their three-dimensional positions.
3 Evaluation Experiment
Target: This study targeted 17 deaf students from the 3rd to 6th grade at the T
Prefectural School for the Deaf, Elementary School Division.
Task and Procedures: After the students viewed IPT, they were given a questionnaire
survey, whose aim was to collect their subjective evaluations of emotional and design
aspects of IPT. Items, which the questionnaire had, were about enjoyment of IPT and
the effectiveness of the textual dialogues. For each item on the survey, the following
5-point scale was used: “strongly agree,” “relatively agree,” “neither agree nor
disagree,” “relatively disagree,” and “strongly disagree.” We investigated the trends
of the responses obtained after separating them into two groups: positive responses
(“strongly agree” and “agree”) and neutral or negative responses (“neither agree nor
disagree,” “disagree,” and “strongly disagree”). The difference between the positive
and the neutral or negative evaluations was computed using Fisher’s exact test (1 × 2).
The results are summarized in Table 1.
The results for three survey items regarding emotional aspects of IPT (Table 1. 1-3)
and six items regarding design aspects (Table 1. 4-8) indicate that the participating
students gave more positive than neutral or negative responses. Significance tests
showed that the number of positive responses significantly exceeded the number of
neutral or negative responses.
Taken together, these results indicate that our system ensures that the text dialogue
information in IPT functions effectively and helpfully, enabling deaf children to
understand the story and enjoy their viewing experience via the dialogue balloons that
supplement the dialogue information.
Neutral
Positive
Negative
**
1.The puppet theater was fun. 16 1
**
2.The story was easy to understand. 17 0
3.I was focused on watching the puppet theater until the end. ** 16 1
**
4. I was able to read each dialogues in the balloons correctly. 15 2
5.I was able to identify quickly which dialogues belonged to the
15 2
apprentice (*character’s name) by looking at balloons. **
6.I was able to identify quickly which dialogues belonged to the
15 2
teacher (*character’s name) by looking at the balloons.**
7.I was able to understand the apprentice’s feelings by reading the
16 1
dialogues in balloons. **
8. I was able to understand the teacher’s feelings by reading the
16 1
dialogues in balloons. **
**
p < 0.01
4 Conclusion
In this study, the results obtained from our experiments showed that the function of
presenting dialogue information could be an effective way for deaf children to acquire
dialogue information as they view a puppet show, and assists them in having an
enjoyable viewing experience.
Acknowledgments. This research was supported by a JSPS the Grants-in-Aid for Scientific Research
(B) (No. 23300309).
References
1. Egusa, R., Wada, K., Namatame, M., Kusunoki, F., Mizoguchi, H., Inagaki, S.:
Development of an Interactive Puppet Show System for Hearing-Impaired People. In: The
Fourth International Conference on Creative Content Technologies, pp. 69–71 (2012)
2. Adachi, T., Goseki, M., Mizoguchi, H., Namatame, M., Kusunoki, F., Egusa, R., Inagaki,
S.: Puppet Theater System for Normal-Hearing and Hearing-Impaired People. In: Nijholt,
A., Romão, T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 461–464. Springer,
Heidelberg (2012)
3. Lambourne, A., Hewitt, J., Lyon, C., Warren, S.: Speech-Based Realtime Subtitling
Services. The International Journal of Speech Technology 7, 269–279 (2004)
Kjetil Falkenberg Hansen1 , Rumi Hiraga2 , Zheng Li1 , and Hua Wang1
1
KTH Royal Institute of Technology
Lindstedtsvägen 24, 10044 Stockholm, Sweden
kjetil@kth.se
2
Tsukuba University of Technology
4-3-15, Amakubo, Tsukuba 305-8520, Japan
Abstract. The Music Puzzle is a computer game for tablets and smart-
phones using sounds for the gameplay. Just like an original picture is
reconstructed from pieces with jigsaw puzzle, an original sound is re-
constructed from musical segments with Music Puzzle. Each segment is
distorted by shifting the pitch and equalization. To finish the game, the
user listens to each segment visualized as pieces on the screen, reorders
them, and corrects their pitch and equalization. The game has a possibil-
ity for deaf and hard of hearing people to improve their residual hearing
ability since the observation shows their concentrating the game with
sounds and preference for music.
1 Introduction
Our experiences tell that deaf and hard of hearing young people like to listen
to music, and they enjoy using smartphones. Also, we are aware of the well-
documented beneficial effects of music [5], and that it is possible to train one’s
hearing to recover sensory abilities and thus music appreciation (e.g. [3,4]). Re-
cently we conducted an experiment using an audio-based computer game called
the “Music Puzzle”, and preliminary results have been reported in [2]. The ex-
periment showed positive results concerning how much time persons with hearing
losses are willing to invest in playful hearing training, and that playing the game
potentially can improve sound perception and thus appreciation of music.
In this paper we will briefly describe the gameplay and the experiment set-up
and game settings. Also some of the more aesthetic considerations and choices
will be discussed. For experiment results, see [2]. The novelty of the work is how
we use music for gaming, and how we investigate differences between solving a
task with music stimuli, speech stimuli or the combination of speech and music.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 540–543, 2013.
c Springer International Publishing Switzerland 2013
distorted sound fragments. This was chosen for several reasons, but mainly it
can be argued that a familiar game can be easier to get started with. Our exper-
iment group, consisting of students with hearing losses, were not used to audio
based games and therefore we chose to keep the task simple, but challenging.
In a jigsaw, the puzzle is the complete picture; traditionally a well-known
image divided into many small pieces. We use a sound recording divided into
several small sound segments. The user can start by listening to the full recording
or even start without knowing the target sound; the latter could be compared
to laying a puzzle without looking at the picture on the box. The puzzle thus
becomes more of a riddle or secret message to be deciphered. The pieces of the
jigsaw must be hooked together correctly in two dimensions with fitting forms,
but in our case the pieces must be ordered temporally. To introduce a further
challenge than just finding the correct order in one dimension, each piece is
randomly changed in pitch and tone quality (hi-pass and lo-pass filtering). This
makes it more complex than a jigsaw puzzle while keeping the concept.
To solve the puzzle the player has to listen very carefully to each sound frag-
ment, choose correct audio filtering and reorder the pieces. This task has proved
to be quite complex involving focussing on details in the sounds to get it right,
which corresponded perfectly with our ambition of making a motivational hear-
ing training game based on sounds, speech and music from everyday life.
The game was written for smartphones with Android 2.2 or above, and later
including other Android devices. Audio programming was done in Pure Data (Pd)
with the libpd library for mobile systems. Pd is a programming environment for
audio and has the necessary functions for pitch shifting and filtering.
(a) (b)
(c) (d)
Fig. 1. The user interface. (a-b) The user sets the game mode by selecting between
four sets of sounds and then between three puzzle types. (c) The user initiates the
game by listening to the sound then shaking the tablet. (d) The game starts with the
pieces randomly positioned on the screen.
automatically correcting pitch and filtering, found at the left bottom corner in
Figure 1(d), and also use four buttons on the top row of the screen to evaluate
a game session, play the original sound, play the current sequence as ordered on
the screen, or dismiss the game.
The difficulty level is determined by several parameters: the length of the sound
file, the number of pieces possible to create from shaking, the range of the pitch
shift, the equalization’s filtering settings, and not least the sound itself. These
parameters impact greatly on each other in a complex way. For instance, a game
consisting of more pieces is not always harder than a game with fewer pieces, and
likewise a short sound file is not always harder than a longer one. Also, the change
of pitch can make two pieces that in the original sounded differently appear very
similar. Furthermore, symmetry effects caused by the division (for instance when
dividing a 4-bar music segment into 2 or 4 pieces) or silent pieces (due to gaps
in the original sound) file can make the task unexpectedly demanding.
Sounds are placed in folders, and settings are defined in a text file. Any sound
recording can be used, depending on the experiment. In our experiments in [2]
we have used read poetry, instrumental music and a combination of the reading
and the music piece (A, B, A+B). In all, we had four sets of each which were not
particularly different. In the earlier experiments [1] we used speech and music
recordings of more varied character. Our experience from selecting the stimuli is
that before dividing the sound, it is hard to know how difficult the task will be,
and this demands that each game and sound should be tested for all the possible
number of pieces. Sounds that seem overly easy, such as a read count-down of
the numbers 5–4–3–2–1, can still be sufficiently challenging.
References
1. Hansen, K.F., Li, Z., Wang, H.: A music puzzle game application for engaging in
active listening. In: SIG Technical Reports: Proceedings of 97th Information Science
and Music (SIGMUS) Research Conference, vol. (7), pp. 1–4 (electronic). Informa-
tion Processing Society of Japan, Tokyo (2012)
2. Hiraga, R., Hansen, K.F.: Sound preferences of persons with hearing loss playing an
audio-based computer game. In: Proceedings of the IMMPD and ACM Multimedia
2013, Barcelona, Spain. ACM Press (in press, 2013)
3. Kraus, N., Skoe, E., Parbery-Clark, A., Ashley, R.: Experience-induced malleability
in neural encoding of pitch, timbre, and timing. Annals of the New York Academy
of Sciences 1169, 543–557 (2009)
4. Särkämö, T., Pihko, E., Laitinen, S., Forsblom, A., Soinila, S., Mikkonen, M., Autti,
T., Silvennoinen, H.M., Erkkil, J., Laine, M., Peretz, I., Hietanen, M., Tervaniemi,
M.: Music and speech listening enhance the recovery of early sensory processing
after stroke. Journal of Cognition and Neuroscience 22(12), 2716–2727 (2010)
5. UK Department for Education. Annex 3: The benefits of music (academic literature
review). In: Department for Culture, Media and Sport (ed.) The Importance of
Music. A National Plan for Music Education, pp. 42–44. Crown Copyright (2011)
1 Introduction
With network-enabled household appliances now a reality, ubiquitous computing
research to enhance the convenience and comfort of everyday life in the home
has become very active. To date, most of the research has been focused on
the living room and kitchen, where electrical appliances are easily installed.
The focus of our research is to bring various ubiquitous computing technologies
into bathrooms and to enhance the everyday act of bathing so that it is more
entertaining. In this paper, we present a number of entertainment applications
for the bathroom that are controlled via embedded touch sensors in the bathtub.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 544–547, 2013.
c Springer International Publishing Switzerland 2013
unit elements, for instance wall, floor, ceiling, bathtub, and are relatively easy
to assemble and remove. The space on the inside of the side of the bathtub can
be accessed by removing a side panel; resulting in easy installation of capacitive
touch-sensors, shown in the picture on the left side of Fig. 1. This space in the
side of the bathtub was designed specifically for additional equipment such as a
Jacuzzi. The picture on the left side of Fig. 1 also shows several electrodes on
the upper inside edge of the bathtub, and a sensor box containing a touch sensor
controller board. This arrangement means that TubTouch can be installed as an
additional system in any such existing bathroom. In addition, electrodes can be
freely installed on the rear side of surfaces, including curved surfaces. Another
advantage is the flexibility of the interactive display and its compatibility with
conventional household environments.
Video Projector
Ceiling
Electrodes
Projected Screen
Capacitive sensors usually respond to contact with water and are therefore
used to measure water levels in tanks. Hence, recent multi-touch input devices
tend to be incompatible with wet environments. However, the basic function of a
capacitive sensor is to react to the presence of dielectric objects. Since water and
the human body have different relative permittivities, TubTouch can indeed be
used to detect human touch, even when wet, in response to each sensor signal.
Japanese bathroom systems also have space in the ceiling to install equipment
such as ventilators, dryers, mist generators, loudspeakers, and audio units. In the
bathroom ceiling, an access hatch is provided for easy access to this space; thus,
a projector can very easily be installed there.
There are three ways to interact with TubTouch: touching, sliding, and prox-
imity to the edge of the bathtub.1, 2 As mentioned above, the proximity value
is measured by reaction to the presence of dielectric objects, such as fingers and
hands in this case. Touch detection is a proximity state that can be determined
1
TubTouch Example 1: http://www.youtube.com/watch?v=lDKR6rTwobM
2
TubTouch Example 2: http://www.youtube.com/watch?v=oiKocZ1IORw
3.1 Bathtuboom
Bathtuboom, shown in Fig. 2(a), is a kind of interactive art system. Each colored
ball projected on the edge of the bathtub is a button that is able to activate
sound phrases and move light shapes on the top of the bathtub. When a bather
touches these buttons simultaneously, the overlapped sound phrases generate
music that can be listened to. Bathers, especially kids who dislike bathing, can
experience some amount of pleasure by touching these balls and listening to the
resulting music.
3.2 Batheremin
Batheremin, shown in Fig. 2(b), is a theremin application. The theremin is a
very famous electronic musical instrument that is controlled by the proximity of
the two hands via capacitive sensors. We designed a bathtub embedded touch
sensor system as a theremin that can be played using two hands.
3.3 BathCount
When taking a bath, many Japanese children play a game, while in the tub of
water, in which they count from one to a few tens until the end of the bath. Chil-
dren learn numbers and counting through these experiences with their parent(s)
in the bathroom. BathCount, shown in Fig. 2(c), is a kind of support system
for this counting experience. When a button is touched, BathCount displays the
number on the bathtub, and speaks the number or plays some sounds. Children
using this system can count numbers with/without a parent, resulting in more
fun at bathtime.
3.4 Bathcratch
Bathcratch[1], shown in Fig. 2(d), is a DJ scratching entertainment system that
partially utilizes TubTouch. Using this application, a bather can scratch like a
DJ by rubbing the edge of the bathtub with a hand, and touching TubTouch
buttons to select rhythm tracks, scratching sounds, and phrases.
4 Conclusion
In this paper, we presented the TubTouch bathroom system, which enables a
bathtub to become an interactive controller using embedded capacitive touch-
sensors, and a number of new entertainment applications associated with it. We
described four of these applications in detail. The TubTouch system and these
applications have been exhibited and demonstrated at several exhibitions and
conferences. The general feedback from people who experimented with the appli-
cations has been very positive; they usually state a desire to have the TubTouch
system and its applications in their own homes.
As a future work, we plan to develop middleware for TubTouch in order to
make applications easier to develop. The middleware will use the TUIO protocol
[2] to divide TubTouch into hardware and software platforms. We hope to spread
this technology, and also hope that more and varied entertainment applications
will be developed for both kids and adults to enhance bathing activities and
make the act of bathing more fun.
References
1. Hirai, S., Sakakibara, Y., Hayakawa, S.: Bathcratch: Touch and sound-based DJ
controller implemented on a bathtub. In: Nijholt, A., Romão, T., Reidsma, D. (eds.)
ACE 2012. LNCS, vol. 7624, pp. 44–56. Springer, Heidelberg (2012)
2. Kaltenbrunner, M.: reacTIVision and TUIO: a tangible tabletop toolkit. In: Proc.
of ITS 2009, pp. 9–16 (2009)
1 Introduction
Several studies have suggested that the impression of food can be affected by “rendi-
tion”, i.e., the surrounding environment such as the appearance of the food and the
dish, not just by its taste [1]. While most renditions are limited to visual effects, we
focused on the aural and haptic rendition of the eating environment. Some studies
have reported on the audio-haptic effects during biting or drinking [2] [3], but we
speculate that we first encounter food with our hands, where there is a room for new
type of audio-haptic rendition.
In this study, we focused on the sound and vibration of liquid being poured from a
Japanese Sake bottle as an audio-haptic rendition of liquid. Sake bottles are known for
their unique “glug” sound and vibration. We believe that these sounds and vibrations
affect the subjective impression of the liquid in the bottle. To examine this idea, we
propose a method that reproduces the vibration of pouring liquid from a Japanese
Sake bottle by measuring and modeling real vibrations.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 548–551, 2013.
© Springer International Publishing Switzerland 2013
2 Method
One of the ways to reprodu uce sensation caused by vibration is to create a modell by
measuring a vibration and reproducing it. Okamura et al. created a vibration moodel
that uses a decaying sinusooidal wave to reproduce the vibro-tactile sensation of ttap-
ping an object by measurin ng the real vibration and modifying reality-based vibrattion
parameters through a seriess of perceptual experiments [4]. We extended this methhod
to reproduce the vibration of
o pouring water.
We surmised that the vibration of water being poured is a function of tilt angle off the
Sake bottle. Fig. 1 shows th he measured vibration of a pouring-water bottle with a tilt
angle of 180° (i.e. the bottlee was upside down). We took a single group of waves frrom
the measured data, which resembled
r a decaying wave, and observed the spectrum m of
the wave using the Fast Fou urier Transform (FFT) (Fig. 2 (Left)).
We found two peaks in th he frequency spectrum. The higher peak was around 2500Hz,
which we speculated was caused by Helmholtz resonance [5]. The lower peak w was
around 20 - 40 Hz, which we w presumed was caused by water surface fluctuation. Innte-
restingly, while both frequeency peaks can be perceived aurally, both can also be pper-
ceived by two types of mechanoreceptors in the skin (the Pacinian corpuscle senses the
higher peak, and the Meissn ner corpuscle the lower peak) [6], which implies that thhese
peaks not only aurally but allso haptically characterize the pouring water experience.
We reproduced the wavee by the following model consisting of two decaying siinu-
soidal waves of different freequencies:
Q t ∑ exp sin 2 (1)
Here, A1 and A2 are thee wave amplitudes, B1 and B2 are the attenuation coeeffi-
quencies of the sinusoidal waves, and t is time. Parameters
cients, f1 and f2 are the freq
were determined by the least-squares method to fit the measurement result. T The
modeled waveform is also shown
s in Fig. 2 (Right).
Fig. 2. (Left) Spectrum density of a single wave at 180°, (Right) Red: Original single wave.
Blue: Modeled wave.
Fig. 3. Sake-b
bottle-shaped device with a vibro-tactile actuator
5 Conclusion
References
1. Ban, Y., Narumi, T., Tanikawa, T., Hirose, M.: Modifying an Identified Size of Objects
Handled with Two Fingers Using Pseudo-Haptic Effects. In: Joint Virtual Reality Confe-
rence of ICAT - EGVE - EuroVR 2012, pp. 1–8 (2012)
2. Zampini, M., Spence, C.: The Role of Auditory Cues in Modulating the Perceived Crispness
and Staleness of Potato Chips. J. Sensory Studies 19(5), 347–363 (2004)
3. Hashimoto, Y., Inami, M., Kajimoto, H.: Straw-like User Interface: Virtual experience of
the sensation of drinking using a straw. In: Ferre, M. (ed.) EuroHaptics 2008. LNCS,
vol. 5024, pp. 484–493. Springer, Heidelberg (2008)
4. Okamura, A.M., Cutkosky, M.R., Dennerlein, J.T.: Reality-Based Models for Vibration
Feedback in Virtual Environments. IEEE/ASME Trans. on Mechatronics 6(3), 245–252
(2001)
5. Selamet, A., Lee, I.: Helmholtz resonator with extended neck. J. Acoust. Soc. Am. 113(4),
1975–1985 (2003)
6. Kandel, E.R., Schwartz, J.H., Jessell, T.M.: Principles of Neural Science, 4th edn. McGraw-Hill
(2000)
1 Introduction
Interactive public displays are already widespread without being yet ubiquitous.
With advances in computer vision technologies and falling prices for displays
and sensor systems, it seems like a safe bet to assume considerable growth in
the upcoming years, in particular for publicity and related areas. Cameras or
other sensors enable proactive behavior of the display and foster implicit and
explicit interaction of users with it. We are currently studying ludic concepts for
public displays that will combine entertainment, storytelling and transmission
of information, based on Living Chernoff Faces that represent data as features of
an abstract face, e.g. the happiness of a mouth correlating to low unemployment
rates. This study shall contribute to an understanding of how emerging pro-
active informative or advertising public displays can be designed to joyfully
attract, engage and convince passers-by without being excessively intrusive.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 552–555, 2013.
c Springer International Publishing Switzerland 2013
Fig. 1. Left: Photo of the proactive avatar of our information kiosk during the shoe fair
of Milano in Fall 2012. Right: a screen shot of currently implemented Chernoff Faces,
with exemplary explanations.
2 Related Work
A starting point for our work has been our own experiences with avatars (an-
thropomorphic user interface agents) for public spaces, for instance the proactive
kiosk that we have created for attracting visitors at the 2012 Milano shoe fair
in the context of the Portuguese SHOEINOV project, cf. Fig. 1. There, we em-
ployed our own high-quality rendering technology together with a movement
detecting camera to drive a very realistic looking, proactive female avatar, aim-
ing to attract potential customers to the boot of our partner. The customers
could then interact with the avatar via a touch screen displaying the shoe cata-
log, but the expected intermediate climax of the interaction was only achieved
when a human representative of the manufacturer initiated the contact to the
curious visitor. As expected (cf. [2]), informal reports of our partner confirm the
high efficacy of the avatar for non-verbally attracting attention and generating
curiosity. Now we are trying to transfer this sensible usage scenario of the avatar
to other situations, and to improve the experience design of such interactive
digital publicity instruments.
Chernoff Faces (cf. [1]) are a way of representing multivariate data on abstract
faces, where facial features such as the shape of the mouth or of the eye brows
stand for the value of data, e.g. the curvature of a mouth that amounts to a
sad expression will stand for a poor average income per person of a certain
country. To the best of our knowledge, this is the first attempt at such a ludic
and pro-active transformation of Chernoff Faces.
Müller et al. [5] describe, along with other interaction design issues, how
passers-by need a motivation to pass from a distinct phase to the next phase,
while coming closer to and intensifying their interaction with a public display.
Our thoughts follow the lines drawn by these authors, but we are particularly
4 Conclusion
References
1. Chernoff, H.: The use of faces to represent points in k-dimensional space graphically.
Journal of the American Statistical Association 68, 361–368 (1973)
2. Christian, A.D., Avery, B.L.: Speak out and annoy someone: experience with in-
telligent kiosks. In: Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI 2000), pp. 313–320. ACM, New York (2000)
3. Iurgel, I.: From Another Point of View: Art-e-Fact. In: Göbel, S., Spierling, U.,
Hoffmann, A., Iurgel, I., Schneider, O., Dechau, J., Feix, A. (eds.) TIDSE 2004.
LNCS, vol. 3105, pp. 26–35. Springer, Heidelberg (2004)
4. Klesen, M., Kipp, M., Gebhard, P., Rist, T.: Staging Exhibitions: Methods and tools
for modelling narrative structure to produce interactive performances with virtual
actors. Special Issue of Virtual Reality on Storytelling in Virtual Environments
(2003)
5. Müller, J., Alt, F., Michelis, D., Schmidt, A.: Requirements and design space for
interactive public displays. In: Proceedings of the International Conference on Mul-
timedia (MM 2010), pp. 1285–1294. ACM, New York (2010)
6. Spaulding, E., Faste, H.: Design-driven narrative: using stories to prototype and
build immersive design worlds. In: Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI 2013), pp. 2843–2852. ACM, New York (2013)
Dongsik Jo, Daehwan Kim, Yongwan Kim, Ki-Hong Kim, and Gil-Haeng Lee
1 Introduction
Recently, augmented reality (AR) systems to overlay virtual objects on the real
world have been widely used for the purposes of location based service, and
many researchers have many efforts to study interaction design of AR objects to
support an intuitive sense of the user [3]. In particular, realistic operation about
the virtual character with context-aware interfaces of real-world environments
are also introduced with a view to offering a seamless environment beyond the
barrier between virtual space and real one [4]. In recent years, markerless-based
tracking with a depth sensor such as Microsoft Kinect have extensively increased,
which allowed user interaction to manipulate the virtual character with detection
of the users motion in real-time [2]. Sam et al. proposed an augmented reality
system to visualize virtual spiders that were controlled behavior and interactivity
using a depth camera [5]. Ha et al. presented a miniature AR system to help
exhibition visitors and to provide DigiLog experience by interactive storytelling.
Here, position of virtual objects was determined by 3D reconstructed models,
and AR objects were visualized with devices such as a phone and an e-book [7].
In this paper, we present a visualization method of virtual characters in rela-
tion to real situations such as miniature environments with wearing an optical
see-through HMD. Our method is capable of executing updated location of the
virtual character through real-time height information of the miniature topog-
raphy estimated by a depth camera. Specially, we combine the sensors among
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 556–559, 2013.
c Springer International Publishing Switzerland 2013
Fig. 1. Active plane detection to find the real desk as an interaction space: (a) real
environments of the desk; (b) normal map to extract using depth information; (c)
candidates among plane areas; and (d) the result to detect an active plane
Figure 1 shows the overall procedure of the active plane detection to find
an interaction space on the real desk using a depth camera. First, we calculate
the normal vectors of all pixels in a depth image by doing cross product of three
neighboring 3D point [1]. Second, we divide all depth pixels into spatially several
neighboring components which can be easily classified by gathering same normal
directions. In order to reduce the depth noise, we simply smooth each depth value
by meaning of around nine pixels before the division. Third, we select our active
plane as a component in the middle of the image as an interaction space. Fourth,
we take the 3D plane equation using randomly selected three points among given
N points obtained from the interaction space. Let P1=(X1 , Y1 , Z1 ), P2=(X2 ,
Y2 , Z2 ), and P3=(X3 , Y3 , Z3 ) be the three points. Three typical plane equations
can be described using P1, P2, and P3 as follows. Here, k represents the number
of a selected point.
aXk + bYk + cZk + d = 0 (1)
We solve these equations using Cramers rule. Last, we convert the xyz coordinate
for the active plane into the criteria coordinate by calculating the transformation
matrix T and multiplying all 3D pixel values P by the matrix.
Pc = T ∗ P (2)
second (FPS). Our system is built on a PC with a 3.07GHz Intel Core i7 CPU,
12 gigabytes of main memory, and a NVIDIA GeForce GTX 260 graphics chip.
References
1. Holz, D., Holzer, S., Rusu, R.B., Behnke, S.: Real-time plane segmentation using
RGB-D cameras. In: Röfer, T., Mayer, N.M., Savage, J., Saranlı, U. (eds.) RoboCup
2011. LNCS, vol. 7416, pp. 306–317. Springer, Heidelberg (2012)
2. Jo, D.S., Kim, Y.W., Cho, E.J., Kim, D.H., Kim, K.H., Lee, G.H.: Tracking and
Interaction Based on Hybrid Sensing for Virtual Environments. ETRI Journal 35(2),
356–359 (2013)
3. Hannah, S., Bruce, T., Rudi, V.: Tangible User Interaction Using Augmented Re-
ality. In: Proceedings of the Third Australasian Conference on User Interfaces, pp.
13–20 (2002)
4. Nadia, M.T., George, P.: Virtual Worlds and Augmented Reality in Cultural Her-
itage. In: Proceedings of the International Workshop on Recording, Modeling and
Visualization of Cultural Heritage (2005)
5. Sam, C.D., Andreas, D., Adrian, C.: An Interactive Augmented Reality System for
Exposure Treatment. In: Proceedings of the 11th IEEE International Symposium
on Mixed and Augmented Reality. IEEE Xplore Digital Library (2012)
6. Silicon micro display, http://www.siliconmicrodisplay.com
7. Ha, T.J., Kim, K.Y., Park, N.Y., Seo, S.C., Woo, W.T.: Miniature Alive: Augmented
reality based interactive digilog experience in miniature exhibition. In: Proceedings
of CHI 2012, pp. 1067–1070. ACM Press (2012)
1 Introduction
To see Mario [1], the star character of Nintendo’s Super Mario Brothers games,
jumping on bricks in a video display is no surprise. However, to see him jumping
on real blocks would be quite surprising. Such a feature in a game would enable
the player to interact more intuitively with the virtual characters and thus enjoy
the game with a higher sense of reality.
Aiming at more seamless spatial mixing of physical objects and virtual images,
we developed a system that provides a tangible and interactive interface for such
mixing. Our “Mid-air Augmented Reality Interaction with Objects (MARIO)”
system enables virtual images to be displayed at various positions and precise
depths in mid-air in combination with physical objects. A game character ap-
pears in mid-air and runs around and jump over the physical blocks which users
have arranged by hands (Fig.1).
With MARIO system, mid-air images can be displayed anywhere inside a rect-
angular parallelepiped space (35 cm (W)×30 cm (D)×25 cm (H)). A high sense
of reality is achieved due to three types of consistency between the mid-air im-
ages and physical objects: high-speed processing for temporal consistency, image
display at precise positions for geometrical consistency, and shadow projection
for optical consistency.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 560–563, 2013.
c Springer International Publishing Switzerland 2013
x
z
Fig. 1. Proposed MARIO system. Users can interact with physical objects and mid-air
images. A virtual character (Hiyoko, a chick) is displayed in mid-air on wooden blocks.
A coordinate system is defined as x for width, y for height, and z for depth.
2 Related Work
Kato and Naemura’s floating image display [2] forms a mid-air image on physical
objects that the user has arranged by hand. The use of a Fresnel lens and display
enables the resulting image to float and move in the vertical plane (the xy-plane)
but not in the horizontal one (the z-axis).
In contrast, Tablescape Plus [3] supports horizontal movement (in the zx-
plane) of visual images on a tabletop interface through the use of image projec-
tion. Users can move tiny stand-up screens on the table to interact with visual
images. However, the images are tethered to the screen surface, so that they
cannot “jump” vertically (the y-axis movement).
The Augmented Urban Model [4] realizes simple object detection through
the use of a depth sensor, enabling ordinary blocks to be used for a tabletop
application. However, the projected images are displayed only from the table
surface (in the zx-plane), not in mid-air.
In contrast to these previous systems, MARIO system enables the display
of images in mid-air that can freely move in both the horizontal and vertical
directions. A mid-air image is displayed in a volumetric area (the xyz-space)
through the combined use of a novel real imaging optics [5] and a linear actuator.
A depth sensor detects the positions and orientations of the physical objects, and
thus mid-air images are seamlessly mixed with the physical objects.
3 System Design
(a) (b)
cm (H)). The shapes of objects which enter this area are estimated from the
depth data. A thin and compact real imaging optics (AIP-350) is used for mid-
air imaging display. The imaging optics is fixed at a 45◦ angle to the table
surface. A 19-inch color display (S1903-TBK) is placed below the optics. A linear
motor actuator (RCP4-RA5C, maximum stroke: 30 cm) connected to the display
changes the imaging position with vertical movement. An LED projector (GP10)
is installed 65 cm above the table to project shadow covering the interaction
area. The coordinate systems of physical objects, mid-air images, and projected
shadows are aligned using manual calibration.
Fig. 3. A mid-air image moves in various depths (10 cm, 25 cm, 20 cm). This picture
is taken with a long exposure to express the movement (ISO100, f/36, 25 sec).
4 Implementation
An image of a Hiyoko character appears in mid-air to run across and jump over
wooden blocks arranged and stacked by the user as a physical terrain (Fig.1). The
character appears to move back and forth in different depths due to the change
of imaging position by the actuator (Fig.3). The movements and positions of the
character change in accordance with the shape of the terrain, so users can play
with Hiyoko by re-arranging the blocks.
5 Conclusion
In this paper, we present a new interactive system combining mid-air images and
physical objects. On the basis of its novelty of interaction, MARIO system was
selected for regular exhibition in Japan National Museum of Emerging Science
and Innovation starting from July 2013. During the exhibition, the system will
be evaluated in terms of mid-air image display and user interaction. Future work
includes more volumetric expression by using integral photography and light field
displays.
References
1. Nintendo, Super Mario Bros., http://mario.nintendo.com
2. Kato, N., Naemura, T.: Floating Image Display with Dual Imaging Optics for Mixed
Reality Environment. Trans. of the VRSJ 12(3), 323–329 (2007) (in Japanese)
3. Kakehi, Y., Naemura, T., Matsushita, M.: Tablescape Plus: Interactive Small-sized
Vertical Displays on a Horizontal Tabletop Display. In: The 2nd IEEE International
Workshop on Horizontal Interactive Human-Computer Systems, pp. 155–162 (2007)
4. Knecht, K., König, R.: Augmented urban model: Bridging the gap between virtual
and physical models to support urban design. In: The 11th International Conference
on Construction Applications of Virtual Reality, pp. 142–152 (2011)
5. Maekawa, S., Nitta, K., Matoba, O.: Transmissive optical imaging device with mi-
cromirror array. In: Proc. of SPIE, vol. 6392, pp. 63920E:1–8 (2006)
Abstract. This paper demonstrates that face-like structures are everywhere, and
can be detected automatically even with computers. Huge amount of satellite
images of the Earth, the Moon, and the Mars are explored and many interesting
face-like structure are detected. Throughout this fact, we believe that science
and technologies can alert people not to easily become an occultist.
1 Introduction
"Face-like" structures have been found in many provinces on the earth or other stars
[5][2]. They are rocks or other geographical structures that look like a human face.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 564–567, 2013.
© Springer International Publishing Switzerland 2013
Because it is quite rare for such shapes to be made without artificial efforts, people
tend to think that they derived from such as unidentified ancient civilizations or space
aliens.
One of the most popular examples of face-like structures is "Face on Mars" (Figure
1) [5]. Some people say that is an ancient ruin made by Martians, and other people
say that is an artifact made by human beings. This mysterious face has been inspired
many sci-fi stories [1][3], even after it turned out just an optical illusion by later
analyses in detail.
Our goal of this research is to demonstrate that such face-like structures are every-
where, and can be detected automatically even with computers. Throughout this fact, we
believe that science and technologies can alert people not to easily become an occultist.
2 Related Work
Onformative1 shows a similar idea in 2013, but our project is original as the upload date
of a Youtube video2 shows that we demonstrated our early result in 2010. In addition
our project includes exploration of not only the Earth, but also the Moon and Mars.
3 System Description
1
http://www.onformative.com/lab/googlefaces/
2
https://www.youtube.com/watch?v=ooKFEmE6JXQ
3
https://sites.google.com/site/geofaceproject/
4
We won the third place awarrd in the Tokyo local competition.
http://marproject.org/
Results
Figure 4 shows a detection example
e that looks like a famous alien called “Grey Alieens”
[9]. This structure is locatedd on the south pole of the moon [10]. We could find m many
face-like structures in the po
olar area because the incidence angle of sunlight was smaaller
in high latitude and inside of craters were mostly dark with shadows. Other examplees of
faces on the south and north h poles of the moon were shown in Figure 5.
Fig. 5. Face-likee structures on the south and north pole of the Moon
References
1. Hoagland, R.C.: The Mo onuments of Mars: A City on the Edge of Forever. Frog Boooks
(1987)
2. http://ryukyushimp po.jp/news/storyid-2333-storytopic-5.html
3. Asuka, A., Mikami, T.: Messiah,
M a Mystery of Lost Face on Mars. Gakusyu Kenkyuusya
(2002)
4. https://developers.google.com/maps/documentation/staticmaps s/
5. http://en.wikipedia.org/wiki/Cydonia_(region_of_Mars)
6. http://ultraist.ha atenablog.com/entry/20110718/1310965532
7. Huang, C., et al.: High-P
Performance Rotation Invariant Multiview Face Detection. IE
EEE
Trans. Pattern Anal. Mach
h. Intell. 29(4), 671–686 (2007)
8. http://libccv.org/
9. http://en.wikipedia.org/wiki/Grey_alien
10. http://wms.lroc.asu.edu/lroc_browse/view/SP_Mosaic
Maarten H. Lamers, Fons J. Verbeek, and Peter W.H. van der Putten
Many of today’s technology heroes and aficionados started their careers by what can
be considered as tinkering. Bill Gates and Steve Jobs are well-known examples,
surrounded by anecdotes of working in garages with small, enthusiastic teams,
supposedly working under playful conditions. For many they exemplify what
tinkering and grass-roots initiatives could lead to. But also the Wii Remote tinkering
projects by Johnny Chung Lee (www.johnnylee.net), shown on Youtube.com, have
captured and sparked the imagination of many.
At the level of today’s technology consumer, there appears to be an increasing
desire to interface our technological power-machines to the real physical world. And
power-machines they are, our personal computers, tablets and smart phones –
equipped with highly advanced man-machine interaction technologies,
communication possibilities, location-determining hardware, acceleration sensors, and
more. However, for all their strengths and possibilities, they do not offer the
connectivity to the physical world around us that many dream of. No smart phone is
currently on offer that drives itself around the house to play with the cat. No tablet is
equipped with motors and sensors that make it suitable to steer a child’s soap-box
cart. No current iPhone models have a user-accessible digital thermometer to play
with. And in a way, this is what we more-and-more expect our technology to do (well,
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 568–571, 2013.
© Springer International Publishing Switzerland 2013
perhaps not exactly this, but similar things) – to connect our computational devices to
the physical world.
This desire to connect may have been always present, but there appears to be more
of a push towards closing the gaps between human and technology, by leveraging
technology in a more personal, private and autonomous manner, under control of the
user.
As a result, tinkering with digital/physical computing systems has gained much
attention over the last few years. For example the Wiring (www.wiring.org.co) and
Arduino (www.arduino.cc) projects offer immensely popular tools for lower- to
intermediate-level software and hardware tinkerers (e.g. [1],[2]), spawning thousands
of interesting home-grown projects. Similar projects are Raspberry Pi
(www.raspberrypi.org), MaKey MaKey (www.makeymakey.com) and, more in the
creative coding domains, Processing (www.processing.org) and OpenFrameworks
(www.openframeworks.cc).
These initiatives gave rise to low-cost rapid prototyping tools that offer rich, if not
full functionality, while hiding complex underlying structures from the developer.
The frequent open-source nature of the projects kindles what is in essence a
community-like support structure, and the ongoing generation of example code and
libraries. All this makes it possible for single medium-skilled developers to master
complex (physical) digital prototyping tools.
Observation 1. In recent years, (physical) digital prototyping was fitted
to the scale of the individual. After years of increasing technological
complexity in the systems around us, the right combination of
technological abstraction and openness has re-enabled individuals to
understand, own and prototype solutions to their own problems and
aims.
2 Scientific Education
The adoption of digital/physical tinkering by individuals has had its effect on science
and education. Scientists increasingly use publicly available low-cost digital
prototyping systems to create measurement tools and other experimental devices (e.g.
[3]). To witness, a Google Scholar (scholar.google.com) query for articles containing
the word “Arduino” in their title, excluding legal document, patents and citations,
yielded a result of 490 scholarly articles1.
Naturally, developments in science and technology resonate in science education
(e.g. [4],[5]) and scientific education (e.g. [6],[7]), although not all experiences are
always positive. Tinkering is found in curricula worldwide, and students realized a
plethora of projects that are disseminated via the web.
An example of successful tinkering by academic students that stands out in our
opinion, is the Amplino project (www.amplino.org), in which students developed a
low-cost Arduino-based polymerase chain reaction (PCR) diagnostic tool for malaria.
1
Query result on September 2, 2013.
Naturally, not all student projects are as successful as the Amplino project. However,
they nonetheless have educational value.
It is the position of the authors that tinkering as a mode of knowledge production has
great value, as acknowledged by the theme of the 10th International Conference on
Advances in Computer Entertainment Technology (ACE 2013). Moreover we propose
that this value extends into the realm of scientific education. As argued above, particular
attention is required for implementing tinkering within research-oriented education. We
are not aware of any initiatives to collectively deal with this topic.
learn more about it. Participants share experiences, develop strategies and tackle
problems, with the explicit goal of consolidating what is known and what is desired,
and building a network for future collaboration.
Participants are encouraged to bring their own experiences and questions into the
discussions. To this end, applicants are invited to fill in a short questionnaire prior to
the workshop. After a plenary introduction and review of issues/questions posed by
participants, separate issues are dealt with by sub-groups within short consecutive
sessions. In a final plenary session, the results are aggregated and possible plans for
future collaboration among participants can be discussed. If possible, the organizers
will aggregate all material into (a) separate paper(s) to be published, consolidating the
workshop results.
References
1. Thompson, C.: Build It. Share It. Profit. Can Open Source Hardware Work? Wired
Magazine 16(11) (2008)
2. Banzi, M.: Getting Started with Arduino. Books & O’Reilly Media, Make (2008)
3. D’Ausilio, A.: Arduino: A Low-cost Multipurpose Lab Equipment. Behavior Research
Methods 44(2), 305–313 (2012)
4. Dougherty, D.: Learning by Making: American Kids Should be Building Rockets and
Robots, Not Taking Standardized Tests. Slate / Future Tense Article (June 4, 2012) (online
resource)
5. Gerstein, J.: The Flipped Classroom: The Full Picture for Tinkering and Maker Education.
Post on User Generated Education Blog (June 16, 2012) (online resource)
6. Brock, J.D., Bruce, R.F., Reiser, S.L.: Using Arduino for Introductory Programming
Courses. Journal of Computing Sciences in Colleges 25(2), 129–130 (2009)
7. Jamieson, P.: Arduino for Teaching Embedded Systems. Are Computer Scientists and
Engineering Educators Missing the Boat? In: Proc. FECS, pp. 289–294 (2010)
8. Doherty, P.C.: The Beginner’s Guide to Winning the Nobel Prize: Advice for Young
Scientists. Columbia University Press (2008)
Petri Lankoski
Södertörn University,
141 89 Huddinge, Sweden
petri.lankoski@sh.se
Abstract. This pilot study looks at how the formal features of character-
driven games can be used to explain player-character engagement. Ques-
tionnaire data (N=206), formal game features (in 11 games), and
ordinal regression were used in the analysis. The results show that inter-
active dialogue and cut-scenes showing the romances between the player-
character and another character relates to higher character engagement
scores, while romance modeling and friendship modeling relate to lower
character engagement scores.
1 Introduction
This is a pilot study that aims to isolate formal game features that contribute
toward player-character engagement. The focus of this study is on single-player
character-driven games. Lankoski’s theory [1] is used as the basis of the study. In
order to evaluate connection between games and engagement, the formal game
features are classified and then evaluated by using ordinal regression and self-
reported engagement scores.
Lankoski [1] argues that player-character engagement relates to goal-driven
engagement and empathic engagement. Here I focus only on empathic engage-
ment. Lankoski suggests that character engagement depends on how the charac-
ter is presented; the kind of access that the player has to the characters actions,
thoughts, and emotions; and how the player evaluates the character in terms of
morals and aesthetics. [1]
The following formal features where selected (c.f., [1]): dialogue vs interac-
tive dialogue, moral choices (no–yes), supporting different play styles (no–yes),
cut-scenes with romantically content – romance in cut-scenes, appearance cus-
tomization (no; possibility to change some aspects of the character, e.g., cloths,
hair style); customizing sex and appearance, character development (no, scripted
character development, player-guided character development), player character
dialogue is voice acted (no–yes), romance/friendship modeling (some modeling,
complex modeling as in Dragon Age: Origins)) and moral choices (no–yes).
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 572–575, 2013.
c Springer International Publishing Switzerland 2013
Most of these features relate to the kind of access the player has to the charac-
ter except for appearance customization, play styles, and character development
that relate more to the aesthetical and moral evaluation of the character (c.f.,
[1]). It is possible that this list does not include all the features that are relevant
to player-character engagement and, hence, the analysis below reveals only the
information on these features in relation to the feature set.
2 Method
Player-character engagement is measured by using a single 5-point Likert scale
question: I identified with my player-character (1: totally disagree, 5: totally
agree). Here the concept of identification is used in the questionnaire as I assumed
that the term is more familiar to players than that of engagement.
A non-proportional quota sampling method was used. The target was to set
to 20 answers for each game. In addition the target for female respondents was
set to 25%.
Respondents were gathered by advertising the study in Facebook, Twitter,
Google+, and pelilauta.fi as well as in two forums dedicated to Assassin Creed:
Brotherhood and Uncharted 2: Among Thieves whenever there was a need to
gather more answers about those games. In addition, to get more answers from
females, the study was advertised in a Finnish girl gamer forum.
The data was gathered by using two questionnaires. One-half of the games
was in the first questionnaire and the other half in the second questionnaire.
I selected popular, rather new games for the study to ensure that the sufficient
amount of answers were obtained. The games are listed in figure 1. The games
were played and judged base on whether they have the aforementioned formal
features.
The data were analyzed by using ordinal regression and mixed effect models.
R [2] and cumulative link mixed models (clmm) from the ordinal package [3]
were used. The Gauss-Hermite quadrature approximation with ten points was
used in the analysis. No structure (except that 1 < 2 < ... < 5) were assumed in
the ordinal scale. Subjects were modeled as a random effect.
Stepwise model selection by using Aikake’s Information Criterion (AIC) was
used. The model with the lowest AIC was selected, However, simpler model was
preferred when the models were not statistically different (by using the likelihood
ratio test).
3 Results
The total number of subjects in the study is 206. However, there can be overlap,
as the data was collected through two anonymous questionnaires. The mean age
of the respondents of the questionnaires is 28.49 (min = 13.00, max = 51.00).
68.9 % of the respondents are male and 31.1 % female. 66% of the respondents are
from Finland, 15 % from Sweden, 4 % from the US, and the rest from different
countries. 21.4 % of the respondents had obtained high school eduction, 5.5 %
*#
!# '
$"#(#
!"
!
% #
!" #%
#!#%
$#*""
!"
!#!
$
$
"#'"
$#,
!)!"
#
!+
!!")'!
""#+
$"&) $%$#
"""""!)!#!
#)!"'$
!#$#
!#+)%"
Fig. 1. Categorization of games. The games with perspective 1st* have a default of a
1st person perspective, but allow playing using a 3rd person perspective.
vocational, 23.3 % college, 24.0 % bachelor, 18.5 % masters, 6.0 % doctoral, and
1.1 % other education. Figure 1 shows the various features of the games.
The predictor variables of the optimal model are interactive dialogue, romance
modeling, friendship modeling, and romance in cut-scenes and subject is a ran-
dom effect.1
The optimal model is significantly better (p < .001) than the null model
containing only a random effect subject. This means that the data is explained
better with the optimal model than by assuming that the player preferences
would explain the data.
The random effect subject has a variance of 2.219 and a standard devia-
tion of 1.489. The strongest positive effect is by interactive dialogue (2.1274,
CI95 = 1.4789 − 2.7759). This means that the games having of interactive di-
alogue are estimated to have higher player engagement with their PCs. Show-
ing romantic engagement in the cut-scenes emph(romance in cut-scenes: 0.6932,
CI95 = 0.1553 − 1.2310) relates to higher scores in the terms of identification.
The effect is considerably smaller than the effect of interactive dialogue. The
games having romance (yes: -1.2650, CI95 = −1.9054 − −0.6245) or friendship
modeling (some: -1.4392, CI95 = −2.2342−−0.6444) have lower player-character
engagement than the games without those features. As an effect, limited romance
modeling (romance some: 0.0843, CI95 = −0.6242 − 0.7929) does not differ from
1
Two models with lower thhe AIC was rejected, because there are problems in the
models: Adding sex to the model decreases the AIC, but this model is not signifi-
cantly better (ΔAIC = .493, p = .1143) and the confidence interval for sex crosses
zero (males and females are not significantly different). Hence, a simpler model is
preferred. A model having a fixed effect with an interaction between table-top role
playing hobby, sex, and romance modeling is significantly better than the optimal
model (ΔAIC = 9.648, p=.0009), but the confidence intervals of the interaction
terms are very wide and cross zero. Hence, the model with interactions was rejected.
4 Discussion
As nonprobability sampling was used, the results are not directly generalizable by
using probability theory. However, the optimal model indicates that the different
backgrounds has no significant role in the results. However, the study includes
only 11 games. The implementation of formal features can have an impact on
the results. Finally, if some formal feature that is relevant to player-character
engagement is not included in the list of features used in the model selection,
the relevance of the feature cannot be evaluated (e.g., the set of games does not
contain pure 1st person games such as Half-Life). To conclude, I believe that the
results are somewhat generalizable to the population outside the sample, but it
is likely that the results are tied to the implementation of the formal features
within the games in this study.
The results indicate that interactive dialogue and showing romantic episodes
in the cut-scenes relate to a higher player-character engagement. Interactive di-
alogue in all the games in this study contains dialogue options that can be used
to present different types of personalities (e.g., in Dragon Age 2 one can select
from diplomatic/helpful, humorous/charming, and aggressive/direct lines). This
allows the players to modulate the character towards their preferences. This can
contribute towards positive evaluation of the character and a higher engage-
ment (c.f., Lankoski[1]). Surprisingly, romance and friendship modeling relate to
lower player-character engagement scores. This might relate to the quality of the
modeling.
In this study I am able to connect formal game features to the identification
self-evaluation scores by using ordinal regression. However, using only one ques-
tion in the questionnaire is being simplistic. Using more nuanced measurements
for evaluating player-character engagement remains to be done in a future work.
References
1. Lankoski, P.: Player character engagement in computer games. Games and Cul-
ture 6, 291–311 (2011)
2. R Development Core Team: R: A Language and Environment for Statistical Com-
puting. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN
3-900051-07-0
3. Christensen, R.H.B.: Ordinal—regression models for ordinal data (2122) R package
version 2012.01-19, http://www.cran.r-project.org/package=ordinal/
Dimitri Slappendel, Fanny Lie, Martijn de Vos, Alex Kopla, and Rafael Bidarra
Train stations, shopping malls and airports: all public places where we spend
a lot of time, waiting for the train, for our friends and for the gate to open.
While waiting, we get bored and we would like to entertain ourselves in order to
kill time. The first thing that comes to our minds is playing a game or social-
izing using our smartphone. People around you are doing the exact same thing.
Wouldn’t it be great if you could play a game with those people, a game which
requires collaboration and interaction with your surroundings?
This is exactly where Paintrix comes into play: gather people, form two teams
and let them collaborate and compete at the same time. Teams have to solve
the same puzzle against the clock. Be faster than your opposing team to win!
How does this work?
Gameplay
The objective of the game is to recreate a colored picture that previously has
been shortly visible on a screen. See Figure 1 for example pictures.
Memorizing the grid is of great importance, as the picture will disappear once
the game starts. Armed with their smartphones as paintbrushes1 they start
painting. The tiles on the field represent the pixels. In order to paint a tile, the
player must first get one color from one of the three available ’color buckets’
that can be found in the corners of the grid: red, blue or yellow. There is also
1
The app required for this game is called ’Paintrix Live’, which can be downloaded
from the Apple App Store and the Android Play Store.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 576–579, 2013.
c Springer International Publishing Switzerland 2013
a special erase bucket available in order to erase a painted tile if a mistake has
been made. After that, they walk (or run) to the tile to be colored and paint it.
A generic arrangement of a field is shown in Figure 2.
Currently, the game is played with two teams competing against each other.
The team that recreates the picture first wins. This means players should first
agree on who does what in order to work efficiently together. Players can leave
and join a team at any moment: making Paintrix ideal for places like train
stations and airports. The picture grid is very scalable and several configurations
are possible to adjust the difficulty of the game. For example, a larger grid could
be used for places that have more space. Sometimes secondary colors are needed.
Just like real paint, secondary colors can be created by combining two primary
colors. See Figure 3 at the end of this document for gameplay photo’s.
At the end of every game each team recieves an amount of Paintrix Points:
these points are calculated and displayed on the screen after a game. The Paintrix
Points are stored in the app where the player can get an overview of their played
games and the earned Paintrix Points. Players can also create their own levels
with the build-in level editor in the app: after approval, their created boards
have a chance to appear when a game starts!
ting. This way, collaboration is more fun because you need to beat the opponents
together.
In contrast to regular apps, Paintrix is not a game where you are glued to the
screen of your smartphone and seperated from your environment and bystanders.
Instead, you are using the smartphone as a tool to paint the tiles on the field.
When two teams are playing against each other, both teams will try their best
to win, but only one of them can be the winner. The better the collaboration, the
faster the team will solve the puzzle. Because most players are very determined to
win, there is a good chance that each individual will play the game very actively:
it is a game where players will be tested both mentally (they have to memorize
the board and quickly devise a strategy to recreate it as fast as possible) and
physically (they need to run around to color the tiles).
Finally, by using the level editor in the app, you can design and submit your
own pictures. Use your imagination to create some cool boards and you might
be able to play your self-created level!
It’s no surprise to be able to paint something on your smartphone... But did
you ever paint using your smartphone as a brush? Try it with Paintrix!
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 580–583, 2013.
© Springer International Publishing Switzerland 2013
• Simple Tokens: Simple Tokens are the most common playing pieces in board-
games(e.g. Checkers, Ludo, Stairs and Ladders, Roulette…). Players arrange a li-
mited amount of playing pieces on the board according to game rules
• Deformable Tokens. Some playing-pieces do not have a constant shape, as they are
made of malleable materials, such us clay, cardboard, cloth...
• Named Tokens. Other kind of playing pieces has a unique role in the game, which
are perceived by the player through their physical appearance. For example, in the
Chess game, the Tower piece has a different appearance and rules than the Pawn.
• Constraint Tokens. A Constraint Token can be described as a playing piece that
acts as a physical constraint of a set of smaller Tokens. Players fill, empty,
rearrange or move the little pieces inside the Constraint Token. For example, in the
Trivial Pursuit game, players fill their playing pieces with little chips in order to
represent their status in the game.
2 Tutorial Organization
The tutorial will last three hours. Participants will follow the complete process of
creating a predefined tangible tabletop game. The game is a Tangible version of the
Fig. 2. Tangible Asteroids game to be created by participants during the tutorial session
classic Asteroids video game from Atari. Playing pieces will consist on cardbooard
space-ships that players wiill manipulate on the tabletop surface in order to clean the
virtual star-field from astero
oids by shutting missiles (see fig. 2).
The tutorial will be org ganized following the different stages that any desiggner
should follow in order to prrototype a game using ToyVision (see fig. 3).
Fig. 3. Different stages off the prototyping process of a tangible game using ToyVision
3 Tutorial Outcom
mes
of tangible games for tabletop devices by offering designers and developers of video-
games with an intuitive environment that isolated them for the technical issues of in-
volving physical objects into a Tangible system.
This tutorial complements the ACE’13 workshop “How to Make Tangible Games
and not Die in the Attempt”; a discussion panel with experts around their experiences
designing and implementing Tangible games for children.
Acknowledgements. We would like to Elisa Ubide, for her help developing the Toy-
Vision Toolkit during her final degree project. This work has been partly financed by
the Spanish Government through the DGICYT contract TIN2011-24660.
References
1. Cooper, N., Keatley, A., Dahlquist, M., Mann, S., Slay, H., Zucco, J., Smith, R., Thomas,
B.H.: Augmented Reality Chinese Checkers. In: Proceedings of the 2004 ACM SIGCHI In-
ternational Conference on Advances in Computer Entertainment Technology, ACE 2004,
vol. 74, pp. 117–126 (2004)
2. Iwata, T., Yamabe, T., Poloj, M., Nakajima, T.: Traditional games meet ICT: a case study
on go game augmentation. In: Fourth International Conference on Tangible, Embedded, and
Embodied Interaction (TEI 2010), pp. 237–240. ACM (2010)
3. Heijboer, M., van den Hoven, E.: Keeping up appearances: interpretation of tangible artifact
design. In: 5th Nordic Conference on Human-Computer Interaction: Building Bridges
(NordiCHI 2008), pp. 162–171. ACM (2008)
4. Kaltenbrunner, M.: reacTIVision and TUIO: a tangible tabletop toolkit. In: Proceedings of
the ACM International Conference on Interactive Tabletops and Surfaces, pp. 9–16. ACM
(November 2009)
5. Marco, J., Cerezo, E., Baldassarri, S.: Playing with toys on a tabletop active surface. In:
Proceedings of the 9th International Conference on Interaction Design and Children, pp.
296–299. ACM (June 2010)
6. Marco, J., Cerezo, E., Baldassarri, S.: ToyVision: a toolkit for prototyping tabletop tangible
games. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive
Computing Systems, pp. 71–80. ACM (June 2012)
7. Rogers, Y., Rodden, T.: Configuring spaces and surfaces to support collaborative interac-
tions. In: O’Hara, K., Perry, M., Churchill, E., Russell, D. (eds.) Public and Situated Dis-
plays, pp. 45–79. Kluwer Publishers (2004)
8. Shaer, O., Jacob, R.J.K.: A specification paradigm for the design and implementation of
tangible user interfaces. ACM Trans. Comput.-Hum. Interact. 16(4), Article 20, 39 pages
(2009)
Abstract. With this creative showcase titled the Ball of Secrets, we demonstrate
an interactive prototype consisting of a unique user interface with an emphasis on
pure play. Through this prototype we created a device that encourages playfulness
and explore sharing and communication via anonymous message posting.
In this paper we describe the concept and the technology used to create the
interface in order to evoke the necessary user experience.
1 Introduction
Play is a form of exploration. While playing a player explores the range of possibilities
made possible by a toy, token or controller within a set of constraints.
Within the context of games the “toy” is an element of the gameplay (in football a
ball is kicked around towards a goal post) and the explorations are goal oriented, go-
verned by a set of rules, bounded by the dimensions of the board or the playing field
and driven by an element of competition.
In pure play situations, that is play that takes place for its own rewards with no
specific goal in mind the toy is the object of play and the explorations are driven by
curiosity and are only bounded by the imagination of the player and the constraints of
the physical and social environment. For example, bouncing a ball around allows the
player to coax the ball to reveal its secrets, e.g. how bouncy it is while learning about
his own strength (how far can I throw the ball), the environment (will the ball bounce
higher from a concrete floor or from a grass field?), and the limits of social accepta-
bility (bouncing a ball where or when you are not supposed to).
Digital artifacts are on the whole far less amenable to this type of playful explora-
tion as their digital innards reveal little about their contents and capabilities and their
interfaces are often designed with a specific set of aims and an exact set of instruc-
tions. The Ball of Secrets is a digital device designed to encourage pure play explora-
tion. Using the familiar and highly tangible shape of a ball and a set of buttons wired
to a fast acting audio board, the Ball of Secrets offers a myriad set of opportunities
through which the player can explore the physical shape and digital functionality of
the ball itself and the secrets in the form of audio recordings that lie within it.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 584–587, 2013.
© Springer International Publishing Switzerland 2013
Our aim in designing the Ball of Secrets is twofold. First, we want to understand
how people interact with the device given its form, functionality and lack of an
explicit purpose or instruction. Second, we want to see whether people would use the
“secret” nature of the ball as a communication medium or leaving messages and per-
sonal details which they might not share otherwise, similar to a digital version of
PostSecret.com[2].
We wish to explore the effect of setting and context on the ways in which people
will interact with the Ball of Secrets. By placing it in different environments such as
shopping malls, schools, conference halls and old age homes, etc., the Ball of Secrets
may take on different interactions and forms of use; it may be a communication device
of the future, an avenue to share stories, disclose secret crushes, or just be an ambi-
guous and fun device. We hope to observe the ways in which context, environment,
personality and society impact and encourage play and exploration.
2 Concept
The Ball of Secrets provides a tangible user interface with memory capabilities to
store audio clips. The prototype is a spherical shaped ball close to the size of a stan-
dard bowling ball. It has two rows of buttons equally aligned across its surface in the
pattern of fingers which are gripping the ball. As the ball is held, subtle holes cover a
microphone facing the player. A speaker positioned on the bottom of the ball is simi-
larly obscured. There are no other visible markings on the surface.
The ball has the ability to store a 10 second audio clip for each combination of but-
tons that is pressed. As the user holds the sphere in his hands, his fingers can press any
combination of the buttons. If there is no audio file stored for a particular combination
of buttons, the microphone activates and displays its status with a red LED. This audio
clip termed a “secret” can only be heard on pressing the same combination of buttons.
With six buttons in the initial prototype, the total number of combinations which can
be stored is 63.
3 Implementation
3.1 Hardware
The Ball of Secrets was built using an Arduino Uno microcontroller[3] together with
an audio wave shield[4], using a SD memory card to store the audio recordings. The
wave shield takes input from an electret microphone, and outputs to a small speaker
mounted on the bottom of the sphere. The Arduino is also connected to six pushbutton
switches, both for powering an internal LED within each switch and for receiving
input.
3.2 Software
The Ball of Secrets primarily uses an adaptation of the already developed WaveRP
library[5]. When activated, the Arduino is constantly reading and debouncing the
digital input coming from each pushbutton. Each pushbutton is assigned to one of the
following values: 1, 2, 4, 8, 16, 32. As the Arduino passes through the debouncing
code, it sums the values of the pressed switches, which will be unique for that combi-
nation. It then passes that integer to the WaveRP library to search for a track at that
index, and, if it does not find one, the recording function is triggered. After an audio
file is played or recorded, the Ball of Secrets returns to its default state afterward.
4 At the Conference
As the ball was designed to be primarily used amongst a gathering of people, we see
the creative showcase section of the conference as an ideal place to observe how the
conference-goers will interact with the device. We propose to display the ball with
minimal supervision and ready to be explored by curious attendees. It will be placed
with instructions to use it and will not be supervised to encourage creative interaction
and reduce discussion about the ball itself. This will help us observe how people ap-
proach the ball, learn about its functions and then use it as per an implicit understand-
ing that they arrive at. To aid in providing context to first-time users the ball will also
be pre-loaded with messages for certain combination of buttons.
References
1. Caillois, R.: Man, Play and Games. University of Illinois Press (2001)
2. PostSecret, http://www.postsecret.com
3. Arduino Uno, http://arduino.cc/en/Main/arduinoBoardUno
4. Wave Shield, http://www.ladyada.net/make/waveshield/
5. Waverp, https://code.google.com/p/waverp/
1 Introduction
Do you have remaining words or messages that you have deep in your heart? Unsent
letter and remaining words bother people’s hearts because of tormenting yearnings.
‘P.S.’ gives people a chance to deliver their inner mind genuinely.
This installation consists of two parts. One is recording the remaining words by
users’ voices. Second is exploring the words on the elastic surface screen with hand
gestures. Through this experience, participants can feel healed by opening and hearing
their heartstrings.
2 Related Works
Elastic screen has been explored by a few researchers as a tool for artwork or data
visualization display. Cassinelli et.al.[1] have implemented the elastic screen as an
interactive art installation. By deforming the screen, the space and time of video
sequence transform. Meanwhile, Yun et al.'s work[2] presents intuitive exploration
through complex and multi-dimensional data structure. It suggests several interaction
methods navigating the human body, or searching with a force directed graph
visualization. An elastic screen also can be a deformable workspace for manipulating
3D virtual objects.[3] Also, recently, Hemsley et.al.[4] designed a set of principles
and interaction examples utilizing the elastic nature of surface. Even though various
researches about elastic screen have been introduced, the majority of them have got
focus on the technical aspects. We developed abstract interaction with physical
changes on the screen. It suggests pushing into an elastic surface as a metaphor for
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 588–591, 2013.
© Springer International Publishing Switzerland 2013
digging deeper into a 'forest' of sound and finding thoughts. This work suggests
elastic screen as a multisensory searching space and intuitive tool for finding sounds
and visual data.
In our lifetime, sometimes we have a moment not to send our own mind to someone
who we cherish - maybe your old teddy bear, friend or family. Whenever you recall
remaining words in your heart, you feel nostalgia for all the things about your friend
and want to talk genuine mind with someone.
If people tell their remaining minds to the microphone, ‘P.S.’ collects voices. All
the collected voices could be explored on the elastic surface screen which is similar to
lake surface for feeling of wave of nostalgia. Through this artwork ‘P.S.’, we want to
ease people who have a huge wave of nostalgia because of remaining words in mind
and sympathize with each other’s nostalgia through hearing their voices carefully.
4 Design
‘P.S.’ consists of two parts - recording booth and multisensory display. First, an user
enters the recording booth and there is a mailbox which looks like a letter envelope.
When an user opens mailbox, he or she can say what's on one's mind through the
mailbox that records the voice. Recorded voices are sent to the multisensory space
and elastic screen presents messages with selected voices responding to a finger.
Multisensory space
4.2 Interaction
A participant in the work is both the person directly involved of a story and explorer
of it. Stories of participants are recorded inside the dark and private booth providing
them for the place to make a personal confession. Opening a letter envelope in the
booth is a trigger to make the recording status on. After talking according to the
instructions, closing the envelope makes the recording status off. Finally, an user can
send the voice by pressing one of 4 expressional buttons which mean 4 emotions- joy,
anger, sorrow, and pleasure.
On the elastic screen, 4 sets of texts are selectively revealed corresponding to the
status of buttons. We suggest an elastic material as the interface to express the feeling
of instigation by remaining words considering the distortion of the surface and
dynamic reaction. We use touching and sweeping gestures as the metaphor of
touching the heart. Also, pressing is used to reach genuine words deeply embedded in
the heart. The deeper the screen is pressed, the fewer voices are left with clearer
sentences and finally one voice lefts in the deepest depth with one sentence since the
very secret words are left in the deepest of one’s own heart.
Texts of poet or lyric -'If I Can' by Emily Dickinson, and 'Unforgettable' by Nat
King Cole et al.- are used as the emotional representation of recorded words. Voices
with sound effects have gridded or circular distribution spatially on screen. An user
can hear 7 voices, which are randomly or sequentially selected among overall 20
voices, maximally at once when touching but not pressing.
Tab
ble 1. Interaction mapping by gestures
Gesture Intteraction
Touch soundd: water drop sound effect
visua
al: blurring text, wave visual effect
Sweep soundd: wave sound effect, controlling voices volume
visua
al: controlling positions of texts, wave visual effect
Press soundd: adding or reducing voices by depth
visua
al: controlling text size bigger or smaller by depth
Fig. 2. 1st step of depth: an Fig. 3. 2nd~3rd step of depth: Fig. 4. 4th step of depthh: an
user touches the elastic screen an user presses the screen user presses the screen and
and hears mixed 4 voices(left) and hears mixed 3~2 voices hears 1 voice (right)
(middle)
5 Conclusion
The 'P.S' suggests the collaborative interaction system that gathers people’s norm mal
voices and transforms themm into a real-time generative artwork. The elastic interfface
and the way of interaction make
m participants connected to the concept of the work. By
digging the surface which is similar to finding or accessing something, a particippant
can hear and share the hearttstrings.
References
1. Cassinelli, A., Ishikawa, M.:
M Khronos projector. In: ACM SIGGRAPH (2005)
2. Yun, K., et al.: ElaScreen: exploring multi-dimensional data using elastic screen. In: C
CHI
2013 Extended Abstracts ono Human Factors in Computing Systems. ACM (2013)
3. Watanabe, Y., Cassinelli, A., Komuro, T., Ishikawa, M.: The Deformable Workspacce: a
Membrane between Real and Virtual Space. In: 3rd IEEE International Workshopp on
Horizontal Interactive Human
H Computer System (TABLETOP), Amsterdam, the
Netherlands, October 1-3 (2008)
4. Hemsley, R., Dand, D.: Obake: Direct Interactions with Deformable Shape Changging
Surfaces. In: CHI, Paris, Frrance, April 27-May 2 (2013)
Christiane Moser
Abstract. Player’s enjoyment is one of the most important goals for games.
Without this, children will not repeatedly play them. In order to meet children’s
needs, it is important to consider them in the development process, for example,
by enabling them to participate actively in the process. Therefore, children will
be enabled to participate in ideation workshops to create creative low-fidelity
prototypes of game ideas that inspire game designers.
1 Introduction
Children are becoming experienced and frequent users of software, services, and
technology and are emerging as an important user group [4]. They encounter and use
software or technologies in their daily lives through, e.g., mobile phones to communi-
cate, computer games for individual or collaborative entertainment, and educational
technologies for learning [7].
As there is not a lot of literature available for the involvement of children through-
out the whole development process of games (next to [7] or [10]), children are rarely
involved. In order to meet their needs, an adequate consideration of them in the de-
velopment process of technologies is necessary [1]. For game developers, it would
thus be an advantage to work together with children to satisfy their range of desires
and needs (e.g., [2], [11]) and not to see them only in the role of game consumers.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 592–599, 2013.
© Springer International Publishing Switzerland 2013
as a foundation. The CCGD approaches illustrate how to guide the involvement and
participation of children aged 10 to 14 years in school classes within the development
process of games. The children’s ideation workshop for the design phase uses game
progress storyboards and creative low-fidelity prototyping and aims at:
Fig. 1. Two examples of game progress storyboards illustrating parts of the game menu
Creative low-fidelity prototypes can be built in groups of four to six children. They
are sketchy, incomplete, and quickly built working models of game aspects. Druin [3]
mentioned in her work that “there is never a need to teach people how to prototype,
since using basic art supplies comes naturally to the youngest and oldest design part-
ners”. Nevertheless, children need a starting point for prototyping [9]. Therefore, this
workshop starts with explaining the game idea and the creation of storyboards to illu-
strate a short sequence of interactions within the game to build the game prototype
afterwards.
Similar to Knudtzon et al. [6] ‘bags of stuff’ approach, the children are provided
with Playmais, Playdough, Lego, and other creative material to create the prototypes.
The prototypes are used to illustrate parts of game levels (see Fig. 2) and re-enact
scenes of the previously created game progress storyboard. Children can try out game
procedures/mechanics and actively discover problems or challenges in the game play.
For very complex game concepts, the low-fidelity prototypes might be problematic, as
the implementation might be too difficult or simply impossible for children [7].
Therefore, the game idea should be easy to explain and understand, but also not too
detailed in order to leave room for imagination (i.e., children should easily come up
with design ideas for the game).
Fig. 2. Two examples of creative low-fidelity prototype illustrating parts of the game
world
In the end, the creative low-fidelity prototypes are filmed while the children play
the gaming sequence of the game progress storyboard for the others. For the filming,
it is necessary that the children assign roles to each other, i.e., one director (storytel-
ler), one or more players (moving the character), and artists moving around the other
game elements. Afterwards, the other children can ask questions, make remarks, or
improvement suggestions.
These videos can be used to envision the game ideas developed by the children for
absent designers and developers [7]. The creative low-fidelity prototype videos
provide insights about children’s thoughts on possible game play, which can help to
better meet their expectations about the game idea and thereby inspire the develop-
ment process.
For the workshop, a multi-purpose room for the amount of children is needed, where
tables for group work can be formed and the children can craft. Two people (i.e., the
workshop lead and assistant) organize and conduct the workshop. The following ma-
terials have to be prepared in advance:
1
http://pubs.iied.org/pdfs/G01508.pdf
Table 1. (continued)
5. 15-20 Preparation for filming (study the scene of the storyboard)
• Explain the outcome of this step and what will happen af-
terwards
• Put the game world ground (the matt/pasteboard) on the
different tables and let the children place the other created
materials on top of it (this should not take longer than 5 mi-
nutes)
• Children should assign roles in the group: 1 director (story-
teller), 1 or more player (moving the character) and the rest
are artists moving around the other game elements
o Help the children to assign roles, if this takes more
than 2-3 minutes, in order to have enough time to
study the play according to their storyboard
• Let them practice the game scene from the storyboard ac-
cording to the directors instructions
6. 10-15 Filming (recording of the played gaming scene)
• Explain the outcome of this step and what will happen af-
terwards
• Walk from group to group with the video camera and tripod
• Let the children act their gaming scene, if necessary let
them react parts of it
• Meanwhile let the other children stand around the table,
watch the play and ask questions afterwards, if there were
obscurities
7. 5-10 Closing
• Wrap-up the workshop
• Ask children about their experiences
• Summarize your own experiences and the outcome of the
workshop
8. 30-90 • Edit recorded videos and provide them to the designers as
well as developers
The objectives of the workshop, the expected outcome, and outcomes of previously
conducted workshops are described in the following sections.
(i.e., understanding) in order to better meet their expectations about game ideas when
building games.
5 Conclusion
The need for child-centered design methods for game development is there. As the
suggested approaches for the children’s ideation workshops have already been suc-
cessfully applied with children ages 10 to 14 years, the proposed workshop holds the
potential for actively involving children in the development process of games with a
win-win situation. On one hand, children are enabled to actively participate. On the
other hand, game designers get insights regarding children’s need and inspiration
from the produced videos of the creative low-fidelity prototypes.
2
http://www.sparklingscience.at/de/projekte/508-games4school-
wissenschafter-innen-entwickeln-spiele-mit-und-f-r-sch-ler-
innen/ or http://www.icts.sbg.ac.at/content.php?id=
1851&m_id=1011&ch_id=1039&ch2_id=1452
Science”). This work is supported by the Austrian project “AIR – Advanced Interface
Research” funded by the Austrian Research Promotion Agency (FFG), the ZIT Center
for Innovation and Technology and the province of Salzburg under contract number
825345.
References
1. Antle, A.N.: Child-based personas: need, ability and experience. Cognition, Technology &
Work 10(2), 155–166 (2008)
2. Brederode, B., Markopoulos, P., Gielen, M., Vermeeren, A., de Ridder, H.: Powerball: the
design of a novel mixed-reality game for children with mixed abilities. In: Proceedings of
the 2005 Conference on Interaction Design and Children (IDC 2005), pp. 32–39. ACM,
New York (2005)
3. Druin, A.: The Role of Children in the Design of New Technology. Behaviour and Infor-
mation Technology 21(1), 1–25 (2002)
4. Jensen, J.J., Skov, M.B.: A review of research methods in children’s technology design. In:
Proceedings of the 2005 Conference on Interaction Design and Children, IDC 2005, pp.
80–87. ACM, New York (2005)
5. Jones, C., McIver, L., Gibson, L., Gregor, P.: Experiences obtained from designing with
children. In: Proceedings of the 2003 Conference on Interaction Design and Children (IDC
2003), pp. 69–74. ACM, New York (2003)
6. Knudtzon, K., Druin, A., Kaplan, N., Summers, K., Chisik, Y., Kulkarni, R., Moulthrop,
S., Weeks, H., Bederson, B.: Starting an intergenerational technology design team: a case
study. In: Proceedings of the 2003 Conference on Interaction Design and Children (IDC
2003), pp. 51–58. ACM, New York (2003)
7. Moser, C.: Child-Centered Game Development (CCGD): Developing Games with Child-
ren at School. Personal and Ubiquitous Computing, Special Issue on Child Computer Inte-
raction, 1–15 (2012)
8. Rouse, R.: Game Design Theory and Practice, 2nd edn. Wordware Publishing Inc., Plano
(2000)
9. Scaiffe, M., Rogers, Y.V.: Kids as Informants: Telling Us What We Didn’t Know or Con-
firming What We Knew Already? In: The Design of Children’s Technology, pp. 27–50.
Morgen Kaufman, San Francisco (1999)
10. Tan, J.L., Goh, D.H.L., Ang, R.P., Huan, V.S.: Child-centered interaction in the design of
a game for social skills intervention. Computers Entertainment 9(1), 17 (2011)
11. Tychsen, A., Canossa, A.: Defining personas in games using metrics. In: Proceedings of
the 2008 Conference on Future Play: Research, Play, Share (Future Play 2008), pp. 73–80.
ACM, New York (2008)
1 Introduction
Paper sumo game is one of the popular tabletop games in Japan. This game follows
sumo rules by using paper figures. Two players play this game with each sumo wrest-
ler figure made of paper as depicted in Fig.1. Each player puts his/her own paper
wrestler facing each other on the sumo ring board that is typically made of a
paper box. The player taps on his/her own-side of the sumo ring board and lets his/her
own wrestler rush and fight with its opponent. The match is decided when either
wrestler has been out from the sumo ring or lost balance and fallen down. Because of
the simplicity, many of Japanese children have played the game. However, it cannot
be played well when paper wrestlers don’t collide or fall down before the game
heated.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 600–603, 2013.
© Springer International Publishing Switzerland 2013
We propose the virtual paper sumo game “Dosukoi-Tap” which solves following
problems of the actual paper sumo game:
This simulation is reproduced by the system employing the multi-finger tracking and
pressure-sensitive device.
2 Dosukoi-Tap
Tsuppari. When a player does swipe toward the opponent, his/her wrestler does an
offence action to push strongly. However it’s easy to lose balance.
Harai. When a player does swipe toward oneself, wrestler moves back to avoid the
Tsuppari of it’s opponent.
In this game, the tapping and the two original operations involve players’ strategic
behaviors. In order to win a match, they must repeat tapping while considering
wrestler’s balance. Although Tsuppari is effective in pushing out an opponent, the
operation carries the risk of falling down when the opponent does Harai.
Tapping operation moves the wrestler. In actual paper sumo, the motion of wres-
tlers is not stable because tapping operations vibrate only sumo ring board made of
paper. The wrestlers fall down easily and also do not easily grapple with each other so
it is difficult for a player to let his/her wrestler fight. The wrestler moves and pushes
one another in response to the force of each tap. The force also affects the wrestler’s
center of gravity balance according to the position of each tap on the ForcePad. Play-
ers must keep the left-right balance to win a match.
3 Concluding Remarks
Dosukoi-Tap has been realized Japanese traditional game paper sumo by employing
multi-finger tracking and pressure-sensitive device. This game can be played by sim-
ple tapping operation like actual paper sumo and solved some of the problems of it.
The original operations have also increased the excitement of the “paper sumo”. The
users who know or do not know actual paper sumo rules easily understand the opera-
tion method, and the strategy is considered because of these simple operations. In
recent years, many traditional games including paper sumo have not been played fre-
quently. These games can be reborn in a more interesting game by solving some of
problems and giving new technology. It will lead us to rediscovery of the fun that we
had lost in these days.
Reference
1. ForcePad, http://www.synaptics.com/solutions/products/forcepad
Abstract. The design principle of Tangible User Interfaces has been applied to
musical interfaces for more accessible usage of computers for musical expres-
sion. This paper proposes DropNotes, a tangible user interface for music com-
position, which utilizes the combination of affordances of familiar objects as
metaphorical procedures to manipulate digital audio. By creating more accessi-
ble and intuitive user interface for music composition, we target opening up a
novel interactive musical expression.
1 Introduction
Tangible music interfaces have eased user experiences to leverage computers for mus-
ical expression, having made digital audio information more accessible. The design
philosophy behind Tangible User Interfaces (TUI) [1] emphasizes representing digital
information manipulation in natural affordances of objects in the physical world, in an
effort to enable users to intuitively interact with the information world. Based on this
concept, a number of tangible music interfaces has been designed. A natural
affordance is assigned in an action to manipulate digital audio information, such as
playing audio with a sequencer, arranging musical constructions, or changing parame-
ters in a synthesizer [2-6]. Design principles of these interfaces have enabled more
accessible and intuitive audio manipulation, and have opened up a new frontier of
musical performances.
Tangible music interfaces have been mainly focused on playing, modifying and ar-
ranging preset sound sources [2, 4-6]. However, to augment capacities of feasible
musical expressions, users need to have an option to jack up sound materials, not
confined using only ready-made ones, so that users can express their versatile inten-
tions and nuances on composing music. Thus, it is significant to offer users a means
to record sound materials according to their musical nuances, in order to ensure
potential diversity for musical expression by means of a tangible music interface.
Furthermore, all procedures for music composition, including recording, choosing,
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 604–607, 2013.
© Springer International Publishing Switzerland 2013
DropNotes consists of a liquid bottle stand, a glass table, a web camera, a projector, a
speaker, and a PC as shown in Figure 1. In this system, multiple procedures to com-
pose music are assigned in metaphorical familiar affordances. Sound source record-
ing, source choosing, musical construction arrangement, and effecting are represented
as putting a funnel into a bottle, tapping off a liquid from the bottle with a dropper,
dropping liquids onto the table, changing the color of droplets, respectively.
Fig. 1. DropNotes System in Use (Left), and System Configurations Diagram (Right)
Fig. 2. Recording (Left-top), Choosing and Picking up Sources (Right-top), Dropping the Liq-
uid on the Glass Table and Projected Sound ID (Left-bottom), Interfusing Color to Alter Filter
Function (Right-bottom)
In this research, we have designed a novel tangible music interface for more accessi-
ble, intuitive and collaborative music composition, especially emphasizing the impor-
tance of recording function. We have represented all procedures to manipulate audio
information in relevant metaphorical affordances of familiar objects, based on the
design principles of TUI. Users can intuitively leverage their own acquired
experiences for audio manipulations. In initial applications, users were capable of
collaboratively composing music with the system. Even non-professional users are
successfully involved in a music composition procedure due to familiarities of meta-
phorical affordances. Moreover, initial applications also indicated that the system also
has potential to work as a platform for diverse kinds of music, because users can
express their musical nuances on sound materials by the recording function.
As to the next study, we need to pursue appropriate interpretations for a set of
droplets to seek more transparent interaction between users and the system. It is also
necessary to grasp the inclination of outcomes of this system through more user expe-
rience studies, for envisioning the potential of novel interactive musical expressions.
References
1. Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and
atoms. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Sys-
tems, Atlanta, Georgia, March 22-27, pp. 234–241 (1997)
2. Jordà, S., Kaltenbrunner, M., Geiger, G., Bencina, R.: “The reacTable*”. In: Proceedings of
the International Computer Music Conference (ICMC 2005), Barcelona, Spain, pp. 579–582
(August 2005)
3. Cameron, A.: Systems Design Limited, “The Art of Experimental Interaction Design”.
Gingko Press (2005)
4. Ishii, H., Mazalek, A., Lee, J.: Bottles as a minimal interface to access digital information.
In: Proceeding of the CHI EA 2001, CHI 2001 Extended Abstracts on Human Factors in
Computing Systems, New York, NY, US, pp. 187–188 (2001)
5. Levin, G.: The Table is The Score: An Augmented-Reality Interface for Real-Time, Tangi-
ble, Spectrographic Performance. In: Proceedings of the International Conference on Com-
puter Music (ICMC 2006), New Orleans, November 6-11 (2006)
6. Blaine, T., Fels, S.: Collaborative Musical Experiences for Novices. Journal of New Music
Research, Swets&Zeitlinger 32(4), 411–428 (2003)
7. Klügel, N., Frieß, M.R., Gloh, G., Echtler, F.: An Approach to Collaborative Music Com-
position. In: Proceedings of the International Conference on New Interfaces for Musical
Expression, pp. 32–35 (2011)
Abstract. Our work is to analyze how the practice of chosen video games may
influence the player's interest. A set of video games was selected by a group of
experts according to their qualities as games and their relations with scientific
knowledge. A focus test experiment has been set up to evaluate the correlation
between teenagers engagement and their interest in scientific domains. The
analysis of the results shows that the desire to pursue the game and the senti-
ment of responsibility has, respectively, a direct and an indirect influence on the
player's scientific interest. Considering that interest is known to be an important
motivational factor in learning, these results have important implications for the
serious games design.
1 Introduction
The present study is a part of the project, Recensement /INMEDIATS1, which main
goals are to understand the relationship between digital entertainment and the interest
in scientific knowledge. Our work is to analyze how the practice of chosen video
games may influence the player's interest.
Interest is defined to be an emotional and motivational variable [1]. Furthermore, it
has been established that the student's personal, situational and topic interest influence
positively his/her learning outcomes [2,3]. Finally, in cognitive science, interest is
considered to be very closely associated to a person's engagement in a given cognitive
activity [4].
Engagement is associated with sensations such as immersion or, even more so,
presence: Essentially, the sensation to "be there" [5,6,7]. In particular, Brockmyer &
al. establish the relation between engagement and emotions, as fear, in their game
engagement questionnaire. Furthermore, some theoretical video game studies ob-
served that emotion can be considered as a component of engagement [8,9]. On the
1
Recensement/INMEDIATS is partly funded by the Government agency ANRU with Univers-
cience, CNAM, the University of Paris 8 as partners.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 608–611, 2013.
© Springer International Publishing Switzerland 2013
other hand, Natkin [10] defined engagement in video games as a feeling of responsi-
bility and Shoenau-fog [11] came to the conclusion that engagement can be conceived
as a desire to pursue the game. Engaging the user in his/her experience of the product
should be, among others, facilitative of cognitive efforts associated with serious as-
pects which might be presented in the product [12].
However, the manner in which a fun and engaging nature of an application, nota-
bly a video game, influences cognitive processes of the user has not yet been estab-
lished. One way to think of such a relationship is to conceive it as being mediated by
some other variable, for example the player’s interest. Considering All these points of
view, the present study attempts to explore the relationship between the player’s en-
gagement in a videogame and his interest in the scientific domain presented in the
game.
2 Method
A set of video games has been selected by a group of experts. That was divided in two
steps: first, from a dedicate data base, games were chosen according to their scientific
aspects and gameplay qualities; Then, these shortlisted games were played by a group
of scientific popularization experts and a pedagogical expert2. The final games chosen
are: The Bridge (Ty Taylor and Mario Castaneda, 2013), World of Goo (2D Boy,
2008), Angry Birds Space HD (Rovio, 2012), Puddle (ENJMIN, Neko/Konami 2012)
and Sim City 4 (Maxis, 2003).
2.2 Participants
One hundred and thirty three teenagers participated in the study, 23 girls and 110
boys. They were ranging in age from 14 to 18 years old (M =15.360; SD =1.292),
were all native speakers of French and had the habit to play video games at least three
hours per week.
2.3 Questionnaires
For this study we designed a pre- and a post-questionnaire. The pre-questionnaire was
designed to assess video games habits and school interests. It consisted of multiple-
choice questions. The post-questionnaire was designed to assess the level of engage-
ment that the video game incited in the participants, as well as participants' interest in
the scientific domain that was suggest by the video game a given participant has
played. The participant's engagement in the video game was assessed on three sepa-
rate dimensions; the player's presence during the game, the desire to pursue playing
the video game and the feeling of responsibility.
2
More details on http://www.cite-sciences.fr/inmediats/
seriousgame/projet.php
2.4 Procedure
The study consisted of 3 stages. In the first stage, the participant was asked to answer
the pre-questionnaire. In the second stage, the participant was assigned to a video
game related to a scientific domain different than his/her favorite school subject.
He/she was asked to play the game autonomously for 30 minutes. In the third and
final stage the participant was invited to answer, right after they played the game, at
the post-questionnaire.
2.5 Results
The post-questionnaire internal stability was tested with the use of Cronbach's alpha
statistics3. For further analysis, only the data from sections with high reliability
(Cronbach's alpha was superior to .650) were retained (Desire to pursue the game,
Feeling of responsibility and Interest in scientific domain), while those from section
Presence were excluded (Cronbach's alpha was inferior to .500).
In order to assess the influence of the two compound variables of player's engage-
ment on his/her interest in the scientific domain presented in the video game he/she
has played, a multiple regression analysis using a step-wise method was conducted
with the Desire to pursue the game and the Feeling of Responsibility as predictor
variables, and the Interest in the scientific domain as a predicted variable.
The multiple regression4 analysis results revealed a moderate to middle positive
correlation among the variables (r=.483 for the correlation between Desire and Inter-
est; r=.357 for the correlation between Responsibility and Interest; r=.504 for the
correlation between Desire and Responsibility)5. Furthermore, Desire, but not Re-
sponsibility, was found to have a significant contribution to predicting Interest.
(F(1,131)= 39.861; p<.001) and to account for approximately 23 per cent of the va-
riance of the predicted variable (R2 =.233; delta R2 =.227)6.
3
Coefficient providing a measure of the internal stability of a given questionnaire..
4
F-test (F and p values): In multiple regression, it explore whether the variables thought of as
predictors have a significant effect on the predicted variable.
5
r: Pearson product-moment correlation coefficient which is a measure of dependence be-
tween two variables. Its values vary between +1 -1; the higher the absolute value of r, the
stronger the dependence/correlation.
6
R2: Coefficient of determination which provides a measure of how well observed outcomes
are replicated by the model.
References
1. Ainley, M., Hillman, K., Hidi, S.: Gender and interest processes in response to literary
texts: Situational and individual interest. Learning and Instruction 12(4), 411–428 (2002)
2. Renninger, K.: Individual interest and its implications for understanding intrinsic motiva-
tion (2000)
3. Schiefele, U.: Topic interest, text representation, and quality of experience. Contemporary
Educational Psychology (1996)
4. Besley, J.C., Roberts, M.C.: Qualitative interviews with journalists about deliberative pub-
lic engagement. Journalism Practice 4(1), 66–81 (2010)
5. Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., Burkhart, K.M., Pidruzny, J.N.:
The development of the Game Engagement Questionnaire: A measure of engagement in
video game-playing. Journal of Experimental Social Psychology 45, 624–634 (2009)
6. Gamberini, L., Barresi, G., Maier, A., Scarpetta, F.: A game a day keeps the doctor away:
A short review of computer games in mental healthcare. Journal of Cyber Therapy and
Rehabilitation 1(2), 127–145 (2008)
7. Regenbrecht, H.T., Schubert, T.W., Friedmann, F.: Measuring the sense of presence and
its relations to fear of heights in virtual environments. International Journal of Human-
Computer Interaction 10(3), 233–249 (1998)
8. McMahan, A.: The Video Game, Theory Reader (2003)
9. Prensky, M.: Digital game-based learning. Comput. Entertain. 1, 21 (2003)
10. Natkin, S.: Interactivity in Games: The Player’s Engagement. In: Nakatsu, R., Tosa, N.,
Naghdy, F., Wong, K.W., Codognet, P. (eds.) ECS 2010. IFIP AICT, vol. 333, pp. 160–168.
Springer, Heidelberg (2010)
11. Schoenau-Fog, H.: Hooked! Evaluating Engagement as Continuation Desire in Interactive
Narratives. In: André, E. (ed.) ICIDS 2011. LNCS, vol. 7069, pp. 219–230. Springer,
Heidelberg (2011)
12. Zichermann, G.: Fun is the Future: Mastering Gamification (October 2010),
http://www.gamesfornature.org/
Abstract. This paper describes an interactive block device – LED Tile – utiliz-
ing 8x8 dot-matrix LEDs which obtain pattern drawing capability. It also ap-
plies magnet connectors for physical connections and signal transmissions, as
well as interacts with accelerometer sensor and audio signal device. The func-
tion of the block device can be defined by the drawn pattern on the matrix LED,
and this capability extends the block system applications. In this paper, we de-
scribe the hardware and software configurations of this block device, as well as
several fundamental and high-level functions of alphanumerical character rec-
ognition. We also describe two applications of this device, such as magic square
and character arrangement.
1 Introduction
Block device systems, which have contained many simple component block devices, have
been evolving from the simple structural devices, such as traditional LEGO blocks, to the
intelligent devices that have small computer, sensors, actuators and communication chan-
nels [1], [2], [3]. Such intelligent block devices allow to build interactive systems. Each
intelligent block device obtains fixed functions such as photo sensor, microcontroller,
while operations in controller can be externally programmed.
The authors have been proposing and developing the block device with 8x8 dot-
matrix LED unit as display output as well as input device by using visible light
source, “LED Tile” system and its applications [4], [5].
In this paper, we propose a new type of block device system where each block can
recognize alphanumerical characters for function definition drawn by visible light on
the matrix LEDs. Each block behaves as it is defined by the drawn patterns, although
it is identical in hardware. We also describe the implementations of two application
games using this capability i.e. Magic Square and Character Arrangement.
In this section, we describe both the hardware and software configurations of the
developed block device, as well as the function definition algorithms by visible light.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 612–615, 2013.
© Springer International Publishing Switzerland 2013
Hardware Specification
Figure 1a shows the developed matrix LED block device named as “LED Tile (LT)”.
The microcontroller (ATmega640, Atmel) and tri-axis acceleration sensor
(MMA7455, Freescale) are mounted on the board that accommodates to the 8x8 dot
matrix LED unit size. The LT accompanies by the four magnet connectors located at
the four edges for easing to connect and disconnect among blocks. The communica-
tion among the connected LTs is carried out in the asynchronous full-duplex serial
protocol to implement system operations across the LTs. The LT unit also holds Li-
thium-Polymer battery (3.7V, 110mAh) and Piezo sounder.
A dot matrix LED unit is applied for displaying and also for photo sensing[6], and
users can ‘draw’ patterns on the dot matrix LED unit surface by visible light like LED
light or laser pointer, by the time-divided operation of display and sensing.
(a) (b)
Fig. 1. Developed matrix LED block device, LT
By using the procedure in [5], the device can detect the drawn pattern by the visible
light, and display the desired pattern independently.
The photosensitivity of LED, or saturated voltage of LED for incident light, de-
pends on the wavelength (color) of the light and the structure of LED. The sensitivity
becomes maximum for the incident light whose wavelength is equal to that of LED
itself. The structure of LED also defines the saturated voltage for incident light; the
generated photo current by incident light will increase the voltage of LED by charging
the junction capacitor, while the leakage current decreases the voltage, which is larger
than voltage of LED, and finally becomes an equilibrium state. The photosensitivity
of LED is basically low compared with the photo sensor (photo diode), and the vol-
tage change of LED by environmental light can be ignored; the voltage change by the
‘bright’ incident light by the user’s intension can be detected to enable user’s interac-
tion with LED by visible light.
surface and then turning on/off of each suitable LED, and displaying patterns such as
alphanumerical characters in the matrix LED. The acceleration parameter can be used
for implementing the user’s interaction, such as shaking.
The Character Recognition Library: This library implements a high level code – main-
ly alphanumerical character recognition. The algorithm is to use vector directions and
Euclidean distances established by two consecutive extracted feature points [7], to distin-
guish different characters which resemble a vector set. We map an individual LED dot
corresponding to a unique coordinate. When drawing the patterns on the matrix LED sur-
face, all points will be saved in an array with the chronological order. They will be then
processed and extracted to a few feature points in the same order. Finally, two consecutive
feature points establish an array of vectors whose every element grasps one in eight differ-
ent directions such as up, down, right, left, upleft, upright, downleft, and downright. This
information will be compared with the stored dictionary data to decide which characters
should be recognized. Figure 2a illustrates a typical example that is vector’s order numbers
when drawing number ‘1’ on the surface of matrix LED.
(a) (b)
Fig. 2. Drawn character and its corresponding vectors
In the case, some characters have the identical directions of all vectors such as ‘a’,
‘d’, ‘q’ characters provided in Figure 2b, the Euclidean distance will be applied to
distinguish all of them. In this example, the 6th and 7th vectors will be considered to
calculate the Euclidean distance.
Figure 3 shows some examples of actually drawn patterns as well as the recognized
and displayed suitable numerical characters.
Fig. 3. Drawn ((a)(b)(c)) and recognized ((d)(e)(f)) patterns. (a)(d) Number 3, (b)(e)Number 5,
and (c)(f) Number 9.
3 Application
The character recognition capability for the drawn pattern on LED can be used for
defining the block function, with the identical device hardware. This is worth to extend
the application of the block devices. In this section, we describe ideas and
implementations of two applications using the LT device and the recognition capability.
4 Conclusion
In this paper, we described the matrix LED block device with pattern draw capability
by visible light, and the character recognition algorithm. We also described two appli-
cations of function-definable block system.
References
1. Gorbet, M.G., Orth, M., Ishii, H.: Triangles: Tangible Interface for Manipulation and explo-
ration of Digital Information Topography. In: Proc. of CHI 1998, pp. 49–56 (1998)
2. Lee, J., Kakehi, Y., Naemura, T.: Bloxels: Glowing Blocks as Volumetric Pixels. ACM
SIGGRAPH 2009 Emerging Technologies ET 331 (2009)
3. Suzuki, H., Kato, H.: AlgoBlock: a Tangible Programming Language for Collaborative
Learning. In: Proc. of ED-Media, p. 770 (1994)
4. Akita, J.: Matrix LED Unit with Pattern Drawing and Extensive Connection. In: Proceed-
ings of SIGGRAPH 2010, Emerging Technologies, CD-ROM (2010)
5. Akita, J.: Interactive Block Device System with Pattern Drawing Capability on Matrix
LEDs. In: The ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)
2012, Interactivity, in DVD-ROM (2012)
6. Hudson, S.: Using Light Emitting Diode Arrays as Touch Sensitive Input and Output
Devices. In: Proc. UIST 2004, pp. 287–290 (2004)
7. http://www.ccs.neu.edu/home/feneric/charrec.html
Abstract. This paper explores the use of modern sensor technologies for
physical interaction in educational games and interactive spaces. The paper
presents a prototype of an educational game developed using a motion capture
controller and two biofeedback sensors (EEG, ECG), proposing a generic
architecture for multi-sensor interactive spaces. Target of this research is to
study further the potential effect of such technologies on educational interactive
games, in two aspects: i) on the involvement of human body and motion in the
process of learning, and recall of knowledge, ii) on assisting the development of
basic social emotional competencies, through the enhanced social affordances
of embodied games.
1 Introduction
The use of computer games in education has been an active field of academic research
for the last twenty years, providing considerable evidence to support the positive
effects of the use of games on the learning outcomes of students. Up until recently,
based on the capabilities of the given technology, academic studies focused mainly on
the conceptual engagement of learners in games. At the same time, scientists have
highlighted the importance of psychological factors that influence children’s learning,
placing the development of social-emotional competences at the core of modern
pedagogy, and creating a growing need in education for tools and instruments to
support and assess these skills. Central to the development of social-emotional
competences is the individual in relation to his or her social environment using all
possible expressive forms. The human body can be seen as part of the human
cognition (Dourish 2001 [1]), and a medium of self-expression and interaction with
the environment and other people, thus its involvement in learning is of key
importance.
The use of motion controllers and biofeedback sensor technologies has great potential
for educational games, contributing to the conceptual engagment of learners [2], as well as
to the social interaction affordances of the game [3]. Motion controllers, offer more
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 616–621, 2013.
© Springer International Publishing Switzerland 2013
Targets of the prototype presented were: i) the testing and demonstration of the
capabilities of some selected commercial sensors, ii) the basic implementation of a
multi-sensor system’s, generic architectural design, and iii) to provide a base for the
collection of some first body and motion data for further study of the concept of using
these data in game interactions.
Inspired by simple traditional children games like hopscotch or jumping rope, main
idea of the prototype conceptualized and developed by the author, was to use a
motion capture sensor to create a board game that would blend characteristics of such
games, such as physical social interaction and body motion, with those of modern
video games, like dynamic computer graphics, sound effects, and fantastic virtual
environments.
1
http://waag.org/en/project/commit
Fig. 1. View
V of the NumHop prototype game scene
wears on his head, detects high level of attention, the superpower level startss to
charge and the player can useu it again. If the player runs out of superpower, he has to
suffer the robot’s hits, whiich reduce the player’s health level. If the player surviives
the attack he can step bacck for a moment and try to relax. When the EEG sennsor
detects high level of meditaation, the health level of the player is increased. The plaayer
is given 3 lives in the beeginning of the game, and bonus lives are awarded afteer a
number of consecutive corrrect answers.
The heart rate value is not directly connected to any element of the game pllay.
There are two reasons that led to this decision. The first one is that because the heeart
sensor has to be a little moiisturized and worn under player’s clothes, it might provee to
be impractical, and time co onsuming to use in a school test session. The second onne is
that designing a certain inteeraction based on heart rate, and mapping this to a crucial
element of the game, requirred to know in advance the expected range of values durring
the game, knowledge and expertise
e that was not available at the time of developmeent.
The presence of the heartt rate value however was thought to be useful as aalso
explained earlier, first for collection
c of the data for further use, and second to see hhow
players respond to this info ormation, if for example by placing the heart rate valuee of
the player, appearing on thee GUI, as what could be conceived by someone as anotther
form of scoring points, mo otivates the player to raise his heart rate by moving m more
intensively.
game, triggering people’s curiosity and offering and engaging experience. On the
negative side, it also showed us that even in a simple game, designing interactions
based on multiple sensors might lead to some degree of difficultness to understand
game mechanics.
Previous studies in the field of sensor technologies and affective interaction have
found various real-time modalities and bio-signals that reveal information about the
emotional state of a person. Expanding this research and intergrading those finding in
games, will allow us to work towards more ambient forms of interaction, where
sensor data is not directly mapped to certain actions, but used by an adaptive game
environment and virtual actors with basic signs of emotional intelligence. Elements
like these would enhance the immersive experience of an interactive story, with
robustly interactive characters, that by extension would help the player to express and
control her own emotions.
References
1. Dourish, P.: Where the action is: The foundations of embodied interaction. MIT Press,
Cambridge (2001)
2. Lindley, S.E., Couteur, J.L., Berthouze, N.L.: Stirring up experience through movement in
game play: effects on engagement and social behaviour. In: Proceedings of the Twenty-
sixth Annual SIGCHI Conference on Human Factors in Computing Systems (CHI 2008),
pp. 511–514. ACM, New York (2008)
3. Johnson-Glenberg, M.C., Birchfield, D., Savvides, P., Megowan-Romanowicz, C.: Semi-
virtual Embodied Learning – Real World STEM Assessment. In: Annetta, L., Bronack, S.
(eds.) Serious Educational Game Assessment: Practical Methods and Models for
Educational Games, Simulations and Virtual Worlds, pp. 225–241. Sense Publications,
Rotterdam (2010)
1 Introduction
In the classic fairy tale, Hansel and Gretel safely return home by tracing back
their path using a trail of pebbles. Natural environments capture history in var-
ious ways from valleys carved out by rivers to the broken twigs that trackers use
to locate animals. Man-made products depict their most frequent used function-
ally in wear, such as chipped off paint or the greasiness of buttons. However, in
modern society of concrete buildings and dynamic touchscreen interfaces, these
environmental clues of use are lost. With the photochromic carpet, we aim to
re-introduce the concept of subtle recording of history in the environment and
visualize patterns-of-use that interact with the user in a ludic way.
The system consists of a large floor painted with passive, mono-stable color
changing ink that is excited by footsteps of custom indoor shoes with build-in
LEDs. Wearing the shoes, users leave trails of dynamic generated patterns. The
patterns persist for minutes and invite users to use the carpet as an expressive
canvas. Using smart materials in the floor, and the shoe as an activation medium,
the system easily scales to very large surfaces, without the limitation of installing
a camera-projector system or sensor networks.
Our current demonstration serves no other purpose than an experiential pro-
totype. Through exhibits we aim to gather user feedback for future applications
in schools and hospitals.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 622–625, 2013.
c Springer International Publishing Switzerland 2013
Fig. 1. The carpet is made from photochromic material that temporarily becomes dark
purple under UV radiation. As shown, near UV LED embedded in shoes activate the
color change long enough to create playful patterns.
2 Related Work
Our effort spans two different research areas: projects that aim to enrich human
interaction with physical objects and projects that augment shoes and floors for
sports, entertainment and as user interface.
The History Table Cloth by Gaver et.al. [2] consists of a table top embed-
ded with electroluminescent material. The table senses objects placed on the
table and lights up around the objects for hours visualizing the flow of objects.
They so investigate digital technologies that aim to engage their users in playful
open-ended interactions rather than functional. Like wise, in “Material Traces“
Rosner et.al. [7] discuss the practice of materiality and implications for HCI,
illustrated by a number of provocative ideas, such as paint that traces rodent
movement in the house. These projects relate to our aim, but ideas are proto-
typed on objects. In contrast, the technology proposed in this demonstration
makes these concepts experiential on large scale environments and includes a
practical way of realization. In computer entertainment interactive floors are
made popular with dance dance revolution [4] in which a floor functions as a
input device for a dancing game. Various interactive floors have been proposed.
For instance, the magic floor [5] aims to capture user motion, however require
complex sensor networks. Other projects take reverse approach by embedding
the sensor technology in the shoe rather than environment. Expressive footware
[6] embeds sensors in various ways to capture sport and dance performance.
An early example of a floor as input and output medium is the artwork Bound-
ary Functions by Scott Snibbe [9]. Using a webcam and projector setup, visitor
activity is captured and graphics are projected accordingly. To address the lim-
itations of resolution and occlusion, the multitoe floor [1] aims to build a floor
with multitouch displays. In that way they explore various interaction tech-
niques activated by feet rather than by fingers. However, current display and
projector technologies have various shortcomings when applying for large sur-
face, high resolution or daylight applications. Using mono-stable color changing
materials solves these issues by exploiting the persistence of image as a physical
frame buffer. In that way, no digital representation is involved in the process of
Fig. 2. The indoor shoes have custom soles attached with Velcro. The soles have an
embedded LED array, and contain an accelerometer and pressure sensor for activation.
Dynamic and static patterns are shown on the left.
Because UV radiation is not safe for skin and eye contact, each shoe includes
a safety mechanism driven by an Arduino. A FSR (Force Sensitive Resistor)
detects whether shoe is pressed against a surface, and the accelerometer detects
if the shoe is flat on the floor and without user induced acceleration. These two
conditions have to be met in order to turn on the LEDs.
References
1. Augsten, T., Kaefer, K., Meusel, R., Fetzer, C., Kanitz, D., Stoff, T., Becker, T.,
Holz, C., Baudisch, P.: Multitoe: high-precision interaction with back-projected
floors based on high-resolution multi-touch input. In: Proceedings of UIST 2010,
pp. 209–218. ACM, New York (2010)
2. Gaver, W., Bowers, J., Boucher, A., Law, A., Pennington, S., Villar, N.: The history
tablecloth: illuminating domestic activity. In: Proceedings of DIS 2006, pp. 199–208.
ACM, New York (2006)
3. Hashida, T., NIshimura, K., Naemura, T.: Hand-rewriting: automatic rewriting sim-
ilar to natural handwriting. In: Proceedings of the 2012 ACM International Con-
ference on Interactive Tabletops and Surfaces, ITS 2012, pp. 153–162. ACM, New
York (2012)
4. Konami: Dance dance revolution, http://www.konami.com/ddr (accessed: June 10,
2013)
5. Paradiso, J., Abler, C., Hsiao, K.Y., Reynolds, M.: The magic carpet: physical sens-
ing for immersive environments. In: CHI 1997 Extended Abstracts on Human Fac-
tors in Computing Systems, CHI EA 1997, pp. 277–278. ACM, New York (1997)
6. Paradiso, J., Hsiao, K., Benbasat, A., Teegarden, Z.: Design and implementation of
expressive footwear. IBM Systems Journal 39(3.4), 511–529 (2000)
7. Rosner, D.K., Ikemiya, M., Kim, D., Koch, K.: Designing with traces. In: Proceed-
ings of CHI 2013, pp. 1649–1658. ACM, New York (2013)
8. Saakes, D., Chiu, K., Hutchison, T., Buczyk, B.M., Koizumi, N., Inami, M., Raskar,
R.: Slow display. In: ACM SIGGRAPH 2010 Emerging Technologies, pp. 22:1–22:1.
ACM, New York (2010)
9. Snibbe, S.: Boundaryfunctions,
http://www.snibbe.com/projects/interactive/boundaryfunctions/ (accessed:
June 10, 2013)
Marco Scirea
IT University of Copenhagen
Abstract. Music is one of the most expressive media to show and ma-
nipulate emotions, but there have been few studies on how to generate
music connected to emotions.
Such studies have always been shunned upon by musicians affirming
that a machine cannot create expressive music, as it’s the composer’s
and player’s experiences and emotions that get poured into the piece. At
the same time another problem is that music is highly complicated (and
subjective) and finding out which elements transmit certain emotions is
not an easy task.
This demo wants to show how the manipulation of a set of features
can actually change the mood the music transmits, hopefully awakening
an interest in this area of research.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 626–629, 2013.
c Springer International Publishing Switzerland 2013
about mood information extraction, we applied the principles they worked with
to instead generate music.
2.1 Intensity
Intensity is defined by how strong the volume of the music is; in music with
low arousal we generally have a lower volume than in the ones that have high
arousal.
2.2 Timbre
Timbre is what we could call the brightness of the music: how much of the
audio signal is composed by bass frequencies. In previous literature MFCC (Mel-
Frequency Cepstral Coefficients) and spectral shape features have been used
to analyze this. For example the brightness of Exuberance music is generally
higher than in Depression, this will result in greater spectral energy in the high
subbands for Exuberance. It’s generally a factor that is very dependent on the
instrumentation choice; in our case we act on the synthesizers, our instruments,
to generate brighter and darker sounds.
2.3 Rhythm
Finally Rhythm has been divided in rhythm strength, regularity and tempo [7].
For example in a high valence/high arousal piece of music we can observe that
the rhythm is strong and steady, while in a low valence/low arousal the tempo
is slow and the rhythm cannot be as easily recognized. We act on these features
in different ways. To influence rhythm strength we change how much the drums
are prominent in the music.
Having the notes generators create notes on the beat or the upbeat creates
different feeling of regularity and irregularity, for example in Contentment music
we will favor a steady rhythm with notes falling on the beats of the measure while
in a Depression one we will give more space to upbeat notes. Finally to influence
the tempo we just act on the BPMs (Beats Per Minute) of the music.
2.4 Remarks
We noticed that these features, originally devised to extract mood information,
were enough to generate different moods. But we also realized that we could
strengthen the impression by introducing dissonances in the music: for Exuber-
ance and Contentment we use a diatonic scale while for Anxious and Depression
an alterated one.
3 The System
The demo has been realized using PD (aka Pure Data) [8], a real-time graphical
programming environment for audio, video, and graphical processing. In PD,
programs are written as graphical graphs called patches, in our demo we used
some patches taken from Brinkmann’s website [3].
The generated music is played by 3 instruments (synthesizers) and a drum
machine. The system consists of five random number generators: four of these
numbers will be converted into notes for the instruments and the drums, while
the last one is used as a sound effect controller. Additionally each of the 4 notes
generators also generate another number that will determine the volume the
note will be played.
These generators create semi-random numbers by adding (in decimal) from a
start value a certain step each tick. Then each generator converts the number to
another base and adds the digits of the result. This value will then become the
note we’ll play; at the same time through a slightly different sum we generate
another number that will control the volume the note will be played.
The numbers, before being sent to the synthesizers that will generate the
note, are filtered so that we can control which notes we want and which we
don’t. This allows us to use dissonances or maintain a diatonic feel to emphasize
moods. The numbers at this point represent notes in MIDI (Musical Instrument
Digital Interface) notation (they can span from 0 to 127), so to filter them we just
use a modulo operation to understand what note they represent. For example,
as there are 12 notes in an octave, we can see that 60 modulo 12 equals 0, this
means that 60 represents a C.
At this point the synthesizers generate the note, by converting the MIDI
notation to a frequency and using PD’s built-in audio wave generator. The syn-
thesizers we use are constructed in a way that we can choose the waveform of
the sound, modulation and if we want to transpose octaves.
After the notes are actually generated we have a patch that controls their
volumes so that we can decide if we want some instruments more prominent
than the others: a mixer. Now we have all the notes at the desired intensity but
before playing them we apply some effects.
At the moment we have implemented only the music generation for the four
quadrants of the bi-dimensional space, we plan to expand it so that the music
can be generated by choosing a point in the plane.
[2] before and after interacting with the demo, this will yield important data in
understanding how effective our generator is.
References
1. Baccianella, S., Esuli, A., Sebastiani, F.: Sentiwordnet 3.0: An enhanced lexical
resource for sentiment analysis and opinion mining. In: Proceedings of the 7th
Conference on International Language Resources and Evaluation (LREC 2010),
Valletta, Malta (May 2010)
2. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and
the semantic differential. Journal of Behavior Therapy and Experimental Psychia-
try 25(1), 49–59 (1994)
3. Brinkmann, M.: Pure data patches, http://www.martin-brinkmann.de/
4. Katayose, H., Imai, M., Inokuchi, S.: Sentiment extraction in music. In: 9th Inter-
national Conference on Pattern Recognition, 1988, pp. 1083–1087. IEEE (1988)
5. Kreutz, G., Ott, U., Teichmann, D., Osawa, P., Vaitl, D.: Using music to in-
duce emotions: Influences of musical preference and absorption. Psychology of
Music 36(1), 101–126 (2008)
6. Lindström, E., Juslin, P.N., Bresin, R., Williamon, A.: Expressivity comes from
within your soul: A questionnaire study of music students’ perspectives on expres-
sivity. Research Studies in Music Education 20(1), 23–47 (2003)
7. Liu, D., Lu, L., Zhang, H.-J.: Automatic mood detection from acoustic music data.
In: Proceedings of the International Symposium on Music Information Retrieval,
pp. 81–87 (2003)
8. Puckette, M., et al.: Pure data: another integrated computer music environment.
In: Proceedings of the Second Intercollege Computer Music Concerts, pp. 37–41
(1996)
9. Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psy-
chology 39(6), 1161–1178 (1980)
10. Thayer, R.E.: The biopsychology of mood and arousal. Oxford University Press on
Demand (1989)
Cristina Sylla1, Sérgio Gonçalves1, Paulo Brito1, Pedro Branco2, and Clara Coutinho3
1
EngageLab/CIEd, University of Minho, Portugal
{sylla,sgoncalves,pbrito}@engagelab.org
2
Department of Information Systems, University of Minho; Portugal
pbranco@dsi.uminho.pt
3
Institute of Education, University of Minho; Portugal
ccoutinho@ie.uminho.pt
Abstract. This work discusses a tangible interface for storytelling that targets
pre-school children and offers a playful experimental space where children can
create their own narratives by placing tangible picture-blocks on an electronic
board. We present the system and report on the findings, describing the extent
to which this interface can motivate and engage children, both in creating narra-
tives, as well as in experimenting different solutions to solve conflicts created
during the story plot.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 630–633, 2013.
© Springer International Publishing Switzerland 2013
Taking into account results and observations from previous studies [6] the system
animates the elements on the screen following the order in which the picture-blocks
are placed, and not their position on the board. When a block is removed from the
platform it just disappears from the screen (fig.1).
The story engine was implemented to animate the elements according to the rules
devised from situations that typically arise in traditional children stories. As the ani-
mations play, children can add or remove elements further developing the narrative.
For example, how can children help the little pig to escape from the witch? Maybe
they decide to place a house where the little pig can hide, hum… or ask other charac-
ters for help? Maybe the fairy… but will she be able to defeat the bad witch?
The concept behind the story engine was to model a world that would be understand-
able for young children. The goal is to allow children to bring their knowledge into
play, while fostering their curiosity by playing out different situations where they
might want to figure out what would happen if...
The story engine is implemented on top of the Unity game engine and it is devel-
oped based on behavior trees, a concept well known in the field of computer games to
model character behavior, reactive decision-making and control of virtual characters.
Each entity on the scene has a corresponding behavior tree that defines the actions of
that element. When the users place the blocks on the platform, the behavior tree gets
the inputs of the entities that are present. The behavior triggered for each entity
depends on the other entities that are also in the scene, and the properties of those
entities.
Adittionally, by pressing the Enter Key, the system automatically generates
snapshots of the created narrative, saving them as digital images.
The created images, which resemble comic book representations, are stored on the
computer and can be directly uploaded to a blog or printed and shared with family and
friends, thus extending the way to share stories with other people involving them into
a collaborative storytelling experience (fig. 2).
A qualitative user study was carried with a group of 15 children, during four sessions,
with the duration of one-hour and a half each. The study revealed that one of the
strongest affordances of the platform is the extent to which it promoted collaboration,
showing that the children were highly motivated to engage and collaborate in differ-
ent storytelling activities, exploring different story plots (fig. 3).
By engaging with each other focusing on creating stories together, children may de-
velop interpersonal language use, as they externalize their feelings and thoughts,
learning to express themselves and to communicate with others, with time becoming
more fluent in their language use [1]. The interaction and collaboration supported by
the system creates a micro-world where children socialize with each other, learning to
share, handle, divide, respect and accept the opinions from each other.
Future development involves a version to run on tablets connecting via Bluetooth,
as well as to broaden and diversify different stets of blocks that explore other themes.
Acknowledgments. A very special Thank You to Colégio Teresiano, Braga the pre-
school children and teachers. This work is funded by FEDER funds through the Oper-
ational Competitiveness Factors Program - COMPETE and by National Funds
through the FCT – Portuguese Foundation for the Science and the Technology within
the Projects: PTDC/CPE-CED/110417/2009, FCOMP-01-0124-FEDER-022674 and the
Doctoral Grant: SFRH /BD/62531/2009.
References
1. Ackermann, E.: Constructing Knowledge and Transforming the World. In: Tokoro, M.,
Steels, L. (eds.) A Learning Zone of One’s Own: Sharing Representations and Flow in
Collaborative Learning Environments, pp. 15–37. IOS Press, Amsterdam (2004)
2. Druin, A.: Cooperative inquiry: Developing new technologies for children with children. In:
Proceedings of CHI 1999, International Conference on Human Factors in Computing
Systems, pp. 592–599. ACM Press, New York (1999)
3. Hunter, S., Kalanithi, J., Merrill, D.: Make a Riddle and TeleStory: Designing Children’s
Applications for the Siftables Platform. In: Proceedings of IDC 2010 International Confe-
rence of Interaction Design and Children, pp. 206–209. ACM Press, New York (2010)
4. Morrow, L.M.: Literacy development in the early years: Helping children read and write,
5th edn. Allyn and Bacon, Boston (2005)
5. Paley, V.G.A.: Child’s Work: The Importance of Fantasy Play. Chicago University Press,
Chicago (2004)
6. Sylla, C., Branco, P., Coutinho, C., Coquet, M.E., Škaroupka, D.: TOK – a Tangible
Interface for Storytelling. In: Proceedings of CHI 2011, International Conference on Human
Factors in Computing Systems, pp. 1363–1368. ACM Press (2011),
doi:10.1145/1979742.1979775.
1 Introduction
Shogi [1] is a variant of chess-like games, which is very popular in Japan. Among
all chess variants, shogi is the most complex one because it allows reuse of cap-
tured pieces as well as its board size is 9x9. Shogi has 10226 possible positions,
whereas the western chess has 10120 . Here, a position (in shogi, called kyokumen)
refers to a status of the game board, defined by locations of all pieces and a turn
of the next move (black or white).
Researchers have been developing many AI programs to play shogi and some
of them defeated professional human players in 2013. Today, stronger programs
are not required by most amateur players but they need improved environments
to enjoy shogi.
Playing shogi on the Internet is very popular. For example, a web service
“Shogi Club 24” [2] provides a networked gaming environment and has more
than 200 thousands registered members. However, it does not provide enough
functions for kansousen. Kansousen is a reviewing discussion after a game. In
case of shogi, kansousen is more popularly conducted than other board games
including the western chess or Go. “Shogi Club 24” only provides text chatting
for kansousen. In case of “81 Dojo” [3], which is an international service for shogi,
it has graphical interfaces to discuss alternative moves (variations). However, it
does not support users to organize and save the discussion records.
2 Project SAKURA
We have been developing a network-based environment for shogi, including fea-
tures to support kansousen [4]. It is called SAKURA (Shogi Archives and Kan-
sousen Utilities for Research and Advice) [5].
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 634–637, 2013.
c Springer International Publishing Switzerland 2013
SAKURA Server
Variations
Client Chat
Player B
3 Kansousen Features
In order to realize the kansousen features, we have designed (a) a set of com-
munication protocol for group discussion and (b) a set of user interfaces for
discussion and to append variations (i.e. candidates of alternative moves that
should be taken at a particular position in the game) and comments to any
positions.
Comment for
a Position
Comment for
a Game
4 Conclusion
SAKURA provides unique features to support kansousen of shogi, which have
not been given by other services.
We are currently implementing interfacing software with which AI programs
can give comments to players in the discussion. In the design, we are paying
attention to the amount of comments and timing when they are given by AI.
Basically shogi players like to consider by themselves for training. Hence com-
ments by AI are given when they are explicitly requested by players and when
the players are closing the discussion without noticing very bad mistakes.
References
1. Russel, J., Cohn, R. (eds.): Shogi. Book on Demand Ltd. (2012)
2. Shogi Club 24, http://www.shogidojo.com/
3. 81 Dojo, http://81dojo.com/
4. Tarumi, H., Hiraga, Y., Hayashi, T.: Groupware Support for Kansousen of Shogi.
In: Proceedings of Fifth International Conference on Collaboration Technologies,
IPSJ, pp. 58–59 (2009)
5. Yamamoto, K., et al.: Database Support for Shogi Learners. In: Proceedings of
2012 Fourth IEEE International Conference on Digital Game and Intelligent Toy
Enhanced Learning, pp. 126–128. IEEE (2012)
1 Introduction
Hospitals are not the most hospitable of places at the best of times and for a child
already suffering from the effects of an illness or an injury they can be downright
antagonistic. It is not only the shock of a new and often noisy place and the assort-
ment of people prodding and asking probing questions that is disturbing it is also the
shuttling from one examination room to another via a maze of doors, corridors and
elevators and the often visible nervousness of accompanying family members.
Hospitals have invested a great deal of effort in making the emergency room a
more welcoming environment for children by building separate emergency room
facilities equipped not only with specialist doctors but with colorful walls, dressed up
equipment, and a variety of toys, games and music intended to calm and distract the
child [1,2] however all of these efforts regardless of their rates of success treat the
symptom (agitation and anxiety) and not the cause (fear and unfamiliarity).
Hospital patients (adults as well as children) are anxious and agitated not only due
to the pain and discomfort brought about by their medical condition but also by their
lack of familiarity with the hospital environment, their lack of knowledge of the med-
ical procedures they are about to undergo and at times their mistrust of hospital staff.
Hospital Hero is a game designed to help children familiarize themselves with the
hospital environment, staff and various medical procedures thereby reducing their
anxiety and making them more cooperative. This in turn enables faster treatment and
better experience for the patient, hospital staff and accompanying family members.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 638–641, 2013.
© Springer International Publishing Switzerland 2013
2 Design Rational
Agitation and anxiety brought about by fear and unfamiliarity can be addressed by
making the unknown known and by providing positive feedback. In the “Teddy Bear
Hospital”, Bloch and Toker [2] have shown that inviting children to bring their teddy
bears to a mock hospital and helping them examine and apply treatments to the teddy
bear, e.g. check the teddy bear heartbeat with a stethoscope, apply bandages and ad-
minister injections lower the perceived anxiety of the children about the prospect of
having to go to the hospital and undergo such procedures themselves. Baldwin’s
MindHabits Trainer game [3] has shown that it is possible to develop games that help
people focus on positive feedback such as smiling faces thereby boosting their self-
confidence and reducing anxiety.
Hospital Hero builds on these earlier works by creating a 3D virtual environment
of a hospital through which the player can familiarize himself with the hospital envi-
ronment, staff, patients and equipment. The gameplay consists of finding lost pa-
tients and collecting a set of medical supplies and equipment, portrayed in the form of
cartoon characters and objects through the maze of corridors and rooms that make up
a cartoon version of a hospital. As the player traverses through the corridor of the
virtual hospital they will encounter various characters. Characters with positive ex-
pressions provide clues about the lost patients and medical supplies and equipment.
The characters players encounter in the game serve two purposes. The first purpose
is to familiarize children with the hospital staff and they type of patients they are like-
ly to encounter. The second purpose is to get the players to actively seek out and
focus on smiling faces in a crowd in an effort to gain the same positive effect shown
by Baldwin in MindHabits [4]. Whenever a sub-task is accomplished (e.g. obtaining
a stethoscope), a positive feedback is also provided in the form of a cute animation.
3 Implementation
The game was developed using the Unity game engine for deployment on tablets and
kiosks. Although developed in Portugal, the game is designed to be bilingual and
can be easily ported to other languages. To cover the wide range of kids, the game
requires minimal language ability. Relaxing ambient sounds and positive sound
effects, upon correct movements, are provided.
A short anxiety test, based on the Modified Short State-Trait Anxiety Inventory
and Talking Mats [5], was added before and after the game to test the effectiveness.
The anxiety test was designed as a part of the game to be non-intrusive. Four
expressions (happy, sad, calm and scared) were created and used in pre- and post-
game anxiety tests. A neutral face was also created for the purpose of showing neutral
results. Approximate results are also shown after each test with three different faces
(sad-with high anxiety, happy-with low anxiety and neutral-middle anxiety) as a form
of feedback. All characters, 3D objects as well as the environment were created in
Autodesk Maya.
Fig. 1. Screen captures of the Hospital Hero during language selection at the beginning of the
game session (left), and during pre-game anxiety test (right)
Fig. 2. Shows a corner of the virtual hospital where a kid will encounter avatars of hospital
staffs and patients with either happy faces or sad faces. When touching a avatar with a
happy face, the avatar offers hints as positive feedbacks, whereas a sad avatar will offer neutral
feedback (right).
4 Discussion
Although the game has not been formally deployed to kiosk in the hospital, the proto-
type has already been tested on tablet with university students to identify usability
problems and interface design issues. A revised prototype was also tested with some
children in a local elementary school to gather preliminary results and to evaluate the
experimental design.
For statistic analysis, the game automatically keeps track a player’s activities in a
log file. In addition to the log file, the front camera of the tablet also records the
facial expression of the player. Another regular video camera was also setup to cap-
ture the whole area in order to study player’s behaviors outside of the game. The
Yale Preoperative Anxiety Scale Modified (YPAS-m) test [6] was conducted by a
shadowing experimenter before/after the game to compare and correlate to the results
from the self-report anxiety level of the in-game anxiety test.
Children reported that they enjoyed the game and did learn new knowledge about
the emergency room. The preliminary results indicate some improvements in terms of
anxiety level. However, we do not have enough subjects for the result to be signifi-
cantly meaningful. The test at the elementary school shows that the system and expe-
rimental setup is ready for deployment at the Children’s Emergency Room. We are
positive about the effectiveness of the game once installed. And we are very eager to
deploy the system in the local hospital in order to collect the real world data for our
long-term study.
Fig. 3. A kid with a transplanted kidney playing the kiosk at the Children ER of Funchal Hos-
pital (left), and another kid play-testing an early prototype of the game on an Andriod tablet
(right)
References
1. Weiland, T.J., Jelinek, G.A., Macarow, K.E., Samartzis, P., Brown, D.M., Grierson, E.M.,
Winter, C.: Original Sound Compositions Reduce Anxiety in Emergency Department Pa-
tients: A Randomised Controlled Trial. Medical Journal of Australia 195(11-12), 694–698
(2011)
2. Bloch, Y.H., Toker, A.: Doctor, Is My Teddy Bear Okay? The “Teddy Bear Hospital” As A
Method To Reduce Children’s Fear of Hospitalization. Israel Medical Association Jour-
nal 10(8-9), 597–599 (2008)
3. Baldwin, M.: Mindhabits: Games for a Positive Outlook. Mindbits (2012)
4. Electric Owl Studios. Interactive Kiosks,
http://www.electricowlstudios.com/interactive-kiosks/kick
5. Nilsson, S., Buchholz, M., Thunberg, G.: Assessing Children’s Anxiety Using the Modified
Short State-Trait Anxiety Inventory and Talking Mats: A Pilot Study. Nursing
Research and Practice (2012), doi:10.1155/2012/932570
6. Kain, Z.N., Mayes, L.C., Cicchetti, D.V., Bagnall, A.L., Finley, J.D., Hofstadter, M.B.: The
Yale Preoperative Anxiety Scale: How Does It Comparewith a “Gold Standard”? (1997)
Imagine yourself being a 7 year old child again for one day, playing around and
exploring freely without any limits. Play is considered as an intrinsically motivated
activity with no direct benefit or goal that is situated outside of daily life that triggers
creativity [2]. In our research, we aim at designing interactive playful solutions with
simple interaction rules that enable children to create their own play and games. We
call this open-ended play [1]. In previous research we have focused on designing
open-ended play objects that stimulate both social and physical play [1, 8]. We
experienced that physical high active play has a possible negative effect on the
creativity of the children. In this paper we therefore investigate the relation between
physical play and creativity in open-ended play. How can we design for a proper
balance between the right physical activity level and children still having enough
energy to be creative? An open-ended play installation named Toinggg was
developed, consisting of three interactive trampolines. A change in the interaction
behavior of Toinggg is built in which should support moments of less active open-
ended play to stimulate creativity. This leads to our research question: Can changes in
children’s activity level influence creativity in an open-ended play environment?
The research presented in this paper is related to open-ended and physical play. An
example of free play is FeetUp [4], a playful accessory integrated in a pair of shoes.
Children get audiovisual feedback whenever they jump or are off the ground. They
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 642–645, 2013.
© Springer International Publishing Switzerland 2013
can freely play with FeetUp anywhere and anytime. It is however a wearable play
object that does not specifically focus on the level of creativity. Concerning physical
play, a related design is the Interactive Trampoline [3], which is a large trampoline
with four satellites, sixteen LEDs build-in in the outer curve and a speaker. The
interaction rules of these elements differ per game. Research with the trampoline
studied physical activity, safety and technology to enrich the play value of the
trampoline. The main differences between Toinggg and the Interactive Trampoline
are the number of trampolines and their size. Instead of one large trampoline, Toinggg
consists of multiple intelligent play objects that create a play environment.
2 Design
the level of creativity we scaled the three aspects Fluency (based on the number of
interpretable, meaningful, and relevant ideas generated), Originality (based on the
uniqueness, non-common way of thinking and level of detail) and Enjoyment (based
on the enjoyment of the users during the test) on a scale from 0-7 [6, 7].
Results show that children played enthusiastically with Toinggg in various ways.
They responded quickly to the sounds of the modules, both verbally: “I am an
elephant”, or non-verbally by expressing a smiling face when an animal sound played.
Sounds were interpreted in many ways and sometimes triggered the imagination of
the children (e.g. the mouse sound was interpreted as a chicken or a Tyrannosaurus).
After individual explorations, the children started to play together. This led to games-
with-rules like changing trampolines and guessing which animal you are.
The children tended to play very physical in the first minutes of the sessions. This
could be related to Functional play. Regarding this type of play, the rest moment was
necessary since the children became very tired. Earlier pilot tests showed that in this
case, children tend to become less concentrated and inspired. As can be seen in Figure
2a, the games (specified in different types) changed clearly on the moment of activity
output change. A peak in the amount of new created games can be found during the rest
moment (minute 6) and even more right after it (minute 7). These are both situations in
which the interaction rules changed. Overall the types of play appeared in the following
order: Functional play, Explorative and Dramatic play and Games-with-rules and
Fig. 2. These charts show (a) the average amount of new games created, divided in types of
play (vertical), counted per minute (horizontal) and (b) the average scores and standard
deviation rated per minute of the seven play sessions scored from 0-7. The vertical lines on the
beginning and end of minute 6 and 12 show the sleep interaction moment.
Dramatic play. The second rest moment (minute 12) created less new games compared
to the first rest moment. This can be related to the fact that the users became familiar
with the interaction rules in both situations (rest and active).
The aspects Fluency, Originality and Enjoyment gave a good impression about the
play behavior of the children. This analysis on the quality of gameplay showed a
similar peak compared to the amount of created games. As can be seen in Figure 2b,
the quality of creativity tended to remain after the rest moment. Observations showed
that this occurred since the richness of the games was explored instead of creating a
new game. Enjoyment seemed to grow even more every time the interaction rules
changed which was visible during the sessions; the children understood the game
better and better. The Originality in games slightly decreased over the rest moments,
since the new situation needed to be explored again. This was for the children a
moment to check if everything was still the same: “Let’s check if the animals didn’t
change position during the night!”
Our results show a positive change in amount of creativity after a change of
activity level. However, the actual peak can be found when the rest moment ends and
the initial interaction rules start again. It seems like the rest moment created a ‘reset
button’ for the children to rest and think about the situation. After the rest moment,
the interaction rules were explored again and creativity of children increased. We
believe these insights can help designers in developing engaging playful solutions.
References
1. Bekker, T., Sturm, J., Eggen, B.: Designing Playful Interactions for Social Interaction and
Physical Play. Personal and Ubiquitous Computing 14(5), 385–396 (2010)
2. Huizinga, J.: Homo Ludens: A Study of the Play Element in Culture. Beacon Press, Boston
(1955)
3. Karoff, H.S., Elbæk, L., Hansen, S.R.: Development of Intelligent Play Practice for
Trampolines. In: 11th Int. Conf. on Interaction Design and Children, pp. 208–211 (2012)
4. Rosales, A., Arroyo, E., Blat, J.: FeetUp: A Playful Accessory to Practice Social Skills
through Free-Play Experiences. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque,
P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 37–44. Springer,
Heidelberg (2011)
5. Rubin, K.H.: The Play Observation Scale (POS). Centre for Children, Relationships and
Culture. University of Maryland (2001)
6. Silva, P.A., Read, J.C.: A Methodology to Evaluate Creative Design Methods: A Study with
the Bad Ideas Method. In: 22nd Conference of the Computer-Human Interaction Special
Interest Group of Australia on Computer-Human Interaction, pp. 264–271 (2010)
7. Torrance, E.P.: The Torrance Tests of Creative Thinking: Norms-Technical Manual.
Personnel Press (1966)
8. de Valk, L., Rijnbout, P., Bekker, T., Eggen, B., de Graaf, M., Schouten, B.: Designing for
Playful Experiences in Open-ended Intelligent Play Environments. In: IADIS International
Conference Game and Entertainment Technologies, pp. 3–10 (2012)
9. Zimmerman, J., Forlizzi, J., Evenson, S.: Research through Design as a Method for
Interaction Design Research in HCI. In: SIGCHI Conference on Human Factors in
Computing Systems, pp. 493–502 (2007)
1 Introduction
A sleeping being – animal, human - is intriguing for many children. It is tempting to wake
it up to see how it reacts, but there is the risk of a negative reaction. This was the
inspiration for ZooMor: a ‘sleeping’ zoomorphic object placed in an open play space,
designed to invite children to play with it, moreover, to develop pretend play around it.
With the last question, we position ZooMor as an interactive object that aims to support
open-ended pretend play, like for example [1]. Design for open-ended play aims to
support children in creating their own play, without predefining the rules of play [2].
ZooMor is an object that has much similarity to relational artifacts. Turkle defines
relational artifacts as ‘objects presenting themselves as having “states of mind” that
are affected by their interactions with human beings’ [3].
The challenge of designing for interactive open-ended play is to provide
opportunities to players. This is fundamentally different from for example game
design, in which the play activity is largely predefined. De Valk’s Stages of Play
provides a framework to design for this open-ended play. In this framework, an
interactive experience in play goes through three stages. First, a potential player must
be attracted to start playing. This first stage is called the invitation phase. Once a
player is attracted to the play design, she or he explores opportunities for interaction
and play in the ‘exploration stage’, wanting to understand aspects such as rules and
affordances. Once a player has acquired sufficient understanding of possibilities for
play, immersion in play may occur: the ‘immersion stage’.
2 Design Iterations
ZooMor was designed to be a zoomorphic creature, its form giving leaving room for
interpretation in order to support open-ended pretend play. Its basic form was
symmetric, with a ‘leg’ part at front and back side, and a heightened back. The form
allowed children to sit on it and crawl underneath it.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 646–648, 2013.
© Springer International Publishing Switzerland 2013
In total three design iterations have been done. All iterations were completed with
a field test in ‘De Ontdekfabriek’, an open play environment. Children from ages
ranging from 3 to 9 could enter freely to play. ZooMor was available for play for
about two hours per study. In every test, 5 to 10 children played with the prototypes
for 5 to 30 minutes. Evaluation of play was done in two ways: analysis of video and
semi-structured short interviews with the children leaving the place. The interactivity
of ZooMor was tested with the Wizard of Oz technique. [4A]
In the first design iteration, emphasis was on the first stage of play, invitation [5].
Applied to ZooMor, it had to sleep but arouse curiosity of children nearby. It is known that
life-like sounds like pulse, breath, snore or cough are strong triggers of zoomorphic
perception [5]. To invite children to interact, we chose the sound of snoring.
The experimenter followed a strict protocol in controlling ZooMor’s behavior. The
initial state in all tests is asleep. After 5 seconds of sleep, it starts snoring loudly. It
wakes up if a child does at least one of the following things: crawl under the belly,
climb on top of it, stroke or pet it, or kick or hit it. It does not react to sound. When it
wakes up, it stops snoring and to give further feedback to the child a blue light under
the belly is switched on. When no one is touching ZooMor for more than 10 seconds,
it will go back into the asleep-state.
The snoring did invite children to come to ZooMor and start playing with it.
Interaction comprises petting, lying down next to it, and sometimes more aggressive
interactions like hitting, kicking, hopping on top of it and shouting occurred. The
sleeping state was very well understood; the awake state with the blue light was not.
Children referred to ZooMor as “sleeping beast”, “giant”, “horse”. One child did not
dare to enter the space because there was “a monster sleeping”.
In the second iteration, the emphasis was on De Valk’s exploration phase. Several
changes were made to the design. First, the poorly understood blue light was replaced
by abstract eyes on one side of the form, to appeal more to zoomorph identification.
Second, the interaction opportunities were extended somewhat: the possibility to
‘wiggle’ the back perpendicular to the spine of ZooMor. This movement could for
example lead to rubbing, stroking, shaking or wiggle while sitting on top of it. The
wiggling movement was also defined as an input that could wake up the creature,
when repeated at least 3 times. Finally, a second ZooMor was added to the scene,
having the same affordances, but in a different color. These changes were successful
in invoking a stronger zoomorphic perception: children were talking about the two
creatures as living things frequently, and started having “inappropriate expectations”
[6]: attributing capabilities to them they did not have. The second design did stimulate
children effectively to explore ZooMor’s behavior. It was not however interesting
enough to create a long lasting and playful experience.
Therefore, the third iteration focused on De Valk’s immersion stage. We aimed for
richer behavior than be awake or asleep, to elicit more variety in play. This was done
through a redesign of the eyes, Fig. 1. These now could be anything in between fully
open and fully closed. The expectation was that this would allow for perceptions like
drowsy, tired, dreamy, and so on. We indeed observed richer and more lasting play,
children assigning meaning to the intermediate states of the eyes.
3 Conclusion
References
1. Beukering, A., De Valk, L., Bekker, M.M.: Wobble: How an Open-ended Play Environment
Supports Different Levels of Social Play. In: Reidsma, D., Katayose, H., Nijholt, A. (eds.)
ACE 2013. LNCS, vol. 8253. Springer, Heidelberg (2013)
2. Bekker, M.M., Sturm, J.: Stimulating Physical and Social Activity through Open-Ended
Play. In: Interact 2009, Uppsala, Sweden, pp. 952–953 (2009)
3. Turkle, S., Breazeal, C., Dasté, O., Scassellati, B.: Encounters with kismet and cog:
Children respond to relational artifacts. In: Digital Media: Transformations in Human
Communication (2006)
4. Green, P., Wei-Haas, L.: The Rapiud Development of User Interfaces: Experience with the
Wizard of Oz Method. In: Proceedings of the Human Factors and Ergonomics Society
Annual Meeting (1985)
5. Schmitz, M.: Concepts for Life-Like Interactive Objects. In: Proceedings of the 5th International
Conference on Tangible, Embedded, and Embodied Interaction, pp. 157–164 (2011)
6. DiSalvo, C., Gemperle, F.: From Seduction to Fulfillment: The Use of Anthropomorphic
Form in Design (2003)
Harko Verhagen, Mirjam Palosaari Eladhari, Magnus Johansson, and Joshua McCoy
1 Introduction
From the beginning of digital games, AI has been part of the main idea of games
containing acting entities, which is to provide the player with “worthy” opponents (NPCs).
The development of multiplayer games has increased the demands put on the NPCs as
believable characters, especially if they are to cooperate with human players. However, the
social aspect of intelligent behaviour has been neglected compared to the development and
use AI for e.g. route planning. In particular, the interplay between intelligent behaviour
that is task-related, the emotions that may be attached to the events in the game world and
the social positioning and interaction of deliberating entities is underdeveloped. This
workshop aims to address this deficiency by putting forward demonstrations of work in
the integration of different aspects of intelligent behaviour, as well as models and theories
that can be used for the emotional and social aspects.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 649–652, 2013.
© Springer International Publishing Switzerland 2013
lists the following: 1) Buy, sell and make stuff. 2) Provide services. 3) Guard
places.4) Get killed for loot. 5) Dispense quests. 6) Supply background information.
7) Do stuff for players. 8) Make the place look busy.
From one perspective the goal of games AI is to support game designers in
providing players with a compelling game experience that supports interactivity and
player choices and adds to replay ability (Bailey and Katchabaw, 2008). The current
way in which people play games might indicate either that players do not easily
become immersed in the games they play or that they play games differently today:
"As more and more people actually finish fewer and fewer games, the opportunity to
extend the life of games becomes important." (Consalvo, 2007, p. 62)
Game research has a growing interest in NPCs. In (Afonso and Prada, 2008) social
relationships between NPCs are seen as the most important for improving the gaming
experience of players. Merrick and Maher (2006) discuss the seemingly static
representation and behavioural repertoire of NPCs in cases where the game world
constantly changes (MMOGs), changes that should also be reflected in the NPCs in order
to be believable, a view that is shared by Lankoski and Björk (2007). One view that is
shared by all referenced sources in this section is that ‘Believability is a basic requirement
for non-player characters of videogames’ (Gomez-Gauchía and Peinado, 2006, p.1).
Game developers develop different aspects of NPCs in new directions as well. In
L.A Noire (Team Bondi, 2011) NPCs' facial expressions add both immersion and
believability, and the behavioural repertoire of the NPCs in Skyrim (Bethesda Game
Studios, 2011) is an example of complex NPC behaviour as are the strategic elements
of the NPCs in Rage (id Software, 2011).
If we take a closer look at the Oz group's list of requirements for believability, we can
see that there are many common traits compared with what other researchers have
reported as important aspects of believability, and this coincides with Carley and
Newell’s (1994) fractionation matrix (Figure 1).
The work of Lankoski and Björk has many overlaps with the Oz group, particularly
as regards the following traits: emotional attachment, contextual conversational
responses, and goal-driven personal development, where the interpretation might
differ slightly but indicates the necessity for emotions, character development/change
and social aspects on an individual plane. Most other traits as identified by Lankoski
and Björk (2007, p.1) fall under the category of the Oz Group's ‘the illusion of life’.
One thing that stands out is that whereas the Oz group details the individual
behaviour of believable characters the social fractionation matrix has little to add to
our understanding of believability in individual characters. The social fractionation
matrix does however add a lot to our understanding of the fine-grained details about
the social believability of and interactions between characters, adding an important
focus to the social dimension. Figure 2 depicts the relation between these sources,
where the theories from the Oz –group (Loyall, 1997, pp. 15-26; Mateas, 1999) end
up in the ‘individual NPC’ part of the figure, accompanied by the overlaps identified
in Lankoski and Björk (2007). Lankoski and Björk also touch upon ‘narrative’, a
subject that has also been studied in depth by Mateas and Stern (2003). It is obvious
that these theories deal with different parts of Figure 2. Little attention, however, is
paid to how the behaviour of groups of NPCs influences immersion, indicating a gap
that needs to be discussed further.
In summary, we look forward to the workshop to discuss the matters outlined here
as well as other topics relevant to increased social believability in games.
References
Afonso, N., Prada, R.: Agents That Relate: Improving the Social Believability of Non-Player
Characters in Role-Playing Games. In: Stevens, S.M., Saldamarco, S.J. (eds.) ICEC 2008.
LNCS, vol. 5309, pp. 34–45. Springer, Heidelberg (2008)
Bailey, C., Katchabaw, M.: An Emergent Framework. In: FuturePlay 2008, Toronto, Ontario,
Canada, November 3-5 (2008)
Bartle, R.: Designing Virtual Worlds. New Riders, Indianapolis (2003)
Bates, J.: The Role of Emotions in Believable Agents (1994)
Bethesda Game Studios. Skyrim (PC). Bethesda Softworks (release date November 11, 2011)
Carley, K.M., Newell, A.: The nature of the social agent. Journal of Mathematical
Sociology 19(4), 221–262 (1994)
Consalvo, M.: Cheating: Gaining advantage in videogames. The MIT Press, Cambridge (2007)
Gómez-Gauchía, H., Peinado, F.: Automatic Customization of Non-Player Characters Using
Players Temperament. In: Göbel, S., Malkewitz, R., Iurgel, I. (eds.) TIDSE 2006. LNCS,
vol. 4326, pp. 241–252. Springer, Heidelberg (2006)
Id Software. RAGE (PC). Bethesda Softworks (release date October 4, 2011)
Lankoski, P., Björk, S.: Gameplay design patterns for believable non-player characters. In:
Proc. 3rd Digital Games Research Association International Conference (2007)
Loyall, A.B.: Believable Agents. Ph.D. thesis. Tech report CMU-CS-97-123, Carnegie Mellon
University (1997)
Mateas, M.: An Oz-Centric Review of Interactive Drama and Believable Agents. Carnegie
Mellon University, Pittsburgh (1999)
Mateas, M., Stern, A.: Façade: An Experiment in Building a Fully-Realized Interactive Drama.
In: GDC 2003 (2003)
Merrick, K., Maher, M.L.: Motivated Reinforcement Learning for Non-Player Characters in
Persistent Computer Game Worlds. In: ACE 2006, June 14-16 (2006)
Team Bondi, Rockstar Leeds, Rockstar Games. L.A. Noire (PC)
Abstract. This workshop deals with the potential that entertainment systems
and games hold for the transportation context. Travelling by car, bus, plane or
by foot can be frustrating and full of negative experiences, but also holds great
potential for innovative entertainment application. New off the shelf technology
offers great potential beyond old-fashioned rear seat entertainment systems with
the sole purpose of keeping kids quiet. The richness of contextual factors and
social situations have so far not sufficiently been exploited, which is why this
workshop aims at discussing potentials for gaming in transportation.
1 Introduction
Since the early days of mankind, games were used to reduce stress, support
amusement, strengthen social ties and simply to have a good time. All of this is often
missing in cars and transportation although especially when we are on the move,
playing a game might cheer us up or relive us from boredom.
Working towards gaming in transportation this workshop is a forum for researchers
and practitioners interested in games & entertainment approaches for the
transportation and automotive domain. We believe that this is an emerging field, since
entertainment technology has the potential to not only entertain travellers but also to
reduce frustration, aggression, and foster positive behavior in all travelling situations.
Games or Gamification can also help shaping the traveller’s behavior for the better
by, for example, keeping them of from driving during rush hour or using less fossil
energy getting from A to B. The potential for entertainment in transportation is also
growing, as (semi) autonomous vehicles will, for example, free up time for drivers
and make opportunities for interactive technology even larger. The potential for
entertainment in public transportation is also great for it being a social place. Gaming
can change the way we travel, make it more fun but also have a societal impact by
directing travelers’ behavior.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 653–655, 2013.
© Springer International Publishing Switzerland 2013
2 Motivation
Although the potential for games in transportation is obvious, there are challenges as
well as opportunities that distinguish gaming in transportation from gaming in other
contexts. Gaming in transportation cannot be like gaming in the living room, but it
also should not be the same. Designing successful entertainment applications for
transportation requires a solid body of research as well as creativity making gaming in
the transportation context a fun experience.
The context of playing games in transportation is highly influential on the
experience that the game elicits. When travelling between locations, the context the
game is played in, changes. Some of these changes affect what can be called
environmental context. Changes in light for example will occur, the landscape moves
past the player and although often not moving the body itself, players move through
an environment with different speeds [1].
Also to consider for games in transportation is the social context. Other travellers
can be a source of annoyance when, for example, space is limited during rush hour.
On the other hand, the presence of other players can allow flexible multiplayer games
that are more entertaining than a single player game can be. But there are still issues
to be solved, for example privacy concerns when playing games in public places with
strangers where we cannot keep our identity hidden and simply log out as it is
possible in online gaming.
Apart from that the space that the game is played in can create very distinct
situations. Playing games in the car requires the usage of a very limited space, most of
which is not reachable when children are seated in child seats. Sharing a space like
the car cabin also holds potential for conflict, since not all passengers might be
interested in listening to the same sound that the game for one player creates [5].
Also in matter of time, transportation is different from playing games in living
rooms or other more traditional contexts. Travel duration is flexible and thus needs an
adaption of the game. Thinking of multiplayer games in public transport, players need
to have the opportunity to join an ongoing game but also to leave it at every moment
when they have reached the stop to get of.
From a more technology driven perspective, the transportation context is rich of
data starting, for example timetables that are public available in many countries
through open data initiatives. Also cars offer a vast amount of sensor data that can
also serve as input for games and related applications. Even when walking we create
data that can be used for meaningful and entertaining games. The challenge is to use
this data in games and therefore make them unique for the context they are played in.
In a way, the transport modality can be seen as controller of future games. This
inclusion of transportation data and the context has the potential to make games
meaningful but needs to be explored further.
This non-exhaustive list of challenges gives a first impression on research
questions and topics that entertainment researchers face in the transportation context.
Solutions must be found on how to use these very special characteristics of the
transportation context and make them a feature in entertainment applications. It is
necessary to create a better understanding of what makes a good game for the
transportation context and which strategies of game design can be applied for games
while travelling. Travelling on the daily commute, for example, is very repetitive and
thus causes boredom [2]. This is an aspect that a game has to take into account so that
a context dependent game, for example, does not get boring simply because the
traveller passes by the same place every day.
While there are promising approaches for gaming in transportation such as
exergames for playing in the tramway [4], more has to be done to use the potential for
gaming when people are on the move.
3 Workshop Objectives
References
1. Brunnberg, L., Juhlin, O.: Keep your eyes on the road and your finger on the trigger -
designing for mixed focus of attention in a mobile game for brief encounters. In: Fishkin,
K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968,
pp. 169–186. Springer, Heidelberg (2006)
2. Obrist, M., Wurhofer, D., Krischkowsky, A., Karapanos, E., Wilfinger, D., Perterer, N.,
Tscheligi, M.: Experiential perspectives on road congestions. In: CHI 2013 Extended
Abstracts on Human Factors in Computing Systems, CHI EA 2013. ACM (2013)
3. Sundström, P., Wilfinger, D., Meschtscherjakov, M., Tscheligi, M., Schmidt, A., Juhlin, O.:
The Car as an Arena for Gaming. In: Adjunct Proc. MobileHCI 2012 (2012)
4. Toprak, C., Platt, J., Mueller, F.: Designing Digital Games for Public Transport. In: Fun and
Games: Extended Proceedings of the 4th International Conference on Fun and Games,
pp. 29–31. IRIT Press (2012)
5. Wilfinger, D., Meschtscherjakov, A., Murer, M., Osswald, S., Tscheligi, M.: Are we there
yet? A probing study to inform design for the rear seat of family cars. In: Campos, P.,
Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part
II. LNCS, vol. 6947, pp. 657–674. Springer, Heidelberg (2011)
Abstract. Kabuki was a popular entertainment in the Edo period in the 19th
century from the 17th century. Kabuki in the Edo period was different from
standing position as a traditional performing arts in modern. Methods of ma-
thematical models in Nowadays has been selected topics in SNS through the In-
ternet, they do not exist in the Edo period. In order to perform the calculation
and measurement of popular artists of the Edo period, we were subject to
publication of the paper medium is the only media at the time (Ukiyoe, poem,
haiku,book) to there. With this approach, the study of past popular actor,
with respect to the reputation of masterpiece can be also from the perspective of
current.
*
Corresponding author.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 656–659, 2013.
© Springer International Publishing Switzerland 2013
Fig. 1. We went to the English translation further the modern translation of the word green line
of ancient documents in the figure. Process of a modern translation of this ancient document is
part of an important method of this study. The following text is an English translation of the
statement process. In the capital, all take their fashion cues from kabuki-particularly those of
high rank, but even the lower orders look to these to pattern their clothing, to the point even of
learning how to tie their obi belts. "From Onnachō Hōki ("A Treasury for Women"), by Sunbo-
kushi Kusada (aka Jōhaku Namura), 1692 Fig.4 from National Diet Library.
Kawaraban and the number of stage performances described in the rankings. The
database used to investigate Edo period trends comprised approximately 630000 pub-
lications from 1855 to 1849 owned by museums and individuals, and held in libraries
and universities in Japan and overseas. [1,2] Data duplications and errors are pre-
vented by applying a numerical value for human operators. We will show the data
which we collected in this study on Web by cooperation of Professor Hideaki Takeda
of National Institute of Informatics in future.[5] The period selected for the study is of
historical importance, covering natural disasters such as the Ansei earthquake (in
which many popular actors perished)and other highly topical events. [3-5]
Fig. 2. Namazu-e
The catfish picture (Fig. 2) was published in large quantities during the great Ansei
earthquake. Therefore, to investigate the number of different Ukiyo-e themed publica-
tions, we must focus on the persons or items of the Edo period. We identified popular
Edo-period Kabuki actors by a variable called “actor reputation mentioned.” The
quality of actors at that time was recorded in Osaka, Kyoto, and Edo Kabuki publica-
tions. However, because the content of the performance differed from one locality to
another, the reputation rankings of Osaka, Kyoto and Edo were biased by their re-
gional differences. Therefore, we restricted our study to Edo regional literature and
the most commonly found relics of that time, Ukiyo-e.[1-5]
In this theory, we present the equation to calculate the intention which is the inten-
tions for people in the society for a certain entailment in Edo. In the case of this paper,
the intention to each actor of Edo Kabuki. Based on the observation of the hit pheno-
mena in Japanese market, we present a theory to explain and predict hit phenomena.
First, instead of the number of people N(t), we introduce here the integrated intention
of individual customer in Edo, Ji(t) defined as follows,
(1)
Kabuki are the very important factor to increase the intention of each person for enjoying
entertainment in Edo. Kabuki are done at 3 Hall in 1849-1855.[5,7,8]We consider the
advertisement effect as an external force term A(t) to the intention as follows,
∑ ∑ ∑ (2)
where is the factor for the direct communication and is the factor for the indi-
rect communication. The factor Cadv corresponds to the strength of the impression
of the media or Concerts exposure for each advertising campaign. Because of the
term of the indirect communication, this equation is a nonlinear equation. If we con-
sider Kabuki, we can solve the equation by including several effect of the advertise-
ment in the following way,
∑ (3)
where each coefficient corresponds to the each advertisement and Concert's counts
and can be determined by using the random number technique introduced in Ref. [6].
The actual formula used in the calculation to analyze the Edo prints response are in-
troduced in detail in Ref.[6-8]
3 Result
Fig. 3. Plot the results for Danjuro Ichikawa. Fig.3 The number of documents pertaining to
Danjuro Ichikawa as a function of time.[4-8].
4 Conclusion
We have been able to use exactly the same mathematical model to analyze and predict
hit phenomena in the past and present as well as the reputation of the artists. While
problems remain with our method in the sense that it is difficult to say that our sample
numbers are complete, we invite the cooperation of volunteers and readers interested
in our study’s attempt, in the hopes that this will lead to the discovery of a new histor-
ical understanding, and that the application of this technique elsewhere in the world
might lead to the appearance of new discoveries and fruitful suggestions. We expect
that in future, the potential of past media for the study of archeology will be unlocked,
leading to a new point of departure for research in the field of archeological research.
References
1. Gerstle, C.A.: Flowers of Edo: Eighteenth-Century Kabuki and Its Patrons. Asian Theatre
Journal 4, 52–75 (1987)
2. Leiter, S.L.: A Kabuki reader: History and performance, pp. 3–359. ME Sharpe Inc. (2002)
3. Edo Kabuki Ukiyo-e for Bigdata database, http://data.linkedopendata.jp/
storage/f/2013-08-30T04%3A47%3A01.086Z/edokabukirefernces.pdf
4. Library and Art Gallery and Museum and database of Ukiyo-e subject of investigation,
http://data.linkedopendata.jp/storage/f/2013-08-
30T05%3A05%3A52.332Z/library-and-art-gallery-
and-museum-and-database-of-ukiyo.pdf
5. Matsumura, F., Kobayashi, I., Kato, F., Kamura, T., Ohmukai, I., Takeda, H.: Producing
and Consuming Linked Open Data on Art with a Local Community. In: Sequeda, J.F.,
Harth, A., Hartig, O. (eds.) Proceedings of the Third International Workshop on Consuming
Linked Data (2012)
6. Ishii, A., et al.: The ‘hit’phenomenon: A mathematical model of human dynamics interac-
tions as a stochastic process. New Journal of Physics 14, 063018 (2012)
7. Kawahata, Y., Genda, E., Ishii, A.: Revenue Prediction of Music Concerts Using the Ma-
thematical Model of Hit Phenomena. In: ICBAKE 2013 (unpublished 2013)
8. Ishiil, A., Ota, S., Koguchi, H., Uchiyama, K.: Quantitative analysis of social popularity for
Japanase pop girl group AKB48 using mathematical model for hit phenona. In: ICBAKE
2013 (unpublished 2013)
In first-person shooting games players interact with virtual worlds through multiple
modalities (first-person graphics, spatial sound, movement) from the perspective of a
virtual character, with the aim of enabling the player to experience the game as the
character. While creating a first-person gaming experience it is not entirely feasible to
cause pain to a player when the character receives an injury, and so games attempt to
replicate aspects of getting an injury without actually causing pain. In this paper we
explore how damage indication methods impact players emotional and cognitive ex-
perience of gameplay, rather than other metrics such as task efficiency (e.g., best
score) – an approach termed “affective ludology” [8]. From this perspective, building
player immersion and a sense of presence in the virtual world (i.e., total immersion
[2]) is an integral part of a successful gaming experience. In this work, we compare
three different damage indication methods in terms of how they impact gaming expe-
rience – this extended abstract is only a summary of the work [10].
Damage indicators are integral to many video games and have a long history, rang-
ing from using abstract health-point systems (such as in ID Software’s Doom fran-
chise), toward more modern and more-realistic methods such as hindering character
senses (vision and audio, as with Activision’s Call of Duty) or movement (as in Ion
Storm’s Deus Ex). Some games add additional information including from which
direction an injury came from (e.g., behind) using, for example, a simple arrow (as in
343 Industry’s Halo 4). We found very little work in the research community on dif-
ferent first-person shooter damage indication methods.
Evaluation of a person’s affective state, and correlating it with measures of immer-
sion and enjoyment, are still active research problems with various facets ranging
from qualitative analysis of written questionnaires and interviews [3], applying heu-
ristics [4], administering subjective questionnaires [1], or using a whole range of bio-
metric and psychometric assessment methods [9]. We draw from this work and apply
some of these methods in our study.
D. Reidsma, H. Katayose, and A. Nijholt (Eds.): ACE 2013, LNCS 8253, pp. 660–664, 2013.
© Springer International Publishing Switzerland 2013
2 Damage Indicators
We investigated three damage indicators, a red flash, paper doll, and X-ray: the red
flash indicator tints the player’s screen red upon injury (Fig. 1, left); the paper doll
indicator places a cut-out character silhouette at the top-left of the screen, flashing the
screen background and associated body part red upon injury (Fig. 1, middle); the nov-
el X-ray indicator overlays X-ray images of the injuries in semi-transparent red, indi-
cating injury to either head, torso, an arm, or legs (Fig. 1, right, and Fig. 2). In all
cases the opacity of the tint or flash is proportional to the severity of the injury. Thus,
the three indicators form points on a range from less immersive design (red flash),
more immersive intention due to providing first-person information (paper doll), and
yet more immersive design due to the nature of presentation (X-ray).
Fig. 1. We compared the three indicators: a red flash (left) indicating an injury was incurred, a
paper doll (middle) with red flash providing further information on where the character was
injured (in the right arm), and an X-ray (right) showing that the character was hit in the left arm.
Fig. 2. The X-ray films used to show injury – left and right hands, head, torso, and foot
3 Study
The post-condition questionnaire asked the participant to rate “how you felt when
your character took damage” and “how you think your character felt when taking
damage,” using the standard Self-Assessment Manikin [7] instrument. We adminis-
tered the Game Experience Questionnaire [6] and further asked how “tough” they felt
the character was and how “strong were the enemies,” and finally asked them to rank
how much they liked the particular damage indicator. This was followed by open-
ended written questions regarding general comments and how each indicator
impacted the perception of character damage, the enjoyability of the game, and the
player-character relationship.
4 Results
Participants reported the additional injury information provided by the X-ray and
paper doll indicators was useful and caused them to think more about their characters,
and their reports strongly suggest that this information, and how it was presented,
contributed to their immersion. They also explicitly related the indicator to the “feel-
ing” of receiving an injury, and for both indicators, talked a great deal more about
character injuries than they did with the red flash. Finally, this finding correlates with
how participants found enemies to be stronger with the red flash and weaker with the
others, suggesting how the immersion can relate to quality of play or even perception
of such. One surprising result was that immersion may actually hinder gaming expe-
rience, rather than improve enjoyability (as in [2]): some participants found the inte-
raction to be too real in the X-ray case, which made them feel bad for their character
and guilty, and some found the pain-type immersion to be demotivating.
The results of our new X-ray indicator were encouraging. There were many signs
of immersion: participants used affective language, talked about feeling the pain, and
talked more about their character’s injuries in comparison to the red flash. In addition,
a majority share rated the X-ray as their favorite, there was a great deal of positive
feedback, and participants wrote more about the X-ray indicator than the others. Par-
ticipants who did not like the indicator primarily cited the obstruction of vision, not-
ing it made them? feel mechanically hindered instead of being injured. This is an
important point to note, as the obstruction was a deliberate design decision intended
to simulate the loss of senses when in pain. This tells us that designing the obstruction
of senses has to be tactfully done to fit well within the game.
From our analysis and results we propose the following recommendations:
Damage indicators have a strong impact on gameplay and Immersion. Small dam-
age indicator design changes can have a large impact on player experience and im-
mersion, so consider this aspect of game design carefully.
Immersion has many dimensions. Increasing immersion in ways that clash with the
game design may have negative experience effects, as X-ray may be too realistic.
Players can reasonably discuss aspects of immersion. Participants were clear and
insightful about their experiences of immersion and gameplay, supporting self-report
as a useful means for future studies.
References
1. Brockmyer, J.H., et al.: The development of the Game Engagement Questionnaire: A
measure of engagement in video game-playing. J. Exper. Social Psych. 45(4), 624–634
(2009)
2. Brown, E., Cairns, P.: A grounded investigation of game immersion. In: Extended abst. of
the 2004 Conf. on Human Factors and Computing Syst., CHI 2004, p. 1297. ACM Press,
New York (2004)
3. Desurvire, H., et al.: Evaluating fun and entertainment: Developing a conceptual frame-
work design of evaluation methods. In: Proceedings of CHI 2007. ACM (2007)
4. Desurvire, H., et al.: Using heuristics to evaluate the playability of games. In: Extended
abst. of the 2004 Conf. on Human Factors and Computing Syst., CHI 2004, p. 1509. ACM
Press, New York (2004)
5. Ermi, L., Mäyrä, F.: Fundamental components of the gameplay experience: Analysing
immersion. Changing Views: Worlds in Play. Selected Papers of the 2005 Digital Games
Res. Assoc.’s Internat. Conf., 15–27 (2005)
6. Ijsselsteijn, W.A., et al.: The game experience questionnaire: Devel. of a self-report meas-
ure to assess player experiences of digital games. FUGA techn. report, deliverable 3.3, TU
Eindhoven (2008)
7. Morris, J.: Observations: SAM: the Self-Assessment Manikin; an efficient cross-cultural
measurem. of emotional response. Jrnl. of Advertising Res. (December 1995)
8. Nacke, L., Lindley, C.A.: Affective ludology, flow and immersion in a first-person shoo-
ter: Measurem. of player experience. The Jrnl. of the Canadian Game Studies Assoc. 3(5)
(2009)
9. Nacke, L.E.: Affective Ludology: Sci. Measurem. of User Experience in Interactive Enter-
tainment. Blekinge Inst. of Technol. (2009)
10. Young, J.E., Shahin, I., Nakane, M.: Embodied damage indicators in first-person shooting
games. University of Manitoba technical report HCI2013-03 (2013),
http://mspace.lib.umanitoba.ca//handle/1993/22118