Facial Expression Recognition Teaching to Preschoolers

with Autism: A Natural User Interface approach

Eirini Christinaki Nikolas Vidakis Georgios Triantafyllidis
TEI of Crete TEI of Crete Aalborg University
echrist@ics.forth.gr vidakis@epp.teicrete.gr gt@create.aau.dk

ABSTRACT patterns of behavior, interests and activities. Children with the

The recognition of facial expressions is important for the same diagnosis of autism may exhibit different symptoms and may
perception of emotions. Understanding emotions is essential in demonstrate markedly different behaviors and skills. The spectrum
human communication and social interaction. Children with autism of symptoms can range from mild to severe and efficient diagnosis
have been reported to exhibit deficits in the recognition of is difficult. Autism is also described as an Autistic Spectrum
affective expressions. Their difficulties in understanding and Disorder (ASD) due to the variability with which the disorder is
expressing emotions lead to inappropriate behavior derived from manifested. In this article autism and ASD will be referred
their inability to interact adequately with other people. Those interchangeably.
deficits seem to be rather permanent in individuals with autism so Social interaction impairments, a core feature of ASD, involve
intervention tools for improving those impairments are desirable. difficulties in understanding and expressing emotions [2]. Social
Educational interventions for teaching emotion recognition from interaction is the mutual influence between individuals, during
facial expressions should occur as early as possible in order to be which people exchange verbal and nonverbal messages, in order to
successful and to have a positive effect. It is claimed that Serious provide and receive information for themselves and for the others.
Games can be very effective in the areas of therapy and education It is a process of production and exchange of signs that usually
for children with autism. However, those computer interventions affects the behavior of people involved in it. Marked impairments
require considerable skills for interaction. Before the age of 6, in the use of nonverbal behaviors such as facial expressions,
most children with autism do not have such basic motor skills in according to DSM-IV-TR, can be responsible for qualitative
order to manipulate a mouse or a keyboard. Our approach takes impaired social interaction. Nonverbal communication is the
account of the specific characteristics of preschoolers with autism expression of emotions, moods, attitudes and the general inner
and their physical inabilities. By creating an educational computer world expressed through the body. It constitutes the largest part of
game, which provides physical interaction by employing natural a communication process and facilitates clarification between
user interface (NUI), we aim to support early intervention and to communicators to define the possible role that each of them has at
foster facial expression learning. each moment during this transaction. Source of nonverbal signals
is the human body and particularly the face, the eyes, the postures,
Categories and Subject Descriptors the body movements and the gestures.
I.4.9 [Image Processing and Computer Vision]: Applications;
K.3.1 [Computers and Education]: Computer Uses in Education; 2. FACIAL EMOTION RECOGNITION
H.5.2 [Information Interfaces and Presentation]: User
Interfaces– natural language, user-centered design. 2.1 Children with Autism
Facial expressions give important clues about emotions and
General Terms provide a key mechanism for understanding, identifying and
Performance; Design; Experimentation. conveying them. Facial expressions underline the feelings of
emotions. Children with autism often fail to recognize the
qualitative differences and associations between various
Keywords expressions of emotions [3]. Due to limited social and emotional
Autism; Facial emotion recognition; Gesture-based interaction; understanding they do not know how to adequate interact with
Kinect; Natural User Interface. other people which sometimes leads to inappropriate behaviors.
Studies have reported that young children with autism experience
1. INTRODUCTION difficulties recognizing expressions and adults with autism are not
Autism, according to the Diagnostic and Statistical Manual of as good as typically developing adults at emotion recognition [4].
Mental Disorders [1], is a Pervasive Developmental Disorder It has also been recorded that individuals with autism are
(PDD) characterized by impairments in social interaction, in significantly impaired and exhibit heterogeneity in their emotion
communication and by restricted, repetitive and stereotyped processing [5], [6], have difficulties in recognizing emotions from
the upper part of the face [7] and exhibit a deficit in overall
emotion recognition from facial expressions [8]. Toddlers with
Permission to make digital or hard copies of all or part of this work for autism focus their attention on a single facial feature and treat
personal or classroom use is granted without fee provided that copies are
people as objects [9]. Visual scanning of faces in autism revealed
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
an increased visual fixation time on mouth region and a significant
otherwise, to republish, to post on servers or to redistribute to lists, less time on eyes region that was associated with impairment in
requires prior specific permission and/or a fee. daily social interaction [10]. There are also evidence that
BCI’13, September 19-21, 2013, Thessaloniki, Greece. individuals with autism and typically developing individuals
Copyright 2013 ACM 978-1-4503-1851-8/13/09 ...$15.00.

141
decode facial expressions through different mechanisms [11]. A and not for effecting changes. Social stories about feelings and
recent study was conducted to evaluate verbal and perceptual skills appropriate emotional responses should provide information about
implicated in the recognition of facial emotions. Individuals with feelings of other characters in the story and the consequences of
ASD, 5-20 years old, were tested for their perception and ignoring those feelings. This can help children with autism to learn
identification of facial emotions. Results indicated unimpaired how to understand emotions in context and how to cope with them.
ability to label basic facial emotions and impaired ability to DIR / Floortime parent intervention [24] can help children with
generalize them [12]. Other studies claim that individuals with autism to connect emotionally and to build social and intellectual
ASDs do not show impaired emotion recognition [13] and these skills. This approach follows the child’s emotional interests and
results indicate the wide variation in the manner that individuals provides one to one, intensive, play - based intervention. SCRET is
are affected. a framework that incorporates practices from other approaches and
evidence-based practices [25]. It is a supporting multidisciplinary
model for children with autism to enhance socio-emotional and
2.2 Typically developing children
communication abilities. The ABA therapy is often used as a
Typically developing children start understanding emotion very
treatment for children with autism. It is a widespread and
early. Five months infants can categorize and recognize facial
recognized method for increasing learning, communication and
expressions such as smiling [14]. Evidence also demonstrates face
appropriate social behavior [26]. Positive reinforcement, reward
processing skills to be present in infants aged 4 to 9 months [15].
for desired behaviors and ignorance for inappropriate behaviors are
The ability to recognize emotions from facial expressions develops
the main characteristics. By rewarding and encouraging desired
gradually with the age [16]. Processing emotion and decoding
behavior, children are more likely to repeat it. According to the
facial signals emerge before their second birthday. At the age of 2
ABA intervention method, teaching children with autism about
they are able to recognize and use terms for the basic emotions. By
emotions can be achieved by providing examples of appropriate
this age they also talk about self and other emotional states. From 2
emotional behavior and then rewarding when a child gives the
to 5 they expand their emotion vocabulary and express more
correct emotional response. Young children with ASDs usually
complex emotions [17]. The ability to recognize emotions from
receive a home-based program, the EIBI, which is based on the
facial expressions is improved gradually with the age and a good
principles and technologies of ABA [27]. Early intervention
accuracy level seems to be acquired by 10 years of age [18]. Full
appears to lessen the effects of autism and children with ASD
proficiency level is reached only after puberty.
appear most able to benefit when intervention begins very early (2-
4 years old) [28], [29]. Toddlers, who attended an early
3. RELATED WORK intervention program, appear to benefit from this, since 31% of
Treatment approaches aim to improve social interaction, conquest those children were functioning in the typical developing range
communication and control inappropriate behavior. Children with after the intervention [30].
autism are more likely to initiate positive interaction after
treatment [19]. Educational interventions and medical management There are presumptions that individuals with autism think in
of the symptoms are the two major categories for treatment. pictures [31]. Children with ASD are reported to have strong
Autism requires a particular educational process to assist children visual processing capability and a good performance on difficult
with ASD in learning basic skills. Education is also considered as a visual search tasks [32], [33]. Augmentative and Alternative
solution for the socio-emotional deficits and training is claimed to Communication (AAC), such as PECS, is frequently used to
improve face processing abilities and strategies in autism [20]. A increase functional communication in children with ASD. PECS is
variety of educational interventions have been proposed for an effective visually-based system for communication via icons
children with autism and many proponents have claimed [34]. Visual supports are also suggested for social skill
developmental improvement and other benefits [21]. development enhancement in young children with ASD [35].
Picture cards with cartoon faces or illustrations of people faces or
photos showing different facial emotional expressions reinforce
3.1 Traditional educational interventions and support learning. Images showing only the face against a plain
Traditional therapy techniques and tools for teaching children with background and clearly depicting a single emotion, usually used to
autism about emotions are social stories. Such techniques and tools help children with autism to understand and recognize basic facial
are: Developmental Individualized and Relationship based (DIR) / expressions. Those cards may be used by calling out the name of
Floortime, Social Communication Emotional Regulation, an emotion and challenging children to give you the correct card or
Transactional Support (SCRET), Applied Behavioral Analysis asking them to look at the face and reflect back the emotion
(ABA), Early Intensive Behavioral Intervention (EIBI), Picture displayed. Cards show people in real situations with natural
Exchange Communication System (PECS) and Treatment and settings and contexts provide visual stimuli for discussion about a
Education of Autistic and Related Communication Handicapped range of emotions or about what people feel, why they may feel
CHildren (TEACCH). that way and possible responses to these feelings.
Social stories developed by Carol Gray [23] are an approach for Also programs that emphasize visual support strategies are
teaching individuals with ASD social behaviors. They are short commonly used in education classrooms. A well-known and
stories that describe and give information about social situations or widely applied model, the TEACCH, is a teaching strategy which
interaction which children may find difficult or confusing. The emphasizes in structured and predictable learning environment. It
stories also outline how to respond in such situations. There are utilizes visual cues to increase independence and to teach new
guidelines and criteria for writing a social story. Their main goal is skills, such as facial emotions, to children with autism. It involves
to provide accurate social information for improving daily schedules, visual materials, individualized treatment and
understanding of events that may lead to more effective responses parental support [36].

142
3.2 Computer interventions be beneficial for children who already have learned labeling and
Many children with autism find computers very engaging and understanding of emotions. The LIFEisGAME is a facial emotion
computer games strongly attract them in a positive way. Computers recognition learning system based on the interaction between
are rich, stable, predictable, consistent teaching tools and they humans and 3D avatars [49]. It is a computer-based approach that
provide a proper and more motivational learning environment for uses real time facial synthesis of 3D characters in order to teach
them [37]. They are great educational tools since children with autistic people to recognize emotions from facial expressions.
autism often experience discomfort with unpredictable social

Text / Audio / Video

Virtual Characters
Mouse / Keyboard

Images / Photos
Test /Evaluation
environment and they prefer a controlled learning environment

Refers to

2D / 3D
Age
[38]. Autistic individuals can enjoy learning and improve their
skills with computer-based intervention. Nowadays computers are Emotion Trainer Autism / Asperger Syndrome school age √ √ 2D √ T
the most adaptable assistive technology devices available for Mind Reading Asperger Syndrome / High Functioning Autism from 8 years old √ √ 2D √ T,A,V

children with autism and various computer games have been What to choose High Functioning Autism teenagers √ √ 3D √ √ T,A
cMotion Autism √ 3D
developed to help them to manipulate their impairments. Some of Let's Face it! Autistic Disorder / Asperger Syndrome / PDD-NOS school age √ √ 2D √
those games focus on their social interaction training and Emotion Mirror Autism √ √ V

especially learning about emotions [39]. Although effects on social LIFEisGAME

The Transporters
ASD from 6 years old √ √ 3D √ A

Autism / Asperger Syndrome 2-8 years old √ Other 3D √ A,V

and emotional skills are mixed, computer intervention is a PixTalk Autism Other 2D √ T

promising practice. Also currently, ample researches in Serious

Games for children with autism have been done. Serious Games Figure 1. Computer and other interventions
are designed for a primary purpose other than pure entertainment,
fun or enjoyment [40]. Michael Zyda defines a Serious Game as: 3.3 Other interventions
“a mental contest, played with a computer in accordance with MIT Media Lab proposes a wearable device [50] that perceives
specific rules that uses entertainment to further government or and reports on social-emotional information in real-time human
corporate training, education, health, public policy, and strategic interaction. The system records and analyzes the facial expressions
communication objectives.” [41]. Serious Games for autism cover and the head movements of the person with whom the wearer is
matters related to education, therapy for communication, interacting. This wearable platform is suggested as an exploratory
psychomotor treatment and social behavior enhancement [42]. and monitoring tool to assist individuals with ASD in perceiving
Autism Serious Games for education are designed to help teachers communication in a natural environment.
or student during the teaching and/or learning process. Studies
The Transporters [51], a DVD for teaching emotion recognition, is
show that Serious Games are very effective in the areas of therapy
an animation series for children with ASD (preschoolers or with
and education for children with autism.
significant learning difficulties). It involves toy vehicles and real
The Emotion Trainer [43] is a multimedia computer program for life faces of actors that show emotional expressions in social
improving school age students’ ability to recognize and predict context. The aim of this program was to teach not just some basic
emotions in others. It has positive effects on users’ understanding emotions but also some more complex ones. Evaluations of this
of emotion, particularly with repeated use. Mind Reading [44] is project showed improvements in emotion comprehension and
an interactive systematic emotion guide for teaching individuals recognition skills [52], [53] but limited efficiency in teaching basic
with Asperger syndrome. Experiments show that Mind Reading is emotion recognition skills to young children with autism with
effective in teaching adults with Asperger syndrome or high- lower range of cognitive ability [54].
functioning autism to recognize complex emotions. Another
PixTalk [55] is a software application for Windows Mobile Smart-
computer training program for teenagers with autism is the “What
phones which can be used as part of ongoing therapy. Teachers
to choose” game [45]. It is software with human and cartoon facial
and caregivers can access a web site and select from an online
expressions, 3D images, text and audio for training individuals
library the images to be downloaded on to the Smart-phone.
with autism in understanding dialogues that contain pragmatic
Children also can browse and select images to express their
subtleties. Children with autism have failed to take into account
intentions, desires, and emotions using PixTalk.
the causal link between facial expressions and the outcome of the
dialogue. cMotion [46] is a computer game that uses virtual
characters to teach emotion recognition and the programming 4. OUR APPROACH
concept to children with autism. The game is designed to teach the Difficulties in identifying and describing feelings are assumed to
users how to recognize facial expressions and manipulate an be an integral part of autism. Educational interventions can be used
interactive virtual character by using a visual drag-and-drop as a tool to help individuals with autism to cope with those
programming interface. A computer-based intervention for face deficits. As the number of children diagnosed with autism
training is the Let’s Face It! (LFI!) program [47]. This program is increased [56], [57], new methods for educating this population
comprised of seven interactive computer games that target the become necessary. In several studies, computer-based programs
specific face impairments associated with autism. Those games are have been widely used with success to teach people with ASD to
organized into a theoretical hierarchy of face processing domains. recognize emotions. However, those computer interventions
The Let’s Face It! Program shows promise as an effective require considerable skills for interaction as the users have to
intervention tool and treatment alternative. Emotion Mirror is a control a mouse or a keyboard. Such abilities are beyond very
project that integrates Computer Expression Recognition Toolbox young children with autism as they have major restriction in their
(CERT) with the program Let’s Face it! [48]. It is a computer efficiency to interact with computers. Our aim is to design and
assisted intervention system to enhance facial expression develop a gesture controlled Serious Game, as an early
perception and production in children with ASD. This game could intervention, for preschoolers (2-6 years old) with autism to teach

143
them basic facial emotions. Our approach takes into account the approach to targets than young adults [61]. With the increase of
specific characteristics of preschoolers with autism and their age, children in this study demonstrated significantly faster and
physical abilities. By creating an educational computer game more accurate use of the mouse which gives evidence of a strong
which proposes physical interaction, we hope to make early link between speed and accuracy skills and developmental
intervention more appealing and to foster learning of emotions. maturity. The same problems occur when players are required to
This software intends to support individuals with autism, their use other game controllers such as joysticks and gamepads.
families and their trainers and also to assist the preschool teaching Various research studies have been conducted to evaluate game
and learning process. controllers with the use of ISO 9241-9 [62] based on Fitts’ law
tasks which is an effective and widely used predictor of
Many individuals with autism show deficits in their perception and
performance and comfort. For point and select task, evaluation
recognition of facial emotions. With the appropriate intervention,
showed that participants prefer using a mouse and results indicated
elimination of those deficits can be achieved. Interventions are
that remote pointing devices perform poorly in terms of
proposed to start as early as possible and children with ASD
throughput, speed and error rate when compared to a mouse [63].
appear most able to benefit when intervention begins at 2-4 years
Touchpads are consider to be a good solution for young children
old. Typically developing children by the age of 2 use term for
but evaluations have also shown inferior performance, increased
basic emotion and from 2-5 years old they expand their
movement time, lower throughput and higher error rates when
vocabulary. So, for children with ASD we assume the age of 2-6
compared with a joystick [64], a trackball [65] and a mouse [66].
(preschool) can be set as the starting level of the emotion teaching
Thus, we propose an interaction based on gestures and a controller
process.
-free interface as a different interaction technique. Our game will
Educating children with ASD and particularly teaching them to allow user to employ gestures for navigation and interaction.
recognize and identify emotions from facial expressions is a
significant challenge and a complex task. The use of educational 5. DESIGN AND IMPLEMENTATION
software for teaching social and emotional skills could help In this paper we present an educational computer-based single
students with ASD to improve those abilities [58]. Learning tasks player game specially designed for Greek preschoolers with
developed in digital environments using information technology autism. The primary objective of this game is to assist young
can motivate the desire to learn in ASD students. As we have children with ASD to identify different emotions from facial
already mentioned in the Related Work, computer interventions expressions. The main goal of our game is to provide recognition
appear to be particularly appropriate for people with ASD for and understanding of facial emotions through early intervention.
several reasons. Traditional interventions previously mentioned
can be transferred into a computerized version. In our game, we As a Serious Game, our development was based on the following
will provide one to one intensive play-based intervention, visual principles: (a) it should have an impact on the player in a real life
support, positive reinforcement and reward, structural and context [67], (b) it is explicitly designed to reach a specific
predictable learning environment and we will incorporate practices purpose beyond the game itself and (c) it aims to teach
from all traditional approaches already mentioned in order to make preschoolers with autism, facial emotion recognition in order to
intervention more efficient. enhance their social interaction.
Many approaches to technology-enhanced intervention rely on Taking into account specific characteristics of autism, we designed
educational methods shown to result in good outcomes. Computer the game in order to meet the needs of students with ASD. A
interventions such as Serious Games could be a good approach as recent study conducted to analyze user needs for Serious Games
they also have positive effects on children with autism. Serious for teaching children with ASD emotions, revealed the
Games have the potential to support the process of learning and to characteristics of the children’s game play behaviors [68]. The
motivate individuals to learn more. They usually run on a personal observation showed repetition, matching instead of learning the
computer and they provide a different style of learning. We will features, lack of holistic face processing and deliberately incorrect
attempt to apply a Serious Game approach to teach children with selection. According to those findings, our game was intentionally
autism facial emotion recognition. designed to avoid such behaviors. For repetition, we decided not to
give them the opportunity to choose the same emotion again and
Previews studies that used Serious Games with children with again. For matching, instead of learning the features, we describe
autism have taken for granted the required skills and in particular the features of facial expressions in a separate level. For lack of
the ability to use the mouse or the keyboard. People with ASD holistic face processing, we describe all the face features that
demonstrate delay in fine motor skills which causes difficulties in reveal the emotion and ask them to look at each feature separately.
grasping and manipulating objects, such as a mouse [59]. Many For deliberate incorrect selection, nothing special happens when
individuals with autism experience motor control problems in their they give a wrong answer.
hands, leading to difficulties in operating the computer mouse. The
difficulties they encounter include moving the computer mouse to Our design also incorporates a theory-driven game design
designated area, corresponding to the location of the cursor to the framework supported by learning and developmental theories [69].
mouse movement, pressing down and lifting up fingers for the The framework is based on the integration of Kolb’s experiential
clicking motion, and clicking the correct button on the mouse to learning model and Piaget’s cognitive model. From this systematic
make selection [60]. Those difficulties are also found in typically approach were extracted six essential elements for designing games
developing individuals. Very young children (autistic and non- to teach children with ASD emotions. Those elements are:
autistic) are unable to functionally operate the computer mouse. matching, recognition, observation, understanding, generalizing
Experiments showed that four and five year old non-autistic and mimicking. We took those elements into consideration during
children make more and less accurate mouse sub-movements on the design process.

144
5.1 Game environment advisable to keep to the basic emotions. In our approach, the basic
The game environment is simple and less detailed in order to avoid emotions selected include happy, sad, angry, scared and surprised.
children’s distraction. Individuals with autism are reported to have Those emotions were chosen because typically developing children
enhanced perception of details [70] which may causes distraction. can recognize and understand them between 2 and 7 years of age.
For those reasons we select black context presented on a white The face stimuli we used are 15 grayscale photographs of male and
background and grayscale stimuli. Black and white contrast may female faces, taken from the CAlifornia Facial Expressions
also help to increase and retain child’s attention and keep them (CAFE) dataset [72]. All images used in the game, met FACS
focused on the screen. criteria [73] and all faces have been certified as “FACS-correct”
[74]. The stimuli are presented on each trial with different pairs of
photos and the goal is to choose the correct image.

5.2 System development

Our game is implemented with the use of MS-Kinect, which is a
motion sensing input device able to track movement and voice, and
even identify faces, without the need for any additional devices.
Kinect is a cheap and simple way to do motion capturing. It offers
simple and reliable skeleton tracking as well as an open source
SDK. By using Visual studios 2010 we were able to use the Kinect
SDK 1.6 released by Microsoft with C# as backend. The advantage
of using C# was that we could integrate XNA Game Studio 4.0 to
develop our game using Kinect. Writing the project’s software in
C# provided another advantage. It made it possible to utilize the
XNA libraries (provided by Microsoft) to create the graphics.
Figure 2. Example of game instruction page
Game begins with an instruction page where the child is informed 5.3 Interaction with the system
what is going to happen, what he/she has to do and how he/she can Interaction may be one of the areas that need to be developed with
do it. Apart from the text on the screen, audio instructions are also extreme care, depending on the activities and skills to be worked
provided. Audio cues are important as the information presented is on each of the games and target group. The computer-based
clear and age-appropriate. When the child feels ready, he/she can interventions that use a keyboard or a mouse for interaction might
choose to start the game. The game provides a structure learning cause problem with the younger children which may not be able to
environment which consists of 3 different levels with increasing use a computer. Our gesture-based interaction approach moves the
difficulty. Breaking the teaching intervention into small learning control of computer from a mouse and keyboard, to the motions of
steps makes the task easier to perform. In the first level children the body via new input devices.
should learn labeling emotions by correlating emotion terms with
images. In the second level they should learn to recognize Our game is designed to use non-touch based NUI and to be
emotions from their description and their association with facial controlled by hand gestures. The gestures are translated into
features. In the third level they should learn to identify the causes control commands. The player has three possible actions in all
of various feelings in different situations, obtained through the use game states, to choose left or right image and move to the next
of social stories. Those three levels provide recognition, matching, play area. These basic actions are implemented with efficient and
observation, understanding and generalization of facial emotions. easy to use gestures. Moving to the next play area requires a two-
hand gesture which is performed by moving both hands above the
head. Selecting the left image requires a one-hand gesture which is
performed by moving the left hand above head. Respectively,
selecting the right image requires a one-hand gesture which is
performed by moving the right hand above head.
During the game, if the player selects the correct or incorrect
stimuli, the system will inform player that he/she gave the correct
or incorrect answer. Each answer provides an audio and a visual
feedback such as operation-related sounds and changing the
images’ color. A voice telling “Bravo” rewards player for the
correct answer and a voice telling “Try again” encourages the
player to try again when the user provides an incorrect answer.
Figure 3. First level -labeling emotions (incorrect selection) [72]
There are no other sound effects because individuals with ASD
Individuals with autism are usually visual learners, which mean may suffer from auditory sensitivity [75], may demonstrate
that they understand written words, photos and visual information oversensitivity to certain sounds, even at low volume and may feel
better than spoken language. Information is good to be presented discomfort when exposed to certain sounds [76]. Visual feedback
through their strongest processing area. When teaching individuals is also provided by changing the image’s color into light green for
with autism about emotions, it is important to keep explanations as the correct answer and into light red for incorrect answer. Light
simple and as concrete as possible. It is also recommended to colors were selected because in ASD occur a reduced chromatic
describe each feeling pictorially by using pictures with clear discrimination that is due to general reduction in sensitivity [77].
outline, minimal details and color [71]. For young children it is Children with autism are less accurate at detecting the differences

145
between colors and have less accurate color perception [78]. [7] Kuusikko, S., Haapsamo, H., Jansson-Verkasalo, E., Hurtig,
Individuals with ASD also experience hypersensitivity which T., Mattila, M. L., Ebeling, H., et al. (2009). Emotion
includes increased light sensitivity and harshness of colors. Acute recognition in children and adolescents with autism spectrum
sensitivity to color is presented by their preference or avoidance of disorders. JADD, 39(6), 938–945.
a particular color [79]. [8] Uljarevic, M., Hamilton, A. (2012). Recognition of Emotions
in Autism: A Formal Meta-Analysis. JADD.
6. DISCUSION AND FUTURE WORK [9] K. A. Pelphrey et al., (2002). "Visual Scanning of Faces in
Increased interest in the potential of technology for users with Autism," JADD, vol. 32, no. 4, pp. 249-261.
autism is motivated by the rapidly growing needs for providing
intervention. Research shows that early intervention can greatly [10] W. Jones et al., (2008). "Absence of preferential looking to
improve the lives of children with autism. Computer-based tools the eyes of approaching adults predicts level of social
have been widely used with success to teach individuals with disability in 2-year-olds with autism spectrum disorder," Arch
autism. However, early intervention cannot be achieved with Gen Psychiatry, vol. 65, no. 8, pp. 946-954.
computers due to lack of skills. Gesture-based interaction aims to [11] Harms MB, Martin A, Wallace GL (2010) Facial emotion
contribute to overcoming this restriction. recognition in autism spectrum disorders: a review of
behavioral and neuroimaging studies. Neuropsychol Rev, vol.
Serious games with NUI interaction are a promising intervention
20, no. 3, pp.290–322.
strategy because they are appealing, motivating for young children
to use and convenient to access. When control is achieved through [12] James W. Tanaka, et al. (2012). The perception and
natural gestures, the user does not have to learn how to perform identification of facial emotions in individuals with autism
actions or how to operate games. Gesture-based interventions in spectrum disorders using the Let’s Face It! Emotion Skills
order to help children with autism to improve their skills must be Battery. JCPP, vol. 53, no. 12, pp. 1259–1267.
carefully designed in accordance with their abilities and needs. [13] Tracy JL, Robins RW, Schriber RA, Solomon M (2011) Is
Kinect offers an unlimited number of opportunities for new emotion recognition impaired in individuals with autism?.
applications such as gesture-based interventions for individuals JADD, vol. 41, no. 1, pp. 102–109.
with autism. As Kinect can support gesture and speech recognition [14] Bornstein, M.H., & Arterberry, M.E. (2003). Recognition,
and offer motion-sensing and interaction, children with autism can discrimination and categorization of smiling by 5-month-old
benefit to improve a broader range of skills. infants. Developmental Science, vol. 6, no. 5, pp. 585–599.
Future work could include the design of special gestures aiming to [15] Schwarzer, G., Zauner, N. (2003). Face processing in 8-
improve children fine motor skills or the design of a Serious Game month-old infants: evidence for configural and analytical
that will enable children to make facial emotion expressions that processing. Vision Research, vol. 43, no. 26, pp. 2783-2793.
the system will be able to detect in order to provide a method [16] Widen, S. C., & Russell, J. A. (2008). Children acquire
called “learn by doing”. emotion categories gradually. Cognitive Development, vol.
23, no. 2, pp. 291–312
7. REFERENCES [17] Wellman, H.M., Harris, P., Banerjee, M., & Sinclair, A.
[1] American Psychiatric Association, Diagnostic and Statistical (1995). Early understanding of emotion. Evidence from
Manual of Mental Disorders, Fourth Edition – Text Revision natural language. Cognition and Emotion, vol.9, pp.117-149.
(DSM-IV-TR), Washington, DC: APA, 2000.
[18] Durand, K., Gallay, M., Seigneuri, A., Robichon, F., &
[2] Susan Williams White, Kathleen Keonig, Lawrence Scahill, Baudouin, J. (2007). The development of facial emotion
(2007). Social Skills Development in Children with Autism recognition: The role of configural information. Journal of
Spectrum Disorders: A Review of the Intervention Research, Experimental Child Psychology, vol. 97, pp. 14–27
JADD, vol. 37, no. 10, pp. 1858-1868.
[19] Nirit Bauminger, (2002), The Facilitation of Social-Emotional
[3] R. Peter Hobson. (1986). The autistic child’s appraisal of Understanding and Social Interaction in High-Functioning
expressions of emotion, JCPP, vol. 27, no. 3, pp. 321–342. Children with Autism: Intervention Outcomes, JADD, vol.32,
[4] Rump, K. M., Giovannelli, J. L., Minshew, N. J., & Strauss, no.4,pp.283-298
M. S. (2009). The development of emotion recognition in [20] Faja, S., Aylward, E., Bernier, R., & Dawson, G. (2008).
individuals with autism. Child Development, vol. 80, no. 5, Becoming a face expert: A computerized face training
pp. 1434–1447. program for high functioning individuals with autism
[5] Hill, E., Berthoz, S., & Frith, U. (2004). Brief report: spectrum disorders. Developmental Neuropsychology, vol.
Cognitive processing of own emotions in individuals with 33, pp. 1–24.
autistic spectrum disorder and in their relatives. JADD, [21] Eikeseth, S. (2009). Outcomes of comprehensive psycho-
vol.34, no.2, pp.229–235. educational interventions for young children with autism.
[6] Matthew D. Lerner, James C. McPartland, James P. Morris. Research in Developmental Disabilities, 30, 158–178.
(2013). Multimodal emotion processing in autism spectrum [22] Carol A. Gray. (2000). The new social story book. Arlington,
disorders: An event-related potential study. Developmental TX: Future Horizons.
Cognitive Neuroscience, vol. 3, pp. 11-21.
[23] Carol A. Gray, Joy D. Grarand. (1993) Social Stories:
Improving responses of student with autism with accurate

146
 social information. Focus on Autistic Behavior, vol. 8, no. 1, [39] Bertram O. Ploog, et al. (2012). Use of Computer-Assisted
pp. 1-10. Technologies (CAT) to Enhance Social, Communicative, and
[24] Kingkaew Pajareya, Kaewta Nopmaneejumruslers. (2012). A Language Development in Children with Autism Spectrum
One-Year Prospective Follow-Up Study of a Disorders. JADD, pp. 1-22.
DIR/Floortime™ Parent Training Intervention for Pre-School [40] Damien Djaouti, Julian Alvarez, Jean-Pierre Jessel, and
Children with Autistic Spectrum Disorders. Journal of The Olivier Rampnoux. (2011). Origins of Serious Games.
Medical Association of Thailand, vol.95, no.9, pp.1184-1193 Serious Games and Edutainment Applications, pp. 25-43.
[25] Prizant, B. M., Wetherby, A. M., Rubin, E., & Laurent, A. C. [41] M. Zyda, (2005). "From visual simulation to virtual reality to
(2003). The SCERTS model: A transactional, family centred games," IEEE Computer Society, No. 9, pp. 25-32.
approach to enhancing communication and socioemotional [42] Mohd Noor, Helmy Adly and Shahbodin, Faaizah and Che
abilities of children with autism spectrum disorders. Infants Pee, Naim (2012) Serious Game for Autism Children: Review
and Young Children, vol. 16, no. 4, pp. 296-316. of Literature. World Academy of Science, Engineering and
[26] Axelrod S, Kates McElrath K, Wine B. (2012) Applied Technology, vol. 64, no. 124, pp. 647-652.
behavior analysis: autism and beyond. Behavioral [43] Silver, M., & Oakes, P. (2001). Evaluation of a new computer
Interventions, Vol.27, pp. 1-15 intervention to teach people with autism or Asperger
[27] Reichow, B. (2012). Overview of meta-analyses on early syndrome to recognize and predict emotions in others.
intensive behavioral intervention for young children with Autism, vol. 5, pp. 299–316.
autism spectrum disorders. JADD, vol. 42, pp. 512–520 [44] Golan O, Baron-Cohen S. (2006). Systemizing empathy:
[28] Tzanakaki P., et al. (2012). How and Why do Parents Choose Teaching adults with Asperger syndrome or high-functioning
Early Intensive Behavioral Intervention for their Young Child autism to recognize complex emotions using interactive
with Autism? ETADD Journal, vol. 47, no. 1, pp. 58 –71 multimedia. Development and Psychopathology, vol. 18, no.
[29] Sally J. Rogers. (1996). Brief Report: Early Intervention in 2, pp. 591-617.
Autism. JADD, vol. 26, no. 2, pp. 243-246. [45] Grynszpan, O., Martin, J.-C., & Nadel, J. (2008). Multimedia
[30] Stahmer A, Akshoomoff N, Cunningham A. (2011). Inclusion interfaces for users with high functioning autism: An
for toddlers with autism spectrum disorders: The first ten empirical investigation, International Journal of Human-
years of a community program. Autism, vol. 15, no. 5, pp. Computer Studies, vol. 66, no. 8, pp. 628-639.
625-641. [46] Samantha L. Finkelstein, Andrea Nickel, Lane Harrison, Evan
[31] Grandin T. Thinking in pictures: and other reports from my A. Suma, Tiffany Barnes, (2009) cMotion: A new game
life with autism. New York: Doubleday; 1995. (book). design to teach emotion recognition and programming logic
to children using virtual humans, IEEE VR2009, pp. 249-250.
[32] Plaisted, K., O'Riordan, M., & Baron-Cohen, S. (1998).
Enhanced visual search or a conjunctive target in autism: A [47] Tanaka, J.W., Wolf, J. M., Klaiman, C., Koenig, K.,
research note. JCPP, vol. 39, no. 5, pp. 777-783 Cockburn, J., Herlihy, L., Brown, C., Stahl, S., Kaiser, M.D.,
and Schultz, R.T. (2010). Using computerized games to teach
[33] O’Riordan, M. A., Plaisted, K. C., Driver, J., & Baron-Cohen, face recognition skills to children with autism spectrum
S. (2001). Superior visual search in autism. Journal of disorder: The Let's Face It! program, JCPP, vol. 51, pp. 944-
Experimental Psychology: Human Perception & Performance, 952.
vol. 27, no. 3 pp. 719–730.
[48] David Deriso, Joshua Susskind, Lauren Krieger, Marian
[34] Ganz, Jennifer B., Simpson, Richard L., Lund, Emily M. Bartlett. (2012). Emotion Mirror: A Novel Intervention for
(2012) The Picture Exchange Communication System Autism Based on Real-Time Expression Recognition.
(PECS): A Promising Method for Improving Communication Computer Vision - ECCV 2012. Workshops and
Skills of Learners with Autism Spectrum Disorders. ETADD Demonstrations, vol. 7585, pp. 671-674.
Journal, vol. 47, no. 2, pp. 176-186.
[49] Jos C. Miranda, Tiago Fernandes, A. Augusto Sousa and
[35] Moody, Amelia K. (2012) Visual supports for promoting er nica C. Orvalho (2 11). Interactive Technology:
social skills in young children: a family perspective. ACEI, Teaching People with Autism to Recognize Facial Emotions,
vol. 88, no. 3. Autism Spectrum Disorders - From Genes to Environment,
[36] Mesibov, G. B., Shea, V., & Schopler, E. (2005). The Tim Williams (Ed.), InTech, Available from:
TEACCH approach to autism spectrum disorders. New York: http://www.intechopen.com/books/autism-spectrum-
Springer Publishers. disorders-from-genes-to-environment/interactive-technology-
teaching-people-with-autism-to-recognize-facial-emotions
[37] Tina R. Goldsmith, Linda A. LeBlanc. (2004). Use of
Technology in Interventions for Children with Autism. JEIBI, [50] Rana el Kaliouby, Alea Teeters and Rosalind W. Picard,.
vol. 1, no. 2, pp. 166-178. (2006). An Exploratory Social-Emotional Prosthetic for
Autism Spectrum Disorders. International Workshop on
[38] Allison L. Wainer, Brooke R. Ingersoll.(2011), The use of
Wearable and Implantable Body Sensor Networks, MIT
innovative computer technology for teaching social
Media Lab, pp. 3-4.
communication to individuals with autism spectrum disorders.
Research in Autism Spectrum Disorders, vol. 5, no. 1, pp. 96–
107.

147
[51] S. Baron-Cohen, O. Golan, E. Chapman and Y. Granader, [64] Douglas, S. A., Kirkpatrick, A. E., & MacKenzie, I. S.
(2007). Transported into a World of Emotion. The (1999). Testing pointing device performance and user
Psychologist, vol. 20, no. 2, pp. 76-77. assessment with the ISO 9241, Part 9 standard. Proceedings
[52] S. Baron-Cohen, O. Golan, and E. Ashwin., (2009). Can of ACM CHI 1999, pp. 215-222.
emotion recognition be taught to children with autism [65] MacKenzie, I. S., Kauppinen, T., & Silfverberg, M. (2001).
spectrum conditions?, Philosophical Transactions of the Accuracy measures for evaluating computer pointing devices.
Royal Society B-Biological Sciences, vol. 364, no. 1535, pp. Proceedings of ACM CHI 2001, pp. 9-16.
3567-3574. [66] Akamatsu, M., & MacKenzie, I. S. (2002). Changes in
[53] Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, applied force to a touchpad during pointing tasks. INT J IND
K., Legget, V., et al. (2010). Enhancing emotion recognition ERGONOM, vol. 29, no. 3, pp. 171-182.
in children with autism spectrum conditions: an intervention [67] Konstantin Mitgutsch and Narda Alvarado. (2012).
using animated vehicles with real emotional faces. JADD, vol. Purposeful by design?: a serious game design assessment
40, no. 3, pp. 269–279. framework. In Proceedings of the International Conference on
[54] Beth T. Williams, Kylie M. Gray, Bruce J. Tonge., (2012). the Foundations of Digital Games (FDG '12). ACM, New
Teaching emotion recognition skills to young children with York, NY, USA, pp. 121-128.
autism: a randomised controlled trial of an emotion training [68] Abirached, B., Zhang, Y. & Park, J.H. (2012). Understanding
programme. JCPP, vol. 53, no. 12, pp. 1268–1276. User Needs for Serious Games for Teaching Children with
[55] De Leo G, Gonzales C, Battagiri P, Leroy G. (2011). A smart- Autism Spectrum Disorders Emotions. Proceedings of World
phone application and a companion website for the Conference on Educational Multimedia, Hypermedia and
improvement of the communication skills of children with Telecommunications 2012, pp. 1054-1063.
autism: clinical rationale, technical development and [69] Ji Hyun Park, Bretagne Abirached and Yan Zhang. (2012). A
preliminary results. Journal of Medical Systems August 2011, framework for designing assistive technologies for teaching
vol. 35, no. 4, pp. 703-711. children with ASDs emotions. In Proceedings of 2012 ACM
[56] J. Baio. (2012). Prevalence of autism spectrum disorders-- annual conference extended abstracts on Human Factors in
Autism and Developmental Disabilities Monitoring Network, Computing Systems, New York, USA, pp. 2423-2428.
14 sites, United States, 2008,. MMWR Surveillance [70] Ashwin, E., Ashwin, C., Rhydderch, D., Howells, J., &
Summaries, vol. 61, no. 3, pp. 1-19. Baron-Cohen, S. (2009). Eagle-eyed visual acuity: An
[57] M. Elsabbagh et al., (2012). Global Prevalence of Autism and experimental investigation of enhanced perception in autism.
Other Pervasive Developmental Disorders, Autism Research, Biological Psychiatry, vol. 65, no. 1, pp. 17–21.
vol. 5, no. 3, pp. 160-179. [71] Dodd S. Understanding Autism. Sydney: Elsevier, 2005.
[58] Lozano-Martínez, Josefina and Ballesta-Pagán, Francisco [72] M. N. Dailey , G. W. Cottrell and J. Reilly, "CAlifornia Facial
Javier and Alcaraz-García, Salvador. (2011). Software for Expressions (CAFE)," 2001. [Online]. Available:
Teaching Emotions to Students with Autism Spectrum http://www.cs.ucsd.edu/users/gary/CAFE/. La Jolla, CA:
Disorder. Revista Comunicar, vol. 18, no. 36, pp. 139-148. Computer, Science and Engineering Department, UCSD.
[59] E. Jasmin, M. Couture, E. Fombonne, E. Gisel,. (2009). [73] P. Ekman and W. V. Friesen, Facial Action Coding System,
Sensori-motor and daily living skills of preschool children Palo Alto: Consulting Psychologist Press, 1978.
with autism spectrum disorders, JADD, vol. 39, no. 2, pp.
231–241. [74] M. L. Smith et al., "Transmitting and Decoding Facial
Expressions," Psychological Science, vol. 16, no. 3, pp. 184-
[60] Karanya Sitdhisanguan , Nopporn Chotikakamthorn , Ajchara 189, 2005.
Dechaboon , Patcharaporn Out,. (2012). Using tangible user
interfaces in computer-based training systems for low- [75] Gomes, Erissandra, Pedroso, Fleming Salvador, & Wagner,
functioning autistic children, Personal and Ubiquitous Mário Bernardes. (2008). Auditory hypersensitivity in the
Computing, vol. 16 no. 2, pp. 143-155. autistic spectrum disorder. Pró-Fono Revista de Atualização
Científica, vol. 20, no. 4, pp. 279-284.
[61] Lane Alison E, Ziviani Jenny M (2 1 ) Factors influencing
skilled use of the computer mouse by school-aged children. [76] Tan, Y.-H., et al.(2012). Auditory abnormalities in children
Computer & Education, vol. 55, no. 3, pp. 1112–1122. with autism. Open J of Psychiatry, vol. 02, no. 1, pp. 33–37.
[62] Ergonomic requirements for office work with visual display [77] A. Franklin et al. (2010). Reduced chromatic discrimination
terminals (VDTs) - Part 9: Requirements for non-keyboard in children with autism spectrum disorders, Developmental
input devices. International Standard, ISO(2000). ISO/DIS Science, vol. 13, no. 1, pp. 188–200.
9241-9 [78] Franklin A, Sowden P, Burley R, et al. (2008). Color
[63] Natapov, D., Castellucci, S. J., and MacKenzie, I. S., (2009). perception in children with autism. JADD, vol. 38, no. 10, pp.
ISO 9241-9 evaluation of video game controllers, 1837-1847
Proceedings of Graphics Interface 2009. Toronto: Canadian [79] Rachel A Coulter. (2009). Understanding the Visual
Information Processing Society, 2009, pp. 223-230. Symptoms of Individuals with Autism Spectrum Disorder
(ASD), J Optom Vis Dev, vol. 40, no. 3, pp. 164-175.

148