robotics
Article
Heart Rate as a Predictor of Challenging Behaviours among
Children with Autism from Wearable Sensors in Social
Robot Interactions
Ahmad Qadeib Alban 1 , Ahmad Yaser Alhaddad 2 , Abdulaziz Al-Ali 1,3 , Wing-Chee So 4 , Olcay Connor 5 ,
Malek Ayesh 1 , Uvais Ahmed Qidwai 1 and John-John Cabibihan 2, *
1
2
3
4
5
*
Citation: Alban, A.Q.; Alhaddad,
A.Y.; Al-Ali, A.; So, W.-C.; Connor, O.;
Ayesh, M.; Ahmed Qidwai, U.;
Cabibihan, J.-J. Heart Rate as a
Predictor of Challenging Behaviours
Department of Computer Science and Engineering, Qatar University, Doha P.O. Box 2713, Qatar
Department of Mechanical and Industrial Engineering, Qatar University, Doha P.O. Box 2713, Qatar
KINDI Computing Research Center, Qatar University, Doha P.O. Box 2713, Qatar
Department of Educational Psychology, Faculty of Education, The Chinese University of Hong Kong,
New Territories, Hong Kong
Step by Step Centre for Special Needs, Doha P.O. Box 47613, Qatar
Correspondence: john.cabibihan@qu.edu.qa
Abstract: Children with autism face challenges in various skills (e.g., communication and social) and
they exhibit challenging behaviours. These challenging behaviours represent a challenge to their
families, therapists, and caregivers, especially during therapy sessions. In this study, we have investigated several machine learning techniques and data modalities acquired using wearable sensors from
children with autism during their interactions with social robots and toys in their potential to detect
challenging behaviours. Each child wore a wearable device that collected data. Video annotations of
the sessions were used to identify the occurrence of challenging behaviours. Extracted time features
(i.e., mean, standard deviation, min, and max) in conjunction with four machine learning techniques
were considered to detect challenging behaviors. The heart rate variability (HRV) changes have
also been investigated in this study. The XGBoost algorithm has achieved the best performance
(i.e., an accuracy of 99%). Additionally, physiological features outperformed the kinetic ones, with
the heart rate being the main contributing feature in the prediction performance. One HRV parameter
(i.e., RMSSD) was found to correlate with the occurrence of challenging behaviours. This work
highlights the importance of developing the tools and methods to detect challenging behaviors
among children with autism during aided sessions with social robots.
among Children with Autism from
Wearable Sensors in Social Robot
Keywords: challenging behaviours; autism; wearables; machine learning; social robots
Interactions. Robotics 2023, 12, 55.
https://doi.org/10.3390/
robotics12020055
Academic Editor: KC Aw
Received: 24 January 2023
Revised: 19 March 2023
Accepted: 21 March 2023
Published: 1 April 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1. Introduction
Autism Spectrum Disorder (ASD) is a developmental and neurological disorder that
causes impairments characterised by difficulties in social communication and restricted
behaviours [1]. Children with ASD exhibit challenging behaviours frequently at varying
intensities and in different forms, such as meltdowns, tantrums, property destruction,
and aggression [2–4]. The prevalence rate of challenging behaviours and aggression among
children with ASD is high [5–7]. It is reported by the Autism Society of Minnesota that
one in every 68 people have ASD [8]. Individuals with ASD exhibit recurrent challenging
behaviours, and these constitute a source of concern for parents and caregivers due to their
sudden occurrences. Challenging behaviours vary in frequency, intensity, duration, and
may put the child or others in an unsafe situation. These behaviours may appear in many
forms, which include [9]:
•
•
Withdrawal: The child may zone out, gaze into space, or performs repetitive actions.
Outward distress: This includes crying in an uncontrolled manner, stomping, screaming, or curling up into a ball.
Robotics 2023, 12, 55. https://doi.org/10.3390/robotics12020055
https://www.mdpi.com/journal/robotics
Robotics 2023, 12, 55
2 of 13
•
Physical and verbal aggression: Destruction of surrounding objects, outbursts, and
self-damaging. These aggressive behaviours include scratching their faces or hands
and pulling their own hair, as well as with skin pinching.
Many children with autism experience distress prior to meltdown. This period that
precedes the meltdown is called the rumble stage [10,11]. Specific behaviours, such as
challenging behaviours, appear at this stage just before a meltdown. These lead to actions
such as head banging (e.g., against an object or using their hands), hand flapping, kicking,
throwing objects, hand biting, screams, and others [5,12]. Being frustrated and the presence of new stimuli are some of the contributing factors that increase the occurrence of
challenging behaviours [13].
Early intervention can help in managing challenging behaviours [14,15]. The advances
in technology are being integrated in the screening of ASD and in therapy sessions to
improve the outcomes [16,17]. Social robots are examples of adopted technologies in
therapy that have reported positive outcomes (e.g., improved communication, motor,
and social skills) among children with autism [18–20].
A social robot represents a new stimulus into a child’s environment that might trigger
an unwanted behaviour. Hence, investigating the perception of a social robot is essential [21,22]. The occurrence of aggression toward social robots was reported during interaction sessions [23,24]. Potential harm could occur due to the manifestation of challenging
behaviours, for example, due to the throwing of a social robot [25]. Improved safer designs
and hardware optimisations approaches are needed to mitigate any potential harms [26–28].
Furthermore, solutions based on machine learning techniques, especially personalised, are
needed to anticipate such behaviours and to provide reactions through a therapist or the
companion social robots [9,29–32].
Physiological changes of the human body can provide indicators about health and
the current emotional state using wearable devices and machine learning techniques.
A previous study found that challenging behaviours are influenced by the physiological
arousal of children with autism [33]. Another study reported a difference in the heart rate
between adults with ASD and normal adults during public speaking [34]. Research that
detects challenging behaviours among children with autism during interactions with social
robots is still limited [9,35,36].
To notify parents or caregivers to the occurrence of these behaviours, the vital signs
and physiological changes should be investigated accordingly to assist in the detection of
challenging behaviours. One approach is to measure the child’s vital signs using wearable
sensors while leveraging machine learning and deep learning algorithms to construct an
accurate detection model. In this study, we conduct experiments using data acquired from
five children with ASD and machine learning techniques to detect challenging behaviours
during interaction sessions with toys and social robots. The contributions of this work are
summarised as follows:
1.
2.
3.
Integration of machine learning techniques and wearable sensors to detect challenging behaviours.
Evaluation of physiological and kinetic features in identifying challenging behaviors.
Analysis on the HR and HRV roles in supporting the detection.
This paper is organised as follows. Section 2 presents the background. Section 3
describes the methods used. Section 4 presents the results, while Section 5 provides the
discussion. Section 6 concludes the study.
2. Background
Prior studies investigated the possibility of integrating wearable sensors and machine
learning techniques to interpret the physiological and kinematic properties of the human
body to predict or detect specific affective patterns of emotions or behaviours [37–40].
The considered modalities and detection objectives using wearables and machine learning
techniques in autism research have varied. Some of the sensors and modalities considered
Robotics 2023, 12, 55
3 of 13
were photoplethysmogram, electrodermal activity or the galvanic skin response, heart
rate, temperature, acceleration, and skin conductance level. There is a growing interest in
incorporating wearable sensors in autism therapy. For example, Fazana et al. presented a
framework that incorporates a set of existing programs for augmentative and alternative
communication with wearable sensors to improve the communication skills, enhance
behaviours, and promote health monitoring of children with autism [41].
Goodwin et al. [42] investigated the detection of stereotypical motor movements
in children with autism using three-axis accelerometers worn on different parts of the
body (i.e., wrists and torso). The data were collected from six individuals with autism.
The recognition performance of the two employed classifiers (i.e., decision tree and support
vector machines) achieved accuracies ranging from 81.2% to 99.1%. Rad et al. [43] used
the same datasets to implement a convolutional neural network and long short-term
memory algorithms. Their results demonstrated that applying deep learning techniques on
the acceleration data would improve the detection of stereotypical motor movements in
real-time conditions. Another work also used accelerometer data to classify challenging
behaviors based on simulated behaviours by a specialist [44]. When tested on a child with
autism, their best machine learning model achieved an accuracy of 69.7%. Another study
explored techniques to detect common motor movements for children diagnosed with
autism [45]. The study investigated the impact of these motor movements on learning and
social interactions using deep learning approaches.
Pollreisz et al. [46] established an emotion recognition system using an Empatica E4
watch to collect electrodermal activity, heart rate, and temperature values from ten young
adults. The reported success rate for emotion recognition was 65% using a decision tree
algorithm. Another study investigated the different emotions encountered throughout
meltdowns to develop an emotion recognition system using machine learning methods [47].
The study concluded that the random forest performed the best (91.27%) when feature
selection techniques were employed. Heart rate variability parameters were also considered (Table 1). Lee et al. [48] measured both the heart rate variability and galvanic skin
response to identify emotions using neural networks. They analyzed the data collected
from participants in the frequency and time domains. They found that changes in some
of the HRV parameters (i.e., RMSSD and SDNN) might lead to elevated activity in the
sympathetic nervous system, which could be interpreted as a sign of fear. Their reported
accuracy of the best trained model was 80.2%.
Table 1. Some of the common heart rate variability (HRV) parameters that were previously considered
in autism research [49].
Parameters
Units
Domain
Description
RMSSD
ms
Time
SDNN
LF
HF
ms
Hz
Hz
Time
Frequency
Frequency
Square root of the mean squared differences between successive RR intervals
Standard deviation of NN intervals
Peak in low frequency range (0.04 to 0.15 Hz)
Peak in high frequency range (0.15 to 0.4 Hz)
Yap et al. [50] investigated the impact of listening to music on the heart rate and
anxiety levels of children with autism to identify which music genre could calm the
children. They devised a mobile application that was connected wirelessly to a pulse sensor
to measure the heart rate. The goal of their application was to improve communication
and learning skills, support emotion regulation, and monitor the heart rate while listening
to music. Lydon et al. [51] focused on investigating the correlation between the heart
rate and challenging behaviours experienced by children with autism. They analyzed the
heart rate data before, during, and after the instances of challenging behaviours in three
children with ASD. They found that such behaviours might increase arousal for some
children with autism. The prediction of challenging behaviours has also been proposed by
J. Nuske et al. [52]. They investigated the possibility of applying the heart rate to predict
Robotics 2023, 12, 55
4 of 13
such behaviours in children with autism based on the statistical analysis for the acquired
data. Forty-one children diagnosed with autism were recruited in their experiments.
The participants wore an electrocardiograph monitor and low-level stress was stimulated.
Considering the intrusive nature of the device and to avoid pulling the ECG electrodes,
vests with pockets were used to house the device while placing the electrodes on the backs
of the children. Their results demonstrated that heart rate changes could be an early sign
for the occurrence of challenging behaviours.
3. Materials and Methods
3.1. Participants
Five male children with autism, with ages ranging between 7 and 10 years old, participated in this study. The participants attended a local centre for special needs in Doha,
Qatar. Parental consent was obtained by the centre. The sessions were conducted with
each child individually with the supervision and assistance of a teacher or their caregiver.
The procedures for this work did not include invasive or potentially hazardous methods and were in accordance with the Code of Ethics of the World Medical Association
(Declaration of Helsinki).
3.2. Stimuli
Social robots and regular children’s toys were used as stimuli in this study. The social
robots were the humanoid Nao robot (SoftBank Robotics, Tokyo, Japan) and a white furred
robotic seal (PARP robots, Itasca, IL, United States). These social robots are shown in
(Figure 1b). The toys consisted of a squishy green rubber ball, multi-colour train, brass
cymbals, and wooden letter blocks that are placed on a toy truck. Further details regarding
the stimuli used can be found in [23].
(a)
(c)
EDA
9.6
7.1
4.5
2
BVP
46
14
-19
-51
-84
1.77
ACC
0.91
(b)
-0.8
-1.65
0.33
HR
0.11
-0.11
-0.33
-0.55
Temp
29.05
28.5
27.95
27.4
26.84
23:27
23:28
23:28
23:29
23:30
23:30
23:31
23:32
23:32
23:33
23:34
23:34
23:35
23:36
23:36
23:37
23:38
23:38
23:39
23:40
23:40
23:41
23:42
23:42
23:43
23:44
23:44
Figure 1. An overview of the adopted methodology in this study. (a) The Empatica E4 wearable
device. (b) One of the children interacting with the social robots (see https://youtu.be/sGNslV2Yuks
(accessed on 20 March 2023)). (c) A sample of the acquired data using the wearable device.
3.3. Wearable Device
A wristband wearable sensor (Empatica E4, Milano, Italy) was used to obtain the data
readings from the children during the experiments (Figure 1a). The E4 wearable sensor
contains a real-time clock and it is capable of recording physiological data signals to an
internal memory (Figure 1c). The physiological signals considered are as follows:
1.
2.
3.
Acceleration (ACC): measures wrist’s motion changes in terms of the acceleration
changes in the x, y, and z directions.
Electrodermal Activity (EDA): determines the change in skin conductance and the
skin’s electrical properties.
Temperature (TEMP): determines the temperature of the skin.
Robotics 2023, 12, 55
5 of 13
4.
5.
Heart Rate (HR): the number of beats per minute.
Blood Volume Pulse (BVP): determines the changes in the blood volume.
3.4. Algorithms
The considered supervised machine learning algorithms were as follows:
1.
2.
3.
4.
Support-Vector Machine (SVM): non-probabilistic binary linear supervised learning
model that can solve and classify both linear and non-linear problems.
Multilayer Perceptron (MLP): learning technique inspired by the biological brain that
consists of layers of artificial neurons that can learn from data.
Decision Tree (DT): an algorithm that predicts the output by moving through the
different discrete decision options that are represented in a tree-like structure until a
conclusion is reached.
Extreme Gradient Boosting (XGBoost): an ensemble supervised machine learning
technique which utilises regularised gradient boosted decision trees to improve the
performance and classification speed.
3.5. Procedures
3.5.1. Annotation
Manual annotation was carried out for each of the five children’s behaviours. This
was conducted with the help of a free annotation software (BORIS, v. 7.10.2, Torino, Italy).
The behaviours were annotated as either ’Challenging’ or ‘Non-challenging’. A challenging
behaviour is considered to be any action that is interfering, repetitive, stimming, and might
inflict harm on oneself or others. Challenging behaviours also included head banging, arm
flapping, ear pulling, kicking, and scratching. The total number of challenging behaviour
instances in all the sessions was 17. The respective percentage of challenging behaviours in
each session was in the range of 1.74% to 18.21%.
3.5.2. Data Preprocessing
To ensure consistency, the data acquired from the wearable device were preprocessed
and the sampling frequency of every acquired data signal was set to 64 Hz. This is crucial
since the different sensors obtain data at different sampling rates. The preprocessing
stage included the outliers’ removal and resampling the training data to ensure that the
classes are equally balanced. A portion equal to thirty percent of the original dataset was
used as the unseen testing set. The initial experiments with the dataset indicated that the
extracted features produced a better performance as compared to the raw features alone.
For this reason, only the time-domain extracted features (i.e., mean, standard deviation,
min, and max) were considered throughout this study.
4. Results
4.1. Machine Learning Models
Four machine learning algorithms were examined based on the evaluation metrics
and prediction speed (Table 2). In the results, challenging behaviors were considered to be
the positive class. The models were developed using Python libraries (i.e., Sklearn [53] and
XGBoost [54]). The depth of the DT algorithm was set to dynamic and the Gini function was
used for the splitting criteria. SVM used a radial basis function kernal with a regularisation
parameter of 0.1, and a gamma parameter was set to the scale. As for the MLP, it contained
one hidden layer that consisted of 100 neurons with weights adjusted using a stochastic
gradient descent at a 0.0001 L2 regularisation. The XGBoost was trained with the logistic
objective and a max depth of 6, with a aplha equal to 1, learning rate of 0.3, and 100 estimators.
Robotics 2023, 12, 55
6 of 13
Table 2. The evaluation metric scores for the four algorithms and their test times (in seconds) needed
to evaluate the test samples.
XGBoost
MLP
SVM
DT
Precision
Recall
F1-Score
Accuracy
Testing Time (s)
0.88
0.67
0.24
0.87
0.99
0.98
0.91
0.92
0.93
0.80
0.38
0.89
0.99
0.97
0.85
0.98
0.24
0.36
2.48
0.29
XGBoost showed a better overall performance compared to other classifiers in terms of
precision (0.88), recall (0.99), F1-Score (0.93), and accuracy (0.99). Additionally, it achieved
the fastest time (i.e., 0.24 s) to predict the test samples. The second best performing
algorithm was DT followed by MLP. SVM achieved the lowest performance and took the
longest time to predict the test samples, which was around 2.5 s. Due to its performance,
the XGBoost has been considered in the upcoming experiments.
4.2. Feature Effects
To measure the contribution of each sensor to the prediction performance, sensor
features were added gradually to the overall feature vector and the results were compared
for the individualised models and combined model (Table 3). With the ACC alone, the personalized models performed poorly in all five participants and so their combined model.
Set 2 considered the effect of adding the HR sensor reading to the feature vector that has
led to a large increase in performance for all participants individually and their combined
model. As for Set 3, adding BVP had little effect on all the models. Adding TEMP improved
the performance of the individual personalised models and their combined model slightly.
Finally, adding EDA in Set 5 has led to a further increase in the overall performance for
most of the models.
Table 3. A summary of the results showing the precision, recall, and F1 scores for the experiments
considering the impact of adding each feature to the feature set for the personalised models of each
participant and their combined generalised model.
Set
Feature
1
2
3
4
5
ACC
Set 1 + HR
Set 2 + BVP
Set 3 + TEMP
Set 4 + EDA
Metric
Prec Rec
F1
Prec Rec
F1
Prec Rec
F1
Prec Rec
F1
Prec Rec
F1
Child 1
Child 2
Child 3
Child 4
Child 5
All
0.35
0.53
0.51
0.37
0.26
0.36
0.50
0.67
0.61
0.49
0.37
0.46
0.63
0.72
0.69
0.75
0.43
0.62
0.78
0.84
0.78
0.85
0.58
0.72
0.62
0.71
0.69
0.75
0.58
0.63
0.77
0.82
0.78
0.85
0.73
0.72
0.79
0.77
0.96
1
0.54
0.82
0.87
0.86
0.97
0.99
0.70
0.89
0.79
0.92
0.96
1
0.57
0.88
0.87
0.96
0.98
0.99
0.73
0.93
0.87
0.89
0.75
0.72
0.64
0.66
1
1
0.90
0.98
0.91
0.86
1
0.97
0.89
0.98
0.97
0.86
0.97
0.99
0.99
0.98
1
0.98
0.97
1
0.99
0.98
1
0.99
4.3. Kinetic vs. Physiological
To understand which category of features are most significant, the kinetic, physiological, and a combination of the two were investigated. The evaluation metrics results
for the two categories and their combined features are depicted in Figure 2. The results
demonstrated that kinetic features alone performed poorly with respect to the physiological
and combined features. The physiological features were found to perform similarly to the
combined features. In spite of this, the overall best performance comes from using the
combined features.
To further investigate how each individual feature contributes to the performance
of the machine learning model, the importance of each individual feature with respect
to the F-score were plotted using the built-in XGBoost tool [54]. The individual plots for
each participant revealed a discrepancy between the importance of each features (Figure 3).
However, HR appears to be the most important factor for the majority of the participants,
Robotics 2023, 12, 55
7 of 13
followed by either EDA or TEMP. The combined plot for the generalised model revealed
that the most important feature was HR then followed by EDA, TEMP, ACC, and BVP
(Figure 4).
1
Kinetic
Physiological
Combined
0.8
0.6
0.4
0.2
0
Precision
Recall
F1-score
Specificity
False Positive
Rate
Figure 2. The evaluation metrics results for the three categories using the best performing algorithm
(i.e., XGBoost).
(a)
(b)
(c)
(d)
(e)
Figure 3. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost)
for each child. (a) Child 1. (b) Child 2. (c) Child 3. (d) Child 4. (e) Child 5.
Robotics 2023, 12, 55
8 of 13
Figure 4. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost)
for the combined model.
4.4. HRV and Challenging Behaviours
To further investigate the importance of heart rate parameters, the heart rate variability
(HRV) based on calculating the RMSSD was considered. The RMSSD was derived from
the interbeat interval signal of the wearable device using a sampling frequency of 64 Hz.
The HRV changes for one of the children during different states were investigated (Figure 5).
The HRV values appear to be highest during a rest state while the lowest during the
occurrence of a challenging behaviour (Figure 5c).
(a)
8:57:41
8:57:00
(Stimulating state)
8:57:30
8:58:00
(b)
8:58:30
(c)
8:58:30 (Rest state)
8:59:00
8:59:30
9:00:49 (Challenging state)
9:00:00
9:00:30
9:01:00
Figure 5. The changes in the HRV (i.e., RMSSD) corresponding to different states. (a) The child is
overwhelmed and stimulated by the bubble gun toy. (b) The child is in a rest state. (c) The child
experiences a challenging behaviour.
A machine learning model was trained that contained an additional feature, called
HRV. The results demonstrated that the importance of HRV for a child that exhibited
more challenging behaviours (Figure 6a) is higher compared to a child that exhibited less
instances of challenging behaviours (Figure 6b). Furthermore, the contribution of HRV
outweighed that of HR in the child exhibiting challenging behaviours and vice versa in
the child experiencing less challenging behaviours. Hence, the detection of challenging
behaviours appears to depend on the changes in HRV.
Robotics 2023, 12, 55
9 of 13
()
()
Figure 6. The importance of HRV in the performance of the machine learning model (i.e., XGBoost).
(a) Represents the feature importance for one of the participants whose challenging behaviours were
more frequent and intense. (b) The feature importance for another participant who displayed less
challenging behaviours.
5. Discussion
To properly understand the occurrence of challenging behaviour, it is vital to analyze
the interactions of the participants throughout their sessions. The children displayed different levels of engagement with the presented stimuli. The participants showed fascination
in the colourful train that produced bubbles, which encouraged their engagement. The fascination and interaction came in various forms that included both facial expressions and
physical movements. Most of the participants did not prefer the white robotic seal, which
could be due to its animal-like appearance. Hence, that might have led to participants
exhibiting challenging behaviours. Furthermore, the robotic humanoid caused confusion
and curiosity as some of the sudden movements produced by it have led to negative
reactions by some of the participants.
In this paper, we investigated the detection of challenging behaviours among children
with autism using wearable sensors to acquire data and machine learning techniques.
While there are many machine learning approaches, not all are suitable to be considered
in such an application. In addition to the prediction accuracy, a system must also make
predictions fast enough for the timely intervention. In our evaluations, the XGBoost
algorithm fulfilled these two criteria. This promising performance and robustness of
XGBoost could be attributed to its ability in recognizing intricate data patterns, identifying
subtle physiological changes, dealing with outliers, handling overfitting, and supporting
parallel processing.
The findings demonstrated that the heart rate (HR) was the most significant contributing feature on the performance of the classifying model for almost all the participants. Our
interpretation is that challenging behaviours are usually accompanied by higher stress
levels, which lead to an increase in the HR above the baseline [55]. Studying the contributions of both kinetic and physiological features in behavioural classification helped us to
better understand the nature of challenging behaviours. More precisely, it was observed
that most challenging behaviours tend to involve some sort of specific hand movements.
And these hand movements were distinguished from regular hand motions through the
accelerometer. Considering HR along with other modalities can offer a valuable decision
support during moment-to-moment treatment planning for individuals with autism.
Another experiment was conducted to elucidate the relationship between the heart
rate variability (HRV) and challenging behaviours. We found that HRV decreased during
stress and stimulating episodes and increased during rest states. With the exception of
one participant, the HRV analysis for the participants demonstrated a strong correlation
between the fluctuations of HRV and the occurrence of challenging behaviours. A possible
explanation for these disparate findings is that children with autism may not have a
stable system for regulating emotions [52]. The children initially interacted with the social
Robotics 2023, 12, 55
10 of 13
robots with fear at different levels and intensities. This variation in emotions has led to
distinct representations of the HRV signal. Hence, HRV can be used as an indicator for the
occurrence of challenging behaviours when it is associated with high reactive interactions
in children with autism. Nonetheless, further research is required on a larger number of
participants to outline the psychological and physiological changes during the exhibition
of challenging behaviours.
Employing wearable sensors allows for lower costs, non-invasive, and less restraining
methods in tracking motor movements and physiological stress for children with autism.
Based on the findings, it is promising to derive HRV parameters from the wearable sensors
to acquire extra information. Dedicated warning techniques that are activated due to an
increase in challenging behaviour episodes would provide a valuable support for children
with autism. The benefits of such systems are magnified for nonverbal individuals who
have restricted means to express their stress to their parents or caregivers [56]. Hence,
parents or caregivers, or even a social robot could intervene early to remove the stimuli
causing that challenging behaviour [9].
One of the main limitations in this study was the low frequency of the observed
challenging behaviours compared to non-challenging ones. While resampling techniques
to balance the challenging and non-challenging behaviours can be used during the training phase, collecting more data that contain more instances of challenging behaviours is
essential to capture the full spectrum of such behaviours. The differences due to gender
was not investigated in this study. Future work will explore ways to collect more data,
including, but not limited to a longer observation period and larger and more diverse
participants’ pool. Another limitation was that data collection was restricted to utilizing a
wrist wearable device within controlled environmental conditions. Children with autism
might not tolerate the wrist wearable device. Hence, they might attempt to remove it,
throw it, or even hurt themselves with it (e.g., in head banging). Future work should
investigate different body locations that are less intrusive to place one or multiple wearable
devices that can recognize different patterns of behaviours at the same time. Additionally,
acquiring data should be conducted under less controlled conditions that are closer to their
daily living scenarios to generalize the observed challenging behaviours in children with
autism throughout the day.
6. Conclusions
The advances in technology can be exploited to help target challenging behaviours
among children with ASD. The combination of wearable sensors to detect behaviours
and social robots to respond have a great impact on the outcomes of therapy sessions.
In this study, we have conducted several investigations using wearable sensors and machine learning techniques to detect challenging behaviours among children with autism.
The wearable sensors acquired different physiological and kinetics signals from five children. Annotated video sessions and time-extracted features were considered to evaluate
the detection models. Four machine learning techniques were evaluated and the best,
based on XGBoost, was considered in further tests. Of all the evaluated machine learning
modes, the XGBoost showed better precision (0.88), recall (0.99), F1-Score (0.93), accuracy
(0.99), and achieved the lowest (i.e., best) time (i.e., 0.24 s) in predicting the test samples.
Features tests were conducted to evaluate the effects of adding each feature to the existing
pool of features. In terms of feature importance, heart rate (i.e., importance score = 405),
followed by electrodermal activity (i.e., importance score = 265), and temperature (i.e.,
importance score = 238) were found to be the most affecting features on the performance of
the prediction model. Testing the categories of features revealed that physiological-based
features provided more useful information to the machine learning model as compared to
kinetic features, hence, improving its performance considerably. The heart rate variability
changes based on the RMSSD parameter was also derived and investigated. This parameter
was found to correlate with challenging behaviours and to be a major contributor to the
prediction performance.
Robotics 2023, 12, 55
11 of 13
The outcomes of this work pave the way toward the development of methods and
tools based on machine learning techniques and wearables technologies that can be used to
detect challenging behaviours and be integrated into social robot-aided sessions.
Author Contributions: A.Q.A. and M.A. annotated the video sessions and analyzed the data; A.Y.A.
and A.Q.A. wrote the first draft; W.C.S. and O.C. established the annotation protocol; A.Q.A., A.Y.A.,
A.A.-A. and J.-J.C. produced the figures; A.A.-A., W.-C.S., O.C., U.A.Q. and J.-J.C. oversaw the
work and edited the manuscript. All authors have read and agreed to the published version of
the manuscript.
Funding: The work was supported by a research grant from QU Marubeni Concept to Prototype
Grant under the grant number M-CTP-CENG-2020-4.
Institutional Review Board Statement: The procedures for this work did not include invasive or
potentially hazardous methods and were in accordance with the Code of Ethics of the World Medical
Association (Declaration of Helsinki).
Informed Consent Statement: Parental consent for each participant was obtained by the centre.
Data Availability Statement: The data presented in this study are available on request from the
corresponding author. The statements made herein are solely the responsibility of the authors.
Conflicts of Interest: The authors declare no conflict of interest.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5® ); American Psychiatric Pub.:
Washington, DC, USA, 2013.
Estes, A.; Munson, J.; Dawson, G.; Koehler, E.; Zhou, X.H.; Abbott, R. Parenting stress and psychological functioning among
mothers of preschool children with autism and developmental delay. Autism 2009, 13, 375–387. [CrossRef] [PubMed]
Matson, J.L.; Gonzalez, M.L.; Rivet, T.T. Reliability of the autism spectrum disorder-behavior problems for children (ASD-BPC).
Res. Autism Spectr. Disord. 2008, 2, 696–706. [CrossRef]
Murphy, O.; Healy, O.; Leader, G. Risk factors for challenging behaviors among 157 children with autism spectrum disorder in
Ireland. Res. Autism Spectr. Disord. 2009, 3, 474–482. [CrossRef]
Kanne, S.M.; Mazurek, M.O. Aggression in children and adolescents with ASD: Prevalence and risk factors. J. Autism Dev. Disord.
2011, 41, 926–937. [CrossRef]
Baghdadli, A.; Pascal, C.; Grisi, S.; Aussilloux, C. Risk factors for self-injurious behaviours among 222 young children with
autistic disorders. J. Intellect. Disabil. Res. 2003, 47, 622–627. [CrossRef]
Bodfish, J.W.; Symons, F.J.; Parker, D.E.; Lewis, M.H. Varieties of repetitive behavior in autism: Comparisons to mental retardation.
J. Autism Dev. Disord. 2000, 30, 237–243. [CrossRef]
Christensen, D.L.; Braun, K.V.N.; Baio, J.; Bilder, D.; Charles, J.; Constantino, J.N.; Daniels, J.; Durkin, M.S.; Fitzgerald, R.T.;
Kurzius-Spencer, M.; et al. Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism
and Developmental Disabilities Monitoring Network, 11 sites, United States, 2012. Morb. Mortal. Wkly. Rep. Surveill. Summ. 2018,
65, 1–23. [CrossRef]
Cabibihan, J.J.; Chellali, R.; So, C.W.C.; Aldosari, M.; Connor, O.; Alhaddad, A.Y.; Javed, H. Social Robots and Wearable Sensors
for Mitigating Meltdowns in Autism—A Pilot Test. In Social Robotics; Springer International Publishing: Berlin/Heidelberg,
Germany, 2018; pp. 103–114.
Liu, W.; Yu, X.; Raj, B.; Yi, L.; Zou, X.; Li, M. Efficient autism spectrum disorder prediction with eye movement: A machine
learning framework. In Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction
(ACII), Xi’an, China, 21–24 September 2015; pp. 649–655.
Linstead, E.; German, R.; Dixon, D.; Granpeesheh, D.; Novack, M.; Powell, A. An application of neural networks to predicting
mastery of learning outcomes in the treatment of autism spectrum disorder. In Proceedings of the 2015 IEEE 14th International
Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, 9–11 December 2015; pp. 414–418.
Matson, J.L.; Nebel-Schwalm, M. Assessing challenging behaviors in children with autism spectrum disorders: A review. Res.
Dev. Disabil. 2007, 28, 567–579. [CrossRef]
Myers, S.M.; Johnson, C.P.; Council on Children with Disabilities. Management of children with autism spectrum disorders.
Pediatrics 2007, 120, 1162–1182. [CrossRef]
Blair, K.S.C.; Lee, I.S.; Cho, S.J.; Dunlap, G. Positive Behavior Support Through Family–School Collaboration for Young Children
With Autism. Top. Early Child. Spec. Educ. 2010, 31, 22–36. [CrossRef]
Neitzel, J. Positive Behavior Supports for Children and Youth with Autism Spectrum Disorders. Prev. Sch. Fail. Altern. Educ.
Child. Youth 2010, 54, 247–255. [CrossRef]
Ennis-Cole, D.L. Technology for Learners with Autism Spectrum Disorders; Springer: Berlin/Heidelberg, Germany, 2015.
Robotics 2023, 12, 55
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
12 of 13
Cabibihan, J.J.; Javed, H.; Aldosari, M.; Frazier, T.W.; Elbashir, H. Sensing Technologies for Autism Spectrum Disorder Screening
and Intervention. Sensors 2016, 17, 46. [CrossRef] [PubMed]
Martinez-Martin, E.; Escalona, F.; Cazorla, M. Socially assistive robots for older adults and people with autism: An overview.
Electronics 2020, 9, 367. [CrossRef]
Alabdulkareem, A.; Alhakbani, N.; Al-Nafjan, A. A systematic review of research on robot-assisted therapy for children with
autism. Sensors 2022, 22, 944. [CrossRef] [PubMed]
Cabibihan, J.J.; Javed, H.; Ang, M., Jr.; Aljunied, S.M. Why robots? A survey on the roles and benefits of social robots in the
therapy of children with autism. Int. J. Soc. Robot. 2013, 5, 593–618. [CrossRef]
Alhaddad, A.Y.; Mecheter, A.; Wadood, M.A.; Alsaari, A.S.; Mohammed, H.; Cabibihan, J.J. Anthropomorphism and its negative
attitudes, sociability, animacy, agency, and disturbance requirements for social robots: A pilot study. In Proceedings of the Social
Robotics: 13th International Conference, ICSR 2021, Singapore, 10–13 November 2021; Springer: Berlin/Heidelberg, Germany,
2021; pp. 791–796.
Katsanis, I.; Alhaddad, A.Y.; Cabibihan, J.J.; Moulianitis, V. Children Perceived Perception of a Mini-Humanoid Social Robot
Based on a Psychometric Scale: A Pilot Study in Greece. In Proceedings of the Social Robotics: 14th International Conference,
CSR 2022, Florence, Italy, 13–16 December 2022; Springer: Berlin/Heidelberg, Germany, 2023; pp. 13–22.
Alhaddad, A.Y.; Javed, H.; Connor, O.; Banire, B.; Thani, D.A.; Cabibihan, J.J. Robotic Trains as an Educational and Therapeutic
Tool for Autism Spectrum Disorder Intervention. In Robotics in Education; Springer International Publishing: Berlin/Heidelberg,
Germany, 2018; pp. 249–262.
Boccanfuso, L.; Barney, E.; Foster, C.; Ahn, Y.A.; Chawarska, K.; Scassellati, B.; Shic, F. Emotional Robot to Examine Differences in
Play Patterns and Affective Response of Children with and without ASD. In Proceedings of the Eleventh ACM/IEEE International
Conference on Human Robot Interaction, Christchurch, New Zealand, 7–10 March 2016; pp. 19–26.
Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Head Impact Severity Measures for Small Social Robots Thrown during Meltdown
in Autism. Int. J. Soc. Robot. 2019, 11, 255–270. [CrossRef]
Cabibihan, J.; Javed, H.; Sadasivuni, K.; Al Haddad, A. Smart Robotic Therapeutic Learning Toy. WIPO Patent WO2018033857,
22 February 2018.
Alhaddad, A.Y.; Cabibihan, J.J.; Hayek, A.; Bonarini, A. Influence of the shape and mass of a small robot when thrown to a
dummy human head. SN Appl. Sci. 2019, 1, 1468. [CrossRef]
Alhaddad, A.Y.; Cabibihan, J.J.; Hayek, A.; Bonarini, A. Safety experiments for small robots investigating the potential of soft
materials in mitigating the harm to the head due to impacts. SN Appl. Sci. 2019, 1, 476. [CrossRef]
Rudovic, O.; Lee, J.; Dai, M.; Schuller, B.; Picard, R.W. Personalized machine learning for robot perception of affect and
engagement in autism therapy. Sci. Robot. 2018, 3, eaao6760. [CrossRef]
Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Real-Time Social Robot’s Responses to Undesired Interactions between Children and
their Surroundings. Int. J. Soc. Robot. 2022, 1–9. . [CrossRef]
Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Influence of Reaction Time in the Emotional Response of a Companion Robot to a
Child’s Aggressive Interaction. Int. J. Soc. Robot. 2020, 12, 1279–1291. [CrossRef]
Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Recognition of aggressive interactions of children toward robotic toys. In Proceedings
of the 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), New Delhi, India,
14–18 October 2019; pp. 1–8.
McDonnell, A.; McCreadie, M.; Mills, R.; Deveau, R.; Anker, R.; Hayden, J. The role of physiological arousal in the management of
challenging behaviours in individuals with autistic spectrum disorders. Res. Dev. Disabil. 2015, 36, 311–322. [CrossRef] [PubMed]
Jansen, L.M.C.; de Wied, C.C.G.; Wiegant, V.M.; Westenberg, H.G.M.; Lahuis, B.E.; van Engeland, H. Autonomic and Neuroendocrine Responses to a Psychosocial Stressor in Adults with Autistic Spectrum Disorder. J. Autism Dev. Disord. 2006, 36, 891–899.
[CrossRef] [PubMed]
Alhaddad, A.Y.; Al-Ali, A.; Pandey, A.K.; Cabibihan, J.J. A Framework for Assistive Social Robots for Detecting Aggression in
Children. In Proceedings of the Social Robotics: 14th International Conference, ICSR 2022, Florence, Italy, 13–16 December 2022;
Springer: Berlin/Heidelberg, Germany, 2023; pp. 74–84.
Alban, A.Q.; Ayesh, M.; Alhaddad, A.Y.; Al-Ali, A.K.; So, W.C.; Connor, O.; Cabibihan, J.J. Detection of challenging behaviours
of children with autism using wearable sensors during interactions with social robots. In Proceedings of the 2021 30th IEEE
International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, BC, Canada, 8–12 August 2021;
pp. 852–857.
Alcañiz Raya, M.; Chicchi Giglioli, I.A.; Marín-Morales, J.; Higuera-Trujillo, J.L.; Olmos, E.; Minissi, M.E.; Teruel Garcia, G.; Sirera,
M.; Abad, L. Application of Supervised Machine Learning for Behavioral Biomarkers of Autism Spectrum Disorder Based on
Electrodermal Activity and Virtual Reality. Front. Hum. Neurosci. 2020, 14, 90. [CrossRef] [PubMed]
Masino, A.J.; Forsyth, D.; Nuske, H.; Herrington, J.; Pennington, J.; Kushleyeva, Y.; Bonafide, C.P. M-Health and autism:
Recognizing stress and anxiety with machine learning and wearables data. In Proceedings of the 2019 IEEE 32nd International
Symposium on Computer-Based Medical Systems (CBMS), Cordoba, Spain, 5–7 June 2019; pp. 714–719.
Sarabadani, S.; Schudlo, L.C.; Samadani, A.A.; Kushski, A. Physiological detection of affective states in children with autism
spectrum disorder. IEEE Trans. Affect. Comput. 2018, 11, 588–600. [CrossRef]
Robotics 2023, 12, 55
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
13 of 13
Williams, R.M.; Gilbert, J.E. Perseverations of the academy: A survey of wearable technologies applied to autism intervention.
Int. J. Hum.-Comput. Stud. 2020, 143, 102485. [CrossRef]
Fazana, F.; Alsadoon, A.; Prasad, P.; Costadopoulos, N.; Elchouemi, A.; Sreedharan, S. Integration of assistive and wearable
technology to improve communication, social interaction and health monitoring for children with autism spectrum disorder
(ASD). In Proceedings of the 2017 IEEE Region 10 Symposium (TENSYMP), Cochin, India, 14–16 July 2017; pp. 1–5.
Goodwin, M.S.; Haghighi, M.; Tang, Q.; Akcakaya, M.; Erdogmus, D.; Intille, S. Moving towards a real-time system for
automatically recognizing stereotypical motor movements in individuals on the autism spectrum using wireless accelerometry.
In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA,
13–17 September 2014; pp. 861–872.
Rad, N.M.; Kia, S.M.; Zarbo, C.; van Laarhoven, T.; Jurman, G.; Venuti, P.; Marchiori, E.; Furlanello, C. Deep learning for automatic
stereotypical motor movement detection using wearable sensors in autism spectrum disorders. Signal Process. 2018, 144, 180–191.
Plötz, T.; Hammerla, N.Y.; Rozga, A.; Reavis, A.; Call, N.; Abowd, G.D. Automatic assessment of problem behavior in individuals
with developmental disabilities. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA,
5–8 September 2012; pp. 391–400.
Rad, N.M.; Furlanello, C. Applying Deep Learning to Stereotypical Motor Movement Detection in Autism Spectrum Disorders.
In Proceedings of the 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), Barcelona, Spain, 12–15
December 2016.
Pollreisz, D.; TaheriNejad, N. A simple algorithm for emotion recognition, using physiological signals of a smart watch. In
Proceedings of the 2017 39th annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC),
Jeju, Republic of Korea, 1–15 July 2017; pp. 2353–2356.
Jarraya, S.K.; Masmoudi, M.; Hammami, M. A comparative study of Autistic Children Emotion recognition based on SpatioTemporal and Deep analysis of facial expressions features during a Meltdown Crisis. Multimed. Tools Appl. 2020, 80, 83–125.
[CrossRef]
Lee, C.; Yoo, S.; Park, Y.; Kim, N.; Jeong, K.; Lee, B. Using neural network to recognize human emotions from heart rate variability
and skin resistance. In Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai,
China, 17–18 January 2006; pp. 5523–5525.
Park, H.; Dong, S.Y.; Lee, M.; Youn, I. The role of heart-rate variability parameters in activity recognition and energy-expenditure
estimation using wearable sensors. Sensors 2017, 17, 1698. [CrossRef]
Yap, C.Y.; Ng, K.H.; Cheah, Y.; Lim, S.Y.; Price, J.; De Vries, M. App4Autism: An Integrated Assistive Technology with Heart Rate
Monitoring for Children with Autism. In Proceedings of the International Visual Informatics Conference, Bangi, Malaysia, 19–21
November 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 498–512.
Lydon, S.; Healy, O.; Dwyer, M. An examination of heart rate during challenging behavior in Autism Spectrum Disorder. J. Dev.
Phys. Disabil. 2013, 25, 149–170. [CrossRef]
Nuske, H.J.; Finkel, E.; Hedley, D.; Parma, V.; Tomczuk, L.; Pellecchia, M.; Herrington, J.; Marcus, S.C.; Mandell, D.S.; Dissanayake,
C. Heart rate increase predicts challenging behavior episodes in preschoolers with autism. Stress 2019, 22, 303–311. [CrossRef]
[PubMed]
Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.;
et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830.
Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the KDD ’16: The 22nd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; ACM:
New York, NY, USA, 2016; pp. 785–794.
Goto, M.; Nagashima, M.; Baba, R.; Nagano, Y.; Yokota, M.; Nishibata, K.; Tsuji, A. Analysis of heart rate variability demonstrates
effects of development on vagal modulation of heart rate in healthy children. J. Pediatr. 1997, 130, 725–729. [CrossRef] [PubMed]
DiStefano, C.; Shih, W.; Kaiser, A.; Landa, R.; Kasari, C. Communication growth in minimally verbal children with ASD: The
importance of interaction. Autism Res. 2016, 9, 1093–1102. [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.