8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom 2017) • September 11-14, 2017 • Debrecen, Hungary

Cognitive workload classification using cardiovascular

measures and dynamic features

Eydis H. Magnusdottir, Kamilla R. Johannsdottir, Christian Bean, Brynjar Olafsson and Jon Gudnason
Center for Analysis and Design of Intelligent Agents, Reykjavik University, Iceland,
Email: eydis07@ru.is, kamilla@ru.is, christianbean@ru.is, brynjaro14@ru.is, jg@ru.is

Abstract—Monitoring cognitive workload has the potential to improve measures to the static cardiovascular signals and using supervised
performance and fidelity in human decision making through a real-time learning methods for classification. A cognitive workload experiment
monitoring model. Multiple studies have shown a successful binary classi-
was conducted with variations in cognitive workload introduced
fication of high and low workload using various methods and often focused
on multiple physiological signals. A more detailed detection of cognitive through different versions of the original Stroop task [15]. Speech-
workload is needed for a meaningful and reliable workload monitoring and cardiovascular information were recorded from 96 participants
tool. This study focuses on trinary workload classification of parameters solving the set tasks by responding verbally to the stimuli. The
extracted from the cardiovascular system. The experiment was validated feature set contained 10 cardiovascular signal parameters for each
with the use of a database containing 96 participants performing tasks
designed to induce slight variations in cognitive workload. Two distinct heartbeat comprising of a detailed hemodynamic profile and derived
supervised learning classifying methods were used and their likelihood delta and delta-delta coefficients for each of these. A separate leave-
score used for the classification schemes of (1) each heartbeat and (2) one-out participant-dependent classifier of the three difficulty levels
each task screen. The results show that the support vector classifier was trained for each participant. Two supervised learning classifiers
outperforms the random forest with the average misclassification rate of
20.44% using the whole screen classification scheme instead of individual
were used; support vector machines (SVM) and random forest (RF)
heartbeat classification. and the likelihood score was evaluated through two classification
schemes, for each heartbeat and for each task screen. The conclusion
is that a finer cognitive workload distinction can be reached with the
I. I NTRODUCTION
combined feature set of cardiovascular signals and delta coefficients,
Monitoring cognitive workload in safety-critical job environ- classifying for each screen with the SVM classifier.
ments, e.g. aviation or surgery, is of paramount importance for
This research is an important contribution towards the aim of
enhanced human performance and decision making. The relationship
making today’s computers capable of reading into the human cogni-
between cognitive workload and performance has been well stud-
tive workload state. In particular, in the field of affective computing
ied [1] and the connection between cognitive workload and physical
where the goal is to simulate empathy towards the user by sensing
health has also been highlighted [2]. The cardiovascular signal,
their emotional and physiological states [16], [17].
measured through heart rate (HR) and heart rate variability (HRV)
or blood pressure, provides a promising method to monitor cognitive The context of the work is presented in Sec. II. The experimental
workload. The measures are non-intrusive and the cardiovascular methodology, cardiovascular measures, feature extraction, classifica-
system is quite sensitive to cognitive stimuli [3], [4]. Moreover, tion setup and design are described in Sec. III. The results are detailed
multiple studies have linked increased cognitive workload to reliable in Sec. IV and the paper is concluded with discussion in Sec. V.
and measurable changes in the cardiovascular signal [5], [6], [7], [8],
[9].
II. C ARDIOVASCULAR SIGNAL ANALYSIS FOR DETECTING
However, despite great success, current methods used to analyze COGNITIVE WORKLOAD
the cardiovascular signal, have not been able to consistently detect
Measurements of cognitive workload are generally gathered
small workload differences [8], [9], [10]. The majority of the work
through three methods; (1) self assessment, (2) behavioral monitoring
is based on comparing averages, calculated for the duration of a
and (3) by measuring physiological signals. More recently the focus
particular task of higher or lower workload. These studies often fail
has been on physiological measures, in particular, cardiovascular
to detect adjacent levels of increasing or decreasing workload [8],
reactivity of the individual to a performed task. Measuring the
[9], [10]. Few studies have attempted to use various machine learn-
individuals cardiovascular reactivity is well suited for the automatic
ing algorithms to classify cognitive workload states (i.e. higher,
monitoring of cognitive workload as it is relatively non-intrusive,
lower) [11], [12], [13], [14]. The results, based on EEG signals
objective, and takes place in real time [18].
along with other physiological measures, show a successful binary
(high, low) classification of cognitive workload. A more detailed Several studies have shown a reliable cardiovascular reactivity
detection of multiple cognitive workload states, particularly based on to increased workload [5], [6], [7], [19], [20], [21]. However, the
the cardiovascular signal, remains a challenge. detection of cognitive workload through cardiovascular reactivity
has to date mainly been binary. That is, current methods detect
The main objective of this work is to provide a trinary classifi-
high and low workload, or task onset and task offset [22]. These
cation of cognitive workload with parameters extracted solely from
methods often fail to distinguish between multiple adjacent levels
the cardiovascular system. This was done by adding time dynamic
of increasing or decreasing workload. For example, in a study by
This work is sponsored ISAVIA, Icelandair and the Icelandic Center for
Wilson [9], HR successfully distinguished take-off and landing from
Research (RANNIS) under the project Monitoring cognitive workload in ATC other segments of flight in a visual and instrumented flight rules
using speech analysis, Grant No 130749051. simulated flights (VFR and IRF). However, neither HR nor HRV

978-1-5386-1264-4/17/$31.00 ©2017 IEEE 000351

E. Huld Magnusdottir et al. • Cognitive workload classification using cardiovascular measures and dynamic features

distinguished between other flight segments (22 in total), although III. M ETHODOLOGY
they all varied in their load level. Furthermore, in a study by Vogt
et al. [8], HR significantly detected increased number of aircrafts in A. Experiments and data
two different simulations (en-route and tower) and increased number A total of 96 participants underwent a session of tasks lasting 45-
of conflicts in en-route simulation. HR however, did not distinguish 50 min. During the session, speech and cardiovascular signals were
higher load of vertical traffic or pilot error in en-route simulation nor recorded, while the participant was engaged in the Stroop tasks with
predictable or unpredictable conflict in tower simulation. 3 min. resting periods between the different Stroop task levels. The
Stroop tasks [15] were designed with a set (35+1) of words of five
The problem is largely methodological. For most parts, workload colors appearing on a screen and the participants task was to State
detection is based on calculating and comparing averages over ex- the color of the words aloud. The cognitive workload levels were
tended task periods that vary in their level of workload. The problem introduced with various levels of congruency, in-congruency and time
with this approach is that it forgoes the information embedded within limits.
the time segments for different tasks such as the body’s natural self-
adjustment mechanism to long term fluctuations, the baroreflex [23]. Level 1 Congruent sets of screens (word and color match) with
It has therefore been suggested that if workload is being detected all the words appearing at the same time.
through comparing means, only the first 30 to 60 seconds should Level 2 In-congruent sets of screens with the alternating two
be considered [23], [24], [25], [26], [27]. Longer time segments levels of 30% and 70% of in-congruency (word and
may reflect a combined activity of two opposing systems, i.e. the color do not match) with all the words appearing at the
sympathetic and the parasympathetic system [23], [27]. same time.
Level 3 Sets of screens with one word appearing at a time at
Another approach is to classify changes in cognitive workload randomly timed intervals of 0.75 sec. and 0.65 sec. Here
state using various machine learning algorithms. Few studies have the same in-congruency setup was used as in Level 2.
shown that a binary (high, low) classification of cognitive workload
is possible by training neural networks using multiple physiological A Latin square technique was used to alternate the order of the
signals, including EEG [11], [12], [13], [14]. Other machine learning cognitive workload levels into six between participants.
algorithms include discrimination analysis and support vectors (SVM)
(see [28], [29], [30]), as well as logic regression and classification B. Cardiovascular measures and delta features
trees [11], and kernel partial least square classifier [31]. For most
parts these multi signal classification methods gain high accuracy The Finometer Pro from Finapres was used to record cardiovas-
for a two level classification (i.e. high, low; present, absent). But a cular signals of the participants during the experiments [34], [35].
more detailed classification seems hard to reach. Brouwer et al. [28] The signals were obtained using a finger cuff and an upper arm cuff
used SVM to detect memory load using the n-back task. They was used for calibration of the reconstructed blood pressure. Ten
trained the model on the first three quarters of task performance and measures for each heartbeat were obtained from the output of the
tested it on the last quarter of task performance. Binary classification Finometer Pro system; 1) Heart rate, 2) Systolic pressure, 3) Diastolic
between all three levels of memory load (0-back, 1-back, 2-back) pressure, 4) Mean pressure, 5) Pulse interval, 6) Stroke volume, 7)
were possible, but the highest accuracy was gained for classifying the Left ventricular pressure energy, 8) Cardiac output, 9) Total peripheral
greater load differences (high, low). Higher accuracy was also gained resistance, 10) Maximum steepness and are designated as ct,i where
when both ERP and spectral power signals were used together. In i ∈ {1, 2, . . . , 10} is the index for the measure and t is the integer
general, most of the prior work has focused on EEG along with other time index of the heartbeat. The ten dimensional feature vector for a
physiological signals, showing high binary classification accuracy. heartbeat at time t is therefore ct = [ct,1 , ct,2 , . . . , ct,10 ]T .
Very little work has been done on classifying cognitive workload The vector ct only includes information about the current heart-
based solely on cardiovascular reactivity. But it has been pointed out beat. Time derivatives are appended to the vector to include informa-
that a classification based on the cardiovascular signal may be more tion about the rate of change of the static cardiovascular measures.
suitable and practical in real task environments compared to using (1)
These dynamic measures are called delta δt and delta-delta δt
(2)
EEG [31]. coefficients and are calculated from the measure ct,i along the time
index by using,
Yin et al. [32], achieved 77.5% classification accuracy for three PN
task-difficulty levels using only the speech signal. In their study, (1) n=1
n(ct+n,i − ct−n,i )
participants performed different variations of the original Stroop task δt,i = PN , (1)
2 n=1
n2
in addition to a standard reading task. They concatenated standard
mel-frequency cepstrum coefficients with prosodic features and used where N denotes the number adjacent heartbeat measures before and
Gaussian mixture models to classify workload performance. Yap et after t used to derive the delta feature. This is set to N = 2 for this
al. [33] continued with this line of research by adding voice source particular work (resulting in a window size of 5). A ten dimensional
(1) (1) (1) (1)
features based on cepstral peak prominence and produced results for delta vector is therefore obtained by δt = [δt,1 , δt,2 , . . . , δt,10 ]T
(2)
three Stroop test levels. and the delta-delta feature vector δt are calculated using the same
(1)
formula but using δt,i instead of ct,i . Three feature sets were
In the current study the focus is set on classification of cardiovas- evaluated in this work, one without delta and delta-delta features,
cular signals with velocity and acceleration information from 5 heart- i.e. just the ten dimensional ct , the second only with the delta
(1)T
beat frames to include time dynamic information. The classification features, i.e. the twenty dimensional dt = [cTt , δt ]T and the
tasks are performed with two types of supervised learning methods third with all the vectors concatenated, i.e. the thirty dimensional
(1)T (2)T
with testing the classification schemes of screens or heartbeat. et = [cTt , δt , δt ]T .

000352
 8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom 2017) • September 11-14, 2017 • Debrecen, Hungary

C. Classifier design and compared with the other feature set combinations and the random
forest classifier. Finally, a further analysis of the performance of
The classifiers designed and implemented in this particular work
individual participants is presented.
are based on the cardiovascular measures either from a single heart-
beat feature vector xt (without delta features, xt = ct or with delta
A. Support vector machines using delta-delta features
features, xt = dt or xt = et ) or a sequence of heartbeats from a sin-
gle screen represented with the data matrix X = [x1 , x2 , . . . , xT ]T . Table I shows the confusion table of how all heartbeats in the
The classifiers return soft scores and are denoted either as yk (xt ) for data set are classified using the support vector machine classifier
the single heartbeat classifier or yk (X) for the sequence classifier. ySV M (xt ) and the delta-delta feature set et . The numbers are
The index k ∈ {1, 2, 3} denotes Stroop level one, two or three obtained by summing individual confusion tables of each participant.
respectively. The vector y(xt ) = [y1 (xt ), y2 (xt ), y3 (xt )]T contains The table also shows the misclassification rate and the mistrust rate
the soft scores for the heartbeat at time index t and the heartbeat of each Stroop level. For example, if there is a 32.29% chance of
classification simply chooses the class with the maximum value misclassification if the heartbeat is known to be from Stroop Level 2
in that vector. For the sequence classification, the soft scores are and if an unknown heartbeat is classified as Stroop Level 2, there
collected in an output matrix Y = [y(x1 ), y(x2 ), . . . , y(xT )]T . The is a 30.99% chance of an error. There errors seem to be evenly
classification is then obtained by summing the soft scores together distributed over the remaining classes which indicates that no level
over the screen to obtain y(X) = [y1 (X), y2 (X), y3 (X)]T and then is dominating the classification. However, Stroop Level 3 seem to
the class according to the maximum value of that vector is chosen. be performing slightly better than the other levels with a 23.13%
misclassification rate and 25.69% mistrust rate. The participant-
Two types of supervised learning classifiers were implemented
average test set misclassification rate is given as 29.26%.
using the statistics toolbox in Matlab: Support Vector Machine
(SVM), ySV M (xt ) and Random Forests (RF), yRF (xt ). A support TABLE I. Confusion matrix for the number of heartbeats classified with
vector machine is fundamentally a binary classifier so to solve the the et feature set classifying with the SVM.
trinary classification problem, three two-class one-vs-all binary SVM
classifiers were implemented, one for each Stroop level. The soft score Stroop Stroop Stroop MCR
output yk (xt ) is the signed distance from the decision boundary for L1 L2 L3 [%]
each of the three classifiers where k ∈ {1, 2, 3}. The class is then Stroop L1 13790 3749 3554 34.62
determined by the one-vs-all classifier which obtains the maximum Stroop L2 3745 16536 4142 32.29
signed distance from the decision boundary. If all scores are negative Stroop L3 3022 3676 22254 23.13
then the class that is the closest to the decision boundary with the Mistrust rate [%] 32.92 30.99 25.69 29.26
least negative score is chosen.
A random forest classifier is trained for each heartbeat using one Table II shows the confusion table of how all screens in the
hundred decision trees. The minimum number of observations in a data set are classified also using the support vector machine classifier
leaf was set to one. The soft score yk (xt ) for the random forest ySV M (xt ) and the delta-delta feature set et . As with the heartbeat
classifier is the proportion of trees in the ensemble predicting class k confusion Table I, the numbers are obtained by summing individual
and is interpreted as the probability of this observation xt originating confusion tables of each participant. The misclassification and mis-
from this class. trust rates are also shown for each Stroop level and the participant-
average test set misclassification rate is reported. The table shows that
D. Classifier training and evaluation the screen based classifier is also balanced with respect to the Stroop
As this work does not deal with differences between individuals, levels.
participant-dependent classifiers were trained in this work. This means TABLE II. Confusion matrix for the number of screens classified with
that 96 separate classifiers were trained and tested, one for each the et feature set classifying with the SVM.
participant. A leave-one-out strategy was used where the test sample
that is left out corresponds to a single screen. The other twenty screens Stroop Stroop Stroop MCR
for that participant were used to train the classifiers. In the case of L1 L2 L3 [%]
single heartbeat classifiers the number of test results corresponded Stroop L1 499 95 78 25.74
to the number of heartbeats contained within the test screen (an Stroop L2 63 440 73 23.61
average of 776) but in the case of the sequence classification for the Stroop L3 38 65 665 13.41
entire screen only one result was obtained. The experiment was then Mistrust rate [%] 16.83 26.66 18.50 20.44
repeated where another screen from the set of twenty-one screens was
reserved for testing. The results were then cumulated over the twenty-
one trials. This procedure was then repeated for each participant in The advantage of using a sequence of heartbeats is clearly seen
the study. by comparing Table I and Table II. The participant-average test set
misclassification rate improves from 29.26% to 20.44%.
IV. R ESULTS
B. Overall performance results
The results are first presented for the support vector machine
using using delta-delta feature sets. We show how we construct a Table III shows the participant-average test set misclassification
confusion table from the leave-one-out experiments and compare the rate for the support vector machine classifier. Each line shows the
single heartbeat classifier with the sequence screen classifier. The results for the three feature sets: without delta features, ct , with
results are then summarized as single average misclassification rates delta features, dt and with delta and delta-delta features et . The two

000353
E. Huld Magnusdottir et al. • Cognitive workload classification using cardiovascular measures and dynamic features

columns show the results for the screen sequence classifier ySV M (X)
20
and the individual heartbeat classifier ySV M (xt ). Each result shows
the average and the standard deviation of the individual participant 15

Beat [freq]
test set misclassification rates. For example the SVM screen classifier
10
using the cardiovascular feature set without delta features, ct , obtains
26.34% misclassification rate on average over all participants and the 5

standard deviation over all participants is 16.95%. 0

0 10 20 30 40 50 60 70 80
TABLE III. Average MCR [%] over all participants comparing different
combinations of classification schemes and feature sets for the SVM 20
classifier.
15

Screen [freq]
Feature set Screen Heartbeat
10
xt y(X) y(xt )
ct 26.34±16.95 36.11±13.88 5
dt 23.42±17.08 33.73±14.11
0
et 20.44±15.48 29.26±13.27 0 10 20 30 40 50 60 70 80
MCR [%]

The table shows that adding delta and delta-delta features im- Fig. 1. Histogram of test set misclassification rate (MCR) [%] for all
participants in the set using the SVM classifier with the delta and delta-delta
proves performance considerably for both the screen based and the features. The upper panel shows the individual heartbeat classifier results and
heartbeat based SVM classifier. The best results of 20.44±15.48% are the lower panel shows the screen-based classifier results.
obtained using delta and delta-delta features, et and the screen based
classifier y(X). These results are also the best results obtained in
this work and correspond to the confusion table presented in Table II. of heartbeats to do the classification. The average misclassification
Further analysis of these results are presented in Section IV-C. rate for the screen-based classification is 22.72% and the range if from
zero to 85.71% (not shown in the figure). The number of participants
Table IV shows the set of results for the random forest classifier. achieving misclassification rate under 10% is 22 participants (out of
The overall results are inferior to the support vector machine results. 96).
For the heartbeat based random forest classifier the performance
improves by adding delta and delta-delta features which is consistent
with the support vector machines results. However the screen based 20

classifier is not improved by adding the delta or delta-delta features. 15

Beat [freq]

TABLE IV. Average MCR [%] over all participants comparing different 10
combinations of classification schemes and feature sets for the RF classifier.
5
Feature set Screen Heartbeat
0
xt y(X) y(xt ) 0 10 20 30 40 50 60 70 80
ct 22.72±15.99 37.21±12.79
dt 23.91±15.69 35.75±12.73 20

et 24.21±15.89 34.50±12.76 15
Screen [freq]

10

5
C. Participant distributions
0
0 10 20 30 40 50 60 70 80
Figure 1 shows the histogram of the test set misclassification MCR [%]
rates for all participants for the support vector machine classifier
and delta and delta-delta features. The first panel shows the results
Fig. 2. Histogram of test set misclassification rate (MCR) [%] for all
for the single heartbeat classifier and the second panel shows the
participants in the set using the RF classifier without the dynamic features.
screen based classifier. The figure corresponds to the last line in The upper panel shows the individual heartbeat classifier results and the lower
Table III and demonstrates how the results improve when the classifier panel shows the screen-based classifier results.
can accumulate the results over an entire screen before making
decision. The figure also shows how widely distributed the results are
depending on the participants, as is evident in the standard deviation. V. D ISCUSSION AND C ONCLUSIONS
The range for the screen based classifier is from zero misclassification
The experiments presented in this work show how the car-
rate to 76.19%. More interesting results evident from the histogram
diovascular system responds to cognitive stimuli when using the
is that 34 participants (out of 96) achieve misclassification rate under
Stroop tests. Support vector machines and Random forests were used
10%.
as classifiers for static cardiovascular measures and their dynamic
Figure 2 shows the histogram of the misclassification rate when features. The results show obvious cardiovascular reactivity to the
using the Random forest classifier without dynamic features. The fig- different cognitive stimuli. The Stroop levels that were tested in this
ure shows how well the performance improves when using a sequence work all demand the same or very similar physical workload from

000354
 8th IEEE International Conference on Cognitive Infocommunications (CogInfoCom 2017) • September 11-14, 2017 • Debrecen, Hungary

the participant since the participant responded verbally. The study niques combining cardiovascular and audio signals performing the
does not include cardiovascular measures from the baseline period same classification tasks. The importance of different time segment
when the participant is completely at rest. The classification results windows is also established in this study by using the delta and delta-
can therefore be interpreted solely on the cognitive workload level delta features, indicating that exploring further the variations in time
that is elicited using the different Stroop levels. The best classifier segments might be beneficial to classification accuracy.
is able to distinguish between low, medium and high Stroop level
with an average of 20.44% test set misclassification rate. Further In conclusion, cognitive workload classification based on the
analysis shows that more than 34 participants out of 96 achieve 10% cardiovascular signal is possible and might provide a reliable, non-
misclassification rate or less. This shows that cognitive workload intrusive monitoring tool. The results provide an important input to the
strongly affects the cardiovascular system. The methodology does literature on workload monitoring as prior work has mainly focused
not however give an insight into what aspect of the cardiovascular on binary (high, low) classification using multiple physiological
measures are affected. signals, including EEG. But as pointed out by Fong [11], basing the
classification on the cardiovascular signal may be more suitable in
The results also showed that the support vector machine classifier the field compared to using the EEG signal. The trinary classification
outperformed the random forest classifier. The best SVM results is accuracy reached in this study is comparable to binary (high, low)
20.44% while the best RF result is 22.72%. Both methods benefit classification accuracy reached in prior work using multiple physio-
greatly from classifying on a whole sequence of heartbeats (screens) logical signals.
instead of individual heartbeats.
The work also introduced dynamic features for the cardiovascular R EFERENCES
methods that have been called delta and delta-delta features in the [1] E. Galy, M. Cariou, and C. Mélan, “What is the relationship between
speech processing community. Both the SVM and RF classifiers that mental workload factors and cognitive load types?” International Jour-
are based on single heartbeats benefited from the addition of dynamic nal of Psychophysiology, vol. 83, no. 3, pp. 269–275, Mar. 2012.
features. However the screen-based RF classifier did not benefit from [2] M. A. Kompier, B. Aust, A. M. van den Berg, and J. Siegrist, “Stress
the addition. prevention in bus drivers: evaluation of 13 natural experiments.” Journal
of occupational health psychology, vol. 5, pp. 11–31, 2000.
Signal classification offers a new approach to cognitive workload [3] B. Mehler, B. Reimer, and M. Zec, “Defining workload in the context
monitoring. The results presented in this study compare well with of driver state detection and HMI evaluation,” in Proceedings of the 4th
International Conference on Automotive User Interfaces and Interactive
other classification work in this field. For most parts, reported results Vehicular Applications. ACM, 2012, pp. 187–191.
are based on a binary classification, sometimes using no vs. some or
[4] M. W. Scerbo, “Stress, workload, and boredom in vigilance: a problem
low vs. high cognitive workload [11], [12], [13], [14], [28]. It is clear and an answer.” Stress, workload, and fatigue, 2001.
from prior work, that high accuracy in binary classification is pos- [5] G. Borghini, L. Astolfi, G. Vecchiato, D. Mattia, and F. Babiloni, “Mea-
sible, in particular, if combining multiple physiological signals. The suring neurophysiological signals in aircraft pilots and car drivers for the
obstacle however, has been to move beyond the binary classification. assessment of mental workload, fatigue and drowsiness,” Neuroscience
& Biobehavioral Reviews, vol. 44, pp. 58–75, Jul. 2014.
In Wilson and Russel [13] a high accuracy was gained using [6] N. Dahlstrom and S. Nahlinder, “Mental workload in aircraft and
an ANN for classifying high and low cognitive workload. Their simulator during basic civil aviation training,” The International journal
classification was based on multiple physiological signals, including of aviation psychology, vol. 19, no. 4, pp. 309–325, 2009.
EEG. The network however, was not capable of classifying accurately [7] R. N. Nolte, R. A. Wright, C. Turner, and R. J. Contrada, “Reported
between more than two cognitive workload states (high and low). fatigue, difficulty, and cardiovascular response to a memory challenge,”
International Journal of Psychophysiology, vol. 69, no. 1, pp. 1–8, 2008.
When the training scenario included four and seven different states
[8] J. Vogt, T. Hagemann, and M. Kastner, “The impact of workload on
based on complexity and number of aircraft, the classifier confused
heart rate and blood pressure in en-route and tower air traffic control,”
adjacent states, not distinguishing between low and medium or Journal of psychophysiology, vol. 20, no. 4, pp. 297–314, 2006.
medium and high. Similarly, Brouwer et al. [28] using a SVM to [9] G. F. Wilson, “An Analysis of Mental Workload in Pilots During Flight
classify mental load, found a high accuracy in classifying high and Using Multiple Psychophysiological Measures,” International Journal
low load, whereas, low to medium or medium to high classification of Aviation Psychology, vol. 12, no. 1, pp. 3–18, Jan. 2002.
was not as accurate. Their classification was also based on EEG and [10] D. B. Kaber, C. M. Perry, N. Segall, and M. A. Sheik-Nainar, “Workload
higher accuracy was gained when two EEG signals (ERP and spectral state classification with automation during simulated air traffic control,”
The International Journal of Aviation Psychology, vol. 17, no. 4, pp.
power) were combined.
371–390, 2007.
The present study goes beyond the state of the art by successfully [11] A. Fong, C. Sibley, A. Cole, C. Baldwin, and J. Coyne, “A comparison
demonstrating a trinary classification of low, medium and high cog- of artificial neural networks, logistic regressions, and classification
trees for modeling mental workload in real-time,” in Proceedings of
nitive workload states. Furthermore, the classification here is based the Human Factors and Ergonomics Society Annual Meeting, vol. 54.
entirely on the cardiovascular signal whereas prior work has for most SAGE Publications Sage CA: Los Angeles, CA, 2010, pp. 1709–1712.
parts focused on EEG often along with other physiological signals. [12] G. F. Wilson, “Real-time adaptive aiding using psychophysiological op-
Few studies have attempted to classify workload based on the speech erator state assessment,” Publication of: Ashgate Publishing Company,
signal. In Magnusdottir et al. [36], three load states were classified 2001.
using vocal tract features and SVM. However, the classification [13] G. F. Wilson and C. A. Russell, “Operator Functional State Classifi-
accuracy was 67,5%. The present results show a better classification cation Using Multiple Psychophysiological Features in an Air Traffic
Control Task,” Human Factors: The Journal of the Human Factors and
accuracy using the cardiovascular signal. In both cases, variation of Ergonomics Society, vol. 45, no. 3, pp. 381–389, Sep. 2003.
the original Stroop task were used to induce cognitive workload. [14] G. F. Wilson, J. Estepp, and I. Davis, “A comparison of performance
As a consequence of these findings, continuing research should aim and psychophysiological classification of complex task performance,”
to explore the synergistic affects of multi-modal measurement tech- in Proceedings of the Human Factors and Ergonomics Society Annual

000355
E. Huld Magnusdottir et al. • Cognitive workload classification using cardiovascular measures and dynamic features

Meeting, vol. 53. SAGE Publications Sage CA: Los Angeles, CA, [35] I. Guelen, B. E. Westerhof, G. L. van der Sar, G. A. van Montfrans,
2009, pp. 141–145. F. Kiemeneij, K. H. Wesseling, and W. J. Bos, “Finometer, finger
[15] J. R. Stroop, “Studies of interference in serial verbal reactions.” Journal pressure measurements with the possibility to reconstruct brachial
of experimental psychology, vol. 18, no. 6, p. 643, 1935. pressure,” Blood pressure monitoring, vol. 8, no. 1, pp. 27–30, 2003.
[16] P. Baranyi and A. Csapo, “Cognitive infocommunications: CogIn- [36] E. H. Magnusdottir, M. Borsky, M. Meier, K. Johannsdottir, and
foCom,” in 2010 11th International Symposium on Computational J. Gudnason, “Monitoring Cognitive Workload Using Vocal Tract and
Intelligence and Informatics (CINTI), Nov. 2010, pp. 141–146. Voice Source Features,” Periodica Polytechnica Electrical Engineering
and Computer Science, May 2017.
[17] P. Baranyi, A. Csapo, and G. Sallai, Cognitive Infocommunica-
tions (CogInfoCom). Springer, Nov. 2015, google-Books-ID: lLjfC-
gAAQBAJ.
[18] F. Psychophysiologie, “Recording methods in applied environments,”
Engineering psychophysiology: Issues and applications, p. 111, 2000.
[19] B. Mehler, B. Reimer, and Y. Wang, “A comparison of heart rate and
heart rate variability indices in distinguishing single-task driving and
driving under secondary cognitive workload,” in Proceedings of the
Sixth International Driving Symposium on Human Factors in Driver
Assessment, Training and Vehicle Design, 2011, pp. 590–597.
[20] R. C. Smith, “Stress, anxiety, and the air traffic control specialist: Some
conclusions from a decade of research,” Federal Aviation Administra-
tion Washington DC Office of Aviation Medicine, Tech. Rep., 1980.
[21] J. Taelman, S. Vandeput, E. Vlemincx, A. Spaepen, and S. Van Huffel,
“Instantaneous changes in heart rate regulation due to mental load in
simulated office work,” European journal of applied physiology, vol.
111, no. 7, pp. 1497–1505, 2011.
[22] E. A. Byrne and R. Parasuraman, “Psychophysiology and adaptive
automation,” Biological psychology, vol. 42, no. 3, pp. 249–268, 1996.
[23] A. Stuiver, K. A. Brookhuis, D. de Waard, and B. Mulder, “Short-
term cardiovascular measures for driver support: Increasing sensitivity
for detecting changes in mental workload,” International Journal of
Psychophysiology, vol. 92, no. 1, pp. 35–41, Apr. 2014.
[24] M. De Rivecourt, M. N. Kuperus, W. J. Post, and L. J. M. Mulder,
“Cardiovascular and eye activity measures as indices for momentary
changes in mental effort during simulated flight,” Ergonomics, vol. 51,
no. 9, pp. 1295–1319, 2008.
[25] D. de Waard, A. Kruizinga, and K. A. Brookhuis, “The consequences
of an increase in heavy goods vehicles for passenger car drivers’
mental workload and behaviour: a simulator study,” Accident Analysis
& Prevention, vol. 40, no. 2, pp. 818–828, 2008.
[26] A. F. Kramer, “Physiological metrics of mental workload: A review of
recent progress,” Multiple-task performance, pp. 279–328, 1991.
[27] A. Stuiver, D. De Waard, K. A. Brookhuis, C. Dijksterhuis, B. Lewis-
Evans, and L. J. M. Mulder, “Short-term cardiovascular responses to
changing task demands,” International Journal of Psychophysiology,
vol. 85, no. 2, pp. 153–160, 2012.
[28] A.-M. Brouwer, M. A. Hogervorst, J. B. Van Erp, T. Heffelaar, P. H.
Zimmerman, and R. Oostenveld, “Estimating workload using EEG
spectral power and ERPs in the n-back task,” Journal of neural
engineering, vol. 9, no. 4, p. 045008, 2012.
[29] M. A. Hogervorst, A.-M. Brouwer, and J. B. van Erp, “Combining and
comparing EEG, peripheral physiology and eye-related measures for
the assessment of mental workload,” Frontiers in neuroscience, vol. 8,
2014.
[30] C. A. Russell and G. F. Wilson, “Feature saliency analysis for operator
state estimation,” in Proceedings of the 11th International Conference
on Human-Computer Interaction, vol. 11. Foundations of Augmented
Cognition, 2005.
[31] L. J. Trejo, N. J. McDonald, R. Matthews, and B. Z. Allison, “Experi-
mental design and testing of a multimodal cognitive overload classifier,”
Foundations of Augmented Cognition, pp. 13–22, 2007.
[32] B. Yin, F. Chen, N. Ruiz, and E. Ambikairajah, “Speech-based cognitive
load monitoring system,” in Acoustics, Speech and Signal Processing,
2008. ICASSP 2008. IEEE International Conference on. IEEE, 2008,
pp. 2041–2044.
[33] T. F. Yap, J. Epps, E. Ambikairajah, and E. H. C. C. E. Member), “For-
mant Frequencies under Cognitive Load: Effects and Classification,”
EURASIP Journal on Advances in Signal Processing, vol. 2011, no. 1,
p. 219253, Dec. 2011.
[34] “Finapres Medical Systems | Products - Finometer R PRO.” [Online].
Available: http://www.finapres.com/Products/Finometer-PRO

000356