See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/335795385

Image-based Motor Imagery EEG Classification using Convolutional Neural

Network

Conference Paper · May 2019

DOI: 10.1109/BHI.2019.8834598

CITATION READS

1 480

8 authors, including:

Tao Yang Kok Soon Phua

Institute for Infocomm Research Institute for Infocomm Research
66 PUBLICATIONS   526 CITATIONS    66 PUBLICATIONS   1,710 CITATIONS   

SEE PROFILE SEE PROFILE

Kai Keng Ang Rosa So

Institute for Infocomm Research Institute for Infocomm Research
186 PUBLICATIONS   5,261 CITATIONS    44 PUBLICATIONS   383 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Prognostic and Monitory EEG-Biomarkers for BCI Upper-limb Stroke Rehabilitation View project

Human Emotion Detection using EEG View project

All content following this page was uploaded by Tao Yang on 12 December 2019.

The user has requested enhancement of the downloaded file.

 Image-based Motor Imagery EEG Classification
using Convolutional Neural Network
Tao Yang Kok Soon Phua Juanhong Yu
Neural & Biomedical Technology Neural & Biomedical Technology Neural & Biomedical Technology
Institute for Infocomm Research Institute for Infocomm Research Institute for Infocomm Research
Singapore 138632 Singapore 138632 Singapore 138632
tyang@i2r.a-star.edu.sg ksphua@i2r.a-star.edu.sg jyu@i2r.a-star.edu.sg

Thevapriya Selvaratnam Valerie Toh Wai Hoe Ng

National Neuroscience Institute National Neuroscience Institute National Neuroscience Institute
Singapore 308433 Singapore 308433 Singapore 308433
thevapriya_selvaratnam@singheath.com. valerie_xl_toh@nni.com ng.wai.hoe@singhealth.com.sg
sg

Kai Keng Ang Rosa Q. So

Neural & Biomedical Technology Neural & Biomedical Technology
Institute for Infocomm Research Institue for Infocomm Research
Singapore 138632 Singapore 138632
kkang@i2r.a-star.edu.sg rosa-so@i2r.a-star.edu.sg

Abstract— Motor Imagery (MI) based Brain Computer Therefore, effective classification of electroencephalography
Interface (BCI) has clinical applications such as rehabilitation or (EEG) signals remains a challenging task.
communication for patients who have lost motor functions.
Accurate classification of motor-imagery based
electroencephalography (EEG) is important in developing such Recently, several groups have applied deep learning
BCI applications. We propose an image-based approach to design algorithms in the classification of EEG signals and have
a convolutional neural network (CNN) to classify EEG signals. In shown that such algorithms can outperform traditional
the proposed method, EEG signals were converted into images machine learning methods [6, 7]. One approach in fitting
based on the locations of scalp electrodes, such that spatial EEG data to a deep convolutional network is taking EEG data
correlation between neighbouring EEG channels was taken into in the form of an image [8, 9], i.e. “EEG-as-image”, each
consideration. We tested the proposed CNN architecture with both
2D and 3D kernels. EEG data were collected from a locked-in ALS
EEG channel is been taken as a pixel of an image. Such an
patient over 5 weeks, when the subject was instructed to perform image approach preserves the spatial features that may be
motor-imagery of right hand movement and idle state. Cross-day significant in classifying MI. In addition, temporal
decoding showed that CNN was able to achieve a 2-class (right vs. information is preserved when the entire EEG signal is
idle) classification accuracy of 68.38±7.29% and 65.94±8.52% represented as a series of images from consecutive time steps.
with 2D and 3D kernels respectively, compared to 55.09±5.74 % C. Tan et al. [8] combined EEG video and optical flow
for the traditionally used filter bank common spatial pattern method to extract multimodal information from EEG data for
(FBCSP) method. We also observed changes in the correlation classification. D. Zhang et al. [9] proposed a convolutional
coefficient of spectral entropy in the EEG data across weeks, and recurrent neural network model with cascade and parallel
these coefficients reveal frequency bands that are important for
decoding. In particular, the CNN architecture with Delta band
structure for the recognition of intention from EEG signal. In
included performed 6.96% higher than that excluding the Delta R. T. Schirrmeister et al.’s work [10], 44 channels of EEG
band. This study shows that the image-based CNN method data went through temporal convolution with 25 linear filters.
improves the classification performance, and the inclusion of The outputs of the temporal convolution were then formed as
Delta band improves classification performance for the current a 44 x 25 array for the subsequent convolution neural network
dataset. (CNN). This layer of 1D convolutional filter increased the
Keywords— Electroencephalography, MI-BCI, Convolutional EEG data from a vector of 44 elements to a 44 x 25 array, and
Neural Network, EEG-as-image. hence a large number of neurons are required to construct the
network.

I. INTRODUCTION Using a similar method to the above described approach, we

applied a CNN architecture to classify motor imagery EEG
Motor imagery (MI) based Brain Computer Interface data that is regarded as a series of images. The proposed
(BCI) has clinical applications in both rehabilitation and method takes into consideration of the spatial correlations
communication [1-3]. It can provide a channel of
among neighbouring EEG channels by accounting for the
communication for patients who have lost their motor function
arrangement of electrodes on the scalp.
due to injury or neurodegenerative disease such as
amyotrophic lateral sclerosis (ALS). However, studies have
shown that the classification accuracy of MI in completely EEG signal was traditionally divided in several bands, and
locked-in ALS patients does not differ from chance level [4]. signal in each band associates with certain brain activities.
K.K. Ang et al. [12] developed a Filter Bank common spatial
pattern (FBCSP) algorithm to extract features for 4 classes
This work is partially supported by BMRC-EDB JCO DP grants
IAF311022 from the Agency for Science, Technology and Research
motor imagery classification in BCI competition IV. In the
Singapore. work [12], band pass filters were applied to the data, and the
common spatial pattern (CSP) were extracted from each normal distribution [15] in Keras. Kernels in each CNN layer
frequency band. Often, Delta band in EEG signal (<4Hz) is are updated during the training, and become data specific
not considered in classification [13], as this band usually kernels upon completion of training. These kernels at each
represents information in lesions, deep sleep or motion layer extract different features from data presented to the
artefacts. However, we show that important differences CNN. Multiple layers of CNN are usually applied. The
between the motor imagery states are contained within Delta optimal feature could lie at an intermediate layer of the
band in our dataset, and investigate the impacts of the Delta network [16]. However, there is only one layer of CNN in the
band to classification accuracy in the proposed method. proposed architecture. Pre-process of training data become
necessary.
Raw EEG signals acquired in our experiment were first
II. METHOD filtered with 10 band pass filters (FIR). The band pass filter
A. Experimental Design was set from 0.5 Hz to 4 Hz, and further to 40 Hz with a step
Motor imagery experiments were conducted on an ALS size of 4 Hz. After band pass filtering, the data was then
patient (aged 29, female) over 5 consecutive weeks (1 day per converted into 6 x 5 x 10 array, where 10 is the number of
week). This subject has been diagnosed with ALS for the past frequency bands which corresponds to the RGB channels in
5 years, and she was in a complete locked-in state during the a coloured image. Training samples are then cropped from
data collection period. The experiments were conducted in each trail as illustrated in Figure 2.
compliance with the SingHealth IRB (CIRS 2016/3092).
Neuroscan amplifier and Quik-cap with 40 electrodes were
used for EEG data acquisition. All channels were referenced
to the A1 and A2 electrodes, which were placed at the left and
right ear lobes respectively. The impedance of each channel
was maintained below 10 kΩ throughout the experiments.
EEG data were acquired at 250 Hz and filtered by a band pass
filter of 0.5 to 40 Hz and a notch filter of 50 Hz in data
acquisition software (Scan 4.5). Figure 2. Cropping training samples using a sliding window of 500
frames on a 6 × 5 × N EEG data. Each training sample was down
sampled from 500 frames to 250 frames during training, i.e. 6 x 5 x
During each experimental trial, real-time feedback of the 250, where 250 is along the temporal dimension
motor imagery decoding results was shown to the subject on
a monitor. In each trial, the subject was instructed to either C. Convolutional Neural Network
move a cursor into a box on the right by performing right
A convolutional neural network with one convolutional
hand motor imagery (right trial), or maintain the cursor within layer was constructed using Keras, as shown in Figure 3. It was
a box by remaining in the idle state (idle trial). Decoding was used for the offline analysis of the acquired EEG data.
performed every 0.1s using 2 s of prior data. A total of 7 to
13 sessions were conducted on each day, with 5-10 trials in
each session.

Thirty channels of EEG signal were used in the analysis.

They were arranged in a 6 x 5 array as shown in Fig. 1. Data
from each trial was converted into a 6 × 5 × N array, where N
is the number of frames in the trial. In the offline analysis, a
sliding window with a width of 500 frames (2 s) and a step
size of 25 frames (0.1 s) was applied to each trial, as
illustrated in Figure 2, to form training samples for the
proposed CNN architecture.

Figure 1. 30 channels of EEG data were rearranged as a 6 × 5 array

according to its location on the scalp; Name of the EEG channels
follow the 10-20 system.

B. Preprocess of Data Figure 3. Convolutional neural network architecture with a 2D/ 3D

kernel for motor imagery classification.
Deep CNN architecture extracts features by training
kernels in its filters. These kernels are initialized by Glorot
 (a) (b) (c) ( d) ( e)
Figure 4. The correlation coefficient of spectral entropy between ‘right’ and ‘idle’ in 5 weeks. Chanel index from 1 to 30 corresponds to
F7, F3, Fz, F4, F8, Ft7, Fc3, Fcz, Fc4, Ft8, T7, C3, Cz, C4, T8, Tp7, Cp3, Cpz, Cp4, Tp8, P7, P3, Pz, P4, P8, O1, Oz, O2, Po1 and Po2.
Theta waves are found in week 1 to 4 [14]. Delta wave is found in week 1to 3. Week 5 has a very low contrast between ‘right’ and ‘idle’ in
terms of the correlation coefficient.

64 filters were used in the convolutional layer, with a 2D day tests are 68.38±7.29% and 65.94±8.52% for 2D and 3D
kernel size of 6 × 5 or a 3D kernel size of 6 × 5 × 5. The kernel respectively.
activation function for the convolutional layers was the Leaky
Rectified Linear Unit (LeakyReLU) with leak coefficient of Performance of the proposed CNN architecture was
0.05. A dropout rate of 0.5 was used to prevent overfitting. compared to the FBCSP algorithm (Fig. 5). The CNN
Mean pooling with a kernel of 75 and stride of 15 were architecture performed better than FBCSP in all 5 cross-day
applied to the filtered data from CNN to reduce the number tests, by 13.29% and 10. 85% on average for 2D and 3D
of features extracted. The Softmax function was applied to kernel respectively.
normalize the probability for classification.

Cross-day tests on the 5 week’s data were performed using

the proposed CNN architecture to understand the
performance of such CNN model in adapting signal variation
among experimental days. In the cross-day tests, EEG data
from one of the week were used to test on the CNN model
trained by data from the rest of 4 weeks, i.e. model used to
decode EEG data in Week 1 were trained using data from
Week 2 to 5.

III. RESULTS
A. EEG Variation over Experimental Period Figure 5. Cross-day validation of CNN. Black, yellow and blue bars
are accuracies obtained using 2D, 3D kernel and FBCSP
EEG signal varies across experimental sessions. Such respectively.
variations lead to inconsistencies in selected features among
experimental sessions which would affect the performance of
classifier. Performance of BCI applications often deteriorates
due to such non-stationarity. Spectral entropy is a measure of
signal disorganization, and its correlation coefficient of two
variables is an indicator of how close these two variables are
linearly related [17]. Figure 4 shows the correlation coefficient
of spectral entropy between ‘right’ and ‘idle’ trials for the data
collected over 5 weeks. High correlation coefficients were
present in data from all weeks except Week 5. The magnitude
of the correlation coefficient varies across the experimental
sessions. On average, the correlation coefficient in Week 4
and 5 are lower compared to the previous 3 weeks. Week 1, 2
and 3 were found with high correlation coefficient in Delta
band.
Figure 6. Classification accuracy of CNN model (2D Kernel) trained
with data including and excluding Delta band in cross-day tests.
B. Performance of CNN
Cross-day tests were performed using the CNN architecture
shown in Figure 3. The CNN was configured with 2D kernel, The correlation coefficients in Delta band were significant in
size of 6 × 5 × 1, and 3D kernel, size of 6 × 5 × 5 separately Weeks 1, 2 and 3 (see Figure 4 a, b and c). Cross-day tests
for the tests. Figure 5 shows the classification accuracies from were conducted using the EEG data including and excluding
the cross-day tests. The average accuracies over the 5 cross- Delta band. These tests were performed on the CNN
architecture with 2D Kernel. The average classification
accuracies of the 5 cross-day tests are 68.38±7.29% and [3] Lazarou I., Nikolopoulos S.,Petrantonakis P. C., Kompatsiaris I.,
Tsolaki M.Lazarou I., “EEG-Based Brain-Computer Interfaces for
61.42±5.90% for CNN trained with Delta band included and Communication and Rehabilitation of People with Motor Impairment:
excluded respectively, showing an improvement in decoding A Novel Approach of the 21 st Century”. Front Hum Neurosci.
accuracy when Delta band was included (Fig. 6). 2018;12:14.. doi:10.3389/fnhum.2018.00014.
[4] M. Marchetti and K. Priftis, “Brain-computer interfaces in amyotrophic
lateral sclerosis: A metanalysis,” Clin Neurophysiol. Vol. 126(6),
IV. DISCUSSIONS AND CONCLUSIONS p.1255-63, Jun. 2015.
[5] Wu W., Chen Z., Gao X., Li Y., Brown E. N., Gao S., "Probabilistic
In this study, we tested the performance of a CNN with both common spatial patterns for multichannel EEG analysis," IEEE
2D and 3D kernel in decoding EEG data and compared with transactions on pattern analysis and machine intelligence, vol. 37, no.
the traditionally used FBCSP method. The average 3, pp. 639-653, 2015
classification accuracies of both 2D and 3D kernel CNN [6] Y. Zhao et al., "On the improvement of classifying EEG recordings
using neural networks," 2017 IEEE International Conference on Big
outperforms FBCSP. No significant difference in Data (Big Data), Boston, MA, 2017, pp. 1709-1711.doi:
classification accuracy was found between 2D and 3D 10.1109/BigData.2017.8258112
kernels. The 3D kernel is an extension of the 2D kernel in [7] S. Sakhavi, C. Guan, and S. Yan, "Learning Temporal Information for
temporal dimension of training samples. Since the subject Brain-Computer Interface Using Convolutional Neural Networks,"
was performing motor imagery consistently during the trial IEEE Transactions on Neural Networks and Learning Systems, 2018.
period, it is not surprising that the 3D kernel would not [8] Tan C., Sun F., Zhang W., Chen J., Liu C., "Multimodal Classification
with Deep Convolutional-Recurrent Neural Networks for
capture additional distinctive features compared to the 2D Electroencephalography," in International Conference on Neural
kernel. Information Processing, 2017, pp. 767-776: Springer.
[9] Zhang D., Yao L., Zhang X.,Wang S., Chen W., Boots R., "EEG-based
The correlation coefficient of spectral entropy of the two Intention Recognition from Spatio-Temporal Representations via
classes of trials (‘right’ and ‘idle’) showed inconsistency in Cascade and Parallel Convolutional Recurrent Neural Networks,"
arXiv preprint arXiv:1708.06578, 2017.
distribution in frequency bands and magnitude of correlation
[10] R. T. Schirrmeister et al., "Deep learning with convolutional neural
coefficient over the experimental period. As shown in Figure networks for EEG decoding and visualization," Human brain mapping,
4, (a), (b) and (c), increased correlation coefficient appeared vol. 38, no. 11, pp. 5391-5420, 2017.
at a frequency below 4 Hz, i.e. Delta band, in data from [11] Yann LeCun, Yosshua Bengio, Geoffrey Hinton “Deep Learing,”
Weeks 1, 2 and 3. The impact of Delta band in decoding is Nature volume 521, pages 436–444 (28 May 2015)
illustrated in Figure 6. Classification accuracy improved [12] K. K. Ang, Z. Y. Chin, C. Wang, C. Guan, and H. Zhang, "Filter bank
common spatial pattern algorithm on BCI competition IV datasets 2a
significantly when Delta band was included in the model and 2b," Frontiers in neuroscience, vol. 6, p. 39, 2012.
training, especially for the first 3 weeks. This result shows [13] Elgendi M., Vialatte F., Cichocki A., Latchoumane C., Jeong J.,
that Delta band signals are important for motor imagery Dauwels J., "Optimization of EEG frequency bands for improved
classification in the current data set. diagnosis of Alzheimer disease, " Conf Proc IEEE Eng Med Biol Soc.
2011;2011:6087-91. doi: 10.1109/IEMBS.2011.6091504.
The magnitudes of correlation coefficients in general across [14] Rosa So Q et al., "Increased Theta Oscillations During Motor Imagery
in a Subject with Late-stage ALS, " Conf Proc IEEE Eng Med Biol Soc.
frequencies and channels are lower in Weeks 4 and 5 (Figure 2018 Jul;2018:1078-1081. doi: 10.1109/EMBC.2018.8512411.
4 d and e) compared to earlier weeks. Lower correlation [15] X. Glorot, Y. Bengio, "understanding the difficulty of training deep
coefficients indicate that the signals collected during these feedforward neural networks," Proceedings of the Thirteenth
weeks were less distinguishable between the motor imagery International Conference on Artificial Intelligence and Statistics,
PMLR 9:249-256, 2010.
tasks, and hence resulted in lower classification accuracy as
[16] Razavian A., Azizpour H., Sullivan J., Carlsson S., "CNN Features off-
shown in Figure 5 Week 4 and 5. This difference might be the-shelf: an Astounding Baseline for Recognition.” CoRR ,
due to lower engagement of the ALS subject during the BCI abs/1403.6382, 2014
sessions in the later weeks. [17] Yang, T., et al. "EEG Channel Selection Based on Correlation
Coefficient for Motor Imagery Classification: A Study on Healthy
In conclusion, an image-based CNN architecture was used for Subjects and ALS Patient." Conf Proc IEEE Eng Med Biol Soc 2018:
1996-1999.
classification of motor imagery EEG signals. We showed that
CNN performs significantly better than FBCSP method on
average using both the 2D and 3D kernels. However, no
significant difference in performance was found between 2D
and 3D kernel in the cross-day tests. As well, future work will
include using data collected over longer periods of time to
statistically validate the significance of these observations.

REFERENCES

[1] Ang K.K, et al., “A large clinical study on the ability of stroke patients
to use an EEG-based motor imagery brain-computer interface.” Clin
EEG Neurosci. 2011 Oct;42(4):253-8.
[2] Broetz D., Braun C., Weber C., Soekadar S. R.,Caria A., Birbaumer,
N.. “Combination of brain-computer interface training and goal-
directed physical therapy in chronic stroke: a case report.” Neurorehab
Neural Re 2010; 24(7): 674-6

View publication stats