Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13188))

Included in the following conference series:

  • 1173 Accesses

Abstract

Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (±15.39) for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.

This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00185, Development of Intelligent Pattern Recognition Softwares for Ambulatory Brain Computer Interface, No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University)).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Pfurtscheller, G., Neuper, C.: Motor imagery and direct brain-computer communication. Proc. IEEE 89(7), 1123–1134 (2001)

    Article  Google Scholar 

  2. Chen, Y., Atnafu, A.D., Schlattner, I., Weldtsadik, W.T., Roh, M.C., Kim, H.J., Lee, S.W., Blankertz, B., Fazli, S.: A high-security EEG-based login system with RSVP stimuli and dry electrodes. IEEE Trans. Inf. Forensics Secur. 11(12), 2635–2647 (2016)

    Article  Google Scholar 

  3. Won, D.O., Hwang, H.J., Kim, D.M., Müller, K.R., Lee, S.W.: Motion-based rapid serial visual presentation for gaze-independent brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 26(2), 334–343 (2018)

    Article  Google Scholar 

  4. Lee, M.H., Williamson, J., Won, D.O., Fazli, S., Lee, S.W.: A high performance spelling system based on EEG-EOG signals with visual feedback. IEEE Trans. Neural Syst. Rehabil. Eng. 26(7), 1443–1459 (2018)

    Article  Google Scholar 

  5. Lee, M.H., et al.: EEG dataset and OpenBMI toolbox for three BCI paradigms: an investigation into BCI illiteracy. GigaScience 8(5), giz002 (2019)

    Google Scholar 

  6. Yuan, H., He, B.: Brain-computer interfaces using sensorimotor rhythms: current state and future perspectives. IEEE Trans. Biomed. Eng. 61(5), 1425–1435 (2014)

    Article  Google Scholar 

  7. Pfurtscheller, G., Brunner, C., Schlögl, A., da Silva, F.L.: Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks. Neuroimage 31(1), 153–159 (2006)

    Article  Google Scholar 

  8. Suk, H.I., Lee, S.W.: Subject and class specific frequency bands selection for multiclass motor imagery classification. Int. J. Imaging Syst. Technol. 21(2), 123–130 (2011)

    Article  Google Scholar 

  9. Ramoser, H., Muller-Gerking, J., Pfurtscheller, G.: Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng. 8(4), 441–446 (2000)

    Article  Google Scholar 

  10. Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., Müller, K.R.: Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process. Mag. 25(1), 41–56 (2008)

    Article  Google Scholar 

  11. Lemm, S., Blankertz, B., Curio, G., Müller, K.R.: Spatio-spectral filters for improving the classification of single trial EEG. IEEE Trans. Biomed. Eng. 52(9), 1541–1548 (2005)

    Article  Google Scholar 

  12. Krauledat, M., Tangermann, M., Blankertz, B., Müller, K.R.: Towards zero training for brain-computer interfacing. PLoS ONE 3(8), e2967 (2008)

    Article  Google Scholar 

  13. Fazli, S., Popescu, F., Danóczy, M., Blankertz, B., Müller, K.R., Grozea, C.: Subject-independent mental state classification in single trials. Neural Netw. 22(9), 1305–1312 (2009)

    Article  Google Scholar 

  14. Kwon, O.Y., Lee, M.H., Guan, C., Lee, S.W.: Subject-independent brain-computer interfaces based on deep convolutional neural networks. IEEE Trans. Neural Networks Learn. Syst. 31(10), 3839–3852 (2019)

    Article  Google Scholar 

  15. Ang, K.K., Chin, Z.Y., Zhang, H., Guan, C.: Mutual information-based selection of optimal spatial-temporal patterns for single-trial EEG-based BCIs. Pattern Recogn. 45(6), 2137–2144 (2012)

    Article  Google Scholar 

  16. Cecotti, H., Graser, A.: Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 433–445 (2011)

    Article  Google Scholar 

  17. Manor, R., Geva, A.B.: Convolutional neural network for multi-category rapid serial visual presentation BCI. Front. Comput. Neurosci. 9, 146 (2015)

    Article  Google Scholar 

  18. Sturm, I., Lapuschkin, S., Samek, W., Müller, K.R.: Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods 274, 141–145 (2016)

    Article  Google Scholar 

  19. Stober, S., Cameron, D.J., Grahn, J.A.: Using convolutional neural networks to recognize rhythm stimuli from electroencephalography recordings. In: Advances in Neural Information Processing Systems, pp. 1449–1457 (2014)

    Google Scholar 

  20. Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks. arXiv preprint arXiv:1511.06448 (2015)

  21. Sakhavi, S., Guan, C., Yan, S.: Learning temporal information for brain-computer interface using convolutional neural networks. IEEE Trans. Neural Networks Learn. Syst. 29(11), 5619–5629 (2018)

    Article  MathSciNet  Google Scholar 

  22. Bang, J.S., Lee, M.H., Fazli, S., Guan, C., Lee, S.W.: Spatio-spectral feature representation for motor imagery classification using convolutional neural networks. IEEE Trans. Neural Networks Learn. Syst. (2021)

    Google Scholar 

  23. LeCun, Y., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, pp. 396–404 (1990)

    Google Scholar 

  24. Barachant, A., Bonnet, S., Congedo, M., Jutten, C.: Multiclass brain-computer interface classification by Riemannian geometry. IEEE Trans. Biomed. Eng. 59(4), 920–928 (2011)

    Article  Google Scholar 

  25. Shi, X., Wang, T., Wang, L., Liu, H., Yan, N.: Hybrid convolutional recurrent neural networks outperform CNN and RNN in task-state EEG detection for Parkinson’s disease. In: 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp. 939–944. IEEE (2019)

    Google Scholar 

  26. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, pp. 807–814 (2010)

    Google Scholar 

  27. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  28. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  29. Tangermann, M., et al.: Review of the BCI competition IV. Front. Neurosci. 6, 55 (2012)

    Article  Google Scholar 

  30. Coyle, D., Satti, A., Prasad, G., McGinnity, T.M.: Neural time-series prediction preprocessing meets common spatial patterns in a brain-computer interface. In: 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2626–2629. IEEE (2008)

    Google Scholar 

  31. Coyle, D.: Neural network based auto association and time-series prediction for biosignal processing in brain-computer interfaces. IEEE Comput. Intell. Magaz. 4(4) (2009)

    Google Scholar 

  32. Lotte, F., Guan, C.: Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Trans. Biomed. Eng. 58(2), 355–362 (2011)

    Article  Google Scholar 

  33. Ang, K.K., Chin, Z.Y., Wang, C., Guan, C., Zhang, H.: Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front. Neurosci. 6, 39 (2012)

    Article  Google Scholar 

  34. Kim, S.H., Yang, H.J., Nguyen, N.A.T., Lee, S.W.: AsEmo: automatic approach for EEG-based multiple emotional state identification. IEEE J. Biomed. Health Inform. 25(5), 1508–1518 (2020)

    Article  Google Scholar 

  35. Lee, S.H., Lee, M., Lee, S.W.: Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication. IEEE Trans. Neural Syst. Rehabil. Eng. 28(12), 2647–2659 (2020)

    Article  Google Scholar 

  36. Lee, D.Y., Lee, M., Lee, S.W.: Decoding imagined speech based on deep metric learning for intuitive BCI communication. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1363–1374 (2021)

    Article  Google Scholar 

  37. Jeong, J.H., Shim, K.H., Kim, D.J., Lee, S.W.: Brain-controlled robotic arm system based on multi-directional CNN-BiLSTM network using EEG signals. IEEE Trans. Neural Syst. Rehabil. Eng. 28(5), 1226–1238 (2020)

    Article  Google Scholar 

  38. Kim, K., et al.: Development of a human-display interface with vibrotactile feedback for real-world assistive applications. Sensors 21(2), 592 (2021)

    Article  Google Scholar 

  39. Zhang, Y., Zhang, H., Chen, X., Lee, S.W., Shen, D.: Hybrid high-order functional connectivity networks using resting-state functional MRI for mild cognitive impairment diagnosis. Sci. Rep. 7(1), 1–15 (2017)

    Article  Google Scholar 

  40. Zhang, Y., et al.: Strength and similarity guided group-level brain functional network construction for MCI diagnosis. Pattern Recogn. 88, 421–430 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea

    Ji-Seon Bang

  2. Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea

    Seong-Whan Lee

Authors
  1. Ji-Seon Bang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seong-Whan Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seong-Whan Lee .

Editor information

Editors and Affiliations

  1. Korea University, Seoul, Korea (Republic of)

    Christian Wallraven

  2. Nanjing University, Nanjing, China

    Qingshan Liu

  3. Osaka University, Osaka, Japan

    Hajime Nagahara

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bang, JS., Lee, SW. (2022). Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13188. Springer, Cham. https://doi.org/10.1007/978-3-031-02375-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-02375-0_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-02374-3

  • Online ISBN: 978-3-031-02375-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation