short-paper

Speech Imagery BCI Training Using Game with a Purpose

Authors:

Abdulrahman Mohamed Selim,

Maurice Rekrut,

Daniel SonntagAuthors Info & Claims

AVI '24: Proceedings of the 2024 International Conference on Advanced Visual Interfaces

Article No.: 43, Pages 1 - 5

https://doi.org/10.1145/3656650.3656654

Published: 03 June 2024 Publication History

Abstract

Games are used in multiple fields of brain–computer interface (BCI) research and applications to improve participants’ engagement and enjoyment during electroencephalogram (EEG) data collection. However, despite potential benefits, no current studies have reported on implemented games for Speech Imagery BCI. Imagined speech is speech produced without audible sounds or active movement of the articulatory muscles. Collecting imagined speech EEG data is a time-consuming, mentally exhausting, and cumbersome process, which requires participants to read words off a computer screen and produce them as imagined speech. To improve this process for study participants, we implemented a maze-like game where a participant navigated a virtual robot capable of performing five actions that represented our words of interest while we recorded their EEG data. The study setup was evaluated with 15 participants. Based on their feedback, the game improved their engagement and enjoyment while resulting in a 69.10% average classification accuracy using a random forest classifier.

References

[1]

Teresa Bailey. 2014. Diagnosing and Treating Developmental Disorders with qEEG and Neurotherapy. In Clinical Neurotherapy. Elsevier, 321–355. https://doi.org/10.1016/B978-0-12-396988-0.00013-1

[2]

Michael Barz, Sebastian Kapp, Jochen Kuhn, and Daniel Sonntag. 2021. Automatic Recognition and Augmentation of Attended Objects in Real-Time Using Eye Tracking and a Head-Mounted Display. In ACM Symposium on Eye Tracking Research and Applications(ETRA ’21 Adjunct). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3450341.3458766 event-place: Virtual Event, Germany.

Digital Library

[3]

Michael Barz and Daniel Sonntag. 2021. Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze. Sensors 21, 12 (June 2021), 4143. https://doi.org/10.3390/s21124143

[4]

Etienne Combrisson and Karim Jerbi. 2015. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. Journal of Neuroscience Methods 250 (July 2015), 126–136. https://doi.org/10.1016/j.jneumeth.2015.01.010

[5]

Martí de Castro-Cros, Marc Sebastian-Romagosa, Javier Rodríguez-Serrano, Eloy Opisso, Manel Ochoa, Rupert Ortner, Christoph Guger, and Dani Tost. 2020. Effects of Gamification in BCI Functional Rehabilitation. Frontiers in Neuroscience 14 (Aug. 2020), 882. https://doi.org/10.3389/fnins.2020.00882

[6]

Jesús S. García-Salinas, Luis Villaseñor-Pineda, Carlos A. Reyes-García, and Alejandro A. Torres-García. 2019. Transfer learning in imagined speech EEG-based BCIs. Biomedical Signal Processing and Control 50 (April 2019), 151–157. https://doi.org/10.1016/j.bspc.2019.01.006

[7]

Bin He. 2018. Introduction. In Neuromodulation. Elsevier, 339. https://doi.org/10.1016/B978-0-12-805353-9.02005-2

[8]

Z.J. Koles. 1991. The quantitative extraction and topographic mapping of the abnormal components in the clinical EEG. Electroencephalography and Clinical Neurophysiology 79, 6 (Dec. 1991), 440–447. https://doi.org/10.1016/0013-4694(91)90163-X

[9]

Patrick Ledwidge, Jeremy Foust, and Adam Ramsey. 2018. Recommendations for Developing an EEG Laboratory at a Primarily Undergraduate Institution. Journal of undergraduate neuroscience education: JUNE: a publication of FUN, Faculty for Undergraduate Neuroscience 17, 1 (2018), A10–A19.

[10]

Seo-Hyun Lee, Minji Lee, and Seong-Whan Lee. 2020. EEG Representations of Spatial and Temporal Features in Imagined Speech and Overt Speech. In Pattern Recognition, Shivakumara Palaiahnakote, Gabriella Sanniti di Baja, Liang Wang, and Wei Qi Yan (Eds.). Vol. 12047. Springer International Publishing, Cham, 387–400. https://doi.org/10.1007/978-3-030-41299-9_30 Series Title: Lecture Notes in Computer Science.

Digital Library

[11]

Ren Li, Jared S. Johansen, Hamad Ahmed, Thomas V. Ilyevsky, Ronnie B Wilbur, Hari M Bharadwaj, and Jeffrey Mark Siskind. 2018. Training on the test set? An analysis of Spampinato et al. [31]. (2018). https://doi.org/10.48550/ARXIV.1812.07697 Publisher: arXiv Version Number: 1.

[12]

Fabien Lotte, Florian Larrue, and Christian Mühl. 2013. Flaws in current human training protocols for spontaneous Brain-Computer Interfaces: lessons learned from instructional design. Frontiers in Human Neuroscience 7 (2013). https://doi.org/10.3389/fnhum.2013.00568

[13]

Stéphanie Martin, Peter Brunner, Chris Holdgraf, Hans-Jochen Heinze, Nathan E. Crone, Jochem Rieger, Gerwin Schalk, Robert T. Knight, and Brian N. Pasley. 2014. Decoding spectrotemporal features of overt and covert speech from the human cortex. Frontiers in Neuroengineering 7 (2014). https://www.frontiersin.org/articles/10.3389/fneng.2014.00014

[14]

Kusuma Mohanchandra and Snehanshu Saha. 2016. A Communication Paradigm Using Subvocalized Speech: Translating Brain Signals into Speech. Augmented Human Research 1, 1 (Dec. 2016), 3. https://doi.org/10.1007/s41133-016-0001-z

[15]

M.R. Nuwer and P. Coutin-Churchman. 2014. Brain Mapping and Quantitative Electroencephalogram. In Encyclopedia of the Neurological Sciences. Elsevier, 499–504. https://doi.org/10.1016/B978-0-12-385157-4.00519-4

[16]

Dipti Pawar and Sudhir Dhage. 2020. Multiclass covert speech classification using extreme learning machine. Biomedical Engineering Letters 10, 2 (May 2020), 217–226. https://doi.org/10.1007/s13534-020-00152-x

[17]

Dipti Pawar and Sudhir Dhage. 2023. EEG-based covert speech decoding using random rotation extreme learning machine ensemble for intuitive BCI communication. Biomedical Signal Processing and Control 80 (Feb. 2023), 104379. https://doi.org/10.1016/j.bspc.2022.104379

[18]

Muhammad Naveed Iqbal Qureshi, Beomjun Min, Hyeong-jun Park, Dongrae Cho, Woosu Choi, and Boreom Lee. 2018. Multiclass Classification of Word Imagination Speech With Hybrid Connectivity Features. IEEE Transactions on Biomedical Engineering 65, 10 (Oct. 2018), 2168–2177. https://doi.org/10.1109/TBME.2017.2786251 Conference Name: IEEE Transactions on Biomedical Engineering.

[19]

Maurice Rekrut, Abdulrahman Mohamed Selim, and Antonio Krüger. 2022. Improving Silent Speech BCI Training Procedures Through Transfer from Overt to Silent Speech. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2650–2656. https://doi.org/10.1109/SMC53654.2022.9945447 ISSN: 2577-1655.

[20]

Maurice Rekrut, Mansi Sharma, Matthias Schmitt, Jan Alexandersson, and Antonio Kruger. 2021. Decoding Semantic Categories from EEG Activity in Silent Speech Imagination Tasks. In 2021 9th International Winter Conference on Brain-Computer Interface (BCI). IEEE, Gangwon, Korea (South), 1–7. https://doi.org/10.1109/BCI51272.2021.9385357

[21]

Joe T. Rexwinkle, Gregory Lieberman, Matthew Jaswa, and Brent J. Lance. 2019. Development of a Game with a Purpose for Acquisition of Brain-Computer Interface Data. (2019). https://doi.org/10.48550/ARXIV.1910.00106 Publisher: arXiv Version Number: 1.

[22]

Alborz Rezazadeh Sereshkeh, Robert Trott, Aurélien Bricout, and Tom Chau. 2017. Online EEG Classification of Covert Speech for Brain–Computer Interfacing. International Journal of Neural Systems 27, 08 (Dec. 2017), 1750033. https://doi.org/10.1142/S0129065717500332

[23]

L. von Ahn. 2006. Games with a purpose. Computer 39, 6 (June 2006), 92–94. https://doi.org/10.1109/MC.2006.196 Conference Name: Computer.

Digital Library

[24]

Hiroki Watanabe, Hiroki Tanaka, Sakriani Sakti, and Satoshi Nakamura. 2020. Synchronization between overt speech envelope and EEG oscillations during imagined speech. Neuroscience Research 153 (April 2020), 48–55. https://doi.org/10.1016/j.neures.2019.04.004

[25]

Xiaolei Xia and Li Hu. 2019. EEG: Neural Basis and Measurement. In EEG Signal Processing and Feature Extraction, Li Hu and Zhiguo Zhang (Eds.). Springer Singapore, Singapore, 7–21. https://doi.org/10.1007/978-981-13-9113-2_2

[26]

Filip Škola, Simona Tinková, and Fotis Liarokapis. 2019. Progressive Training for Motor Imagery Brain-Computer Interfaces Using Gamification and Virtual Reality Embodiment. Frontiers in Human Neuroscience 13 (Sept. 2019), 329. https://doi.org/10.3389/fnhum.2019.00329

Cited By

Nihei AHirano RWatanabe K(2024)Intuitive Brain-Computer Interface Control Using Onomatopoeia for an Enhanced Gaming ExperienceCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678789(208-214)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678789

Index Terms

Speech Imagery BCI Training Using Game with a Purpose
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
    2. Interaction devices

Recommendations

Development of a neuro-feedback game based on motor imagery EEG

Electroencephalogram (EEG) has widely been used to monitor subjects/patients' mental states. Using the monitor results as feedback, neuro-feedback enables patients to learn to regulate their physiological and psychological states so that improvements in ...
Augmented control of an avatar using an SSVEP based BCI
AH '12: Proceedings of the 3rd Augmented Human International Conference

The demonstration shows the usage of an EEG-based brain-computer interface (BCI) for the real-time control of an avatar in World of Warcraft. Visitors can test the installation during the conference after about 5 minutes of training time. World of ...
Endless Runner Game in Virtual Reality Controlled by a Self-paced Brain-Computer Interface Based on EEG and Motor Imagery
Extended Reality
Abstract
A prototype system for an endless runner game controlled by a self-paced brain-computer interface (BCI) based on electroencephalography (EEG) and motor imagery (MI) is presented. In self-paced BCI systems, brain activity can be distinguished ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AVI '24: Proceedings of the 2024 International Conference on Advanced Visual Interfaces

June 2024

578 pages

ISBN:9798400717642

DOI:10.1145/3656650

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

AVI 2024

AVI 2024: International Conference on Advanced Visual Interfaces 2024

June 3 - 7, 2024

Arenzano, Genoa, Italy

Acceptance Rates

AVI '24 Paper Acceptance Rate 21 of 82 submissions, 26%;

Overall Acceptance Rate 128 of 490 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
43
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)4

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nihei AHirano RWatanabe K(2024)Intuitive Brain-Computer Interface Control Using Onomatopoeia for an Enhanced Gaming ExperienceCompanion Proceedings of the 2024 Annual Symposium on Computer-Human Interaction in Play10.1145/3665463.3678789(208-214)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3665463.3678789

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten