Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Adding Human Learning in Brain--Computer Interfaces (BCIs): Towards a Practical Control Modality

Published: 27 May 2015 Publication History

Abstract

In this article, we introduce CLBCI (Co-Learning for Brain--Computer Interfaces), a BCI architecture based on co-learning in which users can give explicit feedback to the system rather than just receiving feedback. CLBCI is based on minimum distance classification with Independent Component Analysis (ICA) and allows for shorter training times compared to classical BCIs, as well as faster learning in users and a good performance progression. We further propose a new scheme for real-time two-dimensional visualization of classification outcomes using Wachspress coordinate interpolation. It allows us to represent classification outcomes for n classes in simple regular polygons. Our objective is to devise a BCI system that constitutes a practical interaction modality that can be deployed rapidly and used on a regular basis. We apply our system to an event-based control task in the form of a simple shooter game in which we evaluate the learning effect induced by our architecture compared to a classical approach. We also evaluate how much user feedback and our visualization method contribute to the performance of the system.

Supplementary Material

MP4 File (a12-kosmyna.mp4)

References

[1]
D. Afergan, E. M. Peck, E. T. Solovey, A. Jenkins, S. W. Hincks, E. T. Brown, R. Chang, and R. J. K. Jacob. 2014. Dynamic difficulty using brain metrics of workload. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 3797--3806.
[2]
B. Alsallakh, A. Hanbury, H. Hauser, S. Miksch, and A. Rauber. 2014. Visual methods for analyzing probabilistic classification data. IEEE Transactions on Visualization and Computer Graphics, 20, 12.
[3]
S. Amiri, A. Rabbi, L. Azinfar, and R. Fazel-Rezai. 2013. A Review of P300, SSVEP, and Hybrid P300/SSVEP Brain Computer Interface Systems. In R. Fazel-Rezai (Ed.). Brain-Computer Interface Systems—Recent Progress and Future Prospects, InTech, Rijeka, Croatia.
[4]
A. Barachant, S. Bonnet, M. Congedo, and C. Jutten. 2012. Multiclass brain-computer interface classification by Riemannian geometry. IEEE Transactions on Biomedical Engineering 59, 4, 920--928.
[5]
A. Barbero and M. Grosse-Wentrup. 2010. Biased feedback in brain-computer interfaces. Journal of Neuroengineering and Rehabilitation 7, 1, 34.
[6]
B. Blankertz, F. Losch, M. Krauledat, G. Dornhege, G. Curio, and K.-R. Müller. 2008. The Berlin brain--computer interface: Accurate performance from first-session in BCI-naïve subjects. IEEE Transactions on Biomedical Engineering 55, 10, 2452--62.
[7]
B. Blankertz and C. Schäfer. 2002. Single trial detection of EEG error potentials: A tool for increasing BCI transmission rates. In Artificial Neural Networks—ICANN 2002. Springer, Berlin, 1137--1143.
[8]
W. R. Boot, A. F. Kramer, D. J. Simons, M. Fabiani, and G. Gratton. 2008. The effects of video game playing on attention, memory, and executive control. Acta Psychologica 129, 3, 387--398.
[9]
D. P. Bos, M. Obbink, A. Nijholt, G. Hakvoort, and M. Christian. 2010. Towards multiplayer BCI games. In BioS-Play: Workshop on Multiuser and Social Biosignal Adaptive Games and Playful Applications. 1--4.
[10]
C. Cajochen, K. Kräuchi, M.-A. von Arx, D. Möri, P. Graw, and A. Wirz-Justice. 1996. Daytime melatonin administration enhances sleepiness and theta/alpha activity in the waking EEG. Neuroscience Letters 207, 3, 209--213.
[11]
H. Cecotti, I. Volosyak, and A. Gräser. 2010. Reliable visual stimuli on LCD screens for SSVEP based BCI. In 18th European Signal Processing Conference (EUSIPCO’10). 919--923.
[12]
G. Chanel, J. Kronegg, D. Grandjean, and T. Pun. 2006. Emotion assessment: Arousal evaluation using EEGs and peripheral physiological signals. In B. A. Gunsel Jain, T. A. Murat, and B. Sankur (Eds). Multimedia Content Representation, Classification and Security. Springer, Berlin, 530--537.
[13]
M. Congedo, M. Goyat, N. Tarrin, G. Ionescu, L. Varnet, B. Rivet, R. Phlypo, N. Jrad, M. Acquadro, and C. Jutten. 2011. “Brain Invaders”: A prototype of an open-source P300-based video game working with the OpenViBE platform. In 5th International BCI Conference. 1--6.
[14]
J. B. F. Van Erp, F. Lotte, and M. Tangermann. 2012. Brain-computer interfaces: Beyond medical applications. Computer, 26--34.
[15]
J. A. Fails and D. R. Olsen Jr. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI’03). ACM, New York, NY, 39--45.
[16]
S. Fairclough. 2011. Physiological computing: Interfacing with the human nervous system. In Sensing Emotions. 1--20.
[17]
M. Fatourechi, A. Bashashati, R. K. Ward, and G. E. Birch. 2007. EMG and EOG artifacts in brain computer interface systems: A survey. Clinical Neurophysiology 118, 3, 480--494.
[18]
R. Fiebrink, P. R. Cook, and D. Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 147--156.
[19]
E. Frank and M. Hall. 2003. Visualizing class probability estimators. In N. Lavra&##x010D;, D. Gamberger, L. Todorovski, and H. Blockeel (Eds). Knowledge Discovery in Databases: PKDD 2003 SE—17. Lecture Notes in Computer Science. Springer Berlin, 168--179.
[20]
J. P. Gee. 2003. What Video Games Have to Teach Us About Learning and Literacy. Comput. Entertain., 1, 1, 20.
[21]
D. Grimes, D. S. Tan, S. E. Hudson, P. Shenoy, and R. P. N. Rao. 2008. Feasibility and pragmatics of classifying working memory load with an Electroencephalograph. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 835--844.
[22]
J. Gruzelier, T. Egner, and D. Vernon. 2006. Validating the efficacy of neurofeedback for optimising performance. Progress in Brain Research 159, 421--31.
[23]
C. Guger, C. Holzner, and C. Groenegress. 2009. Brain computer interface for virtual reality control. In Proceedings of ESANN 2009. 443--448.
[24]
H. Gürkök and A. Nijholt. 2012. Brain--computer interfaces for multimodal interaction: A survey and principles. International Journal of Human-Computer Interaction, 28, 5, 292--307.
[25]
C. R. Hema, M. P. Paulraj, S. Yaacob, A. H. Adom, and R. Nagarajan. 2009. Single trial motor imagery classification for a four state brain machine interface. In 5th International Colloquium on Signal Processing—Its Applications, 2009 (CSPA’09). 39--41.
[26]
L. R. Hochberg, M. D. Serruya, G. M. Friehs, J. A. Mukand, M. Saleh, A. H. Caplan, A. Branner, D. Chen, R. D. Penn, J. and P. Donoghue. 2006. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 7099, 164--71.
[27]
A. Hyvärinen and E. Oja. 1997. A fast fixed-point algorithm for independent component analysis. Neural Computation, 9, 1483--1492.
[28]
Z. Ìscan, Ö. Özkaya, and Z. Dokur. 2011. Classification of EEG in a steady state visual evoked potential based brain computer interface experiment. In Proceedings of the 10th International Conference on Adaptive and Natural Computing Algorithms -- Volume Part II (ICANNGA’11). Springer-Verlag, Berlin, 81--88.
[29]
I. Iturrate, R. Chavarriaga, L. Montesano, J. Minguez, and J. D. R. Millan. 2012. Latency correction of error potentials between different experiments reduces calibration time for single-trial classification. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference. 3288--3291.
[30]
I. Iturrate, L. Montesano, and J. Minguez. 2010. Single trial recognition of error-related potentials during observation of robot operation. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 4181--4184.
[31]
T. Iwata, K. Saito, N. Ueda, S. Stromsten, T. L. Griffiths, and J. B. Tenenbaum. 2005. Parametric embedding for class visualization. In L. K. Saul, Y. Weiss, and L. Bottou (Eds.). Advances in Neural Information Processing Systems 17. MIT Press, Cambridge, MA, 617--624.
[32]
V. Kaiser, G. Bauernfeind, T. Kaufmann, and A. Kreilinger. 2011. Cortical effects of user learning in a motor-imagery BCI training. International Journal of Bioelectromagnetism 13, 2, 60--61.
[33]
C. Kapeller, C. Hintermüller, and C. Guger. 2012. Augmented control of an avatar using an SSVEP based BCI. In Proceedings of the 3rd Augmented Human International Conference (AH’12). ACM, New York, NY, 27:1--27:2.
[34]
A. Kapoor, B. Lee, D. Tan, and E. Horvitz. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1343.
[35]
S. P. Kelly, E. C. Lalor, R. B. Reilly, S. Member, and J. J. Foxe. 2005. Visual spatial attention tracking using high-density SSVEP data for independent brain--computer communication. IEEE Transactions on Neural Systems and Rehabilitation Engineering 13, 2, 172--178.
[36]
N. Kosmyna, F. Tarpin-Bernard, and B. Rivet. 2013. Towards a general architecture for a co-learning of brain computer interfaces. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 1054--1057.
[37]
N. Kosmyna and F. Tarpin-Bernard. 2013. Evaluation and comparison of a multimodal combination of BCI paradigms and eye tracking with affordable consumer-grade hardware in a gaming context. IEEE Transactions on Computational Intelligence and AI in Games, 5, 2, 150--154.
[38]
R. Kus, D. Valbuena, J. Zygierewicz, T. Malechka, A. Graeser, and P. Durka. 2012. Asynchronous BCI based on motor imagery with automated calibration and neurofeedback training. IEEE Transactions on Neural Systems and Rehabilitation Engineering 20, 6, 823--835.
[39]
A. Lécuyer, F. Lotte, R. Reilly, and R. Leeb. 2008. Brain-computer interfaces, virtual reality, and videogames. Computer 42, 10, 66--72.
[40]
J. Legény, R. Viciana Abad, and A. Lécuyer. 2013. Towards contextual SSVEP-based BCI controller: Smart activation of stimuli and controls weighting. IEEE Transactions on Computational Intelligence and AI in Games 5, 2, 111--116.
[41]
P. Lidberg. 2011. Barycentric and Wachspress coordinates in two dimensions: Theory and implementation for shape transformations. Uppsala University, UUDM Project Report.
[42]
A. Llera, V. Gomez, and H. J. Kappen. 2012. Adaptive classification on brain-computer interfaces using reinforcement signals. Neural Computation 24, 11, 2900--2923.
[43]
F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, and B. Arnaldi. 2007. A review of classification algorithms for EEG-based brain-computer interfaces. Journal of Neural Engineering 4, 2, R1--R13.
[44]
F. Lotte, F. Larrue, and C. Mühl. 2013. Flaws in current human training protocols for spontaneous brain-computer interfaces: Lessons learned from instructional design. Frontiers in Human Neuroscience 7 (September), 568.
[45]
M. A. Migut, M. Worring, and C. J. Veenman. 2013. Visualizing multi-dimensional decision boundaries in 2D. Data Mining and Knowledge Discovery, 1--23.
[46]
J. D. R. Millán and J. Mouriño. 2003. Asynchronous BCI and local neural classifiers: An overview of the Adaptive Brain Interface project. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11, 2, 159--161.
[47]
J. Müller-Gerking, G. Pfurtscheller, and H. Flyvbjerg. 1999. Designing optimal spatial filters for single-trial EEG classification in a movement task. Clinical Neurophysiology 110, 787--798.
[48]
G. Müller-Putz and R. Scherer. 2008. Better than random? A closer look on BCI results. International Journal of Bioelectromagnetism 10, 1, 52--55.
[49]
C. Neuper, G. R. Müller-Putz, R. Scherer, and G. Pfurtscheller. 2006. Motor imagery and EEG-based control of spelling devices and neuroprostheses. In C. Neuper and W. Klimesch (Eds.). Event-Related Dynamics of Brain Oscillations. Progress in Brain Research. Elsevier, 393--409.
[50]
L. F. Nicolas-Alonso and J. Gomez-Gil. 2012. Brain computer interfaces, a review. Sensors (Basel, Switzerland), 12, 2, 1211--79.
[51]
K. Nigam and R. Ghani. 2000. Analyzing the effectiveness and applicability of co-training. In Proceedings of the 9th International Conference on Information and Knowledge Management (CIKM’00). ACM, New York, NY, 86--93.
[52]
A. Nijholt, B. Reuderink, and D. Oude Bos. 2009a. Turning shortcomings into challenges: brain-computer interfaces for games. In A. Nijholt, D. Reidsma, and H. Hondorp (Eds.). Intelligent Technologies for Interactive Entertainment SE—15. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. Springer, Berlin, 153--168.
[53]
A. Nijholt, B. Reuderink, and D. Oude Bos. 2009b. Turning shortcomings into challenges: brain-computer interfaces for games. A. Nijholt, D. Reidsma, and H. Hondorp (Eds.). Intelligent Technologies for Interactive Entertainment, 1, 153--168.
[54]
A. Nooh, J. Yunus, and S. Daud. 2011. A review of asynchronous electroencephalogram-based brain computer interface systems. International Conference on Biomedical Engineering and Technology 11, 55--59.
[55]
S. Noorzadeh, B. Rivet, and C. Jutten. 2014. 2D for brain-computer interfaces: Two 3D extensions of the P300-speller. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’14). 2--7.
[56]
G. Pfurtscheller and F. H. Lopes da Silva. 1999. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology, 110, 11, 1842--1857.
[57]
M. F. Pike, H. A. Maior, M. Porcheron, S. C. Sharples, and M. L. Wilson. 2014. Measuring the effect of think aloud protocols on workload using fNIRS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 3807--3816.
[58]
A. Royer, A. Doud, M. Rose, and B. He. 2010. EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18, 6, 581--589.
[59]
R. Scherer, F. Lee, A. Schlogl, R. Leeb, H. Bischof, and G. Pfurtscheller. 2008. Toward self-paced brain--computer communication: navigation through virtual worlds. IEEE Transactions on Biomedical Engineering 55, 2, 675--682.
[60]
E. Schmidt, W. Kincses, and M. Schrauf. 2007. Assessing driver's vigilance state during monotonous driving. In 4th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. 138--145.
[61]
C. Seifert and E. Lex. 2009. A novel visualization approach for data-mining-related classification. In 2009 13th International Conference on Information Visualisation. 490--495.
[62]
V. J. Shute. 2008. Focus on Formative Feedback. Review of Educational Research 78, 1, 153--189.
[63]
Y. Sung, K. Cho, and K. Um. 2012. A development architecture for serious games using BCI (brain computer interface) sensors. Sensors (Basel, Switzerland) 12, 11, 15671--88.
[64]
J. Talbot, B. Lee, A. Kapoor, and D. Tan. 2009. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). 1283--1292.
[65]
D. S. Tan and A. Nijholt (Eds.). 2010. Brain-Computer Interfaces—Applying our Minds to Human-Computer Interaction. Springer, New York, NY.
[66]
E. Thomas, M. Dyson, and M. Clerc. 2013. An analysis of performance evaluation for motor-imagery based BCI. Journal of Neural Engineering, 10, 3, 031001.
[67]
C. Vi and S. Subramanian. 2012. Detecting error-related negativity for interaction design. In SIGCHI Conference on Human Factors in Computing Systems (CHI’12), 493.
[68]
C. T. Vi, I. Jamil, D. Coyle, and S. Subramanian. 2014. Error related negativity in observing interactive tasks. In SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 3787--3796.
[69]
C. Vidaurre, C. Sannelli, K.-R. Müller, and B. Blankertz. 2010. Machine-learning based co-adaptive calibration: A perspective to fight BCI illiteracy. In M. Graña Romay, E. Corchado, and M. T. Garcia Sebastian (Eds.). Hybrid Artificial Intelligence Systems SE—50. Lecture Notes in Computer Science. Springer, Berlin, 413--420.
[70]
Q. Wang, O. Sourina, and M. K. Nguyen. 2010. EEG-based “serious” games design for medical applications. In 2010 International Conference on Cyberworlds. 270--276.
[71]
T. Wang, J. Deng, and B. He. 2004. Classifying EEG-based motor imagery tasks by means of time--frequency synthesized spatial patterns. Clinical Neurophysiology 115, 12, 2744--2753.
[72]
C. Wei, Y. Lin, Y. Wang, Y. Wang, T. Jung, and S. Member. 2013. Detection of steady-state visual-evoked potential using differential canonical correlation analysis. In 6th International IEEE/EMBS Conference on Neural Engineering (NER’13). 57--60.
[73]
J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan. 2002. Brain-computer interfaces for communication and control. Clinical Neurophysiology 113, 6, 767--791.
[74]
E. Yin, Z. Zhou, J. Jiang, F. Chen, Y. Liu, and D. Hu. 2013. A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. Journal of Neural Engineering, 10, 2, 026012.
[75]
X. Zhu, B. Gibson, and T. Rogers. 2011. Co-training as a human collaboration policy. AAAI, 20(Nosofsky 1986).

Cited By

View all
  • (2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
  • (2024)Gamification of motor imagery brain-computer interface training protocols: A systematic reviewComputers in Human Behavior Reports10.1016/j.chbr.2024.10050816(100508)Online publication date: Dec-2024
  • (2024)Expert-Guided Model Cultivation: CoTeaching to Resolve Abstruseness and Enhance Learning PerformanceAdvanced Data Mining and Applications10.1007/978-981-96-0814-0_2(18-32)Online publication date: 3-Dec-2024
  • Show More Cited By

Index Terms

  1. Adding Human Learning in Brain--Computer Interfaces (BCIs): Towards a Practical Control Modality

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Computer-Human Interaction
      ACM Transactions on Computer-Human Interaction  Volume 22, Issue 3
      June 2015
      151 pages
      ISSN:1073-0516
      EISSN:1557-7325
      DOI:10.1145/2785963
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 27 May 2015
      Accepted: 01 January 2015
      Revised: 01 January 2015
      Received: 01 July 2014
      Published in TOCHI Volume 22, Issue 3

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Wachspress coordinates
      2. interactive machine learning

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)51
      • Downloads (Last 6 weeks)14
      Reflects downloads up to 03 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
      • (2024)Gamification of motor imagery brain-computer interface training protocols: A systematic reviewComputers in Human Behavior Reports10.1016/j.chbr.2024.10050816(100508)Online publication date: Dec-2024
      • (2024)Expert-Guided Model Cultivation: CoTeaching to Resolve Abstruseness and Enhance Learning PerformanceAdvanced Data Mining and Applications10.1007/978-981-96-0814-0_2(18-32)Online publication date: 3-Dec-2024
      • (2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
      • (2022)A classification and review of tools for developing and interacting with machine learning systemsProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507310(1092-1101)Online publication date: 25-Apr-2022
      • (2022)Designing human-computer interaction with neuroadaptive technologyCurrent Research in Neuroadaptive Technology10.1016/B978-0-12-821413-8.00006-3(1-15)Online publication date: 2022
      • (2022)Human-in-the-loop machine learning: a state of the artArtificial Intelligence Review10.1007/s10462-022-10246-w56:4(3005-3054)Online publication date: 17-Aug-2022
      • (2021)The Thinking CapExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3451732(1-6)Online publication date: 8-May-2021
      • (2021)Review of the State-of-the-Art of Brain-Controlled VehiclesIEEE Access10.1109/ACCESS.2021.31007009(110173-110193)Online publication date: 2021
      • (2021)A Pilot Study of Using Brain-Computer Interfaces in Classrooms for Promoting Formal Educational ActivitiesProceedings of the Future Technologies Conference (FTC) 2021, Volume 310.1007/978-3-030-89912-7_46(590-609)Online publication date: 25-Oct-2021
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media