research-article

Modality switching and performance in a thought and speech controlled computer game

Authors:

Hayrettin Gürkök,

Mannes PoelAuthors Info & Claims

ICMI '11: Proceedings of the 13th international conference on multimodal interfaces

Pages 41 - 48

https://doi.org/10.1145/2070481.2070491

Published: 14 November 2011 Publication History

Abstract

Providing multiple modalities to users is known to improve the overall performance of an interface. Weakness of one modality can be overcome by the strength of another one. Moreover, with respect to their abilities, users can choose between the modalities to use the one that is the best for them. In this paper we explored whether this holds for direct control of a computer game which can be played using a brain-computer interface (BCI) and an automatic speech recogniser (ASR). Participants played the games in unimodal mode (i.e. ASR-only and BCI-only) and multimodal mode where they could switch between the two modalities. The majority of the participants switched modality during the multimodal game but for the most of the time they stayed in ASR control. Therefore multimodality did not provide a significant performance improvement over unimodal control in our particular setup. We also investigated the factors which influence modality switching. We found that performance and peformance-related factors were prominently effective in modality switching.

References

[1]

J. Bieger and G. G. Molina. Light stimulation properties to influence brain activity: A brain-computer interface application. Technical Report TN-2010-00315, Philips Research, Eindhoven, The Netherlands, 2010.

[2]

G. Bin, X. Gao, Z. Yan, B. Hong, and S. Gao. An online multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis method. Journal of Neural Engineering, 6(4):046002, 2009.

[3]

M. Csikszentmihalyi, S. Abuhamdeh, and J. Nakamura. Flow. In Handbook of Competence and Motivation, pages 598--608. The Guilford Press, New York, NY, USA, 2005.

[4]

H. Gürkök and A. Nijholt. Brain-computer interfaces for multimodal interaction: a survey and principles. International Journal of Human-Computer Interaction, 2011. To appear.

[5]

G. Hakvoort, B. Reuderink, and M. Obbink. Comparison of PSDA and CCA detection methods in a SSVEP-based BCI-system. Technical Report TR-CTIT-11-03, Centre for Telematics and Information Technology, University of Twente, Enschede, The Netherlands, 2011.

[6]

J.-P. Haton. Automatic speech recognition: A review. In Enterprise Information Systems V, pages 6--11. Springer, Dordrecht, The Netherlands, 2005.

[7]

C. S. Herrmann. Human EEG responses to 1--100 Hz flicker: resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Experimental Brain Research, 137(3--4):346--353, 2001.

[8]

H. H. Jasper. Report of the committee on methods of clinical examination in electroencephalography: 1957. Electroencephalography and Clinical Neurophysiology, 10(2):370--375, 1958.

[9]

M. Ko, K. Bae, G. Oh, and T. Ryu. A study on new gameplay based on brain-computer interface. In Proceedings of the DiGRA 2009. Brunel University, London, UK, 2009.

[10]

P. Kortum. HCI Beyond the GUI. Morgan Kaufmann, San Francisco, CA, USA, 2008.

[11]

R. L. Mandryk, K. M. Inkpen, and T. W. Calvert. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology, 25(2):141--158, 2006.

[12]

C. Mühl, H. Gürkök, D. Plass-Oude Bos, M. E. Thurlings, L. Scherffig, M. Duvinage, A. A. Elbakyan, S. Kang, M. Poel, and D. Heylen. Bacteria Hunt: Evaluating multi-paradigm BCI interaction. Journal on Multimodal User Interfaces, 4(1):11--25, 2010.

[13]

J. Nielsen. Usability Engineering. Morgan Kaufmann, San Francisco, CA, USA, 1993.

Digital Library

[14]

A. Nijholt, B. Reuderink, and D. Oude Bos. Turning shortcomings into challenges: Brain-computer interfaces for games. In Intelligent Technologies for Interactive Entertainment, pages 153--168. Springer-Verlag, Berlin/Heidelberg, Germany, 2009.

[15]

A. Nijholt, D. Tan, G. Pfurtscheller, C. Brunner, J. del R. Millán, B. Allison, B. Graimann, F. Popescu, B. Blankertz, and K.-R. Müller. Brain-computer interfacing for intelligent systems. IEEE Intelligent Systems, 23:72--79, 2008.

Digital Library

[16]

S. Oviatt. Taming recognition errors with a multimodal interface. Communications of the ACM, 43:45--51, 2000.

Digital Library

[17]

G. Pfurtscheller, B. Z. Allison, C. Brunner, G. Bauernfeind, T. Solis-Escalante, R. Scherer, T. O. Zander, G. Mueller-Putz, C. Neuper, and N. Birbaumer. The hybrid BCI. Frontiers in Neuroprosthetics, 4:30, 2010.

[18]

D. Tan and A. Nijholt, editors. Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction. Springer-Verlag, London, UK, 2010.

Digital Library

[19]

W. Walker, P. Lamere, P. Kwok, B. Raj, R. Singh, E. Gouvea, P. Wolf, and J. Woelfel. Sphinx-4: A flexible open source framework for speech recognition. Technical Report SMLI TR-2004--139, Sun Microsystems Inc., Menlo Park, CA, USA, 2004.

Digital Library

[20]

M.-S. Yoh, J. Kwon, and S. Kim. NeuroWander: a BCI game in the form of interactive fairy tale. In Proceedings of the Ubicomp'10, pages 389--390, New York, NY, USA, 2010. ACM.

Digital Library

[21]

T. O. Zander and C. Kothe. Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. Journal of Neural Engineering, 8(2):025005, 2011.

Cited By

Auda JGruenefeld UKosch TSchneegass S(2022)The Butterfly Effect: Novel Opportunities for Steady-State Visually-Evoked Potential Stimuli in Virtual RealityProceedings of the Augmented Humans International Conference 202210.1145/3519391.3519397(254-266)Online publication date: 13-Mar-2022
https://dl.acm.org/doi/10.1145/3519391.3519397
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Kosmyna NLécuyer A(2019)A conceptual space for EEG-based brain-computer interfacesPLOS ONE10.1371/journal.pone.021014514:1(e0210145)Online publication date: 3-Jan-2019
https://doi.org/10.1371/journal.pone.0210145
Show More Cited By

Index Terms

Modality switching and performance in a thought and speech controlled computer game
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Buses and high-speed links
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens

Recommendations

User expectations and experiences of a speech and thought controlled computer game
ACE '11: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology

Brain-computer interfaces (BCIs) are often evaluated in terms of performance and seldom for usability. However in some application domains, such as entertainment computing, user experience evaluation is vital. User experience evaluation in BCI systems, ...
Evaluating user experience in a selection based brain-computer interface game a comparative study: a comparative study
ICEC'11: Proceedings of the 10th international conference on Entertainment Computing

In human-computer interaction, it is important to offer the users correct modalities for particular tasks and situations. Unless the user has the suitable modality for a task, neither task performance nor user experience can be optimised. The aim of ...
Exploring Cross-Modality Affective Reactions for Audiovisual Emotion Recognition

Psycholinguistic studies on human communication have shown that during human interaction individuals tend to adapt their behaviors mimicking the spoken style, gestures, and expressions of their conversational partners. This synchronization pattern is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '11: Proceedings of the 13th international conference on multimodal interfaces

November 2011

432 pages

ISBN:9781450306416

DOI:10.1145/2070481

General Chairs:
Hervé Bourlard
Idiap Research Institute, Switzerland
,
Thomas S. Huang
University of Illinois, USA
,
Enrique Vidal
Universitat Politécnica Valéncia, Spain
,
Program Chairs:
Daniel Gatica-Perez
Idiap Research Institute, Switzerland
,
Louis-Philippe Morency
University of Southern California, USA
,
Nicu Sebe
University of Trento, Italy

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI'11

Sponsor:

SIGCHI

ICMI'11: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 14 - 18, 2011

Alicante, Spain

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
309
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Auda JGruenefeld UKosch TSchneegass S(2022)The Butterfly Effect: Novel Opportunities for Steady-State Visually-Evoked Potential Stimuli in Virtual RealityProceedings of the Augmented Humans International Conference 202210.1145/3519391.3519397(254-266)Online publication date: 13-Mar-2022
https://dl.acm.org/doi/10.1145/3519391.3519397
Putze FPutze SSagehorn MMicek CSolovey E(2022)Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and ReuseACM Transactions on Computer-Human Interaction10.1145/349055429:4(1-43)Online publication date: 31-Mar-2022
https://dl.acm.org/doi/10.1145/3490554
Kosmyna NLécuyer A(2019)A conceptual space for EEG-based brain-computer interfacesPLOS ONE10.1371/journal.pone.021014514:1(e0210145)Online publication date: 3-Jan-2019
https://doi.org/10.1371/journal.pone.0210145
Kosmyna NLécuyer A(2017)Designing Guiding Systems for Brain-Computer InterfacesFrontiers in Human Neuroscience10.3389/fnhum.2017.0039611Online publication date: 31-Jul-2017
https://doi.org/10.3389/fnhum.2017.00396
Évain AArgelaguet FCasiez GRoussel NLécuyer A(2016)Design and Evaluation of Fusion Approach for Combining Brain and Gaze Inputs for Target SelectionFrontiers in Neuroscience10.3389/fnins.2016.0045410Online publication date: 7-Oct-2016
https://doi.org/10.3389/fnins.2016.00454
Nijholt APoel M(2016)Multi-Brain BCIProceedings, Part I, 10th International Conference on Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience - Volume 974310.1007/978-3-319-39955-3_8(79-90)Online publication date: 17-Jul-2016
https://dl.acm.org/doi/10.1007/978-3-319-39955-3_8
Kosmyna NTarpin-Bernard FCouture NBastien CDorta T(2014)A combination of brain-computer interaction and eye tracking paradigmes for multimodal interactionProceedings of the 2014 Ergonomie et Informatique Avancée Conference - Design, Ergonomie et IHM: quelle articulation pour la co-conception de l'interaction10.1145/2671470.2671486(110-113)Online publication date: 15-Oct-2014
https://dl.acm.org/doi/10.1145/2671470.2671486
LaViola Jr. JBuchanan SPittman C(2014)Multimodal Input for Perceptual User InterfacesInteractive Displays10.1002/9781118706237.ch9(285-312)Online publication date: 12-Jul-2014
https://doi.org/10.1002/9781118706237.ch9
Liarokapis FVourvopoulos AEne APetridis P(2013)Assessing Brain-Computer Interfaces for Controlling Serious Games2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES)10.1109/VS-GAMES.2013.6624222(1-4)Online publication date: Sep-2013
https://doi.org/10.1109/VS-GAMES.2013.6624222
Kos'myna NTarpin-Bernard F(2013)Evaluation and Comparison of a Multimodal Combination of BCI Paradigms and Eye Tracking With Affordable Consumer-Grade Hardware in a Gaming ContextIEEE Transactions on Computational Intelligence and AI in Games10.1109/TCIAIG.2012.22300035:2(150-154)Online publication date: Jun-2013
https://doi.org/10.1109/TCIAIG.2012.2230003
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten