research-article

Implicit Emotion Communication: EEG Classification and Haptic Feedback

Authors:

Rodrigo Ceballos,

Beatrice Ionascu,

Mohamad EidAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 14, Issue 1

Article No.: 3, Pages 1 - 18

https://doi.org/10.1145/3152128

Published: 20 December 2017 Publication History

Abstract

Today, ubiquitous digital communication systems do not have an intuitive, natural way of communicating emotion, which, in turn, affects the degree to which humans can emotionally connect and interact with one another. To address this problem, a more natural, intuitive, and implicit emotion communication system was designed and created that employs asymmetry-based EEG emotion classification for detecting the emotional state of the sender and haptic feedback (in the form of tactile gestures) for displaying emotions for a receiver. Emotions are modeled in terms of valence (positive/negative emotions) and arousal (intensity of the emotion). Performance analysis shows that the proposed EEG subject-dependent emotion classification model with Free Asymmetry features allows for more flexible feature-generation schemes than other existing algorithms and attains an average accuracy of 92.5% for valence and 96.5% for arousal, outperforming previous-generation schemes in high feature space. As for the haptic feedback, a tactile gesture authoring tool and a haptic jacket were developed to design tactile gestures that can intensify emotional reactions in terms of valence and arousal. Experimental study demonstrated that subject-independent emotion transmission through tactile gestures is effective for the arousal dimension of an emotion but is less effective for valence. Consistency in subject-dependent responses for both valence and arousal suggests that personalized tactile gestures would be more effective.

References

[1]

Omar AlZoubi, Rafael A. Calvo, and Ronald H. Stevens. 2009. Classification of EEG for affect recognition: An adaptive approach. In Australasian Joint Conference on Artificial Intelligence. Springer, 52--61.

Digital Library

[2]

Bradley M. Appelhans and Linda J. Luecken. 2006. Heart rate variability as an index of regulated emotional responding. Review of General Psychology 10, 3, 229.

[3]

Faisal Arafsha, Kazi Masudul Alam, and Abdulmotaleb El Saddik. 2015. Design and development of a user centric affective haptic jacket. Multimedia Tools and Applications 74, 9, 3035--3052.

Digital Library

[4]

Jeremy N. Bailenson, Nick Yee, Scott Brave, Dan Merget, and David Koslow. 2007. Virtual interpersonal touch: Expressing and recognizing emotions through haptic devices. Human--Computer Interaction 22, 3, 325--353.

Digital Library

[5]

Tonio Ball, Markus Kern, Isabella Mutschler, Ad Aertsen, and Andreas Schulze-Bonhage. 2009. Signal quality of simultaneously recorded invasive and non-invasive EEG. Neuroimage 46, 3, 708--716.

[6]

Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25, 1, 49--59.

[7]

Lindsay Brown, Bernard Grundlehner, and Julien Penders. 2011. Towards wireless emotional valence detection from EEG. In Engineering in Medicine and Biology Society (EMBC’11) Annual International Conference of the IEEE. IEEE, 2188--2191.

[8]

Ross Buck. 1994. The neuropsychology of communication: Spontaneous and symbolic aspects. Journal of Pragmatics 22, 3-4, 265--278.

[9]

Jongeun Cha, Mohamad Eid, Ahmad Barghout, ASM Rahman, and Abdulmotaleb El Saddik. 2009. HugMe: Synchronous haptic teleconferencing. In Proceedings of the 17th ACM International Conference on Multimedia. ACM, 1135--1136.

Digital Library

[10]

Monisha Chakraborty and Shreya Das. 2012. Determination of signal to noise ratio of electrocardiograms filtered by band pass and Savitzky-Golay filters. Procedia Technology 4, 830--833.

[11]

Manfred Clynes. 1977. Sentics: The Touch of Emotions. Anchor Press, New York, NY.

[12]

Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In Neural Engineering (NER), 2013 6th International IEEE/EMBS Conference on. IEEE, 81--84.

[13]

Elia Gatti, Giandomenico Caruso, Monica Bordegoni, and Charles Spence. 2013. Can the feel of the haptic interaction modify a user’s emotional state?. In World Haptics Conference (WHC’13). IEEE, 247--252.

[14]

Antal Haans and Wijnand I. Jsselsteijn. 2006. Mediated social touch: A review of current research and future directions. Virtual Reality 9, 2-3, 149--159.

Digital Library

[15]

Ali Israr and Ivan Poupyrev. 2011. Control space of apparent haptic motion. In World Haptics Conference (WHC’11). IEEE, 457--462.

[16]

Tanimu Ahmed Jibril and Mardziah Hayati Abdullah. 2013. Relevance of emoticons in computer-mediated communication contexts: An overview. Asian Social Science 9, 4, 201.

[17]

Min-Ki Kim, Miyoung Kim, Eunmi Oh, and Sung-Phil Kim. 2013. A review on the computational methods for emotional state estimation from the human EEG. Computational and Mathematical Methods in Medicine, vol. 2013, 13 pages.

[18]

Franklin B. Krohn. 2004. A generational approach to using emoticons as nonverbal communication. Journal of Technical Writing and Communication 34, 4, 321--328.

[19]

Peter J. Lang, Margaret M. Bradley, and Bruce N. Cuthbert. 2008. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. Gainesville, FL: University of Florida, USA.

[20]

Paul Lemmens, Floris Crompvoets, Dirk Brokken, Jack Van Den Eerenbeemd, and Gert-Jan de Vries. 2009. A body-conforming tactile jacket to enrich movie viewing. In 3rd Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (World Haptics’09). IEEE, 7--12.

Digital Library

[21]

Rodrigo Lentini, Beatrice Ionascu, Friederike A. Eyssel, Scandar Copti, and Mohamad Eid. 2016. Authoring tactile gestures: Case study for emotion stimulation. World Academy of Science, Engineering and Technology, International Journal of Computer, Electrical, Automation, Control and Information Engineering 10, 11, 1862--1867.

[22]

Antonella Mazzoni and Nick Bryan-Kinns. 2015a. How does it feel like? An exploratory study of a prototype system to convey emotion through haptic wearable devices. In 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN’15). IEEE, 64--68.

Digital Library

[23]

Troy McDaniel, Shantanu Bala, Jacob Rosenthal, Ramin Tadayon, Arash Tadayon, and Sethuraman Panchanathan. 2014. Affective haptics for enhancing access to social interactions for individuals who are blind. In International Conference on Universal Access in Human-Computer Interaction. Springer, 419--429.

[24]

Neurobehavioral Systems Inc. 2017. Presentation. Retrieved November 14, 2017 from https://www.neurobs.com/.

[25]

Lauri Nummenmaa, Enrico Glerean, Riitta Hari, and Jari K. Hietanen. 2014. Bodily maps of emotions. Proceedings of the National Academy of Sciences 111, 2, 646--651.

[26]

Panagiotis C. Petrantonakis and Leontios J. Hadjileontiadis. 2011. A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition. IEEE Transactions on Information Technology in Biomedicine 15, 5, 737--746.

Digital Library

[27]

Abu Saleh Md Mahfujur Rahman, S. K. Alamgir Hossain, and Abdulmotaleb El Saddik. 2010. Bridging the gap between virtual and real world by bringing an interpersonal haptic communication system in second life. In IEEE International Symposium on Multimedia (ISM’10). IEEE, 228--235.

Digital Library

[28]

Jukka Raisamo, Roope Raisamo, and Veikko Surakka. 2013. Comparison of saltation, amplitude modulation, and a hybrid method of vibrotactile stimulation. IEEE Transactions on Haptics 6, 4, 517--521.

Digital Library

[29]

Debra L. Roter, Richard M. Frankel, Judith A. Hall, and David Sluyter. 2006. The expression of emotion through nonverbal behavior in medical visits. Journal of General Internal Medicine 21, S1, S28--S34.

[30]

Katri Salminen, Veikko Surakka, Jani Lylykangas, Jukka Raisamo, Rami Saarinen, Roope Raisamo, Jussi Rantala, and Grigori Evreinov. 2008. Emotional and behavioral responses to haptic stimulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1555--1562.

Digital Library

[31]

Louis A. Schmidt and Laurel J. Trainor. 2001. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition 8 Emotion 15, 4, 487--500.

[32]

Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2012. Multimodal emotion recognition in response to videos. IEEE Transactions on Affective Computing 3, 2, 211--223.

Digital Library

[33]

Kai Sun, Junqing Yu, Yue Huang, and Xiaoqiang Hu. 2009. An improved valence-arousal emotion space for video affective content representation and recognition. In IEEE International Conference on Multimedia and Expo (ICME’09). IEEE, 566--569.

Digital Library

[34]

Dzmitry Tsetserukou, Alena Neviarouskaya, Helmut Prendinger, Mitsuru Ishizuka, and Susumu Tachi. 2010. iFeel_IM: Innovative real-time communication system with rich emotional and haptic channels. In CHI’10 Extended Abstracts on Human Factors in Computing Systems. ACM, 3031--3036.

Digital Library

[35]

Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing 129, 94--106.

Digital Library

[36]

Joyce H. D. M. Westerink, Egon L. Van Den Broek, Marleen H. Schut, Jan Van Herk, and Kees Tuinenbreijer. 2008. Computing emotion awareness through galvanic skin response and facial electromyography. In Probing Experience. Springer, 149--162.

[37]

Yongjae Yoo, Taekbeom Yoo, Jihyun Kong, and Seungmoon Choi. 2015. Emotional responses of tactile icons: Effects of amplitude, frequency, duration, and envelope. In IEEE World Haptics Conference (WHC’15). IEEE, 235--240.

[38]

Xu Zhe and A. C. Boucouvalas. 2002. Text-to-emotion engine for real time Internet communication. In Proceedings of International Symposium on Communication Systems, Networks and DSPs. 164--168.

[39]

Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2016. Identifying stable patterns over time for emotion recognition from EEG. Arxiv Preprint Arxiv:1601.02197.

Cited By

Erat KŞahin EDoğan FMerdanoğlu NAkcakaya ADurdu P(2024)Emotion recognition with EEG-based brain-computer interfaces: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-024-18259-z83:33(79647-79694)Online publication date: 1-Mar-2024
https://doi.org/10.1007/s11042-024-18259-z
Wang XXu BZhang WWang JDeng LPing JHu CLi H(2023)Recognizing emotions induced by wearable haptic vibration using noninvasive electroencephalogramFrontiers in Neuroscience10.3389/fnins.2023.121955317Online publication date: 6-Jul-2023
https://doi.org/10.3389/fnins.2023.1219553
Huang WZhang YWan S(2022)A Sorting Fuzzy Min-Max Model in an Embedded System for Atrial Fibrillation DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355473718:2s(1-18)Online publication date: 5-Aug-2022
https://dl.acm.org/doi/10.1145/3554737
Show More Cited By

Index Terms

Implicit Emotion Communication: EEG Classification and Haptic Feedback
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Emotion Actuator: Embodied Emotional Feedback through Electroencephalography and Electrical Muscle Stimulation
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

The human body reveals emotional and bodily states through measurable signals, such as body language and electroencephalography. However, such manifestations are difficult to communicate to others remotely. We propose EmotionActuator, a proof-of-concept ...
Intensifying Emotional Reactions via Tactile Gestures in Immersive Films

The film industry continuously strives to make visitors’ movie experience more immersive and thus, more captivating. This is realized through larger screens, sophisticated speaker systems, and high quality 2D and 3D content. Moreover, a recent trend in ...
The Impact of Thermal Cues on Affective Responses to Emotionally Resonant Vibrations
ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction

This paper investigates how presenting emotionally resonant vibrotactile stimuli at cool, neutral and warm temperature levels impacts mean ratings for emotional resonance and affective response. Affective vibrotactile stimuli can elicit pleasant or ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 14, Issue 1

February 2018

287 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3173554

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 December 2017

Accepted: 01 August 2017

Revised: 01 June 2017

Received: 01 March 2017

Published in TOMM Volume 14, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

New York University Abu Dhabi

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
656
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)9

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Erat KŞahin EDoğan FMerdanoğlu NAkcakaya ADurdu P(2024)Emotion recognition with EEG-based brain-computer interfaces: a systematic literature reviewMultimedia Tools and Applications10.1007/s11042-024-18259-z83:33(79647-79694)Online publication date: 1-Mar-2024
https://doi.org/10.1007/s11042-024-18259-z
Wang XXu BZhang WWang JDeng LPing JHu CLi H(2023)Recognizing emotions induced by wearable haptic vibration using noninvasive electroencephalogramFrontiers in Neuroscience10.3389/fnins.2023.121955317Online publication date: 6-Jul-2023
https://doi.org/10.3389/fnins.2023.1219553
Huang WZhang YWan S(2022)A Sorting Fuzzy Min-Max Model in an Embedded System for Atrial Fibrillation DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/355473718:2s(1-18)Online publication date: 5-Aug-2022
https://dl.acm.org/doi/10.1145/3554737
Li DYang ZHou FKang QLiu SSong YGao QDong E(2022)EEG-Based Emotion Recognition With Haptic Vibration by a Feature Fusion MethodIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2022.314788271(1-11)Online publication date: 2022
https://doi.org/10.1109/TIM.2022.3147882
Wang XFang XZheng XDeng XLi YWang M(2022)Brain Oscillatory Representations of Vibrotactile Parameters: An EEG Study2022 5th International Conference on Computing and Informatics (ICCI)10.1109/ICCI54321.2022.9756106(035-043)Online publication date: 9-Mar-2022
https://doi.org/10.1109/ICCI54321.2022.9756106
LEI XZHANG TCHEN KZHANG JTIAN YFANG FCHEN L(2019)Psychophysics of wearable haptic/tactile perception in a multisensory contextVirtual Reality & Intelligent Hardware10.3724/SP.J.2096-5796.2018.00121:2(185-200)Online publication date: Apr-2019
https://doi.org/10.3724/SP.J.2096-5796.2018.0012
McDaniel TPanchanathan S(2019)Therapeutic Haptics for Mental Health and WellbeingHaptic Interfaces for Accessibility, Health, and Enhanced Quality of Life10.1007/978-3-030-34230-2_6(149-181)Online publication date: 19-Dec-2019
https://doi.org/10.1007/978-3-030-34230-2_6

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents