Abstract
This paper presents the extraction of the emotional signals from traumatic brain-injured (TBI) patients through the analysis of facial features and implementation of the effective emotion-recognition model through the Pepper robot to assist in the rehabilitation process. The identification of emotional cues from TBI patients is very challenging due to unique and diverse psychological, physiological, and behavioral challenges such as non-cooperation, facial/body paralysis, upper or lower limb impairments, cognitive, motor, and hearing skills inhibition. It is essential to read subtle changes in the emotional cues of TBI patients for effective communication and the development of affect-based systems. To analyze the variations of the emotional signal in TBI patients, a new database is collected in a natural and unconstrained environment from eleven residents of a neurological center in three different modalities, RGB, thermal and depth in three specified scenarios performing physical, cognitive and social communication rehabilitation activities. Due to the lack of labeled data, a deep transfer learning method is applied to efficiently classify emotions. The emotion classification model is tested through closed-field study and installment of a Pepper robot equipped with the trained model. Our deep trained and fine-tuned emotional recognition model composed of CNN-LSTM has improved the performance by 1.47% on MMI, and 4.96% on FER2013 validation data set. In addition, use of temporal information and transfer learning techniques to overcome TBI-data limitations has increased the performance efficacy on challenging dataset of neurologically impaired people. Findings that emerged from the study illustrate the noticeable effectiveness of SoftBank Pepper robot equipped with deep trained emotion recognition model in developing rehabilitation strategies by monitoring the TBI patient’s emotions. This research article presents the technical solution for real therapeutic robot interaction to rehabilitate patients with standard monitoring, assessment, and feedback in the neuro centers.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
https://www.sundhed.dk/sundhedsfaglig/
laegehaandbogen/undersoegelser-og-proever/skemaer/geriatri/mms-mini-mental-status/
https://www.mocatest.org/
References
Adolphs R (2002) Neural systems for recognizing emotion. Curr Opin Neurobiol 12(2):169–177
Albiol A, Monzo D, Martin A, Sastre J, Albiol A (2008) Face recognition using hog-ebgm. Pat Recognit Lett 29(10):1537–1543. https://doi.org/10.1016/j.patrec.2008.03.017
Albuquerque VHC, Damaševičius R, Garcia NM, Pinheiro PR, et al. (2017) Brain computer interface systems for neurorobotics: methods and applications
Balconi M (2012) Neuropsychology of facial expressions. the role of consciousness in processing emotional faces. Neuropsychol Trends 11:19–40
Barman A, Chatterjee A, Bhide R (2016) Cognitive impairment and rehabilitation strategies after traumatic brain injury. Indian J Psychol Med 38(3):172–181. https://doi.org/10.4103/0253-7176.183086
Bellantonio M, Haque MA, Rodriguez P, Nasrollahi K, Telve T, Escalera S, Gonzalez J, Moeslund TB, Rasti P, Anbarjafari G (2017) Spatio-temporal pain recognition in CNN-based super-resolved facial images. Springer International Publishing, Cham, pp 151–162
Bemelmans R, Gelderblom GJ, Jonker P, De Witte L (2012) Socially assistive robots in elderly care: a systematic review into effects and effectiveness. J Am Med Dir Assoc 13(2):114–120
Berretti S, Ben Amor B, Daoudi M, del Bimbo A (2011) 3d facial expression recognition using sift descriptors of automatically detected keypoints. Vis Comput 27(11):1021. https://doi.org/10.1007/s00371-011-0611-x
Breuer R, Kimmel R (2017) A deep learning perspective on the origin of facial expressions. arXiv preprint arXiv:170501842
Burgner-Kahrs J, Rucker DC, Choset H (2015) Continuum robots for medical applications: a survey. IEEE Trans Robot 31(6):1261–1280
Cabibihan JJ, Javed H, Ang M, Aljunied SM (2013) Why robots? a survey on the roles and benefits of social robots in the therapy of children with autism. Int J Soc Robot 5(4):593–618
Calvaresi D, Cesarini D, Sernani P, Marinoni M, Dragoni AF, Sturm A (2017) Exploring the ambient assisted living domain: a systematic review. J Ambient Intell Human Comput 8(2):239–257
Chen L, Zhou M, Su W, Wu M, She J, Hirota K (2018) Softmax regression based deep sparse autoencoder network for facial emotion recognition in human–robot interaction. Inform Sci 428:49–61
Corneanu CA, Simón MO, Cohn JF, Guerrero SE (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568
Dhall A, Goecke R, Lucey S, Gedeon T (2011a) Acted facial expressions in the wild database. Australian National University, Canberra, Australia, Technical Report TR-CS-11 2:1
Dhall A, Goecke R, Lucey S, Gedeon T (2011b) Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), IEEE, pp 2106–2112
Dhall A, Goecke R, Ghosh S, Joshi J, Hoey J, Gedeon T (2017) From individual to group-level emotion recognition: Emotiw 5.0. In: Proceedings of the 19th ACM international conference on multimodal interaction, pp 524–528
Du S, Tao Y, Martinez AM (2014) Compound facial expressions of emotion. Proc Natl Acad Sci 111(15):E1454–E1462. https://doi.org/10.1073/pnas.1322355111
Ekman P, Friesen WV, Ellsworth P (2013) Emotion in the human face: guidelines for research and an integration of findings, vol 11. Elsevier, Amsterdam
Elaklouk AM, Zin NAM, Shapii A (2015) Investigating therapists’ intention to use serious games for acquired brain injury cognitive rehabilitation. J King Saud Univ Comput Inf Sci 27(2):160–169
Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 445–450
Friesen WV, Ekman P, et al. (1983) Emfacs-7: Emotional facial action coding system. Unpublished manuscript, University of California at San Francisco 2(36):1
Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH, et al. (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing, Springer, pp 117–124
Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior JCSJ, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26391–26403
Hassner T, Harel S, Paz E, Enbar R (2015) Effective face frontalization in unconstrained images. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4295–4304
Hugentobler JA, Vegh M, Janiszewski B, Quatman-Yates C (2015) Physical therapy intervention strategies for patients with prolonged mild traumatic brain injury symptoms: a case series. Int J Sports Phys Therapy 10(5):676
Ilyas CMA, Haque MA, Rehm M, Nasrollahi K, Moeslund TB (2018a) Effective facial expression recognition through multimodal imaging for traumatic brain injured patient’s rehabilitation. In: International joint conference on computer vision. Springer, Imaging and Computer Graphics, pp 369–389
Ilyas CMA, Nasrollahi K, Rehm M, Moeslund TB (2018b) Rehabilitation of traumatic brain injured patients: Patient mood analysis from multimodal video. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 2291–2295
Ilyas CMA, Schmuck V, Haque MA, Nasrollahi K, Rehm M, Moeslund TB (2019) Teaching pepper robot to recognize emotions of traumatic brain injured patients using deep neural networks. In: 28th IEEE international conference on robot and human interactive communication (roman)
Jones M, Viola P (2003) Fast multi-view face detection. Mitsubishi Electric Research Lab TR-20003-96 3(14):2
Kahou SE, Pal C, Bouthillier X, Froumenty P, Gülçehre Ç, Memisevic R, Vincent P, Courville A, Bengio Y, Ferrari RC, et al. (2013) Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International conference on multimodal interaction, ACM, pp 543–550
Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1725–1732
Kaya H, Gürpinar F, Afshar S, Salah AA (2015) Contrasting and combining least squares based learners for emotion recognition in the wild. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 459–466
Kim BK, Dong SY, Roh J, Kim G, Lee SY (2016) Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 48–57
Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv:170307140
King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10(Jul):1755–1758
Krishna NM, Sekaran K, Vamsi AVN, Ghantasala GP, Chandana P, Kadry S, Blažauskas T, Damaševičius R (2019) An efficient mixture model approach in brain-machine interface systems for extracting the psychological status of mentally impaired persons using eeg signals. IEEE Access 7:77905–77914
Kulkarni K, Corneanu C, Ofodile I, Escalera S, Baro X, Hyniewska S, Allik J, Anbarjafari G (2018) Automatic recognition of facial displays of unfelt emotions. In: IEEE transactions on affective computing
Kuo CM, Lai SH, Sarkis M (2018) A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 2121–2129
Langhorne P, Bernhardt J, Kwakkel G (2011) Stroke rehabilitation. Lancet 377(9778):1693–1702
Li S, Deng W (2020) Deep facial expression recognition: A survey. In: IEEE transactions on affective computing
Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861
Liu M, Wang R, Li S, Shan S, Huang Z, Chen X (2014a) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th international conference on multimodal interaction, ACM, pp 494–501
Liu P, Han S, Meng Z, Tong Y (2014b) Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1805–1812
Liu X, Vijaya Kumar B, You J, Jia P (2017) Adaptive deep metric learning for identity-aware facial expression recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 20–29
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94–101
Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The japanese female facial expression (jaffe) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16
Maskeliūnas R, Damaševičius R, Segal S (2019) A review of internet of things technologies for ambient assisted living environments. Future Internet 11(12):259
Mavadati M, Sanger P, Mahoor MH (2016) Extended disfa dataset: Investigating posed and spontaneous facial expressions. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1–8
Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160
McKenna K, Cooke DM, Fleming J, Jefferson A, Ogden S (2006) The incidence of visual perceptual impairment in patients with severe traumatic brain injury. Brain Injury 20(5):507–518. https://doi.org/10.1080/02699050600664368
Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), IEEE, pp 558–565
Mohammadi MR, Fatemizadeh E, Mahoor MH (2014) Pca-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092
Müri RM (2016) Cortical control of facial expression. J Comp Neurol 524(8):1578–1585
Ng HW, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 443–449
Oquab M, Bottou L, Laptev I, Sivic J (2014) Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1717–1724
Otberdout N, Kacem A, Daoudi M, Ballihi L, Berretti S (2019) Automatic analysis of facial expressions based on deep covariance trajectories. In: IEEE transactions on neural networks and learning systems
Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, IEEE, p 5
Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition
Perry JC, Andureu J, Cavallaro FI, Veneman J, Carmien S, Keller T (2011) Effective game use in neurorehabilitation: user-centered perspectives. In: Handbook of research on improving learning and motivation through educational games: multidisciplinary approaches, IGI Global, pp 683–725
Rapple L (2008) Lotsa helping hands. FOCUS J Respirat Care Sleep Med p 36
Rees L, Marshall S, Hartridge C, Mackie D, Group MWFTE (2007) Cognitive interventions post acquired brain injury. Brain Injury 21(2):161–200. https://doi.org/10.1080/02699050701201813
Robinson H, MacDonald B, Broadbent E (2014) The role of healthcare robots for older people at home: a review. Int J Soc Robot 6(4):575–591
Rodil K, Rehm M, Krummheuer AL (2018) Co-designing social robots with cognitively impaired citizens. In: The 10th Nordic conference on human–computer interaction, association for computing machinery
Rodriguez P, Cucurull G, Gonzàlez J, Gonfaus JM, Nasrollahi K, Moeslund TB, Roca FX (2017) Deep pain: exploiting long short-term memory networks for facial expression classification. IEEE Trans Cybern 99:1–11
Rudovic O, Lee J, Dai M, Schuller B, Picard RW (2018) Personalized machine learning for robot perception of affect and engagement in autism therapy. Sci Robot 3(19):eaao6760
Šalkevicius J, Damaševičius R, Maskeliunas R, Laukien I (2019) Anxiety level recognition for virtual reality therapy system using physiological signals. Electronics 8(9):1039
Sariyanidi E, Gunes H, Cavallaro A (2014) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pat Anal Mach Intell 37(6):1113–1133
Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816. https://doi.org/10.1016/j.imavis.2008.08.005
Shapi’i A, Zin M, Azan N, Elaklouk AM (2015) A game system for cognitive rehabilitation. In: BioMed research international 2015
Stern Y (2009) Cognitive reserve. Neuropsychologia 47(10):2015–2028
Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pat Recognit Lett 119:49–61
Sutton M (2012) Apps to aid aphasia. ASHA Leader 17(7):32, https://search.proquest.com/docview/1022993653
Tang Y (2013) Deep learning using support vector machines. CoRR abs/1306.0239, arXiv:1306.0239
Taylor RH, Menciassi A, Fichtinger G, Fiorini P, Dario P (2016) Medical robotics and computer-integrated surgery. In: Springer handbook of robotics, Springer, pp 1657–1684
Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pat Anal Mach Intell 23(2):97–115
Tsaousides T, Gordon WA (2009) Cognitive rehabilitation following traumatic brain injury: assessment to treatment. Mount Sinai J Med J Trans Personal Med 76(2):173–181. https://doi.org/10.1002/msj.20099
Uddin MZ, Hassan MM, Almogren A, Alamri A, Alrubaian M, Fortino G (2017) Facial expression recognition utilizing local direction-based robust features and deep belief network. IEEE Access 5:4525–4536
Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of 3rd international workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, Paris, France, p 65
Wan J, Escalera S, Anbarjafari G, Jair Escalante H, Baró X, Guyon I, Madadi M, Allik J, Gorbova J, Lin C, et al. (2017) Results and analysis of chalearn lap multi-modal isolated and continuous gesture recognition, and real versus fake expressed emotions challenges. In: Proceedings of the IEEE international conference on computer vision, pp 3189–3197
Yan J, Zheng W, Cui Z, Tang C, Zhang T, Zong Y (2018) Multi-cue fusion for emotion recognition in the wild. Neurocomputing 309:27–35
Yao A, Shao J, Ma N, Chen Y (2015) Capturing au-aware facial features and their latent relations for emotion recognition in the wild. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, ACM, pp 451–458
Yao A, Cai D, Hu P, Wang S, Sha L, Chen Y (2016) Holonet: towards robust emotion recognition in the wild. In: Proceedings of the 18th ACM international conference on multimodal interaction, pp 472–478
Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3d facial expression database for facial behavior research. In: 7th international conference on automatic face and gesture recognition (FGR06), IEEE, pp 211–216
Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 435–442
Yu Z, Liu Q, Liu G (2018) Deeper cascaded peak-piloted network for weak expression recognition. Vis Comput 34(12):1691–1699
Zeng Z, Pantic M, Roisman GI, Huang TS (2008) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pat Anal Mach Intell 31(1):39–58
Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503
Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203
Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P (2013) A high-resolution spontaneous 3d dynamic facial expression database. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), IEEE, pp 1–6
Zhang Z, Luo P, Loy CC, Tang X (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126(5):550–569
Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11(10):9573–9588
Zhao X, Liang X, Liu L, Li T, Han Y, Vasconcelos N, Yan S (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision, Springer, pp 425–442
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ilyas, C.M.A., Rehm, M., Nasrollahi, K. et al. Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes. Pattern Anal Applic 25, 653–677 (2022). https://doi.org/10.1007/s10044-021-00988-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10044-021-00988-8