article

Advocating a Componential Appraisal Model to Guide Emotion Recognition

Authors:

Marcello Mortillaro,

Klaus R. SchererAuthors Info & Claims

International Journal of Synthetic Emotions, Volume 3, Issue 1

Pages 18 - 32

https://doi.org/10.4018/jse.2012010102

Published: 01 January 2012 Publication History

Abstract

Most models of automatic emotion recognition use a discrete perspective and a black-box approach, i.e., they output an emotion label chosen from a limited pool of candidate terms, on the basis of purely statistical methods. Although these models are successful in emotion classification, a number of practical and theoretical drawbacks limit the range of possible applications. In this paper, the authors suggest the adoption of an appraisal perspective in modeling emotion recognition. The authors propose to use appraisals as an intermediate layer between expressive features input and emotion labeling output. The model would then be made of two parts: first, expressive features would be used to estimate appraisals; second, resulting appraisals would be used to predict an emotion label. While the second part of the model has already been the object of several studies, the first is unexplored. The authors argue that this model should be built on the basis of both theoretical predictions and empirical results about the link between specific appraisals and expressive features. For this purpose, the authors suggest to use the component process model of emotion, which includes detailed predictions of efferent effects of appraisals on facial expression, voice, and body movements.

References

[1]

Arnold, M. B. 1960. Emotion and personality. New York, NY: Columbia University Press.

[2]

Aue, T., Flykt, A.,&Scherer, K. R. 2007. First evidence for differential and sequential efferent effects of goal relevance and goal conduciveness appraisal. Biological Psychology, 74, 347-357. 17052833.

[3]

Aue, T.,&Scherer, K. R. 2008. Appraisal-driven somatovisceral response patterning: Effects of intrinsic pleasantness and goal conduciveness. Biological Psychology, 79, 158-164. 18495321.

[4]

Banse, R.,&Scherer, K. R. 1996. Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70, 614-636. 8851745.

[5]

Bänziger, T., Mortillaro, M.,&Scherer, K. R. 2011. Introducing the Geneva Multimodal Expression corpus for experimental research on emotion perception. Emotion.

[6]

Bänziger, T.,&Scherer, K. R. 2010. Introducing the Geneva Multimodal Emotion Portrayal GEMEP Corpus. In Scherer, K. R., Bäänziger, T.,&Roesch, E. B. Eds., Blueprint for affective computing: A sourcebook pp. 271-294. New York, NY: Oxford University Press.

[7]

Becker-Asano, C. 2008. WASABI: Affect simulation for agents with believable interactivity Doctoral dissertation, University of Bielefeld. Amsterdam, The Netherlands: IOS Press.

[8]

Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T.,&Ton, V. 1997. Matsumoto and Ekman's Japanese and Caucasian Facial Expressions of Emotion JACFEE: Reliability data and cross-national differences. Journal of Nonverbal Behavior, 21, 3-21.

[9]

Calvo, R. A.,&D'Mello, S. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1, 18-37.

Digital Library

[10]

Castellano, G., Caridakis, G., Camurri, A., Karpouzis, K., Volpe, G.,&Kollias, S. 2010. Body gesture and facial expression analysis for automatic affect recognition. In Scherer, K. R., Bäänziger, T.,&Roesch, E. B. Eds., Blueprint for affective computing: A sourcebook pp. 245-255. New York, NY: Oxford University Press.

[11]

Coulson, M. 2009. Expressing emotion through body movement: A component process approach. In Canamero, L.,&Aylett, R. Eds., Animating expressive characters for social interaction Advances in Consciousness Research Series Vol. 74. Amsterdam, The Netherlands: Benjamins.

[12]

Dael, N., Mortillaro, M.,&Scherer, K. R. 2011. Emotion expression in body action and posture. Emotion.

[13]

Dael, N., Mortillaro, M.,&Scherer, K. R. 2012. The body action and posture coding system BAP: Development and reliability. Journal of Nonverbal Behavior.

[14]

Delplanque, S., Grandjean, D., Chrea, C., Coppin, G., Aymard, L.,&Cayeux, I. et¿al. 2009. Sequential unfolding of novelty and pleasantness appraisals of odors: evidence from facial electromyography and autonomic reactions. Emotion Washington, D.C., 93, 316-328. 19485609.

[15]

Devillers, L., Vidrascu, L.,&Layachi, O. 2010. Automatic detection of emotion from vocal expression. In Scherer, K. R., Bäänziger, T.,&Roesch, E. B. Eds., Blueprint for affective computing: A sourcebook pp. 232-244. New York, NY: Oxford University Press.

[16]

Ekman, P. 1992. Facial expressions of emotion: New findings, new questions. Psychological Science, 3, 34-38.

[17]

Ekman, P. 1999. Facial expressions. In Dalgleish, T.,&Power, M. J. Eds., Handbook of cognition and emotion pp. 301-320. New York, NY: John Wiley&Sons.

[18]

Ekman, P.,&Friesen, W. V. 1978. Facial action coding system. Palo Alto, CA: Consulting Psychologists Press.

[19]

Ekman, P., Levenson, R. W.,&Friesen, W. V. 1983. Autonomic nervous system activity distinguishes among emotions. Science, 221, 1208-1210. 6612338.

[20]

Ekman, P., Sorenson, E. R.,&Friesen, W. V. 1969. Pan-cultural elements in facial displays of emotion. Science, 164, 86-88. 5773719.

[21]

El Kaliouby, R.,&Robinson, P. 2004. Real time inference of complex mental states from facial expressions and head gestures. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Workshop on Real Time Computer Vision for Human Computer Interaction Vol. 10, p. 154.

[22]

Ellsworth, P. C.,&Scherer, K. R. 2003. Appraisal processes in emotion. In Davidson, R. J., Scherer, K. R.,&Goldsmith, H. Eds., Handbook of the affective sciences pp. 572-595. New York, NY: Oxford University Press.

[23]

Ellsworth, P. C.,&Smith, C. A. 1988. Shades of joy: Patterns of appraisal differentiating pleasant emotions. Cognition and Emotion, 2, 301-331.

[24]

Frijda, N. H. 1986. The emotions. Cambridge, UK: Cambridge University Press.

[25]

Gunes, H.,&Pantic, M. 2010. Automatic, dimensional and continuous emotion recognition. International Journal of Synthetic Emotions, 1, 68-99.

Digital Library

[26]

Johnstone, T. van Reekum,&Scherer, K. R. 2001. Vocal expression correlates of appraisal processes. In K. R. Scherer, A. Schorr,&T. Johnstone Eds., Appraisal processes in emotion pp. 271-284. New York, NY: Oxford University Press.

[27]

Johnstone, T., van Reekum, C. M., Hird, K., Kirsner, K.,&Scherer, K. R. 2005. Affective speech elicited with a computer game. Emotion Washington, D.C., 54, 513-518. 16366756.

[28]

Kaiser, S.,&Wehrle, T. 2001. Facial expressions as indicators of appraisal processes. In Scherer, K. R., Schorr, A.,&Johnstone, T. Eds., Appraisal processes in emotion pp. 285-300. New York, NY: Oxford University Press.

[29]

Kanade, T., Cohn, J. F.,&Tian, Y. 2000. Comprehensive database for facial expression analysis. In Proceedings of the International Conference on Face and Gesture Recognition pp. 46-53.

[30]

Laukka, P., Elfenbein, H. A., Chui, W., Thingujam, N. S., Iraki, F. K., Rockstuhl, T.,&Althoff, J. 2010. Presenting the VENEC corpus: Development of a cross-cultural corpus of vocal emotion expressions and a novel method of annotating emotion appraisals. In Proceedings of the LREC Workshop on Corpora for Research on Emotion and Affect pp. 53-57.

[31]

Lazarus, R. S. 1991. Emotion and adaptation. New York, NY: Oxford University Press.

[32]

Lewis, M. D. 2005. Bridging emotion theory and neurobiology through dynamic systems modeling. The Behavioral and Brain Sciences, 28, 169-245. 16201458.

[33]

Littlewort, G. C., Bartlett, M. S.,&Lee, K. 2007. Faces of pain: automated measurement of spontaneous facial expressions of genuine and posed pain. In Proceedings of the 9th International Conference on Multimodal Interfaces pp. 15-21.

[34]

Marinier, R. P. 2008. A computational unification of cognitive control, emotion and learning Unpublished doctoral dissertation. University of Michigan, Ann Arbor, MI.

[35]

Marsella, S., Gratch, J.,&Petta, P. 2010. Computational models of emotion. In Scherer, K. R., Bäänziger, T.,&Roesch, E. B. Eds., Blueprint for affective computing: A sourcebook pp. 21-41. New York, NY: Oxford University Press.

[36]

Mortillaro, M., Mehu, M.,&Scherer, K. R. 2011. Subtly different positive emotions can be distinguished by their facial expressions. Social Psychological and Personality Science, 2, 262-271.

[37]

Mortillaro, M., Mehu, M.,&Scherer, K. R. in press. The evolutionary origin of multimodal synchronization in emotional expression. In Altenmüüller, E., Schmidt, S.,&Zimmerman, E. Eds., Evolution of emotional communication: from sounds in nonhuman mammals to speech and music in man. New York, NY: Oxford University Press.

[38]

Naab, P. J.,&Russell, J. A. 2007. Judgments of emotion from spontaneous facial expressions of New Guineans. Emotion Washington, D.C., 7, 736-744. 18039042.

[39]

Nezlek, J. B., Vansteelandt, K., Van Mechelen, I.,&Kuppens, P. 2008. Appraisal-emotion relationships in daily life. Emotion Washington, D.C., 8, 145-150. 18266526.

[40]

Ortony, A., Clore, G. L.,&Collins, A. 1988. The cognitive structure of emotions. Cambridge, UK: Cambridge University Press.

[41]

Osgood, C. E. 1962. Studies of the generality of affective meaning systems. The American Psychologist, 17, 10-28.

[42]

Osgood, C. E. 1964. Semantic differential technique in the comparative study of cultures. American Anthropologist, 66, 171-200.

[43]

Patel, S., Scherer, K. R., Björkner, E.,&Sundberg, J. 2011. Mapping emotions into acoustic space: The role of voice production. Biological Psychology, 87, 93-98. 21354259.

[44]

Pope, L. K.,&Smith, C. A. 1994. On the distinct meanings of smiles and frowns. Cognition and Emotion, 8, 65-72.

[45]

Roseman, I.,&Evdokas, A. 2004. Appraisals cause experienced emotions: Experimental evidence. Cognition and Emotion, 18, 1-28.

[46]

Roseman, I. J.,&Smith, C. A. 2001. Appraisal theory: Overview,assumptions, varieties, controversies. In Scherer, K. R., Schorr, A.,&Johnstone, T. Eds., Appraisal processes in emotion: Theory, methods, research pp. 3-19. New York, NY: Oxford University Press.

[47]

Russell, J. A. 2003. Core affect and the psychological construction of emotion. Psychological Review, 110, 145-172. 12529060.

[48]

Russell, J. A., Bachorowski, J.-A.,&Fernandez-Dols, J.-M. 2003. Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329-349. 12415074.

[49]

Russell, J. A.,&Bullock, M. 1986. On the dimensions preschoolers use to interpret facial expressions of emotion. Developmental Psychology, 22, 97-102.

[50]

Russell, J. A.,&Fernandez-Dols, J. M. 1997. The psychology of facial expression. New York, NY: Cambridge University Press.

[51]

Russell, J. A.,&Mehrabian, A. 1974. Distinguishing anger and anxiety in terms of emotional response factors. Journal of Consulting and Clinical Psychology, 42, 79-83. 4814102.

[52]

Russell, J. A.,&Mehrabian, A. 1977. Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11, 273-294.

[53]

Sander, D., Grandjean, D.,&Scherer, K. R. 2005. A systems approach to appraisal mechanisms in emotion. Neural Networks, 18, 317-352. 15936172.

Digital Library

[54]

Scherer, K. R. 1984. Emotion as a multicomponent process: A model and some cross-cultural data. In P. Shaver Ed., Review of personality and social psychology: Vol. 5. Emotions, relationships and health pp. 37-63. Thousand Oaks, CA: Sage.

[55]

Scherer, K. R. 1994. Toward a concept of ''modal'' emotions. In Ekman, P.,&Davidson, R. J. Eds., The nature of emotion: Fundamental questions pp. 25-31. New York, NY: Oxford University Press.

[56]

Scherer, K. R. 2000. Emotions as episodes of subsystem synchronization driven by nonlinear appraisal processes. In Lewis, M. D.,&Granic, I. Eds., Emotion, development, and self- organization: Dynamic systems approaches to emotional development pp. 70-99. Cambridge, UK: Cambridge University Press.

[57]

Scherer, K. R. 2001. Appraisal considered as a process of multilevel sequential checking. In Scherer, K. R., Schorr, A.,&Johnstone, T. Eds., Appraisal processes in emotion: Theory, methods, research pp. 92-120. New York, NY: Oxford University Press.

[58]

Scherer, K. R. 2009a. The dynamic architecture of emotion: Evidence for the component process model. Cognition and Emotion, 23, 1307-1351.

[59]

Scherer, K. R. 2009b. Emotions are emergent processes: they require a dynamic computational architecture. Philosophical Transactions of the Royal Society B, 364, 3459-3474. 19884141.

[60]

Scherer, K. R. 2010. The component process model: Architecture for a comprehensive computational model of emergent emotion. In Scherer, K. R., Bänziger, T.,&Roesch, E. B. Eds., Blueprint for affective computing: A sourcebook pp. 47-70. New York, NY: Oxford University Press.

[61]

Scherer, K. R., Clark- Polner, E.,&Mortillaro, M. 2011. In the eye of the beholder? Universality and cultural specificity in the expression and perception of emotion. International Journal of Psychology, 46, 401-435. 22126090.

[62]

Scherer, K. R.,&Brosch, T. 2009. Culture-specific appraisal biases contribute to emotion dispositions. European Journal of Personality, 288, 265-288.

[63]

Scherer, K. R.,&Ceschi, G. 2000. Criteria for emotion recognition from verbal and nonverbal expression: Studying baggage loss in the airport. Personality and Social Psychology Bulletin, 26, 327-339.

[64]

Scherer, K. R.,&Ellgring, H. 2007a. Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion Washington, D.C., 7, 113-130. 17352568.

[65]

Scherer, K. R.,&Ellgring, H. 2007b. Multimodal expression of emotion: Affect programs or componential appraisal patterns? Emotion Washington, D.C., 7, 158-171. 17352571.

[66]

Scherer, K. R.,&Grandjean, D. 2008. Facial expressions allow inference of both emotions and their components. Cognition and Emotion, 22, 789-801.

[67]

Scherer, K. R., Schorr, A.,&Johnstone, T. 2001. Appraisal processes in emotion: Theory, methods, research. New York, NY: Oxford University Press.

[68]

Scherer, K. R., Wranik, T., Sangsue, J., Tran, V.,&Scherer, U. 2004. Emotions in everyday life: Probability of occurrence, risk factors, appraisal and reaction pattern. Social Sciences Information. Information Sur les Sciences Sociales, 43, 499-570.

[69]

Siemer, M., Mauss, I.,&Gross, J. J. 2007. Same situation--different emotions: how appraisals shape our emotions. Emotion Washington, D.C., 7, 592-600. 17683215.

[70]

Smith, C. A. 1989. Dimensions of appraisal and physiological response in emotion. Journal of Personality and Social Psychology, 56, 339-353. 2926633.

[71]

Smith, C. A.,&Kirby, L. D. 2004. Appraisal as a pervasive determinant of anger. Emotion Washington, D.C., 4, 133-138. 15222849.

[72]

Smith, C. A.,&Scott, H. 1997. A componential approach to the meaning of facial expressions. In Russell, J.,&Fernandez-Dols, J. Eds., The psychology of facial expression pp. 229-254. Cambridge, UK: Cambridge University Press.

[73]

Valstar, M., Mehu, M., Pantic, M.,&Scherer, K. R. in press. Meta-analysis of the first facial expression recognition and analysis challenge. IEEE Transactions on Systems, Man, and Cybernetics. 21926026.

[74]

Wehrle, T. 1995. The Geneva Appraisal Theory Environment GATE. Unpublished computer software. Geneva, Switzerland: University of Geneva.

[75]

Wehrle, T.,&Scherer, K. R. 2001. Toward computational modelling of appraisal theories. In Scherer, K. R., Schorr, A.,&Johnstone, T. Eds., Appraisal processes in emotion: Theory, methods, research pp. 92-120. New York, NY: Oxford University Press.

[76]

Yeasin, M., Bullot, B.,&Sharma, R. 2006. Recognition of facial expressions and measurement of levels of interest from video. IEEE Transactions on Multimedia, 83, 500-507.

Digital Library

[77]

Zeng, Z., Pantic, M., Roisman, G. I.,&Huang, T. S. 2009. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 311, 39-58. 19029545.

Digital Library

Cited By

Zhang JHilpert BBroekens JJokinen J(2024)Simulating Emotions With an Integrated Computational Model of Appraisal and Reinforcement LearningProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641908(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641908
Seuss DHassan TDieckmann AUnfried MScherer KMortillaro MGarbas J(2023)Automatic Estimation of Action Unit Intensities and Inference of Emotional AppraisalsIEEE Transactions on Affective Computing10.1109/TAFFC.2021.307759014:2(1188-1200)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3077590
Larradet FNiewiadomski RBarresi GMattos LHarle RFarrahi KLane N(2019)Appraisal theory-based mobile app for physiological data collection and labelling in the wildAdjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers10.1145/3341162.3345595(752-756)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3341162.3345595
Show More Cited By

Advocating a Componential Appraisal Model to Guide Emotion Recognition
1. Computing methodologies

Recommendations

The Role of Empathic Traits in Emotion Recognition and Emotion Contagion of Cozmo Robots
HRI '22: Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction

In this online study, we investigated how well people could recognize emotions displayed by video recordings of a Cozmo robot, and the extent to which emotion recognition is shaped by individuals' empathic traits. We also explored whether participants ...
Emotion Recognition Using Physiological Signals
MIDI '15: Proceedings of the Mulitimedia, Interaction, Design and Innnovation

In this paper the problem of emotion recognition using physiological signals is presented. Firstly the problems with acquisition of physiological signals related to specific human emotions are described. It is not a trivial problem to elicit real ...
EmotiW 2016: video and group-level emotion recognition challenges
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

This paper discusses the baseline for the Emotion Recognition in the Wild (EmotiW) 2016 challenge. Continuing on the theme of automatic affect recognition `in the wild', the EmotiW challenge 2016 consists of two sub-challenges: an audio-video based ...

Comments

Information & Contributors

Information

Published In

cover image International Journal of Synthetic Emotions

International Journal of Synthetic Emotions Volume 3, Issue 1

January 2012

63 pages

ISSN:1947-9093

EISSN:1947-9107

Issue’s Table of Contents

Publisher

IGI Global

United States

Publication History

Published: 01 January 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang JHilpert BBroekens JJokinen J(2024)Simulating Emotions With an Integrated Computational Model of Appraisal and Reinforcement LearningProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641908(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641908
Seuss DHassan TDieckmann AUnfried MScherer KMortillaro MGarbas J(2023)Automatic Estimation of Action Unit Intensities and Inference of Emotional AppraisalsIEEE Transactions on Affective Computing10.1109/TAFFC.2021.307759014:2(1188-1200)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3077590
Larradet FNiewiadomski RBarresi GMattos LHarle RFarrahi KLane N(2019)Appraisal theory-based mobile app for physiological data collection and labelling in the wildAdjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers10.1145/3341162.3345595(752-756)Online publication date: 9-Sep-2019
https://dl.acm.org/doi/10.1145/3341162.3345595
Gebhard PSchneeberger TBaur TAndré EAndre EKoenig SDastani MSukthankar G(2018)MARSSIProceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3237383.3237458(497-506)Online publication date: 9-Jul-2018
https://dl.acm.org/doi/10.5555/3237383.3237458
Lingenfelser FWagner JDeng JBrueckner RSchuller BAndré E(2018)Asynchronous and Event-Based Fusion Systems for Affect Recognition on Naturalistic Data in Comparison to Conventional ApproachesIEEE Transactions on Affective Computing10.1109/TAFFC.2016.26351249:4(410-423)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1109/TAFFC.2016.2635124
Jenke RPeer A(2018)A cognitive architecture for modeling emotion dynamicsCognitive Systems Research10.1016/j.cogsys.2018.01.00449:C(128-141)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1016/j.cogsys.2018.01.004
Wang DHe TLi ZCao LDey NAshour ABalas VMccauley PLin YXu JShi F(2018)Image feature-based affective retrieval employing improved parameter and structure identification of adaptive neuro-fuzzy inference systemNeural Computing and Applications10.1007/s00521-016-2512-429:4(1087-1102)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1007/s00521-016-2512-4
Herremans DYang SChuan CBarthet MChew EFazekas GBarthet MStockman T(2017)IMMA-EmoProceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences10.1145/3123514.3123545(1-8)Online publication date: 23-Aug-2017
https://dl.acm.org/doi/10.1145/3123514.3123545
Trabelsi IBouhlel M(2015)Feature Selection for GUMI Kernel-Based SVM in Speech Emotion RecognitionInternational Journal of Synthetic Emotions10.4018/IJSE.20150701046:2(57-68)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.4018/IJSE.2015070104
Jain SAsawa K(2015)Emotional ComputerInternational Journal of Synthetic Emotions10.4018/IJSE.20150701036:2(35-56)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.4018/IJSE.2015070103
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents