The Journal of neuroscience : the official journal of the Society for Neuroscience, Jan 15, 2012
Highly emotional events are associated with vivid "flashbulb" memories. Here we examine... more Highly emotional events are associated with vivid "flashbulb" memories. Here we examine whether the flashbulb metaphor characterizes a previously unknown emotion-enhanced vividness (EEV) during initial perceptual experience. Using a magnitude estimation procedure, human observers estimated the relative magnitude of visual noise overlaid on scenes. After controlling for computational metrics of objective visual salience, emotional salience was associated with decreased noise, or heightened perceptual vividness, demonstrating EEV, which predicted later memory vividness. Event-related potentials revealed a posterior P2 component at ∼200 ms that was associated with both increased emotional salience and decreased objective noise levels, consistent with EEV. Blood oxygenation level-dependent response in the lateral occipital complex (LOC), insula, and amygdala predicted online EEV. The LOC and insula represented complimentary influences on EEV, with the amygdala statistically me...
It is well-known that emotionally salient events are remembered more vividly than mundane ones. O... more It is well-known that emotionally salient events are remembered more vividly than mundane ones. Our recent research has demonstrated that such memory vividness (Mviv) is due in part to the subjective experience of emotional events as more perceptually vivid, an effect we call emotionally enhanced vividness (EEV). The present study built on previously reported research in which fMRI data were collected while participants rated relative levels of visual noise overlaid on emotionally salient and neutral images. Ratings of greater EEV were associated with greater activation in the amygdala and visual cortex. In the present study, we measured BOLD activation that predicted recognition Mviv for these same images 1 week later. Results showed that, after controlling for differences between scenes in low-level objective features, hippocampus activation uniquely predicted subsequent Mviv. In contrast, amygdala and visual cortex regions that were sensitive to EEV were also modulated by subsequ...
For learning meaningful representations of data, a rich source of prior knowledge may come in the... more For learning meaningful representations of data, a rich source of prior knowledge may come in the form of a generative black box, e.g. a graphics program that generates realistic facial images. We consider the problem of learning the inverse of a given generative model from data. The problem is non-trivial because it is difficult to create labelled training cases by hand, and the generative mapping is a black box in the sense that there is no analytic expression for its gradient. We describe a way of training a feedforward neural network that starts with just one labelled training example and uses the generative black box to "breed" more training data. As learning proceeds, the training set evolves and the labels that the network assigns to unlabelled training data converge to their correct values. We demonstrate our approach by learning to invert a generative model of eyes and an active appearance model of faces.
ABSTRACT Motor production may play an important role in learning to recognize facial expressions.... more ABSTRACT Motor production may play an important role in learning to recognize facial expressions. The present study explores the influence of facial production training on the perception of facial expressions by employing a novel production training intervention built on feedback from automated facial expression recognition. We hypothesized that production training using the automated feedback system would improve an individual’s ability to identify dynamic emotional faces. Thirty-four participants were administered a dynamic expression recognition task before and after either interacting with a production training video game called the Emotion Mirror or playing a control video game. Consistent with the prediction that perceptual benefits are tied to expression production, individuals with high engagement in production training improved more than individuals with low engagement or individuals who did not receive production training. These results suggest that the visual-motor associations involved in expression production training are related to perceptual abilities. Additionally, this study demonstrates a novel application of computer vision for real-time facial expression intervention training.
Current theories of emotion perception posit that basic facial expressions signal categorically d... more Current theories of emotion perception posit that basic facial expressions signal categorically discrete emotions or affective dimensions of valence and arousal. In both cases, the information is thought to be directly ''read out'' from the face in a way that is largely immune to context. In contrast, the three studies reported here demonstrated that identical facial configurations convey strikingly different emotions and dimensional values depending on the affective context in which they are embedded. This effect is modulated by the similarity between the target facial expression and the facial expression typically associated with the context. Moreover, by monitoring eye movements, we demonstrated that characteristic fixation patterns previously thought to be determined solely by the facial expression are systematically modulated by emotional context already at very early stages of visual processing, even by the first time the face is fixated. Our results indicate that the perception of basic facial expressions is not context invariant and can be categorically altered by context at early perceptual levels.
According to evolutionary accounts, emotions originated to prepare an organism for action . To in... more According to evolutionary accounts, emotions originated to prepare an organism for action . To investigate this putative relationship between emotion and action we examined the effect of an emotional stimulus on oculomotor actions controlled by the superior colliculus (SC), which has connections with subcortical structures involved in the perceptual prioritization of emotion, such as the amygdala through the pulvinar. The pulvinar connects the amygdala to cells in the SC responsible for the speed of saccade execution, while not affecting the spatial component of the saccade. We tested the effect of emotion on both temporal and spatial signatures of oculomotor functioning using a gap--distractor paradigm. Changes in spatial programming were examined through saccadic curvature in response to a remote distractor stimulus, while changes in temporal execution were examined using a fixation gap manipulation. We show that following the presentation of a task irrelevant fearful face the temporal but not the spatial component of the saccade generation system was affected.
Automatic facial expression recognition (AFER) has undergone substantial advancement over the pas... more Automatic facial expression recognition (AFER) has undergone substantial advancement over the past two decades. This work explores the application of bag of words (BoW), a highly matured approach for object and scene recognition to AFER. We proceed by first highlighting the reasons that makes the task for BoW differ for AFER compared to object and scene recognition. We propose suitable extensions to BoW architecture for the AFER's task. These extensions are able to address some of the limitations of current state of the art appearance-based approaches to AFER. Our BoW architecture is based on the spatial pyramid framework, augmented by multiscale dense SIFT features, and a recently proposed approach for object classification: locality-constrained linear coding and max-pooling. Combining these, we are able to achieve a powerful facial representation that works well even with linear classifiers. We show that a well designed BoW architecture can provide a performance benefit for AFER, and elements of the proposed BoW architecture are empirically evaluated. The proposed BoW approach supersedes previous state of the art results by achieving an average recognition rate of 96% on AFER for two public datasets.
The Journal of neuroscience : the official journal of the Society for Neuroscience, Jan 15, 2012
Highly emotional events are associated with vivid "flashbulb" memories. Here we examine... more Highly emotional events are associated with vivid "flashbulb" memories. Here we examine whether the flashbulb metaphor characterizes a previously unknown emotion-enhanced vividness (EEV) during initial perceptual experience. Using a magnitude estimation procedure, human observers estimated the relative magnitude of visual noise overlaid on scenes. After controlling for computational metrics of objective visual salience, emotional salience was associated with decreased noise, or heightened perceptual vividness, demonstrating EEV, which predicted later memory vividness. Event-related potentials revealed a posterior P2 component at ∼200 ms that was associated with both increased emotional salience and decreased objective noise levels, consistent with EEV. Blood oxygenation level-dependent response in the lateral occipital complex (LOC), insula, and amygdala predicted online EEV. The LOC and insula represented complimentary influences on EEV, with the amygdala statistically me...
It is well-known that emotionally salient events are remembered more vividly than mundane ones. O... more It is well-known that emotionally salient events are remembered more vividly than mundane ones. Our recent research has demonstrated that such memory vividness (Mviv) is due in part to the subjective experience of emotional events as more perceptually vivid, an effect we call emotionally enhanced vividness (EEV). The present study built on previously reported research in which fMRI data were collected while participants rated relative levels of visual noise overlaid on emotionally salient and neutral images. Ratings of greater EEV were associated with greater activation in the amygdala and visual cortex. In the present study, we measured BOLD activation that predicted recognition Mviv for these same images 1 week later. Results showed that, after controlling for differences between scenes in low-level objective features, hippocampus activation uniquely predicted subsequent Mviv. In contrast, amygdala and visual cortex regions that were sensitive to EEV were also modulated by subsequ...
For learning meaningful representations of data, a rich source of prior knowledge may come in the... more For learning meaningful representations of data, a rich source of prior knowledge may come in the form of a generative black box, e.g. a graphics program that generates realistic facial images. We consider the problem of learning the inverse of a given generative model from data. The problem is non-trivial because it is difficult to create labelled training cases by hand, and the generative mapping is a black box in the sense that there is no analytic expression for its gradient. We describe a way of training a feedforward neural network that starts with just one labelled training example and uses the generative black box to "breed" more training data. As learning proceeds, the training set evolves and the labels that the network assigns to unlabelled training data converge to their correct values. We demonstrate our approach by learning to invert a generative model of eyes and an active appearance model of faces.
ABSTRACT Motor production may play an important role in learning to recognize facial expressions.... more ABSTRACT Motor production may play an important role in learning to recognize facial expressions. The present study explores the influence of facial production training on the perception of facial expressions by employing a novel production training intervention built on feedback from automated facial expression recognition. We hypothesized that production training using the automated feedback system would improve an individual’s ability to identify dynamic emotional faces. Thirty-four participants were administered a dynamic expression recognition task before and after either interacting with a production training video game called the Emotion Mirror or playing a control video game. Consistent with the prediction that perceptual benefits are tied to expression production, individuals with high engagement in production training improved more than individuals with low engagement or individuals who did not receive production training. These results suggest that the visual-motor associations involved in expression production training are related to perceptual abilities. Additionally, this study demonstrates a novel application of computer vision for real-time facial expression intervention training.
Current theories of emotion perception posit that basic facial expressions signal categorically d... more Current theories of emotion perception posit that basic facial expressions signal categorically discrete emotions or affective dimensions of valence and arousal. In both cases, the information is thought to be directly ''read out'' from the face in a way that is largely immune to context. In contrast, the three studies reported here demonstrated that identical facial configurations convey strikingly different emotions and dimensional values depending on the affective context in which they are embedded. This effect is modulated by the similarity between the target facial expression and the facial expression typically associated with the context. Moreover, by monitoring eye movements, we demonstrated that characteristic fixation patterns previously thought to be determined solely by the facial expression are systematically modulated by emotional context already at very early stages of visual processing, even by the first time the face is fixated. Our results indicate that the perception of basic facial expressions is not context invariant and can be categorically altered by context at early perceptual levels.
According to evolutionary accounts, emotions originated to prepare an organism for action . To in... more According to evolutionary accounts, emotions originated to prepare an organism for action . To investigate this putative relationship between emotion and action we examined the effect of an emotional stimulus on oculomotor actions controlled by the superior colliculus (SC), which has connections with subcortical structures involved in the perceptual prioritization of emotion, such as the amygdala through the pulvinar. The pulvinar connects the amygdala to cells in the SC responsible for the speed of saccade execution, while not affecting the spatial component of the saccade. We tested the effect of emotion on both temporal and spatial signatures of oculomotor functioning using a gap--distractor paradigm. Changes in spatial programming were examined through saccadic curvature in response to a remote distractor stimulus, while changes in temporal execution were examined using a fixation gap manipulation. We show that following the presentation of a task irrelevant fearful face the temporal but not the spatial component of the saccade generation system was affected.
Automatic facial expression recognition (AFER) has undergone substantial advancement over the pas... more Automatic facial expression recognition (AFER) has undergone substantial advancement over the past two decades. This work explores the application of bag of words (BoW), a highly matured approach for object and scene recognition to AFER. We proceed by first highlighting the reasons that makes the task for BoW differ for AFER compared to object and scene recognition. We propose suitable extensions to BoW architecture for the AFER's task. These extensions are able to address some of the limitations of current state of the art appearance-based approaches to AFER. Our BoW architecture is based on the spatial pyramid framework, augmented by multiscale dense SIFT features, and a recently proposed approach for object classification: locality-constrained linear coding and max-pooling. Combining these, we are able to achieve a powerful facial representation that works well even with linear classifiers. We show that a well designed BoW architecture can provide a performance benefit for AFER, and elements of the proposed BoW architecture are empirically evaluated. The proposed BoW approach supersedes previous state of the art results by achieving an average recognition rate of 96% on AFER for two public datasets.
Uploads
Papers by Josh Susskind