My research interests lie in the general area of emotion and facial expressions. In particular, I am studying morphological and dynamic aspects of spontaneous facial expressions. Also, I am interested in empathy, facial feedback hypothesis and social aspects of facial expressions. Address: Japan
Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face... more Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face communications, typically indicating amusement but sometimes conveying negative emotions such as embarrassment and pain. Although previous studies have suggested that spatial and temporal properties could differ among these various types of smiles, no study has thoroughly analyzed these properties. This study aimed to clarify the spatiotemporal properties of smiles conveying amusement, embarrassment, and pain using a spontaneous facial behavior database. The results regarding spatial patterns revealed that pained smiles showed less eye constriction and more overall facial tension than amused smiles; no spatial differences were identified between embarrassed and amused smiles. Regarding temporal properties, embarrassed and pained smiles remained in a state of higher facial tension than amused smiles. Moreover, embarrassed smiles showed a more gradual change from tension states to the smi...
Automatic facial action detection is important, but no previous studies have evaluated pre-traine... more Automatic facial action detection is important, but no previous studies have evaluated pre-trained models on the accuracy of facial action detection as the angle of the face changes from frontal to profile. Using static facial images obtained at various angles (0°, 15°, 30°, and 45°), we investigated the performance of three automated facial action detection systems (FaceReader, OpenFace, and Py-feat). The overall performance was best for OpenFace, followed by FaceReader and Py-Feat. The performance of FaceReader significantly decreased at 45° compared to that at other angles, while the performance of Py-Feat did not differ among the four angles. The performance of OpenFace decreased as the target face turned sideways. Prediction accuracy and robustness to angle changes varied with the target facial components and action detection system.
In this paper, we introduce the free statistical software, JASP (JASP Team, 2016). For psychologi... more In this paper, we introduce the free statistical software, JASP (JASP Team, 2016). For psychologists who want to use statistical analysis, JASP has several advantages, one of which is that it has a graphical user interface that is easy to operate. Moreover, JASP makes Bayesian statistical analysis accessible. In this paper, we provide examples of how to use JASP using actual Big 5 data
Android robots capable of emotional interactions with humans have considerable potential for appl... more Android robots capable of emotional interactions with humans have considerable potential for application to research. While several studies developed androids that can exhibit human-like emotional facial expressions, few have empirically validated androids’ facial expressions. To investigate this issue, we developed an android head called Nikola based on human psychology and conducted three studies to test the validity of its facial expressions. In Study 1, Nikola produced single facial actions, which were evaluated in accordance with the Facial Action Coding System. The results showed that 17 action units were appropriately produced. In Study 2, Nikola produced the prototypical facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and naïve participants labeled photographs of the expressions. The recognition accuracy of all emotions was higher than chance level. In Study 3, Nikola produced dynamic facial expressions for six basic emotio...
Verbal sounds can be associated with specific meanings, a phenomenon called sound symbolism. Prev... more Verbal sounds can be associated with specific meanings, a phenomenon called sound symbolism. Previous findings of sound symbolism have shown that words including specific consonants or vowels or mouth shapes to pronounce specific vowels associate with specific and subjective physical and emotional evaluations. The purpose of this study was to examine whether each written vowel in a given language was individually associated with specific subjective evaluations. Six hundred and thirteen participants used five-point semantic differential scales for 10 features (size, closeness, thickness, width, weight, height, depth, affection, excitement and familiarity) to rate written Japanese vowels (a, i, u, e, and o). The results showed that the size, closeness, thickness and width of a, u and o were significantly higher than those of i and e, whereas the affection and familiarity of a were higher than the others. These results were consistent with previous findings in which vowels in sound-sym...
Vowels in words have been associated with specific meanings in sound symbolism (Hamano, 1998; New... more Vowels in words have been associated with specific meanings in sound symbolism (Hamano, 1998; Newman, 1933; Sapir, 1929). The purpose of this study was to examine whether each vowel individually involves physical and emotional meanings. Six-hundred and thirteen participants (482 females; M 16.97) rated 5-point semantic differential scales (size, distance, thickness, extent, weight, height, depth, preference, arousal, and familiarity) to presented Japanese vowels (a, i, u, e, and o). Results showed that the size, extent, and thickness of a, u, and o were significantly higher than i and e, whereas the preference and familiarity of a was higher than the others. These results were consistent with previous findings to which vowels in sound-symbolic words were associated with physical (i.e., size, extent, and thickness) and emotional (i.e., preference) evaluations. Our findings suggest that each vowel itself could individually contribute to specifically physical and emotional evaluations....
In the field of affective computing, achieving accurate automatic detection of facial movements i... more In the field of affective computing, achieving accurate automatic detection of facial movements is an important issue, and great progress has already been made. However, a systematic evaluation of systems that now have access to the dynamic facial database remains an unmet need. This study compared the performance of three systems (FaceReader, OpenFace, AFARtoolbox) that detect each facial movement corresponding to an action unit (AU) derived from the Facial Action Coding System. All machines could detect the presence of AUs from the dynamic facial database at a level above chance. Moreover, OpenFace and AFAR provided higher area under the receiver operating characteristic curve values compared to FaceReader. In addition, several confusion biases of facial components (e.g., AU12 and AU14) were observed to be related to each automated AU detection system and the static mode was superior to dynamic mode for analyzing the posed facial database. These findings demonstrate the features o...
Distal facial Electromyography (EMG) can be used to detect smiles and frowns with reasonable accu... more Distal facial Electromyography (EMG) can be used to detect smiles and frowns with reasonable accuracy. It capitalizes on volume conduction to detect relevant muscle activity, even when the electrodes are not placed directly on the source muscle. The main advantage of this method is to prevent occlusion and obstruction of the facial expression production, whilst allowing EMG measurements. However, measuring EMG distally entails that the exact source of the facial movement is unknown. We propose a novel method to estimate specific Facial Action Units (AUs) from distal facial EMG and Computer Vision (CV). This method is based on Independent Component Analysis (ICA), Non-Negative Matrix Factorization (NNMF), and sorting of the resulting components to determine which is the most likely to correspond to each CV-labeled action unit (AU). Performance on the detection of AU06 (Orbicularis Oculi) and AU12 (Zygomaticus Major) was estimated by calculating the agreement with Human Coders. The re...
The physical properties of genuine and deliberate facial expressions remain elusive. This study f... more The physical properties of genuine and deliberate facial expressions remain elusive. This study focuses on observable dynamic differences between genuine and deliberate expressions of surprise based on the temporal structure of facial parts during emotional expression. Facial expressions of surprise were elicited using multiple methods and video recorded: senders were filmed as they experienced genuine surprise in response to a jack-in-the-box (Genuine), other senders were asked to produce deliberate surprise with no preparation (Improvised), by mimicking the expression of another (External), or by reproducing the surprised face after having first experienced genuine surprise (Rehearsed). A total of 127 videos were analyzed, and moment-to-moment movements of eyelids and eyebrows were annotated with deep learning-based tracking software. Results showed that all surprise displays were mainly composed of raising eyebrows and eyelids movements. Genuine displays included horizontal movem...
Extant evidence points toward the role of contextual information and related cross-cultural varia... more Extant evidence points toward the role of contextual information and related cross-cultural variations in emotion perception, but most of the work to date has focused on judgments of basic emotions. The current research examines how culture and situational context affect the interpretation of emotion displays, i.e. judgments of the extent to which ambiguous smiles communicate happiness versus polite intentions. We hypothesized that smiles associated with contexts implying happiness would be judged as conveying more positive feelings compared to smiles paired with contexts implying politeness or smiles presented without context. In line with existing research on cross-cultural variation in contextual influences, we also expected these effects to be larger in Japan than in the UK. In Study 1, British participants viewed non-Duchenne smiles presented on their own or paired with background scenes implying happiness or the need to be polite. Compared to face-only stimuli, happy contexts ...
With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli h... more With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli have been developed over the past two decades. The present research aimed to test existing sets of dynamic facial expressions (published between 2000 and 2015) in a cross-corpus validation effort. For this, 14 dynamic databases were selected that featured facial expressions of the basic six emotions (anger, disgust, fear, happiness, sadness, surprise) in posed or spontaneous form. In Study 1, a subset of stimuli from each database (N = 162) were presented to human observers and machine analysis, yielding considerable variance in emotion recognition performance across the databases. Classification accuracy further varied with perceived intensity and naturalness of the displays, with posed expressions being judged more accurately and as intense, but less natural compared to spontaneous ones. Study 2 aimed for a full validation of the 14 databases by subjecting the entire stimulus set (N = 3...
Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face... more Smiles are universal but nuanced facial expressions that are most frequently used in face-to-face communications, typically indicating amusement but sometimes conveying negative emotions such as embarrassment and pain. Although previous studies have suggested that spatial and temporal properties could differ among these various types of smiles, no study has thoroughly analyzed these properties. This study aimed to clarify the spatiotemporal properties of smiles conveying amusement, embarrassment, and pain using a spontaneous facial behavior database. The results regarding spatial patterns revealed that pained smiles showed less eye constriction and more overall facial tension than amused smiles; no spatial differences were identified between embarrassed and amused smiles. Regarding temporal properties, embarrassed and pained smiles remained in a state of higher facial tension than amused smiles. Moreover, embarrassed smiles showed a more gradual change from tension states to the smi...
Automatic facial action detection is important, but no previous studies have evaluated pre-traine... more Automatic facial action detection is important, but no previous studies have evaluated pre-trained models on the accuracy of facial action detection as the angle of the face changes from frontal to profile. Using static facial images obtained at various angles (0°, 15°, 30°, and 45°), we investigated the performance of three automated facial action detection systems (FaceReader, OpenFace, and Py-feat). The overall performance was best for OpenFace, followed by FaceReader and Py-Feat. The performance of FaceReader significantly decreased at 45° compared to that at other angles, while the performance of Py-Feat did not differ among the four angles. The performance of OpenFace decreased as the target face turned sideways. Prediction accuracy and robustness to angle changes varied with the target facial components and action detection system.
In this paper, we introduce the free statistical software, JASP (JASP Team, 2016). For psychologi... more In this paper, we introduce the free statistical software, JASP (JASP Team, 2016). For psychologists who want to use statistical analysis, JASP has several advantages, one of which is that it has a graphical user interface that is easy to operate. Moreover, JASP makes Bayesian statistical analysis accessible. In this paper, we provide examples of how to use JASP using actual Big 5 data
Android robots capable of emotional interactions with humans have considerable potential for appl... more Android robots capable of emotional interactions with humans have considerable potential for application to research. While several studies developed androids that can exhibit human-like emotional facial expressions, few have empirically validated androids’ facial expressions. To investigate this issue, we developed an android head called Nikola based on human psychology and conducted three studies to test the validity of its facial expressions. In Study 1, Nikola produced single facial actions, which were evaluated in accordance with the Facial Action Coding System. The results showed that 17 action units were appropriately produced. In Study 2, Nikola produced the prototypical facial expressions for six basic emotions (anger, disgust, fear, happiness, sadness, and surprise), and naïve participants labeled photographs of the expressions. The recognition accuracy of all emotions was higher than chance level. In Study 3, Nikola produced dynamic facial expressions for six basic emotio...
Verbal sounds can be associated with specific meanings, a phenomenon called sound symbolism. Prev... more Verbal sounds can be associated with specific meanings, a phenomenon called sound symbolism. Previous findings of sound symbolism have shown that words including specific consonants or vowels or mouth shapes to pronounce specific vowels associate with specific and subjective physical and emotional evaluations. The purpose of this study was to examine whether each written vowel in a given language was individually associated with specific subjective evaluations. Six hundred and thirteen participants used five-point semantic differential scales for 10 features (size, closeness, thickness, width, weight, height, depth, affection, excitement and familiarity) to rate written Japanese vowels (a, i, u, e, and o). The results showed that the size, closeness, thickness and width of a, u and o were significantly higher than those of i and e, whereas the affection and familiarity of a were higher than the others. These results were consistent with previous findings in which vowels in sound-sym...
Vowels in words have been associated with specific meanings in sound symbolism (Hamano, 1998; New... more Vowels in words have been associated with specific meanings in sound symbolism (Hamano, 1998; Newman, 1933; Sapir, 1929). The purpose of this study was to examine whether each vowel individually involves physical and emotional meanings. Six-hundred and thirteen participants (482 females; M 16.97) rated 5-point semantic differential scales (size, distance, thickness, extent, weight, height, depth, preference, arousal, and familiarity) to presented Japanese vowels (a, i, u, e, and o). Results showed that the size, extent, and thickness of a, u, and o were significantly higher than i and e, whereas the preference and familiarity of a was higher than the others. These results were consistent with previous findings to which vowels in sound-symbolic words were associated with physical (i.e., size, extent, and thickness) and emotional (i.e., preference) evaluations. Our findings suggest that each vowel itself could individually contribute to specifically physical and emotional evaluations....
In the field of affective computing, achieving accurate automatic detection of facial movements i... more In the field of affective computing, achieving accurate automatic detection of facial movements is an important issue, and great progress has already been made. However, a systematic evaluation of systems that now have access to the dynamic facial database remains an unmet need. This study compared the performance of three systems (FaceReader, OpenFace, AFARtoolbox) that detect each facial movement corresponding to an action unit (AU) derived from the Facial Action Coding System. All machines could detect the presence of AUs from the dynamic facial database at a level above chance. Moreover, OpenFace and AFAR provided higher area under the receiver operating characteristic curve values compared to FaceReader. In addition, several confusion biases of facial components (e.g., AU12 and AU14) were observed to be related to each automated AU detection system and the static mode was superior to dynamic mode for analyzing the posed facial database. These findings demonstrate the features o...
Distal facial Electromyography (EMG) can be used to detect smiles and frowns with reasonable accu... more Distal facial Electromyography (EMG) can be used to detect smiles and frowns with reasonable accuracy. It capitalizes on volume conduction to detect relevant muscle activity, even when the electrodes are not placed directly on the source muscle. The main advantage of this method is to prevent occlusion and obstruction of the facial expression production, whilst allowing EMG measurements. However, measuring EMG distally entails that the exact source of the facial movement is unknown. We propose a novel method to estimate specific Facial Action Units (AUs) from distal facial EMG and Computer Vision (CV). This method is based on Independent Component Analysis (ICA), Non-Negative Matrix Factorization (NNMF), and sorting of the resulting components to determine which is the most likely to correspond to each CV-labeled action unit (AU). Performance on the detection of AU06 (Orbicularis Oculi) and AU12 (Zygomaticus Major) was estimated by calculating the agreement with Human Coders. The re...
The physical properties of genuine and deliberate facial expressions remain elusive. This study f... more The physical properties of genuine and deliberate facial expressions remain elusive. This study focuses on observable dynamic differences between genuine and deliberate expressions of surprise based on the temporal structure of facial parts during emotional expression. Facial expressions of surprise were elicited using multiple methods and video recorded: senders were filmed as they experienced genuine surprise in response to a jack-in-the-box (Genuine), other senders were asked to produce deliberate surprise with no preparation (Improvised), by mimicking the expression of another (External), or by reproducing the surprised face after having first experienced genuine surprise (Rehearsed). A total of 127 videos were analyzed, and moment-to-moment movements of eyelids and eyebrows were annotated with deep learning-based tracking software. Results showed that all surprise displays were mainly composed of raising eyebrows and eyelids movements. Genuine displays included horizontal movem...
Extant evidence points toward the role of contextual information and related cross-cultural varia... more Extant evidence points toward the role of contextual information and related cross-cultural variations in emotion perception, but most of the work to date has focused on judgments of basic emotions. The current research examines how culture and situational context affect the interpretation of emotion displays, i.e. judgments of the extent to which ambiguous smiles communicate happiness versus polite intentions. We hypothesized that smiles associated with contexts implying happiness would be judged as conveying more positive feelings compared to smiles paired with contexts implying politeness or smiles presented without context. In line with existing research on cross-cultural variation in contextual influences, we also expected these effects to be larger in Japan than in the UK. In Study 1, British participants viewed non-Duchenne smiles presented on their own or paired with background scenes implying happiness or the need to be polite. Compared to face-only stimuli, happy contexts ...
With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli h... more With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli have been developed over the past two decades. The present research aimed to test existing sets of dynamic facial expressions (published between 2000 and 2015) in a cross-corpus validation effort. For this, 14 dynamic databases were selected that featured facial expressions of the basic six emotions (anger, disgust, fear, happiness, sadness, surprise) in posed or spontaneous form. In Study 1, a subset of stimuli from each database (N = 162) were presented to human observers and machine analysis, yielding considerable variance in emotion recognition performance across the databases. Classification accuracy further varied with perceived intensity and naturalness of the displays, with posed expressions being judged more accurately and as intense, but less natural compared to spontaneous ones. Study 2 aimed for a full validation of the 14 databases by subjecting the entire stimulus set (N = 3...
Uploads