Keywords

1 Introduction

Research on cross-modal correspondence between sound and color has a long history in the field of experimental psychology, dating back to the discovery of idiosyncratic sound-color mapping in synesthetes. And synesthetic correspondences between color and auditory stimuli in synesthetes were well-established in the literature [1,2,3,4]. Early cross-modal matching studies, as well as those studies involving the matching of sound with color, suggested that people make reliable associations between certain dimensions of color and sounds. Marks [5, 6] studied on the association of pitch, loudness and lightness, he confirmed that higher pitch and louder sound were associated with lighter color. Stevens [7] and Bonds [8] studied with grey color with sample waveform sounds and demonstrated that people matched brightness and loudness cross-modally, both of children and adults matched light grey color with louder sound and darker grey color with quieter sounds, and most of people consider that the combination of sound and color have a more significant effect on emotional intervention. For example, bright colors and fast rhythm music combinations are exciting and energetic.

Judging and shaping human emotions is not an easy task. Davidson et al. described the two-dimensional model of emotion [9]. From this model we can see that the emotions are determined by their position in the two-dimensional scale, and the two-dimensional scale is spanned by two axis that valence in horizontal axis and arousal in vertical axis. Valence expresses the quality of the unconscious emotion from unpleasant to pleasant. Arousal refers to the level of quantitative activation from calm to excited. The different emotional tags can be drawn on the 2D plane spanned by these two axis across each position, such as can be seen in Fig. 1.

Fig. 1.
figure 1

Graphical representation of the emotion classification model. The emotions are determined by their position in the two-dimensional scale, and the two-dimensional scale is spanned by two axis that valence in horizontal axis and arousal in vertical axis.

Full size image

Researchers usually use different approaches to simulate emotions, such as texts, music, and films, facial expressions, slides, photos, the characteristics of dynamic vision and auditory stimulation make the film seem to be one of the most effective ways to induce emotions [10,11,12,13,14]. Electroencephalogram is a non-stationary time series bio-medical signal that provides information about human brain activities. Compare to the facial expression, human behavior, vocal expression and some physiological signals, EEG is a more reliable approach that could reduce masked emotions. A lot of studies have researched on the question of asymmetrical activation of the cerebral hemisphere. Davidson [15] detected that left frontal EEG activity is related to positive emotion, while right frontal is related to negative. In the EEG study on the reward and punishment of different brain functions, Henriques [16] found a pattern of relative right frontal EEG activity on depressed adults.

2 Materials and Methods

Aim to stimulate subject’s emotions, a group of films clips were used as elicitors, the film clips set includes two clips for each of four target emotional states: joy, fear, sad and relaxation emotions. The selection criteria for film clips were as follows: (a) the length of the scene should be relatively short (3–5 min); (b) the scene is to be understood without explanation; and (c) the scene should induce single desired target emotion of subjects.

To intervene the emotional stimulation of the film, we used a group of music clips and ambient light as intervention factors. Music clips in accordance with the combination of the pitch level and rhythm of speed to get two different type of music including high-fast (HF), low-slow (LS), and select the typical music clips. Ambient light color selection of the most typical red, yellow, blue and green four colors. The selection criteria for music clips were as follows: (a) the length of the music should be relatively short (3–5 min); (b) the music is as unfamiliar as possible, such process as seen in Fig. 2.

Fig. 2.
figure 2

Intervention experiment processing (Color figure online)

Full size image

3 Experiment

3.1 Participants

We recruited a homogeneous population of 10 healthy subjects between 17 and 25 years of age (mean = 20.0 ± 1.7). Some subjects were students of Xi’an jiaotong University and others is Xi’an Gaoxin No.1 Hign School. All participants have normal vision and sound hearing.

3.2 Procedure

The experimental process is divided into two phases including emotional induction and emotional intervention. The emotional induction stage is mainly stimulated by the film segment in a white ambient light environment. The emotional intervention stage carries out emotional adjustment through four different types of music in an ambient light environment, such as seen in Fig. 3. The entire experimental process in a closed and opaque environment, the subjects must wear the EEG cap with EEG recording.

Fig. 3.
figure 3

Experimental environment. During the experiment, using four different films to induce four corresponding emotions. Then light and music work together on the subjects to intervene emotion.

Full size image

3.3 Statistical Analysis Methods

In EEG recording, EPOC device developed by EMOTIV is used to measure EEG activity. The EPOC uses a dry-type sensor and covers 10ch electrodes, such as seen in Fig. 4. This device has high resolution, neuro-signal acquisition, and a processing wireless neuro-headset. The EEG data are sent to a computer through a serial port, and the sampling rate is 128 Hz.

Fig. 4.
figure 4

Electrode position distribution

Full size image

In EEG feature extraction, The EEG data we got from the experiments were analyzed through several procedures, including filtering, independent component analysis (ICA), fast fourier transform (FFT) and wavelet transform etc. Firstly we filtered to eliminate the presence of artifacts and linear trend items, and then carry out independent component analysis to eliminate the eye-electric artifact, and finally we use fast Fourier transform (FFT) to extract domain-frequency characteristics to get power spectra of EEG data of alpha, beta, theta .thus we use the average power spectra of EEG data and the ratio of the power spectral density of the symmetrical electrode in the left and right hemisphere as signal characteristics to represents the change of arousal and valence to evaluate the intervention effect of music and ambient light on the emotional state.

Therefore, we selected the electrodes of AF3, F3, F7, FC5 which in the left frontal area and AF4, F4, F8, FC6 which in the symmetric area to get the average power spectral of alpha band. The average power spectra and the ratio of average power spectra in the left and right hemisphere were calculated to confirm whether the emotion was to be elicited and intervened or not. When the subjects is in a high degree of valance, the ratio of average power spectral in alpha band of the symmetrical electrode decreases. While in a low degree, the ratio increases, such as formula 1.

$$ {\text{x}}_{\text{valance}} = \frac{{\alpha_{{{\text{average}}\;{\text{power}}\;{\text{spectra}}\;{\text{in}}\;{\text{left}}\,{\text{brain}}}} }}{{\alpha_{{{\text{average}}\;{\text{power}}\;{\text{spectra}}\;{\text{in}}\;{\text{right}}\,{\text{brain}}}} }} $$
(1)

4 Results

4.1 Emotion Elicitation

Under the stimulus of the horror movie, the ratio of average power spectra in the left and right hemisphere significantly increases,as shown in Fig. 5(a). For the sadness film, the ratio of average power spectra in the left and right hemisphere significantly increase,as shown in Fig. 5(b). Stimulated by comedy, the ratio significantly decreases, as shown in Fig. 5(c). The ration significantly decreases as stimulated by relaxing film, which is shown in Fig. 5(d).

Fig. 5.
figure 5

The ratio of average power spectra in the left and right hemisphere. Four figures from (a) to (d) present the change of the ratio of average power spectra in the left and right hemisphere with fear, sadness, joy and relaxation emotions. The horizontal axis represents the electrodes, and the vertical axis represents the ratio of average power spectra of alpha band between relaxation state and simulation state.

Full size image

According to the formula 1, subjects’ emotional valence degrees decreased with the stimulation of horror and sadness movies. While stimulated by comedy or relaxing films, subjects’ emotional valence degrees increased, which shows that emotions with fear and sadness need environmental intervention urgently. So, we’re going to focus on the intervention effect of these two negative emotions.

4.2 Emotion Intervention

To intervene the fear and sadness emotions with sound and ambient light according to the cross-modal correspondence between color and auditory. A group of music clips and ambient lights were used as intervention factors. Music clips were divided into two groups in accordance with pitch and rhythm, namely, high-fast (HF), low-slow (LS). Red, yellow, blue and green were selected as the typical ambient light colors. The average power spectra of alpha band were collected from the right and left brain areas, and the ratio of average power spectra were calculated to evaluate the intervention effect of sound and light, as shown in Fig. 6.

Fig. 6.
figure 6

The ratio of average power spectra of the alpha band in AF3/AF4 with different ambient light and music. Four figures from (a) to (d) present the average power spectra of the alpha band in AF3/AF4 under different ambient light and music stimuli. The horizontal axis represents the time, and the vertical axis represents the change of EEG data under different ambient light and music stimuli. Figure (a)–(b) indicate the EEG data collected in fear under HF and LS music, while Figure (c) and (d) indicate the EEG data in sadness under HF and LS music. (Color figure online)

Full size image

Result from the figures, When the subject stayed in fear, red, green and white lights had a big fluctuation value, which indicated that the instantaneous interference effect was obvious, and with the passage of time, the intervention effects of music began to appear, such as HF music and red light combination could gradually make the ratio of average power spectra of the alpha band in AF3/AF4 initially raise then continue to decline, which suggested that HF music and red light combination had a positive intervention effect in valance. While under LS music, white, yellow, and blue light had a positive intervention effect in valance with the change of time.

In the emotion of sadness, subjects of HF group under green and yellow and red ambient lights showed a significant positive intervention effect in valance, the ratio of average power spectra of the alpha band in AF3/AF4 had a tend of decline generally, which indicated that the sadness was significantly relieved. The condition of LS group is similar to the HF group.

Emotional intervention study found that the effect of music on emotional intervention plays the dominant role, followed by color. For negative emotions, high-pitched and fast-paced music functions well in emotional soothing. Summarized the result of the experiment, we can draw on a typical soothing combination of music and colour for negative emotions. Shown as Table 1.

Table 1. Combination of music and light.
Full size table

5 Conclusion

In this study, experiments were carried out to investigate the effect of visual and audio stimuli on human emotions. For negative emotions, high-pitched and fast-paced music functions well in emotional soothing and synesthetic combination of music and color have a more significant effect on emotion intervention. In addition to the valance, there are more studies on arousal. In the fatigue state, the study on arousal is particularly important.

Cross-modal correspondence of music and color not only can be widely used in public environment, such as hospitals, where can calm patients by playing peaceful music, with harmony environment, also can be used in daily work to intervene people’s negative emotions and to reduce adverse consequences or tragedies. Such as the concept design of car exhibited by Toyota in 2017 - Concept-i. which can perceive driver’s emotions, then adjust the color of light and play soothing music. It makes the driving full of fun and reduce the chance of accidents, and truly realize intelligent human-computer interaction.