1. Introduction
Within the brain, impetus inclinations, behavioral reactions, physiological stimulation, states of mind and cognitive procedures are all directly conveyed through emotion. Brain activity and neural pathways are interrelated in a manner that influences mathematical, verbal, perceptive and other forms of intelligence, which further shape emotions [
1,
2]. From a particular response of the body to an instinctive reaction, individual emotional reactions can vary [
3]. Accordingly, the possible extent of congruence between socio-affective circumstances and particular brain areas has been investigated through applying an array of simulation methods in a substantial number of studies [
4,
5].
The dimensional and discrete models are the two affective models that may be adopted to determine and categorize affective states in accordance with a psychological perspective. Key emotions are identified in the discrete model which specific, distinct affective states are connected to; fundamental emotions such as happiness, sadness, surprise, disgust, fear and anger are individually or in some mixture deemed to be responsible for any further emotions [
6,
7]. Adopting a circumplex emotion model, the dimensional model has been pervasively adopted for affective identification application mapping, as a two-dimensional (2D) cognitive-emotional state theory [
8,
9]. A two-scale valence-arousal graph is used for conveying emotions, with emotional strength between calm to excited presented on the vertical axis as ‘arousal’, while the unpleasant to pleasant degree of emotion is conveyed on the horizontal axis as ‘valence’ [
8,
9,
10]. Furthermore, quartiles Q1 to Q4 are the four principal sites of emotional states, with low arousal–high valence (LAHV), low arousal–low valence (LALV), high arousal–low valence (HALV) and high arousal–high valence (HAHV) presented in Q4, Q3, Q2 and Q1 respectively [
11]. Beyond the 2D model, focus-disinterest characteristics have been incorporated into a three-dimensional (3D) cognitive-emotional state model in certain studies [
12].
Various emotional states have been produced through the adoption of varied techniques in studies. Consequently, audio-visual, auditory or visual stimuli have all been adopted in different instances. As in brain-computer interfacing (BCI) research, diminishing or expanding the sensorimotor rhythm amplitude is the process for auditory and visual stimuli [
13]. Additionally, compared to music-based audio stimuli, brain signals more straightforwardly convey picture-based visual stimuli [
14]. So as to provoke a particular affective state in the most effective manner, auditory and visual stimulus’ amalgamated impact has been acknowledged in studies, thus establishing the optimal context for affective identification. Regarding automatic emotion identification, audio-visual stimuli have also been applied [
4]. In contrast with alternative stimuli methods, audio-visual production of emotional states has been found to be superior and more pervasively adopted [
13,
15,
16]. Therefore, brief audio-visual film excerpts were adopted to elicit emotion in this research.
Emotional changes would be elicited using different physiological signals such as galvanic skin response (GSR) [
15], electrodermal activity (EDA) [
17], blood volume pressure (BVP) [
18], and skin temperature (ST) [
19], evoked potentials (EP) [
20], electrocardiogram (ECG) [
21], electromyogram (EMG) [
22], and electroencephalogram (EEG) [
23,
24,
25,
26,
27,
28,
29,
30]. Clinically, EEG signals have been widely used as useful indicators of different mental states such as epilepsy, Alzheimer’s disease (AD) and vascular dementia (VaD) [
31,
32,
33,
34,
35].
As the brain is a complex structure that has a dynamic behavior, electrical activities including emotional states, can be reflected by using EEG. EEG is a neurophysiological tool used to monitor and identify brain changes [
36]. EEG is a widely available, cost-effective, and non-invasive tool that tracks information processing with milliseconds precision and high temporal resolution [
36,
37]. A typical clinical EEG frequency ranges from 0.01 Hz to approximately 70 Hz [
38]; the corresponding waveforms have an amplitude of a few µVolt to approximately 100 µVolt [
39]. EEG background waveforms also convey valuable information. Thus, these waveforms can be classified into five specific frequency power bands: the delta band (
), the theta band (
), the alpha band (
), the beta band (
), and the gamma band (
) [
40,
41]. Studies on EEG signal processing have been conducted to identify the brain activity patterns involved in cognitive science, neuropsychological research, clinical assessments, and consciousness research [
42,
43,
44,
45,
46,
47]. Recently, EEG has been widely used to assess and evaluate the human emotional states with excellent time resolution [
3,
15,
28,
29,
30,
48]. EEG can provide useful information of emotional states that have been described as a potential biomarker to evaluate different emotional responses from multi-channel EEG datasets over the brain regions [
38]. A key advantage of the multi-channel EEG signal processing is to interpret EEG changes during different emotional states over the brain regions. Thus, numerous studies have been performed including this study to deal with issue [
49,
50,
51]. For instance, Nattapong et al. have proposed a continuous music-emotion recognition approach based on brain wave signals [
52]. Olga et al. have applied a combined music therapy process with the real-time EEG-based human emotion recognition algorithm to identify the current emotional state based on neurofeedback and adjust the music therapy based on the patient’s needs [
53].
As the brain neurons are controlled by linear and non-linear phenomena, several linear techniques including traditional spectral powers have been used to analyze the smoothness of the EEG as a time series and to elicit the emotional information from the EEG signals. The higher order statistics (HOS) features, namely skewness and kurtosis, have been applied as well to measure the presence of transients in the signal [
52,
54,
55,
56]. Due to the capability of the brain to perform sophisticated emotional tasks and to investigate the complex dynamic information that is reflected from the brain cortex, several researchers have used non-linear methods for automatic detection of emotions through EEG signals [
57]. Previous emotion studies have used a small number of features, mostly relative powers [
3], Hurst [
15], Hjorth parameters [
58], Fractal Dimension (FD) [
59], and statistical features [
60,
61]. Moreover, entropy has been considered as the most prevalent methods to evaluate the presence or absence of long-range dependence on physiological signal analysis including approximate entropy (
), sample entropy (
) and permutation entropy (
) which are relatively robust to noise and powerful enough to quantify the complexity of a time series [
62]. Amplitude-aware permutation entropy (
) has demonstrated efficiency in discriminating between calmness and distress [
63,
64,
65]. Fuzzy entropy (
) was proposed for EEG analysis in [
66,
67].
is slightly faster than
, however the latter is more consistent and less dependent on the data length [
68]. Azami et al. has considered the advantages of
over
and recently has introduced refined composite multiscale
(
) [
68,
69,
70]. In
, the entropy stability is improved, the signals’ length sensitivity is reduced and the coarse-grained process of
smoothens the signals. Hence,
has been used in this study.
EEG signal contains useful information on physiological states of the brain and has proven to be a potential biomarker to realize the linear and non-linear behavior of the brain [
31,
32,
71,
72,
73]. Therefore, motivation of this work is twofold. First, in order to investigate alternate information from multi-channel emotional EEG datasets, linear spectral conventional analysis, non-linear entropy method and temporal feature were performed to obtain the potential EEG emotional biomarkers. Second, the obtained biomarkers may be further considered to provide additional information to illustrate the EEG spectro-spatial
, entropy-spatial
and temporo-spatial
profiles for the seven emotional states over the brain regions.
The preprocessing stage has been used to limit the unwanted frequency from the EEG dataset. Spectral biomarkers were computed by employing the absolute powers
of
,
,
,
, and
. Moreover, to quantify the complexity of brain functions, entropy biomarkers across multi-channel EEG signals have been measured using
. Furthermore, the temporal biomarker was reported by amplitude envelope which was extracted using Hilbert transform, and the amplitude values were investigated by skewness (
) to get
. Three stages of two-way analysis of variance (ANOVA) were conducted to obtain the spectral, entropy and temporal biomarkers followed by Pearson’s correlations to get the spatial profiles that are related to anger, anxiety, disgust, happiness, sadness, surprise and neutral emotional states over the brain regions. The valance-arousal circumplex model was employed in this study to represent and recognize human emotions due to its effectiveness in viewing the emotions while audio-visual video clips were used [
74]. To the author’s best knowledge this study has two contributions: firstly, it is the first use of a combination of certain features to develop spectral, entropy and temporal biomarkers towards
,
and
profile identification for the seven emotional states over the brain regions; secondly, the EEG elicitation protocol and EEG measurement procedure have never been used before for emotion data acquisition.
2. Materials and Methods
This study is intended to be focused on the potential EEG emotional biomarkers and profiles that obtained from EEG datasets.
Figure 1 illustrates the block diagram of the proposed study.
2.1. EEG Acquisition and Recording
A transportable Emotiv EPOC EEG 14-channel headset (Emotive Systems, Inc., San Francisco, CA, USA) was adopted in order to evaluate 14 EEG electrodes (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4) overall with 2 ground electrodes which were provided by the driven right leg (DRL) mastoid and common mode sense (CMS) left mastoid. The Emotiv EPOC EEG uses sponge-based electrodes which were located based on the 10–20 system. The electrode information was filtered through a 0.5–70 Hz band-pass filter. A 128 Hz sampling frequency was used with a resolution of 0.51 mV.
40 university students agreed to participate in this research (
Table 1). A subject appraisal was carried out for each individual to guarantee that no previous psychiatric or neurological problems had been suffered, with the participant then providing their agreement to participation through signing an informed consent document, before the study proceeded. Subjects were presented with different brief film excerpts alongside audio that aimed to be emotionally engaging, after which a self-assessment questionnaire (SAQ) was filled in by the subjects to provide their assessment and scoring of emotional reactions to the excerpts. The subsequent video excerpt was presented after a 45 s pause. This process is presented in
Figure 2 [
75].
Respondents rated their responses in the SAQ according to the level of emotion felt, from 5 = very high; 4 = high; 3 = medium; 2 = low, to 1 = very low, thus providing a five-point scale [
48]. This enabled the neutral circumstances and six affective states—anger, anxiety, sadness, disgust, surprise and happiness. Rottenberg’s suggestions were followed in order to identify appropriate affective film excerpts [
75]; one film excerpt had a duration of four minutes, which was the longest, with the others differing in length.
In order to assist with the presentation of the affective video excerpts to the subjects, the University of Vienna’s virtual emotion presenter program was applied. Further information source documentation and arbitrary presentation is permitted through the program. The anthropology research laboratory was the location of the research; the sound for the film was played to the subjects at a consistent and reasonable volume through a stereo system; film excerpts were viewed on an LCD display; the laboratory had consistent natural light sources, with the VEP also adopted as explained above. As the 40 subjects viewed the affective video excerpts, monitoring of the EEG electrode signals ensued.
2.2. Preprocessing Stage
Brain responses and artefacts may have intersected due to the latter residing within the frequency bands of the EEG waves. In terms of EEG signal preprocessing, a significant aspect is noise eradication. Standard filtering may be an aspect of the preprocessing phase, with EEG signals seeing the introduction of further software filters—band pass and notch filters for example—to carry out this process. As a means of restricting EEG signal frequencies in accordance with [
76], a higher cutoff frequency around 64 Hz and a low cutoff of 0.5 Hz (3 dB) was adopted for the band pass filter. A 50 Hz cutoff frequency was adopted for the notch filter; eliminating A/C electricity line interference is the typical reason for doing so [
38]. Three 10 s trials per video excerpt comprised the overall video, with every 10 s trial comprising of 1280 information points, in order to carry out additional filtered EEG data processing.
2.3. Features Extraction
Comprehending various affective states’ associations is assisted by EEG signals as a significant source of brain function data. In terms of identifying particular emotional actions, the EEG signal offers various quantifiable measures. Accordingly, affective EEG biomarkers were derived from a number of characteristics, primarily distinguished into temporal, entropy and spectral power characteristics, as a means of determining the principal characteristics that enable the EEG data to be matched with affective states, while also allowing improved explication of the brain areas’ altering affective states. Additionally, the brain areas’ and seven affective states’ spectro-spatial , entropy-spatial and temporo-spatial profiles were determined through combining the quantified biomarkers via Pearson’s correlations.
2.3.1. Spectral Biomarker
Through investigating the impact on different brain areas of various multi-channel EEG signals’ frequency bands, such spectral assessments as a linear characteristic for appraising affective alterations have been pervasively adopted. Multi-channel EEG alterations have been quantified via the AbsP characteristic as aspects of brain rhythms. Meanwhile, Welch’s technique was applied to calculate the EEG information’s power spectral density (PSD), with the specific frequency bands of gamma (
: 32 ≤
≤ 60) Hz, beta (
: 16 ≤
≤ 32) Hz, alpha (
: 8 ≤
≤ 16) Hz, theta (
: 4 ≤
≤ 8) Hz, and delta (
: 0 ≤
≤ 4) Hz being distinguished as particular frequency bands for the EEG signals’ PSD [
77]. A single band’s level of EEG activity autonomous of different bands’ activity is indicated by the
, with Equation (1) adopted to determine it [
78].
Every film excerpt’s last 30 s were divided into three 10 s parts comprising of 1280 information points per part, providing 3840 information points overall from which the EEG signal information’s features were ascertained.
2.3.2. Entropy Biomarker
The obtained EEG signals, inclusive of the , have been assessed by applying the non-linear entropy method, given that complex mental procedures may be undertaken by the brain.
In order to calculate the
based on mean
for
,
, where
For a defined scale factor
and embedding dimension,
,
and
for each of
are separately calculated. Next, the average of values of
and
on
are computed, respectively. Finally, the
is computed as in Equation (2):
The embedding dimension
,
power
, and tolerance
for all of the approaches were respectively chosen as
,
,
,
, and
is the standard deviation of the original time series [
68].
2.3.3. Temporal Biomarker
In contrast with alternative brain imaging approaches, greater temporal resolution and altered temporal changeability over a particular time period are provided by EEG signals, which provide its clinical advantage. Accordingly, precise millisecond readings of electro-physiological alterations may be derived from EEG. Consequently, temporal data analysis enables the formulation of EEG biomarkers.
The Hilbert transformation’s adoption enables the application of the amplitude envelope to define the temporal biomarkers. Skewness () was calculated to get in relation to the distribution for every EEG channel, once the amplitude envelope had been established.
Therefore, to compute the temporal biomarkers set, first the EEG signal
, for channel
and
is the time-domain index, the temporal envelope is then extracted using the Hilbert transform
as in Equation (3) [
79].
Let
be the
central moment of the
distributions. The
is defined as in Equation (4).
is the normalized 3rd order moment of amplitude distribution. If the distribution is symmetrical, then
is zero. By contrast, large
values are associated with the asymmetry degree of amplitude distribution [
80,
81,
82].
2.4. Statistical Analysis
Enhanced comprehension of brain states is a requisite outcome of the approach taken to the EEG dataset’s mapping. IBM USA’s SPSS program version 25 was adopted to undertake statistical analysis. Resultantly, four recording areas relating to the cerebral cortex’s scalp region formed the basis of the initial categorization of the 40 fit participants’ EEG dataset. The dimension of the feature matrix was (40 subjects × 14 EEG Channels × 7 emotional states) = 3920 attributes for each of spectral, entropy and temporal biomarkers. The different brain regions’ alternative affective states’ profiles and affective biomarkers could be directly conveyed by the divergences in brain areas, facilitated by the mean characteristics of the area. The area mean’s derived characteristics were used to categorize the various brain regions’ differences, for example occipital (O1 and O2 channels), parietal (P7 and P8 channels), temporal (T7 and T8 channels) and frontal (AF3, F7, F3, FC5, F4, FC6, F8, and AF4 channels). Subsequently, Levene’s test for homoscedasticity was applied, and the Kolmogorov–Smirnov test was performed to test the normality assumption required by the ANOVA statistical test. Two methods of statistical analysis were applied. One established the extent to which brain areas had varying affective states in relation to temporal, entropy and spectral characteristics, namely the analysis of variance (ANOVA) test. The brain areas’ various connectivity characteristics were established through Pearson’s correlation.
2.4.1. ANOVA
There were three aspects to the ANOVA test. Firstly, the distinctive characteristics were subject to a two-way ANOVA test; the dependent variable related to the spectral biomarker and was the AbsP characteristic, while the independent variables were the four brain areas (occipital, parietal, temporal and frontal) as well as the seven emotional states (anger, anxiety, disgust, happiness, sadness, surprise and neutral).
Secondly, the characteristic was subjected to the two-way ANOVA test, with the independent variables being the brain areas and seven affective states, while the entropy biomarker was the dependent variable.
Thirdly, the dependent variable of the seven affective states’ amplitude envelope distributions’ was subject to the two-way ANOVA test, with the temporal biomarker’s independent variable being the seven affective states and four brain areas.
Duncan’s test was applied in order to provide the post hoc contrast, with ˂ 0.05 established as each statistical assessments’ level of significance. Resultantly, the seven affective states and the brain areas’ possible temporal, entropy and spectral biomarkers are conveyed in this part. The Bonferroni post hoc test has been conducted to examine multiple comparisons for each group of tests, including the seven emotional states and the four brain regions.
2.4.2. Pearson’s Correlations
As a means of analyzing and revealing the spatial variability and distribution changes in three different ways along the EEG signals’ length, the spectral, entropy and temporal biomarkers obtained during the previous section will be integrated into the brain spatial information, thus enabling an appropriate understanding of emotional significance. Consequently, three stages of Pearson’s correlation were implemented for developing spectro-spatial , entropy-spatial and temporo-spatial profiles. These patterns offered a concise, consolidated method of EEG profile representation over the brain regions relating to anger, anxiety, disgust, happiness, sadness, surprise and neutral emotions.
During each of the three sessions, Pearson’s correlation coefficient was calculated so as to establish the biomarkers’ correlations, including for the neutral emotional state and six emotion states. Each correlation analysis under the Pearson’s correlation method was calculated at reflecting statistical significance. All correlation sessions were implemented for every participant.
Determination of every specific affective state’s profile— for anger, for anxiety, for disgust, for happiness, for sadness, for surprise)—as well as for the neutral affective state, was undertaken during Pearson’s correlation’s initial application.
A 2nd session of Pearson’s correlation, the profile— for anger, for anxiety, for disgust, for happiness, for sadness, for surprise—as well as for the neutral affective state, were computed.
A 3rd session of Pearson’s correlation, the profile— for anger, for anxiety, for disgust, for happiness, for sadness, for surprise)—as well as for the neutral affective state, were obtained.
4. Discussion
EEG’s utility as a clinical tool for analyzing functional changes associated with different emotional states (anger, anxiety, disgust, happiness, sadness, surprise and neutral) across different brain areas (frontal, temporal, parietal and occipital scalp) is of considerable interest. In this regard, the research has established a novel conceptual connection between the
,
and
profiles and the aforementioned emotional states across the brain regions. Conventional filters were employed to provide a preprocessing stage. A total of 14 channels across the various scalp regions were recorded while participants viewed seven brief emotional audio-visual video clips. The various domain features during this research, including spectral, entropy and temporal features, were computed so as to illustrate key EEG biomarkers relating to several emotional states. To provide more in-depth investigation,
,
and
EEG emotional profiles were developed through the multivariate addition of spectral, entropy and temporal characteristics to spatial information. Overall, from the visual inspection of the spectral and temporal biomarkers, it was found that the frontal regions are particularly responsible for emotion detection while experiencing anger and anxiety in the upper-right quadrant of the valence-arousal circumplex model, whereas sadness and disgust appear in the lower-right left quadrant of the valence-arousal circumplex model. Surprise and happiness were situated in the upper-left quadrant of the valence-arousal circumplex model. The entropy biomarkers evidenced that the temporal regions were especially activated in the detection of emotion while experiencing anxiety and surprise, in the valence-arousal circumplex model’s upper-right and upper-left quadrants respectively.
Table 8 presents the most highly correlated emotions with neutral for the
,
and
profiles across the four brain regions. It was evidenced that the frontal, temporal, parietal and occipital lobes are primarily responsible for anger, anxiety and sadness elicitations. Emotions such as surprise are detectable in the frontal and parietal lobes whereas happiness and disgust may be elicited from the temporal and occipital lobes respectively. Accordingly, such findings imply that the combination of spectral, entropy and temporal feature sets could provide and convey more reliable biomarkers as a means of identifying
,
and
profiles for anger, anxiety, disgust, happiness, sadness, surprise and neutral emotional states across the frontal, temporal, parietal and occipital scalp brain areas. The
profile is significant in representing how anger emotions correspond to all brain lobes, while the
profile is significant for representing anxiety emotions. Additionally, the
profile is important for representing the sadness emotions.
Regarding the neuro-scientific perspective, all of the obtained results are consistent with the frontal, temporal, parietal and occipital brain lobes’ principal functions. The frontal lobe is deemed to be the emotional control center [
83,
84], while temporal lobes are linked to emotional perception [
85]. Resultantly, to obtain greater insight into EEG emotional states, we incorporated several features from spectral, entropy and temporal aspects, enabling the identification of the most reliable EEG emotional biomarkers, as well as the development of the
,
and
profiles as benchmarks for deeper inspection.
To sum up, emotions play a critical role in our day-to-day lives. Emotion investigation can gain a deeper understanding of human complex behavior. Emotions like happiness are considered as positive emotions that have been linked to a variety of outcomes including increased longevity and increased marital satisfaction [
82]. Conversely, anger, anxiety and sadness are often thought of as negative emotions that have been linked to decreased life expectancy and may even have an impact on physical health [
83,
84]. Therefore, to capture and characterize people’s everyday emotional experiences, many recent scientific works validate the use of EEG as a diagnostic tool that is widely used in everyday life. So far, spectral, entropy and temporal biomarkers and
,
and
EEG emotional profiles might be valuable physiological information that help in improving emotional investigation procedure.