Simulation of Human Ear Recognition Sound Direction Based on Convolutional Neural Network

1 Introduction

In the field of artificial intelligence, computer hearing [8] remains in an early stage of research than computer vision [10]. With the development of technology, computer hearing has become an important research topic in the field of artificial intelligence. It is one of the essential symbols of intelligent robots, and it is also an important means to realize human-computer interaction and interaction with the environment [5].

There are three mainstream methods for sound source localization: time delay estimation method [9], the beam forming method [7] and a machine learning method [16]. The delay estimation is built on the generalized correlation function based on periodic cross-spectral density and the generalized cross-correlation algorithm based on the cross-power spectrum. This algorithm was suggested by Knapp et al. In order to get a more accurate time delay estimation, people focus on the design of the microphone array, such as linear arrays [6], circular arrays [15], distributed arrays [3], and non-coplanar arrays of any shape [1]. The sound source localization method based on time delay estimation is firstly to estimate the time difference of sound reaching different microphones. Then, the position of the sound source can be found by a geometric relationship. The advantage is that the amount of calculation is unimportant, but it will produce cumulative errors and cannot be used for multi-source positioning. The second method is the beam forming method. It is primarily divided into the subspace method and beam scanning method [4, 11, 17, 20]. The subspace method mainly performs feature decomposition on the covariance matrix of the output data of an array and obtains a noise subspace orthogonal to the signal subspace corresponding to the signal component. Sound localization is performed using orthogonally between the signal and noise subspaces. Classical methods include multi-signal classification [19] and estimation of signal parameters by rotational invariance technique [21]. The beam scanning method can position the array signal in a specific direction. The classical method is the steering response power phase transformation. The benefit of this method is that its spatial resolution is not affected by the sampling frequency, and arbitrary precision can be achieved under certain conditions. However, if directional noise occurs in practical applications, and its energy is not too dissimilar from the sound source, it is judged that the sound source may use the noise source as the sound source according to the largest features in the correlation matrix. Besides, this method needs to search the entire space to determine the reliable source, and the accuracy of the estimation is related to the degree of subdivision of the space, and the calculation is complicated.

Machine learning is a popular method in recent years, and they mainly include the two parts of feature extraction and machine learning methods. Extracted features mainly include time delay estimation feature, covariance matrix, and short-time power spectral density function spectrum. Machine learning methods mainly include support vector machines, multi-layer perceptron, Gaussian mixture models, Convolutional Neural Network and so on. In [14] the covariance matrix of the received signal of the microphone array is mainly used as the feature input. The feed-forward neural network model was chosen for sound source localization. In [13], the delay feature (GCC) is extracted by the generalized cross-correlation method, and the support vector machine is employed for positioning. In [18], the delay feature is further extracted, The network selects convolutional neural networks and multilayer perceptrons for reliable source coordinate localization. However, these methods use a complex microphone matrix and do not achieve end-to-end sound direction recognition.

2 Overview of the principle of acoustic scattering

It is well known that in terms of sound source localization, the number of microphones required is three or more. This article uses a dual microphone with input characteristics including time delay (GCC) and raw data. To some extent, when the sound source distance is the same, the raw data and time delay of the sound source into the sensor in the front and back directions are basically the same.

The basic principle of this paper is the sound scattering feature. As shown in Figure 1. When the sound wave emitted by the sounder reaches the simulated human head, when an obstacle such as the auricle and the irregular surface of the human face is encountered, part of the sound wave is deviated from the original predetermined propagation path and spreads from the periphery of the obstacle. This makes the difference in the audio data received by the microphone. With this feature we try to use dual microphones for sound source orientation.

Figure 1

Schematic diagram of acoustic scattering

3 The proposed approach

In this paper, our model includes two parts of feature extraction and convolutional neural network. In terms of feature extraction, we use the dual-channel raw data and time delay estimates as feature input for CNN, respectively. On CNN, we propose two kinds of network structures Net1 and Net2. The specific operation process is shown in Figure 2.

Figure 2

Sound direction recognition model flow chart

3.1 Data acquisition and preprocessing

In this section, we will describe step one following Figure 3. The collection of audio signals is selected in an anechoic room environment and an ordinary room environment. The sound is emitted in four different directions of the simulated human head, and these auditory data are collected by Pulse.

Figure 3

Data acquisition and pre-processing flowchart

For the original dataset,we intercept directly from the raw data in each direction, and the intercept length is L. All data form a dataset in binary form. The dataset mainly includes four parts, namely, a training set, a training set label, a test set, and a test set label.

For the GCC dataset, we need to go through the process of feature extraction. The specific operation is: extracting the delay feature by using the generalized cross-correlation algorithm based on the cross-power spectrum from the original data with the intercept length L. The feature extraction method will be introduced in the next part.

3.2 Feature extraction

In this part, we will introduce the extraction method of the delay feature. The commonly used time delay estimation methods have Generalized Cross-Correlation, Least Mean Square [2] and Cross-power Phase [12]. In this paper, we use the Cross-power Phase to get time delay estimation. This method was first proposed by Knapp et al., as shown in Figure 4:

Figure 4

Delay feature extraction flow chart

First, the original data of length L is transformed into a frequency domain signal by fast Fourier transform. The mutual power spectral density function G_xy (τ) is found.

Next, we perform frequency domain weighting on it. However, some of our audio data is collected in a common room and there may be interference from noise and reverberation, which do not make it possible to observe peaks. In order to make peaks easier to observe, we first need to perform a filtering process on the signals, that is, weighing in the frequency domain, it can strengthen the spectral component of the source signal in the received signal, achieve the function of improving the signal to noise ratio, and then can obtain the higher TDOA delay estimation accuracy. We use Path as the weighting function to get:

(1) R x y g = ∫ − ∞ + ∞ A G x 1 x 2 ( τ ) e − j 2 π τ d τ

Where A = 1 G x 1 x 2 ( τ ) represents a generalized cross-correlation weighting function.

Finally, the generalized cross-correlation function based on the cross-power spectrum can be obtained through the inverse Fourier transform. The peak is the delay between the arrival of the sound and the two microphones.

After obtaining the generalized cross-correlation function, we will extract the effective values to form the feature matrix. First measure the distance d between the two microphones, According to the formula:

(2) Δ τ m a x = d v F s

Where v = 340m/s represents the velocity of sound in the air, F_s represents the sampling frequency. Therefore, the number of useful values is 2 τ_max. The feature matrix X_xy is:

(3) X x y = R x y ( − Δ τ max ) , R x y ( − Δ τ max + 1 ) . . . R x y ( Δ τ max − 1 ) , R x y ( Δ τ max ) T

In this paper, we intercept the length of the original data 256 ÷ 65536 = 3.9ms. The distance d between the two microphones was measured to be 13cm. Then the number of valid values is:

(4) Δ τ max = 0.13 340 × 65536 = 25

We select 50 valid values to form the data set X_xy , in the form of:

(5) X x y = R x y ( − 25 ) , R x y ( − 24 ) . . . R x y ( 24 ) , R x y ( 25 ) T

3.3 CNN architecture

This paper designs two architecture, namely Net1 and Net2. Net1 contains two hidden layers. The extracted features pass through the input layer, then they will go through the first hidden layer and are input to the second hidden layer. After the matching operation, they are input into the fully connected layer. Finally, the classification result will output via the Softmax layer, Net2 only contains a hidden layer, and its parameters are shown in Figure 5.

Figure 5

Schematic diagram of the convolutional neural network structure. Net1 contains two convolutional layers, while Net2 contains only one convolutional layer

In speech recognition, convolutional neural networks still perform very well.Speech recognition technology is used to model the relationship between spoken speech signals and textual content. However, speech signals are diverse, and we need to be able to distinguish between different signals for sound source localization. For example, different speakers, different timbres and tones, different sound sources, and environmental factors, the convolutional neural network uses the local connection and weight sharing to make it have good translation invariance. We can use this function to overcome the diversity of speech signals. Another aspect is the choice of the number of network layers. This article designed Net1 and Net2. Their hidden layers are two layers and one layer. The reason for not using a deeper network structure lies in the pooling layer. In terms of speech recognition, the largest pooled result is better than the mean pooling, which can reduce the number of output nodes, thereby reducing the amount of calculation and increasing the speed of operation. This is highly critical in practical applications. Besides, choosing maximum pooling can increase the robustness of speech features. After the maximum pooling method is selected, if the number of network layers is too much, it will lead to the loss of features, resulting in a lower recognition rate of experimental results.

In order to maintain the map and features of activated neurons by functions, preserve features and remove some of the data redundancy. We use activation functions to add non-linear factors. In this paper, four commonly used activation functions are selected: sigmoid, tanh, ReLU, and Leaky-ReLU. The cause is that the ReLU activation function is ineffective for negative numbers. In the time delay estimation matrix, we can clearly see that the negative number occupies a large proportion, so we choose different activation functions to compare the recognition accuracy.

4 Experimental evaluation

4.1 Experimental preparation and data acquisition

Experimental environments are the near-field environment of an anechoic chamber and the near-field environment of an ordinary room. As shown in Figure 6. Lab equipment includes Simulated Head, Pulse, Computer, Tape Measure, Router, Marker, and Audio Materials.

Figure 6

The left picture shows the ordinary room environment, and the right picture shows the anechoic room environment

Take an anechoic room environment as an example. As shown in Figure 7. First connect the device, place the simulation head in the middle of the anechoic chamber, connect the sensor on the head of the simulation to the Pulse and connect it to the computer through the router. Audio is recorded at 45 degrees, 90 degrees (Left), 180 degrees (Back), 270 degrees (Right), and 360 degrees (Front), respectively, based on the front of the human head. The distances are 40cm, 80cm, 120cm, and 200cm, respectively, and they are marked; finally,the audio material is played to record sound signals. The types and duration of recorded sounds are shown in Table 1:

Figure 7

Schematic diagram of experimental equipment connection

Table 1

The type and duration of the audio.

Distance	Experimental environment
	Anechoic room	Ordinary room
40cm	Speaking, Pulse gun, MP3—each 3min	Speaking, Pulse gun, MP3—each 3min
80cm	Speaking, Pulse gun, MP3—each 3min	Speaking, Pulse gun, MP3—each 3min
120cm	Speaking, Pulse gun, MP3—each 3min	Speaking, Pulse gun, MP3—each 3min
200cm	Speaking, Pulse gun, MP3—each 3min	Speaking, Pulse gun, MP3—each 3min

4.4 Analysis of results

4.4.1 Analysis of recognition results of different features

In an anechoic room environment, the comparison of the delay features and the identification results of the raw data is shown in Figure 8.

Figure 8

Based on the comparison of different feature recognition results

We can clearly see that the recognition results of the two-channel raw data are much better than the GCC features. It is well known that in conventional sound source localization methods, at least three microphones are required for positioning using delay estimation. Therefore, the time delay estimation is used as the input of the neural network, and the model cannot distinguish the front and back (left and right). Because the features in both directions are almost identical. However, when we used the two-channel raw data as the input of the neural network, we got a good recognition result. We speculate that the possible reason is that the sounds in different directions may differ more significantly in the raw data. Therefore, the recognition accuracy based on the original data is higher than the delay feature. In the discussion that follows, we focus on the raw data.

4.4.2 Influence of different indoor environments on experimental resultsn

Our experimental environment is divided into anechoic chambers and ordinary room. The reason why the ordinary room is chosen is that they are closer to the real environment. They are characterized by the possibility of noise and reverberation. The experimental results are shown in Figure 9:

Figure 9

Recognition results in different indoor environments based on raw data

As can be seen from Figure 8, the recognition rate of the ordinary room is lower than that of the anechoic chamber environment. Moreover, the recognition rate is getting lower as the distance increases. The possible reason is that the ordinary room has more noise and reverberation interference than the anechoic chamber. Therefore, this model is intended to be applied to actual engineering, and audio is also pre-processed to eliminate noise and reverberation interference.

4.4.3 The effect of different data sizes on the recognition accuracy

In the previous section, we discussed that the recognition rate of the ordinary room is lower than that of anechoic room. Therefore, we used different data sizes for experiments. The reason is that under the premise of ensuring accuracy, using smaller data as much as possible can greatly reduce the recognition time and improve efficiency. The result is shown in Figure 10.

Figure 10

The result is compared after changing the data size

In the figure, after changing the data size, the maximum recognition result is obtained at different distances. Although we have chosen larger data, it seems that the recognition accuracy has not improved much. However, we have found that when we choose larger data, the accuracy of the training set is high, which can reach almost 1. The test accuracy can reach more than 0.9 in the anechoic room environment, but not in the ordinary room, Perhaps overfitting is one of the reasons. We will continue to explore in the next work.

4.4.4 The influence of network structure on the recognition accuracy

Taking an anechoic room environment as an example, we compare the effect of different network structures on the recognition accuracy. The classification results on two different networks are shown in Figure 11.

Figure 11

Different network identification results of raw data

For the original feature, Net1 is about 0.1 more accurate than Net2. This phenomenon seems to indicate that the deep model is more practical. However, using the deep model is not realistic due to data size and maximum pooling restrictions. Moreover, our test accuracy can reach 0.92. For delay data, the accuracy of Net1 and Net2 is not really satisfactory.

Another improvement for the network is activating the function. For raw data, tanh is chosen as the activation function to obtain the highest recognition accuracy. Moreover, ReLU and Leaky-ReLU also have very good results. However, the accuracy of the sigmoid is much lower. The possible reason for this is that the input is far from the origin of coordinates and the gradient of the function becomes very small, almost zero. Eventually, it leads to gradient saturation. For the delay feature, although the recognition accuracy of the sigmoid is not very different from that of other activation functions, the sigmoid requires an exponential operation and has a long running time.

5 High resolution directional model based on probability distribution

In this article, we only conducted positioning experiments for the four directions of the simulated human head. In order to achieve high-resolution orientation, we hope to expand the direction of the sound source to eight and more directions, but we do not want to increase Multi-directional training data, therefore, we propose a high-resolution positioning model based on probability distribution.

Firstly, the network model is trained by using four directions of audio data. When the training is completed, we input a test data of 90 degrees, and perform numerical processing on the final output unit through the Softmax layer to obtain a sound source probability of P1 in the direction of 90 degrees. The probability of the sound source in the 180-degree direction is P2, the probability of the sound source in the 270-degree direction is P3, the probability of the sound source in the 360-degree direction is P4, and when the recognition is correct, there is: P1 > P2, P3, P4.

Based on this well-trained network model, we want to predict more directions. We guess that there is a certain relationship between the directions of the source points. If such an algorithm is found, the positive direction is related to other directions. And it gives a new method of discriminating between right and wrong in the oblique direction. It is a possibility to use the n-class model to predict the 2n-class problem. Therefore, we design an algorithm to convert the probability of the model output into an unknown direction. And we will call this algorithm a high-resolution algorithm, the specific steps are as follows:

Step one: First, we define the number classifications n of the model which will be trained, divide 360° equally into n-class and collect sound data in n directions for feature extraction. Then we make a data set and train the neural network, and finally save this well-trained network model.

Step two: We divide 360° equally into 2n-class, and collect the sound data between n and n+1 direction’s center, a total of n data is collected. The feature of the new data is extracted by the extraction feature of step one. The data set is only included in the test set. The new test set is input into the trained network model, and the model outputs n probability values. We record them as P1 P2. . . P_n.

Step three: Using the vector decomposition method, multiply the probabilities of the n directions by the angle between the data directions of the two acquisitions. Then we let the vector sum in the two adjacent directions, and finally get the probability of the center direction, the expression is as follows:

Where j is 1, 2, 3...n, P_n is the probability of direction n and N_i is the probability of oblique direction i.

Step four: Based on the high-resolution algorithm, we add a new filter condition. According to the empirical algorithm, when the probability of an oblique direction is greater than a confidence probability N_v, we can think that it is correct, that means, N_i and N_v are compared. If N_i > N_v, the judgment is correct.

When we are making the oblique direction test set, we know the label corresponds to each data, that means, the direction of the sound source. The purpose of setting the confidence probability is to give a value range based on the real position, if we finally obtained the probability which is within this range by the high resolution algorithm. We can determine that the classification is correct. A schematic diagram of the algorithm is shown in Figure 12.

Figure 12

Schematic diagram of high resolution algorithm

In order to verify the accuracy of the high-resolution algorithm, we designed a set of experiments: Taking n as 4 as an example and collecting four oblique data. Finally we obtained the results by the above algorithm are shown in Table 9.

It can be seen from Table 9 that when the confidence probability is equal to 0.3 or 0.4, the result of the direction discrimination both can reach a higher probability. Thus the results verify our previous guess that there is a certain relevancy between the directions of the sound source points and also validates the practicality of our algorithm.

In this article, we use the dual-channel raw data as the input feature of the neural network to design a dual microphone model that simulates the human ear, At the same time, the high-resolution directional model doubles the recognition accuracy of the sound source direction. The model based on the convolutional neural network shows good performance, and the accuracy on the test set reaches more than 0.9. When the confidence probability is set to 0.3, the recognition accuracy of the high-resolution directional model reaches 0.99. The dual microphone model that simulates the human ear has a wide range of application prospects in the project. The reduction in the number of microphones can reduce the consumption of materials and cost of the enterprise, and the dual microphone model can also achieve the recognition accuracy of the microphone array. The simulation of the human ear enhances the credibility of the dual microphone system, making it more widely used. Among them, the high-resolution directional model is a practical application of simulating the human ear, which makes the recognition accuracy of the direction of sound source much improved.

6 Conclusions

In this article, a two-microphone sound direction recognition model is proposed. Through this article,we can have the following benefits:

1. The theoretically untrustworthy dual microphone model can be used to identify the direction of the sound, thus achieving some of the functions of the human ear.

2. We can use the convolutional neural network to identify the sound direction. When the input is two-channel raw data, we can get a very good result.

3. The anti-noise and anti-reverberation of this model are not strong. If it is carried out in an ordinary room, the audio data must be pre-processed to remove noise interference.

4. When the size of the selected data is larger, the improvement of the test accuracy is not obvious, but we found that the training accuracy can be improved to almost 1 during the training.

Finally, we performed high-resolution directional experiments based on probability distributions, which are no longer limited to four directions. A high resolution algorithm is designed to achieve higher resolution accuracy. In the next job, it also enhances the model’s noise immunity and reverberation capabilities, allowing it to be used in complex environments.