Optimizing text-independent speaker

recognition using an LSTM neural network

Master Thesis in Robotics
Joel Larsson

October 26, 2014

 Abstract

In this paper a novel speaker recognition system is introduced. Automated

speaker recognition has become increasingly popular to aid in crime investi-
gations and authorization processes with the advances in computer science.
Here, a recurrent neural network approach is used to learn to identify ten
speakers within a set of 21 audio books. Audio signals are processed via spec-
tral analysis into Mel Frequency Cepstral Coefficients that serve as speaker
specific features, which are input to the neural network. The Long Short-
Term Memory algorithm is examined for the first time within this area, with
interesting results. Experiments are made as to find the optimum network
model for the problem. These show that the network learns to identify the
speakers well, text-independently, when the recording situation is the same.
However the system has problems to recognize speakers from different record-
ings, which is probably due to noise sensitivity of the speech processing algo-
rithm in use.

Keywords - speaker recognition, speaker identification, text-independent,

long short-term memory, lstm, mel frequency cepstral coefficients, mfcc, re-
current neural network, speech processing, spectral analysis, rnnlib, htktoolkit
Contents

I Introduction 3
1 Introduction 4
1.1 Defining the Research . . . . . . . . . . . . . . . . . . . . . . 7

II Background 9
2 Neural Networks 10
2.1 Recurrent Neural Networks . . . . . . . . . . . . . . . . . . . 10
2.2 Long Short-Term Memory . . . . . . . . . . . . . . . . . . . . 11
2.2.1 Fundamentals . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.2 Information Flow . . . . . . . . . . . . . . . . . . . . . 13
2.2.3 Algorithm Outline . . . . . . . . . . . . . . . . . . . . 13
2.2.3.a Forward Pass . . . . . . . . . . . . . . . . . . 14
2.2.3.b Backward pass . . . . . . . . . . . . . . . . . 15
2.2.3.c Update Weights . . . . . . . . . . . . . . . . 17

3 Sound Processing 18
3.1 Speech From A Human Perspective . . . . . . . . . . . . . . . 18
3.1.1 Speech Production . . . . . . . . . . . . . . . . . . . . 18
3.1.2 Speech Interpretation . . . . . . . . . . . . . . . . . . 19
3.2 Automatic Feature extraction . . . . . . . . . . . . . . . . . . 20
3.2.1 The Speech Signal . . . . . . . . . . . . . . . . . . . . 20
3.2.2 Analyzing the signal . . . . . . . . . . . . . . . . . . . 21
3.2.2.a Mel Frequency Cepstral Coefficients . . . . . 22

III Experiment Setup 27

4 Model 28
4.1 Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . 29
4.3 Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . 30

1
5 Experiments 32
5.1 Does size matter? . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Will the classifications be robust? . . . . . . . . . . . . . . . . 33

IV Results and Discussion 35

6 Results 36
6.1 Size/Depth Experiments . . . . . . . . . . . . . . . . . . . . . 36
6.2 Robustness Experiments . . . . . . . . . . . . . . . . . . . . . 40

7 Discussion 46
7.1 Discussion of Results . . . . . . . . . . . . . . . . . . . . . . . 46
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2
 Part I

Introduction

3
Chapter 1

Introduction

Finding a way to make computers understand the human languages has

been a subject for research for a long period of time. It is a crucial point
in the quest for smooth computer-human interaction. Human-like behavior
in technology has always fascinated us and the ability to speak is standard
for computers and robots in science fiction stories. Nowadays, the fruits of
research can be seen in everyday life as speech recognition has become a
common feature in smart phones. For instance Apple’s Siri and especially
Google’s Voice Search shows, as of 2014, remarkably good results in under-
standing human speech, although not perfect.

An area related to speech recognition is speaker recognition. Speaker recog-

nition is easiest explained as the ability to identify who is speaking, based on
audio data. Speech recognition, on the other hand, is the ability to identify
what is said. A person’s voice is highly personal. It has gotten its spe-
cific sound due to the unique physics of the body of the individual. These
characteristics are transferred into the sound waves and can be extracted as
a set of features, which a computer can learn to associate with a specific
person. Compared to speech recognition, speaker recognition is a somewhat
less explored field, but have many possible applications. Both now and in
the future. For instance it can be used as a form of verification where high
security is needed, or aid humanoids in communication with people. Nowa-
days, and most likely even more in the future, speaker recognition is used in
forensics as to aid in the analysis of phone calls between criminals [26, 33],
as another example.

There are a set of concepts to get acquainted with regarding speaker recog-
nition. These concepts can be a bit difficult to grasp at first because of
their similarity to each other. However, the following paragraphs will try to
explain their differences [4, 6]. Roughly, there are two phases involved with
speaker recognition: enrollment and verification. In the enrollment phase
speech is collected from speakers and features are extracted from it. In the
second phase, verification, a speech sample is compared with the previously

4
recorded speech to figure out who is speaking. How these two steps are
made differ between applications. It is common to categorize applications as
speaker identification and speaker verification. Identification tasks involve
identifying an unknown speaker among a set of speakers, whereas verification
involves trying to verify that the correct person is speaking. Identification is
therefore a bigger challenge.

Speaker recognition is usually divided into two different types: text-dependent

and text-independent recognition. The difference between these lies in the
data sets from which decisions are made. Text dependent recognition is re-
ferred to as speaker recognition where the same things needs to be said in
the enrollment and verification phase. For instance it could be a password
in an authentication process. So, text-dependent recognition is typically
used in a speaker verification application. Text-independent recognition, on
the other hand, is recognition where the speech in the enrollment and ver-
ification phases can be different. What is more, it does not require any
cooperation from the speaker. Therefore it can also be made without the
persons knowledge. This type of recognition is instead typically used within
speaker identification applications.

The most common method used for recognizing human speech has for the
past decades been based on Hidden Markov Models (HMM) [1–3]. This is
because of their proficiency in recognizing temporal patterns. Temporal pat-
terns can be found in most real life applications where not all data is present
at start but instead is revealed over time - sometimes with very long time in
between important events.

Neural networks have become increasingly popular within this field the recent
years due to some advances in research. When compared to Feed Forward
Neural Networks (FFNN), Recurrent Neural Networks (RNN) have the abil-
ity to perform better in tasks involved with sequence modeling. Nonetheless,
RNNs have historically been unable to recognize patterns over longer peri-
ods of time because of their gradient based training algorithms. Usually they
cannot connect output sequences to input sequences separated by more than
5 to 10 time steps [30]. The most commonly used recurrent training algo-
rithms are Backpropagation Through Time (BPTT), as used by Rumelhart
and McClelland [29], and Real Time Recurrent Learning (RTRL), used by
Robinson and Fallside [28]. Even though they give very successful results
for some applications, the limited ability to bridge input-output time gaps
gives the trained networks difficulties when it comes to temporal information
processing.

The designs of the training algorithms are such that previous outputs of
the network gets more or less significant for each time step. The reason
for this is that errors get scaled in the backwards pass by a multiple of the
network nodes activations and weights. Therefore, when error signals are
propagated through the network they are likely to either vanish and get for-

5
gotten by the network, or blow up in proportion in just a few time steps.
This flaw can lead to an oscillating behavior of the weights between nodes
in a network. It can also make the weight updates so small that the network
needs excessive training in order to find the patterns, if ever [21, 24]. In
those cases it is impossible for the neural network to learn about patterns
that repeat themselves with a long time lag in between their occurrences.

However, during the last decade, great improvements regarding temporal

information processing has been made with neural networks. A new type
of RNN called Long Short-Term Memory (LSTM) [21] was introduced, ad-
dressing the problem with gradient descent error propagation in BPTT and
RTRL described above. In its architecture LSTM makes use of unbounded,
self-connected internal nodes called memory cells to store information over
time. The information flow through the memory cells is controlled by gating
units. Together, a memory cell combined with an input gate and an output
gate form a memory block. These memory blocks form the recurrent hidden
layer of the network. This architecture proved to function exceptionally well
with temporal patterns, being able to quickly learn how to connect data with
time lag in the order of 1000 time steps. Even with noisy input and without
loosing the ability to link data adjacent in time. The algorithm needed some
fine tuning to reach its full potential though.

The very strength of the algorithm proved to also introduce some limita-
tions, pointed out by Gers et al. [13]. It could be shown that the standard
LSTM algorithm, in some situations where it was presented to a continuous
input stream, allowed memory cell states to grow indefinitely. These situ-
ations can either lead to blocking of errors input to the cell, or make the
cell behave as a standard BPTT unit. Presented, by Gers et al. [13], was an
improvement to the LSTM algorithm called forget gates. By the addition of
this new gating unit to the memory block, memory cells were able to learn
to reset themselves when their contents have served their purpose, hence
solving the issue with indefinitely growing memory cell states.

Building upon LSTM with forget gates, Gers et al. [14] developed the al-
gorithm further by adding so called peephole connections. The peephole
connections were giving direct connection between the memory cell and the
gating units within a memory block making it able to view its current in-
ternal states. The addition of these peephole connections proved to make it
possible for the network to learn very precise timing between events. The
algorithm was now really robust and promising for use in real world applica-
tions where timing is of the essence, for instance in speech or music related
tasks.

Long Short-Term Memory has brought about a change to the top of speech
recognition algorithms, as indicated by several research papers [12, 16, 17, 30,
34]. It has not only shown to outperform more commonly used algorithms,
like Hidden Markov Models, but it has also directed research in this area

6
towards more biologically inspired solutions [20]. Apart from the research
made with LSTM within the speech recognition field, the algorithm’s ability
to learn precise timing has been tested in the area of music composition,
Coca et al. [7], Eck and Schmidhuber [11], and handwriting recognition [19]
with very interesting results. Inspired by the great achievements described
above, the thought behind this thesis came about.

1.1 Defining the Research

The main goal of this thesis was to investigate a neural network’s ability to
identify speakers from a set of speech samples. The specific type of neural
network used for this matter was a Bidirectional Long Short-Term Memory
(BDLSTM) based Recurrent Neural Network [16]. To the author’s knowl-
edge, the performance of an LSTM based network had until this day never
been examined within the speaker recognition field. However, it was the
author’s belief that this architecture could excel in this field as it had done
within the closely related speech recognition area.

In order for a computer to be able to distinguish one speaker from another,

the sound waves has to be processed in such a way that features can be
extracted from them [9, 22, 25, 31]. The most commonly utilized method to
model sound waves in speech/speaker recognition is to transform them into
Mel Frequency Cepstrum Coefficients (MFCC) [10]. The MFCC’s are then
combined into a feature vector that is used as input to the LSTM network. In
this thesis the MFCC’s were extracted via the Hidden Markov Model Toolkit
(HTK) - an open source toolkit for speech recognition [35].

The data sets used for training and testing were gathered from a set of
audio books. These audio books were narrated by ten English speaking
adult males and contained studio recorded, emotionally colored speech. The
speaker identification system created was text-independent and tested on
excerpts of speech from different books read by the same speakers. This was
done as to test the systems robustness.

LSTM is capable of both online and offline learning techniques [20]. How-
ever, in this thesis the focus will be on online learning. Thus, the weights of
the network will be updated at every time step during training. The network
is to be trained using the LSTM learning algorithm proposed by Gers et al.
[14]. Experiments regarding the parameters: size, depth, network architec-
ture and the classification robustness will be carried out within the scope of
this thesis. These experiments will constitute a path towards optimization
of the system and the results will be evaluated based on the classification
error of the networks in use.

The first part of this thesis, Introduction, introduces the subject and defines

7
the research. In the second part, Background, the fundamentals of neural
networks and the LSTM architecture are outlined. This part also explains
sound processing and specifically MFCC extraction in detail. The third part,
Experiment Setup, describes how and what experiments were made during
this research. The fourth part, Results and Conclusions, states the results
from the experiments and the conclusions drawn from these.

8
 Part II

Background

9
Chapter 2

Neural Networks

This section contains an introduction to Recurrent Neural Networks and

also a description of the Long Short-Term Memory architecture used in this
thesis.

2.1 Recurrent Neural Networks

A Recurrent Neural Network (RNN) is a type of neural network that in
contrast to Feed Forward Neural Networks (FFNN) makes use of cyclic con-
nections between its nodes. This structure makes it possible for a RNN to
form a sort of memory of internal states. Having this information available
means that the RNN cannot only map input to output directly, but instead
can make use of virtually all previous inputs to every output. Thus, RNNs
are very good to use in applications where contextual information is impor-
tant for the network to make correct classifications. In particular they can
be used favorably for time series prediction, for instance in finance, and for
time series classification as for example when detecting rhythms in speech
or music.

Unfortunately traditional RNNs function better in theory than in practice

because the contextual information can only be held in a network’s "mem-
ory" for a limited amount of time. So sequences of inputs way back in history
cannot be taken into account. Typically, RNNs can build their contextual
information upon no more than the last ten times steps. That is because
they suffer from problems with vanishing or exploding gradients [5]. The
problem arise when training the network with gradient based training algo-
rithms, such as Backpropagation Through Time (BPTT) [29] or Real Time
Recurrent Learning (RTRL) [28]. Many attempts to deal with these prob-
lems have been made, for instance by Bengio et al. [5]. However, the solution
that has proven to give the best results up til now is named Long Short-Term
Memory, introduced by Hochreiter and Schmidhuber [21].

10
Figure 2.1: A simple RNN structure. The grey boxes show the boundary of
each layer. Nodes in the network are represented by the blue circles. The
arrows represent the connections between nodes. Recurrent connections are
marked with red color.

2.2 Long Short-Term Memory

A recurrent network can be said to store information in a combination of
long- and short-term memory. The short-term memory is formed by the
activation of units, containing the recent history of the network. The long-
term memory is instead formed by the slowly changing weights of the unit
transitions that are holding experience based information about the system.
Long Short-Term Memory is an attempt to extend the time that a RNN
can hold important information. Since its invention [21] LSTM has gotten
improved with several additions to its structure. The enhancements have, as
mentioned earlier, been forget gates [13] and peephole connections [14]. The
architecture described here is called bidirectional LSTM (BLSTM), which
has been implemented for the purposes of this thesis. This particular version
of LSTM was first introduced by Graves and Schmidhuber [15] and contains
all the earlier improvements.

2.2.1 Fundamentals
In this part the fundamentals of the LSTM structure will be described along
with the importance of each element and how they work together.

Instead of the hidden nodes in a traditional RNN, see Figure 2.1, an LSTM
RNN makes use of something called memory blocks. The memory blocks
are recurrently connected units that in themselves hold a network of units.

11
 Figure 2.2: An LSTM memory block with one memory cell.

Inside these memory blocks is where the solution to the vanishing gradient
problem lies. The memory blocks are made up of a memory cell, an input
gate, an output gate and a forget gate. See Figure 2.2. The memory cell is
the very core of the memory block, containing the information. To be able
to preserve its state when no other input is present the memory cell has a
selfrecurrent connection. The forget gate guards this selfrecurrent connec-
tion. In this way it can be used to adaptively learn to discard the cell state
when it has become obsolete. This is not only important to keep the network
information up to date, but also because not resetting the cell states can in
some occations, with continuous input, make them grow indefinitely. This
would defeat the purpose of LSTM [13]. The input gate determines what
information to store in the cell, that is, protects the cell from unwanted in-
put. The output gate, on the other hand, decides what information should
flow out of the from the memory cell and therefore prohibits unwanted flow
of information in the network.

The cell’s self-recurrent weight and the gating units constructs altogether
a constant error flow through the cell. This error flow is referred to as Con-
stant Error Carousel (CEC) [21]. The CEC is what makes LSTM networks
able to bridge inputs to outputs with more than 1000 time steps in between
them and thereby extending the long range memory capacity by a hundred-
fold compared to conventional RNNs. Having access to this long history of
information is also the very reason that LSTM networks can solve problems

12
that earlier was impossible with RNNs.

2.2.2 Information Flow

The following will be a description of how the information flows through the
memory block, from input to output. For simplicity reasons, the description
covers a memory block containing only one memory cell. See Figure 2.2 to
be able to easier follow this explanation.

Incoming signals first get summed up and squashed through an input ac-
tivation function. Traveling further towards the cell the squashed signal gets
scaled by the input gate. The scaling of the signal is the way that the in-
put gate can guard the cell state from getting interfered with by unwanted
signals. So, to prohibit the signal from reaching the cell, the input gate
simply multiplies the signal with a scaling factor of, or close to, zero. If the
signal is let past the gate then the cell state gets updated. Similarly, the
output from the memory cell gets scaled by the output gate in order to pro-
hibit unnecessary information from disturbing other parts of the network. If
the output signal were to be allowed through the output gate then it gets
squashed through an output activation function before leaving the memory
block.

In the occasion that an input signal is not let through to update the state
of the memory cell, the cell state is preserved to the next time step by the
cell’s self-recurrent connection. The weight of the self-recurrent connection
is 1, so usually nothing gets changed. However, the forget gate can interfere
on this connection to scale the cell value to become more or less important.
So, if the forget gate finds out that the cell state has become obsolete, it can
simply reset it by scaling the value on the self-recurrent connection with a
factor close to zero.

All the gating units have the so called peephole connections where they can
access the cell state directly. This helps them to learn to precisely time differ-
ent events. The gating units also have connections to other gates, themselves
and block inputs and outputs. All these weighted information gets summed
up and used to set the appropriate gate opening in every time step. This
functionality is optimized in the training process of the network.

2.2.3 Algorithm Outline

The following part will explain the algorithm outline for a bidirectional
LSTM network trained with the full gradient backpropagation through time
algorithm. This type of LSTM was first introduced by Graves and Schmid-
huber [15] and the following description is heavily based upon their work.

13
 Table 2.1: Description of symbols

Symbol Meaning
Wij Weight to unit i from unit j
The time step at which a function is evaluated (if
τ
nothing else is stated)
xk (τ ) Input x to unit k
yk (τ ) Activation y of unit k
E(τ ) Output error of the network
tk (τ ) Target output t of unit k
ek (τ ) Error output e of unit k
k (τ ) Backpropagated error  to unit k
S Input sequence used for training
The set of all units in the network that may be con-
nected to other units. That is, all units who’s ac-
N
tivations are visible outside the memory block they
belong to.
C The set of all cells
c Suffix indicating a cell
ι Suffix indicating an input gate
φ Suffix indicating a forget gate
ω Suffix indicating an output gate
sc State s of cell c
f The function squashing the gate activation
g The function squashing the cell input
h The function squashing the cell output
α Learning rate
m Momentum

2.2.3.a Forward Pass

In the forward pass all of the inputs are fed into the network. The inputs
are used to update activations for the network units so that the output can
be predicted. Note that this description will be carried out in the required
execution order, as the order of execution is important for an LSTM network.

• Start by resetting all the activations, i.e. set them to 0.

• Proceed with the updating of the activations by supplying all the input
data to the network and execute the calculations (2.1) - (2.9) in a
sequential manner for each memory block:

Input x to input gate ι:

X X
xi = wιj yj (τ − 1) + wιc sc (τ − 1) (2.1)
j∈N c∈C

14
Activation y of input gate ι:

yι = f (xι ) (2.2)

Input x to forget gate φ:

X X
xφ = wφj yj (τ − 1) + wφc sc (τ − 1) (2.3)
j∈N c∈C

Activation y of forget gate φ:

yφ = f (xφ ) (2.4)

Input x to cell c: X
∀c ∈ C, xc = wcj yj (τ − 1) (2.5)
j∈N

State s of cell c:
sc = yφ sc (τ − 1) + yι g(xc ) (2.6)
Input x to output gate ω:
X X
xω = wωj yj (τ − 1) + wωc sc (τ ) (2.7)
j∈N c∈C

Activation y of output gate ω:

yω = f (xω ) (2.8)

Output y of cell c:
∀c ∈ C, yc = yω h(sc ) (2.9)

2.2.3.b Backward pass

In the backward pass the predicted output is compared to the wanted output
for the specific input sequence. The error is then fed back through the
network and the derivative of the error function is calculated.

• First, reset all the partial derivatives, i.e. set their value to 0.
• Then calculate and feed the output errors backwards through the net,
starting from time τ1 .

The errors are propagated throughout the network making use of the stan-
dard BPTT algorithm. See the definitions below, where E(τ ) is the output
error of the net, tk (τ ) is the target value for output unit k and ek (τ ) is the
error for unit k, all at time τ . k (τ ) is the backpropagation error of unit k
at time τ .

15
Partial derivative δk (τ ) definition:

∂E(τ )
def ine δk (τ ) =
∂xk
Error of output unit k at time τ , ek (τ )

yk (τ ) − tk (τ ) k ∈ output units
ek (τ ) =
0 otherwise
Initial backpropagation error of unit k at time τ1 , k (τ1 ):

k (t1 ) = ek (t1 )
Backpropagation error of unit k at time τ − 1, k (τ − 1):
X
k (τ − 1) = ek (τ − 1) + wjk δj (τ )
j∈N

To calculate the partial derivatives of error function the calculations (2.10)

- (2.15) should be carried out for each memory block in the network:

Error of cell c, c :
X
∀c ∈ C, c = wjc δj (τ + 1) (2.10)
j∈N

Partial derivative of error of output gate ω, δω :

X
δω = f 0 (xω ) c h(sc ) (2.11)
c∈C

Partial derivative of the nets output error E with respect to state s of cell c,
∂E
sc (t):

∂E ∂E
(t) = c yω h0 (yc )+ (τ +1)yφ (τ +1)+δι (τ +1)wιc +δφ (τ +1)wφc +δω wωc
sc ∂sc
(2.12)
Partial derivative of error of cell c, δc :
∂E
∀c ∈ C, δc = yι g 0 (xc ) (2.13)
∂sc
Partial error derivative of the forget gate φ, δφ :
X ∂E
δφ = f 0 (xφ ) yc (τ − 1) (2.14)
∂sc
c∈C

Partial error derivative of the input gate ι, δι :

X ∂E
δι = f 0 (xι ) g(xc ) (2.15)
∂sc
c∈C

16
Now, calculate the partial derivative of the cumulative sequence error by
summing all the derivatives:

Definition of the total error Etotal when network is presented to the input
sequence S:
Xτ1
def ine Etotal (S) = E(τ )
τ =τ0

Definition of the partial derivative of the cumulative sequence error, ∆ij (S):
τ1
∂Etotal (S) X
def ine ∆ij (S) = = δi (τ )yj (τ − 1)
∂wij τ =τ +1
0

2.2.3.c Update Weights

The following is the standard equation for calculating gradient descent with
momentum. This is used after the forward and backward passes have been
carried out to update the weights between the nodes in the network. In this
thesis online learning is used, so the weights are updated after each time
step. The learning rate is denoted by α and the momentum by m.

∆wij (S) = α∆ij (S) + m∆wij (S − 1) (2.16)

17
Chapter 3

Sound Processing

This chapter will describe how sound waves from speech can be processed so
that features can be extracted from them. The features of the sound waves
can later be presented to a neural network for speaker recognition.

3.1 Speech From A Human Perspective

This section will briefly describe how sound is produced and interpreted by
humans. The biological way of doing it is important to understand as it helps
in comprehending how the different sound processing techniques function.

3.1.1 Speech Production

The acts of producing and interpreting speech are very complex processes.
Through millions of years of evolution we humans have learned to control
them to the extent we can today. All the sounds that we produce are con-
trolled by the contraction of a set of different muscles. To explain it simply,
the muscles in the lungs are pushing air from the lungs up through the glot-
tis, the opening of the larynx that is controlled be the vocal cords. The
vocal cords works to open or shut the glottis, creating vibrations that gen-
erate sound waves when the air passes through it. This part of the process
is referred to as phonation. The sound waves are then modified on its way
through the vocal tract by the use of our tongue, lips, cheeks and jaws etc.
before they are released into the air for interpretation [27]. The modifications
we do with different parts of the vocal tract are what we use to articulate
the sounds we produce. So, the production of speech is usually divided into
the three main parts:

• Respiration - where the lungs produce the energy needed, in the form
of a stream of air.

• Phonation - where the larynx modifies the air stream to create phona-
tion.

18
 • Articulation - where the vocal tract modulates the air stream via a set
of articulators.

Because all people are of different sizes and shapes, no two people’s voices
sound the same. The way our voices sound also varies depending on the peo-
ple in our surrounding, as we tend to adapt to the people around us. What
is more, our voices change as we grow and change our physical appearance.
So, voices are highly personal and invariant.

3.1.2 Speech Interpretation

Sound waves that are traveling through the air gets caught up into the shape
of the outer ear. Through a process called auditory transduction the sound
waves are then converted into electrical signals that can be analyzed by the
brain and interpreted into different words and sentences. The following ex-
plains this process in short.

When sound enters the ear it soon reaches the eardrum, or tympanic mem-
brane, which is a cone shaped membrane that picks up the vibrations created
by the sound [36]. Higher and lower frequency sounds makes the eardrum
vibrate faster and slower respectively, whereas the amplitude of the sound
makes the vibrations more or less dramatic. The vibrations are transferred
through a set of bones into a structure called the bony labyrinth. The bony
labyrinth holds a fluid that starts to move with the vibrations and thereby
pushes towards two other membranes. In between these membranes there is
a structure called the organ of corti, which hold specialized auditory nerve
cells, known as hair cells. As these membranes move, the hair cells inside
the organ of corti gets stimulated and fires electrical impulses to the brain.
Different hair cells get stimulated by different frequencies and the higher the
amplitude of the vibrations, the easier the cells get excited.

Through a persons upbringing it is learned how different sounds, or excita-

tion of different nerve cells, can be connected to each other to create words
and attach meaning to them. Similarly, it is learned that a certain set of
frequencies, i.e. someone’s voice, usually comes from a specific person and
thus we connect them to that person. In this way it is learned to recognize
someone’s voice. However, there are several systems in the human brain
involved with the interpretation and recognition of speech. For instance we
include body language and listen for emotions expressed in the speech to get
more contextual information when we determine what words actually were
said to us. And when trying to determine who is speaking, in a situation
where we cannot see the person, we rely heavily on what is being said and
try to put that into the right context and figure it out in that way.

The melody with which people speak, prosody, is dependent on language and
dialect but also change dependent on the mood of the speaker, for example.
However, every person make their own variations to it. They use a limited
set of words and often say things in a slightly similar way. All these things

19
Figure 3.1: The spectogram representation of the word "acting", pronounced
by two different speakers.

we learn and attach to the specific person. Because of all this contextual
information we can for instance more or less easily distinguish one person’s
voice from another. Unfortunately, all the information about the context is
usually not available when a computer tries to identify a speaker from her
speech. Therefore automatic speaker recognition pose a tricky problem.

3.2 Automatic Feature extraction

3.2.1 The Speech Signal
Speech can be thought of as a continuous analog signal. Usually it is said
that the smallest component in speech is phonemes. Phonemes can be de-
scribed as the basic sounds that is used to make up words. For instance
the word "fancy" consists of the five phonemes: /f/ /ā/ /n/ /s/ /y/. How-
ever, the relationship between letter and phoneme is not always one to one.
There can for example be two or three letters corresponding to one sound or
phoneme, e.g /sh/ or /sch/. The number of phonemes used vary between

20
languages as all languages have their own set of words and specific ways to
combine sounds into them.

How the phonemes are pronounced is highly individual. The intuitive way
of thinking when beginning to analyze speech signals might be that they
can easily be divided into a set of phonemes. That each phoneme have a
distinct start and ending that can be seen just by looking at the signal. Un-
fortunately that is not the case. The analogous nature of speech makes the
analyzing of it more difficult. Phonemes tend to be interleaved with one an-
other and therefore there are usually no pauses with silence in between them.
Some phonemes, such as /d/, /k/ and /t/, will make a silence before they
are pronounced though. It is because the glottis is completely shut in the
process of pronouncing them. This makes it impossible for air to be exhaled
from the lungs and hence there is no sound produced. This phenomenon can
be seen in figure 3.1.

How the phonemes are pronounced is influenced by our emotions, rhythm of

speech and dialects etc. Furthermore, all humans are unique. Everyone has
their own shape of the vocal tract and larynx and also their own abilities
to use their muscles to alter the shape of these. Because of this fact, the
sounds produced differ between individuals, see figure 3.1. Another thing
that affects the speech signal is peoples sloppiness. When people get tired
or too comfortable speaking with someone they tend to become sloppy with
regards to articulation, making phonemes and words more likely to float to-
gether. Thus, words can be pronounced differently depending on situation
and mindset of the speaker. Additionally, someone’s illness can also bring
about changes to their voice. These dissimilarities in pronunciation would
correspond to difference in frequency, amplitude and shape of the signal.
Therefore the speech signal is highly invariant and difficult to analyze in a
standardized way that makes it possible to identify people with 100 percent
success.

3.2.2 Analyzing the signal

There are a several ways to analyze a sound signal and all techniques have
their own limitations and possibilities. Roughly the different analyzing meth-
ods can be divided into temporal analysis methods and spectral analysis
methods [22]. These will here be descriped shortly.

In temporal analysis the characteristics of the sound wave itself is exam-

ined. This has its limitations. From the waveform it is only possible to
withdraw simple information, such as periodicity and amplitude. However,
these kinds of calculations are easily implemented and do not require much
computational power to be executed. The simplicity of the information that
can be gained from the waveform makes it less usable though. In a real life
situation the amplitude of a speech signal, for example, would differ highly
between situations and be dependent on the mood of the speaker, for in-

21
stance. Also when we speak we tend to start sentences speaking louder than
we do in the end of them, as another example. Thus, due to the fact that
speech is highly variable by its nature, the temporal analysis methods are
not used very often in real life applications. Not in this thesis either.

The more often used technique to examine signals is spectral analysis. Using
this method, the waveform itself is not analyzed, but instead the spectral
representation of it. This opens up for richer, more complex information
to be extracted from the signal. For example spectral analysis makes it
possible to extract the parameters of the vocal tract. Therefore it is very
useful in speaker recognition applications, where the physical features of
one’s vocal tract is an essential part of what distinguishes one speaker from
another. Furthermore, spectral analysis can be applied to construct very
robust classification of phonemes because information that disturb the valu-
able information in the signal can be disregarded. For example excitation
and emotional coloring of speech can be peeled off from the signal to leave
only the information that is concerning the phoneme classification. Of course,
the information regarding emotional coloring can be used for other purposes.

The facts presented regarding spectral analysis methods make them useful
for extracting features for utilization in real life applications. In comparison
with temporal analysis, the spectral analysis methods are computationally
heavy. Thus the need for computational power is greater with spectral than
temporal analysis techniques. Spectral analysis can also be sensitive to noise
because of its dependency on the spectral form.

There are several commonly used spectral analysis methods to extract valu-
able features from speech signals. Within speaker recognition, Linear Pre-
diction Cepstral Coefficients and Mel Frequency Cepstral Coefficients have
proven to give the best results [23]. The features are used to create feature
vectors that will serve as input to a classification algorithm in speech/speaker
recognition applications. In this thesis the features will serve as input to a
bidirectional Long Short-Term Memory neural network.

3.2.2.a Mel Frequency Cepstral Coefficients

Mel Frequency Cepstral Coefficients (MFCC) are among the most, if not the
most, commonly used features in speech recognition applications [31]. They
were introduced by Davis and Mermelstein [8] in 1980 and have been used in
state of the art research since then. Especially in the field of speaker recog-
nition. The MFCCs can effectively be used to represent features of speech
signals that are important for the vocal tract information. The features
are withdrawn from the short time power spectrum and will in a good way
represent the characteristics of the signal that are emotionally independent.
There are drawbacks, though. MFCCs are sensitive to noise and in speaker
recognition applications, where there can be a lot of background noise, this
may pose a problem [23].

22
The following will be a short outline of the steps in the process of acquiring
the Mel Frequency Cepstral Coefficients from a speech signal. The steps
presented below will be described in more detail further on.

• Divide the signal into short frames.

• Approximate the power spectrum for each frame.

• Apply the mel filterbank to the power spectra and sum the energy in
every filter.
• Logarithmize the filterbank energies.

• Calculate the DCT of the logaritmized filterbank energies.

• Discard all DCT coefficients exept coefficients 2-13.

The coefficients left are the ones that form the feature vectors exploited for
classification purposes. Usually features called Delta and Delta-Delta fea-
tures that are added to the feature vectors. These features are also known as
differential and acceleration coefficients and are the first and second deriva-
tives of the previously calculated coefficients.

The first step in the process is to divide the signal into short frames. This
is done because of the variable nature of the speech signal. To ease in the
classification process the signal is therefore divided into time frames of 20-40
milliseconds, where the standard is 25 milliseconds. During this time period
the signal is considered not to have changed that much and therefore the
signal will for instance not represent two spoken phonemes in this time win-
dow. The windows are set with a step of around 10 milliseconds in between
the start of two consecutive windows, making them overlap a bit.

When the signal has been split up into frames we should estimate the power
spectrum for each frame by calculating the periodogram of the frames. This
is the process where it is examined which frequencies are present in every
slice of the signal. Similar work is made by the hair cells inside the cochlea,
in the organ of corti in the human ear.

23
Figure 3.2: The Mel scale. Based on peoples judgment, it was created by
placing sounds with different pitch on what was perceived as equal melodic
distance from each other.

Table 3.1: Description of symbols

Symbol Meaning
N Number of samples in one frame.
Number of discrete points in a Discrete Fourier Transform
K
of a frame.
i Indicates frame.
si (n) Time domain signal si at sample n, in frame i.
Si (k) Discretized signal Si at point k, in frame i.
h(n) Analysis window h(n) at sample n.
Pi (k) Periodogram estimate Pi at point k, in frame i.
di Delta coefficient di of frame i.
ci±M Static coefficient c of frame i ± M , where M is usually 2.

First the Discrete Fourier Transforms (DFT) of the frames are determined:
N
X
Si (k) = si (n)h(n)e−2πkn/N 1≤k≤K (3.1)
n=1

From the DFTs the spectral estimation periodogram is given by:

1 2
Pi (k) = |Si (k)| (3.2)
N

24
Now, the result from this should be an estimation of the signals power spec-
trum from which the power of present frequencies can be withdrawn.

The next step in the process would be to filter the frequencies of the peri-
odogram, in other words combine frequencies close to each other into groups
of frequencies. This is done to correspond to limitations in the human hearing
system. Humans are not very good at distinguishing frequencies in the near
vicinity to each other. This is especially true for higher frequency sounds. At
lower frequencies we have a better ability to differentiate between sounds of
similar frequency. To better simulate what actually can be perceived by the
human ear, the frequencies are therefore grouped together. This also peels
away unnecessary information from the signal and hence makes the analysis
less computationally heavy.

To better model the human perception of sounds the Mel-frequency scale

was introduced by Stevens and Volkmann [32]. The scale relates the per-
ceived frequency to the actual frequency in a way that makes it fairly linear
up to 1000 Hz, which corresponds to 1000 mel, and logarithmic afterwards,
see Figure 3.1. This is a fairly good approximation of how sounds of dif-
ferent frequency are perceived by humans. Up to a 1000 Hz we can better
distinguish one frequency from another, but for higher frequency sounds this
ability degrades with the increasing frequency. The Mel scale gives infor-
mation about how small steps in frequency can be in order for humans to
perceive them as different frequency sounds. This information is used when
filtering the frequencies of the periodogram. By summing up nearby fre-
quencies into the closest of the distinguishable frequencies of the Mel scale,
the perceivable information of the sound under analyzing can be withdrawn.
Figure 3.3 shows how frequencies are filtered.

The standard number of filters applied is 26, but may vary between 20 to
40 filters. Once the periodogram is filtered, it is known how much energy
is present in each of the different frequency groups, also referred to as fil-
terbanks. The energy calculated to be present in each filterbank is then
logarithmized to create a set of log filterbank energies. This is made because
loudness is not perceived on a linear scale by the human ear. In general, to
perceive a sound to be double the volume of another, the energy put into it
has to be eight times as high.

The cepstral coefficients are finally acquired by taking the Discrete Cosine
Transform (DCT) of the log filterbank energies. The calculation of the DCTs
is needed because the filterbanks are overlapping, see Figure 3.3, making the
filterbank energies connected to each other. Taking the DCT of the log fil-
terbank energies decorrelates them so that they can be modeled with more
ease. Out of the 20 - 40 coefficients acquired from the filterbanks, only the
lower 12 - 13 are used in speech recognition applications. These are com-
bined into a feature vector that can serve as input to, for instance, a neural

25
Figure 3.3: The mel frequency filter applied to withdraw the perceivable
frequencies of the sound wave.

network. The reason to not use all of the coefficients is that other coefficients
have very little, or degrading, impact on the success rate of the recognition
systems.

As mentioned before, Delta and Delta-Delta features can be added to these

feature vectors for increased knowledge-base and performance. The Delta
coefficients are calculated using the equation below:
PM
m=1 m(ci+m − ci−m )
di = PM (3.3)
2 2
m=1 m
The Delta-Delta coefficients are calculated using the same equation, though
the static coefficients ci±M should be substituted by the Delta coefficients.

26
 Part III

Experiment Setup

27
Chapter 4

Model

This chapter will describe how experiments were implemented within this
research and what parameters and aids were used in order to carry the them
out.

4.1 Data Sets

For a neural network to be able to make classifications, it needs to be trained
on a set of data. The amount of data needed to accurately classify speakers
differs, depending on the application. When trying to recognize speakers
from a test set which is highly invariant with regards to recording circum-
stances, the training set needs to be bigger than if the test set recording
situations are similar. To clarify, recordings may have a smaller or greater
diversity when it comes to background noise or emotional stress, for instance.
So, a greater diversity in the test set comes with a demand for larger training
set. What is more, the need for a bigger training data set also increase with
the number of speakers you will need to recognize speech from.

The data sets used in this research were constituted of excerpts from au-
dio books. 21 books, narrated by ten different people, were chosen as the
research base. The first ten books were all randomly selected, one from each
of the narrators. From each of these ten books, an excerpt of ten minutes was
randomly withdrawn as to constitute the training data set. Thus, the total
training data set consisted of 100 minutes of speech divided evenly upon ten
different speakers. Out of these 100 minutes, one minute of speech from every
speaker were chosen at random to make up a validation set. The validation
set was used to test whether improvement had been made throughout the
training process. That way it could be determined early on in the training
if it was worth continuing with the same parameter setup. Time was of the
essence. Though chosen at random, the validation set remained the same
throughout the whole research. As did the training data set.

For the purpose of this thesis there were three different data sets used for

28
testing the ability of the network. The test sets used were completely set
apart from the training set. So, not a single frame were existing in both the
training set and any of the test sets. The first test set (1) was compound of
five randomly selected one minute excerpts from each of the ten books used
in the training data set. Thus it consisted of 50 minutes of speech, spread
evenly among the ten speakers.

The remaining two test sets were used to see if the network could actually
recognize the speakers voices in a slightly different context. So the narra-
tors were all the same, but the books were different from the ones used in
the training set. The second test set (2) consisted of five randomly cho-
sen one minute excerpts from eight different books, narrated by eight of the
ten speakers. In total test set (2) consisted of 40 minutes of speech that
were evenly spread out among eight speakers. The third test set (3) was
the smallest one and consisted of five randomly selected one minute excerpts
from three of the speakers. Thus it was compound of 15 minutes of speech,
spread evenly across three narrators. These excerpts were withdrawn from
three books, different from the ones used in the other data sets. They were
books that came from the same series of books as some of the ones used
in the training set. In that sense it was thought that they would be more
similar to the ones used for training. Therefore it was the author’s belief
that test set (3) might be of less challenge for the network than test set (2),
but still a bigger challenge than (1).

The narrators of the selected books were all adult males. It was thought
that speakers of the same sex would be a greater challenge for the network,
compared to doing the research with a research base of mixed female and
male speakers. The language spoken on all of the audio books is English.
However, some speakers use a British accent and some an American. The
excerpts contained emotionally colored speech. All the audio files used were
studio recorded. Thus, they would not represent a real life situation with
regards to background noise, for example.

4.2 Feature Extraction

The sound waves must be processed and converted into a set of discrete fea-
tures that can be used as input to the LSTM neural network. In this thesis,
the features withdrawn from the sound waves were Mel Frequency Cepstral
Coefficients (MFCC) together with their differentials and accelerations, i.e
Delta and Delta-Delta coefficients. By their characteristics they represent
features of speech signals that are important for the phonetic information.
These features are withdrawn from the short time power spectrum and rep-
resent the characteristics of the signal that are emotionally independent.

The features were extracted from the sound waves by processing a 25 mil-
lisecond window of the signal. This 25 millisecond window form a frame.
The window were then moved 10 milliseconds at a time until end of the sig-

29
nal. Thus, the frames overlap each other to lessen the risk of information
getting lost in the transition between frames. From every frame, 13 MFCC
coefficients were extracted using a set of 26 filter-bank channels. To better
model the behavior of the signal, the differentials and accelerations of the
MFCC coefficients were calculated. All these features were combined into a
feature vector of size 39. The feature vectors served as input to the neural
network.

The feature extraction was made using the Hidden Markov Model Toolkit
(HTK) [35]. This library can be used on its own as a speech recognition
software, making use of Hidden Markov Models. However, only the tools
regarding MFCC extraction were used during this research. Specifically the
tools HCOPY and HLIST were used to extract the features and aid in the
creation of data sets.

4.3 Neural Network

A Recurrent Neural Network (RNN) was used to execute the speaker recog-
nition. The specific type of neural network implemented for the purpose of
this thesis was a bidirectional Long Short-Term Memory (BLSTM) RNN.
This type of network is a biologically plausible model of a neural network
that has a proven capability to store long range contextual information. That
way, it is possible to learn to bridge long time gaps between rarely occurring
events.

The difference between an ordinary RNN and an LSTM RNN lies within
the hidden layer of the neural network. Ordinary hidden layer units are ex-
changed with LSTM memory blocks. The memory blocks consists of at least
one memory cell, an input gate, an output gate and a forget gate. In this
research only one memory cell per memory block where used. The memory
cell was constituted of a linear unit whereas the gates where made up of
sigmoid units. Also the input and output squashing functions were sigmoid
functions. All of the sigmoid units ranged from -1 to 1. The activation of
the gates controlled the input to, and output of, the memory cell via mul-
tiplicative units. So, for example, the memory cells output was multiplied
with the output gates activation as to give the final output of the memory
block.

As for the network architecture, the network consisted of an input layer,

a number of hidden layers and an output layer. The input layer was of size
39, so that a whole feature vector was input to the network at once. That
is, every feature coefficient corresponded to one node in the input network.
The hidden layer was constituted by different setups of recurrently connected
LSTM memory blocks. The number of memory blocks, as well as the number
of hidden layers, were the parameters experimented with within the scope of
this thesis.

30
To create the neural network, a library called RNNLIB [18] was used. The
library was developed by Alex Graves, one of the main contributors to the
LSTM structure of today. It provided the necessary configuration possibili-
ties for the purpose of this thesis in a convenient way.

During the experiments the feature vectors were input to the network, one
vector at a time. The feature vectors correlated with one of the audio files,
from one speaker, were seen as a whole sequence that corresponded to one
target. Thus, the network was trained so that every sequence of speech
had one target speaker. So the network was used for sequence classification
rather than frame-wise classification.

31
Chapter 5

Experiments

This chapter will describe which experiments were carried out during this
thesis.

5.1 Does size matter?

In this first experiment it was tested whether the size and depth of the neu-
ral network matters. The depth is the number of hidden layers whereas the
size is the number of memory blocks within the hidden layers of the LSTM
network. Adding more hidden layers and more memory blocks within them
increases the complexity of the network. Simply adding one extra memory
block will add a large amount of extra weights to the network because of all
the recurrent connections.

It was the author’s belief that the higher network complexity would bet-
ter model a problem like this. However, it was thought that, at some point,
the increasing of network complexity would rather slow computations down
than aid in making the classifications more correct.

These experiments were executed with three data sets: one for training,
one for validation and one for testing. All audio was excerpted from ten
audio books. The training data set consisted of 10 minutes of speech from
ten different speakers. Totally 100 minutes of speech. The validation set
was a compound of one minute of speech from the same speakers. It was a
subset of the training set. The test set consisted of five minutes of speech
from each of the ten speakers. Totally 50 minutes of speech from the same
books as those in the training set. The training set and test set did not
overlap, though.

32
 Table 5.1: Network setups

Network Hidden Layers Memory Blocks

LSTM2x5 2 5,5
LSTM2x25 2 25,25
LSTM2x50 2 50,50
LSTM4x5 4 5,5,5,5
LSTM4x25 4 25,25,25,25
LSTM4x50 4 50,50,50,50
LSTM6x5 6 5,5,5,5,5,5
LSTM6x25 6 25,25,25,25,25,25
LSTM6x50 6 50,50,50,50,50,50

5.2 Will the classifications be robust?

The second question that this research was searching for an answer to was
how robust the network was. In other words, will the network be able to
accurately classify the speakers voices in a slightly different context? In this
thesis the different context was represented by other books.

These experiments were executed with four data sets: one for training, one
for validation and two for testing. All audio was excerpted from 21 au-
dio books. The training data set was made up from ten audio books and
consisted of ten minutes of speech from ten different speakers. Totally 100
minutes of speech. The validation set was a compound of one minute of
speech from the same speakers. It was a subset of the training set. Both
the training and validation set in this experiment were similar to the ones in
the previous experiments. However, this experiment was executed with two
test sets that where different from the other experiments. This was thought
to test the robustness of the network’s ability to correctly classify who was
speaking.

The first of the test sets used within this experiment consisted of five minutes
of speech from eight of the ten speakers. Totally 40 minutes of speech. These
excerpts came from a set of books different from the ones used to build the
training set. The books where written by other authors, but had a similar
genre. This set was thought to be of greatest challenge to the network as
the books were in a completely different setting from those in the training set.

The second of the test sets used within this experiment consisted of five
minutes of speech from three of the ten speakers. Totally 15 minutes of
speech. These excerpts came from a set of books different from the ones
used to build the training set and other test sets. The books where written
by the same authors as those in the training set. They were in the same

33
genre and even the same series of books. Therefore, it was thought that
they would be quite similar to the books used in the training set. This set
was thought to be of a smaller challenge to the network as the books had a
similar setting to those in the training set.

For these experiments the network architecture found to give the best results
in the previous experiments was used, LSTM4x25 figure 6.5. It consisted of
39 input units, four hidden layers and softmax output layer.

34
 Part IV

Results and Discussion

35
Chapter 6

Results

This chapter will state all the results from the experiments carried out within
the scope of this research.

6.1 Size/Depth Experiments

During the depth and size experiments it was investigated whether the num-
ber of hidden layers as well as the number of memory blocks within them
would affect the classification results in any way. Nine different LSTM neu-
ral network setups were tested on a speaker identification task. The setups
tested consisted two, four and six hidden layers with 5, 25 and 50 memory
blocks within each layer. Every hidden layer setup was tested with each
number of memory blocks. Apart from the hidden layers, the networks also
made use of an input and an output layer.

There were ten speakers used for the speaker identification task. The train
set contained ten minutes of speech from each speaker, divided into 5-minute
excerpts. The validation set contained one 1-minute sample of speech from
each speaker, taken from the train set. The test set consisted of 5 minutes of
speech from each speaker, divided into 1-minute excerpts. None of the sam-
ples used in the test set were present in the set the networks were trained on.
However, they were withdrawn from the same audio books. The experiments
were discontinued when the improvement of the training had stagnated or
reached perfect results.

Generally it can be said, by looking at the results, that greater network

depth did degrade the classification results. From figure 6.1, figure 6.2 and
figure 6.3 it can be seen that the networks using six hidden layers performed
poorly on the speaker recognition task. Independent of the size of the hid-
den layers. Among these, the lowest classification error was reached with
LSTM6x50, seen in figure 6.3. Still, it only reached down to a classifica-
tion error of 80 percent on the validation set, which is not usable for any

36
Figure 6.1: A graph over how classification errors change during training of
an LSTM network using six hidden layers consisting of five memory blocks
each.

application. The LSTM6x50 network was, in contrast to the other LSTM6

networks, at least able to learn to adapt weights to the training set.

The increasing network depth also came with excessive training times. Not
only did the time to train a network on one epoch of training data increase,
but also the number of epochs required to reach a good result. Especially
for the training of the more complex six hidden layer networks, which took
around ten hours to execute on a mid range laptop computer. This is worth
mentioning as the networks with just two hidden layers were trained in below
15 minutes.

Taking a look at figures 6.4, 6.5 and 6.6 shows that the networks with a
depth of four performs better than those with a depth of six. LSTM4x25,
figure 6.5, was able to identify all ten speakers correctly within the validation
set after 100 epochs of training. However, the network’s classification error
on the test set did stagnate at 6 percent. Thus 47 out of the 50 1-minute
samples was correctly classified as one of the ten speakers by the LSTM4x25
network.

37
Figure 6.2: A graph over how classification errors change during training of
an LSTM network using six hidden layers consisting of 25 memory blocks
each.

The LSTM4x50 network, figure 6.6, made progress a lot faster than LSTM4x25,
reaching a 0 percent classification error on the validation set after only 40
epochs of training. On the test set, however, the LSTM4x50 network man-
aged to obtain 12 percent classification error after 48 epochs of training,
which can be considered a fairly good result. It corresponds to 44 correctly
classified 1-minute samples out of 50.

Evaluating the performance of the LSTM4x5 network, figure 6.4, it can be

seen that the performance is as poor as with the deeper networks. Despite
excessive training, the network’s classification error did not go down any fur-
ther than to 80 percent on the validation set. The same results were reached
on the training and test set. Thus the network was not able to adapt its
weights to the training set, which says that this network setup did not fit
well to model this problem.

The most well performing network setups, with regards to the validation set,
turned out to be the ones using only two hidden layers, figures 6.7, 6.8

38
Figure 6.3: A graph over how classification errors change during training of
an LSTM network using six hidden layers consisting of 50 memory blocks
each.

and 6.9. They were also the network setups that needed the least amount
of training. The LSTM2x5 network, figure 6.7, did achieve 100 percent cor-
rect classifications within the validation set after only 19 epochs of training.
LSTM2x50 was not long after as it reached it in 20 epochs. Nevertheless,
none of these networks were able to reach perfect results on the test set,
where they reached a minimum classification error of 8 and 24 percent re-
spectively. These results correspond to 46 and 38 correctly classified speaker
samples out of 50. The LSTM2x5 result can be seen as a good achievement.

The LSTM2x25 network, figure 6.8, made the fastest progress of all net-
work setups. Though, the progress stagnated and did not give any less
classification error than 10 percent on the validation set. This result was
obtained after nine epochs of training, however. The minimum classification
error achieved on the test set was 14 percent. This corresponds to 43 out of
50 correct classifications on the 1-minute samples, which is fairly good.

39
Figure 6.4: A graph over how classification errors change during training of
an LSTM network using four hidden layers consisting of five memory blocks
each.

6.2 Robustness Experiments

The robustness experiments were run as to examine whether the speaker
identification system created was able to identify the speakers, which it had
been trained to identify, in a slightly different context. The different context
was within this research, as working with excerpts from audio books, exem-
plified by the use of audio books different from the ones used for training of
the networks. For these experiments, the network setup with the best results
from the previous size/depth experiments, LSTM4x25, was used.

In the first experiment the network was tested on a set of 40 1-minute ex-
cerpts divided among eight of the ten speakers present in the training set.
This was thought to be the hardest of all tests as the samples were with-
drawn from books that were completely set apart from the ones utilized in
the training set. The result of this experiment were extremely poor. Only
two out of the 40 1-minute samples were correctly classified, thus giving a
classification error of 95 percent.

During the second experiment it was investigated whether the LSTM4x25

40
Figure 6.5: A graph over how classification errors change during training of
an LSTM network using four hidden layers consisting of 25 memory blocks
each.

network could identify the speakers’ voices within one additional data set.
This time the set consisted of 15 1-minute samples that were excerpted from
three books that were narrated by three of the speakers on which the system
had been trained on. These books were in the same series as the ones used
for training and therefore were of the same genre and writing style. There-
fore it was thought that this set would be of less challenge than the previous
one. Nevertheless, the results from this experiment were even worse than in
the first. Not a single one of the 15 voice samples was recognized properly
by the network. So the classification error was 100 percent. Thus it turned
out that the network was not able to perform well in any of these tasks.

41
Figure 6.6: A graph over how classification errors change during training of
an LSTM network using four hidden layers consisting of 50 memory blocks
each.

42
Figure 6.7: A graph over how classification errors change during training of
an LSTM network using two hidden layers consisting of five memory blocks
each.

43
Figure 6.8: A graph over how classification errors change during training of
an LSTM network using two hidden layers consisting of 25 memory blocks
each.

44
Figure 6.9: A graph over how classification errors change during training of
an LSTM network using two hidden layers consisting of 50 memory blocks
each.

45
Chapter 7

Discussion

In this chapter the results of the research will be discussed and some con-
clusions will be drawn from the results. The chapter will also cover what
possible future work within this area could be.

7.1 Discussion of Results

During this research a text-independent speaker identification system has
been built and the performance of it has been evaluated. The system was
implemented with a neural network approach, with the use of a Bidirectional
Long Short-Term Memory network. To the author’s knowledge, this type of
recurrent neural network has never been utilized within the speaker recog-
nition area until this day, but has previously proven to be successful within
other applications in which the learning of temporal information is impor-
tant.

The system was trained and tested against a database of ten speakers. The
data sets were made up of excerpts from audio books. A total of 21 books
were used within this research. The audio files were processed into Mel
Frequency Cepstral Coefficients and their first and second derivatives as to
create feature vectors that were used as inputs to the neural network. Each
vector contained 39 features that were withdrawn from the short time power
spectrum of the audio signals.

During the experiments it was investigated whether the size and depth of
the neural network would have any effect on its ability to identify speakers.
Nine network setups were tested, making use of two, four and six hidden
layers and 5, 25 and 50 memory blocks within each of them. It turned out
that, within this application at least, a greater depth rather degrades the
performance of the system than enhances it. The networks using six hidden
layers did all perform badly, independent of size, even though the one us-
ing 50 hidden blocks did perform the least bad. Among the networks using
four hidden layers, the smallest network did not perform well. However, the

46
larger ones gave good results. In fact, the network using 25 hidden blocks
gave the best results out of all network setups, achieving an identification
rate of 94 percent on the test set and 100 percent on the validation set.

Overall, the network setups using only two hidden layers performed the best.
Among these networks the smaller ones proved to give better results though.
So, of the least deep networks, the smallest network performed the best,
among the middle-depth networks the middle size gave the best results and
among the deepest networks, the largest size performed the best. Thus it
seems that the size of the network does not on it self affect the performance
of the system, but in conjunction with depth, it does. So, to create a well
functioning system one needs to match the size and the depth of the network.
It is also the author’s belief that it is important to match the complexity of
the network with the complexity of the database and problem.

Another thing found during the experiments is that training time will be
heavily affected by the complexity of network. The network complexity, i.e.
number of weights, is in turn mostly dependent on the depth of the network.
The difference in training time between the least complex and the most com-
plex network was around ten hours for the same number of training epochs.
The general performance of the four and six layer networks was also lower
than with the less deep networks. Thus, if time is of the essence, it is not
really worth going for a more complex network model. Anyhow, four hidden
layers seems to be some kind of roof for the network to even sort a problem
of this size out.

The experiments carried out to try how well the network could recognize
the speakers in a slightly different context did not achieve satisfactory re-
sults, mildly said. No more than five percent of the audio samples were
classified properly, which obviously is not good enough. So, the robustness
of the networks speaker recognition ability was low. Nevertheless, the net-
work could identify the speakers with sound accuracy within the previous
experiments, where training and test data sets also were different from each
other, but built from the same recordings. It is the author’s belief that the
MFCC’s sensitivity to noise is the reason for this. Probably the background
noise and "setting" of each recording did affect the MFCCs too much for the
network to be able to identify the same speaker in two different recordings.

To sum things up; the Bidirectional Long Short-Term Memory neural net-
work algorithm proved to be useful also within the speaker recognition area.
Without any aid in the identification process, an BLSTM network could,
from MFCCs, identify speakers text-independently with 93 percent accuracy
and text-dependently with a 100 percent accuracy. This was done with a net-
work using four hidden layers containing 25 memory blocks each. It should
be noted that if the recordings are too different from each other, when it
comes to background noise etc., the network will have big problems identi-
fying speakers accurately. Therefore, if great accuracy is needed and audio

47
data is of diverse quality, this type of system, alone, may not be suitable for
text-independent speaker identification without further speech processing.

7.2 Future Work

Future work in this area could be to extend the data sets with more speakers.
It would be interesting to see if there is some kind of a limit to how many
speakers it is possible to recognize, in a situation where computational power
will not limit the size of the network or give undesirably long training times.
Also it would be interesting to see how well the network performs with less
available audio data for training and testing.

What is more, future work could be to investigate whether normalizing the

audio data would aid with the issue the network has to identify speakers
in between recordings. It is thought that this technique would increase the
robustness of a system like this one.

Another interesting possibility that would be fun to explore further is music

recognition using a system similar to the one created within this thesis.

48
Bibliography

[1] A. A M Abushariah, T.S. Gunawan, O.O. Khalifa, and M. A M

Abushariah. English digits speech recognition system based on hidden
markov models. In Computer and Communication Engineering (IC-
CCE), 2010 International Conference on, pages 1–5, May 2010. doi:
10.1109/ICCCE.2010.5556819.

[2] A. Alwan. Modeling speech production and perception mechanisms

and their applications to synthesis, recognition, and coding. In Signal
Processing and Its Applications, 1999. ISSPA ’99. Proceedings of the
Fifth International Symposium on, volume 1, pages 7 vol.1–, 1999. doi:
10.1109/ISSPA.1999.818096.

[3] M. Aymen, A. Abdelaziz, S. Halim, and H. Maaref. Hidden markov

models for automatic speech recognition. In Communications, Comput-
ing and Control Applications (CCCA), 2011 International Conference
on, pages 1–6, March 2011. doi: 10.1109/CCCA.2011.6031408.
[4] Homayoon Beigi. Fundamentals of speaker recognition. Springer, 2011.

[5] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term
dependencies with gradient descent is difficult. Neural Networks, IEEE
Transactions on, 5(2):157–166, 1994.
[6] Joseph P Campbell Jr. Speaker recognition: A tutorial. Proceedings of
the IEEE, 85(9):1437–1462, 1997.

[7] A.E. Coca, D.C. Correa, and Liang Zhao. Computer-aided music com-
position with lstm neural network and chaotic inspiration. In Neural
Networks (IJCNN), The 2013 International Joint Conference on, pages
1–7, Aug 2013. doi: 10.1109/IJCNN.2013.6706747.

[8] S. Davis and P. Mermelstein. Comparison of parametric representa-

tions for monosyllabic word recognition in continuously spoken sen-
tences. Acoustics, Speech and Signal Processing, IEEE Transactions
on, 28(4):357–366, Aug 1980. ISSN 0096-3518. doi: 10.1109/TASSP.
1980.1163420.

[9] John R Deller, John G Proakis, and John HL Hansen. Discrete-time

processing of speech signals. IEEE New York, NY, USA:, 2000.

49
[10] Nidhi Desai, Prof Kinnal Dhameliya, and Vijayendra Desai. Feature ex-
traction and classification techniques for speech recognition: A review.
International Journal of Emerging Technology and Advanced Engineer-
ing, 3(12), December 2013.
[11] Douglas Eck and Jürgen Schmidhuber. A first look at music composition
using lstm recurrent neural networks. Istituto Dalle Molle Di Studi Sull
Intelligenza Artificiale, 2002.
[12] F. Eyben, S. Petridis, B. Schuller, G. Tzimiropoulos, S. Zafeiriou, and
M. Pantic. Audiovisual classification of vocal outbursts in human con-
versation using long-short-term memory networks. In Acoustics, Speech
and Signal Processing (ICASSP), 2011 IEEE International Conference
on, pages 5844–5847, May 2011. doi: 10.1109/ICASSP.2011.5947690.
[13] F.A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: con-
tinual prediction with lstm. In Artificial Neural Networks, 1999. ICANN
99. Ninth International Conference on (Conf. Publ. No. 470), volume 2,
pages 850–855 vol.2, 1999. doi: 10.1049/cp:19991218.
[14] Felix A. Gers, Nicol N. Schraudolph, and Jürgen Schmidhuber.
Learning precise timing with lstm recurrent networks. J. Mach.
Learn. Res., 3:115–143, March 2003. ISSN 1532-4435. doi:
10.1162/153244303768966139. URL http://dx.doi.org/10.1162/
153244303768966139.
[15] A. Graves and J. Schmidhuber. Framewise phoneme classification with
bidirectional lstm networks. In Neural Networks, 2005. IJCNN ’05.
Proceedings. 2005 IEEE International Joint Conference on, volume 4,
pages 2047–2052 vol. 4, July 2005. doi: 10.1109/IJCNN.2005.1556215.
[16] A. Graves, N. Jaitly, and A.-R. Mohamed. Hybrid speech recogni-
tion with deep bidirectional lstm. In Automatic Speech Recognition and
Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278, Dec
2013. doi: 10.1109/ASRU.2013.6707742.
[17] A. Graves, A.-R. Mohamed, and G. Hinton. Speech recognition with
deep recurrent neural networks. In Acoustics, Speech and Signal Process-
ing (ICASSP), 2013 IEEE International Conference on, pages 6645–
6649, May 2013. doi: 10.1109/ICASSP.2013.6638947.
[18] Alex Graves. Rnnlib: A recurrent neural network library for sequence
learning problems. http://sourceforge.net/projects/rnnl/.
[19] Alex Graves and Jürgen Schmidhuber. Offline handwriting recogni-
tion with multidimensional recurrent neural networks. In D. Koller,
D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural
Information Processing Systems 21, pages 545–552. MIT Press, 2009.
[20] Alex Graves, Douglas Eck, Nicole Beringer, and Jürgen Schmidhuber.
Biologically plausible speech recognition with lstm neural nets. In in
Proc. of Bio-ADIT, pages 127–136, 2004.

50
[21] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.
1997.
[22] Manish P Kesarkar. Feature extraction for speech recognition. Elec-
tronic Systems, EE. Dept., IIT Bombay, 2003.
[23] S. Chougule M. Chavan. Speaker features and recognition techniques: A
review. International Journal Of Computational Engineering Research,
May-June 2012(3):720–728, 2012.
[24] Derek Monner and James A Reggia. A generalized lstm-like training
algorithm for second-order recurrent neural networks. Neural Networks,
25:70–83, 2012.
[25] Climent Nadeu, Dušan Macho, and Javier Hernando. Time and fre-
quency filtering of filter-bank energies for robust hmm speech recogni-
tion. Speech Communication, 34(1):93–114, 2001.
[26] Tuija Niemi-Laitinen, Juhani Saastamoinen, Tomi Kinnunen, and Pasi
Fränti. Applying mfcc-based automatic speaker recognition to gsm and
forensic data. In Proc. Second Baltic Conf. on Human Language Tech-
nologies (HLT’2005), Tallinn, Estonia, pages 317–322, 2005.
[27] Lawrence J Raphael, Gloria J Borden, and Katherine S Harris. Speech
science primer: Physiology, acoustics, and perception of speech. Lippin-
cott Williams & Wilkins, 2007.
[28] A. J. Robinson and F. Fallside. The utility driven dynamic error prop-
agation network. Technical report, 1987.
[29] D. Rumelhart and J. McClelland. Parallel Distributed Processing: Ex-
plorations in the Microstructure of Cognition: Foundations. MIT Press,
1987.
[30] Haşim Sak, Andrew Senior, and Françoise Beaufays. Long short-term
memory based recurrent neural network architectures for large vocabu-
lary speech recognition. arXiv preprint arXiv:1402.1128, 2014.
[31] Urmila Shrawankar and Vilas M Thakare. Techniques for feature extrac-
tion in speech recognition system: A comparative study. arXiv preprint
arXiv:1305.1145, 2013.
[32] Stanley S Stevens and John Volkmann. The relation of pitch to fre-
quency: a revised scale. The American Journal of Psychology, 1940.
[33] Tharmarajah Thiruvaran, Eliathamby Ambikairajah, and Julien Epps.
Fm features for automatic forensic speaker recognition. In INTER-
SPEECH, pages 1497–1500, 2008.
[34] M. Wollmer, F. Eyben, J. Keshet, A. Graves, B. Schuller, and G. Rigoll.
Robust discriminative keyword spotting for emotionally colored sponta-
neous speech using bidirectional lstm networks. In Acoustics, Speech and

51
 Signal Processing, 2009. ICASSP 2009. IEEE International Conference
on, pages 3949–3952, April 2009. doi: 10.1109/ICASSP.2009.4960492.
[35] Steve J Young and Sj Young. The HTK hidden Markov model toolkit:
Design and philosophy. Citeseer, 1993.
[36] Willard R Zemlin. Speech and hearing science, anatomy and physiology.
1968.

52