CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky

Lecture 10: Acoustic Modeling

IP Notice:

Outline for Today

Speech Recognition Architectural Overview Hidden Markov Models in general and for speech
Forward Viterbi Decoding

How this fits into the ASR component of course

Jan 27 HMMs, Forward, Viterbi, Jan 29 Baum-Welch (Forward-Backward) Feb 3: Feature Extraction, MFCCs, start of AM (VQ) Feb 5: Acoustic Modeling: GMMs Feb 10: N-grams and Language Modeling Feb 24: Search and Advanced Decoding Feb 26: Dealing with Variation Mar 3: Dealing with Disfluencies

Outline for Today

Acoustic Model
Increasingly sophisticated models Acoustic Likelihood for each state:
Gaussians Multivariate Gaussians Mixtures of Multivariate Gaussians CI Subphone (3ish per phone) CD phone (=triphones) State-tying of CD phone

Where a state is progressively:

If Time: Evaluation
Word Error Rate

Reminder: VQ
To compute p(ot|qj)
Compute distance between feature vector ot
and each codeword (prototype vector) in a preclustered codebook where distance is either Euclidean Mahalanobis and take its codeword vk

Choose the vector that is the closest to ot And then look up the likelihood of vk given HMM state j in the B matrix

Bj(ot)=bj(vk) s.t. vk is codeword of closest vector to ot Using Baum-Welch as above

Computing bj(vk)

Slide from John-Paul Hosum, OHSU/OGI

feature value 2 for state j feature value 1 for state j bj(vk) = number of vectors with codebook index k in state j = 14 = 1 number of vectors in state j 56 4

Summary: VQ
Training:
Do VQ and then use Baum-Welch to assign probabilities to each symbol

Decoding:
Do VQ and then use the symbol probabilities in decoding

Directly Modeling Continuous Observations

Gaussians
Univariate Gaussians
Baum-Welch for univariate Gaussians

Multivariate Gaussians
Baum-Welch for multivariate Gausians

Gaussian Mixture Models (GMMs)

Baum-Welch for GMMs

Better than VQ
VQ is insufficient for real ASR Instead: Assume the possible values of the observation feature vector ot are normally distributed. Represent the observation likelihood function bj(ot) as a Gaussian with mean j and variance j2

1 ( x ) f ( x | , ) = exp( ) 2 2 2

Gaussians are parameters by mean and variance

Reminder: means and variances

For a discrete random variable X Mean is the expected value of X
Weighted sum over the values of X

Variance is the squared average deviation from mean

Gaussian as Probability Density Function

Gaussian PDFs
A Gaussian is a probability density function; probability is area under curve. To make it a probability, we constrain area under curve = 1. BUT
We will be using point estimates; value of Gaussian at point.

Technically these are not probabilities, since a pdf gives a probability over a interval, needs to be multiplied by dx As we will see later, this is ok since the same value is omitted from all Gaussians, so argmax is still correct.

Gaussians for Acoustic Modeling

A Gaussian is parameterized by a mean and a variance: Different means

P(o|q):
P(o|q)

P(o|q) is highest here at mean

P(o|q is low here, very far from mean)

Using a (univariate Gaussian) as an acoustic likelihood estimator

Lets suppose our observation was a single real-valued feature (instead of 39D vector) Then if we had learned a Gaussian over the distribution of values of this feature We could compute the likelihood of any given observation ot as follows:

Training a Univariate Gaussian

A (single) Gaussian is characterized by a mean and a variance Imagine that we had some training data in which each state was labeled We could just compute the mean and variance from the data:

1 i = ot s.t . ot is state i T t =1
T 1 i 2 = (ot i ) 2 s.t . qt is state i T t =1

Training Univariate Gaussians

But we dont know which observation was produced by which state! What we want: to assign each observation vector ot to every possible state i, prorated by the probability the the HMM was in state i at time t. The probability of being in state i at time t is t(i)!!
T

(i)o
t

T t 2 ( i )( o ) t t i t =1 T

i =

t =1 T

(i)
t t =1

2i =

(i)
t t =1

Multivariate Gaussians
Instead of a single mean and variance :
1 ( x ) 2 f ( x | , ) = exp( ) 2 2 2

Vector of observations x modeled by vector of means and covariance matrix 1 1 T 1 f ( x | , ) = exp ( x ) ( x ) D /2 1/ 2 2 (2 ) | |

Multivariate Gaussians
Defining and

= E ( x)
= E [( x )( x )
T

So the i-jth element of is:

= E [( x i i )( x j j )]
2 ij

Gaussian Intuitions: Size of

= [0 0] = [0 0] = [0 0] = I = 0.6I = 2I As becomes larger, Gaussian becomes more spread out; as becomes smaller, Gaussian more Text compressed and figures from Andrew Ngs lecture notes for CS229

From Chen, Picheny et al lecture slides

[1 0] [0 1]

[.6 0] [ 0 2]

Different variances in different dimensions

Gaussian Intuitions: Offdiagonal

As we increase the off-diagonal entries, more correlation between value of x and value of y

Text and figures from Andrew Ngs lecture notes for CS229

Gaussian Intuitions: offdiagonal

As we increase the off-diagonal entries, more correlation between value of x and value of y

Text and figures from Andrew Ngs lecture notes for CS229

Gaussian Intuitions: offdiagonal and diagonal

Decreasing non-diagonal entries (#1-2) Increasing variance of one dimension in diagonal (#3)

Text and figures from Andrew Ngs lecture notes for CS229

In two dimensions

From Chen, Picheny et al lecture slides

But: assume diagonal covariance

I.e., assume that the features in the feature vector are uncorrelated This isnt true for FFT features, but is true for MFCC features, as we saw las time Computation and storage much cheaper if diagonal covariance. I.e. only diagonal entries are non-zero Diagonal contains the variance of each dimension ii2 So this means we consider the variance of each acoustic feature (dimension) separately

Diagonal covariance
Diagonal contains the variance of each dimension ii2 So this means we consider the variance of each acoustic feature (dimension) 2 separately D

b j (ot ) =
d =1

o 1 td jd exp 2 2 jd 2 jd 1
D

b j (ot ) = 2

1
D 2

d =1

jd

1 exp( 2 d =1 2

(otd jd ) 2

jd

Baum-Welch reestimation equations for multivariate Gaussians

Natural extension of univariate case, where now i is mean vector for state i:
T

i =

(i)o
t t =1 T

(i)
t t =1

i2 =

(i)(o
t t =1

t T

i )(ot i )
t

(i)
t =1

But were not there yet

Single Gaussian may do a bad job of modeling distribution in any dimension:

Solution: Mixtures of Gaussians

Figure from Chen, Picheney et al slides

Mixture of Gaussians to model a function

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0 4

Mixtures of Gaussians
M mixtures of Gaussians:

1 1 T 1 f ( x | jk , jk ) = c jk exp ( x jk ) ( x jk ) D /2 1/ 2 2 (2 ) | jk | k =1
For diagonal covariance:
M D 2 o 1 td jkd exp 2 jkd
D

b j (ot ) = c jk
k =1
M

1 2 2jkd
D

d =1

b j (ot ) =
k =1

c jk 2
D 2

d =1

jkd

1 exp( 2 d =1 2

( x jkd jkd ) 2

jkd

GMMs
Summary: each state has a likelihood function parameterized by:
M Mixture weights M Mean Vectors of dimensionality D Either
M Covariance Matrices of DxD

Or more likely
M Diagonal Covariance Matrices of DxD which is equivalent to M Variance Vectors of dimensionality D

Training a GMM
Problem: how do we train a GMM if we dont know what component is accounting for aspects of any particular observation? Intuition: we use Baum-Welch to find it for us, just as we did for finding hidden states that accounted for the observation

Baum-Welch for Mixture Models

By analogy with earlier, lets define the probability of being in state j at time t with the kth mixture component accounting for ot:
N

Now,

tm ( j ) =
T

t 1

( j )aij c jm b jm (ot ) j ( t )

i= 1

F (T )
T T tm

jm =

tm

( j )o t c jm =
tk

t =1 T M

( j) jm =
tk

t =1

tm

( j )(ot j )(ot j )T
T M tm

t =1 T M

t =1 k =1

( j)

t =1 k =1

( j)

t =1 k =1

( j)

2/4/09

34

How to train mixtures?

1) 2) 3) 4) Choose M (often 16; or can tune M dependent on amount of training observations) Then can do various splitting or clustering algorithms One simple method for splitting: Compute global mean and global variance Split into two Gaussians, with means (sometimes is 0.2) Run Forward-Backward to retrain Go to 2 until we have 16 mixtures

2/4/09

35

Embedded Training
Components of a speech recognizer:
Feature extraction: not statistical Language model: word transition probabilities, trained on some other corpus Acoustic model:
Pronunciation lexicon: the HMM structure for each word, built by hand Observation likelihoods bj(ot) Transition probabilities aij

2/4/09

CS 224S Winter 2007

36

Embedded training of acoustic model

If we had hand-segmented and hand-labeled training data With word and phone boundaries We could just compute the
B: means and variances of all our triphone gaussians A: transition probabilities

And wed be done! But we dont have word and phone boundaries, nor phone labeling

2/4/09

CS 224S Winter 2007

37

Embedded training
Instead:
Well train each phone HMM embedded in an entire sentence Well do word/phone segmentation and alignment automatically as part of training process

2/4/09

CS 224S Winter 2007

38

Embedded Training

2/4/09

CS 224S Winter 2007

39

Initialization: Flat start

Transition probabilities:
set to zero any that you want to be structurally zero Set the rest to identical values
The probability computation includes previous value of aij, so if its zero it will never change

Likelihoods:
initialize and of each state to global mean and variance of all training data

2/4/09

40

Embedded Training
Given: phoneset, pron lexicon, transcribed wavefiles
Build a whole sentence HMM for each sentence Initialize A probs to 0.5, or to zero Initialize B probs to global mean and variance Run multiple iteractions of Baum Welch
During each iteration, we compute forward and backward probabilities

Use them to re-estimate A and B Run Baum-Welch til converge

2/4/09 41

Viterbi training
Baum-Welch training says:
We need to know what state we were in, to accumulate counts of a given output symbol ot Well compute I(t), the probability of being in state i at time t, by using forward-backward to sum over all possible paths that might have been in state i and output ot.

Viterbi training says:

Instead of summing over all possible paths, just take the single most likely path Use the Viterbi algorithm to compute this Viterbi path Via forced alignment
2/4/09
CS 224S Winter 2007

42

Forced Alignment
Computing the Viterbi path over the training data is called forced alignment Because we know which word string to assign to each observation sequence. We just dont know the state sequence. So we use aij to constrain the path to go through the correct words And otherwise do normal Viterbi Result: state sequence!
2/4/09
CS 224S Winter 2007

43

Viterbi training equations

Viterbi
n ij ij = a ni
For all pairs of emitting states, 1 <= i, j <= N

Baum-Welch

n ( s . t . o = v ) j t k (v ) = b j k nj
Where nij is number of frames with transition from i to j in best path And nj is number of frames where state j is occupied
2/4/09
CS 224S Winter 2007

44

Viterbi Training
Much faster than Baum-Welch But doesnt work quite as well But the tradeoff is often worth it.

2/4/09

CS 224S Winter 2007

45

Viterbi training (II)

Equations for non-mixture Gaussians
1 T i = ot s.t . qt = i N i t =1

Viterbi training for mixture Gaussians is

1 2 i = ( ot i ) s.t. qt = i N i t =1
2

more complex, generally just assign each observation to 1 mixture

CS 224S Winter 2007

2/4/09 46

Log domain
In practice, do all computation in log domain Avoids underflow
Instead of multiplying lots of very small probabilities, we add numbers that are not so small.

Single multivariate Gaussian (diagonal ) compute:

In log space:

2/4/09

CS 224S Winter 2007

47

Log domain
Repeating:

With some rearrangement of terms

Where:

Note that this looks like a weighted Mahalanobis distance!!! Also may justify why we these arent really probabilities (point estimates); these are really just distances.

2/4/09

CS 224S Winter 2007

48

Evaluation
How to evaluate the word string output by a speech recognizer?

Word Error Rate

Word Error Rate = 100 (Insertions+Substitutions + Deletions) -----------------------------Total Word in Correct Transcript Aligment example: REF: portable **** PHONE UPSTAIRS last night so HYP: portable FORM OF STORES last night so Eval I S S WER = 100 (1+2+0)/6 = 50%

NIST sctk-1.3 scoring softare: Computing WER with sclite

http://www.nist.gov/speech/tools/ Sclite aligns a hypothesized text (HYP) (from the recognizer) with a correct or reference text (REF) (human transcribed)
id: (2347-b-013) Scores: (#C #S #D #I) REF: was an engineer HYP: was an engineer Eval: 9 3 1 2 SO I i was always with **** **** MEN UM and they ** AND i was always with THEM THEY ALL THAT and they D S I I S S

Sclite output for error analysis

CONFUSION PAIRS Total With >= (%hesitation) ==> on the ==> that but ==> that a ==> the four ==> for in ==> and there ==> that (%hesitation) ==> and (%hesitation) ==> the (a-) ==> i and ==> i and ==> in are ==> there as ==> is have ==> that is ==> this (972) 1 occurances (972) 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 6 6 5 4 4 4 4 3 3 3 3 3 3 3 3 3 -> -> -> -> -> -> -> -> -> -> -> -> -> -> -> ->

Sclite output for error analysis

17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 -> it ==> that -> mouse ==> most -> was ==> is -> was ==> this -> you ==> we -> (%hesitation) ==> -> (%hesitation) ==> -> (%hesitation) ==> -> (%hesitation) ==> -> a ==> all -> a ==> know -> a ==> you -> along ==> well -> and ==> it -> and ==> we -> and ==> you -> are ==> i -> are ==> were it that to yeah

Better metrics than WER?

WER has been useful But should we be more concerned with meaning (semantic error rate)?
Good idea, but hard to agree on Has been applied in dialogue systems, where desired semantic output is more clear

Summary: ASR Architecture

Five easy pieces: ASR Noisy Channel architecture
1) Feature Extraction:
39 MFCC features

2) Acoustic Model:
Gaussians for computing p(o|q)

3) Lexicon/Pronunciation Model
HMM: what phones can follow each other

4) Language Model 5) Decoder

N-grams for computing p(wi|wi-1) Viterbi algorithm: dynamic programming for combining all these to get word sequence from speech!
55

ASR Lexicon: Markov Models for pronunciation

Pronunciation Modeling

Generating surface forms:

Eric Fosler-Lussier slide

Dynamic Pronunciation Modeling

Slide from Eric Fosler-Lussier

Summary
Speech Recognition Architectural Overview Hidden Markov Models in general
Forward Viterbi Decoding

Hidden Markov models for Speech Evaluation