Adaptive Language Modeling
with a Set of Domain Dependent Models
Yangyang Shi, Pascal Wiggers, and Catholijn M. Jonker
Interactive intelligence Group, Delft University of Technology
Mekelweg 4, 2628CD, Netherlands
shiyang1983@gmail.com
Abstract. An adaptive language modeling method is proposed in this paper.
Instead of using one static model for all situations, it applies a set of specific
models to dynamically adapt to the discourse. We present the general structure
of the model and the training procedure. In our experiments, we instantiated the
method with a set of domain dependent models which are trained according to
different socio-situational settings (ALMOSD). We compare it with previous topic
dependent and socio-situational setting dependent adaptive language models and
with a smoothed n-gram model in terms of perplexity and word prediction
accuracy. Our experiments show that ALMOSD achieves perplexity reductions up
to almost 12% compared with the other models.
1 Introduction
A language model judges whether a sequence of words is a fluent sentence in a language
or not. Statistical language models do so by modeling probability distributions over
word sequences. This paper focuses on language models that adapt to the domain to
which they are applied.
Current state of the art statistical language models are smoothed n-grams, maximum
entropy models, and more recently recurrent neural networks [1]. All of these models
rely on a relatively small history of previous words to predict the next word in a
sentence and do not take into account the larger context of the conversation at hand.
In other words, these models assume that the same word distributions can be used in all
situations.
However, when dealing with different tasks or situations, people cognitively adapt
their language [2]. For example, consider the syntax and lexicon employed in a formal
article, versus a casual conversation. In this paper, we study models that take into
account the diversity and variability of language over context.
Taking contextual information into account is beneficial to language models, as
is shown by topic dependent and socio-situational setting dependent language models [3,4,5,6,7]. Such models typically interpolate between multiple domain specific language models. This leads to robust models that favor coherent discourses. The price for
this robustness is the fact that through interpolation the results of component models
that match the current discourse may be weighed down by non-matching components.
Therefore, a challenge is to create statistical language models that can dynamically select the domain specific models that best fit the current fragment of discourse. In this
way, it can avoid the impact of non-matching component.
P. Sojka et al. (Eds.): TSD 2012, LNCS 7499, pp. 472–479, 2012.
c Springer-Verlag Berlin Heidelberg 2012
Adaptive Language Modeling with a Set of Domain Dependent Models
473
Theoretically, better performance can be achieved if a specific model would be
available that matches the statistical regularities in the current discourse [8]. For
example, a model trained on texts on economics and marketing will get better results in
predicting texts from the Wall Street Journal than a general model.
Clearly, the chance of having a model that exactly matches the current discourse is
small, but what if we have available a rich set of models that correspond to different
topics and/or different types of discourse, can we then dynamically select one model
among those models most suited for the current discourse? And would this outperform
an interpolation approach? These are the questions investigated in this paper.
The paper is organized as follows. In the next section, related work is discussed. In
section 3, we give the theoretical background and motivation of our ALMOSD model,
and discuss the structure of the model as well as the procedures we used for training
a specific instance of this adaptive language modeling approach. In Section 4, we
compare a smoothed n-gram model, a topic dependent and a socio-situational setting
dependent language model with ALMOSD in terms of perplexity and word prediction
accuracy. Finally, based on the results, conclusions are drawn in Section 5.
2 Related Work
Several adaptive language modeling approaches have been proposed in the literature. [9] classifies the adaptation methods into three categories according to the underlying philosophy: model interpolation, constraint specification and meta-information
(e.g. semantic knowledge, topic knowledge, syntactic infra-structure) extraction.
In model interpolation, [10] proposed the class based n-gram language models. Their
adaptation strategy is to assign words with similar meaning and syntactic function into
one class.
In constraint specification adaptive language models should satisfy the features
extracted from the adaptation data. Exponential models trained using a maximum
entropy approach, separately assign different weights for each feature [11].
[6] and [12] proposed a mixture language models adaptation method. A collection
of sub-models are trained on separate pre-defined domains. Mixture language models
linearly interpolate these sub-models. However, as discussed in [13], in actual usage,
the mixture language model doesn’t work well, partly because of the complicated
smoothing. [7,14] explicitly model the domain as a variable in their language models.
In this case, it avoids complicated smoothing, as only one model needs to be trained.
In meta-information extraction, there has been much previous research in applying
topic information in language modeling [4,6].
All these adaptation methods finally generate one general language model to capture
the diversity of natural language. In ALMOSD, we use different models to represent
different domains of natural language.
3 The Model
As discussed by [15], psychophysical evidence for the existence of parallel processing
channels in human processing of speech has been found, especially in dealing with
474
Y. Shi, P. Wiggers, and C.M. Jonker
Table 1. Perplexity results of specific sub-models (SM), a smoothed trigram (ST), a topic
dependent (TM) and socio-situational setting dependent ( SSM) model
comp
a
b
c&d
e
f
g
h
i
j
k
l
m
n
o
socio-situational setting
Spontaneous conversations (’face-to-face’)
Interviews with teachers of Dutch
Spontaneous telephone dialogues
Simulated business negotiations
Interviews/ discussions/debates
(political) Discussions/debates/ meetings
Lessons recorded in the classroom
Live (eg sports) commentaries (broadcast)
Newsreports/reportages (broadcast)
News (broadcast)
Commentaries/columns/reviews (broadcast)
Ceremonious speeches/sermons
Lectures/seminars
Read speech
SM ST TM SSM
220
196
188
110
274
287
298
275
345
366
425
398
573
222
214
191
153
284
372
315
425
369
573
440
444
705
226
213
192
154
283
366
314
402
367
560
435
436
682
221
212
190
152
281
349
313
369
361
563
434
434
695
unexpected words. This evidence inspired us to design an adaptive language modeling
which can automatically select one from a set of domain dependent models.
The other source of inspiration is the behavior of language models. It is well
known that domain-specific language models perform much better on a given domain
than general-purpose models. This is illustrated in Table 1, where perplexity results
are shown for data from 14 different domains. Perplexity is a measurement of the
performance of language models. A better model returns lower perplexity on the same
test data set. The perplexity is calculated according to:
1
P P = 2− t
log P(w1 w2 ...wt )
,
(1)
where w1 w2 . . . wt is the data in the test set.
The results in the column marked SM in Table 1 are obtained by domain specific
models, trained on similar data from the same domain, whereas all results the column
marked ST are obtained by a smoothed trigram trained on data from all domains.
The columns marked TM and SSM show the results obtained with two sentence-level
mixture models the first of which contains topic-specific submodels found by automatic
clustering and the second of which uses the domain-specific models of column SM as
components (the details of these models will be explained in the next section).
These results suggest that a model that predicts the next word according to a
distribution that fits that of a domain specific model will outperform a static model as
well as mixture models. So, if k is the current domain and Pk a corresponding model
(e.g. on of the models listed in the third column of Table 1), then for an adaptive model
P̂ the following should hold:
P̂(wi |w1 . . . wi−1 ) = Pk (wi |w1 . . . wi−1 ),
(2)
Adaptive Language Modeling with a Set of Domain Dependent Models
475
where wi is the word at position i . Note that this formulation allows the selection of a
different k for every word in a discourse.
The question then is how to select the right specific model Pk at every point in time.
For this purpose we introduce the function () that predicts the domain k based on the
word history:
P̂(wi |(w1 . . . wi−1 ) = k) = Pk (wi |w1 . . . wi−1 ).
(3)
We chose to use the set of specific models themselves to implement this function, i.e.
the model that best matches the history seen sofar, is the one used to predict the next
word:
(4)
(w1 . . . wi−1 ) = arg max Pk (w1 . . . wi−1 ).
k
An alternative way to think of this model is as a model that puts all word histories
that best match a particular model k in one equivalence class. This implies that in
general, there is no need to use the sub-models themselves to select the appropriate
distribution for prediction, any suitable function of the word history will do. Also, there
is no restriction on the submodels that can be used, one could for example include a
general purpose model or mixture models as components as well to deal with those
cases in which no appropriate specific model is available.
3.1 Models Training
All sub-models used in this paper are interpolated trigrams trained with a two phase
procedure. For all models the same vocabulary is used. The process of training of the
sub-models is as follows:
Initial Training. Initially, a unigram, bigram and trigram model are trained on all
training data using MLE. Next these models are interpolated:
p̂(wi |wi−2 wi−1 ) = λ1 p(wi |wi−2 wi−1 ) + λ2 p(wi |wi−1 ) + λ3 p(wi ),
(5)
where wi is the i -th word in a sentence. The interpolation weights λ1 , λ2 and λ3 are
estimated using a held-out data set. This model is used as our baseline smoothed trigram
model (ST) and as a basis for all other models.
Sub-model Training. In the second phase, the complete data set is partitioned into a set
of sub-domains (Table 1). Each domain specific model is trained on the corresponding
subset. The final component based model is obtained by interpolating this model with
the general model of phase 1:
P(wi |wi−1 ...wi−n ) = θC PC (wi |wi−1 ...wi−n ) + θ S PS (wi |wi−1 ...wi−n ),
(6)
where θC + θ S = 1, θC , θ S ≥ 0, PC , PS represent the probability learned in complete
training and subset training, respectively.
In initial training the complete data trained models actually are the average of the
distributions of each subset. They avoid overfitting in some degree, but they also ignore
the characteristics of each subset. The sub-models which are the interpolation of the
complete data and subsets data, highlight the distribution of each subset. At the same
time, they are controlled by the complete data distribution to avoid overfitting.
476
Y. Shi, P. Wiggers, and C.M. Jonker
4 Experiments
4.1 Data
The Corpus Spoken Dutch (Corpus Gesproken Nederlands; CGN) [16,17], an 8 million
word corpus of contemporary Dutch spoken in Flanders and Netherlands is used in
our experiments. This data set is made up of 15 components, each related to a sociosituational setting. The socio-situational settings used in this paper are shown in Table 1.
In the experiment, 80% of the data in every component was randomly selected
for training, 10% for development testing and 10% for evaluation. A vocabulary with
44,368 words was created, which contains all unique words that occur more than once in
the training data. All words in the test data that are not in the vocabulary were replaced
by an out-of-vocabulary token.
4.2 Topic and Socio-situational Setting Dependent Language Models
We compared our model with a baseline smoothed trigram, but also with two mixture
models: one based on topic-dependent models (TM) that were found by automatic
clustering of the data and one based on the same set of socio-situational models
that make up the components of our model (SSM). Both are sentence level mixture
models [6]:
N
p(wi |w1 . . . wi−1 , ti ),
(7)
p(T )
p(w1,N ) =
T
i=1
and
p(T ) = p(t1 )
N
p(ti |ti−1 ),
(8)
i=2
where ti represents the topics or socio-situational settings at time i .
In these models, the current topic or socio-situational setting is dependent on the
previous one; the current word is dependent on a history of two words and the previous
topic or socio-situational setting. For the details of combining topic information in
dynamic language models see [4,6].
4.3 Results
Table 2 shows the performance of the models on the entire test set in terms of perplexity. The interpolated models perform only slightly better than the smoothed trigram. AL MOSD clearly outperforms the three other models with a perplexity reduction of 11.91%.
In addition to perplexity, we also use word prediction accuracy to measure the performance of the language models. Word prediction has many applications in natural language
processing, such as augmentative and alternative communication, spelling correction,
word and sentence auto completion, etc. Typically word prediction provides one word
or a list of words which fit the context best. This function can be realized by statistical
language models as a side product. Looking at this from the other side, word prediction
accuracy actually provides a measurement of the performance of language models [18].
Adaptive Language Modeling with a Set of Domain Dependent Models
477
Table 2. Comparison of the models in terms of perplexity (ppl)
model
perplexity
smoothed n-gram ST
277
274
topic-based mixture model TM
272
socio-situational setting mixture model SSM
ALMOSD
244
Table 3. Comparison of the models in terms of word prediction accuracy (wpa) per component
of the data set and for the entire test set
comp
a
b
cd
e
f
g
h
i
j
k
l
n
o
overall
ALMOSD
ST
TM
SSM
16.14
14.90
18.10
19.07
13.98
14.93
13.42
15.70
13.88
18.26
12.38
12.61
12.64
16.09
15.35
14.83
17.40
18.02
14.22
14.00
13.41
13.44
12.75
15.54
12.86
12.48
11.85
15.36
15.36
14.83
17.42
18.02
14.25
14.14
13.40
13.48
12.95
15.51
12.86
12.51
11.95
15.38
15.37
14.87
17.41
17.97
14.27
14.23
13.40
13.53
13.17
15.62
12.98
12.61
11.98
15.39
Table 3 compares ALMOSD, the smoothed trigram model ST, the topic dependent
adaptive language models (TM) and the socio-situational setting dependent adaptive
language model (SSM) in terms word prediction accuracy. The models are compared on
the entire test sets as well as per component of the CGN listed in Table 1. It can be seen
that one the entire data set, the ALMOSD model outperforms the three other models.
The model also performs best on most individual components. It does especially well
on component k that contains broadcast news (a word prediction accuracy of 18.26%
vs a prediction accuracy of 15.62% by the socio-situation setting mixture model).
For these results it should be noted that our models were trained on data from the
same set of domains that they were tested on, in case of out-of-domain data, a difference
between our model and mixture based models is to be expected. However, note that
there is no restriction on the set of models that can be included as components. To
handle out-of-domain data, one could include for example a mixture model component.
5 Conclusion
Arguing that domain specific language models perform better than general purpose
models, we propose an adaptive language modeling method called ALMOSD that
478
Y. Shi, P. Wiggers, and C.M. Jonker
combines a set of domain dependent models with a function that selects the most
appropriate domain dependent model for the current situation. In particular, for the task
of word prediction the function selects that component model that fits the word history
better than the other component models. The prediction of the selected model is chosen
as the next word prediction of ALMOSD. At every point in time the model that assigns
the highest probability to the word history is chosen.
Our experiments we show that ALMOSD is able to reduce perplexity by 11.91%
compared to a smoothed n-gram model. It also outperforms the other models tested on
a word prediction task.
The architecture of ALMOSD makes it easy to experiment with other functions to
select the appropriate component model for the current situation. For example, we will
experiment with a limited history horizon, e.g., looking back at most 20 sentences.
Other ideas are to experiment with (combinations of) dynamic classification functions
of domains.
Furthermore, the architecture of ALMOSD also makes it easy to plug in other models,
e.g., richer or models for other specific domains than used in our experiments.
References
1. Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., Khudanpur, S.: Recurrent neural network
based language model. In: INTERSPEECH, pp. 1045–1048 (2010)
2. Foster, P., Skehan, P.: The influence of planning and task type on second language
performance. Studies in Second Language Acquisition 18, 299–323 (1996)
3. Wiggers, P.: Modelling Context in Automatic Speech Recognition. Ph.D. thesis, Delft
University of Technology (2008)
4. Wiggers, P., Rothkrantz, L.: Combining Topic Information and Structure Information in a
Dynamic Language Model. In: Matoušek, V., Mautner, P. (eds.) TSD 2009. LNCS, vol. 5729,
pp. 218–225. Springer, Heidelberg (2009)
5. Iyer, R., Ostendorf, M.: Modeling long distance dependencies in language: Topic mixtures
versus dynamic cache models. IEEE Trans. Speech Audio Process. 7, 236–239 (1999)
6. Iyer, R., Ostendorf, M., Rohlicek, J.R.: Language modeling with sentence-level mixtures.
In: HLT 1994: Proceedings of the Workshop on Human Language Technology, pp. 82–87.
Association for Computational Linguistics, Morristown (1994)
7. Shi, Y., Wiggers, P., Jonker, C.M.: Language modelling with dynamic bayesian networks
using conversation types and part of speech information. In: The 22nd Benelux Conference
on Artificial Intelligence, BNAIC (2010)
8. Shi, Y., Wiggers, P., Jonker, C.M.: Combining Topic Specific Language Models. In:
Habernal, I., Matoušek, V. (eds.) TSD 2011. LNCS, vol. 6836, pp. 99–106. Springer,
Heidelberg (2011)
9. Bellegarda, J.: Statistical language model adaptation: review and perspectives. Speech
Communication 42, 93–108 (2004)
10. Brown, P.F., Pietra, V.J.D., de Souza, P.V., Lai, J.C., Mercer, R.L.: Class-based n-gram
models of natural language. Computational Linguistics 18, 467–479 (1992)
11. Rosenfeld, R.: A maximum entropy approach to adaptive statistical language modelling.
Computer Speech & Language 10, 187–228 (1996)
12. Seymore, K., Rosenfeld, R.: Using story topics for language model adaptation. In: Kokkinakis, G., Fakotakis, N., Dermatas, E. (eds.) EUROSPEECH. ISCA (1997)
Adaptive Language Modeling with a Set of Domain Dependent Models
479
13. Adda, G., Jardino, M., Gauvain, J.L.: Sixth European Conference on Speech Communication
and Technology, Eurospeech 1999, budapest, Hungary, September 5-9. ISCA (1999)
14. Wiggers, P., Rothkrantz, L.J.M.: Topic-based language modeling with dynamic bayesian
networks. In: Proceedings of the Ninth International Conference on Spoken Language
Processing, pp. 1866–1869 (2006)
15. Hermansky, H.: Dealing with Unexpected Words in Automatic Recognition of Speech. In:
Habernal, I., Matoušek, V. (eds.) TSD 2011. LNCS, vol. 6836, pp. 1–15. Springer, Heidelberg
(2011)
16. Hoekstra, H., Moortgat, M., Schuurman, I., van der Wouden, T.: Syntactic annotation for
the spoken dutch corpus project (cgn). Computational Linguistics in the Netherlands 2000,
73–87 (2001)
17. Nelleke, O., Wim, G., Frank Van, E., Louis, B., Jean-pierre, M., Michael, M., Harald, B.:
Experiences from the spoken dutch corpus project. In: Proceedings of the Third International
Conference on Language Resources and Evaluation, pp. 340–347 (2002)
18. van den Bosch, A.: Scalable classification-based word prediction and confusible correction.
Traitement Automatique des Langues 46, 39–63 (2006)