Linked Latent Dirichlet Allocation in Web Spam Filtering
István Bíró
Dávid Siklósi
Jácint Szabó
∗
András A. Benczúr
Data Mining and Web search Research Group, Informatics Laboratory
Computer and Automation Research Institute of the Hungarian Academy of Sciences
{ibiro, sdavid, jacint, benczur}@ilab.sztaki.hu
ABSTRACT
Latent Dirichlet allocation (LDA) (Blei, Ng, Jordan 2003)
is a fully generative statistical language model on the content and topics of a corpus of documents. In this paper
we apply an extension of LDA for web spam classification.
Our linked LDA technique takes also linkage into account:
topics are propagated along links in such a way that the
linked document directly influences the words in the linking document. The inferred LDA model can be applied for
classification as dimensionality reduction similarly to latent
semantic indexing. We test linked LDA on the WEBSPAMUK2007 corpus. By using BayesNet classifier, in terms of
the AUC of classification, we achieve 3% improvement over
plain LDA with BayesNet, and 8% over the public link features with C4.5. The addition of this method to a log-odds
based combination of strong link and content baseline classifiers results in a 3% improvement in AUC. Our method
even slightly improves over the best Web Spam Challenge
2008 result.
Categories and Subject Descriptors
H.3 [Information Systems]: Information Storage and Retrieval; I.2.7 [Computing Methodologies]: Artificial Intelligence—Natural Language Processing
General Terms
text analysis, feature selection, document classification, information retrieval
Keywords
Web content spam, latent Dirichlet allocation
1. INTRODUCTION
∗This work was supported by the EU FP7 project LiWA
– Living Web Archives and by grants OTKA NK 72845,
ASTOR NKFP 2/004/05
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
AIRWeb ’09, April 21, 2009 Madrid, Spain.
Copyright 2009 ACM 978-1-60558-438-6 ...$5.00.
Identifying and preventing spam is cited as one of the top
challenges in web search engines in [16, 22]. As all major
search engines incorporate anchor text and link analysis algorithms into their ranking schemes, web spam appears in
sophisticated forms that manipulate content as well as linkage [14].
In this paper we demonstrate the applicability of topic
based natural language models for Web spam filtering. Several such generative models [8, 17, 4] have been developed
in the field of information retrieval. One of the most successful generative topic models is latent Dirichlet allocation
(LDA) developed by Blei, Ng and Jordan [4], which is a fully
generative graphical model with astonishing performance in
various tasks. Several LDA extensions are known with wide
range of applications in the fields of language processing,
text mining and information retrieval, including categorization, keyword extraction, similarity search and statistical
language modeling.
Recently several models extend LDA to exploit links between web documents or scientific papers [7, 10, 9, 20]. In
these models the term and topic distributions may be modified along the links. All these models have the drawback
that every document is thought of either citing or cited, in
other words, the citation graph is bipartite, and influence
flows only from cited documents to citing ones.
In this paper we apply the recently developed linked
LDA model [3], in which each document can cite to and be
cited by others and thus be influenced and influence other
documents. Linked LDA is very similar to the citation influence model of Dietz, Bickel and Scheffer [9] with the main
difference that in linked LDA the citation graph is not restricted to be bipartite. This fact and its consequences are
the main advantage of linked LDA, namely, that the citation
graph is homogeneous with no need for a citing and a cited
copy of of each document, and finally, that influence may
flow along paths of length more than one, a fact that gives
power to learning over graphs [24, 18]. In addition, linked
LDA gives a flexible model of all possible aspects including
cross-topic relations and link selection. The model may also
distinguish between sites with strong, weak or even no influence from its neighbors. The linked LDA model is described
in full detail in Section 2.1.
We demonstrate the applicability of linked LDA for Web
spam filtering. The inferred topic distributions of documents are used as features. To assess the prediction power
of these features, we test the linked LDA method in combination with the WEBSPAM-UK2007 public features1 and
1
http://www.yr-bcn.es/webspam/datasets/uk2007/features/
ϕ ∼ Dir(β)
SVM over tf.idf. Using a log-odds based random forest to
aggregate these classifiers, the inclusion of linked LDA into
the public and tf.idf features yields an improvement of 3%
in AUC. For a detailed explanation, see Section 3.
β
k
1.1 Related results
Spam hunters use a variety of content based features to
detect web spam [11, 12, 5, 21]; a recent measurement of
their combination appears in [6]. Perhaps the strongest
SVM based content classification is described in [1]. An
efficient method for combining several classifiers is the use
of log-odds averaging [19]. In this paper we apply a modification of this method for combination: a random forest over
the log-odds of the classifiers.
The first probabilistic models that jointly model text and
link as well as the influence of topics along links is PHITS [7]
and the mixed membership model [10]. Later, several similar link based LDA models were introduced, including the
copycat and the citation influence models [9] and the linkPLSA-LDA and pairwise-link-LDA models [20]. These two
results extend LDA over a bipartition of the corpus into citing and cited documents such that influence flows along links
from cited to citing documents. They are shown to outperform earlier methods [9, 20]. While these models generate
topical relation for hyperlinked documents, in a homogeneous corpus one has to duplicate each document and infer
two models for them. This is in contrast to the linked LDA
model that treats citing and cited documents identically.
Our results improve over our related multicorpus LDA
model [2] for Web spam detection. Multicorpus LDA separately builds LDA models for the collection of spam and
normal sites, then take the union of the resulting topic collections and make inference with respect to this aggregated
collection of topics for every unseen document d. The total
probability of spam topics in the topic distribution of d may
serve as a spamicity-measure. In this paper we do not make
experiments on multicorpus LDA, but make comparison to
the measurements presented in [2], see Subsection 3.2.
α
2.1 Linked LDA
The linked LDA model extends LDA to model the effect
of a hyperlink between two documents on the topic and term
distributions. The key idea, summarized as a Bayes net in
Figure 2, is to modify the topic distribution of a position on
the word plate based on a link from the current document on
the document plate. Linked LDA relies on the LDA distributions ϕz and ϑd , but involves an additional distribution χd
z∼ϑ
w ∼ ϕz
m
Figure 1: LDA as a Bayesian network
on the set Sd = {d and its outneighbors} for every document
d, sampled from Dir(γd ), where γd is a positive smoothing
vector on Sd .
As also seen in the Bayes net of Figure 2, in the linked
LDA model the words of the documents are drawn as follows.
For every word position i of document d, we
• draw an influencing document r ∈ Sd from χd ,
• draw a topic z from ϑr (instead of ϑd as in LDA),
• draw a term from ϕz and fill into the position.
ϕ ∼ Dir(β)
β
k
z ∼ ϑs
γ
χ ∼ Dir(γ)
α
ϑ ∼ Dir(α)
2. METHOD
In the classical latent Dirichlet allocation model [4] we
have a vocabulary V consisting of terms, a set T of k topics
and a set D of m documents of arbitrary length. For every
topic z ∈ T a distribution ϕz on V is sampled from Dir(β),
where β ∈ RV+ is a positive smoothing parameter. Similarly,
for every document d a distribution ϑd on T is sampled from
Dir(α), where α ∈ RT+ is a positive smoothing parameter.
The words of the documents are drawn as follows: for every word position of document d a topic z is drawn from
ϑd , and then a term is drawn from ϕz and filled into that
position. The notation is summarized in the widely used
Bayesian network representation of LDA in Figure 1. Inference is mostly done by Gibbs sampling or variational inference.
ϑ ∼ Dir(α)
s∼χ
w ∼ ϕz
m
Figure 2: Linked LDA as a Bayesian network
Note that for sake of a unified treatment, d itself can be
an influencing document of itself. Note that the citation
influence model [9] has a nonuniform solution such that for
every word a Bernoulli trial is used to decide whether the
influencing document is d itself or one of its outneighbors.
For inference a Gibbs sampling procedure can be constructed (described in [3]), along the same lines as for LDA
[15].
3.
EXPERIMENTS
We make experiments on the WEBSPAM-UK2007 corpus2 . In order to define features for hosts, we aggregate
the words appearing in all HTML pages of the host to form
one document per host in a bag of words model. We keep
only alphanumeric characters and the hyphen but remove
all words containing a hyphen not between two alphabetical
2
http://barcelona.research.yahoo.net/webspam/datasets/uk2007/
words. After stemming by TreeTagger3 and removing stop
words by the Onix list4 , the most frequent 100,000 words
form the vocabulary.
Our experiments include the 6000 hosts having a WEBSPAMUK2007 spam or normal label along with an additional 39,000
hosts linked by one of these. We weight directed links between hosts by their multiplicity, and for every site we keep
only at most 10 outlinks with largest weight. After linked
LDA inference is run, we have the ϑ topic distribution vectors as features to the classification.
In our experiments we perform two-class spam classification. We use the linear kernel SVM, C4.5 decision tree and
Bayes net implementations of the machine learning toolkit
Weka [23].
As the simplest baseline we use the public features5 with
C4.5 decision tree and the tf.idf vectors with SVM. For
tf.idf, only terms appearing in at least half of the sites are
kept. Another baseline is formed by the ϑ topic distributions
of the original LDA model as features for classification by
BayesNet. Both the baseline and the linked LDA based classifiers are trained on the WEBSPAM-UK2007 training labels
(3900 sites) and are evaluated on the WEBSPAM-UK2007
test labels (2027 sites) for direct comparability with the Web
Spam Challenge 2008 results.
Our combination of the classifiers is inspired by the logodds averaging by Lynam and Cormack [19]. We first make a
10-fold cross validation on the WEBSPAM-UK2007 training
labels to score every host by every classifier. Then for every
classifier we calculate the log-odds as a feature, which is the
logarithm of the fraction of the number of spams with lower
score over the number of normal sites with higher score.
Finally, we train a random forest over this (quite small)
feature set and give predictions for the WEBSPAM-UK2007
test labels.
3.1 LDA parameters
For LDA inference the following parameter settings are
used. The number of topics is chosen to be k = 30 and k =
90. The Dirichlet parameter vector β is constant 200/|V |,
and α is constant 50/k. For a document d, the smoothing
parameter vector γd is chosen in such a way that
γd (c) ∝ w(d → c) for all c ∈ Sd , c 6= d and
γd (d) ∝ 1 +
X
w(d → c)
c∈Sd ,c6=d
P
such that c∈Sd γd (c) = |d|/p, where |d| is the number of
word positions in d (the document length), w(d → c) denotes
the multiplicity of the d → c link in the corpus, and p is a
normalization parameter. We tried three values p = 1, 4, 10.
We have developed an own C++-code for LDA and linked
LDA, which is publicly available6 . The computations were
run on Linux machines with 50GB RAM and multicore 64bit 3.2 GHz Xeon processors with 2MB cache.
We performed 50 iterations for Gibbs sampling as several
measurements indicate that both the AUC value on classifying over the topic distributions and the likelihood stabilizes
after 50 iterations [13, 3].
3.2
Results
The results of the classification can be seen in Tables 13, the evaluation metric is AUC. For linked LDA only the
parameter choice p = 4, k = 30 was included in the combination, as it gave the best result.
k = 30
k = 90
p=1
0.768
0.764
p=4
0.784
0.777
p = 10
0.783
0.773
Table 1: Classification accuracy measured in AUC
for linked LDA with various parameters, classified
by BayesNet.
features
LDA with BayesNet
tf.idf with SVM
public (link) with C4.5
public (content) with C4.5
AUC
0.766
0.795
0.724
0.782
Table 2: Classification accuracy measured in AUC
for the baseline methods.
features
tf.idf & LDA
tf.idf & linked LDA
public & LDA
public & linked LDA
public & tf.idf
public & tf.idf & LDA
public & tf.idf & linked LDA
public & tf.idf & LDA& linked LDA
AUC
0.827
0.831
0.820
0.829
0.827
0.845
0.854
0.854
Table 3: Classification accuracy measured in AUC
by combining the classifications of Tables 1 and 2
with a log-odds based random forest. For linked
LDA the parameters are chosen to be p = 4, k = 30.
The tables indicate that linked LDA slightly outperforms
LDA by about 3% using BayesNet, showing the predicting
power of the links in the corpus. Most notably, linked LDA
achieved 8% improvement over the public link features with
C4.5, and it is at par with the public content features. The
addition of linked LDA to the log-odds based combination
of the public and tf.idf based classifiers results in a 3% improvement in AUC.
We trained the classifiers on the WEBSPAM-UK2007 training and evaluated them on the WEBSPAM-UK2007 testing
labels as in the Web Spam Challenge 2008 setup. Thus
we can compare these measurements to the Challenge results7 . The present methods improve a lot over the 0.796
AUC achieved by our research group using the multicorpus
LDA model. The winner, Geng et al., managed to have an
AUC of 0.848, and even this value is improved by our public
& tf.idf & linked LDA combination, with AUC 0.854.
3
http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/
4
http://www.lextek.com/manuals/onix/stopwords1.html
5
http://www.yr-bcn.es/webspam/datasets/uk2007/features/ 7 http://webspam.lip6.fr/wiki/6
pmwiki.php?n=Main.PhaseIIIResults
http://www.ilab.sztaki.hu/~ibiro/linkedLDA/
Conclusion and future work
In this paper we applied the newly introduced linked LDA
model [3] to Web spam classification. In our experiments
linked LDA outperformed LDA and other baseline classifications by about 3-8% in AUC. Combining tf.idf, the public
and the linked LDA features with a log-odds based random
forest we achieved an AUC of 0.854, beating the Web Spam
Challenge 2008 winner (0.848). As another experiment we
are currently measuring the quality of the inferred linked
LDA edge weights χ, by using it in a stacked graphical classification procedure, for both the link graph and the cocitation graph.
[13]
[14]
[15]
[16]
4. REFERENCES
[1] J. Abernethy, O. Chapelle, and C. Castillo. WITCH:
A New Approach to Web Spam Detection. In
Proceedings of the 4th International Workshop on
Adversarial Information Retrieval on the Web
(AIRWeb), 2008.
[2] I. Bı́ró, J. Szabó, and A. A. Benczúr. Latent Dirichlet
Allocation in Web Spam Filtering. manuscript, 2008.
[3] I. Bı́ró, J. Szabó, and A. A. Benczúr. Very Large Scale
Link Based Latent Dirichlet Allocation for Web
Document Classification. manuscript,
http://www.ilab.sztaki.hu/~ibiro/linkedLDA/,
2009.
[4] D. Blei, A. Ng, and M. Jordan. Latent Dirichlet
allocation. Journal of Machine Learning Research,
3(5):993–1022, 2003.
[5] A. Bratko, B. Filipič, G. Cormack, T. Lynam, and
B. Zupan. Spam Filtering Using Statistical Data
Compression Models. The Journal of Machine
Learning Research, 7:2673–2698, 2006.
[6] C. Castillo, D. Donato, A. Gionis, V. Murdock, and
F. Silvestri. Know your neighbors: web spam
detection using the web topology. Proceedings of the
30th annual international ACM SIGIR conference on
Research and development in information retrieval,
pages 423–430, 2007.
[7] D. Cohn and T. Hofmann. The Missing Link-A
Probabilistic Model of Document Content and
Hypertext Connectivity. Advances in Neural
Information Processing Systems, pages 430–436, 2001.
[8] S. C. Deerwester, S. T. Dumais, T. K. Landauer,
G. W. Furnas, and R. A. Harshman. Indexing by
latent semantic analysis. Journal of the American
Society of Information Science, 41(6):391–407, 1990.
[9] L. Dietz, S. Bickel, and T. Scheffer. Unsupervised
prediction of citation influences. In Proceedings of the
24th international conference on Machine learning,
pages 233–240. ACM Press New York, NY, USA, 2007.
[10] E. Erosheva, S. Fienberg, and J. Lafferty.
Mixed-membership models of scientific publications,
2004.
[11] D. Fetterly, M. Manasse, and M. Najork. Spam, damn
spam, and statistics – Using statistical analysis to
locate spam web pages. In Proceedings of the 7th
International Workshop on the Web and Databases
(WebDB), pages 1–6, Paris, France, 2004.
[12] D. Fetterly, M. Manasse, and M. Najork. Detecting
phrase-level duplication on the world wide web. In
[17]
[18]
[19]
[20]
[21]
[22]
[23]
[24]
Proceedings of the 28th ACM International Conference
on Research and Development in Information
Retrieval (SIGIR), Salvador, Brazil, 2005.
T. Griffiths and M. Steyvers. Finding scientific topics.
Proceedings of the National Academy of Sciences,
101(suppl 1):5228–5235, 2004.
Z. Gyöngyi and H. Garcia-Molina. Web spam
taxonomy. In Proceedings of the 1st International
Workshop on Adversarial Information Retrieval on the
Web (AIRWeb), Chiba, Japan, 2005.
G. Heinrich. Parameter estimation for text analysis.
Technical report, Technical Report, 2004.
M. R. Henzinger, R. Motwani, and C. Silverstein.
Challenges in web search engines. SIGIR Forum,
36(2):11–22, 2002.
T. Hofmann. Unsupervised Learning by Probabilistic
Latent Semantic Analysis. Machine Learning,
42(1):177–196, 2001.
Z. Kou and W. W. Cohen. Stacked graphical models
for efficient inference in markov random fields. In
SDM 07, 2007.
T. Lynam, G. Cormack, and D. Cheriton. On-line
spam filter fusion. Proc. of the 29th international
ACM SIGIR conference on Research and development
in information retrieval, pages 123–130, 2006.
R. Nallapati, A. Ahmed, E. Xing, and W. Cohen.
Joint Latent Topic Models for Text and Citations. In
Proceedings of the 14th ACM SIGKDD international
conference on Knowledge discovery and data mining.
ACM Press New York, NY, USA, 2008.
A. Ntoulas, M. Najork, M. Manasse, and D. Fetterly.
Detecting spam web pages through content analysis.
In Proceedings of the 15th International World Wide
Web Conference (WWW), pages 83–92, Edinburgh,
Scotland, 2006.
A. Singhal. Challenges in running a commercial search
engine. In IBM Search and Collaboration Seminar
2004. IBM Haifa Labs, 2004.
I. H. Witten and E. Frank. Data Mining: Practical
Machine Learning Tools and Techniques. Morgan
Kaufmann Series in Data Management Systems.
Morgan Kaufmann, second edition, June 2005.
X. Zhu, J. Kandola, Z. Ghahramani, and J. Lafferty.
Nonparametric transforms of graph kernels for
semi-supervised learning. Advances in Neural
Information Processing Systems, 17:1641–1648, 2005.