8037_9789814335454_tp.

indd 1 26/2/15 12:15 pm

 IISc Lecture Notes Series ISSN: 2010-2402

Editor-in-Chief: Gadadhar Misra

Editors: Chandrashekar S Jog
Joy Kuri
K L Sebastian
Diptiman Sen
Sandhya Visweswariah

Published:
Vol. 1: Introduction to Algebraic Geometry and Commutative Algebra
by Dilip P Patil & Uwe Storch

Vol. 2: Schwarz’s Lemma from a Differential Geometric Veiwpoint

by Kang-Tae Kim & Hanjin Lee

Vol. 3: Noise and Vibration Control

by M L Munjal

Vol. 4: Game Theory and Mechanism Design

by Y Narahari

Vol. 5 Introduction to Pattern Recognition and Machine Learning

by M. Narasimha Murty & V. Susheela Devi

Dipa - Introduction to pattern recognition.indd 1 10/4/2015 1:29:09 PM

 World Scientific

8037_9789814335454_tp.indd 2 26/2/15 12:15 pm

 Published by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

Library of Congress Cataloging-in-Publication Data

Murty, M. Narasimha.
Introduction to pattern recognition and machine learning / by M Narasimha Murty &
V Susheela Devi (Indian Institute of Science, India).
pages cm. -- (IISc lecture notes series, 2010–2402 ; vol. 5)
ISBN 978-9814335454
1. Pattern recognition systems. 2. Machine learning. I. Devi, V. Susheela. II. Title.
TK7882.P3M87 2015
006.4--dc23

2014044796

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library.

Copyright © 2015 by World Scientific Publishing Co. Pte. Ltd.

All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,
electronic or mechanical, including photocopying, recording or any information storage and retrieval
system now known or to be invented, without written permission from the publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance
Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy
is not required from the publisher.

In-house Editors: Chandra Nugraha/Dipasri Sardar

Typeset by Stallion Press

Email: enquiries@stallionpress.com

Printed in Singapore

Dipa - Introduction to pattern recognition.indd 3 10/4/2015 1:29:09 PM

 Series Preface
World Scientific Publishing Company - Indian Institute of Science Collaboration

IISc Press and WSPC are co-publishing books authored by world renowned sci-
entists and engineers. This collaboration, started in 2008 during IISc’s centenary
year under a Memorandum of Understanding between IISc and WSPC, has resulted
in the establishment of three Series: IISc Centenary Lectures Series (ICLS), IISc
Research Monographs Series (IRMS), and IISc Lecture Notes Series (ILNS).

This pioneering collaboration will contribute significantly in disseminating current

Indian scientific advancement worldwide.

The “IISc Centenary Lectures Series” will comprise lectures by designated

Centenary Lecturers - eminent teachers and researchers from all over the world.

The “IISc Research Monographs Series” will comprise state-of-the-art mono-

graphs written by experts in specific areas. They will include, but not limited to,
the authors’ own research work.

The “IISc Lecture Notes Series” will consist of books that are reasonably self-
contained and can be used either as textbooks or for self-study at the postgraduate
level in science and engineering. The books will be based on material that has been
class-tested for most part.

Editorial Board for the IISc Lecture Notes Series (ILNS):

Gadadhar Misra, Editor-in-Chief (gm@math.iisc.ernet.in)

Chandrashekar S Jog (jogc@mecheng.iisc.ernet.in)

Joy Kuri (kuri@cedt.iisc.ernet.in)
K L Sebastian (kls@ipc.iisc.ernet.in)
Diptiman Sen (diptiman@cts.iisc.ernet.in)
Sandhya Visweswariah (sandhya@mrdg.iisc.ernet.in)

Dipa - Introduction to pattern recognition.indd 2 10/4/2015 1:29:09 PM

May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page vii

Table of Contents

About the Authors xiii

Preface xv

1. Introduction 1
1. Classifiers: An Introduction . . . . . . . . . . . . . . 5
2. An Introduction to Clustering . . . . . . . . . . . . . 14
3. Machine Learning . . . . . . . . . . . . . . . . . . . 25

2. Types of Data 37
1. Features and Patterns . . . . . . . . . . . . . . . . . 37
2. Domain of a Variable . . . . . . . . . . . . . . . . . 39
3. Types of Features . . . . . . . . . . . . . . . . . . . 41
3.1. Nominal data . . . . . . . . . . . . . . . . . . 41
3.2. Ordinal data . . . . . . . . . . . . . . . . . . . 45
3.3. Interval-valued variables . . . . . . . . . . . . 48
3.4. Ratio variables . . . . . . . . . . . . . . . . . . 49
3.5. Spatio-temporal data . . . . . . . . . . . . . . 49
4. Proximity measures . . . . . . . . . . . . . . . . . . 50
4.1. Fractional norms . . . . . . . . . . . . . . . . 56
4.2. Are metrics essential? . . . . . . . . . . . . . . 57
4.3. Similarity between vectors . . . . . . . . . . . 59
4.4. Proximity between spatial patterns . . . . . . 61
4.5. Proximity between temporal patterns . . . . . 62

vii
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page viii

viii Table of Contents

4.6. Mean dissimilarity . . . . . . . . . . . . . . . . 63

4.7. Peak dissimilarity . . . . . . . . . . . . . . . . 63
4.8. Correlation coefficient . . . . . . . . . . . . . . 64
4.9. Dynamic Time Warping (DTW) distance . . . 64

3. Feature Extraction and Feature Selection 75

1. Types of Feature Selection . . . . . . . . . . . . . . . 76
2. Mutual Information (MI) for Feature Selection . . . 78
3. Chi-square Statistic . . . . . . . . . . . . . . . . . . 79
4. Goodman–Kruskal Measure . . . . . . . . . . . . . . 81
5. Laplacian Score . . . . . . . . . . . . . . . . . . . . . 81
6. Singular Value Decomposition (SVD) . . . . . . . . 83
7. Non-negative Matrix Factorization (NMF) . . . . . . 84
8. Random Projections (RPs) for Feature
Extraction . . . . . . . . . . . . . . . . . . . . . . . 86
8.1. Advantages of random projections . . . . . . . 88
9. Locality Sensitive Hashing (LSH) . . . . . . . . . . . 88
10. Class Separability . . . . . . . . . . . . . . . . . . . 90
11. Genetic and Evolutionary Algorithms . . . . . . . . 91
11.1. Hybrid GA for feature selection . . . . . . . . 92
12. Ranking for Feature Selection . . . . . . . . . . . . . 96
12.1. Feature selection based on an optimization
formulation . . . . . . . . . . . . . . . . . . . . 97
12.2. Feature ranking using F-score . . . . . . . . . 99
12.3. Feature ranking using linear support vector
machine (SVM) weight vector . . . . . . . . . 100
12.4. Ensemble feature ranking . . . . . . . . . . . . 101
12.5. Feature ranking using number
of label changes . . . . . . . . . . . . . . . . . 103
13. Feature Selection for Time Series Data . . . . . . . . 103
13.1. Piecewise aggregate approximation . . . . . . 103
13.2. Spectral decomposition . . . . . . . . . . . . . 104
13.3. Wavelet decomposition . . . . . . . . . . . . . 104
13.4. Singular Value Decomposition (SVD) . . . . . 104
13.5. Common principal component loading based
variable subset selection (CLeVer) . . . . . . . 104
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page ix

Table of Contents ix

4. Bayesian Learning 111

1. Document Classification . . . . . . . . . . . . . . . . 111
2. Naive Bayes Classifier . . . . . . . . . . . . . . . . . 113
3. Frequency-Based Estimation of Probabilities . . . . 115
4. Posterior Probability . . . . . . . . . . . . . . . . . . 117
5. Density Estimation . . . . . . . . . . . . . . . . . . . 119
6. Conjugate Priors . . . . . . . . . . . . . . . . . . . . 126

5. Classification 135
1. Classification Without Learning . . . . . . . . . . . 135
2. Classification in High-Dimensional Spaces . . . . . . 139
2.1. Fractional distance metrics . . . . . . . . . . . 141
2.2. Shrinkage–divergence proximity (SDP) . . . . 143
3. Random Forests . . . . . . . . . . . . . . . . . . . . 144
3.1. Fuzzy random forests . . . . . . . . . . . . . . 148
4. Linear Support Vector Machine (SVM) . . . . . . . 150
4.1. SVM–kNN . . . . . . . . . . . . . . . . . . . . 153
4.2. Adaptation of cutting plane algorithm . . . . 154
4.3. Nystrom approximated SVM . . . . . . . . . . 155
5. Logistic Regression . . . . . . . . . . . . . . . . . . . 156
6. Semi-supervised Classification . . . . . . . . . . . . . 159
6.1. Using clustering algorithms . . . . . . . . . . . 160
6.2. Using generative models . . . . . . . . . . . . 160
6.3. Using low density separation . . . . . . . . . . 161
6.4. Using graph-based methods . . . . . . . . . . 162
6.5. Using co-training methods . . . . . . . . . . . 164
6.6. Using self-training methods . . . . . . . . . . . 165
6.7. SVM for semi-supervised classification . . . . 166
6.8. Random forests for semi-supervised
classification . . . . . . . . . . . . . . . . . . . 166
7. Classification of Time-Series Data . . . . . . . . . . 167
7.1. Distance-based classification . . . . . . . . . . 168
7.2. Feature-based classification . . . . . . . . . . . 169
7.3. Model-based classification . . . . . . . . . . . 170
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page x

x Table of Contents

6. Classification using Soft Computing Techniques 177

1. Introduction . . . . . . . . . . . . . . . . . . . . . . 177
2. Fuzzy Classification . . . . . . . . . . . . . . . . . . 178
2.1. Fuzzy k-nearest neighbor algorithm . . . . . . 179
3. Rough Classification . . . . . . . . . . . . . . . . . . 179
3.1. Rough set attribute reduction . . . . . . . . . 180
3.2. Generating decision rules . . . . . . . . . . . . 181
4. GAs . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.1. Weighting of attributes using GA . . . . . . . 182
4.2. Binary pattern classification using GA . . . . 184
4.3. Rule-based classification using GAs . . . . . . 185
4.4. Time series classification . . . . . . . . . . . . 187
4.5. Using generalized Choquet integral with
signed fuzzy measure for classification
using GAs . . . . . . . . . . . . . . . . . . . . 187
4.6. Decision tree induction using
Evolutionary algorithms . . . . . . . . . . . . 191
5. Neural Networks for Classification . . . . . . . . . . 195
5.1. Multi-layer feed forward network
with backpropagation . . . . . . . . . . . . . . 197
5.2. Training a feedforward neural network
using GAs . . . . . . . . . . . . . . . . . . . . 199
6. Multi-label Classification . . . . . . . . . . . . . . . 202
6.1. Multi-label kNN (mL-kNN) . . . . . . . . . . 203
6.2. Probabilistic classifier chains (PCC) . . . . . . 204
6.3. Binary relevance (BR) . . . . . . . . . . . . . 205
6.4. Using label powersets (LP) . . . . . . . . . . . 205
6.5. Neural networks for Multi-label
classification . . . . . . . . . . . . . . . . . . . 206
6.6. Evaluation of multi-label classification . . . . 209

7. Data Clustering 215

1. Number of Partitions . . . . . . . . . . . . . . . . . 215
2. Clustering Algorithms . . . . . . . . . . . . . . . . . 218
2.1. K-means algorithm . . . . . . . . . . . . . . . 219
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xi

Table of Contents xi

2.2. Leader algorithm . . . . . . . . . . . . . . . . 223

2.3. BIRCH: Balanced Iterative Reducing
and Clustering using Hierarchies . . . . . . . . 225
2.4. Clustering based on graphs . . . . . . . . . . . 230
3. Why Clustering? . . . . . . . . . . . . . . . . . . . . 241
3.1. Data compression . . . . . . . . . . . . . . . . 241
3.2. Outlier detection . . . . . . . . . . . . . . . . 242
3.3. Pattern synthesis . . . . . . . . . . . . . . . . 243
4. Clustering Labeled Data . . . . . . . . . . . . . . . . 246
4.1. Clustering for classification . . . . . . . . . . . 246
4.2. Knowledge-based clustering . . . . . . . . . . 250
5. Combination of Clusterings . . . . . . . . . . . . . . 255

8. Soft Clustering 263

1. Soft Clustering Paradigms . . . . . . . . . . . . . . . 264
2. Fuzzy Clustering . . . . . . . . . . . . . . . . . . . . 266
2.1. Fuzzy K-means algorithm . . . . . . . . . . . 267
3. Rough Clustering . . . . . . . . . . . . . . . . . . . . 269
3.1. Rough K-means algorithm . . . . . . . . . . . 271
4. Clustering Based on Evolutionary Algorithms . . . . 272
5. Clustering Based on Neural Networks . . . . . . . . 281
6. Statistical Clustering . . . . . . . . . . . . . . . . . . 282
6.1. OKM algorithm . . . . . . . . . . . . . . . . . 283
6.2. EM-based clustering . . . . . . . . . . . . . . . 285
7. Topic Models . . . . . . . . . . . . . . . . . . . . . . 293
7.1. Matrix factorization-based methods . . . . . . 295
7.2. Divide-and-conquer approach . . . . . . . . . 296
7.3. Latent Semantic Analysis (LSA) . . . . . . . . 299
7.4. SVD and PCA . . . . . . . . . . . . . . . . . . 302
7.5. Probabilistic Latent Semantic Analysis
(PLSA) . . . . . . . . . . . . . . . . . . . . . . 307
7.6. Non-negative Matrix Factorization
(NMF) . . . . . . . . . . . . . . . . . . . . . . 310
7.7. LDA . . . . . . . . . . . . . . . . . . . . . . . 311
7.8. Concept and topic . . . . . . . . . . . . . . . . 316
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xii

xii Table of Contents

9. Application — Social and Information Networks 321

1. Introduction . . . . . . . . . . . . . . . . . . . . . . 321
2. Patterns in Graphs . . . . . . . . . . . . . . . . . . . 322
3. Identification of Communities in Networks . . . . . . 326
3.1. Graph partitioning . . . . . . . . . . . . . . . 328
3.2. Spectral clustering . . . . . . . . . . . . . . . . 329
3.3. Linkage-based clustering . . . . . . . . . . . . 331
3.4. Hierarchical clustering . . . . . . . . . . . . . 331
3.5. Modularity optimization for partitioning
graphs . . . . . . . . . . . . . . . . . . . . . . 333
4. Link Prediction . . . . . . . . . . . . . . . . . . . . . 340
4.1. Proximity functions . . . . . . . . . . . . . . . 341
5. Information Diffusion . . . . . . . . . . . . . . . . . 347
5.1. Graph-based approaches . . . . . . . . . . . . 348
5.2. Non-graph approaches . . . . . . . . . . . . . 349
6. Identifying Specific Nodes in a Social Network . . . 353
7. Topic Models . . . . . . . . . . . . . . . . . . . . . . 355
7.1. Probabilistic latent semantic analysis
(pLSA) . . . . . . . . . . . . . . . . . . . . . . 355
7.2. Latent dirichlet allocation (LDA) . . . . . . . 357
7.3. Author–topic model . . . . . . . . . . . . . . . 359

Index 365
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xiii

About the Authors

Professor M. Narasimha Murty completed his B.E., M.E., and

Ph.D. at the Indian Institute of Science (IISc), Bangalore. He joined
IISc as an Assistant Professor in 1984. He became a professor in 1996
and currently he is the Dean, Engineering Faculty at IISc. He has
guided more than 20 doctoral students and several masters students
over the past 30 years at IISc; most of these students have worked in
the areas of Pattern Recognition, Machine Learning, and Data Min-
ing. A paper co-authored by him on Pattern Clustering has around
9600 citations as reported by Google scholar. A team led by him
had won the KDD Cup on the citation prediction task organized by
the Cornell University in 2003. He is elected as a fellow of both the
Indian National Academy of Engineering and the National Academy
of Sciences.

Dr. V. Susheela Devi completed her PhD at the Indian Institute

of Science in 2000. Since then she has worked as a faculty in the
Department of Computer Science and Automation at the Indian
Institute of Science. She works in the areas of Pattern Recogni-
tion, Data Mining, Machine Learning, and Soft Computing. She has
taught the courses Data Mining, Pattern Recognition, Data Struc-
tures and Algorithms, Computational Methods of Optimization and
Artificial Intelligence. She has a number of papers in international
conferences and journals.

xiii
May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xv

Preface

Pattern recognition (PR) is a classical area and some of the important

topics covered in the books on PR include representation of patterns,
classification, and clustering. There are different paradigms for pat-
tern recognition including the statistical and structural paradigms.
The structural or linguistic paradigm has been studied in the early
days using formal language tools. Logic and automata have been
used in this context. In linguistic PR, patterns could be represented
as sentences in a logic; here, each pattern is represented using a set
of primitives or sub-patterns and a set of operators. Further, a class
of patterns is viewed as being generated using a grammar; in other
words, a grammar is used to generate a collection of sentences or
strings where each string corresponds to a pattern. So, the classifi-
cation model is learnt using some grammatical inference procedure;
the collection of sentences corresponding to the patterns in the class
are used to learn the grammar. A major problem with the linguistic
approach is that it is suited to dealing with structured patterns and
the models learnt cannot tolerate noise.
On the contrary the statistical paradigm has gained a lot of
momentum in the past three to four decades. Here, patterns are
viewed as vectors in a multi-dimensional space and some of the
optimal classifiers are based on Bayes rule. Vectors corresponding
to patterns in a class are viewed as being generated by the underly-
ing probability density function; Bayes rule helps in converting the
prior probabilities of the classes into posterior probabilities using the

xv
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xvi

xvi Preface

likelihood values corresponding to the patterns given in each class.

So, estimation schemes are used to obtain the probability density
function of a class using the vectors corresponding to patterns in the
class. There are several other classifiers that work with vector repre-
sentation of patterns. We deal with statistical pattern recognition in
this book.
Some of the simplest classification and clustering algorithms are
based on matching or similarity between vectors. Typically, two pat-
terns are similar if the distance between the corresponding vectors is
lesser; Euclidean distance is popularly used. Well-known algorithms
including the nearest neighbor classifier (NNC), K-nearest neighbor
classifier (KNNC), and the K-Means Clustering algorithm are based
on such distance computations. However, it is well understood in the
literature that distance between two vectors may not be meaning-
ful if the vectors are in large-dimensional spaces which is the case in
several state-of-the-art application areas; this is because the distance
between a vector and its nearest neighbor can tend to the distance
between the pattern and its farthest neighbor as the dimensionality
increases. This prompts the need to reduce the dimensionality of the
vectors. We deal with the representation of patterns, different types
of components of vectors and the associated similarity measures in
Chapters 2 and 3.
Machine learning (ML) also has been around for a while;
early efforts have concentrated on logic or formal language-based
approaches. Bayesian methods have gained prominence in ML in
the recent decade; they have been applied in both classification and
clustering. Some of the simple and effective classification schemes
are based on simplification of the Bayes classifier using some accept-
able assumptions. Bayes classifier and its simplified version called
the Naive Bayes classifier are discussed in Chapter 4. Tradition-
ally there has been a contest between the frequentist approaches
like the Maximum-likelihood approach and the Bayesian approach.
In maximum-likelihood approaches the underlying density is esti-
mated based on the assumption that the unknown parameters are
deterministic; on the other hand the Bayesian schemes assume that
the parameters characterizing the density are unknown random vari-
ables. In order to make the estimation schemes simpler, the notion
April 8, 2015 13:2 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-fm page xvii

Preface xvii

of conjugate pair is exploited in the Bayesian methods. If for a given

prior density, the density of a class of patterns is such that, the pos-
terior has the same density function as the prior, then the prior and
the class density form a conjugate prior. One of the most exploited in
the context of clustering are the Dirichlet prior and the Multinomial
class density which form a conjugate pair. For a variety of such con-
jugate pairs it is possible to show that when the datasets are large
in size, there is no difference between the maximum-likelihood and
the Bayesian estimates. So, it is important to examine the role of
Bayesian methods in Big Data applications.
Some of the most popular classifiers are based on support vector
machines (SVMs), boosting, and Random Forest. These are discussed
in Chapter 5 which deals with classification. In large-scale applica-
tions like text classification where the dimensionality is large, linear
SVMs and Random Forest-based classifiers are popularly used. These
classifiers are well understood in terms of their theoretical properties.
There are several applications where each pattern belongs to more
than one class; soft classification schemes are required to deal with
such applications. We discuss soft classification schemes in Chapter 6.
Chapter 7 deals with several classical clustering algorithms including
the K-Means algorithm and Spectral clustering. The so-called topic
models have become popular in the context of soft clustering. We
deal with them in Chapter 8.
Social Networks is an important application area related to PR
and ML. Most of the earlier work has dealt with the structural
aspects of the social networks which is based on their link structure.
Currently there is interest in using the text associated with the nodes
in the social networks also along with the link information. We deal
with this application in Chapter 9.
This book deals with the material at an early graduate level.
Beginners are encouraged to read our introductory book Pattern
recognition: An Algorithmic Approach published by Springer in 2011
before reading this book.

M. Narasimha Murty
V. Susheela Devi
Bangalore, India
May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 1

Chapter 1

Introduction

This book deals with machine learning (ML) and pattern recognition
(PR). Even though humans can deal with both physical objects and
abstract notions in day-to-day activities while making decisions in
various situations, it is not possible for the computer to handle them
directly. For example, in order to discriminate between a chair and
a pen, using a machine, we cannot directly deal with the physical
objects; we abstract these objects and store the corresponding rep-
resentations on the machine. For example, we may represent these
objects using features like height, weight, cost, and color. We will
not be able to reproduce the physical objects from the respective
representations. So, we deal with the representations of the patterns,
not the patterns themselves. It is not uncommon to call both the
patterns and their representations as patterns in the literature.
So, the input to the machine learning or pattern recognition sys-
tem is abstractions of the input patterns/data. The output of the
system is also one or more abstractions. We explain this process
using the tasks of pattern recognition and machine learning. In pat-
tern recognition there are two primary tasks:
1. Classification: This problem may be defined as follows:
• There are C classes; these are Class1 , Class2, . . . , ClassC .
• Given a set Di of patterns from Classi for i = 1, 2, . . . , C.
D = D1 ∪ D2 . . . ∪ DC . D is called the training set and mem-
bers of D are called labeled patterns because each pattern
has a class label associated with it. If each pattern Xj ∈ D is

1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 2

2 Introduction to Pattern Recognition and Machine Learning

d-dimensional, then we say that the patterns are d-dimensional

or the set D is d-dimensional or equivalently the patterns lie
in a d-dimensional space.
• A classification model Mc is learnt using the training patterns
in D.
• Given an unlabeled pattern X, assign an appropriate class label
to X with the help of Mc .
It may be viewed as assigning a class label to an unlabeled pattern.
For example, if there is a set of documents, Dp , from politics class
and another set of documents, Ds , from sports, then classification
involves assigning an unlabeled document d a label; equivalently
assign d to one of two classes, politics or sports, using a classifier
learnt from Dp ∪ Ds .
There could be some more details associated with the definition
given above. They are

• A pattern Xj may belong to one or more classes. For example, a

document could be dealing with both sports and politics. In such
a case we have multiple labels associated with each pattern. In the
rest of the book we assume that a pattern has only one class label
associated.
• It is possible to view the training data as a matrix D of size n × d
where the number of training patterns is n and each pattern is
d-dimensional. This view permits us to treat D both as a set and
as a pattern matrix. In addition to d features used to represent
each pattern, we have the class label for each pattern which could
be viewed as the (d + 1)th feature. So, a labeled set of n pat-
terns could be viewed as {(X1 , C 1 ), (X2 , C 2 ), . . . , (Xn , C n )} where
C i ∈ {Class1 , Class2 , . . . , ClassC } for i = 1, 2, . . . , n. Also, the
data matrix could be viewed as an n × (d + 1) matrix with the
(d + 1)th column having the class labels.
• We evaluate the classifier learnt using a separate set of patterns,
called test set. Each of the m test patterns comes with a class
label called the target label and is labeled using the classifier learnt
and this label assigned is the obtained label. A test pattern is
correctly classified if the obtained label matches with the target
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 3

Introduction 3

label and is misclassified if they mismatch. If out of m patterns,

mc are correctly classified then the % accuracy of the classifier is
100 × mc
m .
• In order to build the classifier we use a subset of the training
set, called the validation set which is kept aside. The classifica-
tion model is learnt using the training set and the validation set
is used as test set to tune the model or obtain the parameters
associated with the model. Even though there are a variety of
schemes for validation, K-fold cross-validation is popularly used.
Here, the training set is divided into K equal parts and one of them
is used as the validation set and the remaining K −1 parts form the
training set. We repeat this process K times considering a different
part as validation set each time and compute the accuracy on the
validation data. So, we get K accuracies; typically we present the
sample mean of these K accuracies as the overall accuracy and also
show the sample standard deviation along with the mean accuracy.
An extreme case of validation is to consider n-fold cross-validation
where the model is built using n−1 patterns and is validated using
the remaining pattern.

2. Clustering: Clustering is viewed as grouping a collection of

patterns. Formally we may define the problem as follows:
• There is a set, D, of n patterns in a d-dimensional space.
A generally projected view is that these patterns are unlabeled.
• Partition the set D into K blocks C1 , C2 , . . . , CK ; Ci is called
the ith cluster. This means Ci ∩ Cj = φ and Ci = φ for i = j
and i, j ∈ {1, 2, . . . , K}.
• In classification an unlabeled pattern X is assigned to one of
C classes and in clustering a pattern X is assigned to one of
K clusters. A major difference is that classes have semantic
class labels associated with them and clusters have syntactic
labels. For example, politics and sports are semantic labels;
we cannot arbitrarily relabel them. However, in the case of
clustering we can change the labels arbitrarily, but consistently.
For example, if D is partitioned into two clusters C1 and C2 ;
so the clustering of D is πD = {C1 , C2 }. So, we can relabel C1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 4

4 Introduction to Pattern Recognition and Machine Learning

as C2 and C2 as C1 consistently and have the same clustering

(set {C1 , C2 }) because elements in a set are not ordered.

Some of the possible variations are as follows:

• In a partition a pattern can belong to only one cluster. However,
in soft clustering a pattern may belong to more than one cluster.
There are applications that require soft clustering.
• Even though clustering is viewed conventionally as partitioning a
set of unlabeled patterns, there are several applications where clus-
tering of labeled patterns is useful. One application is in efficient
classification.
We illustrate the pattern recognition tasks using the two-dimensional
dataset shown in Figure 1.1. There are nine points from class
X labeled X1, X2, . . . , X9 and 10 points from class O labeled
O1, O2, . . . , O10. It is possible to cluster patterns in each class sep-
arately. One such grouping is shown in Figure 1.1. The Xs are clus-
tered into two groups and the Os are also clustered into two groups;
there is no requirement that there be equal number of clusters in
each class in general. Also we can deal with more than two classes.
Different algorithms might generate different clusterings of each class.
Here, we are using the class labels to cluster the patterns as we are
clustering patterns in each class separately. Further we can represent

F2 X8 X9
X6 O9
X7 t2 O10
b O8
X5 O7
X4 X3 t1 O6O5
X1 X2
O1 O3O4
O2

a F1

Figure 1.1. Classification and clustering.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 5

Introduction 5

each cluster by its centroid, medoid, or median which helps in data

compression; it is sometimes adequate to use the cluster representa-
tives as training data so as to reduce the training effort in terms of
both space and time. We discuss a variety of algorithms for clustering
data in later chapters.

1. Classifiers: An Introduction
In order to get a feel for classification we use the same data points
shown in Figure 1.1. We also considered two test points labeled t1
and t2 . We briefly illustrate some of the prominent classifiers.

• Nearest Neighbor Classifier (NNC): We take the nearest

neighbor of the test pattern and assign the label of the neighbor
to the test pattern. For the test pattern t1 , the nearest neighbor
is X3; so, t1 is classified as a member of X. Similarly, the nearest
neighbor of t2 is O9 and so t2 is assigned to class O.
• K-Nearest Neighbor Classifier (KNNC): We consider K-
nearest neighbors of the test pattern and assign it to the class
based on majority voting; if the number of neighbors from class X
is more than that of class O, then we assign the test pattern to
class X; otherwise to class O. Note that NNC is a special case of
KNNC where K = 1.
In the example, if we consider three nearest neighbors of t1
then they are: X3 , X2 , and O1 . So majority are from class X and
so t1 is assigned to class X. In the case of t2 the three nearest
neighbors are: O9 , X9 , and X8 . Majority are from class X; so, t2
is assigned to class X. Note that t2 was assigned to class O based
on NNC and to class X based on KNNC . In general different
classifiers might assign the same test pattern to different classes.
• Decision Tree Classifier (DTC): A DTC considers each feature
in turn and identifies the best feature along with the value at which
it splits the data into two (or more) parts which are as pure as
possible. By purity here we mean as many patterns in the part are
from the same class as possible. This process gets repeated level
by level till some termination condition is satisfied; termination is
affected based on whether the obtained parts at a level are totally
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 6

6 Introduction to Pattern Recognition and Machine Learning

pure or nearly pure. Each of these splits is an axis-parallel split

where the partitioning is done based on values of the patterns on
the selected feature.
In the example shown in Figure 1.1, between features F1 and
F2 dividing on F1 based on value a gives two pure parts; here
all the patterns having F1 value below a and above a are put
into two parts, the left and the right. This division is depicted in
Figure 1.1. All the patterns in the left part are from class X and
all the patterns in the right part are from class O. In this example
both the parts are pure. Using this split it is easy to observe that
both the test patterns t1 and t2 are assigned to class X.
• Support Vector Machine (SVM): In a SVM, we obtain either
a linear or a non-linear decision boundary between the patterns
belonging to both the classes; even the nonlinear decision boundary
may be viewed as a linear boundary in a high-dimensional space.
The boundary is positioned such that it lies in the middle of the
margin between the two classes; the SVM is learnt based on the
maximization of the margin. Learning involves finding a weight
vector W and a threshold b using the training patterns. Once we
have them, then given a test pattern X, we assign X to the positive
class if W t X + b > 0 else to the negative class.
It is possible to show that W is orthogonal to the decision
boundary; so, in a sense W fixes the orientation of the decision
boundary. The value of b fixes the location of the decision bound-
ary; b = 0 means the decision boundary passes through the origin.
In the example the decision boundary is the vertical line passing
through a as shown in Figure 1.1. All the patterns labeled X may
be viewed as negative class patterns and patterns labeled O are
positive patterns. So, W t X + b < 0 for all X and W t O + b > 0
for all O. Note that both t1 and t2 are classified as negative class
patterns.

We have briefly explained some of the popular classifiers. We can

further categorize them as follows:

• Linear and Non-linear Classifiers: Both NNC and KNNC

are non-linear classifiers as the decision boundaries are non-linear.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 7

Introduction 7

Similarly both SVM and DTC are linear in the example as the
decision boundaries are linear. In general, NNC and KNNC are
nonlinear. Even though Kernel SVM can be nonlinear, it may be
viewed as a linear classifier in a high-dimensional space and DTC
may be viewed as a piecewise linear classifier. There are other lin-
ear classifiers like the Naive Bayes Classifier (NBC ) and Logistic
Regression-based classifiers which are discussed in the later chap-
ters.
• Classification in High-dimensional Spaces: Most of the
current applications require classifiers that can deal with high-
dimensional data; these applications include text classification,
genetic sequence analysis, and multimedia data processing. It is
difficult to get discriminative information using conventional dis-
tance based classifiers; this is because the nearest neighbor and
farthest neighbors of a pattern will have the same distance values
from any point in a high-dimensional space. So, NNC and KNNC
are not typically used in high-dimensional spaces. Similarly, it is
difficult to build a decision tree when there are a large number of
features; this is because starting from the root node of a possibly
tall tree we have to select a feature and its value for the best split
out of a large collection of features at every internal node of the
decision tree. Similarly, it becomes difficult to train a kernel SVM
in a high-dimensional space.
Some of the popular classifiers in high-dimensional spaces are
linear SVM , NBC , and logistic regression-based classifier. Classi-
fier based on random forest seems to be another useful classifier
in high-dimensional spaces; random forest works well because each
tree in the forest is built based on a low-dimensional subspace.
• Numerical and Categorical Features: In several practical
applications we have data characterized by both numerical and cat-
egorical features. SVM s can handle only numerical data because
they employ dot product computations. Similarly, NNC and
KNNC work with numerical data where it is easy to compute
neighbors based on distances. These classifiers require conversion
of categorical features into numerical features appropriately before
using them.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 8

8 Introduction to Pattern Recognition and Machine Learning

Classifiers like DTC and NBC are ideally suited to handle

data described by both numerical and categorical features. In the
case of DTC purity measures employed require only number of
patterns from each class corresponding to the left and right parts
of a split and both kinds of features can be split. In the case of
NBC it is required to compute the frequency of patterns from a
class corresponding to a feature value in the case of categorical
features and likelihood value of the numerical features.
• Class Imbalance: In some of the classification problems one
encounters class imbalance. This happens because some classes
are not sufficiently represented in the training data. Consider, for
example, classification of people into normal and abnormal classes
based on their health status. Typically, the number of abnormal
people could be much smaller than the number of normal peo-
ple in a collection. In such a case, we have class imbalance. Most
of the classifiers may fail to do well on such data. In the case
of the abnormal (minority) class, frequency estimates go wrong
because of small sample size. Also it may not be meaningful to
locate the SVM decision boundary symmetrically between the two
support planes; intuitively it is good to locate the decision bound-
ary such that more patterns are accommodated in the normal
(majority) class.
A preprocessing step may be carried out to balance the data
that is not currently balanced. One way is to reduce the number
of patterns in the majority class (undersampling). This is typically
done by clustering patterns in the majority class and representing
clusters by their prototypes; this step reduces a large collection of
patterns in the majority class to a small number of cluster repre-
sentatives. Similarly, pattern synthesis can be used to increase the
number of patterns in the minority class (oversampling). A simple
technique to achieve it is based on bootstrapping; here, we consider
a pattern and obtain the centroid of its K nearest neighbors from
the same class. This centroid forms an additional pattern; this
process is repeated for all the patterns in the training dataset. So,
if there are n training patterns to start with we will be able to gen-
erate additional n synthetic patterns which means bootstrapping
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 9

Introduction 9

over the entire training dataset will double the number of training
patterns.
Bootstrapping may be explained using the data in Figure 1.1.
Let us consider three nearest neighbors for each pattern. Let us
consider X1; its 3 neighbors from the same class are X2, X4, and
X3. Let X1
be the centroid of these three points. In a similar
manner we can compute bootstrapped patterns X2
, X3
, . . . , X9

corresponding to X2, X3, . . . , X9 respectively. In a similar manner
bootstrapped patterns corresponding to Os also can be computed.
For example, O2, O3, O6 are the three neighbors of O1 and their
centroid will give the bootstrapped pattern O1
. In a general set-
ting we may have to obtain bootstrap patterns corresponding to
both the classes; however to deal with the class imbalance problem,
we need to bootstrap only the minority class patterns. There are
several other ways to synthesize patterns in the minority class.
Preprocessing may be carried out either by decreasing the size
of the training data of the majority class or by increasing the size
of training data of the minority class or both.
• Training and Classification Time: Most of the classifiers
involve a training phase; they learn a model and use it for classi-
fication. So, computation time is required to learn the model and
for classification of the test patterns; these are called training time
and classification/test time respectively. We give the details below:

− Training: It is done only once using the training data. So, for
real time classification applications classification time is more
important than the training time.
∗ NNC : There is no training done; in this sense it is the
simplest model. However, in order to simplify testing/
classification a data structure is built to store the training
data in a compact/compressed form.
∗ KNNC : Here also there is no training time. However, using a
part of the training data and the remaining part for valida-
tion, we need to fix a suitable value for K. Basically KNNC
is more robust to noise compared to NNC as it considers
more neighbors. So, smaller values of K make it noise-prone
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 10

10 Introduction to Pattern Recognition and Machine Learning

whereas larger values of K, specifically K = n makes it decide

based on the prior probabilities of the classes or equivalently
based on the frequency of patterns from each of the classes.
There are variants of KNNC that take into account the dis-
tance between a pattern and each of the K neighbors; contri-
bution of neighbors too far away from the pattern is ignored.
∗ DTC : The simple version of decision tree is built based
on axis-parallel decisions. If there are n training patterns,
each represented by a d-dimensional vector, then the effort
involved in decision making at each node is of O(d n log n);
this is because on each feature value we have to sort the n
pattern values using O(n log n) time and there are d features.
It gets larger as the value of d increases; further, the tree is
built in a greedy manner as a feature selected leads to a
split and it influences the later splits. A split made earlier
cannot be redone. There are other possible ways of splitting
a node; one possibility is to use an oblique split which could
be considered as a split based on a linear combination of val-
ues of some selected features. However, oblique split based
decision trees are not popular because they require time that
is exponential in n.
∗ SVM : Training an SVM requires O(n3 ) time.

− Testing: Several researchers examine the testing/classification

time more closely compared to the training time as training
is performed only once whereas testing is carried out multiple
times. The testing times for various classifiers are:

∗ NNC : It is linear in the number of patterns as for each test

pattern we have to compute n distances, one for each training
pattern.
∗ KNNC : It requires O(nK) time for testing as it has to update
the list of K neighbors.
∗ DTC : It requires O(log n) effort to classify a test pattern
because it has to traverse over a path of the decision tree
having at most n leaf nodes.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 11

Introduction 11

∗ SVM : Linear SVM takes O(d) time to classify a test pattern

based on primarily a dot product computation.
• Discriminative and Generative Models: The classification
model learnt is either probabilistic or deterministic. Typically
deterministic models are called discriminative and the probabilis-
tic models are called generative. Example classifiers are:
− Generative Models: The Bayesian and Naive Bayes models are
popular generative models. Here, we need to estimate the pro-
bability structure using the training data and use these models
in classification. Because we estimate the underlying probabi-
lity densities it is easy to generate patterns from the obtained
probabilistic structures. Further, when using these models for
classification, one can calculate the posterior probability asso-
ciated with each of the classes given the test pattern. There are
other generative models like the Hidden Markov Models and
Gaussian mixture models which are used in classification.
− Discriminative Models: Deterministic models including DTC ,
and SVM are examples of discriminative models. They typically
can be used to classify a test pattern; they cannot reveal the
associated probability as they are deterministic models.
• Binary versus Multi-Class Classification: Some of the classi-
fiers are inherently suited to deal with two-class problems whereas
the others can handle multi-class problems. For example, SVM
and AdaBoost are ideally suited for two-class problems. Classi-
fiers including NNC , KNNC , and DTC are generic enough to deal
with multi-class problems. It is possible to combine binary classifier
results to achieve multi-class classification. Two popular schemes
for doing this are:
1. One versus the Rest: If there are C classes then we build a
binary classifier for each class as follows:
− Class1 versus the rest Class2 ∪ Class3 · · · ∪ ClassC
− Class2 versus the rest Class1 ∪ Class3 · · · ∪ ClassC
..
.
− ClassC versus the rest Class1 ∪ Class2 · · · ∪ ClassC−1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 12

12 Introduction to Pattern Recognition and Machine Learning

There are a total of C binary classifiers and the test pattern is

assigned a class label based on the output of these classifiers.
Ideally the test pattern will belong to one class Classi ; so the
corresponding binary classifier will assign it to Classi and the
remaining C − 1 classifiers assign it to the rest. One problem
with this approach is that each of the binary classifiers has
to deal with class imbalance; this is because in each binary
classification problem we have patterns of one of the classes
labeled positive and the patterns of the remaining C − 1 classes
are labeled negative. So, there could be class imbalance with
the positive class being the minority class and the negative class
being the majority class.
2. One versus One: Here out of the C classes two classes are con-
sidered at a time to form a binary classifier. There are a total
of C(C−1)
2 choices and as many binary classifiers to be built.
These are:
− Class1 versus Class2 , Class1 versus Class3, . . . , Class1
versus ClassC
− Class2 versus Class3 , Class2 versus Class4, . . . , Class2
versus ClassC
..
.
− ClassC−1 versus ClassC
A pattern is assigned to class Ci based on a majority voting.
• Number of Classes: Most of the classifiers work well when the
number of classes is small. Specific possibilities are:
− Number of Classes C is Small: Classifiers that work well are
∗ SVM : It is inherently a binary classifier; so, it is ideally suited
for dealing with a small number of classes.
∗ NBC : It can estimate the associated probabilities accurately
when the data is dense and it is more likely when the number
of classes is small.
∗ DTC : It works well when the number of features is small. In
such a case we cannot have a large number of classes because
if there are C leaf nodes in a binary tree then the number
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 13

Introduction 13

of internal nodes (decision nodes) will be C − 1; each such

node corresponds to a split on a feature. If each feature is
used for splitting at m nodes on an average and there are
l features then l × m ≥ C − 1; So, C ≤ l × m + 1. As a
consequence Random Forest also is more suited when the
number of classes is small.
∗ Bayes Classifier: For a pattern if the posterior probabilities
are equal for all the classes then the probability of error is
1 − C1 = C−1
C and if there is one class with posterior proba-
bility 1 and the remaining C − 1 classes having a zero pos-
terior probability then the probability of error is zero. So,
the probability of error is bounded by 0 ≤ P (error) ≤ C−1C .
So, if C = 2, then the upper bound is 12 . If C → ∞ then
P (error) is upper bounded by 1. So, as C changes the bound
gets effected.
− Number of Classes C is Large: Some of the classifiers that
are relevant are
∗ KNNC : It can work well when the number of neighbors con-
sidered is large and the number of training patterns n is large.
Theoretically it is possible to show that it can be optimal as
K and n tend to ∞, K slower than n. So, it can deal with
a large number of classes provided each class has a sufficient
number of training patterns. However, the classification time
could be large.
∗ DTC : It is inherently a multi-class classifier like the KNNC .
So, it can work well when n > l × m + 1 > C.
• Classification of Multi-label Data: It is possible that each
pattern can have more than one class label associated with it.
For example, in a collection of documents to be classified into
either sports or politics, it is possible that one or more documents
have both the labels associated; in such a case we have multi-
label classification problem which is different from the multi-class
classification discussed earlier. In a multi-class case, the number of
classes is more than two but each pattern has only one class label
associated with it.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 14

14 Introduction to Pattern Recognition and Machine Learning

One solution to the multi-label problem is to consider each

subset of the set of the C classes as a label; in such a case we have
again a multi-class classification problem. However, the number of
possible class labels is exponential in the number of classes. For
example, in the case of two class set {sports, politics}, the possible
labels correspond to the subsets {sports}, {politics}, and {sports,
politics}. Even though we have two classes, we can have three class
labels here. In general for a C class problem, the number of class
labels obtained this way is 2C − 1. A major problem with this
process is that we need to look for a classifier that can deal with
a large number of class labels.
Another solution to the multi-label problem is based on using
a soft computing tool for classification; in such a case we may
have the same pattern belonging to different classes with different
membership values, based on using fuzzy sets.

2. An Introduction to Clustering
In clustering we group a collection, D, of patterns into some K clus-
ters; patterns in each cluster are similar to each other. There are a
variety of clustering algorithms. Broadly they may be characterized
in the following ways:

• Partitional versus Hierarchical Clustering: In partitional

clustering a partition of the dataset is obtained. In the hierar-
chical clustering a hierarchy of partitions is generated. Some of
the specific properties of these kinds of algorithms are:
− Partitional Clustering: Here, the dataset D is divided into
K clusters. It is achieved such that some criterion function is
optimized. If we consider the set {X1 , X2 , X3 } then the possi-
bilities for two clusters are:
1. C1 = {X1 , X2 }; C2 = {X3 }.
2. C1 = {X1 , X3 }; C2 = {X2 }.
3. C1 = {X2 , X3 }; C2 = {X1 }.
So, the number of partitions of a dataset of three patterns is 3.
This number grows very fast as the set size and number of
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 15

Introduction 15

clusters increase. For example, to cluster a small dataset of

19 patterns into 4 clusters, the number of possible partitions
is approximately 11,259,666,000. So, exhaustive enumeration
of all possible partitions to find out the best partition is not
realistic. So, each clustering algorithm is designed to ensure that
only an appropriate subset of the set of all possible partitions
is explored by the algorithm.
For example, one of the most popular partitional clustering
algorithms is the K-means algorithm. It partitions the given
dataset into K clusters or equivalently it obtains a K-partition
of the dataset. It starts with an arbitrary initial K-partition
and keeps on refining the partition iteratively till a convergence
condition is satisfied. The K-means algorithm minimizes the
squared-error criterion; it generates K spherical clusters which
are characterized by some kind of tightness. Specifically it aims
to minimize the sum over all the clusters the sum of squared
distances of points in each cluster from its centroid; here each
cluster is characterized and represented by its centroid. So, by
its nature the algorithm is inherently restricted to generate
spherical clusters. However, based on the type of the distance
function used, it is possible to generate different cluster shapes.
For example, it can generate the clusters depicted in Figure 1.1.
Another kind of partitional algorithm uses a threshold on
the distance between a pattern and a cluster representative to
see whether the pattern can be assigned to the cluster or not. If
the distance is below the threshold then the pattern is assigned
to the cluster; otherwise a new cluster is initiated with the pat-
tern as its representative. The first cluster is represented by the
first pattern in the collection. Here, threshold plays an impor-
tant role; if it is too small then there will be a larger number
of clusters and if it is large then there will be a smaller number
of clusters. A simple algorithm that employs threshold as spec-
ified above is the leader algorithm; BIRCH is another popular
clustering algorithm that employs a threshold for clustering.
− Hierarchical Clustering: In hierarchical clustering we generate
partitions of size 1 (one cluster) to partitions of n clusters while
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 16

16 Introduction to Pattern Recognition and Machine Learning

clustering a collection of n patterns. They are either agglomera-

tive or divisive. In the case of agglomerative algorithms we start
with each pattern in a cluster and keep merging most similar
pair of clusters from n − 1 clusters; this process of merging the
most similar pair of clusters is repeated to get n−2, n−3, . . ., 2,
and 1 clusters. In the case of divisive algorithms we start with
one cluster having all the patterns and divide it into two clusters
based on some notion of separation between the resulting pair
of clusters; the cluster with maximum size out of these clusters
is split into two clusters to realize three clusters. this splitting
process goes on as we get 4, 5, . . . , n − 1, n clusters.
A difficulty with these hierarchical algorithms is that they
need to compute and store a proximity matrix of size O(n2 ).
So, they may not be suited to deal with large-scale datasets.
• Computational Requirements: Computational requirements
of the clustering algorithms include time and space requirements.
− Computation Time: The conventional hierarchical algorithms
require O(n2 ) to compute distances between pairs of points. It is
possible to show that the K-means clustering algorithm requires
O(nKlm) where K is the number of clusters, l is the number of
features, and m is the number of iterations of the algorithm. The
Leader algorithm is the simplest computationally; it requires
one scan of the dataset.
− Storage Space: Hierarchical algorithms require O(n2 ) space to
store the proximity matrix which is used in clustering. The
K-means algorithm requires O(Kl) to store the K cluster cen-
ters each in l-dimensional space; in addition we need to store
the dataset which requires O(nl) space. The leader algorithm
has space requirements similar to the K-means algorithm.
• Local Optimum: Several partitional clustering algorithms
including the K-means algorithm can lead to a local minimum of
the associated criterion function. For example, the K-means algo-
rithm may reach the local minimum value of the squared error cri-
terion function if the initial partition is not properly chosen. Even
though it is possible to show equivalence between the K-means
type of algorithms and threshold based clustering algorithms, there
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 17

Introduction 17

may not be an explicit criterion function that is optimized by the

leader like algorithms.
It is possible to show the equivalence between some of the hier-
archical algorithms with their graph-theoretic counterparts. For
example, the agglomerative algorithm can merge two clusters in
different ways:
1. Single-Link Algorithm (SLA): Here two clusters Cp and Cq are
merged if the distance between a pair of points Xi ∈ Cp and
Xj ∈ Cq is the smallest among all possible pairs of clusters. It
can group points into clusters when two or more clusters have
the same mean but different covariance; such clusters are called
concentric clusters. It is more versatile than the K-means algo-
rithm. It corresponds to the construction of the Minimal Span-
ning Tree of the data where an edge weight is based on the
distance between the points representing the end vertices and
clusters are realized by ignoring the link with the maximum
weight. Here, a minimum spanning tree of the data is obtained.
However, the algorithm is bound to generate a minimal span-
ning tree where the minimal spanning tree is a spanning tree
with the sum of the edge weights being a minimum; a spanning
tree is a tree that connects all the nodes.
2. Complete-Link Algorithm (CLA): Here, two clusters Cp and Cq
are merged if the distance between them is minimum; the dis-
tance between the two clusters is defined by the maximum of
the distance between points Xi ∈ Cp and Xj ∈ Cq for all Xi and
Xj . This algorithm corresponds to the generation of completely
connected components.
3. Average-Link Algorithm (ALA): Here, two clusters Cp and Cq
are merged based on average distance between pairs of points
where one is from Cp and the other is from Cq .
• Representing Clusters: The most popularly used cluster repre-
sentative is the centroid or the sample mean of the points in the
cluster. It may be defined as
|C|
1

µC = Xi .
|C|
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 18

18 Introduction to Pattern Recognition and Machine Learning

For example, in Figure 1.1, for the points X6, X7, X8, X9 in one of
the clusters, the centroid is located inside the circle having these
four patterns. The advantage of representing a cluster using its
centroid is that it may be centrally located and it is the point
from which the sum of the squared distances to all the points in the
cluster is minimum. However, it is not helpful in achieving robust
clustering; this is because if there is an outlier in the dataset then
the centroid may be shifted away from a majority of the points in
the cluster. The centroid may shift further as the outlier becomes
more and more prominent. So, centroid is not a good representative
in the presence of outliers. Another representative that could be
used is the medoid of the cluster; medoid is the most centrally
located point that belongs to the cluster. So, medoid cannot be
significantly affected by a small number of points in the cluster
whether they are outliers or not.
Another issue that emerges in this context is to decide whether
each cluster has a single representative or multiple representatives.
• Dynamic Clustering: Here, we obtain a partition of the data
using a set, Dn , of patterns. Let the partition be πn where n is
the number of patterns in D. Now we would like to add or delete
a pattern from D. So, the possibilities are:
− Addition of a Pattern: Now the question is whether we can
reflect the addition of a pattern to Dn in the resulting cluster-
ing by updating the partition πn to πn+1 without re-clustering
the already clustered data. In other words, we would like to use
the n + 1th pattern and πn to get πn+1 ; this means we generate
πn+1 without reexamining the patterns in Dn . Such a clustering
paradigm may be called incremental clustering. This paradigm
is useful in stream data mining. One problem with incremen-
tal clustering is order dependence; for different orderings of the
input patterns in D, we obtain different partitions.
− Deletion of a Pattern: Even though incremental clustering
where additional patterns can be used to update the current
partition without re-clustering the earlier seen data is popular,
deletion of patterns from the current set and its impact on the
partition is not examined in a detailed manner in the literature.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 19

Introduction 19

By dynamic clustering, we mean updating the partition after

either addition or deletion of patterns without re-clustering.
• Detection of Outliers: An outlier is a pattern that differs from
the rest of the patterns significantly. It can be either an out of range
or within range pattern. Outliers are typically seen as abnormal
patterns which differ from the rest. Typically, they are detected
based on either looking for singleton clusters or by using some
density based approach. Outliers are patterns that lie in sparse
regions.
In a simplistic scenario clustering could be used to detect out-
liers because outliers are elements of small size clusters. Also there
are density-based clustering algorithms that categorize each pat-
tern as a core pattern or a boundary pattern and keep merging
the patterns to form clusters till some boundary patterns are left
out as noise or outliers. So, clustering has been a popularly used
tool in outlier detection.
• Missing Values: It is possible that some feature values in a subset
of patterns are missing. For example, in a power system it may
not be possible to get current and voltage values at every node;
sometimes it may not be possible to have access to a few nodes.
Similarly while building a recommender system we may not have
access to the reviews of each of the individuals on a subset of
products being considered for possible recommendation. Also in
evolving social networks we may have links between only a subset
of the nodes.
In conventional pattern recognition, missing values are esti-
mated using a variety of schemes. Some of them are:
− Cluster the patterns using the available feature values. If the
pattern Xi has its jth feature value xij missing then
|C|


xij = xpj ,
p=1

where C is the cluster to which Xi belongs.

− Find the nearest neighbor of Xi from the given dataset using
the available feature values; let the nearest neighbor be Xq .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 20

20 Introduction to Pattern Recognition and Machine Learning

Then

xij = xqj + δ,

where δ is a small quantity used to perturb the value of xqj to

obtain the missing value.

In social networks we have missing links which are predicted using

some link prediction algorithm. If Xi and Xj are two nodes in the
network represented as a graph, then similarity between Xi and
Xj is computed. Based on the similarity value, whether a link is
possible or not is decided. A simple local similarity measure may
be explained as follows:

− Let N N Set(Xi ) = Set of nodes adjacent to Xi ,

− Let N N Set(Xj ) = Set of nodes adjacent to Xj ,
− Similarity-Score (Xi , Xj ) = |N N Set(Xi ) ∩ N N Set(Xj )|.

Here, the similarity between two nodes is defined as the number of

neighbors common to Xi and Xj . Based on the similarity values
we can rank the missing links in decreasing order of the similarity;
we consider a subset of the missing links in the rank order to link
the related nodes.
• Clustering Labeled Data: Conventionally clustering is associ-
ated with grouping of unlabeled patterns. But clustering may be
viewed as data compression; so, we can group labeled patterns
and represent clusters by their representatives. Such a provision
helps us in realizing efficient classifiers as explained earlier using
the data in Figure 1.1.
Clustering has been effectively used in combination with a vari-
ety of classifiers. Most popularly clustering has been used along
with NNC and KNNC to reduce the classification time. It has
been used to improve the speed of training SVM to be used in
classification; clustering is used in training both linear SVM and
nonlinear SVM . The most popular classifier that exploits cluster-
ing is the Hidden Markov Model (HMM).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 21

Introduction 21

• Clustering Large Datasets: Large datasets are typically

encountered in several machine learning applications including bio-
informatics, software engineering, text classification, video analy-
tics, health, education and agriculture. So, the role of clustering to
compress data in these applications is very natural. In data mining,
one generates abstractions from data and clustering is an ideal tool
for obtaining a variety of such abstractions; in fact clustering has
gained prominence after the emergence of data mining.
Some of the prominent directions for clustering large data-
sets are:

− Incremental Clustering: Algorithms like Leader clustering and

BIRCH are incremental algorithms for clustering. They require
to scan the dataset only once. Sometimes additional processing
is done to avoid order dependence.
− Divide-and-conquer Clustering: Here, we divide the dataset of
n patterns into p blocks so that each block has approximately
n
p patterns. It is possible to cluster patterns in each block sepa-
rately and represent them by a small number of patterns. These
clusters are merged by clustering their representatives using
another clustering step and affecting the resulting cluster labels
on all the patterns. It is important to observe that Map-Reduce
is a divide-and-conquer approach that could be used to solve a
variety of problems including clustering.
− Compress and Cluster : It is possible to represent the data using
a variety of abstraction generation schemes and then cluster the
data. Some possibilities are:
∗ Hybrid clustering: Here, using an inexpensive clustering algo-
rithm we compress the data and cluster the representatives
using an expensive algorithm. Such an approach is called
hybrid clustering.
∗ Sampling: The dataset size is reduced by selecting a sample
of the large dataset and then cluster the sample but not the
original dataset.
∗ Lossy and non-lossy compression: In several applications it is
adequate to deal with the compressed data. Compression may
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 22

22 Introduction to Pattern Recognition and Machine Learning

be achieved by using a variety of lossy and non-lossy compres-

sion schemes; then clustering is performed on the compressed
data. Some of the non-lossy compression schemes include run-
length coding, and Huffman coding. Similarly, lossy compres-
sion schemes include representing the data using frequent
itemsets, using some dimensionality reduction techniques.
∗ Use a compact data structure: Some of the data structures
that are used in clustering are Cluster Feature Tree (CF-
Tree), Frequent Pattern Tree (FP-Tree), and Inverted Files.
BIRCH constructs CF -tree using a single dataset scan and
uses it to reflect the clustering. F P -tree is constructed using
two dataset scans and implicitly stores all the frequent item-
sets. The inverted file structure is very popular in information
retrieval.
• Soft Clustering: Most of the current day applications require
soft clustering where a pattern belongs to more than one cluster.
There are a variety of soft clustering approaches; these include:

1. Fuzzy clustering: Here, a pattern is assigned to different clusters

with different membership values. It is originally designed to
deal with linguistic uncertainty. If the sum of the membership
values of a pattern to different clusters adds up to 1, then it
is fuzzy-set theoretic clustering; otherwise we have possibilis-
tic clustering. Fuzzy K-means is one of the most popular soft
clustering algorithms.
2. Rough clustering: Here, patterns are divided into equivalence
classes based on the notion of discernibility; if two or more pat-
terns are not discernible then they belong to the same equiva-
lence class. Each cluster is described using a lower and an upper
approximation. All the equivalence classes that form subsets of
the cluster form the lower approximation and all the equivalence
classes that overlap with the cluster form the upper approxima-
tion. A pattern definitely belongs to a cluster if the pattern is in
the lower approximation of the cluster and a pattern may belong
to two or more clusters if it is in the upper approximation of a
cluster.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 23

Introduction 23

3. Neural networks: In a neural network used for clustering each

output node indicates a cluster to which a pattern belongs.
Typically, we employ a soft output function in the interval [0, 1];
so, each output node might indicate some degree to which the
input pattern might belong to the corresponding cluster. Using
such an output structure it is possible to realize some kind of
soft clustering.
4. Evolutionary algorithms: In clustering using the evolutionary
algorithms, each element of the population corresponds to a
clustering. If each string/element of the population is inter-
preted as corresponding to K cluster centroids/representatives,
then each pattern may belong to more than one cluster
based on its closeness (similarity) to the centroids of various
clusters.
• Topic Models: Topic is defined as a probability distribution over
words/terms in a collection. For example, in a small collection
of short documents let the terms and corresponding probability of
occurrence be as shown in Table 1.1. This may be viewed as a topic
cricket where the terms shown in the table have occurred. Note
that if we use it in an unsupervised setting, then we have to provide
the topic its name manually. For example, we cannot automatically
assign the label cricket to the probability mass function shown in
Table 1.1. Also, topic as we understand it intuitively corresponds to
some semantic label. So, it may be good to use the name probability
distribution instead of topic.

Table 1.1. A probabil-

ity distribution.

Term Probability

Cricket 0.4
Ball 0.2
Bat 0.15
Umpire 0.12
Wicket 0.08
Run 0.05
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 24

24 Introduction to Pattern Recognition and Machine Learning

Another notion that could be used to describe a cluster is

concept. Typically, concept is a logical description;
 a popular  usage
1 3 ∧ f
is conjunction
 of disjunctions.
  For example, f 1 = v 1 ∨ v 1 2 =
2 4 5 ··· ∧ f 1 may be viewed as a concept; here f
v2 ∨ v2 ∨ v2 l = vl i
is the ith feature and the value vij corresponds to jth element of
the domain of feature fi . We have assumed that the domain has a
finite set of values for the sake of simplicity.
There are topic models that are popularly used in document
clustering and Latent Dirichlet Allocation (LDA) is one of the
recent models.
− LDA: It is a model where each document is made up of at most
K clusters/topics. Each topic is an assignment of probabilities
to words in the collection. Given the document collection or
the words in the collection, the problem is to obtain the latent
clusters/topics and use them to describe the documents. Each
topic is viewed as a multinomial distribution and by using the
conjugate prior in the form of a Dirichlet, we get the posterior
also to be Dirichlet. It is possible to view this process as fac-
torizing the document matrix. There are some simpler models
like Probabilistic Latent Semantic Indexing (PLSI) that can be
used as modeling the collection of documents. It is possible
to show equivalence between P LSI and Non-negative matrix
factorization (N M F ).
− NMF: Here, the problem is to factorize the document-term
matrix A into 2 factors B(>0) and C(>0) such that

A − BC2F is minimized.

It is possible to view these models as performing soft clustering;

this is because a document may belong to more than one topic.
• Knowledge-based clustering: Even though clustering is viewed
as an unsupervised learning paradigm, there is a well-known the-
orem called the theorem of the ugly duckling which shows that
we need to use extra-logical information to discriminate a pair
of patterns from another pair based on the similarity between
the pairs. In short, it is not possible to have unsupervised
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 25

Introduction 25

classification; it is essential to use knowledge in the process of

clustering.
There are knowledge-based clustering schemes that are based
on the notion of concept; specifically they are called conceptual
clustering methods. Also Wikipedia knowledge is used in clustering
documents semantically to get more meaningful clusters. In the
conceptual clustering framework we get both the clusters and their
descriptions using concepts. Note that clustering based on topic
models also generates cluster descriptions in the form of topics.
• Cluster Validity: One of the evasive features of clustering is
the notion of cluster validity. If we get a clustering of the given
dataset then we would like to know whether the resultant clus-
tering and the number of clusters is acceptable. A host of cluster
validity indices are proposed and used to measure the quality of
clustering. However, there is a strong opinion that these measures
are monotonic. That means as the number of clusters is increased
the validity measure will monotonically increase or decrease. For
example, if we consider the squared error criterion then when the
number of clusters is 1 the value of the squared error is maximum
and when the number of clusters is n then the squared error is
minimum (zero). Further, an index suitable for one type of clusters
may not be good to deal with other type of clusters. For example,
the squared error is good for evaluating spherical clusters.

3. Machine Learning
We have seen the two important pattern recognition tasks which are
classification and clustering. Another task is dimensionality reduc-
tion which is required in both classification and clustering. Some of
the other tasks in machine learning are regression or curve fitting,
ranking, and summarization. We deal with them in this section.

• Dimensionality reduction

Feature extraction/selection is almost an integral part of both classi-

fication and clustering. In classification we need to reduce the dimen-
sionality to ensure that the classifiers work well and in clustering also
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 26

26 Introduction to Pattern Recognition and Machine Learning

we need to reduce the dimensionality to reduce the clustering time

and space requirements of the algorithms. So, typically a prepro-
cessing step is carried out to reduce the dimensionality. Some of the
associated issues are:

• Distance between patterns: In both classification and clustering we

need to compute distance between pairs of patterns. For example,
in NNC the test pattern is classified based on the class label of
its nearest neighbor which is obtained based on some type of dis-
tance. Similarly, clustering using the K-means algorithm requires
distance between a pattern and each of the K centroids. These dis-
tances may not be meaningful if the patterns are represented in a
high-dimensional space. So, we need to reduce the dimensionality.
• Feature selection: Typically, a data matrix of size n × l is used to
represent n patterns in a l-dimensional space or using l features.
One way of reducing the dimensionality is by selecting a subset of
the given set of features or select some d out of l features. This
selection is done in different ways.
− Filter methods: In these methods features are evaluated based
on some criterion function that is independent of the classifier.
One popular function that is effectively used in text analysis is
the mutual information. It measures some kind of information
that is mutual between a class and a feature. Based on the
mutual information the features are ranked and some d top
ranked features out of the given l features are used to learn the
classifier. So, we may use the class labels but not the classifier
in selecting features here.
− Wrapper methods: Here, features are selected based on the clas-
sifier used. For each subset of features one can build the classifier
and it is used to evaluate the quality of the selected feature set.
An example is the genetic algorithm-based feature selection;
each string in a population corresponds to a subset of the fea-
tures. Specifically, each string is of l bits corresponding to l
given features. A 1 in the ith position corresponds to selecting
the ith feature and a 0 means the ith feature is not selected. So,
each string corresponds to a subset. The fitness of the string is
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 27

Introduction 27

obtained by using the classifier built using the selected features;

so, if the selected subset leads to a higher classification accuracy
then the fitness of the corresponding string is high.
− Embedded methods: Here, the subset of features is obtained as a
consequence of building the classifier. For example, in building
a DTC, the decision tree built will have features at different
internal or decision nodes of the tree. Features used at the root
and other top nodes of the decision tree are relatively more
important than those at the other nodes. So, selection of fea-
tures can be done using the decision tree; top level features
are selected. Similarly, in a linear SVM the learning involves
obtaining a weight vector W and a threshold b. It is possi-
ble to use the entries of W to rank features; if the value is
positive and large, then the presence of the feature will make
the pattern move towards the positive class and a large nega-
tive value will mean that the pattern having the feature will
more likely be a negative class pattern. So, one can retain
features that have large corresponding components in W and
ignore the remaining features. So, both DTC and SVM may
be viewed as providing features in an embedded manner for
selection.

• Feature extraction: Here, we extract new features that are either

linear or nonlinear combinations of the given features and the
extracted features are used in Pattern recognition. Some of the
extraction schemes are:

− Principal components: Principal components are vectors in the

directions of maximum variance. They are computed by using
the eigenvectors of the data covariance matrix. If there are l
features then the covariance matrix is of size l × l and is a
symmetric matrix. So, the eigenvalues are real and eigenvectors
are orthogonal to each other. The first principal component is
the eigenvector corresponding to the maximum eigenvalue. The
next principal component is the eigenvector of the covariance
matrix corresponding to the next largest eigenvalue and so on.
This is an unsupervised learning scheme and so can be used
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 28

28 Introduction to Pattern Recognition and Machine Learning

both in classification and clustering. In text mining it is popu-

larly used under the name latent semantic analysis.
− Random projections: Here, the rows of n×l data matrix are pro-
jected to a d-dimensional space by multiplying the data matrix
with a matrix of size l × d where d << l. The entries of the
l × d matrix are randomly selected based on some distribu-
tional assumptions. It is possible to show that the projection to
the lower dimensional space will protect the distance and dot
product between a pair of patterns.
− Clustering and feature extraction: Because we know that the
row-rank of a matrix is equal to its column-rank, there is an
implicit relation between clustering and feature extraction. If
we consider the soft clustering schemes based on matrix factor-
ization including latent semantic indexing (LSI), probabilistic
latent semantic indexing (PLSI), LDA and N M F , then there
is linear feature extraction scheme corresponding to each of
them. For example, consider the factorization of matrix A of
size n × l into matrices B of size n × d and C of size d × l; the
B matrix describes d topics/clusters. Equivalently we can view
them as d features and the B matrix gives the description of
the n patterns in the d or extracted feature space. So, implicitly
we have feature extraction in these cases; each factorization is
based on a different optimization problem and so can give rise
to extracting different types of features.
• Function learning: In classification, we assign a pattern to one
of a finite number of classes. However, in function learning, we may
need to assign real values to patterns. For example, the value of a
share of a company could be given as a positive real number; the
predicted temperature on a day in the summer at a geographical
location or the predicted intensity of earthquake are all positive
real numbers. In such cases we learn functions using the training
data and use the function to predict values in the future. Learning
classification function or a classifier may be viewed as a special
case of function learning. So, there are a host of techniques that
are used both in classification and general function learning. For
example, decision trees, SVM s, and AdaBoost-based schemes have
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 29

Introduction 29

been successfully used in both classification and regression (general

function learning).
• Ranking: Here, instead of classifying we would like to get some
kind of probability with which we can assign a pattern to a class.
For example, a search engine provides as a output ranked set of
snippets corresponding to a query entered by the user. Here, the
relevant documents based on some similarity measure are shown
in a ranked fashion; top ones are more relevant to the query.
• Summarization: When the data is large we would like to have
a summary or abstraction of the data instead of all the details.
Different abstractions possible are: cluster representatives, decision
tree, set of rules, and a concise abstract of one or more documents.

In this book we deal with the following topics:

1. Types of Data: We consider both the categorical and numer-

ical features and their characterization along with some of the
associated proximity measures.
2. Feature Extraction and Feature Selection: Dimensionality
reduction is an important and integral part of pattern recogni-
tion and machine learning tasks. Here, we discuss some of the
important schemes for feature selection and extraction.
3. Bayesian Learning: Bayesian approaches were popular in pat-
tern recognition. In machine learning, Bayesian approaches are
gaining prominence in both classification and clustering. We dis-
cuss Naive Bayes classifier and the notion of conjugate prior and
its role in estimation of densities.
4. Classification: We consider some of the important classifiers
including KNNC , SVM , logistic regression, random forests, and
classification in high-dimensional spaces. We also discuss some of
the modifications to these classifiers.
5. Soft Classification: Soft classification techniques such as
fuzzy classifiers, rough classifiers, classification using evolutionary
strategies and neural networks are discussed.
6. Data Clustering: Clustering is a compression tool; it is used as
a preprocessor to handle efficient classification. We consider algo-
rithms for both partitional and hierarchical clustering. We also
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 30

30 Introduction to Pattern Recognition and Machine Learning

discuss incremental algorithms like Leader and BIRCH. We

consider clustering algorithms that are used on graph datasets.
We deal with the role of frequent itemsets in clustering and com-
binations of outputs generated by clustering algorithms. The role
of knowledge-based clustering, labeled clustering, and divide-and-
conquer clustering are discussed.
7. Soft Clustering: Soft clustering has become an important area
of research in machine learning. In addition to the conventional
paradigms, the role of topic models in realizing soft clusters and
their descriptions is examined.
8. Social and Information Networks: We consider the applica-
tion of various techniques discussed in social networks. Specifi-
cally, we consider some of the important concepts in analyzing
social networks including community detection, modularity, link
prediction, and topic models.

Research Ideas
1. What are the primary differences between pattern recognition, machine learn-
ing, and data mining. Which tasks are important in each of these areas?

Relevant References
(a) J. Han, M. Kamber and J. Pei, Data Mining: Concepts and Techniques,
Third Edition. New York: Morgan Kauffmann, 2011.
(b) K. Murphy, Machine Learning: A Probabilistic Perspective. Cambridge,
MA: MIT Press, 2012.
(c) M. N. Murty and V. Susheela Devi, Pattern Recognition: An Algorithmic
Approach. London: Springer, 2011.
2. How do we evaluate classifiers performance? What is the best validation
scheme? How to deal with class imbalance?

Relevant References
(a) V. Lopez, A. Fernandez and F. Herrera, On the importance of the vali-
dation technique for classification with imbalanced datasets: Addressing
covariate shift when data is skewed. Information Sciences, 257:1–13, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 31

Introduction 31

(b) Q.-Y. Yin, J.-S. Zhang, C.-X. Zhang and N.-N. Ji, A novel selective ensem-
ble algorithm for imbalanced data classification based on exploratory
under-sampling. Mathematical Problems in Engineering, 2014.
(c) Y. Sun, A. K. C. Wong and M. S. Kamel, Classification of imbalanced
data: A review. International Journal of Pattern Recognition and Artificial
Intelligence, 23:687–719, 2009.
3. How do we represent clusters? Is it essential that there should be one repre-
sentative per cluster? Is it possible to solve it using an optimization scheme?

Relevant References
(a) D. Bhattacharya, S. Seth and T.-H. Kim, Social network analysis to detect
inherent communities based on constraints. Applied Mathematics and
Information Sciences, 8:1–12, 2014.
(b) W. Hamalainen, V. Kumpulainen and M. Mozgovoy, Evaluation of clus-
tering methods for adaptive learning systems. In Artificial Intelligence in
Distance Education, U. Kose and D. Koc (eds.). Hershey, PA: IGI Global,
2014, pp. 237–260.
(c) P. Franti, M. Rezaei and Q. Zhao, Centroid index: Cluster level similarity
measure. Pattern Recognition, 47:3034–3045, 2014.
4. How do we compute similarity between a pair of patterns that employ both
numerical and categorical features? Can distance/dot product based methods
work well with such patterns?

Relevant References
(a) Y.-M. Cheung and H. Jia, Categorical-and-numerical-attribute data clus-
tering based on a unified similarity metric without knowing cluster number.
Pattern Recognition, 46:2228–2238, 2013.
(b) I. W. Tsang, J. T. Kwok and P.-M. Cheung, Core vector machines: Fast
SVM training on very large data sets. JMLR, 6:363–392, 2005.
(c) A. Ahmad and G. Brown, Random projection random discretization
ensembles — ensembles of linear multivariate decision trees. IEEE
Transactions on Knowledge Data and Engineering, 26:1225–1239,
2014.
5. In the context of so-called generative models, why should one synthesize
patterns? Which classifiers exploit the synthetic patterns better?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 32

32 Introduction to Pattern Recognition and Machine Learning

Relevant References
(a) L. Plonsky, J. Egbert and G. T. Laflair, Bootstrapping in applied linguistics:
Assessing its potential using shared data. Applied Linguistics, 2014.
(b) P. S. Gromski, Y. Xu, E. Correa, D. I. Ellis, M. L. Turner and R. Goodacre,
A comparative investigation of modern feature selection and classification
approaches for the analysis of mass spectrometry data. Analytica Chimica
Acta, 829:1–8, 2014.
(c) H. Seetha, R. Saravanan and M. N. Murty, Pattern synthesis using multiple
Kernel learning for efficient SVM classification. Cybernetics and Infor-
mation Technologies, 12:77–94, 2012.
6. Is it meaningful to combine several binary classifiers to realize multi-class
classification?

Relevant References
(a) V. Sazonova and S. Matwin, Combining binary classifiers for a multi-class
problem with differential privacy. Transactions on Data Privacy, 7:51–70,
2014.
(b) A. Kontorovich and R. Weiss, Maximum margin multiclass nearest neigh-
bors, arXiv:1401.7898, 2014.
(c) T. Takenouchi and S. Ishii, A unified framework of binary classifiers
ensemble for multi-class classification. Proceedings of ICONIP, 2012.
(d) K. Hwang, K. Lee, C. Lee and S. Park, Multi-class classification using
a signomial function. Journal of the Operational Research Society,
doi:10.105.7/jors.2013.180, Published online on 5 March 2014.
7. Which classifier is ideally suited to deal with a large number, say 1000, classes?
Can one design one?

Relevant References
(a) K. Mei, P. Dong, H. Lei and J. Fan, A distributed approach for large-scale
classifier training and image classification. Neurocomputing, 144:304–
317, 2014.
(b) P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus and Y. LeCun,
OverFeat: Integrated recognition, localization and detection using convo-
lutional networks, arXiv:1312.6229, 2014.
(c) T.-N. Doan, T.-N. Do and F. Poulet, Large scale visual classification with
many classes. Proceedings of MLDM, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 33

Introduction 33

(d) M. Ristin, M. Guillaumin, J. Gall and L. van Gool, Incremental learning of

NCM forests for large-scale image classification. Proceedings of CVPR,
2014.
8. How do we classify patterns that have multiple labels?

Relevant References
(a) B. Akhand and V. S. Devi, Multi-label classification of discrete data, IEEE
International conference on fuzzy systems. FUZZ-’13, 2013.
(b) J. Xu, Fast multi-label core vector machine. Pattern Recognition, 46:885–
898, 2013.
(c) J. Read, L. Martino and D. Luengo, Efficient Monte Carlo methods for
multi-dimensional learning with classifier chains. Pattern Recognition,
47:1535–1546, 2014.
(d) J. Lee and D.-W. Kim, Feature selection for multi-label classification using
multivariate mutual information. Pattern Recognition Letters, 34:349–357,
2013.
9. It is possible to show equivalence between threshold based algorithms like
leader and number of clusters based algorithms like the K -means algorithm.
Is it possible to axiomatize clustering to derive such equivalences?

Relevant References
(a) R. Chitta and M. N. Murty, Two-level k-means clustering algorithm for
k-tau relationship establishment and linear-time classification. Pattern
Recognition, 43:796–804, 2010.
(b) M. Ackerman, S. Ben-David and D. Loker, Towards property-based clas-
sification of clustering paradigms. Proceedings of NIPS, 2010.
(c) M. Meila, Comparing clusterings — An axiomatic view. Proceedings of
ICML, 2005.
10. In social networks that evolve over time the notion of outlier may have to be
redefined. How do we achieve it?

Relevant References
(a) N. N. R. R. Suri, M. N. Murty and G. Athithan, Characterizing temporal
anomalies in evolving networks. Proceedings of PAKDD, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 34

34 Introduction to Pattern Recognition and Machine Learning

(b) M. Gupta, J. Gao, C. Aggawal and J. Han, Outlier Detection for Temporal
Data. San Rafael: Morgan and Claypool Publishers, 2014.
(c) L. Peel and A. Clauset, Detecting change points in the large-scale structure
of evolving networks, arXiv:1403.0989, 2014.
11. What is the most appropriate scheme for clustering labeled data?

Relevant References
(a) V. Sridhar and M. N. Murty, Clustering algorithms for library comparison.
Pattern Recognition, 24:815–823, 1991.
(b) Q. Qiu and G. Sapiro, Learning transformations for clustering and classi-
fication, arXiv:1309.2074, 2014.
(c) A. Kyriakopoulou, Theodore Kalamboukis: Using clustering to enhance
text classification. Proceedings of SIGIR, 2007.
12. How do we exploit Map-Reduce framework to design efficient clustering
schemes?

Relevant References
(a) A. Ene, S. Im and B. Moseley, Fast clustering using MapReduce. Proceed-
ings of KDD, 2011.
(b) R. L. F. Cordeiro, C. Traina Jr., A. J. M. Traina, J. Lopez, U. Kang
and C. Faloutsos, Clustering very large multi-dimensional datasets with
MapReduce. Proceedings of KDD, 2011.
(c) S. Fries, S. Wels and T. Seidl, Projected clustering for huge data sets in
MapReduce. Proceedings of EDBT, 2014.
13. What is the best way to exploit knowledge in clustering? Which components
of the clustering system are more sensitive to the usage of knowledge?

Relevant References
(a) A. Srivastava and M. N. Murty, A comparison between conceptual clus-
tering and conventional clustering. Pattern Recognition, 23:975–981,
1990.
(b) X. Hu, X. Zhang, C. Lu, E. K. Park and X. Zhou, Exploiting Wikipedia as
external knowledge for document clustering. Proceedings of KDD, 2009.
(c) W. Pedrycz, Knowledge-Based Clustering: From Data to Information
Granules. New Jersey: John Wiley & Sons, 2005.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch01 page 35

Introduction 35

14. How do we generate a formal framework that can be used to summarize

documents?

Relevant References
(a) N. Karthik and M. N. Murty, Obtaining single document summaries using
latent Dirichlet allocation. Proceedings of ICONIP, 2012.
(b) B. Piwowarski, M. R. Amini and M. Lalmas, On using a quantum physics
formalism for multi-document summarization. Journal of the American
Society for Information Science and Technology, 63:865–888, 2012.
May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 37

Chapter 2

Types of Data

In statistical machine learning, pattern recognition and data mining,

data is represented as a pattern matrix or data matrix. We illustrate
it using the data in Figure 2.1 which is represented using the matrix
shown in Table 2.1. Note that in Table 2.1, there are eight patterns
which are represented using height in feet and weight in Kilograms.
There are two classes labeled chair and human corresponding to a
possible collection of chairs and humans; each class has four patterns
in this example collection. Each pattern is represented as a point in
the two-dimensional space, where weight is the first (X1 ) feature and
height is the second (X2 ) feature.

1. Features and Patterns

Typically, columns of the data matrix correspond to features. A fea-
ture is a property or characteristic of a pattern. For example, weight
and height are the two different features characterizing the chairs and
humans in the collection and class label is the dependent feature that
provides the semantic labels of the objects considered in Table 2.1.
In social sciences, the terms variable and characteristic are used in
the place of feature and in data mining area field and attribute are
popularly used instead of feature.
The rows of the data matrix correspond to patterns in the
collection. A pattern is also called point, vector, and sample in
pattern recognition, and corresponds to record, transaction, and
instance in data mining. So, a pattern is described by a collection of

37
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 38

38 Introduction to Pattern Recognition and Machine Learning

O
X2 O
O
O

XXX
XXX

X — Chair X1
O — Human

Figure 2.1. Example set of patterns.

Table 2.1. An example pattern matrix of eight patterns.

Pattern number Weight (in kgs) Height (in feet) Class label

1 10 3.5 Chair
2 63 5.4 Human
3 10.4 3.45 Chair
4 10.3 3.3 Chair
5 73.5 5.8 Human
6 81 6.1 Human
7 10.4 3.35 Chair
8 71 6.4 Human

feature values. In general, for machine-based pattern recognition and

machine learning, we deal with the pattern representations rather
than the patterns themselves. For example, on a computer, we rep-
resent the chairs and humans based on their weight and height.
This amounts to two levels of lossy compression. Observe that
we cannot reproduce either a chair or a human from their respective
representations based on height and weight alone; this corresponds
to one level of abstraction. Further, the exact values of height and
weight cannot be measured in practice; we use values obtained using
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 39

Types of Data 39

instruments that measure these quantities with a finite precision. For

example, when we say that a person’s height is 5
10

, it is not the
exact value of height; it is an abstraction based on the precision of
the measuring device employed.

2. Domain of a Variable
Typically, each feature is assigned a number or a symbol as its
value. For example, the weight of the chair shown in the first row
of Table 2.1 is 3.5 kgs where 3.5 is a real number; weight of other
objects in the Table are all numbers. However, color of a chair could
be black, red, or green which we are not using in the representation
used in Table 2.1. So, color assumes symbolic values. It is possible
for the features to assume values that could be trees or graphs and
other structures. Specifically, a document collection is popularly rep-
resented as a document–term matrix, where each document (pattern)
corresponds to a row and each term in the collection corresponds to a
column. It is also possible to represent a document collection using an
inverted index as is done by search engines for information retrieval.
In the inverted index, a list of the documents in which a term occurs
is stored for each term in the collection. Even though such an index
is primarily used for information retrieval, it is possible to use the
index structure in other machine learning tasks like classification,
clustering, ranking and prediction in general.
A feature assumes values from a set: For example, the feature
weight has its values from the set of positive reals in general and an
appropriate subset in particular. Such a set is called the domain of
the feature. It is possible that weight can be measured in terms of
either kilograms or pounds. In both the cases, the domain of weight
is the set of positive reals. Also, it is possible that different features
can use the same domain of values. For example, a feature like height
also has set of positive reals as its domain. For some features, these
assignments are artificial or man-made; for others they are natural.
For example, it is possible to use either a number or a symbol as the
value of the feature ID or Identification of a person; the only require-
ment here is being able to discriminate between two individuals based
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 40

40 Introduction to Pattern Recognition and Machine Learning

on their ID values. Typically, we assign a positive integer as the ID

value; this is an artificial assignment. Instead of a positive integer,
we could have assigned a negative integer, a string of alphanumeric
characters as the value of ID or even a real number to stand for the
value of ID. However, it is natural to assign a non-negative integer
value to the feature number of dependents of any individual. Note
that number of dependents cannot be negative.
In pattern recognition (PR), it is possible to use an appropriate
abstraction and still achieve an acceptable recognition accuracy. Most
of the PR systems use compressed data for both training and clas-
sification. It is possible that the values used to represent a variable
may not satisfy one or more properties of the variable. For example,
consider the four objects shown in Table 2.2.
We may classify an object as light if its weight is less than 40 kgs,
otherwise we classify it as heavy. Note that the first two patterns are
light and the remaining two are heavy. The weight of the third pattern
is four times that of the second pattern. Consider a transformed set
of values for the four objects as shown in the third column of the
table. So, the transform captures the order of the values, but not
the ratio property. For example, the light objects have their trans-
formed weight values below a threshold value of 70 and if the value is
above 70, then the object is heavy. So, the first two objects are light
and the remaining two objects are heavy based on the transformed
values also. The transform is based on a monotonic function, which
is a function f :
→
such that for x, y ∈
, x ≤ y ⇒ f (x) ≤ f (y),
where
is the set of real numbers. However, the ratio property is not
preserved by the transformation; ratio of weights (second column) of
third and second objects is 4, where the ratio of the transformed

Table 2.2. Classification based on weight values.

Pattern number Weight (in kgs) Transformed weight Class label

1 10 12.5 light
2 15 47 light
3 60 93 heavy
4 85 120 heavy
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 41

Types of Data 41


values of these two objects is 1.98 93 47 . So, such a transformed
data may be adequate for pattern classification based on choosing or
learning the appropriate threshold value either in the input or in the
transformed domain. This example illustrates the property that the
way we measure and represent an attribute may not match its pro-
perties. However, classification is still possible. We discuss properties
associated with different types of variables in the next section.

3. Types of Features
There are different types of features or variables. We may categorize
them as follows:

1. Nominal variable: The simplest variable where the domain has

distinct values.
2. Ordinal variable: The domain of this variable is an ordered set;
so, the values are ordered.
3. Interval variable: The domain is an ordered set where the dif-
ferences between values have a meaningful interpretation.
4. Ratio variable: The domain is similar to that of the interval
variable where not only differences, but ratios are also meaningful.

In addition, we have temporal data, that is data which varies with

time and spatial data that varies spatially. Next, we examine different
types of data in detail.

3.1. Nominal data

A nominal feature fN assumes values from a set, that is the domain
of fN denoted by D(fN ). So, distinct objects can have different val-
ues that are drawn from the set D(fN ). Here, different elements of
D(fN ) are distinct and they are not ordered as D(fN ) is a set. Some
examples of nominal features are:

• Type of Curve: A possible domain of this feature is

{line, parabola, circle, ellipse}.
• Type of Publication: The domain of this variable will include
technical report, journal paper, conference paper, and book.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 42

42 Introduction to Pattern Recognition and Machine Learning

• TV Manufacturer : The domain of this attribute could be

{Sony, Philips, Samsung, LG, Videocon, Onida}.

It is possible that a nominal variable is either binary or non-binary.

A binary feature has a domain with two elements. Some examples of
binary nominal variables are:

• Gender : domain = {male, female}.

• Beverage available: domain = {tea, coffee}.

The most popular application area where nominal data is routinely

encountered is information retrieval. Here, a document is typically
viewed as a bag of words; so, it is a multiset without any ordering on
the values or elements of the set. We illustrate it with an example.

Example 1. Let us consider the following two documents.

• D1 : The good old teacher teaches several courses

• D2 : In the big old college

We represent the documents as multisets given by:

R1 = {The, good, old, teacher, teaches, several, courses} and

R2 = {In, the, big, old, college},

where R1 and R2 are representations of the documents D1 and D2

respectively. Typically, several simple operations are performed to
reduce the total number of terms. For example, converting uppercase
characters to lowercase gives us the following representations:

R1 = {the, good, old, teacher, teaches, several, courses} and

R2 = {in, the, big, old, college}.

Note that “The” is converted to “the” and “In” is transformed to

“in”. Further, by stemming, that is by transforming the words to
their stemmed forms, we can replace “teacher” and “teaches” by
“teach” and “courses” by “course” to get

R1 = {the, good, old, teach, teach, several, course} and

R2 = {in, the, big, old, college}.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 43

Types of Data 43

It is possible to represent a collection of documents by the union

of their multisets. In the current example, the collection of the two
documents may be equivalently represented by using a set of term
frequency pairs.

R = {(the, 2), (good, 1), (old, 2), (teach, 2), (several, 1), (course, 1),
(in, 2), (big, 1), (college, 1)}

In R, the elements are ordered pairs of the form (term, frequency),

where the first entry in the tuple is the term and the second entry is
its cumulative frequency. For example, the pair (the, 2) indicates that
the term “the” has occurred twice in the collection; note that “the”
has occurred once in each of the two documents. So, its cumulative
frequency is 2 (1+1). Similarly, “good” has occurred once in D1 and
it is absent in D2 ; so, its cumulative frequency is 1. Such a set of
term frequency pairs, or the histogram is shown in Figure 2.2. Here,
terms are shown on the horizontal axis and corresponding cumulative
frequencies are shown on the vertical axis.
Note it is possible to view each document also as a histogram of
term frequency values. For example, the histograms corresponding to
documents D1 and D2 , after casefolding and stemming are given by

Histogram(D1 ): {(the, 1), (good, 1), (old, 1), (teach, 2), (several, 1),
(course, 1)}.
Histogram(D2 ): {(in, 1), (the, 1), (big, 1), (old, 1), (college, 1)}.

Let us consider, in general, a collection D of n documents, given by

D = {D1 , D2 , . . . , Dn }.

the good old teach several course in big college

Figure 2.2. Histogram of term frequencies.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 44

44 Introduction to Pattern Recognition and Machine Learning

In summary, each document is a bag of words. Equivalently, it

may be viewed as a histogram of term-frequency values. Note that
mapping the document to a histogram this way is equivalent to con-
verting the underlying multiset into a set of ordered pairs (or tuples),
where for each term that occurs in the document we include a tuple
of (term, frequency). For example, in D1 (teach, 2) and in D2 (big, 1)
are such tuples. Here, document is a nominal variable and its value
is a specific document that may be viewed as a bag of words or as a
histogram or equivalently as a set of term-frequency tuples. There is
no order among the documents (histograms); so, the variable docu-
ment is nominal. More fundamentally, the terms which are elements
of the domain of the variable term are not ordered. So, the variable
term is nominal.

3.1.1. Operations on nominal variables

On such nominal variables, one can perform some operations; these
include comparison, mode and entropy. We illustrate this with an
example.

Example 2. Consider a dataset of 10 objects which are character-

ized by only one nominal variable; let the nominal variable be color.
The objects and their colors are given in the following set, where obji
stands for object i.

{(obj1, blue), (obj2, blue), (obj3, red), (obj4, green), (obj5, blue),
(obj6, green), (obj7, blue), (obj8, red), (obj9, blue), (obj10, green)}

Considering the set, we can say that the domain of the nomi-
nal variable color is Dcolor = {blue, red, green}; we can discriminate
objects based on the value the variable takes from the domain. For
example, obj1 and obj2 are identical and are different from obj3. In
a similar manner we can compare any pair of objects in the set based
on the value assumed. Further, note that there are 5 blue, 3 green
and 2 red objects in the collection, which means that the set can be
represented by a histogram given by {(blue, 5), (red, 2), (green, 3)}.
Once we have the histogram, we can obtain the mode and entropy
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 45

Types of Data 45

as follows:

• Mode: Mode is the most frequent value of the variable. In the

example, we can observe from the histogram that blue has the
highest frequency and so the mode is blue.
• Entropy: Entropy is a function of the frequencies of values. It
characterizes in some sense impurity of the dataset; if the variable
assumes only one value in the whole dataset, then the dataset
is pure and the entropy is zero. On the contrary, if the variable
assumes all the values with almost equal frequency in the set, then
the entropy is maximum. Shannon’s entropy is the most popular
characterization of entropy. It is given, for the dataset D by


d
Entropy (D) = − pi log pi ,
i=1

where pi is the probability of value i and d is the size of the domain

of the variable under consideration.
So, for the data shown in Example 2, the values of probabilities
obtained based on their frequencies of occurrence are:
5 3 2
P (blue) = = 0.5; P (green) = = 0.3; P (red ) = = 0.2.
10 10 10
For these probability values, the entropy is 0.4472 (using logarithm
to the base 10).

3.2. Ordinal data

In the case of ordinal data, the elements of the domain of the variable
are ordered, in addition to being distinct. Note that nominal vari-
ables satisfy only the property of their values being distinct. Some
examples of ordinal features are:

• Height of an object: domain = {very tall, tall, medium, short, very

short}.
• Ranking of documents. For example, quality of a document based
on a scale from 1–9; typically, reviewers are asked to rank a paper
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 46

46 Introduction to Pattern Recognition and Machine Learning

submitted for possible publication. Also, search engines provide an

output of ranked documents against a query posed by a user.
• Sentiment mined from a collection of documents (perhaps tweets)
on a product: domain = {very negative, negative, neutral, positive,
very positive}.

3.2.1. Operations possible on ordinal variables

As an ordinal variable has domain whose elements are distinct, all the
operations on nominal variables are possible on ordinal variables also.
So, comparison, mode, and entropy are possible. In addition, ordering
among the values permits operations like median and percentile.

• Median is the most centrally located value in the domain. For

example, medium value of variable Height; neutral for variable sen-
timent; and value 5 for the variable ranking based on a scale from
1 to 9 may be viewed as the median values in each case.
• Percentile makes sense when the values of the variable are ordered.
Top ten percentile indicates the value below which 90% of the
values are located.

It is possible to convert a nominal variable into an ordinal variable by

imposing some meaningful ordering. For example, even though val-
ues of ordinal variable color are not ordered, it is possible to impose
an order on the values based on the requirements of the application
considered. For example, the values of color can be ordered as Violet,
Indigo, Blue, Green, Yellow, Orange, and Red based on their wave-
lengths. We can illustrate this notion using the document data in the
next example.

Example 3. Consider the two documents and their representations

shown in Example 1. Observe the corresponding frequency details
shown in Figure 2.2. In this figure, the terms are ordered based on
the order in which they appeared in the documents and the sequence
in which the documents are processed. Each term may be viewed as
the value of a nominal variable; it is nominal because the ordering
employed has no ordinal flavor.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 47

Types of Data 47

An important ordering that has a great potential in Information

Retrieval is based on the frequency of the terms in the collection. Note
that there are nine terms in the collection as shown in Figure 2.2. It is
possible to order these terms based on their frequency of occurrence
in the collection. Further, if two or more terms have same frequency of
occurrence, then corresponding terms are arranged in lexicographic
order. Such a histogram is characterized by

Histogram(D) = {(in, 2), (old, 2), (teach, 2), (the, 2), (big, 1), (col-
lege, 1), (course, 1), (good, 1), (several, 1)}.

The corresponding histogram is shown in Figure 2.3. Such a fre-

quency ordered representation gives us an ordinal variable based on
the frequency of terms. Note that big (the fifth term) is viewed as
the median value among the nine terms. Another possibility is the
lexicographic ordering.
In this context, it is important to note that the fundamental law
in information retrieval is the Zipf ’s law; it is an empirical law. If
we rank the terms based on their frequency of occurrence in a large
collection of documents, with the most frequent term having the least
(first) rank and the least frequent term having the largest (last) rank,
then the frequency and rank of terms are related by
C
f (ti ) = ,
i
where f (ti) is the frequency of the ith term ti , C is the total number
of terms in the collection, and i is the rank of ti . This means the
frequency of the first term is twice that of the second term in the
collection. This law holds for large size collections.

old teach the big college course good in several

Figure 2.3. Histogram of ordered term frequencies.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 48

48 Introduction to Pattern Recognition and Machine Learning

Table 2.3. Representation of two documents in a binary form.

Big College Course Good In Old Several Teach The

D1 0 0 1 1 0 1 1 1 1
D2 1 1 0 0 1 1 0 0 1

3.2.2. Binary variables

It is possible that a nominal or an ordinal variable could be binary.
For example, consider D1 and D2 . We can represent them as binary
strings of fixed length based on the presence or absence, in a docu-
ment, of each term in the collection. There are nine distinct terms
in these two documents. They are: big, college, course, good, in, old,
several, teach, the in the lexicographically sorted order. Using these
nine terms, the two documents can be represented as binary strings
as shown in Table 2.3.

3.3. Interval-valued variables

For interval-valued variables, the differences between values are
meaningful. This is in addition to the properties satisfied by an ordi-
nal variable. However, zero value is not properly defined. Also, prod-
ucts and ratios are well defined here. An example is: Temperature in
Celcius and Fahrenheit:
Here, zero degrees Celcius or Fahrenheit do not mean no heat. How-
ever, 10◦ C is five degrees more than 5◦ C; specifically the difference
between 10◦ C and 5◦ C is the same as the difference between 5◦ C
and 0◦ C. In a sense, interval carries the meaning in the case of these
variables and permits a variety of operations. Here, 10◦ C is not two
times that of 5◦ C. However, the ratio of intervals makes sense; for
example, the difference between 10◦ C and 0◦ C is twice the difference
between 10◦ C and 5◦ C.

3.3.1. Operations possible on interval-valued variables

Mean and Standard Deviation are two possible operations on these
variables. For example, mean or average temperature is a popularly
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 49

Types of Data 49

used statistic. Here, (sample) mean and (sample) standard deviation

of a collection of values, x1 , x2 , . . . , xn , of the variable are given by

1
n
Mean = xi ,
n
i=1
 1
1 
n 2

Standard Deviation = (xi − Mean)2 .

n−1
i=1

Example 4. Let us consider temperature over 5 consecutive days

in summer in Bengaluru to be 35◦ C, 36◦ C, 36◦ C, 37◦ C, 36◦ C. Then
the mean value is 36◦ C and the variance is 0.4. Another example of
the interval-valued type is calender dates. Here, also differences make
sense, but not ratios.

3.4. Ratio variables

A ratio variable, in addition to properties like distinctness, order,
addition and subtraction, permits usage of multiplication and divi-
sion. Ratio variables are the most popular in Data Mining, Pattern
Recognition, and Machine Learning. A popular dichotomy of vari-
ables which is of recent origin, is to classify a variable as categorical
or numerical. Nominal and ordinal variables make up for the cate-
gorical data whereas numerical variables are typically of ratio type.
Interval-valued variables are so rare that all the numerical data is tac-
itly assumed to be of ratio type. Examples of ratio variables include
weight and height of physical objects.

3.5. Spatio-temporal data

There are several applications where the data is not static or fixed; it
is dynamic. Dynamic data is routinely understood as time varying.
However, in a more generic way spatial variations can also be con-
sidered as dynamic. There are several applications where the data is
spatio-temporal (variations in space or time or both). We consider
some details next. In such dynamic datasets, we have the data to be
one of the following:
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 50

50 Introduction to Pattern Recognition and Machine Learning

1. Spatial Data: In some applications, learning the predictive mod-

els is influenced by the spatial information of the data. For exam-
ple, in predicting earthquakes, it is possible that for all other con-
ditions being equal, an area in some geographical location has a
higher probability of being earthquake prone whereas some other
area in a different geographical location may have a lower proba-
bility. Also, a search engine might retrieve different sets of results
for queries posed from different geographical locations. For exam-
ple, for a query posed from outside India to a search engine on
“Center for Artificial Intelligence”, “Centre for Artificial Intelli-
gence and Robotics at www.utm.my/cairo” may be ranked higher;
however, for the same query posed from Bengaluru, India it is
highly likely that “Center for Artificial Intelligence and Robotics,
Bangalore” will show up earlier. There are several other applica-
tions where spatial variations are important.
2. Temporal Data: Data that varies with time is called temporal
data. Time series data is popular and here the successive time
intervals are equally spaced or regular. Speech signal is a popular
example of time series data. Another example of temporal data
is encountered in web clicks; such a data is an example of stream
data where the clicks can be made at irregular intervals of time.
3. Spatio-temporal Data: In some applications, data varies both
with space and time. For example a search engine provides ranked
results to a query. These results might change with time and also
based on geographical location from which the user queried the
search engine. Another important application area is earth sci-
ences; for example, climatic conditions might change with time
and space.

4. Proximity measures
Matching is of paramount importance in several areas of computer
science and engineering. Matching trees, graphs, sequences, strings,
and vectors is routinely handled by a variety of algorithms. Match-
ing is either exact or approximate. Exact matching is popular in
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 51

Types of Data 51

editors, databases, compilers and in operating system tools like grep.

However, approximate matching is popular in pattern recognition
and machine learning. A popular representation of patterns is by
using vectors of feature values. Proximity between vectors is char-
acterized by Similarity and Distance (Dissimilarity). Typically, sim-
ilarity between two patterns is a numerical value in the range [0, 1];
similarity is higher when the matching between the objects is higher.
Popularly distance functions are used to characterize dissimilarity.
Similarity is captured by using a monotonically decreasing function
of distance; larger the distance between a pair of patterns lower the
similarity and smaller the distance, higher the similarity.
It is tacitly assumed that the distance function, d(x, y) where x
and y are patterns, is a metric which satisfies the following properties:
1. Positive Reflexivity
d(x, y) > 0 if x = y and d(x, y) = 0 if x = y.
This property has an intuitive appeal. Distance between a pair
of objects is never negative. Further, if two objects (or more
appropriately their representations) are identical, then they are
highly similar and so the distance between them is minimum which
is 0.
2. Symmetry
d(x, y) = d(y, x) for all x and y.
This property has practical implications which may be explained
as follows. Consider a dataset X given by
X = {X1 , X2 , . . . , Xn }.
The corresponding distance/dissimilarity matrix is shown in
Table 2.4. Here, d(Xi , Xj ) which corresponds to the distance
between patterns Xi and Xj is represented as d(i, j) for the sake
of simplicity. By observing that each of the diagonal entries in
the matrix is of the form d(i, i) with a value 0 as specified by
Property 1 and because of symmetry d(i, j) = d(j, i) for 1 < i,
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 52

52 Introduction to Pattern Recognition and Machine Learning

Table 2.4. Distance/dissimilarity matrix.

d(1,1) d(1,2) d(1,3) ··· d(1,n)
d(2,1) d(2,2) d(2,3) ··· d(2,n)
d(3,1) d(3,2) d(3,3) ··· d(3,n)
.. .. .. ..
. . . ··· .
d(n,1) d(n,2) ··· d(n,n)

Table 2.5. Symmetric distance matrix.

0
d(2,1) 0
d(3,1) d(3,2) 0
.. .. .. ..
. . . ··· .
d(n,1) d(n,2) ··· 0

j < n, we can simplify the matrix in Table 2.4 to the lower-

triangular matrix given in Table 2.5. Note that the value of any
entry, d(i, j), in the upper-triangular portion can be obtained by
looking at the corresponding entry d(j, i), which has the same
value as d(i, j), in the lower-triangular portion. This means that
we can achieve reduction in both the time required to compute the
distances and the space required to store the corresponding values.
n(n−1)
For n patterns, we need to compute 2 distances to specify
2
the lower-triangular matrix rather than n distances required to
characterize the entire matrix shown in Table 2.4. This means
that there is a reduction by more than a factor of 2. In a similar
sense, one can analyze to show that the space required to store the
possible entries in the lower-triangular matrix shown in Table 2.5
also reduces by a factor of 2. This is an advantage in using a
distance function that is symmetric. Note that such a property is
intrinsic to a pair of nodes in a friendship network. If we interpret
such a network as a graph, then the graph is undirected.

3. Triangular Inequality

d(x, z) ≤ d(x, y) + d(y, z) for all x, y, and z.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 53

Types of Data 53

This inequality is so-called because any three patterns (or equiva-

lently points) Xi , Xj , and Xk could be viewed as three vertices of
a triangle and it is well-known that length of any side of a triangle
is bounded by the sum of the lengths of the remaining two sides of
the triangle. Note that length of the side (edge) uv corresponding
to vertices u and v is characterized by the distance between points
u and v.
Triangle inequality is exploited in pattern recognition in deriv-
ing a variety of properties associated with pattern classifiers
including the Nearest Neighbor classifier, and divide-and-conquer
clustering.

A distance function that satisfies the above three properties is a

metric. Such a function is called distance measure.

(i) Distance measures

Various distance functions are defined and used on data represented

using a collection of features. It is convenient to start with distances
between binary vectors.

(ii) Distance between binary vectors

Let us consider two binary strings x (= (x1 , x2 , . . . , xd )) and y (= (y1 ,

y2 , . . . , yd )). The distance between such a pair of strings is obtained
by matching xi and yi for all i and considering the corresponding
numbers which are given by:

n11 = Number of times xi = yi = 1 for i = 1, 2, . . . , d

n00 = Number of times xi = yi = 0 for i = 1, 2, . . . , d
n10 = Number of times xi = 1 and yi = 0 for i = 1, 2, . . . , d
n01 = Number of times xi = 0 and yi = 1 for i = 1, 2, . . . , d

Using these matching counts, we can define the distance between

x and y as

n10 + n01
d(x, y) = .
n11 + n10 + n01 + n00
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 54

54 Introduction to Pattern Recognition and Machine Learning

Example 5. We illustrate it using the binary representation of the

documents shown in Table 2.3. Here, n11 = 2, n10 = 4, n01 = 3, and
n00 = 0. So, distance between D1 and D2 is given by

7
d(D1 , D2 ) = .
9
(iii) Similarity between binary strings

Typically, similarity, s(x, y), between a pair of vectors x and y ranges

between 0 and 1.

s(x, y) = 1 − d(x, y).

This means that in the example above

2 7
s(D1 , D2 ) = , as d(D1 , D2 ) = .
9 9

Note that

n10 + n01
s(x, y) = 1 −
n11 + n10 + n01 + n00
n11 + n00
= .
n11 + n10 + n01 + n00

A more popular measure of similarity is the Jaccard coefficient,

J(x, y) which ignores n00 and is given by

n11
J(x, y) = .
n11 + n10 + n01

Note that J(D1 , D2 ) = 29 . Also, s(x, y) and J(x, y) need not be same.
For example, if there are two 5 bit binary strings x and y given by

x = 1 0 0 0 1 and
y=01001
then s(x, y) = 35 = 0.6 and J(x, y) = 1
3 = 0.33.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 55

Types of Data 55

(iv) Minkowski distance

A family of distance measures is the Minkowski measure and between
a pair of patterns x and y it is given by
 p 1
 q

dq (x, y) = |xi − yi|q .

i=1

1. The simplest distance in this case is the one with q = 1; the

corresponding distance is called the L1 -norm (because q = 1) or
city-block distance (because it is the sum of the absolute differences
across the variables; it is similar to the distance traveled to reach
a house from another in a city block). It is specified by

p
d1 (x, y) = |xi − yi |.
i=1

2. Euclidean distance or L2 -norm (because q = 2) is the most pop-

ular among the dissimilarity measures used in pattern recog-
nition. Euclidean distance between two patterns x and y in a
p-dimensional space is given by
 p 1
 2
2
d2 (x, y) = (xi − yi ) .
i=1

Euclidean distance is popular because:

• It is easy for human comprehension and visualization as it char-
acterizes the crow-flying distance.
• It is translation invariant. This may be explained using two
patterns x and y in a p-dimensional space represented by x =
(x1 , x2 , . . . , xp ) and y = (y1 , y2 , . . . , yp ). The translated versions
of x and y are x
and y
given, in general, by x
= (x1 + h1 , x2 +
h2 , . . . , xp + hp ) and y
= (y1 + h1 , y2 + h2 , . . . , yp + hp ) where
h1 , h2 , . . . , hp are real numbers. Note that the euclidean distance
between x and y is the same as that between x
and y
. In other
words, d2 (x, y) = d2 (x
, y
) which shows that euclidean distance
does not change with translation.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 56

56 Introduction to Pattern Recognition and Machine Learning

• It is rotation invariant. For example, let x = (x1 , x2 ) and y =

(y1 , y2 ) be two points in a two-dimensional space. It is possible
to view them as two diagonally opposite vertices of a rectangle
so that the line xy is a diagonal. By rotating the line around its
midpoint by 90◦ , we get the line x
y
where x
= (x1 , y2 ) and
y
= (y1 , x2 ). Note that this forms the other diagonal. So, the
euclidean distances are equal as the diagonals in the rectangles
are of equal length. That is d2 (x, y) = d2 (x
, y
).
• However, a weakness is that it is not scale-invariant. For exam-
ple, let us consider two two-dimensional patterns x = (x1 , x2 )
and y = (y1 , y2 ), x = y. If “x” and “y” are scaled versions
of x and y by a factor α, such that x” = (αx1 , αx2 ) and
y” = (αy1 , αy2 ), then we can observe that d(x, y) = d(x”, y”),
in general, where d(.,.) is the euclidean distance; they are equal
if and only if α = 1.
3. Another popular distance measure is the Max distance or
L∞ -norm: It is the maximum of the absolute differences among
all the features. Given two patterns x and y, the max distance is
defined as:

d∞ (x, y) = max |xi − yi |.

i

This simplification is possible because as q → ∞ the qth power of

maxi |xi − yi | is larger than the qth power of any of the other p − 1
components. This means that the sum over the qth powers of all
the p terms can be approximated by the qth power of maxi |xi −yi|
as the other p − 1 terms are negligible. So, the 1q th power of the
sum can be approximated by maxi |xi − yi|.

4.1. Fractional norms

In applications involving high-dimensional patterns, it is observed
that Minkowski distances between patterns do not give a meaningful
characterization of dissimilarity. One class of distance measures is
based on fractional norm. Here in the norm given by
1
dq (x, y) = [Σpi=1 |xi − yi |q ] q
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 57

Types of Data 57

the value of q is a fraction. We consider a simple example to illustrate.

Example 6. Consider two four-dimensional patterns X1 and X2

given by

X1 = (1, 1.16, 1, 0) and X2 = (1.36, 1, 1, 0).

1
Then d0.5 (X1 , X2 ) = [0.6 + 0.4 + 0 + 0] 0.5 = 12 = 1.
It is not difficult to see that

d1 (X1 , X2 ) = 0.36 + 0.16 = 0.52 and

d2 (X1 , X2 ) = [0.362 + 0.162 ]0.5 = 0.39.

To illustrate it further let us consider another pair of patterns X3

and X4 given by

X3 = (1, 2, 1, 0) and X4 = (3, 0, 1, 0).

The corresponding distances are

1 √
d0.5 (X3 , X4 ) = [20.5 + 20.5 ] 0.5 = [ 8]2 = 8.

Further, observe that

d1 (X3 , X4 ) = 2 + 2 = 4 and
√
d2 (X3 , X4 ) = [22 + 22 ]0.5 = 8 = 2.828.

4.2. Are metrics essential?

There are a variety of distance functions that are not metrics; they
violate one or more of the properties of metric distances. Such
non-metric distance functions have been popularly used in pattern
recognition and machine learning. We will examine some of these
distance functions. Specifically, we will see how each of the three
properties of a metric get violated by distance functions. Let us con-
sider examples for each.

1. Violation of Positive Reflexivity: Let us consider a distance func-

tion that returns negative values. Specifically, consider a two-class
problem in a one-dimensional space.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 58

58 Introduction to Pattern Recognition and Machine Learning

Example 7. Let the training data consisting of one-dimensional

patterns be
Class 1: 1, 3, 8, 12 Class 2: 57, 63, 44
Let the distance function be given by

− | x − y | if | x − y |< T
d(x, y) = .
(x − y)2 Otherwise
Now consider a test pattern with value 40 and let T be 20; the dis-
tances between the test pattern and the training patterns respec-
tively are:
Class 1: 1521, 1369, 1024, 784 Class 2: −17, 529, −4
Now consider k nearest neighbors based on the distances; so, the
3(k = 3) nearest neighbors are 57, 44, and 63 respectively. All the
three are from Class 2. Hence by using the class label of the major-
ity class, we classify the test pattern to Class 2. This illustrates
that distances can act meaningfully even if positive reflexivity is
violated. However, in practice violation of positive reflexivity is
not common.
2. Violation of Symmetry: The most popularly used distance func-
tion that violates symmetry is the Kullback–Leibler distance (or
KL divergence) defined between two probability distributions. If
p and q are discrete distributions over the same variable such that
p = {p1 , p2 , . . . , pl } and q = {q1 , q2 , . . . , ql }, then the KL distance
between p and q is
 l

qi
KL(p, q) = − pi log 2 .
pi
i=1

Even though KL(p, q) is non-negative and is equal to 0 when p = q,

KL(p, q) = KL(q, p) in general; so, KL divergence does not satisfy
symmetry.
3. Violation of Triangular Inequality: Even though euclidean dis-
tance is a metric, the squared euclidean distance is not a metric; it
violates the triangular inequality. We illustrate it with an example.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 59

Types of Data 59

Example 8. Consider three points x, y, and z. Let the euclidean

distances between pairs of points be

d2 (x, y) = 2; d2 (y, z) = 3; d2 (x, z) = 4.

Consider the squared euclidean distances between the correspond-

ing pairs of points; they are given by

d22 (x, y) = 4; d22 (y, z) = 9; d22 (x, z) = 16.

Note that the squared euclidean distance does not satisfy the tri-
angular inequality because d22 (x, y) + d22 (y, z) = 13 and d22 (x, z) =
16 which means sum of lengths of two sides of the triangle (13) is
less than the length of the third side (16) violating the triangular
inequality.
However, the squared euclidean distance is symmetric. This
is because basically, euclidean distance is a metric and so it sat-
isfies symmetry. As a consequence, for any two points u and v,
d2 (u, v) = d2 (v, u) which means d22 (u, v) = d22 (v, u) ensuring that
the squared euclidean distance is symmetric. Similarly, one can
show that squared euclidean distance satisfies positive reflexivity.

4.3. Similarity between vectors

One of the most popular similarity functions between a pair of vectors
x and y is the cosine of the angle between the vectors x and y and
is given by
(x · y)
cos(x, y) = ,
x y
where x · y is the dot product between x and y and x is the length
of x or equivalently the euclidean distance between the origin and x.
This similarity function is used popularly in text mining and infor-
mation retrieval.

Example 9. Consider an example where x and y are two six-

dimensional vectors given by

x = (3, 1, 5, 1, 0, 0) and y = (1, 0, 1, 0, 0, 2).

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 60

60 Introduction to Pattern Recognition and Machine Learning

In this case

x · y = 3 ∗ 1 + 1 ∗ 0 + 5 ∗ 1 + 1 ∗ 0 + 0 ∗ 0 + 0 ∗ 2 = 8,
√ 1
x = (3 ∗ 3 + 1 ∗ 1 + 5 ∗ 5 + 1 ∗ 1 + 0 ∗ 0 + 0 ∗ 0) 2 = 36 = 6,
1 √
y = (1 ∗ 1 + 0 ∗ 0 + 1 ∗ 1 + 0 ∗ 0 + 0 ∗ 0 + 2 ∗ 2) 2 = 6 = 2.245.

8
So, cos(x, y) = 6 ∗ 2.245 = 0.594.

Some of the issues associated with computing the cosine values are:

1. If we normalize x and y so that they are made unit norm vectors,

then there is no need to divide the dot product with the norms of
vectors x and y while computing the cosine value. For example,
normalizing x, that is dividing each of the six components of x by
x , we get

1
x
= (3, 1, 5, 1, 0, 0) = (0.5, 0.166, 0.83, 0.166, 0, 0),
6

where x
is the normalized version of x. Similarly, the normalized

version y
of y is

1
y
= (1, 0, 1, 0, 0, 2) = (0.445, 0, 0.445, 0, 0, 0.89).
2.245

So, the dot product between x
and y
is given by

cos(x
, y
) = (0.5 ∗ 0.445 + 0.166 ∗ 0 + 0.83 ∗ 0.445

+ 0.166 ∗ 0 + 0 ∗ 0 + 0 ∗ 0.89) = 0.592.

So, it can simplify the cosine computation if the data are initially
normalized. Data is routinely normalized when classifiers based
on neural networks and support vector machines are used.
2. It is possible to approximate the cosine computation by ignoring
the smaller values in a normalized vector; we replace such small
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 61

Types of Data 61

values by 0. For example, in vector x
and y
we can ignore values

below 0.4. This will mean the approximate vectors x
and y
are:

x
= (0.5, 0, 0.83, 0, 0, 0) and y
= (0.445, 0, 0.445, 0, 0, 0.89).

In such a case, the approximate value of cos(x, y) is

0.5 ∗ 0.445 + 0.83 ∗ 0.445 = 0.592.

Such approximations, which can reduce the number of multipli-

cations while computing the dot product between two normalized
vectors, are not uncommon in computing the cosine of the angle
between two document vectors where each vector is very high-
dimensional. For example, dimensionalities in the range 50,000 to
100,000 are very common in document classification.
3. A variant of cosine similarity is the Tanimoto similarity given by
x·y
t(x, y) = .
x·x+y·y−x·y
Note the equivalence between the Tanimoto similarity and
Jaccard coefficient in the case of binary vectors by noting that

x · y = n11 , x · x = n11 + n10 , and y · y = n11 + n01 .

4.4. Proximity between spatial patterns

A simple scheme to process spatial patterns is by assuming that each
pattern is represented as a vector with additional features to pro-
vide the location details. Let us consider the data given in Table 2.6
corresponding to books where copies of the same book are sold in
different locations in the world. Note that based on Location we
can get the same two groups {1, 3, 5} and {2, 4, 6}, where the
first group of books are published in USA and the second group is
published in India. Further, the prices of books in the first group
are higher than the prices of books in the second group. In other
words, the distance between any pair of books from either group 1
or group 2 is small if the location feature or price are given more
importance. It is possible to assign different weightages to different
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 62

62 Introduction to Pattern Recognition and Machine Learning

Table 2.6. Book prices in different countries.

Title of the book Authors Publisher Location Price

1. Introduction to Tan, Steinbach and Addison- USA US$95.12

Data Mining Vipin Kumar Wesley
2. Introduction to Tan, Steinbach and Pearson India Rs. 525
Data Mining Vipin Kumar
3. Data Mining Han, Kamber Elsevier USA US$59.47
Concepts and
Techniques
4. Data Mining Han, Kamber Elsevier India Rs. 450
Concepts and
Techniques
5. Fundamentals of Brassard, Bratley Prentice-Hall USA US$51
Algorithmics
6. Fundamentals of Brassard, Bratley Prentice-Hall India Rs. 225
Algorithmics

features; perhaps a higher weight may be assigned to the Location

feature.

4.5. Proximity between temporal patterns

One of the application areas where time varying data is popularly
analyzed is speech processing. Typically speech signal is viewed as
a semi-stationary time series. A time series may be represented as
{(p1 , t1 ), (p2 , t2 ), . . . , (pm , tm )} where pi is the value at time ti , the
values are recorded at regular intervals of time. In speech analysis
and synthesis, it is assumed that the vowel sounds are periodic and
consonants are generated by white noise. Some of the recognition
tasks here include speaker recognition and speech recognition.
One way of carrying out classification and clustering on time
series data is to find a measure of proximity between time series. Once
this is taken care of, any distance-based classification or clustering
technique can be used. Let us look at some proximity measures for
time series data. We have already examined some of the popularly
used proximity measures which are given below.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 63

Types of Data 63

1. Minkowski distance
2. Cosine similarity
3. KL distance: This is an assymetric distance function. The sym-
metric version is:
d(a, b) + d(b, a)
D(a, b) = ,
2
where d(a, b) is the conventional KL distance between a and b.

4.6. Mean dissimilarity

The mean dissimilarity can be defined as a distance function

1
d
dm = dissim(pk , qk ),
d
k=1

|pk −qk |
where dissim(pk , qk ) = |pk |+|qk | .

Example 10. Let us consider two time series P : 1, −3, 2, 5 and

Q: 1, 2, 3, 5. Then the mean dissimilarity is 14 02 + 55 + 15 + 10
0
=
1.2
4 = 0.3.

4.7. Peak dissimilarity

Peak dissimilarity between the kth values pk and qk is given by

| pk − q k |
dp (pk , qk ) = .
2.max(| pk |, | qk |)

The peak dissimilarity between two time series P and Q will

then be

1
d
dp (P, Q) = dp (pk , qk ).
d
k=1

Example 11. Again considering the two time series P : 1, −3, 2, 5

and Q: 1, 2, 3, 5. Then the peak dissimilarity is 14 2 ∗0 1 + 2 ∗5 3 + 2 ∗1 3 +
0 1
2 ∗ 5 = 4 = 0.25.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 64

64 Introduction to Pattern Recognition and Machine Learning

4.8. Correlation coefficient

If P and Q are two d-dimensional time-series, the correlation coeffi-
cient CC(P, Q) is
d
(Pi − µP )(Qi − µQ )
CC(P, Q) = i=1 ,
sP .sQ
where µP is the mean of P and is given by

1
d
µP = Pi ,
d
i=1

sP is the scatter of P and is given by

 d 0.5

2
sP = (Pi − µP ) .
i=1

Similarly, µQ and sQ are the mean and scatter of Q respectively.

Based on the similarity coefficient CC(P, Q), one possible dis-
tance measure is

1 − CC(P, Q) β
dCC1 (P, Q) = ,
1 + CC(P, Q)
where an appropriate value of β is to be chosen. β is a value greater
than zero; typically β = 2.
Another distance measure is

dCC2 (P, Q) = 2(1 − CC(P, Q)).

Example 12. Considering P : 1, −3, 2, 5 and Q: 1, 2, 3, 5, note that

µP = 54 and µQ = 11 4 . Further, sP = 5.7 and sQ = 2.95. So,
dCC1 (P, Q) = 0.025 and dCC2 (P, Q) = 0.54.

4.9. Dynamic Time Warping (DTW) distance

In most cases, the DTW distance is superior to Euclidean distance for
classification and clustering of time series. Consider two time series
P of dimension dP and Q of dimension dQ . The intention is to align
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 65

Types of Data 65

these two time series. This is done by constructing a matrix of size

dP × dQ . The alignment between the ith value in P, Pi , and jth value
in Q, Qj , is calculated as

d(Pi , Qj ) = |Pi − Qj |.

We have suggested the Manhattan distance or L1 norm here; it

is possible to use other distances measures like the L2 norm.
It is necessary to find the best match between the two sequences.
For this we need to find a path through the matrix that minimizes
the total cumulative distance between them. A warping path through
the matrix is a contiguous set of matrix elements that characterizes
a mapping between P and Q. The kth element of a warping path W
is wk = (i, j)k and

W = w1 , w2 , . . . , wk , . . . , wm where max (dP , dQ ) ≤ m < dP + dQ + 1.

We need to find the optimal path W ∗ which is the path that

minimizes the warping cost. In other words
 
m
 
DT W (dP , dQ ) = W ∗ = min  (wi ).
i=1

Finding all paths is a time-consuming process. A dynamic pro-

gramming approach is used to evaluate the cumulative distance
DST (i, j). This will be

DST (i, j) = d(pi , qj ) + min{DST (i − 1, j), DST (i, j − 1),

DST (i − 1, j − 1)}.

This means that the cumulative distance DST (i, j) is the sum of
the distance in the current cell d(i, j) and the minimum of the cumu-
lative distances of the adjacent elements, DST (i−1, j), DST (i, j−1),
and DST (i − 1, j − 1).
To decrease the number of paths considered and to speed up the
calculations, some constraints are considered. The paths considered
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 66

66 Introduction to Pattern Recognition and Machine Learning

should satisfy the following conditions:

1. Boundary condition: The path must start in w1 = (1, 1) and end

in wm = (dP , dQ ).
2. Continuity condition: The path must form a sequence which
means that the indices i and j can only increase by 0 or 1 on
each step along the path. If we are at point (i, j), the next point
can only be (i + 1, j + 1) or (i + 1, j) or (i, j + 1).
3. Monotonic condition: The path cannot go backwards in the
indices. Both the i and j indices can either stay the same or
increase. They can never decrease.
4. Slope constraint condition: The warping path cannot be too steep
or too shallow. This means that moving for a long time along one
dimension is not allowed. This is expressed as the ratio ab which
gives the slope of the path. If we move b steps in the x direction,
it is necessary to make the next step in the y direction.
5. Adjustment window condition: The warping path cannot drift
very far from the diagonal. The distance the path can wander
is limited to a window of size w directly above and to the right of
the diagonal.

4.9.1. Lower bounding the DTW distance

Since DTW computation is very demanding in terms of time, it would
help if we could find a lower bounding function which will prune
sequences which could not possibly be the best match. The lower
bounding measure should be fast to compute and should be a rela-
tively tight lower bound. We have seen that the number of warping
paths which are covered are limited by not allowing the path to stray
too far away from the diagonal. The part of the matrix the warping
path is allowed to visit is called the warping window or a band. Some
of the warping bands used are:

1. Sakoe–Chiba Band
2. Itakura Parallelogram

They are illustrated in Figure 2.4.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 67

Types of Data 67

Q Q

P P

Sakoe−Chiba Band Itakura Parallelogram

Figure 2.4. Limiting scope of the warping path.

The Sakoe–Chiba Band or the Itakura Parallelogram is used to

create a bounding envelope above and below the time series P. Then
the sum of the distances from every part of the time series Q not
falling within the bounding envelope, to the nearest orthogonal edge
of the bounding envelope, is returned as its lower bound. This helps
to prune off a number of expensive DTW computations and reduce
the complexity of the DTW algorithm.
We need to constrain the indices of the warping path wk = (i, j)k
such that j − Ri ≤ i ≤ j + Ri where Ri defines the allowed range
of warping for a given point in a sequence. For the Sakoe–Chiba
Band, R is independent of i but for the Itakura Parallelogram, R is
a function of i. R can be defined as

Ri = d, 0 ≤ d ≤ m,

where Ri is the height above the diagonal in the y direction as well as

the width to the right of the diagonal in the x direction. Any global
constraint can be defined using R. For example, a Sakoe–Chiba Band
of overall width 11 or a width of 5 above and to the right of the
diagonal can be represented as

5 if 1 ≤ i ≤ m − 5,
Ri =
m − i if m − 5 < i ≤ m.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 68

68 Introduction to Pattern Recognition and Machine Learning

An Itukara Parallelogram can be defined as

   

 2
1≤i≤
3

 3i m ,
3
Ri =      

 3 2 3

 m − i m < i ≤ m.
8 5 8

A simple algorithm for computing the DTW distance between P

of size dP and Q of size dQ is based on the folowing steps.

Algorithm for Computing DTW values

• Input — P : P1 , P2 , . . . , PdP , Q: Q1 , Q2 , . . . , QdQ , and w(window
size)
• Output — DT W (dP , dQ )
• Steps of the Algorithm

1. Initialize for i = 1 to dP
for j = 1 to dQ DTW(i, j) = ∞
DTW(0, 0) = 0
2. Update for i = 1 to dP
for j = max (1, i − w) to min(dQ , i + w)
DTW(i, j) =| Pi − Qj | + min(DT W (i − 1, j), DT W (i, j − 1),
DT W (i − 1, j − 1))

We illustrate it using a simple example.

Example 13. Let P : 11223 and Q: 1233. Let us look at i = 1 and

compute the DTW values for different values of j.

1. i = 1, j = 1: DT W (1, 1) = |1−1| + min(DT W (0, 1), DT W (1, 0)+

DT W (0, 0)) = 0 + min(∞, ∞, 0) = 0.
2. i = 1, j = 2: DT W (1, 2) = |1−2| + min(DT W (0, 2), DT W (1, 1)+
DT W (0, 1)) = 1 + min(∞, 0, ∞) = 1.
3. i = 1, j = 3: DT W (1, 3) = |1 − 3| + min(DT W (0, 3), DT W (1, 2),
DT W (0, 2)) = 2 + min(∞, 1, ∞) = 3.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 69

Types of Data 69

Table 2.7. DTW(i, j) values for all possible

i (1 to 5) and j (1 to 4).

j \i 0 1 2 3 4 5

0 0 ∞ ∞ ∞ ∞ ∞
1 ∞ 0 0 2 3 5
2 ∞ 1 1 0 0 1
3 ∞ 3 3 1 1 0
4 ∞ 5 5 2 2 0

Table 2.8. DTW(i, j) values for legal values

of i and j with w = 1.

j \i 0 1 2 3 4 5

0 0 ∞ ∞ ∞ ∞ ∞
1 ∞ 0 0 ∞ ∞ ∞
2 ∞ 1 1 0 ∞ ∞
3 ∞ ∞ 3 1 1 ∞
4 ∞ ∞ ∞ 1 2 1

4. i = 1, j = 4: DT W (1, 4) = |1 − 3| + min(DT W (0, 4), DT W (1, 3),

DT W (0, 3)) = 2 + min(∞, 3, ∞) = 5.
By assuming a value of w = 5 we get the results shown in Table 2.7.
Here, we give values for DTW(i, j) for all possible i and j. Note
that the cumulative cost DTW(5, 4) = 0 and the path goes through
(1, 1), (2, 1), (3, 2), (4, 2), (5, 3), and (5, 4). The corresponding entries
(0 values) are shown in bold face.
By selecting a value of w = 1, we get the values shown in
Table 2.8. Here, values outside the range are chosen to be ∞. All
the values in the legal region are shown in boldface.

Research Ideas
1. It is pointed out in Section 1 that pattern classification can be carried out
using approximate values of features. Is it possible to work out bounds on the
approximation for an acceptable level of classification?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 70

70 Introduction to Pattern Recognition and Machine Learning

Relevant References
(a) L. E. Ghaoui, G. R. G. Lanckriet and G. Natsoulis, Robust classification
with interval data. Technical Report UCB/CSD-03-1279, Computer Sci-
ence Division, University of California, Berkeley, 2003.
(b) Robust Classification, www.ims.nus.edu.sg/Programs/semidefinite/files/
IMS2006 Lect2.ppt [accessed on 25 October 2014].
(c) A. Ben-Tal, S. Bhadra, C. Bhattacharyya and J. Saketha Nath, Chance
constrained uncertain classification via robust optimization. Mathematical
Program, 127(1):145–173, 2011.
(d) A. Takeda, H. Mitsugi and T. Kanamori, A unified classification model
based on robust optimization. Neural Computation, 25:759–804, 2013.
2. Even though the domain of a variable is a large or/and possibly infinite set,
it may make sense to restrict the domain size to build a variety of classifiers.
For example, in document classification it is possible to ignore some terms as
illustrated by stemming in Section 3.1. How to exploit such a reduction in the
domain size in classification?

Relevant References
(a) A. Globerson and N. Tishby, Sufficient dimensionality reduction. Journal
of Machine Learning Research, 3:1307–1331, 2003.
(b) C. D. Manning, P. Raghavan and H. Schutze, Introduction to Information
Retrieval. Cambridge: Cambridge University Press, 2008.
(c) T. Berka, web.eecs.utk.edu/events/tmw11/slides/Berka.pdf, Dimensional-
ity reduction for information retrieval using vector replacement of rare
terms, 2011.
(d) D. Wang and H. Zhang, Inverse-category-frequency based supervised term
weighting schemes for text categorization. Journal of Information Science
and Engineering, 29:209–225, 2013.
3. Given the training dataset, how do we learn an appropriate distance/similarity
function that could be used in classification? Is it possible to use different
similarity functions in different regions of the feature space?

Relevant References
(a) M. Gonen and E. Alpaydn, Multiple Kernel learning algorithms. Journal
of Machine Learning Research, 12:2211–2268, 2011.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 71

Types of Data 71

(b) V. K. Garg and M. N. Murty, Feature subspace SVMs (FS-SVMs) for

high dimensional handwritten digit recognition. IJDMMM, 1(4):411–436,
2009.
(c) D. Ramanan and S. Baker, Local distance functions: A taxonomy, new
algorithms, and an evaluation. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 33:794–806, 2011.
(d) C.-M. Hsu and M.-S. Chen, On the design and applicability of distance
functions in high-dimensional data space. IEEE Transactions on TKDE,
21:523–536, 2009.
(e) J. H. Lee, K. T. McDonnell, A. Zelenyuk, D. Imre and K. Mueller,
A structure-based distance metric for high-dimensional space exploration
with multidimensional scaling. IEEE Transactions on Vizualization and
Computer Graphics, 20:351–364, 2013.

4. In the definition of Minkowski distance defined in Section 4, what is the impact

on classifiers if the value of q is a negative real number. Such distances could
be called negative norms.

Relevant References

(a) C.-M. Hsu and M.-S. Chen, On the design and applicability of distance
functions in high-dimensional data space. IEEE Transactions Knowledge
and Data Engineering, 21(4):523–536, 2009.
(b) C. C. Aggarwal, A. Hinneburg and D. A. Keim, On the surprising behavior
of distance metrics in high dimensional spaces. ICDT, 420–434, 2001.

5. It is assumed that either similarity or dissimilarity between patterns is ade-

quate; it is possible to derive one from the other. How do you characterize
them as separate functions and use a linear/nonlinear combination of them in
classification.

Relevant References

(a) M. Martin-Merino, Learning a combination of heterogeneous dissimilar-

ities from incomplete knowledge. Proceedings of ICANN, (3), 2010.
(b) K. C. Gowda and T. V. Ravi, Divisive Clustering of symbolic clustering
using the concept of both similarity and dissimilarity. Pattern Recognition,
28:1277–1282, 1995.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 72

72 Introduction to Pattern Recognition and Machine Learning

(c) Z. S. Xu and J. Chen, An overview of distance and similarity measures of

intuitionistic fuzzy sets. International Journal of Uncertainty, Fuzziness
and Knowledge-Based Systems, 16:529–555, 2008.
6. Is there an equivalence between categorical features and numerical features
in terms of classification. For example, Support Vector Machines (SVMs) and
Artificial Neural Networks (ANNs) are useful when the features are numerical
and frequent itemset based classification algorithms are ideally suited to deal
with categorical features.

Relevant References
(a) N. Lee and J. Kim, Conversion of categorical variables into numerical
variables via Bayesian network classifiers for binary classifications. Com-
putational Statistics & Data Analysis, 54:1247–1265, 2010.
(b) Natural Language Understanding, www.cs.stonybrook.edu/ ychoi/cse507/
slides/04-ml.pdf.
(c) I. W. Tsang, J. T. Kwok and P.-M. Cheung, Core vector machines: Fast
SVM training on very large data sets. JMLR, 6:363–392, 2005.
7. In Section 4.2, a variety of distance functions that violate one or more of
the metric properties have been detailed. Analyze further to rank the distance
functions based on the type of property violated.

Relevant References
(a) T. Skopal and J. Bustos, On nonmetric similarity search problems in com-
plex domains. ACM Computing Surveys, 43:34–50, 2011.
(b) M. Li, X. Chen, X. Li, B. Ma and P. Vitanyi, The similarity metric. IEEE
Transactions on Information Theory, 50:3250–3264, 2004.
8. Give example algorithms where the triangular inequality satisfied by the dis-
tance measure is exploited to simplify the learning algorithm.

Relevant References
(a) S. Guha, A. Meyerson, N. Mishra, R. Motwani and L. O’Callaghan, Clus-
tering data streams: Theory and practice. IEEE Transactions on Knowledge
Data and Engineering, 15:515–528, 2003.
(b) D. Arthur and S. Vassilvitskii, k-means++: The advantages of careful seed-
ing. SODA: 1027–1035, 2007.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch02 page 73

Types of Data 73

9. In high-dimensional spaces it is useful to approximate the distance/similarity

computation. For example, computation of the cosine of the angle between two
high-dimensional document vectors is approximated by ignoring the entries
that are smaller than a threshold.

Relevant References
(a) S. V. Dongen and A. J. Enright, Metric distances derived from cosine
similarity and Pearson and Spearman correlations, arXiv:1208.3145v1,
2012.
(b) C. D. Manning, P. Raghavan and H. Schutze, Introduction to Information
Retrieval. Cambridge: Cambridge University Press, 2008.
10. How do we speedup the computation of DTW values further.

Relevant References
(a) T. Giorgino, Computing and visualizing dynamic time warping alignments
in R: The dtw package, cran.r-project.org/web/packages/dtw/vignettes/
dtw.pdf.
(b) Y. Sakurai, M. Yoshikawa and C. Faloutsos, FTW: Fast similarity search
under the time warping distance. PODS: 326–337, 2005.
May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 75

Chapter 3

Feature Extraction and Feature Selection

Feature extraction is the process of determining the features to be

used for learning. The description and properties of the patterns are
known. However, for the classification task at hand, it is necessary
to extract the features to be used. It may involve carrying out some
arithmetic operations on the features like linear combinations of the
features or finding the value of a function. Feature selection is the
process of discarding some of the features of the patterns and using
only a subset of the features.
Feature extraction and feature selection are very important prob-
lems in machine learning. To enhance the performance of the machine
learning algorithm, it is necessary that the right or discriminating
features are used. Use of irrelevant features may lead to unnecessary
computation. In addition, due to the peaking phenomenon, as the
number of features increases, a larger training dataset is required to
get good classification accuracy. With a fixed dataset, the accuracy
increases with the number of features upto a point, beyond which, if
the number of features increases, the classification accuracy begins
to drop.
Feature selection also reduces the time complexity of the train-
ing process. If the data size is very large, the training time will be
very high. In some cases, the training may not be completed in an
acceptable time period. If feature selection is first carried out, then
the training time may come within acceptable limits.
In addition, the distances in high dimensional spaces may not
capture the intended similarity appropriately. In high dimensional

75
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 76

76 Introduction to Pattern Recognition and Machine Learning

spaces, distances between a point X and its nearest neighbor nn(X)

and its furthest neighbor fn(X) are shown to satisfy d(X, nn(x)) =
d(X, fn(X)) where d(X, nn(X)) is the nearest neighbor distance from
X and d(X, fn(X)) is the distance of furthest neighbor from X.
If feature selection is carried out, this problem is mitigated as the
dimensions are reduced.

1. Types of Feature Selection

There are three feature selection methods. They are:

1. Filter methods
2. Wrapper methods
3. Embedded methods

The filter methods compute a score for each feature and then
select features according to the score. The wrapper methods score
feature subsets by seeing their performance on a dataset using a clas-
sification algorithm. The embedded methods select features during
the process of training.
The wrapper method finds the feature subset by the method of
search. For every subset generated, its performance is evaluated on a
validation dataset using a classification algorithm. The feature subset
giving the best performance on the validation dataset is selected.
Some of the methods used are:

1. Exhaustive enumeration
2. Branch and bound technique
3. Sequential selection
(a) Sequential Forward Selection
(b) Sequential Backward Selection
(c) Sequential Floating Forward Selection
(d) Sequential Floating Backward Selection
4. Min–max approach
5. Stochastic techniques like genetic algorithms (GA)
6. Artificial Neural Networks
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 77

Feature Extraction and Feature Selection 77

This chapter explains the filter method where the interaction

between the features is considered. Consider a dataset with patterns
which are d-dimensional. Feature selection is the task of selecting k
features where 1 ≤ k < d. So if we have a feature set {f1 , f2 , . . . , fd },
this entails finding the score of each feature fi. This score represents
the degree of relevance, preference or importance of the feature. The
similarity between any two features fi and fj is then found using
these scores. This is used to remove the redundancy in the selected
features. In other words, the selected features should complement
each other. This means that the total similarity score over all the
selected features should be as small as possible.
Then an algorithm is used to maximize the total importance
scores and minimize the total similarity scores of a set of features.
An evaluation measure is used to find the importance score of each
feature. This is discussed in the sections which follow.
The process of selecting those features with largest total impor-
tance scores can be represented as


max [wi xi − sij xi xj ],
i

for j = i
s.t. x ∈ {0, 1} i = 1, . . . , d,
 i
i xi = k,

where xi is 1 if the ith feature is chosen and 0 if it is not chosen,

wi is the importance score of feature i
sij is the similarity between feature fi and fj ,
i.e.

1 if xi and xj are similar
sij =
0 otherwise
d is the total number of features
and k is the number of features selected.
If xi and xj are similar, then the objective function will be


max xi (wi − xj ).
i
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 78

78 Introduction to Pattern Recognition and Machine Learning

2. Mutual Information (MI) for Feature Selection

This function is based on the information gain and takes into account
how features work together. MI is used to measure the dependencies
between features and classes. MI between term t and class l measures
how much information the presence or absence of a term contributes
to making the correct classification decision on the class l. It is com-
puted as
P (ut , ul ) P (ūt , ul )
MI = −P (ut , ul )log 2 − P (ūt , ul )log 2
P (ut )P (ul ) P (u¯t )P (ul )
P (ut , ūl ) p(ūt , ūl )
− P (ut , ūl )log2 − P (ūt , ūl )log2 ,
P (ut )P (ūl ) P (ūt )P (ūl )
where
ut means that the document contains the term t; and
ūt means the document does not contain the term t;
ul means the document is in class l and;
ūl means the document is not in class l.
In other words, this can be written as
Nut ul NN ut ul
MI = log2
N (Nut ūl + Nut ul )(Nut ul + Nu¯t ul )
Nu¯t ul NN u¯t ul
+ log2
N ((Nu¯t ul + Nu¯t ūl )(Nut ul + Nu¯t ul )
NN ut ūl NN ut ūl
+ log2
N (Nut ul + Nut ūl )(Nut ūl + Nu¯t ūl )
Nu¯t ūl NN u¯t ūl
+ log2 ,
N (Nu¯t ul + Nu¯t ūl )(Nut ūl + Nu¯t ūl )
where
Nut ul = number of documents where the term is present which
belongs to the class,
Nu¯t ul = number of documents belonging to the class where the term
is absent,
Nut ūl = number of documents where the term is present which does
not belong to the class,
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 79

Feature Extraction and Feature Selection 79

Nu¯t ūl = number of documents where both the term and class are
absent,
N = Total number of documents.
If the distribution of the term in the whole document is the
same as its distribution in the class then MI = 0. If MI is large,
it means the term is in a document if and only if the document is in
the class. It makes sense to keep only informative terms and elimi-
nate non-informative terms so that the performance of the classifier
improves.
In the filter approach, a filter is used to discard features having
a low value of MI. We can also use the backward filter which discards
features if its value of MI with the class is less than some
with
probability p. The forward filter also can be used which includes a
feature if the MI is greater than
with a probability p.

3. Chi-square Statistic
The chi-square statistic is used to determine if a distribution
of observed frequencies differs from the theoretical expected fre-
quencies. This non-parametric statistical technique uses frequencies
instead of using the mean and variances, since it uses categorical
data.
The chi-square statistic is given by


χ2 = |(Ni − Ei )2 /Ei |,
i

where χ2 is the chi-square statistic, N is the observed frequency

and E is the expected frequency. The chi-square statistic sums the
discrepancy of the observed number of times each outcome occurs
and the expected number of times each outcome occurs for each
category. The discrepancy is computed as the square of the difference
between N and E divided by E.
The chi-square statistic can be computed to find the goodness of
fit or to test for independence of two sets of categories.
The chi-square test for independence is used to determine if two
variables are independent. The values calculated from the formula
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 80

80 Introduction to Pattern Recognition and Machine Learning

are compared with the values in the chi-square distribution table.

In the chi-square distribution table, for each degree of freedom, the
probability levels are given for different values of χ2 . Looking at the
probability levels, if it is below 5%, it is below the significance level
which implies that the two distributions are the same.
In feature selection, we test the independence between a term
and a class. The following χ2 value is calculated:
(Nut ul − Eut ul )2 (Nu¯t ul − Eu¯t ul )2
χ2 (D, t, l) = +
Eut ul Eu¯t ul
(Nut ūl − Eut ūl )2 (Nu¯t ūl − Eu¯t ūl )2
+ + ,
Eut ūl Eu¯t ūl
where
ut means that the document contains the term t, and
ūt means the document does not contain the term t;
ul means the document is in class l and,
ūl means the document is not in class l;
N = observed frequency and,
E = expected frequency.
This means
(N00 − E00 )2 (N01 − E01 )2 (N10 − E10 )2
χ2 (D, t, l) = + +
E00 E01 E10
(N11 − E11 )2
+ ,
E11
where
N11 = number of documents which contains term t and belongs to
class l;
N10 = number of documents which contain term t and do not
belong to class l;
N01 = number of documents which do not contain term t and
belong to class l;
N00 = number of documents which do not contain term t and do
not belong to class l.

If the χ2 value is larger than the one in the table giving χ2 distri-
butions for the degree of freedom, then it means that we need to reject
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 81

Feature Extraction and Feature Selection 81

the hypothesis that they are independent. This means that since the
two are dependent, the occurrence of the term makes the occurrence
of the class more likely. This means the term is useful as a feature.

4. Goodman–Kruskal Measure
The Goodman–Kruskal measure λ measures the interaction between
a feature and a class. If there are two classes + and −, the measure
for a feature i is
v
j=1 max(nj+ , nj− ) − max(n+ , n− )
λi = ,
n − max(n+ , n− )
where
ni+ = number of instances for which the value of a feature is Vi
and the class is ‘+’,
ni− = number of instances for which the value of a feature is Vi
and the class is ‘−’,
v = number of discrete values taken by the feature.

Domain of the input feature = {V1 , V2 , · · · , Vv }

n+ = number of instances of class ‘+’,
n− = number of instances of class ‘−’,
n = total number of instances.
The value of λi varies from 0 to 1, where λi = 0 means there is
no predictive gain in using the feature i to predict the class. λi = 1
means there is perfect predictivity in using feature i.

5. Laplacian Score
The Laplacian Score measures importance of a feature by its ability
for locality preserving. The features are given a score depending on
their locality preserving power. The algorithm is based on finding a
nearest neighbor graph for the set of nodes and finding the Laplacian
of the graph. Using the Laplacian, the Laplacian score is calculated
for every feature.
The algorithm is as follows:

1. Construct the nearest neighbor graph G for the set of points. For
every pair of points i and j, if xi is one of the k-nearest neighbors
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 82

82 Introduction to Pattern Recognition and Machine Learning

of xj or if xj is one of the k nearest neighbors of xi , then an edge

is drawn between i and j.

xi −xj
2
2. If nodes i and j are connected, put Wij = exp− k , where k
is a user-defined constant. If i is not connected to j, Wij = 0.
3. For a feature i, let the feature vector be
fi = (fi1 , fi2 , . . . , fid ).
The unit matrix I = (1, . . . , 1)T .

The matrix D is given by

D = diag(WI ).

The Laplacian L = D − W .

fiT DI
Then f˜i = fi − I T DI
I.

4. The Laplacian score of the ith feature is

f˜iT Lf˜i
Li = . (1)
f˜iT Df˜i

The justification for Eq. (1) is as follows. Considering all pairs

of points j and k, Wjk measures the similarity between the jth and
kth node. A good feature is one on which two data points are close
to each other if and only if there is an edge between the two points.
The criterion for choosing a feature using the above principle can be
written as:
 2
jk (fij − fik ) Wjk
Li = , (2)
var(fi )

where var(fi ) is the variance of the ith feature.

The numerator of Eq. (2) can be written as


 
(fij − fik )2 Wjk = fij2 + fik
2
− 2fij fik Wjk
jk jk

= 2fiT Df i − 2fiT Wf i = 2fiT Lf i .

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 83

Feature Extraction and Feature Selection 83

The denominator of Eq. (2) can be written as

var(fi ) = (fij − µi )2 Djj ,
j
 

Djj fiT DI
µi = fij  = .
j Djj I T DI
j

Removing the mean from the samples, we get

f T DI
f˜i = fi = iT I,
I DI

2
var(fi ) = f˜ij Djj = f˜iT Df˜i .
j

Putting f˜iT Lf̃ i = fiT Lf i and substituting in Eq. (2), we get

Eq. (1).

6. Singular Value Decomposition (SVD)

SVD is the decomposition of a rectangular matrix. Given a rectan-
gular matrix P which is n × m, it can be decomposed as

P = AΣB T ,

where A is a (n × k) matrix, B T is a k × m matrix and Σ is a k × k

square matrix. The rank of Σ is k and the diagonal elements have
singular values and the rows can be adjusted so that α1 ≥ α2 ≥ · · · ≥
αk > 0. A column P.i of P , which is a m vector can be expressed as
a linear combination of the m basis vectors of A (A.1 , A.2 , . . . , A.m )
using the singular values in Σ(α1 , α2 , . . . , αm ) and the ith column
B.iT of B T .
One method of using SVD for feature selection is to approximate
P by Pk1 where k1 < k. In Pk , the singular values which are zero or
close to zero are removed and the remaining k1 singular values are
retained. So we get
T
Pk1 = Am×k1 Σk1×k1Bk1×n .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 84

84 Introduction to Pattern Recognition and Machine Learning

The k1 singular values which are larger are chosen. High singular
values correspond to dimensions which have more variability. The
dimensions with lower singular values correspond to dimensions with
less variability which may not be discriminative features for learning.
Classification which has to be performed on P which is an m × n
matrix can now be performed on the matrix ΣB T which is a k1 × n
matrix where k1 < m.
It is also possible to find a basis vector A which will transform
any vector from the original vector space to the new vector space.
For a training dataset T , the resulting SVD T = ÃΣ̃B̃ T will yield a
set of basis vectors which can be used for the range of P . Here T is
m × r where r is the number of points in the training dataset. To use
the resulting SVD for T to transform P , it is necessary to project
the columns of P onto a subspace spanned by the first k1 columns of
Ã. A is transformed by computing (Ã.1 , Ã.2 , . . . , Ã.k1 )T A. Hence the
original patterns can be transformed using (Ã.1 , Ã.2 , . . . , Ã.k1 )T and
classification can be carried out on the transformed patterns.

7. Non-negative Matrix Factorization (NMF)

NMF is a feature extraction algorithm that decomposes multivariate
data by creating a user-defined number of extracted features which
results in a reduced representation of the original data. NMF decom-
poses a data matrix X into the product of two lower rank matrices B
and H so that X is approximately equal to BH i.e. X ≈ BH . NMF is
an iterative procedure which starts with some initial values of B and
H which are modified iteratively so that the product approaches X.
The procedure terminates when the approximation error converges.
It can also be terminated after a specified number of iterations. The
new features are a linear combination of the original features. NMF
does not allow negative entries in the matrices B and H. So if X is
a n × m matrix, then since finally X = BH, B is a n × r matrix and
H is an r × m matrix. The r columns of B can be called the bases
and then X will consist of original feature vectors and H is the newly
learned feature vector based on the basis matrix B. If r is chosen to be
smaller than n, then data compression and dimensionality reduction
takes place.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 85

Feature Extraction and Feature Selection 85

An NMF factorization can be defined as the optimization

problem:

minB,H D(XBH ),

such that B, H ≥ 0.
In this formulation, B and H have to be non-negative.
D(XBH ) means the divergence of X from BH and is the cost
function of the problem. If BH is actually Y , then

 xij

D(XY ) = xij log − xij + yij .
yij
i,j

This gives a measure of the error resulting in factorizing X into BH .

The objective function can be written as


n

m
O= [Xij log(BH )ij − (BH )ij ], (3)
i=1 j=1

where the value Xij is generated by adding Poisson noise to the prod-
uct (BH )ij . The objective function O is subject to the non-negativity
constraint i.e. all the non-zero elements of B and H are positive. An
iterative procedure is used to modify the initial values of B and H
so that the product approaches X. This procedure terminates when
the approximation error converges or after a user-defined number of
iterations. The update formula for B and H is as follows:

Xij
Bik = Bik Hkj ,
(BH )ij
j

Bik
Bik =  ,
l Blk

Xij
Hkj = Hkj Bik .
(BH )ij
i

The learned bases using NMF are not orthonormal to each other.
This is because to satisfy the non-negativity constraint the bases
cannot be orthonormal. One way of handling this is to consider the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 86

86 Introduction to Pattern Recognition and Machine Learning

learned non-orthonormal bases and orthonormalize the bases. This

is found to give better results for feature extraction.

8. Random Projections (RPs) for Feature Extraction

In RP, the original high-dimensional data is projected onto a
lower-dimensional subspace using a random matrix whose columns
have unit lengths. The original d-dimensional data is projected
to a k-dimensional subspace through the origin, using a random
k × d matrix P whose columns have unit lengths. If we have
n d-dimensional samples, X1 , X2 , . . . , Xn , it is converted to n
k-dimensional samples using a projection matrix P . This can be
written as

Yk×n = Pk×d Xd×n .

P is not orthogonal and hence cannot be called a projection.

Normally, if P is not orthogonal, it causes significant distortions in
the data. But since P is sufficiently close to being orthogonal, in a
high-dimensional space, the directions are almost orthogonal. The
choice of P is what is to be studied.
Unlike other methods such as PCA, random projections are easier
to do as they just require a matrix multiplication with a matrix
that can be generated without much difficulty. In PCA, we need to
compute the covariance matrix, decompose it into its singular value
form and choose the top k-eigenvectors. This entire process requires
a significant amount of effort.
The Euclidean distance between two points x1 and x2 after the
random projection is approximated by the scaled Euclidean distance

d
P x1 − P x2 .
k

d
The scaling fraction k is required due to the decrease in the
dimensionality of the data. This fraction is called the Johnson–
Lindenstrauss (J–L) scaling term.
In many cases where RP is used, the elements pij are Gaussian
distributed. Using a simpler distribution, according to Achlioptas the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 87

Feature Extraction and Feature Selection 87

elements pij can take the values


 1

 +1 with probability

 2s
√  1
pij = s. 0 with probability 1 − . (4)

 s



−1 with probability 1
2s
The value chosen for s is 1 or 3. The choice of pij as given
above uses integer arithmetic and results in a sparse matrix which
gives additional saving in time. The use of the J-L scaling to take
into account the reduction in dimensionality ensures that the inter-
pattern distances are preserved when random projections is used. The
choice of k, the reduced number of dimensions needs to be selected.
The L-J result gives a bound on k but it is possible to get a much
tighter bound. It is also possible to find k by carrying out experiments
on a validation set.
While using s = 1 or s = 3 gives sparse√random projections, it is
also possible to use s >> 3 such as s = d or s = logd d , where d is
the number of dimensions, leading to very sparse random projections
due to which the computation is speeded up significantly.
Some characteristics of Random Projections are now discussed.
In RP, the original matrix XεRn×d is multiplied with a random
matrix P εRd×k consisting of i.i.d N(0,1) entries. Let the rows of
X be denoted by {ai }ni=1 εRd . Let {bi }ni=1 εRk denote the rows of the
projected data. We can then write:
1
bi = √ P T ai .
k
Considering only the first two rows, if we denote

d

d
2
m1 = a1  = a21,i ; m2 = a2  = 2
a22,i , (5)
i=1 i=1

d
u= aT1 a2 = a1,j a2,j ; d = a1 − a2 2 = m1 + m2 − 2u. (6)
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 88

88 Introduction to Pattern Recognition and Machine Learning

Then
2 2
E(b1 2 ) = a1 2 ) = m1 ; var(b1 2 ) = m , (7)
k 1
2 2
E(b1 − b2 2 ) = d; var(b1 − b2 2 ) = d , (8)
k
1
E(bT1 b2 ) = u; var(bT1 b2 ) = (m1 m2 + u2 ). (9)
k
This shows that distances between two points and inner prod-
ucts can be computed in k dimensions. If k  d then there is a lot
of saving in time and space. Looking at var(b1 2 ), var(b1 − b2 2 )
and var(bT1 b2 ) in Eqs. (7), (8) and (9), it can be seen that Random
Projections preserve the pairwise distances in expected sense.

8.1. Advantages of random projections

Using random projections results in projecting the data onto a ran-
dom lower dimensional subspace. It is found to give results compa-
rable to other conventional dimensionality reduction methods such
as Principal Component Analysis (PCA) and SVD. At the same
time, use of RP is computationally less expensive. If a sparse ran-
dom matrix is used, the sparseness can be used to give additional
saving computationally. In additional, random projections preserves
the inter-pattern distances. RP can be used in a number of different
applications. Since RP preserves the distances between patterns, an
application such as high dimensional clustering is highly suited to
using RP.

9. Locality Sensitive Hashing (LSH)

LSH is a set of techniques for performing approximate search in high
dimensions. It is used for grouping points in space into ‘buckets’
based on some distance metric operating on the points. Points that
are close to each other under the chosen metric are mapped to the
same bucket with high probability. Two points are considered to be
close to each other if, after a projection of the points on to specified
dimensions, the two points remain close together. In other words, if
random projection operation is used to map the data points from a
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 89

Feature Extraction and Feature Selection 89

high-dimensional space to a low-dimensional subspace, then points

which are close to each other in the low dimensional subspace are
grouped together. For the purpose of grouping, the projection is put
into a set of hash bins so that nearby points in the original space will
fall into the same bin. The hash function for a point v is given by
 
x·v+b
h(v) = , (10)
w
where w is the width of each quantization bin, b is a random variable
uniformly distributed between 0 and w and x is a vector with com-
ponents that are selected at random from a Gaussian distribution.
The value of w influences the number of points that fall into each
bucket. Increasing w will increase the number of points that fall into
each bucket.
Then for any two points p and q in Rd that are close to each
other, there is a high probability P1 that they fall into the same bin.

PH [h(p) = h(q)] ≥ P1 for p − q ≤ R1 .

For points p and q in Rd that are far apart, there is a low prob-
ability P2 that they fall into the same bin. That is,

PH [h(p) = h(q)] ≤ P2 for p − q ≥ cR1 = R2 .

. is the L2 norm and R2 > R1 and P1 > P2 .

Further the dot product in Eq. (10) can be done k times, thereby
magnifying the difference between P1 and P2 . The ratio of the proba-
 k
bilities is increased since PP12 > PP12 . The k dot products are carried
out on v to transform it into k real numbers. Using Eq. (10), the k
inner products are quantized into the set of hash bins. In all the k dot
products, if the points and its nearest neighbor fall into the same bin,
then success is achieved. This has a probability of P1k of occurring.
It can be seen that as k increases, the probability decreases. This is
repeated a number of times for different random projections so that
the true nearest neighbor is found with an arbitrary high probability.
So if there are k dot products and M projections, then kM neighbors
are pooled to find the nearest neighbor.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 90

90 Introduction to Pattern Recognition and Machine Learning

Each data point is placed in a hash bucket described by k integer

values using the process of projection and quantization. Hence, it can
be seen that LSH performs probabilistic dimensionality reduction of
high dimensional data. Similar items are mapped to the same bin
and can be taken to be a single feature. One of the main appli-
cations of LSH is to provide an efficient nearest neighbor search
algorithm. Depending on the metric used, items are mapped to the
bins. Thus, for high-dimensional datasets, fast approximate nearest-
neighbor search can be carried out using locality sensitive hashing.
LSH can be viewed as random feature selection.

10. Class Separability

A measure is used for class separability for each feature. This measure
is calculated by using the Kullback–Leibler (KL) distance between
histograms of feature values. For each feature, for discrete valued
features, each value forms a bin. In the case of numeric features,
√
discretization is carried out using n equally spaced fields, where n
is the size of the training data. For each bin, the bin count is divided
by the total number of elements to get the probability that a feature
takes a value in each of the bins. We get pj (b = i | c = c1 ) which is
the probability that the jth feature takes a value in the ith bin given
a class c1 . For each feature j, the class separability is


c

c
CS j = δj (i, k),
i=1 k=1

where c is the number of classes and δj (i, k) is the KL distance

between the histograms of the two classes i and k and is given by


s 
pj (b = k | c = i)
δj (i, k) = pj (b = k | c = i) log ,
pj (b = k | c = k)
k=1

where s gives the number of bins. When the CS j values are sorted
in decreasing order, if the difference between the class separability of
two features differs by a very small value, say 0.001, then the feature
having the smaller distance is eliminated.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 91

Feature Extraction and Feature Selection 91

11. Genetic and Evolutionary Algorithms

Genetic and evolutionary algorithms are robust algorithms based on
the principles of natural selection and survival of the fittest. They
are optimization techniques which find the best solution from a num-
ber of competing solutions which forms the population of candidate
solutions.
The genetic algorithm (GA) consists of a population of chromo-
somes or strings. For feature selection, each chromosome consists of
d elements where if d is the number of features. Each element is 1
if that particular feature is selected and 0 if it is not selected. For
example, if d = 6, and a chromosome is 011001, it means that feature
2, 3 and 6 are selected. The fitness function for a chromosome is the
number of correct classifications on a validation set using the features
selected.
One method of feature selection using GAs is explained below.
The evolutionary algorithm uses a search strategy which experiments
with different combinations of features being chosen where each is a
candidate solution or a string. There is a population of strings. Each
such combination is evaluated. Operators are used to improve the
candidate solutions over time and the candidate solution which gives
the best evaluation is chosen and the feature subset pertaining to
this string is selected. The population in the GA consists of strings
which are binary in nature. Each string (or chromosome) is of length
d, with each position i being zero or one depending on the absence or
presence of feature i in the selected set. This means that each feature
subset is coded as a d-element bit string or binary valued vector.
Each string in the population is a feature selection vector α where
each α = α1 , . . . , αd where αi assumes a value 0 if the ith feature
is excluded and 1 if it is present in the subset. Each chromosome is
evaluated to compute its fitness by determining its performance on
the training set. This is done as explained below.
The penalty function p(e) is
e−t
exp m − 1
p(e) = ,
e−1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 92

92 Introduction to Pattern Recognition and Machine Learning

where
e = error rate,
t = feasibility threshold,
m = scale factor.
This penalty function is monotonic with respect to e. If e < t, then
p(e) is negative and, as e approaches zero, p(e) slowly approaches its
minimal value.
If e = t, then p(e) = 0 and
if e = t + m, then p(e) = 1.
For greater values of the error rate the penalty function quickly rises
towards infinity.

The score J(a) is given by

J(a) = l(a) + p(e(a)),

where a = (α1 , . . . , αd ) is a bit string representing a feature sub-

set. l(a) is the number of features absent in the feature subset. Let
Π = {a1 , . . . , an } denote the population of feature selection vectors.
Each feature selection vector is a string in the population. Since we
require a minimum of the score J(a), the fitness function is
 
f (ai ) = (1 +
) max J(aj ) − J(ai ) ,
aj
Π

where
is a small positive constant.

11.1. Hybrid GA for feature selection

This algorithm uses a guided hybrid GA for feature selection which
reduces the number of fitness function evaluations required. The
chromosome or string consists of d bits where d is the dimension-
ality of the data. Each bit of the string corresponds to one feature
and is 1 if the feature is included and 0 if it is excluded. The eval-
uation of a string s consists of two parameters c(s) which is the
cost function value and f (s) which is based on the performance
of the string on a validation set. The cost function c(s) can be
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 93

Feature Extraction and Feature Selection 93

written as:

c(s) = ns1 + p,

where p is the penalty given by

max(FA )−f (s)
p=2 γ − 1.

ns1 gives the number of included features in s, max(FA ) gives the

maximum of the f value considering all chromosomes in the complete
set of chromosomes A, and γ is a threshold value. The value f (s) of
the string s is given by
var(O − Ocv (s))
f (s) = 1 − ,
var(O)
where O is the response variable, and Ocv (s) gives the predicted
response from a cross-validation set. In other words,
m − mc
f (s) = 1 − ,
m
where
m = number of patterns in validation set;
mc = number of correctly classified patterns.
It can be seen that if max(FA ) − f (s) ≤ γ, then 0 ≤ p ≤ 1 holds.
If max(FA ) − f (s) γ, p increases rapidly. A steady state GA is
used. The set A is the archive which stores all the strings SA with
their modeling accuracy FA and cost function values CA . At each
iteration, there is a subset of A which is the reproductive population
R. This is the population subject to the genetic operators to produce
new strings. A subset of R denoted by B forms the set of strings from
which one string is selected for local feature elimination.
The algorithm can be written as:

1. Generate a number of initial random strings. Let one of the

strings consist of all ones (i.e. all features are included). These
strings from the archive A.
2. Evaluate A to generate FA and CA .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 94

94 Introduction to Pattern Recognition and Machine Learning

3. Set it = 1 and tr = A.

4. While convergence criterion is not met do.
5. Extract R from A.
6. Select two strings from R.
7. Apply crossover to get the children s1 and s2 .
8. Apply mutation on s1 and s2 .
9. Evaluate s1 and s2 .
10. Add s1 and s2 with f (s1 ), f (s2 ), c(s1 ) and c(s2 ) to A.
11. if it > δ then.
12. if (A > β + tr) then.
13. Train classifier C using A. Set tr = A.
14. end if.
15. Extract B from R.
16. Select one string s and do feature elimination on s using G.
17. Add evaluated strings to A.
18. end if.
19. it = it + 1.
20. end while.

In the above algorithm, upto δ iterations, the GA operates with

its operations of selection, crossover, and mutation. After δ iterations,
feature elimination is carried out in addition to GA operations. It can
also be seen that after every β iterations, the classifier C is retrained
with the A existing at that time. The convergence criterion would
be either that the cost function or modeling accuracy values are not
improving with iterations or that the maximum iterations have been
reached.
The genetic operators, selection, crossover and mutation are
explained below.
Selection involves selecting two parents from R. The first parent
s1 is chosen randomly from R. Binary tournament selection is used
to select the second parent. Two strings are selected from R and the
string with the smaller dissimilarity to s1 is chosen as the second
parent s2 .
The strings s1 and s2 are subject to subset size oriented
commonality crossover (SSOCF). This type of crossover maintains,
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 95

Feature Extraction and Feature Selection 95

on an average, the same number of included features in the off-

spring as either of the parents. The first child has features which
have the set of common elements of s1 and s2 which are of included
features. The first child also has features of x(s1 , s2 ) with probability
nc1 −nu1(s1 ,s2 )
nx(s1 ,s2 ) · x(s1 , s2 ) are the set of elements that are not common
among s1 and s2 . nu1(s1 ,s2 ) gives the number of common elements
between s1 and s2 which are of included features.
nc1 gives the number of elements of included features in the first
child.
Selecting R from A is done based on properties of the strings
such as whether it is inert, whether it is dominated by other strings
and its crowding index.
The crowding index of string s is given by:

|A|
1

cr(s) = nu1 (s, SA (i)),
ns1
i=1

ns1 gives the number of features included in s, nu1(s, SA (i)) is

the number of common elements that are included between s and
the strings in A.
For a string s if the full set of “leave one variable out” Slo in A
is such that nSlo = ns1 , then s is inert.
A string s1 is dominated i.e. dm(s1 ) = 1 if there is at least one
other string s2 in A where s12 ⊂ S11 and c(s2 ) ≤ c(s1 ). This means
that if dm(s1 ) = 1, there is another string in A which has a subset
of included features and of better or equal performance.
The procedure to find the subset R from A is as follows. The
strings in A are arranged in decreasing order of the CA values. The
first string is then put into R. For the other strings the following
procedure is used. For a string i, check if the string is dominated
or inert. If not, compute the dissimilarities Di between this string
and all the strings in SR . Also find the crowding index Cr (i). A(i)
is selected if its crowding index CR(i) < min(CR R ) and if there are
less than γ strings in R with a smaller dissimilarity to SA (i). Strings
are added to R till FA (i) < median(FA ).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 96

96 Introduction to Pattern Recognition and Machine Learning

Note that the dissimilarity between two strings s1 and s2 is

given by
nu1 (s1 , s2 )
d(s1 , s2 ) = 1 −  1 1 ,
ns1 .ns2
where nu1 (s1 , s2 ) gives the number of common elements that are
included in the feature set in s1 and s2 , n1s1 gives the number of
included features in s1 and n1s2 is the number of included features
in s2 .
The set B contains strings which can be selected for feature elim-
ination. The strings of R which are not inert are put into B. To select
among these strings, the individuals which are most dissimilar to the
inert individuals are extracted and put into B.
The individual to be selected from B for feature elimination is
chosen
  by using tournament selection. The size of the tournament is
|B|
k where k is a constant having a default of 5.
C is a classifier trained on the dataset available in A.

12. Ranking for Feature Selection

Ranking of features entails computing a score for each of the features
and sorting the features according to the scores. The order in which
the features occur are considered rather than the scores themselves.
Ranking of the features can be used to define a filter for feature
selection. Based on the ranking, some features can be eliminated. The
criteria used for ranking of the features is usually the performance
of the feature subset on an input training set. The ranking method
generally is a two-step process.

1. Each individual feature is used to construct a predictor of the

label.
2. The features are ranked based on the errors of these predictors.

Most approaches use the same training set both for constructing
the predictor and for evaluating its error.
An evaluation measure such as mean average precision (MAP) or
Normalized discount cumulative gain (NDCG) is used. The features
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 97

Feature Extraction and Feature Selection 97

are then sorted according to the score and this ordered list of features
is considered for feature selection.
Some feature ranking algorithms are discussed here.

12.1. Feature selection based on an optimization

formulation
In this method, the feature selection depends on the important score
of each feature fi and the similarity between pairs of features fi
and fj .
When the features selected are used on a validation set, the
results are used to evaluate the ranking. Some measures used here are
Mean Average Prediction (MAP), Normalized Discount Cumulative
Gain (NDCG) and Kendal’s similarity score. These are discussed
below.

1. MAP
MAP measures the precision of the ranking results. If there are
two classes, the positive and negative class, precision measures the
accuracy of the top n results to a query. It is given by:
no. of positive instances within top n
P (n) = .
n
Average precision of a query is

N
P ∗ pos(n)
Pav = ,
no. of positive instances
n=1

where N is the maximum number of instances, pos(n) is a binary

function which indicates whether the instance at position i is pos-
itive. MAP is Pav averaged over all queries considered.
2. NDCG
NDCG measures ranking accuracy. Given a query,


n
2R(i) − 1
NDCG(n) = Yn ,
log(1 + j)
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 98

98 Introduction to Pattern Recognition and Machine Learning

where n denotes position, R(i) denotes the score for rank j, Yn

is a normalization factor which ensures that a perfect ranking’s
NDCG at position n is 1. Finally, NDCG is averaged over all
queries.
3. Similarity between Features
Similarity between two features is measured on the basis of their
ranking result. The similarity between two features is repre-
sented by the similarity between the ranking results they produce.
Kendall’s similarity is used. The similarity between two features
fi and fj for a query q is given by
|{(xa , xb )εD|xa ≺fi xb and xa ≺fj xb }|
Kq (fi , fj ) = ,
|{(xa , xb )εD}|
where D denotes all the pairs of instances (xa , xb ) with respect
to the query q, xa ≺fi xb denotes that instance xa is ranked
ahead of instance xb by feature fi . The Kendall’s score is aver-
ages over all queries to get K(fi , fj ). It is to be noted that
K(fi , fj ) = K(fj , fi ).

The optimization formulation can be represented as:


max i wi yi
 
min i j
=i s(fi, fj )yi yj
such that yi ε{0, 1}, i = 1, . . . , m
Σi yi = t,

where t is the total number of features, yi = 1 indicates that the

feature fi is selected and yi = 0 indicates that the feature fi is not
selected, wi is the importance score of feature fi and s(fi, fj ) is the
similarity between features fi and fj . To calculate wi , MAP or NDCG
is used and for similarity, s(fi , fj ) = K(fi , fj ).
In the above formulation, there are two objectives: To maximize
the sum of the importance scores of each feature and to minimize
the sum of the similarity scores between pairs of features.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 99

Feature Extraction and Feature Selection 99

Converting this multi-objective programming problem to a

single-objective programming problem is done as follows:
  
max i wi yi − K1 i j
=i s(fi, fj )yi yj
such that yi ε{0, 1}, i = 1, . . . , m

i yi = t.

K1 decides the relative importance given to the first term and

the second term of the objective function.
The optimization is a typical 0–1 integer programming prob-
lem. The solution using exhaustive search has high time complexity.
A greedy search algorithm can be used as shown below.
The algorithm is as follows:

1. Construct an undirected graph G0 in which each node represents

a feature, the weight of node fi is wi and the weight of the edge
between fi and fj is s(fi , fj ).
2. Let the set F = Φ.
3. For i = 1 . . . t
(a) Select node with largest weight. Let the node be fki .
(b) A punishment term is subtracted from the weight of the other
nodes which depends on their dissimilarity with fki as follows:
wj = wj − s(fki , j) ∗ 2K1 , j = ki.
(c) Add fki to F and remove it from the graph G with all the

edges connected to it. In other words, Fi+1 = Fi {fki } and
Gi+1 = Gi \{fki }.
4. Fi gives the set of features.

12.2. Feature ranking using F-score

A simple and effective way to carry out feature selection is to use
the F-score. This score is based on the statistics of the patterns and
does not depend on the class labels. The F-score measures the dis-
crimination of two sets of real numbers. The F-score of every feature
is computed using the information in the training data. The F-score
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 100

100 Introduction to Pattern Recognition and Machine Learning

of the jth feature F (j) is

 2  2
(+) (−)
xj − xj + xj − xj
F (j) =
1
n+  (+) 
(+) 2 1
n−  (−)  ,
(−) 2
n+ −1 i=1 xi,j − xj + n− −1 i=1 x i,j − (x)j

(11)

where
n+ = number of positive instances,
n− = number of negative instances,
xj = average of jth feature in the whole dataset,
(+)
xj = average of jth feature in the positive labeled instances,
(−)
xj = average of jth feature in the negative labeled instances,
(+)
xi,j = jth feature of the ith positive instance,
(−)
xi,j = jth feature of the ith negative instance.

The numerator of Eq. (11) indicates the discrimination between

the positive and the negative labeled instances, and the denomi-
nator indicates the discrimination within the positive labeled set
and the negative labeled set. A larger F-score is more discrimi-
native. If Fi > Fj , it means that the feature i is more discrimina-
tive than feature j. The F-score considers each feature separately
and therefore does not take into account the interaction between
features.

12.3. Feature ranking using linear support vector

machine (SVM) weight vector
This method does feature ranking using weights from linear SVM
models. The SVM finds a separating hyperplane between two classes.
Given a training dataset, SVM solves the following unconstrained
optimization problem:

1
l
minw,b wt w + C E(w, b; Xi , yi ), (12)
2
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 101

Feature Extraction and Feature Selection 101

where E(w, b; Xi , yi ) is a loss function, C is a penalty factor and the

loss function is

max(1 − yi (wt φ(Xi ) + b), 0) (13)

or

max(1 − yi (wt φ(Xi ) + b), 0)2 , (14)

where φ is a function used when the SVM is nonlinear. If Eq. (13) is

used as the loss function it is called the L1-loss SVM. If Eq. (14) is
used as the loss function it is called the L2-loss SVM.
A linear SVM has φ(X) = X. For any test pattern x, the decision
function is

f (X) = sign(wt φ(X) + b). (15)

A kernel function K(Xi , Xj ) = φ(Xi )t φ(Xj ) can be used to train

the SVM. In the case of linear SVM, the kernel function will be
K(Xi , Xj ) = Xit Xj . Another kernel is the radial basis function (RBF)
which can be written as

K(Xi , Xj ) = exp(−αXi − Xj 2 ), where α > 0.

Using the linear SVM, w ∈ Rn can be used to decide the importance

of each feature. If |wi | > |wj | then ith feature is more important than
the jth feature. The features can be ranked according to the values
|wj |. Further, training a linear SVM is simpler.
The linear SVM algorithm can be written as:

1. Use grid search to find the parameter C for Eq. (12).

2. Train a L2-loss SVM using the best value of C.
3. Sort the features according to the absolute value of the weights.

12.4. Ensemble feature ranking

A number of different approaches can be used for feature ranking. To
get a better ranking of the features, the ranking obtained by different
methods can be combined.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 102

102 Introduction to Pattern Recognition and Machine Learning

12.4.1. Using threshold-based feature selection techniques

Techniques based on Information Gain, Chi-square statistic,
F-measure etc. can be used to find the feature ranking. Then these
rankings are combined. If we have rankings F1 , . . . , FT where T is
the number of ranking methods used, to find the overall ranking of
a class, the arithmetic mean is used.

12.4.2. Evolutionary algorithm

One technique for ensemble feature ranking extracts feature ranking
using independent runs of an evolutionary learning algorithm called
ROGER (ROC-based Genetic Learner). It is called ROC-based as
it optimizes the Area Under the Curve (AUC) based on Receiver
Operator Characteristics (ROC). An earlier technique uses GAs to
find good feature subsets where the fitness function is the accuracy
obtained on a validation set using kNN classifier and only the selected
features. The feature subsets are then combined.
The ROGER algorithm finds a hypothesis h which mea-
sures the weighted L1 distance to a point p in Rd . Each string
of the GA has d weights and d coordinates of the point p i.e.
the string is (w1 , w2 , . . . , wd , p1 , . . . , pd ). This is associated with the
hypothesis


d
h(X) = wi ∗ |xi − pi|.
i=1

The fitness of the string is computed using Wilcoxon statistic

giving the fitness of the string, F = p(h(Xi ) > h(Xj ) | yi > yj )
i.e. it is the probability that the result of the hypothesis of Xi is
greater than that of Xj when the class label of Xi i.e. yi is greater
than the given class label of Xj i.e. yj . This is equivalent to the AUC
criterion.
ROGER is used a number of times to get the feature rankings
F1 , . . . , FT where T is the number of times the independent runs of
the GA are carried out. Then the ensemble feature ranking for the
ith feature is the number of features j which are ranked before i by
more than half the rankings.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 103

Feature Extraction and Feature Selection 103

12.5. Feature ranking using number of label changes

This method is a simple and fast method for ranking the features.
The data is projected on to the axis being considered. It is then
arranged in increasing order. Now while scanning this data, the class
label of every point is noted. While scanning, the number of label
changes is noted. If the number of label changes is less, the feature
yields more information. The points with the same value are arranged
so that all those of a single class label occur together. If the number
of label changes is found for every feature, the feature which has the
least number of label changes is the best attribute and so on. This
method can be used to rank the features.

13. Feature Selection for Time Series Data

A good method of indexing time series is by using Spatial Access
Methods (SAMs) such as R trees. Since most SAMs degrade rapidly
at dimensionalities greater than 8–12, it is meaningful to carry out
dimensionality reduction on the time series data. In time series data,
since we consider a series of values in a sequence, the correlation
between the features are very strong. While carrying out feature
selection, it is necessary to maintain the existing correlation between
the features while removing features.

13.1. Piecewise aggregate approximation

A very simple method of dimensionality reduction is to use piecewise
aggregate approximation. If the time series has a dimensionality of
d and it is required to reduce the dimensionality to k, then the data
is divided into k equi-sized frames. For each of the k frames, the
mean value of the data falling in the frame is taken as the data
in the reduced representation. If the original time series data is
X = (x1 , . . . , xd ) and the time series with the dimensionality reduced
be X̄ = (x̄1 , . . . , x̄k ). Then the ith element in the k dimensional time
series will be
d
i
k

k

x̄i = xj .
d
j= kd (i−1)+1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 104

104 Introduction to Pattern Recognition and Machine Learning

13.2. Spectral decomposition

In spectral decomposition, the time series is represented by a super-
position of a finite number of sine/cosine waves, where each wave is
represented by a complex number, the Fourier coefficient. The time
series is in the frequency domain. When the time series is in the
frequency domain, it can be decomposed into d sine/cosine waves,
i.e. d Fourier coefficients. Many of these Fourier coefficients will be
of very low amplitude and can therefore be discarded without loss of
much information. To carry out dimensionality reduction of a time
series X of length d, the Discrete Fourier Transform (DFT) of X
is calculated. The transformed vector of coefficients is truncated at
d
2 as each coefficient is a complex number with real and imaginary
parts to the coefficient.

13.3. Wavelet decomposition

The wavelet transform is used to transform the data to yield wavelet
coefficients. The first few coefficients contain an overall, coarse
approximation of the data and the later coefficients refer to specific
areas of the data in high detail. To carry out dimensionality reduction
of a time series X of length d, the Discrete Haar Wavelet Transform
(DWT) is calculated and the first k values are chosen and the other
coefficients are discarded.

13.4. Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD) transforms the time series data
so that the data has maximum possible variance with respect to
the first axis, the second axis has the maximum possible variance
orthogonal to the first, the third axis has maximum possible variance
orthogonal to the first two axes and so on. To carry out dimension-
ality reduction of a time series X of length d, the SVD is found and
the first k coefficients are chosen to represent the time series.

13.5. Common principal component loading based

variable subset selection (CLeVer)
This method is based on common principal component analysis
(CPCA) and retains the correlation information among the features.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 105

Feature Extraction and Feature Selection 105

The algorithm first entails finding the principal components (PCs)

and the descriptive common principal components (DCPCs). Finding
the PCs involves finding k out of d PCs where k < d. A threshold
δ is used, and k is the minimum value such that the ratio of the
variances determined by its first k PCs to the total variance exceeds
the threshold δ. The final value of k is the maximum of these values.
After this, all the time series have to be described by their first k
principal components.

Research Ideas
1. It is possible to view feature selection as a specialization of either linear or
nonlinear feature extraction. Under what conditions can feature extraction be
preferred over feature selection? The improved performance could be in terms
of space, time and/or accuracy.

Relevant References
(a) V. S. Devi and M. N. Murty, Pattern Recognition: An Introduction.
Hyderabad, India: Universities Press, 2012.
(b) M. N. Murty and V. S. Devi, Pattern Recognition: An Algorithmic
Approach. New York: Springer, 2012.
(c) Jens Kresten, Simultaneous feature selection and Gaussian mixture model
estimation for supervised classification problems. Pattern Recognition,
47:2582–2595, 2014.
(d) D. Zhang, J. He, Y. Zhao, Z. Luo and M. Du, Global plus local: A complete
framework for feature extraction and recognition. Pattern Recognition,
47:1433–1442, 2014.
(e) G. Wang, Q. Song, H. Sun, X. Zhang, B. Xu and Y. Zhou, A feature subset
selection algorithm automatic recommendation method. JAIR, 47:1–34,
2013.
2. MI has been popularly exploited in feature selection. How can we reduce the
number of features selected by such a method to get better accuracy? Will it
help in improving the scalability of the feature selection scheme?

Relevant References
(a) G. Herman, B. Zhang, Y. Wang, G. Ye and F. Chen, Mutual information-
based method for selecting informative feature sets. Pattern Recognition,
46(12):3315–3327, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 106

106 Introduction to Pattern Recognition and Machine Learning

(b) H. Liu, J. Sun, L. Liu and H. Zhang, Feature selection with dynamic mutual
information. Pattern Recognition, 42:1330–1339, 2009.
(c) P. M. Chinta and M. N. Murty, Discriminative feature analysis and selection
for document classification. Proceedings of ICONIP, 2012.
(d) J. Dai and Q. Xu, Attribute selection based on information gain ratio in
fuzzy rough set theory with application to tumor classification. Applied
Soft Computing, 13:211–221, 2013.
3. Feature selection based on MI and Chi-Square test perform reasonably well
on large datasets. How do you compare them?

Relevant References
(a) C. D. Manning, P. Raghavan and H. Schutze, Introduction to Information
Retrieval. Cambridge: Cambridge University Press, 2008.
(b) P. M. Chinta and M. N. Murty, Discriminative feature analysis and selection
for document classification. Proceedings of ICONIP, 2012.
(c) S. R. Singh, H. A. Murthy and T. A. Gonsalves, Feature selection for text
classification based on gini coefficient of inequality. JMLR Workshop and
Conference Proceedings, 10:76–85, 2010.
4. Principal components are the eigenvectors of the covariance matrix of the data.
The first principal coefficient is in the maximum variance direction; second
component is orthogonal to the first and corresponds to the next variance
direction and so on. Show using a simple two-dimensional example that the
second principal component is better than the first principal component for
discrimination. However, there are popular schemes like the latent semantic
indexing which use the principal components of the high-dimensional data
successfully. What could be the reason behind this?

Relevant References
(a) M. N. Murty and V. S. Devi, Pattern recognition, Web course, NPTEL,
2012, http://nptel.iitm.ac.in/courses.php [accessed on 2 November 2014].
(b) S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer and
R. Harshman, Indexing by latent semantic analysis. Journal of the Amer-
ican Society for Information Science, 41:391–407, 1990.
(c) M. Prakash and M. N. Murty, A genetic approach for selection of (near-)
optimal subsets of principal components for discrimination. PR Letters,
16:781–787, 1995.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 107

Feature Extraction and Feature Selection 107

(d) S. Karamizadeh, S. M. Abdullah, A. A. Manaf, M. Zamani and A. Hooman,

An overview of principal component analysis. Journal of Signal and Infor-
mation Processing, 4:173–175, 2013.
5. In the case of NMF, we factorize a data matrix X into B and H . The associated
optimization problem is not simple (not convex). If we know in addition to
X either B or H , then we can have a simple optimization problem. Consider
situations where B is known or H known and examine the role of NMF.

Relevant References
(a) D. D. Lee and H. S. Seung, Learning the parts of objects by non-negative
matrix factorization. Nature, 401:788–791, 1999.
(b) D. D. Lee and H. S. Seung, Algorithms for non-negative matrix factoriza-
tion. Advances in Neural Information Processing Systems, 13:556–562,
2001.
(c) C. C. Aggarwal and C. K. Reddy, Data Clustering: Algorithms and Appli-
cations. New York: CRC Press, 2014.
(d) C. Thurau, K. Kersting, M. Wahabzada and C. Bauckhage, Convex non-
negative matrix factorization for massive datasets. Knowledge and Infor-
mation Systems, 29:457–478, 2011.
6. One of the issues with NMF is that the resulting factorization may lead to a
local minimum of the optimization function considered. What are the different
ways of improving upon this?

Relevant References
(a) A. Korattikara, L. Boyles, M. Welling, J. Kim and H. Park, Statistical
optimization of non-negative matrix factorization. Proceedings of The
Fourteenth International Conference on Artificial Intelligence and Statis-
tics, JMLR: W&CP 15, 2011.
(b) F. Pompili, N. Gillis, P.-A. Absil and F. Glineur, Two algorithms for ortho-
gonal nonnegative matrix factorization with application to clustering.
CoRR abs/1201.0901, 2014.
(c) V. Bittorf, B. Recht, C. Re and J. A. Tropp, Factoring nonnegative matrices
with linear programs. CORR abs/1206.1270, 2013.
7. It is possible to project d-dimensional patterns to k -dimensional patterns using
random projections where the random entries come from a Gaussian with zero
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 108

108 Introduction to Pattern Recognition and Machine Learning

mean and unit variance. Then if Xi and Xj are a pair of patterns in the d space
and the corresponding patterns after projection into the k space are Xi and
Xj , then it is possible to show that with probability greater than or equal to
1 − n−β
(1 −
)Xi − Xj 2 ≤ Xi − Xj 2 ≤ (1 +
)Xi − Xj 2
4+2β
given positive
and β and k is any number greater than kmin = 2 3 log n.
2
− 3
How do we appreciate the role of various quantities like β,
, n, and k ? What
can happen to the bounds when these parameters are varied within their legal
ranges?

Relevant References
(a) A. K. Menon, Random projections and applications to dimensionality
reduction. BS (advanced) thesis, School of Info. Tech., University of Syd-
ney, 2007.
(b) P. Li, T. J. Hastie and K. W. Church, Very sparse random projections.
Proceedings of KDD, 2006.
(c) R. J. Durrant and A. Kaban, Sharp generalization error bounds for
randomly-projected classifiers. Proceedings of ICML, 2013.
8. Even though GAs have been used in feature selection and extraction, algo-
rithms based on GAs cannot scale up well, specifically steady-state GA may
be very slow in converging. How to make them scale-up well?

Relevant References
(a) I. Rejer and K. Lorenz, Genetic algorithm and forward method for feature
selection in EEG feature space. JTACS, 7:72–82, 2013.
(b) A. Ekbal and S. Saha, Stacked ensemble coupled with feature selection
for biomedical entity extraction. Knowledge-Based Systems, 46:22–32,
2013.
(c) D. Dohare and V. S. Devi, Combination of similarity measures for time
series classification using genetic algorithms. IEEE Congress on Evolu-
tionary Computation, 2011.
(d) D. Anand, Article: Improved collaborative filtering using evolutionary
algorithm based feature extraction. International Journal of Computer
Applications, 64:20–26, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 109

Feature Extraction and Feature Selection 109

(e) I. Guyon, S. Gunn, M. Nikravesh and L. A. Zadeh, Feature Extraction:

Foundations and Applications. New York: Springer, 2006.
9. It is often claimed that noisy and boundary patterns qualify as support vectors.
Further the weight vector W obtained by an SVM is


W = αi yiXi ,
Xi ∈ S

where S is the set of support vectors, yi is the class label of Xi which is either
+1 or −1, and αi is the Lagrange variable associated with Xi . So, how can
such a weight vector W be useful in ranking features?

Relevant References
(a) Y.-W. Chang and C.-J. Lin, Feature ranking using linear SVM. JMLR
Workshop and Conference Proceedings, pp. 53–64, 2008.
(b) Y.-W. Chen and C.-J. Lin, Combining SVMs with various feature selec-
tion strategies. In Feature Extraction, Foundations and Applications,
I. Guyon, S. Gunn, M. Nikravesh and L. Zadeh (eds.). New York: Springer,
2006.
(c) J. Wang, S. Zhou, Y. Yi and J. Kong, An improved feature selection based
on effective range for classification. The Scientific World Journal, 1–8,
2014.
(d) H. Li, C.-J. Li, X.-J. Wu and J. Sun, Statistics-based wrapper for fea-
ture selection: An implementation on financial distress identification with
support vector machine. Applied Soft Computing, 19:57–67, 2014.
10. Feature selection based on F -score has been effectively used in several prac-
tical applications. What is the reason for its success.

Relevant References
(a) H.-Y. Lo et al., An ensemble of three classifiers for KDD cup 2009:
Expanded linear model, heterogeneous boosting, and selective naive bayes.
JMLR: Workshop and Conference Proceedings, 7:57–64, 2009.
(b) Y.-W. Chen and C.-J. Lin, Combining SVMs with Various Feature selection
strategies. In Feature Extraction, Foundations and Applications. Berlin:
Springer, 2006, pp. 315–324.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch03 page 110

110 Introduction to Pattern Recognition and Machine Learning

(c) J. Xie, J. Lei, W. Xie, Y. Shi and X. Liu, Two-stage hybrid feature selection
algorithms for diagnosing erythemato-squamous diseases. Health Infor-
mation Science and Systems, 1:10, 2013.
10. Time series data can be large in several applications. How to extract features
for meaningful classification?

Relevant References
(a) P. K. Vemulapalli, V. Monga and S. N. Brennan, Robust extrema features
for time–series data analysis. IEEE Transactions on PAMI, 35:1464–1479,
2013.
(b) M. G. Baydogan, G. Runger and E. Tuv, A bag-of-features framework to
classify time series. IEEE Transactions on PAMI, 35:2796–2802, 2013.
(c) Q. Wang, X. Li and Q. Qin, Feature selection for time series model-
ing. Journal of Intelligent Learning Systems and Applications, 5:152–164,
2013.
(d) B. D. Fulcher and N. S. Jones, Highly comparative, feature-based time-
series classification, CoRR abs/1401.3531, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 111

Chapter 4

Bayesian Learning

Bayesian approach is well established and well studied in literature.

Even though it is classical, it has gained a lot of prominence more
recently. An important reason for this is that domain knowledge can
be suitably used to help in learning; using such prior knowledge will
lead to meaningful estimation of parameters using even small size
datasets. Also, when a small quantity of data is available, Bayesian
approaches work better than their counterparts which are based on
the data alone. Bayesian learning is used in both supervised and
unsupervised settings. We consider the supervised case in this chap-
ter and consider the unsupervised case in a later chapter.

1. Document Classification
In the Bayesian approach, typically we exploit the Bayes rule to
convert the prior probabilities to posterior probabilities based on the
data under consideration. For example, let us consider a collection
of documents
D = {(d1 , C), (d2 , C), . . . , (dn1 , C), (dn1 +1 , C), . . . , (dn , C)},
where we have n1 documents from class C and n − n1 documents
from class C. Now a new document d can be classified using the

111
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 112

112 Introduction to Pattern Recognition and Machine Learning

Bayes rule as follows:

1. Obtain the posterior probabilities P (C|d) and P (C|d) using the
Bayes rule
P (d|C)P (C)
P (C|d) = ,
P (d|C)P (C) + P (d|C)P (C)
P (d|C)P (C)
P (C|d) = .
P (d|C)P (C) + P (d|C)P (C)
2. Assign d to class that has larger posterior probability; that is
assign d to class

C if P (C|d) > P (C|d),

C if P (C|d) > P (C|d).

Some of the important features of the classifier are:

1. It is possible to show that the Bayes classifier is optimal; it mini-
mizes the average probability of error or error-rate.
2. Bayes classifier is basically probabilistic and theoretical; it can be
made practical and useful based on statistics. In the absence of
the availability of values of various probabilities required to obtain
the posterior probabilities, it is not possible to use it in practice.
3. Estimation of the probabilities is achieved using the dataset D.

A major difficulty is in using it when the data is small in size. Let

us consider a simple example.

Example 1. Consider the collection of six documents shown in

Table 4.1. There are six example documents shown here. There are
three documents from class sports and two from politics. The sixth
document d6 is a test document; we need to find the class label to be
associated with it. In order to use the Bayes classifier, here we need to
know P (Sports|d6 ) and P (P olitics|d6 ). Let us consider P (Sports|d6 )
given by
P (d6 |Sports )P (Sports )
P (Sports|d6 ) = .
P (d6 |Sports )P (Sports ) + P (d6 |Politics)P (Politics)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 113

Bayesian Learning 113

Table 4.1. A dataset of six documents.

Document Description Class

d1 Cricket Mumbai Sports

d2 Dravid Ball Cricket Sports
d3 Ball Tendulkar Pawar Sports
d4 Pawar Tendulkar Patil Mumbai Politics
d5 Sonia Singh Pawar Delhi Politics
d6 Tendulkar Patil Pawar Delhi ?

Noting that out of the five labeled patterns, three are from sports,
an estimate for P (Sports) = 35 . Similarly, P (politics) = 25 . However,
frequency based estimate for P (d6 |Sports) is zero because none of the
training documents in the class sports matches d6 ; similarly estimate
of P (d6 |P olitics) becomes zero. So, it becomes difficult to obtain a
meaningful estimate of either P (d6 |Sports) or P (d6 |P olitics).

2. Naive Bayes Classifier

One may argue that these probability estimates are not meaningful
because the training dataset size is small. However, even when the
training dataset is large in size, it is possible that a lengthy (a large
number of distinct words) document can have the probability esti-
mate to be zero. This has led to the development of the Naive Bayes
Classifier where the probability estimation is simplified by assuming
class-conditional independence among the words in a document. The
resulting simplification may be specified as


P (d|C) = P (wi |C).
wi ∈d

Here, the independence of various words appearing in the document

d is conditional; it depends on class C.
Finally we need to compute P (Sports|d6 ) and P (P olitics|d6 )
which require
• Prior Probabilities: P (Sports) and P (P olitics); these quantities
could be estimated as explained earlier to be P (Sports) = 35 and
P (politics) = 25 .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 114

114 Introduction to Pattern Recognition and Machine Learning

• P (d6 |Sports) and P (d6 |P olitics): In the current example these

values are estimated by using class-conditional independence as
follows:

P (d6 |C) = P (T endulkar|C) × P (P atil|C) × P (P awar|C)

× P (Delhi|C).

In the current example, one may view C to be Sports and C to

be P olitics. So, the Maximum Likelihood (ML) estimates based
on the frequency of occurrence are
— P (T endulkar|Sports) = 16 . This is because out of the three doc-
uments corresponding to Sports there are six words totally and
T endulkar appeared once.
— P (P atil|Sports) = 06 = 0. Such an estimate will mean that
P (Sports|d6 ) takes a value zero(0).
In order to modify such estimates which can lead to zero values
for posterior probabilities a smoothing component is introduced
to ensure that the probability estimates neither become zero nor
one. This is achieved by using the following smoothed version to
estimate.
Number of occurrences of w in C + 1
P (w|C) = ,
Total number of words in C + |V |

where |V | is the size of the vocabulary or the number of distinct

words in the collection. In the given example the vocabulary set
V is

V = {Cricket, M umbai, Dravid, Ball, T endulkar, P awar, P atil,

Sonia, Singh, Delhi}.

So, |V | = 10 and the estimates of conditional probabilities are

— P (T endulkar|Sports) = P (P awar|Sports) = 18 ,
1
— P (P atil|Sports) = P (Delhi|Sports) = 16 ,
2
— P (T endulkar|P olitics) = P (P atil|P olitics) = 17 ,
2 3
— P (Delhi|P olitics) = 17 ; P (P awar|P olitics) = 17 .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 115

Bayesian Learning 115

Now we can use these estimates and the prior probabilities given
by P (Sports) = 35 and P (P olitics) = 25 to obtain the posterior pro-
babilities using the following

P (Sports|d6 )
P (d6 |Sports)P (Sports)
=
P (d6 |Sports)P (Sports) + P (d6 |P olitics)P (P olitics)
= 0.24,
P (P olitics|d6 )
P (d6 |P olitics)P (P olitics)
=
P (d6 |Sports)P (Sports) + P (d6 |P olitics)P (P olitics)
= 0.76.

• So we assign to d6 class label P olitics as the corresponding poste-

rior probability is larger; it is 0.76.

3. Frequency-Based Estimation of Probabilities

A simple and popularly used estimate for probability of an event is
based on the frequency of occurrence of the event. Let us consider a
simple example of tossing a coin.

Example 2. Let a coin be tossed n times out of which let the num-
ber of times head shows up be nh , then probability of the coin show-
ing head, P (h), is estimated using
nh
P (h) = .
n
Specifically let the coin be tossed six times out of which head shows
up four times, so, P (h) = 46 = 23 . However, in another coin tossing
experiment, if there are 0 (zero) heads out of five tosses of the coin,
then the probability P (h) = 05 = 0. This is the problem with the fre-
quency based estimation scheme; the estimate may not be accurate
when the experiment is conducted a smaller number of times or
equivalently when the dataset size is small.
One way to improve the quality of the estimate is to integrate
any prior knowledge we have in the process of estimation. A simple
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 116

116 Introduction to Pattern Recognition and Machine Learning

scheme is to use some kind of uniform prior. For example, in the

case of coin tossing it is safe to assume that the probability of a
head or a tail is equally likely to be 12 . In such a case the estimate
will be
nh + 1
P (h) = .
n+2
Such an estimate will take values in the open interval (0, 1). For
example, if there are 0 heads out of five tosses of the coin then the
estimate for P (h) is 0+1
5+2 which is not zero even though it could
be small. In a generic setting the probability of an event e, P (e) is
given by
ne + 1
P (e) = ,
n+t
where ne is the number of trails favoring event e out of a total of n
trails and t is the total number of outcomes. Here, we have a uniform
prior with each of the t outcomes having equal probability of 1t . In
most practical applications the value of t is much smaller compared
to the value of n. Also, from an asymptotic view point t is fixed, but
n can keep growing. So, as n → ∞, P (e) tends to nne which is the ML
estimate.
Based on the above discussion, we have the following observa-
tions:

• In frequency based estimation of probabilities, one may encounter

zero probabilities. One can avoid such zero or non-discriminative
estimates by softening the estimates using some prior knowledge.
• One of the simplest schemes is based on employing uniform prior;
in such a case when the number of trails n is large then the result-
ing estimate tends to the simple frequency based estimate.
• Bayesian estimation schemes could be viewed as generalizations
where the decisions are based on posterior probabilities obtained
by combining knowledge in the form of priors.

We next consider the use of posterior probabilities in simple

classification.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 117

Bayesian Learning 117

4. Posterior Probability
The most important feature of Bayesian learning is to exploit Bayes
rule to convert the prior probability into the posterior probability.
Specifically let C be the class label and d be the observed document.
Then
• Prior probability: P (C)
• Posterior probability: P (C|d); it is the probability of the class after
observing d.
• Using Bayes rule, we have
P (d|C) × P (C)
P (C|d) = .
P (d|C) × P (C) + P (d|C) × P (C)
Once we have the posterior probabilities, we can assign d to class C
if P (C|d) > P (C|d); else assign d to C. Equivalently, we assign d
to class C if PP (C|d)
(C|d)
> 1. We can simplify the expressions if P (C|d)
and P (C|d) are exponential functions by assigning d to class C if
logP (C|d) > logP (C|d). We consider an example involving univari-
ate normal densities next; note that univariate normal is a member
of the exponential family of distributions.

Example 3. Let us consider that the densities of classes C and C

be univariate normal. Specifically let

P (d|C) ∼ N (µ, 1) and

P (d|C) ∼ N (µ, 1).
Here, we consider that µ and µ are the means of the densities of the
classes C and C respectively and the variances are equal to 1. Let us
assume further that the prior probabilities P (C) and P (C) be equal
to 12 . Now P (C|d) is given by
P (d|C)
P (C|d) =
P (d|C) + P (d|C)
 
exp − 12 (x − µ)2
=    .
exp − 12 (x − µ)2 + exp − 12 (x − µ)2
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 118

118 Introduction to Pattern Recognition and Machine Learning

Note that P (C|d) has the same denominator as P (C|d); they differ
only in their numerator values. So,
 
P (C|d) exp − 12 (x − µ)2
=  .
P (C|d) exp − 12 (x − µ)2
In this ratio we have exponentials in both the numerator and denom-
inator; so, it is good to consider the comparison of log(P (C|d)) and
log(P (C|d)). We assign d to class C if

log(P (C|d)) > log(P (C|d)).

Equivalently assign d to class C if

(x − µ)2 < (x − µ)2

or

(µ + µ) < 2x assuming µ > µ.

So, assign d to class C if

(µ + µ)
x> .
2
Similarly assign d to C if
(µ + µ)
x< .
2
Decide arbitrarily if x = µ+µ2 . It appears as though such a simple
decision-based classifier is realized because the data distributions are
univariate. However, similar decisions can be arrived at even when
the distributions are multivariate. Specifically, when classes C and
C are multivariate normal with means µ and µ respectively and the
covariance matrices are equal and are equal to aI where a is a non-
negative real number and I is the identity matrix.
The resulting classifier is called the Minimum Distance Classifier
(MDC). If the classes have µ and µ as the means of the two classes
C and C respectively then a pattern X is assigned to class C if

(X − µ)t Σ−1 (X − µ) < (X − µ)t Σ−1 (X − µ).

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 119

Bayesian Learning 119

If we assume in addition that Σ−1 = σ12 I, then the above inequality

becomes
(X − µ)t (X − µ) (X − µ)t (X − µ)
< .
σ2 σ2
This means assign X to C if the Euclidean distance between X
and µ is smaller than that between X and µ; otherwise assign X to C.
In addition to using the Bayes rule in classification it is possible to
use it in density estimation, which we consider next.

5. Density Estimation
We have noted earlier that in order to use the Bayes classifier, it is
important to have the prior probabilities and the probability density
function of each class. One of the simplest ways is to assume that the
form of the density function is known and the parameter underlying
the density function is unknown. Estimation of the density function
under these conditions is called parametric estimation. In the fre-
quentist test approach, the parameters are assumed to be unknown
but deterministic. On the contrary, in the Bayesian approach the
parameters are assumed to be random variables. We examine para-
metric estimation using these schemes in this section.

• Bernoulli Random Variable: Let us consider a binary random

variable that can take a value of either 1 or 0. Let us assume
that the probability that the random variable assumes value 1
be p1 . Let the probability of the random variable assuming value 0
be p0 . It is easy to see that p0 = 1 − p1 . So, one of either p1 or
p0 is adequate to capture the behavior. Now the probability mass
function is given by
1−X
Bernoulli(X|p1 ) = pX
1 p0 so,
Bernoulli(X|p1 ) = p1 if X = 1 and
Bernoulli(X|p1 ) = p0 = (1 − p1 ) if X = 0.
It is possible to show that the expected value of X is p1 as
E[X] = 1 × p1 + 0 × p0 = p1 .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 120

120 Introduction to Pattern Recognition and Machine Learning

Similarly, it is possible to show that the variance of X is p1 p0 as

follows:

Variance[X] = E[X 2 ] − (E[X])2 = (12 × p1 + 02 × p0 ) − p21

= p1 − p21 = p1 (1 − p1 ) = p1 p0 .

Next we consider estimation of the parameter p1 from the data.

We consider the ML scheme for estimating the parameter p1 .
— Maximum-likelihood estimation: Let there be n patterns drawn
independently from the unknown Bernoulli density. Let the
collection of these patterns be

D = {X1 , X2 , . . . , Xn }.

The functional form of the density is known here; once we know

the value of the parameter p1 , we know everything about the
class density. Because of the independence, we have

n
1−Xi
P (D|p1 ) = pX
1 (1 − p1 )
i
.
i=1

Typically we choose the value of p1 corresponding to the max-

imum value of the likelihood function P (D|p1 ). Equivalently
one can get the optimal value of p1 by maximizing the loga-
rithm of the likelihood function. Note that the logarithm of the
likelihood function is

n
logP (D|p1 ) = [Xi log p1 + (1 − Xi )log(1 − p1 )].
i=1

By differentiating it with respect to p1 and equating the deriva-

tive to zero we get


n 
n
Xi n− Xi
i=1 i=1
− = 0.
p1 1 − p1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 121

Bayesian Learning 121

By simplifying the above expression we get the estimate of p1

to be

n
Xi
i=1
p1 = , (1)
n

which is an intuitively acceptable estimate; it is the propor-

tion of times the random variable assumes a value 1. Next we
consider the Bayesian estimation scheme.
— Bayesian estimation: In the case of ML estimation we assume
that the parameter is unknown but deterministic; once we know
the parameter we have the knowledge of the entire density
function. In the case of the Bayesian estimation we treat the
parameter as a random variable and we assume that the prior
density of the random variable is known. This is the place to use
the domain knowledge; it is tacitly assumed that the domain
knowledge is integrated into the model using a suitable form
for the prior density. We use the Bayes rule to compute the
posterior density using prior density and the likelihood func-
tion. The posterior density of the parameter p1 , after observing
data D is given by

P (D|p1 )P (p1 )
P (p1 |D) = 
P (D|p1 )P (p1 )dp1
P
n
P
n
Xi n− Xi
p1i=1
× (1 − p1 ) i=1
= P
n
P
n
 Xi n− Xi
i=1
p1 × (1 − p1 ) i=1 dp1
P
n
P
n
Xi n− Xi
i=1
p1 × (1 − p1 ) i=1 × (n + 1)!
× n    .
 n
Xi ! × n − Xi !
i=1 i=1

It is assumed that the prior density of p1 is uniform in the

range [0, 1]; so, P (p1 ) = 1. In simplifying the integral in the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 122

122 Introduction to Pattern Recognition and Machine Learning

denominator, the following identity is used:

p! × q!
xp × (1 − x)q dx = .
(p + q + 1)!
Note that posterior density of p1 depends both on the data D
and also on the prior density P (p1 ). This is the fundamental
difference between the ML estimate and the Bayesian estimate;
ML estimate depends only on the data as given by (1) whereas
the Bayesian estimate exploits the knowledge in the form of the
prior density. One can use the posterior density in estimating
P (X|D) as shown below:

P (X|D) = P (X, p1 |D)dp1

= P (X|p1 , D)P (p1 |D)dp1

= P (X|p1 )P (p1 |D)dp1 .

The last equation emerges because once p1 is given, everything

is known about the density of X; nothing else is required. In
the current example of Bernoulli distribution we know the pos-
terior density and P (X|p1 ) is Bernoulli in terms of the parame-
ter p1 . Using these details and the above equality we can obtain
P (X|D) as follows:
(n + 1)!
P (X|D) =  n   
 n
Xi ! n − Xi !
i=1 i=1
„n « „ «
P P
n
Xi n− Xi
(1−X)
× 1 (1 − p1 )
pX p1 i=1
(1 − p1 ) i=1 dp1

(n + 1)!
=   

n 
n
Xi ! n − Xi !
i=1 i=1
P „ «
X+
n
Xi n+1− X+
P
n
Xi
× p1 i=1
(1 − p1 ) i=1 dp1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 123

Bayesian Learning 123

(n + 1)!
=   

n 
n
Xi ! n − Xi !
i=1 i=1
 
n 
n
(X + i=1 Xi )! n + 1 − (X + Xi ) !
i=1
× .
(n + 2)!
By simplifying the above expression we get
    
n  n
X+ Xi ! n+1− X + Xi !
P (X|D) =  n  ×  
i=1 i=1
.
 n
Xi ! n− Xi ! (n + 2)
i=1 i=1

Note that X is a Bernoulli random variable; it can take a value

of either 1 or 0. When X = 1, we can simplify the above expres-
sion, by substituting 1 for X, to
 

n
1+ Xi
i=1
P (X = 1|D) = . (2)
n+2
Similarly the probability for X taking 0 is
 
n n
n+1− Xi 1+ Xi
i=1 i=1
P (X = 0|D) = =1− . (3)
n+2 n+2
Note the similarity between the ML estimation of p1 as shown
in (1) and the Bayesian estimate given in (2). When n is large

Xi can be large and so the numerator in (2) may be approx-

imated by Xi and n + 2 in the denominator tends to n.
Such an approximation of (2) leads to the estimate in (1). So,
when the dataset size (n) is large, there is no difference between
the ML estimate and the Bayesian estimate; this is the typical
behavior. So, in big data applications where the dataset size is
large, ML estimate is good enough; Bayesian estimate may not
reveal any additional characteristics in big data analytics.
In the above example we have assumed that the prior dis-
tribution of p1 is uniform in the range [0, 1]. One can use
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 124

124 Introduction to Pattern Recognition and Machine Learning

other distributions for the prior distribution of the random

variable p1 . Consider again the estimate shown in (2) which is
 

n
1+ Xi
i=1
P (X = 1|D) = .
n+2

n
This estimate can still make sense when Xi = 0; this can
i=1
happen, for example, when n = 0. In such a case, the prob-
ability P (X = 1|D) is 12 ; it means that a priori X taking a
value 1 or 0 are equally likely. This is the typical behavior of
the Bayes estimate; it ensures that the probabilities are in the
open interval (0, 1).
— Choice of the prior density: In the above example of Bernoulli
random variable we have used uniform density for the prior. It
simplified the computation of
1. Posterior density: In this case the posterior has a simple
form and depends on the likelihood function.
2. Estimation of P (X|D): It helped in obtaining a closed form
expression for P (X|D).
Even though uniform prior leads to a simple form for the result-
ing densities, it may not be able to capture the domain know-
ledge properly. For example, when we consider the frequency
of each term occurring in a large collection of documents, the
corresponding distribution is not uniform. It is a power law
degree distribution which is characterized by Zipf’s law. It was
observed by Zipf that the frequency of the ith term in the
collection, fi , is given by

C
fi = , (4)
ri
where ri is the rank of the ith term; most frequent term has
the smallest rank of 1 and hence its frequency is the maximum
given by C and the least frequent term has its rank value to
be C and hence its frequency is 1 C C .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 125

Bayesian Learning 125

Similarly in scale free networks the degree distribution is

observed to be satisfying power law distribution, not uniform.
Specifically it is

M
nk = ,
kα
where nk is the number of nodes of degree k and M and α are
some constants. So, using an appropriate prior density that
may not be uniform may make sense based on the context.
In such a case it may be difficult to get simple closed form
expressions for the densities in general. However, it is theo-
retically possible to simplify the analysis by assuming suitable
complementary or conjugate forms for the prior density. Even
when closed form expression is not obtained for the density
P (X|D), it is possible to consider some simplifying scenarios
where mode, mean or maximum values of the posterior densi-
ties are used. In order to explain the notion of conjugate prior
we consider another popular distribution.
• Binomial distribution: It is one of the popularly encountered distri-
butions. For example, consider tossing a coin n times out of which
we get nh heads and nt tails; let the probability of head in a single
toss be ph . Then the probability of this event is given by

n
Bin(nh heads out of n tosses) = nh pnhh (1 − ph )n−nh .

More generally it is viewed as the number of successes in n trials

with the probability of success being ps and the probability of
failure being pf = (1 − ps ). So, probability of k successes in n trials
is given by

n
Bin(n, k) = k pks (1 − ps )(n−k) .

It can be shown that

— Mean, E[k] = nps and
— Variance, E[(k − E[k])2 ] = nps pf .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 126

126 Introduction to Pattern Recognition and Machine Learning

Note that both Bernoulli and binomial distributions P have

n
similar
i=1 Xi
functional
Pn form in terms of the parameter given by p1 (1 −
p1 )n− i=1 Xi . This form helps in choosing an appropriate prior den-
sity; a popular notion in this context is conjugate prior. We have
observed based on Bayes rule that the posterior density depends on
the prior density and the likelihood function. For a given data distri-
bution and correspondingly the likelihood function, a prior density
is conjugate if resulting posterior density has the same form of the
distribution as the prior.

6. Conjugate Priors
Note that both the Bernoulli and Binomial distributions have like-
lihood functions, based on parameter q ∈ [0, 1], proportional to
q a (1 − q)b , where a and b are constants. This suggests that we choose
a prior density that is also proportional to q a (1 − q)b so as to ensure
that the posterior has a similar form; Beta density has such a form;
the probability density is given by

Γ(a + b) a−1
Beta(q|a, b) = q (1 − q)b−1
Γ(a)Γ(b)
∞ ∞
where Γ(a) = v a−1 e−v dv = (a − 1) v a−2 e−v dv
0 0
= (a − 1)Γ(a − 1).

Note that any probability density function p(x) satisfies the following
properties:

1. p(x)
 ≥ 0 for all x,
2. p(x) dx = 1.

So, for the Beta distribution we have

1
Γ(a + b) a−1
q (1 − q)b−1 dq = 1
0 Γ(a)Γ(b)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 127

Bayesian Learning 127

or equivalently
1
Γ(a + b)
q a−1 (1 − q)b−1 dq = 1,
Γ(a)Γ(b) 0

which means that

1
Γ(a)Γ(b)
q a−1 (1 − q)b−1 dq = . (5)
0 Γ(a + b)

It is possible to use this equality and the property of the Gamma

distribution to show that the mean of the Beta distributed random
a
variable is a+b . It may be shown as follows:

1
Γ(a + b) a−1
E[q] = q q (1 − q)b−1 dq
0 Γ(a)Γ(b)

Γ(a + b) 1 a
= q (1 − q)b−1 dq
Γ(a)Γ(b) 0
Γ(a + b) Γ(a + 1)Γ(b)
= ×
Γ(a)Γ(b) Γ(a + b + 1)
Γ(a + b) a × Γ(a)Γ(b)
= × .
Γ(a)Γ(b) (a + b)Γ(a + b)

By canceling terms that are present in both the numerator and the
denominator, we get

a
E[q] = .
a+b

Similarly it is possible to show that the variance is (a+b)2ab

(a+b+1)
.
Here, a and b may be viewed as hyper-parameters that characterize
the distribution of q. We know that the posterior is given by

P (D|q)P (q)
P (q|D) =  1 .
0 P (D|q)P (q)dq
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 128

128 Introduction to Pattern Recognition and Machine Learning

Noting that the denominator is independent of q it can be treated as

a constant C1 . So,

P (q|D) = C1 P (D|q)P (q).

Noting that P (q) is Beta and the likelihood is based on Bernoulli,

we can get the posterior to be

P
n P
n
Xi n− Xi
P (q|D) = C1 q i=1 (1 − q) i=1 C2 q a−1 (1 − q)b−1 ,

Γ(a+b)
where C2 is Γ(a)Γ(b)
; by using C for C1 × C2 , we get

P
n P
n
a+ Xi −1 n+b− Xi −1
P (q|D) = Cq i=1 (1 − q) i=1 .

So, the posterior is again a Beta density and the hyper-parameters

n 
n
are a + Xi and b + n − Xi . The notion of conjugate prior is
i=1 i=1
useful because it makes the posterior have a closed form expression
and further it ensures that both the prior and the posterior have the
same functional form for the density functions.
In the case of document collections, a frequently used assumption
is that the terms occur in a document independent of each other; we
are interested in the number of occurrences of the term in a document
but not in the order in which terms occur in the document. Bag of
words model is the name of such a model; it is the most popular
model in information retrieval. In such a case if there are k distinct
terms t1 , t2 , . . . , tk in the document collection and there are m terms
in a document such that term ti occurs mi times in the document
for i = 1, 2, . . . , k, then the distribution of the terms is explained by
the multinomial density.
By assuming that pi is the probability of term ti occurring in
the document, Multinomial may be viewed as a generalization of
the Binomial distribution where k = 2 and the mass function of the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 129

Bayesian Learning 129

multinomial random variable is given by

m


k
M ult(m1 , m2 , . . . , mk |p, m) = m1 m2 . . . mk pm i
i ,
i=1

m

where m1 m2 . . . mk = m1 !mm!
2 !···mk !
and p = (p1 , p2 , . . . , pk )t .
The expected value of mi is given by

m k
m!
E[mi ] = mi pi i dmi
m1 !m2 ! · · · mk !
i=1
 


k
m − 1!  pi j dmj 
m
= mpi
m1 !m2 ! · · · (mi − 1)!mk !
j=i,j=1

×pimi −1 dmi
= mpi × 1 = mpi .

Observe that this result is similar to the estimate of mean of the bino-
mial; similarly it is possible to show that variance of mi is mpi (1−pi ).
We discuss next how the Dirichlet prior is the conjugate to multi-
nomial; this result has been significantly exploited in the machine
learning literature during the past decade in the form of soft cluster-
ing based on latent Dirichlet allocation and its variants. It is possible
to show that the likelihood function corresponding to multinomial is
given by


k
M
P (D|p) ∝ pj j ,
j=1

where Mj is the number of times tj occurred in the collection, D, of

documents. So, we require the prior to have a similar form which is
satisfied by the Dirichlet distribution. So the prior density is given by

Γ(a1 + a2 +, . . . , +ak )
aj −1
k
P (p) = pj .
Γ(a1 ), . . . , Γ(ak )
j=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 130

130 Introduction to Pattern Recognition and Machine Learning

Now the posterior P (p|D) is given by

P (p|D) ∝ P (D|p)P (p)


k
Mj +aj −1
∝ pj .
j=1

Thus the posterior is also Dirichlet with parameters Mj + aj . We will

examine how this is exploited in the so-called Topic Model based on
latent Dirichlet allocation in the chapter on Soft Clustering.
It is possible to summarize some of the important conjugate pri-
ors as shown in Table 4.2.
It is very important to note that the notion of conjugate prior
helps in simplifying the process of calculating the posterior; it helps
in getting a closed form expression for the posterior. It does not
guarantee that the resulting expressions are practically beneficial or
semantically correct. For example, it may be appropriate to use power
law prior while dealing with the frequencies of terms in a collection
of documents, not the Dirichlet prior.
Another issue associated with the Bayesian estimation is that
there may not be any difference between the parameter estimated
using the Bayesian scheme and the ML scheme when the size of the

Table 4.2. A collection of conjugate priors.

Likelihood Prior Posterior

Bernoulli, Binomial Beta Beta

∝ pk (1 − p)n−k ∝ pa−1 (1 − p)b−1 ∝ pk+a−1 (1 − p)n+a−k−1

Multinomial Dirichlet Dirichlet

Qk Qk Q
k
∝ pm
i
i
∝ pai i −1 ∝ pmi
i +ai −1

i=1 i=1 i=1

Exponential, Poisson Gamma Gamma !

n
P n
P
−λ Xi −λ b+ Xi
∝ λn e i=1 ∝ λa−1 e−bλ λn+a−1 e i=1

Normal Normal Normal

(µ−µ0 )2 (µ−µn )2
Qn
−
(Xi −µ)2 −
2σ2
− 2
∝ e 2σ2 ∝e 0 ∝e 2σn

i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 131

Bayesian Learning 131

dataset is large. We have seen this behavior in a variety of cases when

conjugate priors are used. So, in big data analysis it may make sense
to use the simple scheme of ML based on frequencies first.
Another issue is that of realization. Bayesian scheme is inherently
probabilistic; however, in order to obtain results in a practical setting,
it requires some empirical methods.

Research Ideas
1. In Section 2, we have discussed the Naive Bayes classifier (NBC) which assumes
that the terms are independent of each other given the class. It is not difficult
to realize that this could be a gross simplification. Then why should NBC
work well?

Relevant References
(a) I. Rish, An empirical study of the Naive Bayes classifier. International Joint
Conferences on Artificial Intelligence workshop on empirical methods in
artificial intelligence, 2001.
(b) L. Jiang, D. Wang and Z. Cai, Discriminatively weighted Naive Bayes and
its application in text classification. International Journal of Artificial Intel-
ligence Tools, 21(1), 2012.
(c) C. D. Manning, P. Raghavan and H. Schutze, Introduction to Information
Retrieval. Cambridge: Cambridge University Press, 2008.
2. It is possible to use classifiers in feature selection. How does one use the NBC
in feature selection?

Relevant References
(a) C.-H. Lee, F. Gutierrez and D. Dou, Calculating feature weights in Naive
Bayes with Kullback–Leibler Measure. IEEE International Conference on
Data Mining, 2011.
(b) J. Chen, H. Huang, S. Tian and Y. Qu, Feature selection for text classification
with Naive Bayes. Expert Systems with Applications, 36(3):5432–5435,
2009.
(c) Z. Zeng, H. Zhang, R. Zhang and Y. Zhang, A hybrid feature selection
method based on rough conditional mutual information and Naive Bayesian
classifier. ISRN Applied Mathematics, 2014:1–11, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 132

132 Introduction to Pattern Recognition and Machine Learning

(d) Microsoft Naive Bayes Algorithm Technical Reference, http:// www.msdn.

microsoft.com/en-us/library/cc645902.ASPX.
3. It may be meaningful to assume class-conditional independence of phrases
instead of terms. This is because phrases carry more information than terms for
classification. Can we use frequent itemsets or other abstractions to characterize
phrases?

Relevant References
(a) S. Dey and M. N. Murty, Using discriminative phrases for text catego-
rization. 20th International Conference on Neural Information Processing,
2013.
(b) M. Yuan, Y. X. Ouyang and Z. Xiong, A text categorization method using
extended vector space model by frequent term sets. Journal of Information
Science and Engineering, 29:99–114, 2013.
(c) D. Gujraniya and M. N. Murty, Efficient classification using phrases gener-
ated by topic models. In Proceedings of International Conference on Pattern
Recognition, 2012.
4. It is possible to extend the MDC discussed in Section 4 to deal with more
than two classes. Specifically, if there are n classes corresponding to the n
training patterns, then each class may be viewed as drawn from a normal density
with mean at the point and the covariance matrix is of the form 0I. In such a
case the MDC converges to Nearest Neighbor Classifier. Is this interpretation
meaningful?

Relevant References
(a) R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification, Second
Edition. New York: Wiley Interscience, 2000.
(b) J. Ye, Multiple Closed-Form local metric learning for K-nearest neighbor
classifier. CoRR abs/1311.3157, 2013.
(c) J. Liu, X. Pan, X. Zhu and W. Zhu, Using phenological metrics and the
multiple classifier fusion method to map land cover types. Journal of Applied
Remote Sensing, 8: 2014.
5. We have discussed conjugate priors in Section 6. There could be other kinds
of priors that may help in getting closed form expressions. How can they be
used?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 133

Bayesian Learning 133

Relevant References
(a) M. I. Jordan, Jeffrey’s Priors and Reference Priors, Lecture 7, 2010.
(www.cs.berkeley.edu/jordan/courses/260-spring10/.../lecture7.pdf)
(b) R. Yang and J. O. Berger, Estimation of a covariance matrix using the
reference prior. The Annals of Statistics, 22(3):1195–1211, 1994.
(c) M. D. Branco, M. G. Genton and B. Liseo, Objective Bayesian analysis of
skew-t distributions. Scandinavian Journal of Statistics Theory and Appli-
cations, 40(1):63–85, 2013.
(d) C. Hu, E. Ryu, D. Carlson, Y. Wang and L. Carin, Latent Gaussian models
for topic modeling. JMLR Workshop and Conference Proceedings, 2014.
6. It is analytically convenient to assume that the prior is Dirichlet and the like-
lihood to be multinomial in finding clusters of documents. However, because
we know that frequency distribution of terms satisfies power law as specified
by Zipf, does it make sense to consider other forms of prior densities?

Relevant References
(a) D. M. Blei, Probabilistic topic models. Communications of the ACM,
55(4):77–84, 2012.
(b) C. Wang and D. M. Blei, Variational inference in non-conjugate models.
Journal of Machine Learning Research, 14(1):1005–1031, 2013.
(c) D. Newman, E. V. Bonilla and W. Buntine, Improving topic coherence with
regularized topic models. Proceedings of Neural Information Processing
Systems, 2011.
7. Most of the times priors are selected based on analytical tractability rather than
the semantic requirement. For example, Dirichlet is convenient mathematically
to deal with the frequency distribution of terms in a document collection where
the likelihood is characterized by multi-nomial. However, Zipf’s curve based
on empirical studies gives a better prior in this case. Similarly Wikipedia offers
a rich semantic input to fix the prior in case of clustering and classification of
documents. How does such empirical data help in arriving at more appropriate
Bayesian schemes?

Relevant References
(a) C. M. Bishop, Pattern Recognition and Machine Learning. Singapore:
Springer, 2008.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch04 page 134

134 Introduction to Pattern Recognition and Machine Learning

(b) X. Hu, X. Zhang, C. Lu, E. K. Park and X. Zhou, Exploiting Wikipedia as

external knowledge for document clustering. Proceedings of the 15th ACM
SIG KDD, 389–396, 2009.
(c) J. A. Hansen, E. K. Ringger and K. D. Seppi, Probabilistic explicit topic
modeling using Wikipedia. Lecture Notes in Computer Science, 8105:69–
82, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 135

Chapter 5

Classification

Classification is the task of assigning a class label to an input pattern.

The class label indicates one of a given set of classes. The classifica-
tion is carried out with the help of a model obtained using a learning
procedure. According to the type of learning used, there are two
categories of classification, one using supervised learning and the
other using unsupervised learning. Supervised learning makes
use of a set of examples which already have the class labels assigned
to them. Unsupervised learning attempts to find inherent struc-
tures in the data. Semi-supervised learning makes use of a small
number of labeled data and a large number of unlabeled data to learn
the classifier.

1. Classification Without Learning

A set of training patterns which are labeled is available but there
is no learning procedure to generate a model of the training data.
This type of classification is a non-parametric form of classification
where the inbuilt distribution of the data is not explicitly used.
A popular classification algorithm which carries out classification
without learning is the Nearest Neighbor (NN) algorithm which
is generally called the 1NN classifier. The pattern to be classified
is compared to all the patterns in the training data and the dis-
tance between the test pattern and the other patterns is determined.

135
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 136

136 Introduction to Pattern Recognition and Machine Learning

The test pattern is given a class label which is the class label of the
pattern closest to it in the training data. If there are n training pat-
terns (X1 , w1 ), (X2 , w2 ), . . . , (Xn , wn ), we need to classify a pattern
P , and if DP i is the distance between P and pattern Xi , then if
DP k = min DP i where i = 1, 2, . . . , n, then P is assigned the class of
pattern Xk which will be wk .
Other algorithms based on the NN rule are the k-nearest
neighbor (kNN) algorithm, the modified k-nearest neighbor
(mkNN) algorithm, and the r-nearest neighbor (rNN) algo-
rithm. All these algorithms do not need to develop any model for
classification using the training data. Hence, no learning takes place
except for fixing the parameter k. The value of k is crucial to the
performance of the classifier. It can therefore be seen that these
algorithms do not need any time for learning the classification model.
Classification algorithms which carry out classification without going
through the learning phase have no design time (or training time).
These algorithms are robust. 1NN classifier has an error rate less than
twice the bayes error rate which is the optimal error rate asymptot-
ically. Similarly, the kNN classifier gives the optimal (bayes) error
rate asymptotically.
Consider the set of points in Figure 5.1. It is a two-dimensional
dataset with two features f1 and f2 . The nearest neighbor or the 1NN
classifier assigns the label of the closest neighbor to a test point.
Test data P will be classified as belonging to class ‘square’ as its
closest neighbor belongs to that class. Q will be classified as belonging
to the class ‘circle’ as it is closest to point 7. R will be classified as
belonging to class ‘circle’ as it is closest to point 7. In the case of
point P , there is no ambiguity and the 1NN classifier works well. In
the case of point Q, even though it is closest to class ‘circle’, since it
is on the boundary of the class ‘cross’ and class ‘circle’, there is some
ambiguity. If kNN is used with k = 5, the points Q and R are labeled
as belonging to class ‘cross’ and not class ‘circle’. In the case of
mkNN, the distances of the test pattern from the k neighbors is also
taken into account. Out of the k neighbors, if dmin is the distance of
the closest neighbor and dmax is the distance of the furthest neighbor
out of the k neighbors, then the weight given to the class of neighbor
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 137

Classification 137

4
f2
17
18
3
14 15 16
P 13
2
3 x 6 x 9 o
o 12
8 o o
1 1x 5 x Q 11
2 x o
4 x R 7 o
10

1 2 3 4 5
f1

Figure 5.1. Example dataset.

i is
(dmax − di )
wi = .
(dmax − dmin )
The NN weight w1 is set to 1. The score of every class is initialized
to 0, i.e. scorei = 0, i = 1, . . . , c.
For every neighbor i out of the k neighbors, if the point belongs
to class j, scorej is incremented by wi . After doing this for the k
neighbors, the test pattern belongs to the class having the largest
score. In the case of pattern R, using kNN it is given the class label
‘cross’. Using mkNN, since 7 is the closest neighbor, class ‘circle’
is given a weightage of 1. The other points 4, 2, and 5 belong to
class ‘cross’. The 5th neighbor is quite far away from R. If the score
aggregated to class ‘cross’ is more than the score for class ‘circle’, R
will be assigned to class ‘cross’. It can be seen that often, using kNN
may classify the test pattern differently as compared to mkNN.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 138

138 Introduction to Pattern Recognition and Machine Learning

These classifiers require time linear in the sample size for classifi-
cation. This goes up as the training data size goes up. In this context,
if the training dataset size can be reduced, the time required for clas-
sification can be reduced. This reduction can be accomplished either
by reducing the number of training patterns, reducing the number of
features, or both. Reducing the number of training patterns can be
done by carrying out prototype selection which includes condensation
algorithms and editing algorithms. There are a number of algorithms
including the popular Condensed Nearest Neighbor (CNN) algorithm
and the Modified Condensed Nearest Neighbor (MCNN) algorithm.
The CNN is an order-dependent algorithm and different orderings of
the input data give different condensed sets. As a result, you cannot
be guaranteed to get the optimal condensed set. MCNN mitigates
this problem by suitably modifying the algorithm to make it order
independent.
The CNN starts with a set of all patterns Data and a condensed
set Condensed which is empty. The first pattern in Data is put into
Condensed. After this, the following set of statements are repeated
till there is no change in Condensed in an iteration.

1. For every pattern x in Data, find its NN in Condensed.

2. If closest neighbor does not have the same class label as x, add x
to Condensed.

It can be seen that the first pattern presented to the algo-

rithm will be put in the condensed set. For example, in Figure 5.1,
if the patterns are presented to the algorithm in the order
x1 , x2 , x3 , x4 , x5 , x6 , x7 , . . . , pattern x1 belonging to class ‘cross’ will
be first put into Condensed. x2 , x3 , x4 , x5 , and x6 will be left out and
x7 which belongs to class ‘circle’ will be put into Condensed. So in the
first iteration, the first pattern presented to the algorithm from each
class will be included in Condensed, along with other patterns. Even
in the subsequent iteration, the patterns included in Condensed will
depend on the patterns already included. Hence, it is evident that
the patterns put into Condensed depend on the order in which the
patterns are presented to the algorithm. It is therefore an order-
dependent algorithm. Obviously, when different permutations of
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 139

Classification 139

the data give different condensed sets, the CNN algorithm does not
give an optimal condensed set.
The MCNN algorithm is a modification of the CNN algorithm
making it an order-independent algorithm. In this algorithm,
in each iteration, one pattern which is a typical pattern of a class
is added to Condensed. So in each iteration, c patterns are added
to Condensed, one from each class. The condensed set is used to
classify the training set. The misclassified patterns of each class are
used to find the next typical pattern for each class which is added to
Condensed. This is continued till there are no misclassified patterns.
It is to be noted that in a particular iteration, a class which has no
misclassified patterns will not have any pattern added to Condensed.
It can be seen that in the above algorithm, all the patterns are con-
sidered. Finding the typical pattern of a class and classification of
the training set using the condensed set do not depend on the order
in which the patterns are presented to the algorithm. MCNN is an
order-independent algorithm which gives better results than CNN.
However, the MCNN has a higher time complexity. Since it needs to
be run only once for a dataset to get the condensed set which can
be used for 1NN classification, the time taken should not matter if
it gives a better condensed set. It is to be noted that both CNN
and MCNN work by classifying the training dataset by using the
condensed dataset. Using the final Condensed set obtained, both
the algorithms result in 100% classification accuracy on the training
dataset.

2. Classification in High-Dimensional Spaces

Classification in high-dimensional spaces suffers from the problems of
large data. Since the number of features is large, the space and time
complexity goes up. Besides, the distance metrics generally used like
the Lk metric do not work well in high-dimensional space.
One obvious solution to high-dimensional data is to reduce it
to low-dimensional data by carrying out feature selection. It then
becomes a problem of classification in low-dimensional spaces. It is
also possible to use techniques like Fisher’s linear discriminant, which
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 140

140 Introduction to Pattern Recognition and Machine Learning

reduces the problem to a linear problem with the discriminant which

can be used for classification. This also helps to mitigate the curse
of dimensionality problem.
Classification algorithms which use the distance metric to com-
pare patterns have to use distance metrics which work well in
high-dimensional space. The usual Minkowski’s metric does not do
well. Using the Lk metric, the distance to the nearest data point
approaches the distance to the farthest data point. This leads to
difficulty in using NN classifier since the concept of distances to dif-
ferent patterns does not work well. This means that the contrast in
distances to different points from a particular test pattern P becomes
non-existent. The difference in distance between the NN and any
other points in the dataset could become very small. This means that
when we consider a small value
, if Dnn is the distance from P to
the NN then most of the points in the dataset fall within a distance
(1 +
)Dnn from P . This is called the unstable phenomenon. This
is shown in Figure 5.2. Since Dmax is the distance to the furthest
point, all the points in the dataset fall between the inner and the
outer hyper-sphere.
Due to the unstable phenomenon which occurs in high-
dimensional spaces, it is necessary to find meaningful distance

Dnn

Dmax

Dmax <= (1 + e) Dnn

Figure 5.2. Example of an unstable phenomenon.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 141

Classification 141

metrics in high-dimensional spaces. The following sections discuss

some distance metrics which can be used for high-dimensional space.

2.1. Fractional distance metrics

It is possible to show the following result:



Xd
k k
Dmax − Dnn
k
if limd→∞ var = 0, then d d
→k 0, (1)
E[
Xd
k ] Dnn k
d

where E[X] is the expected value of a random variable X, var[X] is

k
the variance of X, Dmax d
is the furthest distance of the points in the
k
dataset from a point P when using Lk norm, Dnn d
is the distance of
the nearest point from the dataset when using Lk norm.
This basically says that for X
Rd when using the Lk norm, as
the dimensionality d increases if variance of a random variable X
divided by its expectation tends to zero, then the difference between
the maximum and minimum distance tends to zero. This shows that
the difference between the maximum and minumum distances to a
given test pattern does not increase as fast as the nearest distance
to any point in high-dimensional space. Since the distance metric
does not show enough variation in the distance to NN and furthest
neighbor, it becomes meaningless to use such a distance metric as d
k
Dmax k
−Dnn
d d
increases. The ratio k
Dnn
is called the relative contrast.
d
Further, it can be proved that for a uniform distribution of n
points
  
  
C 1 k
Dmax − Dnn
k
1 ≤ limd→∞ E d
1 1
d

(k + 1) k 2k + 1 dk−2
  

C(n − 1) 1
≤ 1 , (2)
(k + 1) k 2k + 1

where C is some constant.

For an arbitrary distribution of n points, it can be shown that
 
k
Dmax − D k
nnd
Ck ≤ limd→∞ E d
1 ≤ (n − 1) · Ck . (3)
− 21
d k
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 142

142 Introduction to Pattern Recognition and Machine Learning

The Lk norm can be written as

1
Lk (X, Y ) = [Σdi=1 (
xi − y i
k )] k . (4)
In high-dimensional spaces smaller values of k do better than
larger values of k. According to Eqs. (2) and (3), if we consider the
difference between the maximum distance and minimum distance
1 1
from a test pattern P , |Dmax −Dnn |, it grows as d k − 2 with increasing
dimensionality d. This means that for the L1 norm, the above expres-
sion diverges to ∞. For the L2 norm or euclidean distance (ED), the
expression is bounded by constants and for higher values of k, it con-
verges to zero. As a result of this, only the L1 metric among the Lk
family is found to have increasing difference between the nearest and
farthest neighbors of a query point as the dimensionality increases.
For the L2 metric, |Dmax − Dnn | converges to a constant and for any
distance metric Li where i ≥ 3, |Dmax − Dnn | converges to zero as
dimensionality d increases. Thus, for high-dimensional data, the L1
metric is preferable to L2 metric. Since lower values of k do better,
fractional distance metrics in which k is a fraction smaller than 1
can be considered. So the fractional distance function distfd can be
considered where
1
distfd = (Σdi=1 (xi − y i )f ) f , (5)
where f
(0, 1).
It is found that fractional distance metrics show better relative
contrast than integral distance metrics. Experimentation has been
done with a range of values from L0.1 to L10 and also for L∞ and
it is found that the classification accuracy decreases with increasing
values of k in the Lk norm.
In the case of fractional metrics, for the case of uniform distribu-
tion and if f = 1l , where l is an integer, we get
  
  
f f
C 1 Dmax d − Dnnd
1 ≤ limd→∞ E 1
2·f +1 −1
(f + 1) f df 2
  

C · (n − 1) 1
≤ 1 . (6)
(f + 1) f 2·f +1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 143

Classification 143

This shows that the absolute difference between the maximum

distance and minimum distance using fractional distance metric
1
−1
increases at the rate of d f 2 . It can be seen that the smaller the
fraction, the greater is the rate of divergence between the maximum
and the minimum distance.
To examine the relative contrast, we use the following equation:
  
f f
1 Dmax − Dnn
C· ≤ limd→∞ E d d

2·f +1 f
Dnn d

1
≤ C · (n − 1) · . (7)
2·f +1

From Eqs. (1)–(3), (6), and (7), it can be seen that fractional
distance metrics provide better contract than the integer valued dis-
tance metrics.

2.2. Shrinkage–divergence proximity (SDP)

The proximity between two points is computed by finding the prox-
imity on each attribute and adding them up. SDP magnifies the
variation in the distance for every attribute to avoid the unsta-
ble phenomenon. If we consider two data points P and Q whose
proximity is to be found, for each attribute, the attribute’s prox-
imity is almost zero if the projected attribute values of the two
data points are close to each other. To distinguish between the two
points on this attribute, the projected attributes of the two data
points are made more far apart to make it easy to distinguish them.
In SDP, the exponential function is used to spread out the points
and increase discrimination. To take care of this, SDP uses weighted
attributes.
Consider a function f defined as


 0, if 0 ≤ x < p,

fp,q (x) = x, if p ≤ x < q, (8)


ex , otherwise.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 144

144 Introduction to Pattern Recognition and Machine Learning

If we have two d-dimensional points P = (p1 , p2 , . . . , pd ) and Q =

(q1 , q2 , . . . , qd ), then the general form of the SDP function between
P and Q is defined as:

SDP G (P, Q) = Σdi=1 wi fsi1 ,si2 (Dd (pi , qi )), (9)

where wi is the weightage given to the ith attribute and depends

on the importance given to the attribute. One way of finding the
values of wi would be to give it a value of σ1i where σi is the standard
deviation of attribute i. Any feature weighting algorithm can be used
to find the values of wi .
The parameter si1 is called the shrinkage threshold for attribute
i and si2 is called the divergence threshold for attribute i. The values
of these two parameters are fixed empirically so that more stable per-
formance is obtained. It is found that si1 can be fixed to a very small
value such as 0.005σi and si2 is taken from the interval [25σi , 60σi ].
If the parameters wi are taken to be 1, then the SDP G will
degenerate to

SDPs1 ,s2 (p, q) = Σdi=1 fs1 ,s2 (Dd (pi , qi )). (10)

It is to be noted that if Dd is the Lk metric, then the SDP has

the following properties:

1. SDPs1 ,s2 is equivalent to L1 , as s1 → 0 and s2 → ∞.

2. SDPs1 ,s2 (p, q) = 0 if and only if 0 ≤ Dd (pi , qi ) < s1 ∀i.
3. SDPs1 ,s2 (p, q) ≥ des2 if Dd (pi , qi ) ≥ s2 ∀i.

The first property shows that SDP is a general form of the L1

metric. The third property shows that the magnifying effect of SDP
is very sharp. The SDP function is not a metric as SDPs1 ,s2 (p, q) = 0
does not mean that p = q for s1 > 0 and the triangular inequality
does not hold in general.

3. Random Forests
Random forests or decision tree forests consist of a number of deci-
sion trees. It is an ensemble classifier. The prediction of the decision
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 145

Classification 145

trees is combined to get the overall prediction of the decision forest.

Each tree is built by considering only a random subset of the fea-
tures for splitting at the nodes. Because only a subset of features
is used, the decision tree forest can handle a larger number of
features.
If there are N training patterns, a random sample of N is selected
with replacement. Generally 23 rd of N is chosen at random. A decision
tree is built using the random sample. This decision tree is built by
considering only a subset of the features for splitting at the nodes. If
d is the dimensionality of the data, d1 features are chosen at random
where d1 < d.
By using the above procedure, a number of decision trees are
built leading to a decision tree forest. A new pattern is assigned a
class label by using all the decision trees in the forest, and combined
to give the final class label.
The selection of a random subset of features for each decision
tree makes this a random subspace method. This prevents overfitting
leading to stochastic discrimination.
One advantage of decision tree forests is that the 13 rd patterns
which are left out for building a decision tree (which are called the
“out of bag” (OOB) samples) can be used as a test sample for that
tree. By using the OOB samples, the error rate of prediction for each
tree is computed. The average error rate for all the trees gives the
generalization error of the decision tree forest.
Some of the choices to be made for building a random forest are
as follows:

1. The forest size or the number of decision trees F .

2. Type of decision to be made at each node: Using a single feature
at each node leads to hyper-rectangular regions. It is also possi-
ble to use oblique split (linear combinations of some features) or
multivariate split (nonlinear combination of some features).
3. Parameter to be used to choose the best feature to split at a node.
Generally, information gain or gini index is used. The information
gain at a node N is calculated as shown further. At a node N , if
we have Ni as the number of instances of class i, then the impurity
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 146

146 Introduction to Pattern Recognition and Machine Learning

i(N ) is:

c
i(N ) = − fi log fi ,
i=1

where
Ni
fi = c .
j=1 Nj

In the case of two classes, we get

i(N ) = −fp log fp − fn log fn ,

N
where fp is the fraction of positive examples given by Np and fn
is the fraction of negative examples given by NNn . N is the total
number of instances, Np is the number of instances of positive
class, and Nn is the number of instances of negative class.
If there are s splits pertaining to a decision on an attribute a,
then the impurity for split k is

c
I(ak) = − faki log faki ,
i=1

where faki gives the fraction of instances of attribute a in split k

of class i and is given by faki = NNaki
ak
.
Nak is the total number of instances along split k and is given

by Nak = cj=1 Nakj where Naki gives the instances of class i with
attribute a in split k.
The information gain for node N for split on attribute a is
given by

s
Nak
IG(N a) = I(N ) − I(ak).
N
k=1

Using variance impurity, the impurity at the node N is given by

i(N ) = f i ∗ fj ,
i,j,i=j

i = 1 . . . c, j = 1 . . . c.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 147

Classification 147

This can also be written as

 
1
c
i(N ) = 1− fj2 .
2
j=1

This is called the Gini impurity. In the case of the two-class prob-
lem, we get

i(N ) = fp ∗ fn .

4. Each leaf node is associated with a predictor model. The predictor

model to be used has to be chosen. It could be the conditional
probability p(c | x) where c refers to the class.
5. The method for injecting randomness in each tree. This is done by
choosing a random subset of the dataset to be considered to build
the tree and choosing only a subset of the features at random to
be considered for the splitting. If there are N patterns and N1
patterns are considered for building a tree, then the amount of
randomness is controlled by the ratio NN1 . The parameter ρ gives
the number of patterns in the subset of patterns chosen. If ρ = N ,
then there is no randomness in the system. Also ρ = 1 gives the
maximum randomness.

In view of the above, some of the key parameters of random forests

are:

1. Forest size F .
2. Number of patterns considered for each tree N 1 which gives the
randomness parameter ρ.
3. Number of random features (d1 ) considered for each tree.

If each leaf i gives a class conditional posterior probability pi (c|x),

the combined output will be
1 F
pi (c | x) = Σ pi (c | x). (11)
F i=1
The random forest can be used for regression when the predictor
takes on numerical values instead of class labels at the leaf nodes.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 148

148 Introduction to Pattern Recognition and Machine Learning

This depends on the predictor model which can be constant, polyno-

mial and linear, and probabilistic linear. For example, a polynomial
model would correspond to

y(x) = ΣN 1 i
i=1 ai x .

It is probabilistic linear when the output at a leaf i is of the form

pi (y | x).
Some of the strengths of random forests are as follows:

1. It is found to give better results than decision trees.

2. It is robust to outliers and noise.
3. A probabilistic output is provided.
4. Works well even when more than two classes are involved.
5. The overall performance of the random forest can be guaged by
finding the generalization error. It is found that random forests
generalize well to unseen data.
6. The algorithm can be parallelized to give improved speed.

Random forests can be used not only for supervised classification

but also for clustering and semi-supervised classification.

3.1. Fuzzy random forests

In a fuzzy random forest, there is a forest of fuzzy decision trees
generated randomly. To construct a fuzzy random forest, 23 rd of the
training examples are selected randomly. The other 13 rd of the data
are called OOB data. Using the selected training examples, a fuzzy
tree is constructed.
Every example has a weight of 1 at the root node. At the root
node, there will be a number of examples of each class. These exam-
ples are distributed to the branches. Each example is along each
branch by an amount obtained as the product of its weight and
the membership degree to the node. The information gain for each
attribute at that node is calculated. The attribute with the highest
information gain is chosen as the attribute to be split at the node.
The split is made at the node. The above procedure is used at the
new nodes created by the split.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 149

Classification 149

To classify a new pattern, it is checked with every fuzzy tree in

the forest, and the decision of each tree is taken and combined to get
the final classification. Combine the information from the leaf nodes
reached in each tree to get another decision forest from where the
decision is taken.
For a new pattern p, each tree i gives a weight to each class c for
the leaf node j reached which is W (i, c). Here, the value
Ec
W [i, c] = C ,
m=1 Em

where Ej is the number of examples of class j.

The combination of these decisions can be made in a number of
ways.
1. Simple majority vote: The class label which has maximum weight
is assigned to the test pattern in each tree. These are combined
to find the class label with the majority vote. For every tree t
and for a class c, Utc is 1 if weight Wtc is maximum in the tree
otherwise Utc is zero. Then we get

i=T
Vj = Uij , j = 1, . . . , c, (12)
i=1

where T is the number of trees and c is the number of classes. The

new pattern is assigned to the class
c = argmaxj Vj .
2. Weighted vote: For every class, the weights obtained from each
tree are combined. The class having the maximum weight is
assigned to the test pattern. We get

i=T
Zj = Wij , j = 1, . . . , C, (13)
i=1

where T is the number of trees and C is the number of classes.

The new pattern is assigned to the class
c = argmaxj Zj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 150

150 Introduction to Pattern Recognition and Machine Learning

4. Linear Support Vector Machine (SVM)

Linear SVM is an extremely fast machine learning algorithm for very
large datasets. A cutting plane algorithm is used to get an SVM which
has a linear complexity. This means that it is a scalable algorithm as
the model scales linearly with the size of the dataset. It is found to
give superior performance when the data size is very large. Besides, it
can be used for multi-class problems, whereas SVMs are suitable for
two-class problems. Linear SVM is popularly used to classify high-
dimensional patterns.
We explain the working and the features of the linear SVM using
a simple two-dimensional example shown in Figure 5.3. There are
two lines in the two-dimensional space; the points (2, 0)t and (3, 1)t
are on one line and another point (0, 0)t is on the second line. In
high-dimensional spaces, we will have hyperplanes instead of lines.
The first line in the example figure is given by

x1 − x2 − 1 = 1.

Note that both the points (2, 0)t and (3, 1)t satisfy this equation.
Also if we consider the point (1, −1)t , it satisfies the above equation
as it falls on the corresponding line. The second line is characterized

O
X(3,1)
O
X
X
(0,0)O X(2,0)

Figure 5.3. Two-dimensional dataset.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 151

Classification 151

by

x1 − x2 − 1 = −1.

The point (0, 0)t satisfies this equation. Also, point (1, 1)t satisfies
this equation as it falls on this line. The first line may be represented
as W t X + b = 1 where W and X are vectors and b is a scalar. In the
two-dimensional case

W = (w1 , w2 )t ; X = (x1 , x2 )t .

In this example, W = (1, −1)t and b = −1. The second line is of the
form W t X + b = − 1. In the two-dimensional case, these lines are
called support lines. In the high-dimensional case, we have support
planes characterized by W t X+b = 1 and W t X+b = −1. It is possible
to show that the normal distance or margin between these two planes
2
is ||W || . In classification based on SVM, we use the patterns from two
different classes, positive and negative classes, to learn W and b.
Any point X from the positive class satisfies the property that
W t X + b ≥ 1 with support vectors (some kind of boundary vectors)
from the positive class satisfying W t X +b = 1. Similarly, points from
the negative class satisfy W t X + b ≤ −1 with support vectors from
the negative class satisfying W t X + b = −1. So, the margin is con-
cerned with distance between the two support planes; we would like
2
to maximize the margin. So, we would like to maximize ||W || . Equiv-
2
alently, we can minimize ||W2 || .
By assuming a point Xi in positive class has the label yi = 1 and
a point Xj in the negative class has the class label yj = −1. So, the
optimization problem has constraints. The optimization problem is

||W ||2
Minimize ,
2
such that yi (W t Xi + b) ≥ 1 for all the patterns Xi , i = 1, 2, . . . , n.
The Lagrangian associated with the optimization problem is

1 n
L(W, b) = ||W ||2 − αi [yi (W t Xi + b) − 1].
2
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 152

152 Introduction to Pattern Recognition and Machine Learning

By taking the gradient of the Lagrangian with respect to W and


equating to zero, we get W = αi yiXi , where α’s are Lagrange
Xi ∈S
variables. There are several standard methods and software pack-
ages to find the α’s. Further by differentiating with respect to b and
n
equating to zero, we get αi yi = 0.
i+1
By looking at the pattern (3, 1)t on the first line and (1, 1)t on the
second line, we get W = α1 (3, 1)t − α2 (1, 1)t by plugging in the values

in the equation W = αi yi Xi . One can find the values of α’s using
Xi ∈S
some standard methods. For α1 = 1 and α2 = 2, we get W = (1, −1)t .
Then we can use W and the points (3, 1)t and (1, 1)t along with the
constraints, we get b = − 1. So, in the example situation, the decision
boundary between the two classes is in between the two support lines
and is given by x1 − x2 = 1. It is possible to observe the following:

• W is orthogonal to the decision boundary and is directed toward

the positive class. So, W decides the orientation of the decision
boundary. Any point falling in the left-hand side (negative half)
is labeled negative and any point falling on the right-hand side of
the decision boundary is labeled as positive. For example, for the
point X = (0, 1)t , the value of W t X + b is −2(<−1) and so it
is classified as a member of the negative class. The point (3, 0)t
has the value to be 2(>1) and so is classified as a member of the
positive class.
• b is the threshold that decides the location of the decision boundary
with respect to the origin. When b is negative as in the current two-
dimensional example, then the decision boundary is below the line
x2 = x1 which passes through the origin. In the current example,
origin is in the negative side of the decision boundary.
• Learning the SVM classifier involves obtaining the weight vector
W and the threshold b value from the training patterns. Typically,
a subset, S, of the set of training patterns is the set of support
vectors. Elements of S are adequate to obtain W and b based

on W = αi yi Xi . This requires learning the values of α for
Xi ∈S
each element of S. The value of b is obtained from the constraints
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 153

Classification 153

corresponding to the elements in S; specifically we obtain b using

yi (W t Xi + b) = 1 for Xi ∈ S.
• So, the whole learning boils down to learning α’s for the patterns.
If a pattern Xj is not a support vector, then the value of αj is
zero (0). If some Xi ∈ S, then the αi is either positive or zero.
So, typically positive αi ’s identify the important support vectors
which contribute to the vector W .

Most of the above observations are applicable in the high-dimensional

case also. Linear SVM is adequate to deal with linearly separable
classes. If there are some mis-classifications because of positive points
falling in the negative half or some negative points falling in the posi-
tive half, then the optimization problem is updated to accommodate
such noise.

4.1. SVM–kNN
A hybrid method is used here where SVM is not applied to the entire
training data. A coarse and quick categorization is done by finding
the kNN. Then SVM is performed on the smaller set of examples
which are more relevant.
The algorithm is as follows:

1. Find a set of k1 neighbors to the query where k1 > k by using a

crude distance function.
2. Find k NN by using a more accurate distance function on the k1
samples.
3. If all the k neighbors have the same label, the query is labeled.
4. If not, find the pairwise distance of the query and the k neighbors.
5. Convert the pairwise distance matrix into a kernel matrix using
the kernel trick.
6. Apply DAGSVM on the kernel matrix to label the query.

Applying DAGSVM to a large dataset is intractable. When

DAGSVM is used on only the limited number of examples, the classi-
fication can be done in a reasonable time. DAGSVM can also be used
for multi-class problems. It has been shown experimentally that this
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 154

154 Introduction to Pattern Recognition and Machine Learning

method when applied to very large datasets with high dimensionality

gives accurate results in a reasonable time.

4.2. Adaptation of cutting plane algorithm

While SVM formulation involves a large number of examples n and
a large number of features d, each example has only f features where
f << d. The cutting-plane algorithm for training linear SVMs has a
complexity of O(f n). This algorithm is based on an alternative but
equivalent formulation of the SVM optimization problem.
The primal form of the SVM optimization problem can be written
as

C
n
1
minw,ξ≥0 wT w + ξi , (14)
2 n
i=1

s.t. ∀iε1, . . . , n : yi (wT xi ) ≥ 1 − ξi .

Another formulation of this problem uses only one slack variable
ξ for all constraints. It can be written as
1
minw,ξ≥0 wT w + Cξ (15)
2
 
s.t. ∀cε{0, 1}n : n1 wT ni=1 ci yi xi ≥ n1 ni=1 ci − ξ.
The Wolfe dual of Eq. (15) can be written as

1
n

c
1
maxα > 0 αc − αc αc
xTc xc
(16)
n 2
cε{0,1}n cε{0,1 c
ε{0,1}n

s.t.

αc ≤ C,
cε{0,1}n

where

1
n
xc = ci yi xi .
n
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 155

Classification 155

Using Eq. (15), the algorithm given below can be used for training
the SVM.

1. The input to the algorithm is the set of patterns X = ((x1 , y1 ),

. . . , (xn , yn )), the constant C and
.
2. Working set WS = φ
3. repeat
4. (w, ξ) = argminw,ξ≥0 12 wT w + Cξ
 
s.t. ∀cεWS : n1 wT ni=1 ci yi xi ≥ n1 ni=1 ci − ξ
5. for i  = 1, . . . , n do
1 yi (wT xi ) < 1
6. ci =
0 otherwise
7. end for

8. WS = WS {c}

9. until n1 ni=1 ci yi (wT xi ) ≤ ξ +

10. return(w, ξ).

This algorithm has a working set WS which gives the constraints

being considered. It starts with the working set being Φ i.e. with an
empty set of constraints. In every iteration, it finds the most violated
constraint for Eq. (15) and adds it to the working set WS. The
stopping criterion depends on the training loss ξ.
is a small value
which gives the error which can be tolerated in this value. Usually,
0.1% is an acceptable training error. The optimization problem in
line 4 can be done using the Wolfe dual as given in Eq. (16). This dual
requires O(|WS|2 ) elements of the Hessian to be calculated which will
take O(|WS|2 f n) time. Since d ≤ f n, the dual is independent of n
and f . So, the overall time complexity per iteration is O(fn).

4.3. Nystrom approximated SVM

The primal form of the SVM can be written as

1 n
minw,b wT w + C η(w; φ(xi ), yi ). (17)
2
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 156

156 Introduction to Pattern Recognition and Machine Learning

The dual form can be written as

1
minα αT Qα − eT α (18)
2
s.t. αT y = 0 and 0 ≤ αi ≤ C∀i,
where Q is a kernel matrix such that Q = K(X, X) = {K[xi , xj )}ij
where K(xi , xj ) = φ(xi )T φ(xj ).
The computation of Q takes O(n2 d) where n is the number of
samples and d is the dimensionality. Nystrom approximation is of the
form Q = P Wd P T where P = K(X, B)εRnXd and BεRdXd consists
of n1 n examples which are called the projection bases. Wd = W −1
where W = K(B, B).
Finding a basis matrix for the problem is not trivial. One way of
selecting the basis is to randomly sample instances from the train-
ing set to form the basis matrix. Random sampling gives the reduced
support vector machine (RSVM). Generally, a stratified random sam-
pling is carried out where equal number of patterns are selected from
each class to avoid imbalance in the dataset. Another method is to
find a basis set that maximizes the accuracy of the model on the
dataset.

5. Logistic Regression
There are several linear models that have been successful in classi-
fying large datasets. We have seen one such model that is the linear
SVM. Another popular model that has been used on large datasets
is the logistic regression model. Here, we model the ratio of the like-
lihoods. Specifically, we assume that
P (X|C1 )
ln is linear in X.
P (X|C2 )
Observing that X is a vector, our assumption would result in the
log-likelihood ratio being equal to some scalar of the form W t X + q,
where W is a vector and q is a scalar; this expression is linear in X.
This may be easily achieved in the following case.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 157

Classification 157

Example 1. Let the likelihood values of X be obtained based on the

assumption that both the classes C1 and C2 are normally distributed.
Further, to simplify the analysis, let us assume

• X is univariate that is X is a scalar.

• Let the standard deviation be the same for both the classes; let it
be σ.
• Let the means of C1 and C2 be µ1 and µ2 respectively.

P (X|C1 )
Then the likelihood ratio P (X|C2 ) is given by

 
exp − 12 (X−µ 1)
2
 
σ 2 1
 2
 = exp − 2 {(X − µ1 ) − (X − µ2 ) } 2
2
exp − 12 (X−µ
σ 2
2) 2σ
 
µ1 − µ2
= exp {2X + (µ1 + µ2 )} .
2σ 2

So,

P (X|C1 ) µ1 − µ2
ln = {2X + (µ1 + µ2 )}.
P (X|C2 ) 2σ 2

The right-hand side of the above equation can be viewed as W X + b

where W = µ1σ−µ 2
2 −µ2
and q = µ12σ 2 (µ1 + µ2 ). In a multi-dimensional
case, we will have W and X to be vectors of the same size and q
to be a scalar. We can show that the logarithm of the likelihood
ratio will be linear in X, specifically of the form W t X + q where W
and X are d-dimensional vectors and the underlying distributions are
Gaussian with different means and the same covariance matrix. This
is a situation where the logistic regression is optimal for classification.
So, we have

P (X|C1 )
ln = W t X + q,
P (X|C2 )
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 158

158 Introduction to Pattern Recognition and Machine Learning

or correspondingly by considering the ratio of the posterior proba-

bilities, with P (C1 ) and P (C2 ) as the prior probabilities
P (C1 |X) P (X|C1 ) P (C1 )
ln = ln + ln
P (C2 |X) P (X|C2 ) P (C2 )
P (C1 )
= W t X + q + ln
P (C2 )
= W t X + b,

where b = q + ln PP (C1)
(C2 ) . This gives us a linear form for both the log-
arithm of the likelihood ratio and the logarithm of the ratio of the
posterior probabilities. We can simplify further to write P (C1 |X) in
terms of W t X + b as follows. We have
P (C1 |X)
ln = W t X + b.
P (C2 |X)
In a two-class problem P (C2 |X) = 1 − P (C1 |X). So,


P (C1 |X)
ln = W t X + b.
1 − P (C1 |X)
This implies, by taking exponentiation on both sides, that
P (C1 |X)
= exp(W t X + b).
1 − P (C1 |X)
By simplifying further, we get

P (C1 |X)[1 + exp(W t X + b)] = exp(W t X + b).

So,
exp(W t X + b)
P (C1 |X) = .
(1 + exp(W t X + b))

By dividing the numerator and the denominator by exp(W t X + b),

we get
1
P (C1 |X) = .
1 + exp[−(W t X + b)]
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 159

Classification 159

The right-hand side of the above equation is the sigmoid function.

1
The above equation is of the form P (C1 |X) = 1+exp(−t) where t =
t 1
W X +b and s(t) = 1+exp(−t) is the sigmoid function; s(t) takes values
in the range 0 to 1.
Given a set of n training patterns {X1 , X2 , . . . , Xn } where each
pattern Xi is labeled using C1 or C2 , if Xi is from class C1 , then
we choose yi = 1 and if Xi is from class C2 , then yi = 0. Because
P (C2 |X) = 1 − P (C1 |X) and P (C1 |X) = s(W t X + b), we can have
the following least squares learning algorithm. Obtain W and b cor-
responding to

n
min (P (C1 |Xi ) − yi)2 .
i=1

Because P (C1 |Xi ) = s(W t Xi + b), we can write the optimization

problem as find W and b that minimize

1
n
{s(W t Xi + b) − yi }2 .
2
i=1

6. Semi-supervised Classification
It is generally assumed that there is a large amount of labeled training
data. In reality, this may not be true. Often, there is some labeled
data and a large amount of unlabeled data. This maybe due to the
fact that when preparing the training data, getting labels maybe a
difficult task or the cost of obtaining labels maybe very costly. This
maybe due to one of the following reasons:

1. Getting labels may require a lot of work.

2. It may require an expert to find the labels.
3. It may require some expensive device or expensive computation.

The classification of such data is called semi-supervised classifica-

tion. The techniques used should take advantage of the large amount
of unlabeled data but make do with just a small set of labeled data.
While the unlabeled data by itself may not be sufficient to carrying
out classification, they may contain discriminative information which
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 160

160 Introduction to Pattern Recognition and Machine Learning

is useful for classification. Since semi-supervised learning generally

requires less human effort and helps to achieve higher classification
accuracy, it is of interest to us. Generalization error is reduced by
using the unlabeled data.
The problem can be described as follows. In the training data,
we have l labeled data i.e.
Xl = {(x1 , θ1 ), (x2 , θ2 ), . . . , (xl , θl )}
and the remaining data is unlabeled data
Xu = {xl + 1 , . . . , xn }.
Usually the unlabeled data is very much larger than the labeled
data.

i.e. n − l >> l.

Many of the semi-supervised learning algorithms make use of the

cluster assumption which states that the decision boundary should
not cross high density regions but should instead lie in low density
regions.
Given below are some of the techniques used for semi-supervised
learning.

6.1. Using clustering algorithms

This is a simple algorithm which uses existing clustering algorithms.
It is as follows:

1. The clustering algorithm is run on the labeled and unlabeled data.

2. All the points in each cluster is labeled by the majority class of
all the points in the cluster.
3. The newly labeled data and the labeled data are used for
classification.

6.2. Using generative models

The decision boundary obtained using the labeled data and the
unlabeled data is different from the decision boundary obtained
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 161

Classification 161

by using only the labeled data. This is because we get a mixture

model.
The algorithm for generative models is as follows:

1. Choose a generative model p(x, y|θ).

2. Find the maximum likelihood estimation (MLE) on labeled and
unlabeled data
θ ∗ = argmaxθ p(Xl , Yl , Xu |θ).
3. Compute class distribution using Bayes’ rule:
p(x, y|θ ∗ )
p(y|x, θ ∗ ) = . (19)
Σy
p(x, y  |θ ∗ )
One method of finding the MLE i.e. finding θ ∗ is by using
the expectation maximization (EM) algorithm. An EM algorithm
is an iterative method for finding the maximum likelihood estimate
(MLE).
The steps for using EM to find θ ∗ are as follows:
1. Use the labeled data to find p(y|θ) which is the proportion of
data with label y and p(x|y, θ) which is the mean and covariance
of data with label y.
2. Repeat
(a) Compute the expected labels p(y|x, θ) for all the unlabeled
data and assign the class labels.
(b) Update MLE θ with the labeled data and the newly labeled
unlabeled data.
3. Until there is no change in θ.

6.3. Using low density separation

In this method, the cluster assumption is used. This states that the
decision boundary should not cross high density regions, but should
reside in low density regions. Graph-based distances are derived from
the data points which help in identifying the low density regions
between clusters. After this, an SVM is trained so that the decision
boundary is placed in low density regions. To find the low density
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 162

162 Introduction to Pattern Recognition and Machine Learning

regions, a graph is drawn using the data points in the dataset. Edges
are drawn between nodes which are NN. This graph is then used to
identify the low density regions.

6.4. Using graph-based methods

This makes use of a graph drawn using the labeled and unlabeled
training patterns as vertices and undirected edges connecting the
vertices with a weight attached to it. For an edge between patterns
xi and xj , there is an edge with a weight wij which reflects the
proximity of xi and xj . There are a number of parameters which are
used for wij .
1. Gaussian edge weight function:
2 /σ2 )
wij = e(−xi −xj  . (20)
The graph is fully connected but as the distance between the
points increases, the weight decays.
2. kNN edge weight function:
In this function, wij is 1 if xi is one of the kNN of xj . Otherwise
wij is 0.
If wij is large, it implies that f (xi ) and f (xj ) are likely to be the
same. The graph energy of a function f is:
2
i,j=1 wij (f (xi ) − f (xj )) .
Σl+u (21)
Looking at the graph energy, we can order all the weight func-
tions f . It is advantageous to use f values of top ranked functions.
To find an f function which fits the data well and ranks high, we
need to minimize the function:
1
argminf Σli=1 c(f (xi ), yi ) + λ1
f
2 + λ2 Σl+u 2
i,j=1 wij (f (xi ) − f (xj )) ,
f
(22)
where c(f (x), y) is a convex loss function such as the hinge loss or
the squared loss. λ1 and λ2 are weights assigned to the two terms.
This convex optimization problem can be solved.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 163

Classification 163

Some graph-based methods are described below.

1. Mincut
In the two-class problem, the positive labels act as sources and
the negative labels as sinks. The minimum set of edges are found
such that all flow from the source to the sink is blocked. Then
all nodes connected to the source are labeled positive and those
connected to the sink are labeled negative.
Another definition of mincut is the mode of a Markov ran-
dom field with positive and negative labels. Mincut minimizes the
function which combines the loss function and regularizer. It can
be written as
1
M∗ (yi − yi|L) + wij (yi − yj )2 , (23)
2
iεL i,j

subject to yi ε{0, 1}.

The first term is a quadratic loss function where M is a very
large constant. The second term is the regularizer.
2. Discrete Markov Random Fields
The marginal probabilities of the discrete Markov random fields
are computed. A sampling technique is used.
Another approach is to use a continuous relaxation to the
discrete Markov random fields by using the harmonic function
method. This uses a quadratic loss function and a regularizer
based on the Laplacian ∆ function given by

M∗ (fi − yi )2 + f T ∆f, (24)
iεL

∆ is the Laplacian given by ∆ = D − W ,

D is the degree matrix and is a diagonal matrix.

n
Dii = Wij ,
j=1

and W is the n × n weight matrix for both labeled and unlabeled

data which is positive and symmetric.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 164

164 Introduction to Pattern Recognition and Machine Learning

3. The local and global consistency method uses a loss function

n
(fi − yi )2 ,
i=1

1 1
and a normalized Laplacian D − 2 ∆D− 2 is used in the regularizer
giving the function
1 1
f T D− 2 ∆D − 2 f. (25)

4. Tikhonov Regularizer
This algorithm uses the loss function and the Tikhonov regularizer
giving
1
(fi − yi)2 + βf T ∆f. (26)
l
i

∆ can be replaced by ∆m where m is an integer. This is the

smoothness matrix. β is a parameter where βεR.

6.5. Using co-training methods

In this method, two classifiers are used. The labeled data is split
into two parts and is used as labeled data for the two classifiers
respectively. Each classifier adds labeled data to the other classifier.
The algorithm is as follows:
(1) (2)
1. Split labeled dataset into two sets: Xl and Xl . The features are
(1)
split into two parts where Xl is the labeled data using feature
(2)
set 1 and Xl is the labeled data using feature set 2.
(1) (2)
2. Use Xl to train classifier C (1) and Xl to train classifier C (2).
3. Classify Xu separately with C and C (2).
(1)

4. Add C (1) ’s k-most confident patterns from Xu to C (2) ’s labeled

data.
5. Add C (2) ’s k-most confident patterns from Xu to C (1) ’s labeled
data.
6. Repeat Steps 3, 4, and 5.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 165

Classification 165

This method assumes that there is a split in the features which

may not be the case. It is less susceptible to error than self-training.
Some variations of co-training are given below:

1. Co-EM:
• For every pattern x, each classifier probabilistically gives a label.
• Add (x, y) (where y is the label) with the weight P (y|x).
2. For different feature splits
• Create random feature splits.
• Apply co-training.
3. Use multiple classifiers
• Train multiple classifiers using labeled data.
• Classify unlabeled data with all the classifiers.
• Unlabeled data is labeled according to majority vote.

6.6. Using self-training methods

This is the simplest semi-supervised learning method. This involves
using the labeled data and building a classifier. This is used to clas-
sify the unlabeled data. The sample in the unlabeled set with the
highest confidence is taken and added along with its label to the
labeled data. This is repeated so as to increase the size of the labeled
dataset.
The algorithm is as follows:

1. Choose a classification method. Train the classifier C using Xl .

2. Use C to classify the unlabeled data x ∈ Xu .
3. Pick x∗ with the highest confidence and add (x∗ , C(x∗ )) to the
labeled data.
4. Repeat Steps 2 and 3 as many times as required.

This method is simple and can be used with any existing classifier
but it is possible that the mistakes made in classification will keep
reinforcing themselves.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 166

166 Introduction to Pattern Recognition and Machine Learning

6.7. SVM for semi-supervised classification

While the standard SVM maximizes the margin in the labeled data,
in semi-supervised SVM, the margin is maximized for the unla-
beled data. This is also called Transductive Support Vector Machine
(TSVM). All the 2u possible labelings of the unlabeled data is enu-
merated. For each of the labelings, a standard SVM is built. The SVM
with the largest margin is picked. Finding the TSVM solution in this
way is NP-hard and the algorithms cannot handle large datasets.
Another formulation of semi-supervised SVM or TSVM is as
follows:

1. We have as input the kernal K, weights λ1 and λ2 , the labeled

data (Xl , Yl ) and the unlabeled data Xu .
2. Solve the optimization problem for f (x) = h(x) + b, h(x) ∈ Hk .
The optimization problem can be formulated as follows:

minf Σli=1 (1 − yi f (xi ))+ + λ1
h
2HK + λ2 Σni=l+1 (1 − |f (xi )|)+ .

(27)

The last term arises from assigning the label sign f (x) to the
unlabeled points.
3. Classify a test pattern x by sign(f (x)).

6.8. Random forests for semi-supervised classification

Here the small number of labeled data and the large number of unla-
beled data has to be used to build the decision trees in the forest.
The features chosen at a node depends on the information gain. In
the case of semi-supervised learning, the information gain at node j
has two component, Iju and Ijl . So we get

Ij = Iju + βIjl . (28)

Here, Ijl depends only on the labeled data. Iju depends on both
the labeled and unlabeled data. β is a weighting factor.
|Yji |
Ijl = E(Yj ) − Σi=1,2,...,s E(Yji ), (29)
|Yj |
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 167

Classification 167

where E gives the entropy, Yj is the subset of patterns at node j, Yji

is the subset of patterns at the ith split and s gives the number of
branches at the node j.
|Yji |
Iju = log|cov(Yj )| − Σi=1,2,...,s log|cov(Yji )|. (30)
|Yj |
Here, Yj refers to all the labeled and unlabeled data and cov(Yj )
refers to the covariance matrix of the relative displacement vector for
all the points belonging to Yj .
By using this mixed information gain, it is possible to build a
forest of decision trees for semi-supervised learning.

7. Classification of Time-Series Data

Time series classification is the classification of multivariate data
when the values of one or more variables are in the form of a sequence.
Most of the data recorded is in the form of time series, for example,
electricity demand, stocks and shares prices, weather data such as
rainfall and temperature, medical data such as ECG and blood pres-
sure, etc. In fact, other forms of data can also be meaningfully con-
verted to a time series format. These include text, DNA, video, audio,
images, etc. Time series occur in medical, scientific, and business
domains. Some applications of classification of time series data are
1. The electroencephalogram (EEG) signals are used to classify
whether the patient is neurologically healthy or is suffering from
neurological disorders such as epilepsy, etc.
2. Signature verification is used to classify whether the signature is
genuine or not. This task is called anomaly detection.
3. The daily, weekly, and monthly activity of the stock prices can be
used to predict future stock market prices.
4. Weather data like temperature, humidity, rainfall, etc. can be used
to predict weather.
Generally, a time series t = t1 , . . . , tr is an ordered set of r data
points. The data points are typically measured at successive points
of time spaced at uniform time intervals. Time series classification is
the task of learning a classifier C, which is a function that maps a
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 168

168 Introduction to Pattern Recognition and Machine Learning

time series t to a class label l, i.e. C(t) = l where l ∈ L, the set of

class labels.
The time series classification methods can be divided into three
large categories:

• Distance-based classification: Distance-based methods compute

the distance between pairs of time series. The method used to
measure the distance is crucial to the performance of the classifi-
cation algorithm.
• Feature-based classification: Feature-based methods transform the
time series data into feature vectors and then apply conventional
classification methods. Feature selection plays an important role
in these methods as it decreases the dimensionality of the data.
• Model-based classification: Model-based methods use a model
such as Hidden Markov Model (HMM), Artificial Neural Network
(ANN), Recurrent Neural Network (RNN), etc. to classify time
series data.

7.1. Distance-based classification

Some well-known similarity measures for time series data are
Euclidean Distance (ED), dynamic time warping (DTW) distance,
longest common subsequence (LCSS), etc. Once the distance compu-
tation is fixed, any of the standard classification algorithms can be
used. In fact, the NN classifier is found to perform well in distance-
based classification.
Some of the similarity measures used for time series data have
been discussed in Chapter 2.
For symbolic sequences, such as protein sequences and DNA
sequences, alignment-based distances can be adopted. An optimum
alignment score can be computed between two sequences. The
Needleman–Wunsch algorithm computes the optimum global align-
ment score between two sequences using dynamic programming.
The Smith–Waterman algorithm and BLAST measure the similarity
between two sequences by considering the most similar regions but
not considering the alignment on the full length. These are local
alignment algorithms.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 169

Classification 169

7.2. Feature-based classification

The time series data is transformed into a vector of features generally
using feature selection. Conventional classification methods such as
decision trees, neural networks, etc. can be used on the transformed
data. Some feature selection methods for time series data have been
discussed in Chapter 3.
One method used for classification of time series data is the SVM.
The SVM is used to map a sequence into a feature space and find the
maximum-margin hyperplane to separate the two classes. Sometimes,
a kernel function is found which corresponds to a high-dimension
feature space. Some kernels that have been used are k-spectrum ker-
nel or string kernel, polynomial kernel, Fisher’s kernel, and diffusion
kernel.
For symbolic data, each element is treated as a feature. Fea-
ture selection is carried out by selecting a short sequence segment
of k consecutive symbols as a feature. These are called the k-grams.
By using k-grams as features, time series data can be classified by
using conventional classification methods. It is also possible to select
a small informative subset of freatures from the k-grams. Another
method of finding features in symbolic data is to find features which
are short sequence elements which satisfy the following criteria:

1. They are frequent in at least one class.

2. They are distinctive in at least one class.
3. They are not redundant.

After finding features, a classification algorithm is used.

Another feature extraction technique is to transform the time
series into the frequency domain and then carry out dimensional-
ity reduction. Transforming the time series into frequency domain
can be done using discrete fourier transform (DFT), discrete wavelet
transform (DWT) or singular value decomposition (SVD).
Another method used for feature extraction is the kernel method
(KM). SVM is the KM used. By calculating the inner product of
the input vectors in high dimension (which represent the input time
series), linear decision boundaries can be drawn between the classes.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 170

170 Introduction to Pattern Recognition and Machine Learning

It is possible that features are extracted from the time series.

The features extracted are the mean µ, the standard deviation σ,
the kurtosis ks, and the skew sk. These are given by the following
equations:
n
x(i)
µ = i=1 , (31)
n
n
(x(i) − µ)2
σ = sqrt i=1 , (32)
n
n
(x(i) − µ)3
sk = i=1 , (33)
nσ 3
n
(x(i) − µ)4
ks = i=1 − 3. (34)
nσ 4
Second-order features are found by transforming the time series
using a user-specified value C. The new time series is

xˆ(i) = x(i + C) − x(i), 1 ≤ i ≤ n − C.

The same four statistical features of Eqs. (31)–(34) are used to

find the second-order features. The eight first-order features and
second-order features form the feature vector and is used for clas-
sification of the time series. Multi-layered perceptron (MLP) neural
network can be used for classification. Since the feature vector is of
length 8, the input to the network is of size eight and will have much
less neurons and weights as compared to an MLP which inputs the
entire time series as the input.

7.3. Model-based classification

Model-based classification assumes that the time series are generated
by an underlying model. The model finds the probability distribution
of the time series. A model is defined for the time series data and
the probability distributions are described by a set of parameters.
In the training stage, the parameters of the model are learned. Once
the parameters of the model are learned, a new time series is assigned
to the class with the highest likelihood.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 171

Classification 171

Some of the statistical models that can be used are Naive Bayes,
Gaussian, Poisson, Markov, and hidden markov model (HMM).
When using HMM, training examples are used to learn the tran-
sition probabilities between the states. An HMM consists of a set of
states, an alphabet, a probability transition matrix T = (tij ) and a
probability emission matrix M = (mik ). In state i, the system has a
probability of tij of moving to state j and a probability mik of emit-
ting symbol k. For each class, an HMM is built using the training
data. A new pattern is given the class label of the model which fits
the data the best.
An ANN can also be used. Two types of ANN used are MLP
and RNN.

Research Ideas
1. Is it possible to design better condensation algorithms compared to CNN and
MCNN in terms of space and condensation time requirements?

Relevant References
(a) V. S. Devi and M. N. Murty, An incremental prototype set building tech-
nique. Pattern Recognition, 35:505–513, 2002.
(b) V. S. Devi and M. N. Murty, Pattern Recognition: An Introduction.
Hyderabad, India: Universities Press, 2012.
(c) M. N. Murty and V. S. Devi, Pattern Recognition: An Algorithmic
Approach. London: Springer, 2012.
(d) S. Gracia, J. Derrac, J. R. Cano and F. Herrera, Prototype selection for near-
est neighbor classification: Taxonomy and empirical study. IEEE Trans-
actions on PAMI, 34:417–435, 2012.
2. The usual distance metrics such as ED do not work well in high-dimensional
spaces. Can we find metric or non-metric distance functions such as fractional
norms that work well in high-dimensional spaces?

Relevant References
(a) C. C. Aggarwal, Re-designing distance functions and distance-based appli-
cations for high dimensional data. SIGMOD Record, 30:13–18, 2001.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 172

172 Introduction to Pattern Recognition and Machine Learning

(b) K. Beyer, J. Goldstein, R. Ramakrishnan and U. Shaft, When is nearest

neighbors meaningful? Proceedings of Seventh International Conference
Database Theory, pp. 506–515, 2000.
(c) C.-M. Hsu and M.-S. Chen, On the design and applicability of distance
functions in high-dimensional data space. IEEE Transactions on Knowl-
edge and Data Engineering, 21:523–536, 2009.
(d) L. Chen and R. Ng, On the marriage of lp-norms and edit distance.
Proceedings of VLDB, 2004.
3. Why should random forests exhibit superior performance over some of the
other random feature selection and extraction schemes like Latent Semantic
Hashing (LSH) and random projections?

Relevant References
(a) L. Breiman, Random forests. Machine Learning, 45(1):5–32, 2001.
(b) X. Z. Fern and C. E. Brodley, Random projection for high-dimensional
data clustering: A cluster ensemble approach. Proceedings of ICML, 2003.
(c) A. Andoni and P. Indyk, Near-optimal hashing algorithms for approxi-
mate nearest neighbors in high dimensions. Communications of the ACM,
51:117–122, 2008.
(d) Y. Ye, Q. Wu, H. Z. Huang, M. K. Ng and X. Li, Stratified sampling for
feature subspace selection in random forests for high dimensional data.
Pattern Recognition, 46:769–787, 2013.
4. Bagging and boosting are two useful techniques to improve classifiers perfor-
mance. How can one combine them in classification using random forests?

Relevant References
(a) L. Breiman, Bagging predictors. Machine Learning, 24:123–140, 1996.
(b) T. K. Ho, The random subspace method for constructing decision forests.
IEEE Transactions on PAMI, 20:832–844, 1998.
(c) P. J. Tan and D. L. Dowe, Decision forests with oblique decision trees.
Proceedings of MICAI, 2006.
5. Like the fuzzy random forests, is it possible to consider random forests based
on other soft computing tools?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 173

Classification 173

Relevant References
(a) Q.-H. Hu, D.-R. Yu and M.-Y. Wang, Constructing rough decision forests,
in D. Slezak et al. (eds.). Berlin, Heidelberg: Springer-Verlag, 2005,
pp. 147–156. LNAI 3642.
(b) H. Shen, J. Yang, S. Wang and X. Liu, Attribute weighted Mercer kernel-
based fuzzy clustering algorithm for general non-spherical data sets. Soft
Computing, 10:1061–1073, 2006.
(c) A. Verikas, A. Gelzinis and M. Bacauskiene, Mining data with random
forests: A survey and results of new tests. Pattern Recognition, 44:2330–
2349, 2011.
6. What is the reason behind the success of linear SVM classifier in dealing with
classification in high-dimensional spaces?

Relevant References
(a) D. Liu, H. Qian, G. Dai and Z. Zhang, An iterative SVM approach to
feature selection and classification in high-dimensional datasets. Pattern
Recognition, 46:2531–2537, 2013.
(b) M.-H. Tsai, Y.-R. Yeh, Y.-J. Lee and Y.-C. Frank Wang, Solving nonlinear
SVM in linear time? A Nystrom approximated SVM with applications to
image classification. IAPR Conference on Machine Vision Applications,
2013.
(c) T. Joachims, Training linear SVMs in linear time. Proceedings of KDD,
2006.
(d) G.-X. Yuan, C.-H. Ho and C.-J. Lin, Recent advances of large-scale linear
classification. Proceedings of the IEEE, 100:2584–2603, 2012.
7. The so-called nonlinear SVM employs the kernel trick to obtain a linear deci-
sion boundary in a higher-dimensional space, thus effectively increasing the
dimensionality of the patterns. However, the random forest classifier consid-
ers a random subspace at a time to construct a decision tree which forms a
part of the forest. Also, there are plenty of other dimensionality reduction
techniques that perform well in classification. How can one reconcile to the
fact that both increase in the dimensionality (kernel SVM) and decrease in the
dimensionality (random forests and other classifiers) improve the classification
performance?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 174

174 Introduction to Pattern Recognition and Machine Learning

Relevant References
(a) M.-H. Tsai, Y.-R. Yeh, Y.-J. Lee and Y.-C. F. Wang, Solving nonlinear
SVM in linear time? A Nystrom approximated SVM with applications to
image classification. IAPR Conference on Machine Vision Applications,
2013.
(b) S. Haykin, Neural Networks and Learning Machines, Vol. 3. Upper Saddle
River: Pearson Education, 2009.
(c) G. Seni and J. F. Elder, Ensemble methods in data mining: Improving
accuracy through combining predictions. Synthesis Lectures on Data
Mining Knowledge Discovery, 2:1–126, 2010.
(d) X. Hu, C. Caramanis and S. Mannor, Robustness and regularization of
support vector machines. JMLR, 10:1485–1510, 2009.
(e) N. Chen, J. Zhu, J. Chen and B. Zhang, Dropout training for support vector
machines. arXiv:1404.4171v1, 16th April 2014.
8. Can we pose the semi-supervised classification problem as a simpler optimiza-
tion problem?

Relevant References
(a) I. S. Reddy, S. K. Shevade and M. N. Murty, A fast quasi-Newton method
for semi-supervised support vector machine. Pattern Recognition, 44:
2305–2313, 2011.
(b) X. Chen, S. Chen, H. Xue and X. Zhou, A unified dimensionality reduction
framework for semi-paired and semi-supervised multi-view data. Pattern
Recognition, 45:2005–2018, 2012.
(c) X. Ren, Y. Wang and X.-S. Zhang, A flexible convex optimization model
for semi-supervised clustering with instance-level constraints. Proceed-
ings of ISORA, 2011.
9. Is it possible to consider semi-supervised dimensionality reduction which can
help in efficient and effective classification?

Relevant References
(a) K. Kim and J. Lee, Sentiment visualization and classification via semi-
supervised nonlinear dimensionality reduction. Pattern Recognition, 47:
758–768, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 175

Classification 175

(b) R. Cai, Z. Zhang and Z. Hao, BASSUM: A bayesian semi-supervised

method for classification feature selection. Pattern Recognition, 44:811–
820, 2011.
(c) X. Kong and P. S. Yu, Semi-supervised feature selection for graph classi-
fication. Proceedings of KDD, 2010.
(d) K. Dai, H.-Y. Yu and Q. Li, A semisupervised feature selection with support
vector machine. Journal of Applied Maths, 2013, 2013.
10. How to scale up classification algorithms dealing with temporal data?

Relevant References
(a) S. Laxman and P. Sastry, A survey of temporal data mining. Sadhana,
31:173–198, 2006.
(b) Z. Xing, J. Pei and E. Keogh, A brief survey on sequence classification.
SIGKDD Explorations, 12:40–48, 2010.
(c) N. Piatkowski, S. Lee and K. Morik, Spatio-temporal random fields: Com-
pressible representation and distributed estimation. Machine Learning,
93:115–139, 2013.
11. It is possible to view patterns as transactions and use frequent itemset-based
classifiers. What is the role of frequent itemsets in classification?

Relevant References
(a) H. Cheng, X. Yan, J. Han and P. S. Yu, Direct discriminative pattern mining
for effective classification. Proceedings of ICDE, 2008.
(b) M. N. Murty and V. Susheela Devi, NPTEL Lecture Notes on Pat-
tern Recognition, http://nptel.ac.in/courses.php [accessed on 2 November
2014].
(c) B. Fernando, E. Fromont and T. Tuytelaars, Mining mid-level features for
image classification. International Journal of Computer Vision, 108:186–
203, 2014.
12. One way to reduce space and time requirements in classification is to compress
the data and design classifiers in the compressed domain. How to realize such
classifiers in practice?

Relevant References
(a) D. Xin, J. Han, X. Yan and H. Cheng, Mining compressed frequent pattern
ets. Proceedings of VLDB, 2005.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch05 page 176

176 Introduction to Pattern Recognition and Machine Learning

(b) T. R. Babu, M. N. Murty and S. V. Subrahmanya, Compression Schemes

for Mining Large Datasets: A Machine Learning Perspective. New York:
Springer, 2013.
(c) M. Danieletto, N. Bui and M. Zorzi, RAZOR: A compression and classi-
fication solution for the internet of things. Sensors, 14:68–94, 2014.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 177

Chapter 6

Classification using Soft Computing Techniques

1. Introduction
Hard classifiers or the classical classification techniques make a hard
or definite decision on the class label of the test patterns. The many
classifiers we have discussed till now fall under this. However, several
current day applications require each pattern to belong to one or
more classes. For example, a document may belong to both sports
and politics. Such applications motivate the need for soft classifica-
tion. A soft classifier either gives the degree of classification of the
test pattern to every class label, or may classify the test pattern
as belonging to more than one class. Classifiers based on genetic
algorithms (GAs) or neural networks start with some random values
(for the candidate solution or the weights) and depending on the
performance on training patterns, these values are adapted.
Some of these methods which we will discuss in this chapter are
as follows:
1. Fuzzy Classifier: In this classifier, each pattern belongs to every
class with a membership value. To predict the class label of a
test pattern, its fuzzy membership to every class is determined
and the class to which its membership is maximum is the class
chosen. Further, in a multi-label scenario, classes could be ranked
based on the respective membership values and more than one
class label could be assigned to the test pattern based on the
ranking.
2. Rough Classifier: Here, every pattern belongs to the lower
approximation of one class or to the upper approximation of more

177
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 178

178 Introduction to Pattern Recognition and Machine Learning

than one class. This type of classifier is suitable when the patterns
in the domain can belong to more than one class.
3. Genetic Algorithms (GAs) for Classification: GAs work
with a population of candidate solutions which are initialized ran-
domly. Each chromosome in the population is evaluated to see
how well it carries out the problem to be solved. This is called the
fitness function. The advantage of GAs is that they are generally
used for optimization and can handle problems which are complex,
non-differentiable and are multi-modal and multi-objective. Local
minima is easily avoided because of working with a population
of chromosomes. GAs for classification usually attempt to find
a dividing hyperplane between classes, find the set of rules for
classification etc.
4. Neural Networks: The neural network is inspired by the neural
system in human beings and the neurons in the brain. The neu-
ral network consists of the input layer, 0–2 hidden layers and an
output layer. The weights in the network are adjusted so as to get
the correct class label when the training patterns are input to the
network.
5. Multi-class Classification: The patterns in this scenario belong
to more than one class. If we have a label set L = {c1 , . . . , ck } then
each pattern belongs to a subset of L. One such application is when
the news items in the newspaper have to be classified. A news item
can belong to say both politics and movies if a movie star is in
politics. The task is much more complex here as it is necessary to
not only predict the subset of labels but also the ranking of the
labels.

2. Fuzzy Classification
In the conventional classification algorithms, each pattern to be clas-
sified belongs to one class. This is a crisp classification paradigm. In
fuzzy classification, each pattern belongs to each class with a mem-
bership value. If we consider a pattern P , µP 1 , µP 2 , . . . , µP C are the
membership values of pattern P to classes 1, 2, . . . , C. This can be
converted into crisp classification by assigning pattern P to the class
to which its membership value is highest.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 179

Classification using Soft Computing Techniques 179

2.1. Fuzzy k-nearest neighbor algorithm

To classify a test pattern, the k-nearest neighbors are found. These
neighbors have a fuzzy membership degree to the class labels. A test
pattern y belongs to a class C to the degree given by:


C(y) = R(x, y)C(x),
x
N

where R(x, y) gives the similarity between x and y and is defined as:

y − x
−2/(m−1)
R(x, y) =  −2/(m−1)
.
j
N
y − j

2.1.1. Fuzzy kNN classification

This is a modification of the kNN classifier. It is used to find the
membership function µP i , which is the membership value of the pat-
tern P to each class i. When a pattern P is to be assigned a class
label, its k closest neighbors are found. If the jth nearest neighbor
belongs to class i, pij is set to 1. This is done for the first k-nearest
neighbors. The values for all other patterns are set to zero. Then, the
membership values are calculated as follows:
 
k 1
j=1 pij 2
d(P,Xj ) m−1
µP i =   , i = 1, . . . , C. (1)
k 1
j=1 2
d(P,Xj ) m−1

The constant m generally takes the value 2. If µP l is the largest

membership value of pattern P to the classes, pattern P is assigned
the class label l.

3. Rough Classification
Here, we use the notion of an upper and lower approximation of a set
to define approximate classification. An approximation space A con-
sists of A = (U, R) where U is the universe and R is a binary equiv-
alence relation over U , which is called the indiscernibility relation. If
(p, q)εR, then p and q are indiscernible in A. Equivalence classes of
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 180

180 Introduction to Pattern Recognition and Machine Learning

R are called elementary sets in A. A finite union of elementary sets

in A is called a definable set in A.
If we consider a set P ⊂ U , an upper approximation AP , is the
least definable set in A containing set P . A lower approximation of
the set P is AP , which is the greatest definable set in A contained
in P . The boundary set of P in A is given by B(X) = AP − AP .
Now if we consider a family of subsets of U i.e. C =
{P1 , P2 , . . . , Pn }, Pi ⊂ U , then the lower approximation of C in
A is the family

AC = {AP1 , AP2 , . . . , APn }

and the upper approximation is the family

AC = {AP1 , AP2 , . . . , APn }.

If C is a partition of U , then
 
n
Pi Pj = Φ for every i, j, 1 ≤ i, j ≤ n, Pi = U.
i=1

In this case, C is a classification of U and Pi are the classes of C.

An information system can be expressed as I = (U, F, V, τ ) where
U is the universe of I, F is the set of features or attributes, V gives
the set of values of the attributes and τ given by τ : U XF → V is a
description function such that τ (p, f )εVf for every f εF and pεU .
As an example, if we consider Table 6.1 which is an information
function τ , then

U = {p1 , p2 , . . . , p7 }, F = {a, b, c}, Va = {0, 1},

Vb = {0, 1, 2, 3, 4}, Vc = {0, 1, 2, 3}.

3.1. Rough set attribute reduction

In an information system, some attributes do not provide any addi-
tional information about the objects in U and can be removed to
make the decision process simpler and more cost-effective. If a clas-
sification task maps a set of variables F to a set of labels L then a
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 181

Classification using Soft Computing Techniques 181

Table 6.1. Information function τ .

U a b c

p1 0 4 1
p2 1 2 0
p3 1 1 3
p4 0 0 2
p5 1 3 2
p6 0 4 3
p7 0 3 0

reduct D is defined as D ⊂ F , such that τ (F, L) = τ (D, L). A reduct

set is the set D = {P εP S(F ): τ (P, L) = τ (F, L)}.
A minimal reduct Dr is any reduct D such that |Dr | ≤ |P |, ∀P εD.
It is the reduct of least cardinality.

3.2. Generating decision rules

Let the set of attributes consist of n independent attributes and one
dependent attribute, i.e. F = {f1 , . . . , fn } independent attributes and
a dependent attribute d. Suppose we have a partition {P1 , . . . , Pu }
induced from F and let {Q1 , . . . , Qv } be a partition induced from d.

With each Pi , the set Si = {Qj : Pi Qj = φ}. This means that

if pεPi , then pεQj1 or . . . or pεQjv .

Each class Pi corresponds to a feature vector (ai )i≤n where pεPi

only if fq1 = a1 , . . . , fqn = an and pεQ1 only if fd = bj for some
bj εVd . This means that the rule is of the form.
If fq1 = a1 and . . . and fqn = an

then fd (p) = bj1 or . . . or fd (p) = bjj . (2)

If the pattern Pi belongs to only one partition Qj then the value

of d in the Eq. (2) is unique. Otherwise, fd (x) may belong to any class
contained in Si and there is a proper disjunction on the right-hand
side of Eq. (2).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 182

182 Introduction to Pattern Recognition and Machine Learning

Once the decision rules in the form of Eq. (2) are generated,
simplification is done so that as many of the condition attribute val-
ues are removed as possible without losing the required information.
This process is called value reduction. All the rules are kept in a
set Rule. One rule at a time is taken and copied to r. A condition is
removed from r and rule r is checked for decision consistency with
every rule belonging to Rule. If r is inconsistent, then the dropped
condition is restored. This is repeated for every condition of the rule.
The rule r after all these processes is the generalized rule. If r is
included in any rule belonging to Grule, r is discarded. If any rule in
Grule is included in r, these rules are removed from Grule. After all
the rules in Rule is processed, we get the rules in Grule. This set of
rules are called maximally general or minimal length.

4. GAs
GAs are a robust method of carrying out optimization based on
the principles of natural selection and genetics. It is a search pro-
cess where a candidate solution is used with an evaluation function.
A candidate solution is a chromosome or string and there is a popula-
tion of strings. These are evaluated and the next generation of strings
are generated by using the operations of selection, crossover and
mutation. After repeating this procedure for a number of iterations,
the candidate solution generated which gives the best evaluation is
the final solution.
The GA has been used for carrying out classification. The fol-
lowing sections discuss some algorithms using GA for classification.

4.1. Weighting of attributes using GA

This algorithm is equivalent to the weighted kNN classifier. The
strings of the GA consist of d elements giving real valued weights.
These are values between 0 and 1 which give the relative weighting
to each attribute. For example, if there are 10 features then a string
in the population would be of the form:

[0.45 0.20 0.93 0.11 0.56 0.77 0.32 0.45 0.69 0.85].
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 183

Classification using Soft Computing Techniques 183

This means that the first feature is given a weight of 0.45,

feature 2 is given a weight of 0.2 etc. The goal is to find the weighting
which gives the optimal performance on the training data. If Xi and
Xj are two d-dimensional patterns, the weighted euclidean distance
is used which is:

k=d


Dij =
wk (Xik − Xjk )2 ,
k=1

wk gives the weight assigned to the kth attribute and is obtained

from the string in the GA being used at that point of time.
The fitness function is calculated by using the weights and using
kNN algorithm to carry out classification. The simplest fitness func-
tion would be to use the error in classification, i.e.

total − correct
fitness = ,
total
where correct refers to the number of patterns correctly classified
and total refers to the total number of patterns.
Another fitness function that can be used gives weights which
result in the maximum class separation. This is given by:

total − correct nm /k
fitness = a ∗ +b∗ ,
total total

where nm is the number of the nearest neighbors (out of the k-nearest

neighbors) which are not used in any subsequent classification and is
called the minority set of the nearest neighbors.
Another fitness evaluation used is to use ranking of the chromo-
somes. The first criterion used is the number of misclassified sam-
ples(e). If this is equal then other factors like the number of k neigh-
bors which are of the same class (k same neighbors (ns)), the total
distance to the k same neighbors (ds), the number of k neighbors
which are not of the same class (k different neighbors (nd)) and the
total distance to the k different neighbors (dd) are used. The criteria
for chromosome A to be ranked higher than chromosome B are as
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 184

184 Introduction to Pattern Recognition and Machine Learning

follows in the order given:

neA < neB or;

neA = neB and
nsA ≥ nsB and
dsA < dsB and
ddA > ddB or;
neA = neB and
   
dsA ddA dsB ddB
+ < + .
nsA ndA nsB ndB
Since it is necessary to work on a population of strings, the algo-
rithm is time-consuming but parallelism can be used. A simple way
of carrying out the parallelism is to use different processors to carry
out the evaluation of the strings. If the evaluation process is time con-
suming, as usually is the case, this would help in reducing the time
required for the algorithm. Parallelism can be done by distributing
individuals in a landscape and the reproduction is biased according to
the spatial location of individuals. Micrograined parallelism only
carries out the fitness calculation in parallel by passing the strings to
individual processors. This type of parallelism helps in speeding up
the process if the fitness calculation dominates the GA calculations.

4.2. Binary pattern classification using GA

In this approach the string in the GA consists of rules of the form

condition
: class − label
.
The condition part would give values for each dimension and for
each class. So the rule would be of the form
(y11 , . . . , y1i , . . . , yid ), . . . , (yj1 , . . . , yji , . . . , yjd ), . . . ,
(yc1 , . . . , yci , . . . , ycd ): ω.

Here, i = 1 . . . d, j = 1 . . . c and ω gives the class label. Since

it is a binary dataset, every yji ε 0,1*, where * is the do not care
symbol. Each string in the population refers to one rule. To evaluate
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 185

Classification using Soft Computing Techniques 185

the string, it is matched against the training dataset. A 0 in the string

matches a 0 in the training pattern, a 1 matches a 1 and a do not care
symbol * matches either a 0 or a 1. Each training pattern is matched
to each of the class feature vector and is classified as belonging to the
class with the largest number of matches. The class of the training
pattern is compared to the class label ω given in the string. If they
match then the classification is correct. The fitness function of the
rule is based on the classification accuracy of the rule. The fitness
function is calculated as follows:

correct invalid
fitness = +a∗ ,
total total

where invalid gives the number of attributes which have the same
value in the rule for all classes, i.e. if y1i = y2i . . . = yci , then the
attribute i is invalid. After running the GA for a number of iterations,
the rule which gives the best fitness is chosen.

4.3. Rule-based classification using GAs

Another formulation has each individual in the population consisting
of a fixed number of rules. Each individual is a complete solution
candidate. In each individual, there are a fixed number of rules. For
each rule, each attribute has as many elements as the number of
discrete values it takes. So for an attribute which has five discrete
values, each rule in an individual has five elements corresponding to
the five discrete values and each element takes the values 0 or 1. For
every rule, it is said to belong to class i if

No. of class i matched by rule No. of class i in training data

> .
No. matched by rule No. of training data

The fitness function consists of four parameters. The fitness of an

individual I is given by:

Fitness(I) = Error rate(I) + Entropy(I)

+ Rule consistency(I) + Hole(I).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 186

186 Introduction to Pattern Recognition and Machine Learning

1. Error rate: Each training pattern is classified by using all the rules
in an individual and the majority class label is assigned to the
pattern. This is compared with the class label given and the error
rate of the individual or string is the percent of training patterns
which are misclassified.
2. Entropy: In the training patterns that a rule R matches, if pi is
the percent of patterns belonging to class i, then


n
Entropy(R) = − pi log2 pi.
i=1

Since an individual consists of a number of rules, the overall

entropy of the individual is the average of the entropy of the
individual rules.
L
Entropy(Ri )
Entropy(individual) = i=1 ,
L
where L is the number of rules.
3. Rule consistency: Preference is given to the individual which has
the more consistent rule set.

Rule − consistency(individual)
= −pcorr log2 pcorr − (1 − pcorr )log2 (1 − pcorr ),

where pcorr is the proportional of rules in an individual whose class

label is the same as that of the training example.
4. Hole ratio: This is a measure of the coverage of the rules on the
training data. The hole ratio is 1-coverage. In the case of the
binary classification problem, it will be:
No(P + ) + No(N − )
Hole(Individual) = 1 − ,
n
where P + are those examples whose actual class is positive and is
classified as positive by at least one rule in the individual, and N −
are those examples whose actual class is negative and is classified
as negative by at least one rule in the individual and n is the total
number of training patterns.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 187

Classification using Soft Computing Techniques 187

4.4. Time series classification

If the time series has a large dimension, then feature selection can
be carried out. After that, when there is a reduced set of features,
support vector machines (SVM) is used for classification.
One method of carrying out the feature selection is to use genetic
programming. The chromosome is in the form of a tree structure
which gives a feature vector. The fitness of an individual is com-
puted as:

1

C
Fitness = n i ai ,
n
i=1

where n is the total number of samples, ni is the number of samples

in class i, C is the total number of classes and ai gives the accuracy
for class i. These values are obtained by using an SVM and measuring
the performance obtained on the training data.
Cross-validation can also be used. If we make N partitions,
Pi , i = 1, . . . , N , N different SVMs are trained on Pi which is the
complement of partition Pi and is tested on the partition Pi . The
cross-validation fitness is given by:

1

N
fitnesscv = a(Pi , Pi ),
N
i=1

where a(Pi , Pi ) is the accuracy obtained of Pi after training on Pi .

4.5. Using generalized Choquet integral with signed

fuzzy measure for classification using GAs
A GA is used to solve the optimization problem for a Choquet-
integral classification problem. A Choquet Hyperplane H is found
which separates points of the two classes.
The signed fuzzy measure µ is defined as a set function:

µ: P (X) → (−∞, ∞),

where µ(φ) = 0.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 188

188 Introduction to Pattern Recognition and Machine Learning

The values of the set function µ on the non-empty sets in P (X)

are denoted by µi , i = 1, . . . , 2n − 1. If we have a two-class prob-
lem where the sample points belong to the same feature attributes
x1 , . . . , xd , the ith sample point of the positive class denoted by
ri is:

ri = (fi (x1 ), fi (x2 ), . . . , fi (xd )).

The Choquet Hyperplane H is expressed as:


H: (c) (p + qf )dµ − A = 0,

p = (p1 , . . . , pd ) and q = (q1 , . . . , qd ) are d-dimensional vectors where

pi ε[0, ∞) with min i pi = 0, and |qi |ε[0, 1] with max i |qi | = 1. A is
a real number. So all the unknown parameters can be assumed to
be in [−1, 1). These parameters are determined so as to maximize
the total sum of signed distances of the training samples from the
Choquet Hyperplane H. Samples belonging to the positive class will
be on one side of H and the samples belonging to the negative class
will be on the other side of H.
A Choquet integral is used to find the Choquet distance of points
which can be used for classification. The distance of a point ri
which belongs to the positive class from the Choquet Hyperplane
is given by:

(c) (p + qf )dµ − A
di = , ij = 1, 2, . . . , m.
µ21 + µ22 + · · · + µ22n −1

If ri belongs to the negative class the distance will be given by:


A − (c) (p + qf )dµ
di = 2 , ij = 1, 2, . . . , m ,
µ1 + µ22 + · · · + µ2n −1

where m is the number of instances of positive class and m is the

number of instances from the negative class.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 189

Classification using Soft Computing Techniques 189

The total signed Choquet distance is:

m

m
Dist = di + di
i=1 i
=1
m  
 
i=1 ((c) (p + qf )dµ − A) − m i=1 ((c) (p + qf )dµ − A)
=  .
2n −1 2
i=1 µ i
(3)

Dist has to be maximized. Misclassified samples will have a neg-

ative value in the above formula. The above formula will not work
well if the dataset is unbalanced, i.e. if m = m .
If the datasets are not of the same size, a revision has to be
made in Eq. (3) where a large penalty coefficient is added for each
misclassified instance. So the distance is:




m

m
Distmod = ci di + ci
di
i=1 i
=1
m  
  
i=1 ci ((c) (p + qf )dµ − A) − m i=1 ci
((c) (p + qf )dµ − A)
=  .
2n −1 2
i=1 µi
(4)

Here
 
c if (c) (p + qf )dµ < A
ci =
1 otherwise

for i = 1, 2, . . . , m and
 
 c if (c) (p + qf )dµ > A
ci
= .
1 otherwise

4.5.1. Representation of chromosomes in the GA

All the unknown parameters of the hyperplane are included in a
chromosome. Each gene is a bit string. The unknown parameters are
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 190

190 Introduction to Pattern Recognition and Machine Learning

µ1 , µ2 , . . . , and vectors p and q and A. The population consists of a

number of chromosomes (N ).

4.5.2. Fitness of the chromosomes

To find the fitness function, each chromosome c has the parame-
ters µ1 , µ2 , . . . , µm , p, q, and A which are used in Eq. (4) to find the
signed Choquet distance Dc using the data samples. The fitness of a
chromosome is given by
Dc − Dmin
Fc = ,
Dmax − Dmin
where

Dmin = minc=1,...,pop Dc ; Dmax = maxc=1,...,pop Dc .

4.5.3. Selection of chromosomes

Chromosomes are chosen for the next generation according to
Roulette Wheel Selection where the slice of each chromosome c in
the roulette wheel is the probability pc given by:
Fc
pc = N ,
i=1 Fi

where Fc is the fitness of the cth chromosome.

4.5.4. Generation of new chromosomes

Two new chromosomes are generated by choosing mutation or
crossover at random using a two-point probability distribution
(α, 1 − α). This is repeated N2 times to generate N chromosomes. The
fitness of the N new chromosomes are calculated and they are added
to the already existing N chromosomes to form (2N ) chromosomes.
Top N chromosomes with better fitness form the new generation.
The process of selection, generating new chromosomes and form-
ing the next generation is repeated till there is no significant improve-
ment in the maximum fitness. When the iterations are stopped,
the chromosome with the maximum fitness in the last generation
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 191

Classification using Soft Computing Techniques 191

is chosen. The values of µ1 , µ2 , . . . , p, q and A from this chromosome

are chosen to form the Choquet Hyperplane H.

4.6. Decision tree induction using

Evolutionary algorithms
Decision trees have been designed using Evolutionary algorithms.
These are stochastic algorithms where the solution is found by the
method of search. We work on a population of chromosomes where
each chromosome is a representation of the decision tree and is a
candidate solution. The fitness function reflects how well the decision
tree classifies patterns and generally is the classification accuracy on
a validation set. In the following subsections, a number of represen-
tation schemes, various fitness functions, different types of selection,
crossover and mutation have been discussed.

4.6.1. Representation schemes

The candidate decision tree can be encoded as a fixed length linear
string or a tree-encoding scheme can be used.
The axis-parallel decision tree which is the most common type of
decision tree uses a tree encoding where each node is represented by
a 4-tuple
node = {index , is − terminal , operator , value}.
The first value is the attribute index (attribute to be repre-
sented), the second value is whether the node is a terminal or a
non-terminal node, the third value is the operator to be used such
as (<, >, =) and the fourth is the value to be tested by the attribute
(this is only in a non-terminal node). These nodes have pointers to
the children nodes. For example, a node could be represented as:
[1 0 > 5}.
The above represents the rule f1 > 5.
Another way of encoding also uses a tree-encoding representing
each node as a 7-tuple as follows:
node = {index , label , par , l, r, S},
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 192

192 Introduction to Pattern Recognition and Machine Learning

where label is the class label (meaningful only if the node is a ter-
minal node), par is a pointer to the parent node, l and r are the
pointers to the left and right children respectively. S is an array
where S[0] stores the attribute id and S[1] stores the threshold for
the feature S[0] < S[1] which is boolean giving the outcomes “yes”
or “no”.
In another tree-based formulation, the nodes are either terminal
or function nodes. Terminal nodes consist of an attribute id, attribute
value or a class label. Function nodes consist of a 4-tuple given by:

node = {index , value, l, r},

where value gives the attribute value, l and r are children nodes.
Fixed length encoding is difficult to use for decision trees which
are non-binary. Generally only binary trees are considered for this.
One method, divides the genes into caltrops. Each caltrop consists
of the subtree formed using the node as the root and its two chil-
dren. A non-terminal node is identified by an attribute index and the
terminal node is identified by the value zero.
Another method of using fixed length encoding encodes each node
by two values. A node is represented by:

node = {index , value}.

This shows that the test performed at the node is attribute −

index ≥ value.

4.6.2. Initialization of Population

For fixed length encoding, a random initialization is generally used.
In a tree-based encoding, attributes are chosen at random and the
split values are chosen from a predefined list or range. The depth of
the decision tree is chosen randomly from a range which is generally
[2, maxdepth] where maxdepth gives the maximum depth upto which
a tree can grow. This is called the full method. Another strategy is
to use varying distances from root to the leaves. This is called the
grow method. It is also possible to use a mixture of trees generated
by either the full or the grow method.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 193

Classification using Soft Computing Techniques 193

Another strategy restrains the initial population to a 2-level deci-

sion tree which has only a root and the leaves. It is assumed that the
evolutionary algorithms will generate trees with greater depth. Here
initialization can be done by using each of the attributes as the root
and generating a number of chromosomes with different split values
for the attributes. The 2-level decision trees can be combined to get
trees of more depth.
The values which are chosen for the splits for the attributes is
generally not chosen completely randomly but is taken from the set
of values observed in the training set.
The initial population can also be generated by using a tradi-
tional decision tree algorithm on samples of the training set and
using these decision trees as the initial population.

4.6.3. Fitness value

The most common fitness measure used is the classification accuracy
η on a validation set. It is calculated as:
corr
η= ,
n
where corr gives the number of correctly classified samples and n is
the total number of samples. Sometimes η 2 is used.
Another fitness measure used is the J-measure which uses the
quality of the rules that describe a class as the measure. For a k-class
problem, there are k rules which are of the form (if Yi then ωi ), where
Yi gives a set of disjunctions along the paths used to label instances
belonging to class ωi . Then the J-measure is:

k  
p(ωi | Yi )
J= p(Yi )p(ωi | Yi )log ,
p(ωi )
i=1

where p(Yi ) is the fraction of instances which satisfy condition Yi ,

p(ωi ) is the fraction of instances belonging to class ωi and p(ωi |Yi ) is
the fraction of instances that satisfy Yi and belong to class ωi . If the
value of J is higher, it means the classification accuracy of the tree
is higher.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 194

194 Introduction to Pattern Recognition and Machine Learning

Another measure is the distance score which can be used when

the class labels are sequential integer values. The distance score Dscore
is given by:
 2
1

n
ωi − ωi
Dscore = 1− ,
n ωmax − ωmin
i=1

where n is the number of instances, ωi is the actual class of the ith

instance, ωi is the predicted class of the ith instance, ωmax is the
maximum value of the class labels and ωmin is the minimum value of
the class labels.
The above measures are all single objective fitness functions.
Multi-objective fitness functions try to balance the accuracy and sim-
plicity of the decision tree. Simplicity entails finding a decision tree
of less depth or smaller number of nodes. A cost-sensitive measure
find a balance between sensitivity and specificity.
One such measure combines the classification accuracy and tree
size as follows:
f (T ) = w1 ∗ ηT − w2 ∗ sizeT ,
where w1 and w2 are weights. The measure f (T ) needs to be maxi-
mized. A cost-sensitive approach replaces the efficiency in the above
formula by the misclassification cost. The size size(T ) could be the
number of nodes in the tree or could include both tree depth and
number of attributes in each path of the tree.
Another measure is the Minimum Description Length (MDL)
given by:
MDL = ECL + TCL,
where ECL = Error coding length and TCL = Tree coding length.
They are given by:

 
nol
ECL = log 2 ,
el
lleaves
TCL = (ni + nl ) + ni log 2 s + nl log2 k,
where nol is the number of instances in leaf node l, el is the number
of instances misclassified in leaf node l, ni is the number of internal
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 195

Classification using Soft Computing Techniques 195

nodes, nl is the number of leaf nodes, s is the number of splits and

k is the number of classes.
Another multi-objective fitness function is one based on the recall
and a variation of precision which penalizes false positives more
severely.

4.6.4. Genetic operators

One of the popular methods used for selection is the tournament
selection. In tournament selection, two individuals are selected and
the individual with the better fitness is put into the next gener-
ation of strings. This is repeated N times where N is the size of
the population. Another popular method of selection used is the
roulette wheel selection. Rank-based selection is also used where
the selection is based not on the actual fitness but a ranking of the
individuals.
For fixed length representation, the 1-point crossover is used. For
tree-based representations, nodes are selected in the two nodes to be
crossed over, and the entire subtrees corresponding to these nodes
are exchanged generating two children.
Some methods of mutation are replacing one subtree by another
random subtree, replacing a subtree by a random leaf node or replac-
ing a leaf node by a random subtree. It is also possible to replace the
attribute, the test value or both of a chosen node. Another method
is to replace a randomly selected node by another node which is
already present in the tree and not a random subtree. One strategy
uses two methods of mutation, ‘switch’ and ‘translocation’. ‘Switch’
swaps children of the same parent and ‘translocation’ swaps children
from different parents but in the same tree level. Another strategy
also changes logical operations like >, <, =, =, etc. where one logical
operator is replaced by another.

5. Neural Networks for Classification

Many artificial neural network (ANN) models have been proposed
for pattern classification. The multi-layer feed forward network with
back propagation is one of the popular methods. ANNs carry out
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 196

196 Introduction to Pattern Recognition and Machine Learning

classification by adjustment of the network weights using training

data. ANN classification is a model which exhibits learning by exam-
ple. It can thus be seen that classification using ANN requires a
large training time or design time for large datasets. Once a trained
network is available, the classification time for new patterns is very
small. Another feature of neural network learning is that the learnt
neural network acts like a black box. The new pattern is input and
the class label is output. This basically means that there is no expla-
nation ability in the classifier.
Neural networks work on the principle of the nervous system in
the human brain. It basically consists of nodes which are like the
neurons in the human brain and edges between nodes. The way the
network is formed and the connections between the nodes determine
the behavior of the network. Figure 6.1 shows a node in the neural
network. As we can see, the node has one or more edges as input to
the node.
A two-step process takes place in the node. First an aggregation
is made of each of the inputs along the input edges multiplied by
the weight along the edges. As shown in Figure 6.1. If there are four
inputs X1 , X2 , X3 , and X4 to node j and the weights along these
edges are w1j , w2j , w3j and w4j , then the aggregation results in:

4


bj = wij ∗ Xj .
i=1

X1 W1j

W2j
X2
bj = ΣX i Wij f(bj )
W3j

X3
node j
X4 W4j

Figure 6.1. An example of node in a neural network.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 197

Classification using Soft Computing Techniques 197

An activation function is applied to bj giving f (bj ) as the output of

the neuron.
There are a number of activation functions. The unit step func-
tion is as follows:

0 if bj < T
f (bj ) = .
1 if bj ≥ T
The sign function is:

−1 if bj < T
f (bj ) = ,
+1 if bj ≥ T
where T is the threshold.
The sigmoidal function is:
1
f (bj ) = .
1 + exp(−bj )
This gives a value between 0 and 1.

5.1. Multi-layer feed forward network

with backpropagation
This is the most popular method of classification. The input layer
consists of d neurons where d is the dimensionality or the number of
features. There are one or two hidden layers and the output layer. The
nodes in a layer are connected to all the nodes in the next layer. The
number of output neurons depend on the number of class labels. If
there are C classes, it should be able to represent C distinct outputs.
Figure 6.2 shows a multi-layer feed-forward neural network with one
hidden layer.
I1 , . . . , Id are the d input units, H1 , . . . , Hh are the h hidden
units and O1 , . . . , Oo are the o output units. All the input units are
connected to all the hidden units and these edges are initialized to
random weights between 0 and 1. So we have weights uij , i = 1, . . . , d
and j = 1, . . . , h. All the hidden units are connected to the output
units with the weights given by vij , i = 1, . . . , h and j = 1, . . . , o. The
input to a unit is the aggregate of the input along each input edge
times the weight on the edge. An activation function is applied to this
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 198

198 Introduction to Pattern Recognition and Machine Learning

O1 O2 Oo

........

V=vij , i=1 to h
j=1 to o

H H ........ Hh
1 2

U=uij , i=1 to d,
j=1 to h

........
I1 I2 I
d

Figure 6.2. Architecture of multi-layer feed forward neural network.

total input to get the output of a unit. Popular activation functions

are the sigmoidal function which give a value between 0 and 1, the
unit step function which gives the output as either 0 or 1 and the
sign function which gives the output as −1 or +1. The training data
is fed into the network one by one. Since we know the class label of
the instance, if the predicted class label does not match the actual
class label, then the backpropagation algorithm is used to update
the weights in the network. The weight between the hidden layer
and output layer is updated as follows:

∆vij = α ∗ δj ∗ Hi
and vij = vij + ∆vij ,

where

δj = (Ej − Oj )fj (aj ).

α is the learning rate and is in the range (0,1). Hi is the output

of the hidden unit i, Ej is the actual output which is the actual class
label of the instance. Oj is the output which gives the predicted class
label of the instance. aj is the input of the output unit j and fj (aj )
gives the integration of the activation function on aj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 199

Classification using Soft Computing Techniques 199

The weights between the input layer and the hidden layer is
updated as follows:

∆uij = α ∗ δj ∗ Ii
and uij = uij + ∆uij ,

where


K
∆uij = fj (aj ) δk vjk .
k=1

Ii is the ith input; aj is the input of the hidden unit j; fj (aj )
gives the integration of the activation function applied to aj ; K is
the number of neurons in the next layer.
The learning rate α plays a critical role in the training. If α is
too low, the convergence of the weights to the optimum is very slow
and if α is too high, the weight values oscillate or get stuck in a local
minimum. To tackle this problem, a momentum term β can be added
to make the updation equation as:
t
∆vij = α ∗ δj ∗ Hi + β ∗ ∆vij
t−1
,
t−1
where ∆vij is the incremental change in vij done in the previous
iteration. α and β are values between 0 and 1. Usually α is a small
value like say 0.01 and β will have a larger value like 0.7 or 0.8.
Looking at the error in classification, the two values can be adjusted.
As mentioned earlier, the number of inputs to the input layer
is equal to the number of features and the output is the number
of classes. For example, in a digit recognition problem, there are
10 classes which consist of the digits 0–9. If the digits are represented
as an image with 8 × 8 pixels, then there are 64 features being input
to the neural network, the number of inputs being 64.

5.2. Training a feedforward neural network using GAs

A feedforward neural network is generally trained using the back-
propagation algorithm. The neural network is initialized by taking
random weights along all the edges in the network. When a training
pattern is input to the network, if the predicted class label does not
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 200

200 Introduction to Pattern Recognition and Machine Learning

match the actual class label, the weights in the network are updated
using the backpropagation algorithm. Instead of using the backprop-
agation algorithm, a GA is used to find the best weights which fit
the network.

5.2.1. Representation and initialization

The network design is first done and the weights in the network are
encoded as a list of real numbers in the chromosome. The number
of elements in the string depends on the number of inputs, number
of outputs, the number of hidden layers and the number of hidden
nodes. For example, if there are 5 input units, 10 hidden units and
2 output units, the number of values will be:
N = (5 + 1) ∗ 10 + (10 + 1) ∗ 2 = 82.
The initial weights are chosen at random, uniformly distributed
in the interval +/− the inverse of the square root of the fan-in (num-
ber of input edges) to the neuron. If the neuron
 has a fan-in of 7,
1 1
then the weight is chosen to be in − √7 ; √7 . The fitness function
is determined by running the training dataset through the network,
determining the predicted class labels and finding the sum of squares
of the errors in classification.

5.2.2. Operators used

Three operators are used, mutation, crossover, and gradient. But
mutation takes a chromosome and randomly changes some of the
elements in the chromosome. The crossover operator takes two chro-
mosomes and creates two children by exchanging some of the genetic
material in the two parents. The gradient operator adds a multiple
of the gradient with respect to the evaluation function to all the
elements of a chromosome.
1. Mutation:
(a) Unbiased mutation: For a particular chromosome, with a
probability p = 0.1, this operator replaces each entry with
a random value chosen from the initialization probability
distribution.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 201

Classification using Soft Computing Techniques 201

(b) Biased mutation: For a particular chromosome, with a prob-

ability of p = 0.1, this operator adds a random value chosen
from the initialization probability distribution.
(c) Node-based mutation: A number of nodes n which are not
input nodes are selected in the network and for each of these
nodes, ingoing links are taken and this operator adds to these
link weights, a random value from the initialization probabi-
lity distribution. Generally, n is a small number like say 2.
(d) Mutation of weakest nodes: A node has zero error if its output
links are all zero and since this does not contribute anything
to the network, it is a weak node. The strength of a hidden
node is the difference between the evaluation of the network
as it is and its evaluation with the output of this node set to
zero. After calculating the strength of the hidden nodes, this
operator chooses m weakest nodes and performs mutation of
the ingoing and outgoing links of the node.
2. Crossover:
(a) Crossover of two chromosomes: This operator takes a child
chromosome and for each position, picks one of the values
from the two parent chromosomes and uses this value.
(b) Node-based Crossover: For each node encoded by the child
chromosome, this operator finds the corresponding node in
one of the two parents’ networks. The weight of each ingoing
link to the parent’s node is put as the weight in the corre-
sponding link of the child’s network.
(c) Layer-wise crossover: This operator exchanges the weights on
links connected to two nodes in the same layer. If we consider
two nodes P and Q in the same layer, if node R is connected
to node P by an ingoing or outgoing link, node R is also
connected to node Q. The weight on the link between R and
P is exchanged with the weight on the link between node R
and node Q.
3. Gradient Operator:
The gradient for every instance in the training set is calculated
and summed up to get the total gradient. The total gradient is
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 202

202 Introduction to Pattern Recognition and Machine Learning

divided by the magnitude to get the normalized gradient. The

chromosome chosen is changed in the direction of the normalized
gradient by an amount ∆s. This value is adapted according to
the performance. If the evaluation of the child is worse than the
parent, then we take ∆s = ∆s ∗ 0.4. If the child is better than the
parent then ∆s = ∆s ∗ 1.4.

When the algorithm is started, all the above operators are given
equal probabilities. In the course of a run, the performance of the
operators is observed and the probability of these operators being
selected is increased if the operator is doing well and decreased if the
operator is doing poorly.

6. Multi-label Classification
In this type of problem, each instance is associated with a set of
labels. It is necessary to predict the set of labels of a test pattern using
the training instances with known label sets. For example, when we
are classifying the newsitems in a newspaper, the same article may
belong to Politics and Entertainment say if a filmstar is standing
for election. When classifying a scene, the same image may have a
set of labels say hill, river, tree etc. When carrying out sentiment
analysis of a document, the same document may express the senti-
ment sadness, anger and interest. There are a number of applications
such as annotation of images and video, text classification, functional
genomics and music categorization into emotions.
Let X εRd be the d-dimensional instance domain and let
L = {l1 , . . . , lq } be the set of labels or classes. Multi-label learning
entails learning a function h : X → 2L which maps each instance
xεX to a set of labels. This is called multi-label ranking since the
order of the labels is important. In other words, we need to predict
the labels with their ranking.
Given in Table 6.2 is an example dataset where each pattern
can belong to more than one class. The dataset has four features
fi , i = 1, . . . , 4 and labels lj , 1 ≤ j ≤ 5. The features have normalized
values. The labels for each instance gives the subset of classes to
which the instance belongs. The ranking of the class labels is also
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 203

Classification using Soft Computing Techniques 203

Table 6.2. An example multi-label dataset.

f1 f2 f3 f4 1 2 3 4 5

0.98 0.43 0.23 0.12 2 1

0.78 0.55 0.31 0.88 5 2 1 4
0.69 0.43 0.29 0.41 4
0.71 0.33 0.19 0.53 3 4 1
0.89 0.29 0.10 0.73 2 5 4 1 3
0.83 0.37 0.37 0.36 1 3

taken into account. This means that for the first pattern, the first
label is 2 and the second label is 1. For a new instance, it is necessary
to predict the subset of class labels to which the instance belongs
ranked in the correct order.

6.1. Multi-label kNN (mL-kNN)

mL-kNN is a multi-label lazy learning approach which is derived
from the k-nearest neighbor algorithm. For a test instance, first the
k-nearest neighbors are found. Then according to the number of
neighboring instances belonging to each class, maximum a posteriori
(MAP) principle is used to determine the set of labels for the test
instances.
For a test instance p, the k nearest neighbors are considered. Let
N (p) denote the set of the k-nearest neighbors. If lp is the category
vector for p, lp (c) (cεL) takes the value 1 if c is in the label set
for p and 0 otherwise. Based on the label sets of the neighbors, we
calculate:


Cp (c) = li (c), for c = l1 , . . . , lq ,
i
N (p)

where Cp (c) counts the number of neighbors of p belonging to the

cth class. Let E1c be the event that p has label c and E0c be the event
that p does not have label c. Also let Ijc denote the event that among
the kNN’s of p, there are exactly j instances having label c. Then
  
lp (c) = argmax b
{0,1} P Ebc ICc p (c) , c = l1 , . . . , lq .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 204

204 Introduction to Pattern Recognition and Machine Learning

This is determined using MAP principle.

This equation can be rewritten using Bayes rule as
    
P Ebc P ICc p (c) Ebc
lp (c) = argmax b
{0,1}  
P ICc p (c)
    
= argmax b
{0,1} P Ebc P ICc p (c) Ebc .

So the category
  vector lp can be determined using the prior pro-
 c is the label and b = {0, 1} and the posterior
babilities P Ebc where
c  c
probability P Ij Eb for j = 0, 1, . . . , k. These values can be directly
estimated from the training set.

6.2. Probabilistic classifier chains (PCC)

This method uses the principle of the classifier chain (CC). In CC,
a classifier hi is trained for each label and assigns a scoring function
fi . For a new instance p to be classified, h1 predicts ω1 , h2 predicts
ω2 taking the instance p and the predicted value ω1 . We go on in this
way and hi predicts ωi using ω1 , . . . , ωi−1 .
In PCC, given a query p, the probability of a label combination
ω = (ω1 , . . . , ωm ) is computed using the product rule:

m
Pp (ω) = Pp (ω1 ). Pp (ωi |ω1 , . . . , ωi−1 ). (5)
i=2

To find the joint distribution of labels, the m functions fi (·) is

learnt on an augmented input space PX{0, 1}i−1 , using ω1 , . . . , ωi−1
as additional features, i.e.

fi: PX{0, 1}i−1 → [0, 1]

(p, ω1 , . . . , ωi−1 ) → P (ωi = 1|p, ω1 , . . . , ωi−1 ).

This makes Eq. (5)


m
Pp (ω) = f1 (p). fi (p, ω1 , . . . , ωi−1 ).
i=2

Given Pp an optimal prediction can be derived.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 205

Classification using Soft Computing Techniques 205

6.3. Binary relevance (BR)

BR learns L classifiers, one for each of the class labels. The original
dataset is transformed into L datasets. Each dataset contains all the
instances in the original dataset with the label for each instance being
either positive or negative. In the ith dataset, if the label set for a
instance contains the ith label it is labeled positively; otherwise it is
labeled negatively. To classify a new pattern, it is assigned a class
label by all the L datasets and the union of these labels is the labelset
predicted.

6.4. Using label powersets (LP)

In multi-label learning, one of the methods uses LP where every
distinct combination of labels that exist in the training set is treated
as a different class and single-label classification task is carried out.
Though this makes the task of classification simpler, if the number of
classes is large, the number of labelsets appearing in the training set
maybe very large. This increases the computational cost and makes
the learning difficult as the number of training examples of each
labelset will be very small. To take care of this problem, the initial
set of labels is broken up into small random subsets called labelsets
and then LP is used on these labelsets. This method is called RAkEL
(Random k labelsets). The size of the labelsets is specified using k.
Consider the finite set of classes to be C = {ωi : i = 1, . . . , L}.
Each instance xi is associated with a set of labels Yi where Yi ⊆ C.
A labelset S ⊆ C where k = |S| is called the k-labelset. Two types
of labelsets can be used: (a) Disjoint labelsets and (b) overlapping
labelsets.
In the disjoint version, each labelset
 is of size k. The class labelset
L
C is randomly divided into l = k disjoint labelsets, Si , i = 1, . . . , l.

Since the labelsets are disjoint, li=1 Si = φ. Then, l multi-label
classifiers hi , i = 1, . . . , l are learnt using LP. Each classifier hi is a
single-label classification task having as class labels all the subsets of
Si that are in the training set.
For the classifier hi , the dataset Di contains all the training
instances in the original set but the labels will be the intersection
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 206

206 Introduction to Pattern Recognition and Machine Learning


of the given labels and Si . So Di = {(xj , Yj Si ), j = 1, . . . , n}
where n is the total number of patterns. Some patterns may have
the empty set as the label set. Given a new multi-label instance p,
the binary prediction hi of all classifiers for all labels ωj εSi are found
and used to find the multi-class classification vector.
In the case of overlapping labelsets, Ck is the set of all distinct
k-labelsets of C. The size of C k is |C k | = Lk . If we require l classi-
fiers and the label set size is k, we need to first select l l-labelsets
Si , i = 1, . . . , l from the set C k via random sampling without replace-
ment. Here the labelsets may overlap. Then l multi-label classifiers
hi , i = 1, . . . , l are learnt using LP. To classify a new instance p, every
classifier hi gives a binary prediction for each label in the correspond-
ing labelset Si . Taking all the decisions for the l models, the mean of
the predictions is found for each label ωj εC and decides on the label
if the value is greater than 0.5.

6.5. Neural networks for Multi-label classification

The multi-layer feedforward neural network which is used for single-
label instances is adapted to handle multi-label classification. The
neural network algorithm used is called Backpropagation for Multi-
Label Learning (BP-MLL). If there are instances X such that X ε∇d .
Let there be n training patterns {(x1 , Y1 ), (x2 y2 ), . . . , (xn , Yn )}. If
we have a set of labels C = {ωi , i = 1, . . . , l}, then Yi ⊆ C. The
architecture of BP-MLL is shown in Figure 6.3. The neural network

c1 c2 c l
..............

V=v
h,t
..............
bo b bm
1 U=us,h

..............
ao a a a
1 2 d

Figure 6.3. Architecture of BP-MLL neural network.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 207

Classification using Soft Computing Techniques 207

has d input units each pertaining to one dimension of the feature

vector and there are l output units, each corresponding to one class
label. The hidden layer has m nodes. The input layer is completely
connected to the hidden layer where the weight is usg where 1 ≤ s ≤ d
and 1 ≤ g ≤ m. The weights between the hidden units and the output
is vgt where 1 ≤ t ≤ l. The bias for the hidden units γg is shown as
an extra input unit a0 with a fixed value of 1. The bias parameter βl
for the output units is shown as an extra hidden unit b0 with a fixed
value of 1.
The goal of learning of the labels is to minimize the error on
all the n training patterns. If ei is the error on xi , then the overall
error is:


n
E= ei .
i=1

l  i 
i 2
j=1 oj − dj , where oj is the actual output of the
Here, ei = i

network on pattern xi on the jth class and dij is the desired output
on pattern xi on the jth class. The desired output is either +1(if
jεYi ) or −1(if j ∈
/ Yi).
Another formulation for the overall error is:


n

n
1
  
E= ei = exp − oip − oiq .
|Yi ||Ȳi |
i=1 i=1 (p,q)Yi X Ȳi

This equation adds up the error on each training instance (xi , Yi ).

Ȳi is the complementary set of Yi . oip − oiq is the difference between
the actual output of the network on a label belonging to Yi and
one label not belonging to it. The bigger this difference, the better
is the performance of the network. The term |Y ||1Ȳ | normalizes the
i i
summation of the differences in the output.
The actual output of the jth output unit is
 

m
oj = f  bg vgj + βj .
g=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 208

208 Introduction to Pattern Recognition and Machine Learning

The activation function is the “tanh” function:

ex − e−x
f (x) = .
ex + e−x
bg is the output of the gth hidden unit and is given by
 d 


bg = f as usg + γs .
s=1

The change to be made in the weights ∆v and ∆u is given by:

∂Ei
∆vgt = −α = αdt bg , (6)
∂vgt
∂Ei
∆usg = −α = αeg as . (7)
∂usg

The change in biases is

∆βt = αdt and ∆γg = αeg . (8)

eg is the error of the gth hidden unit and is given by

∂Ei
eg = −  d 
∂ s=1 as vsg

l
= (dt vgt )(1 + bg )(1 − bg )
t=1

and α is the learning rate and the value is (0.0,1.0).

For training, the training instances are fed to the network one
at a time and for each instance, Eqs. (6)–(8) are updated. Feeding
the training instances once through the neural network is called a
training epoch. The training epoch is repeated a number of times.
The training process is repeated till the global error E does not
increase or for a fixed number of training epochs.
When a new pattern is fed into the network after training, we get
the outputs ct , t = 1, . . . , l. A threshold is used and if ct is greater
than the threshold, then the label is included in the labelset for the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 209

Classification using Soft Computing Techniques 209

instance i.e. Y = {t|ct > threshold , tεC}. The total number of weights
and biases in the network is given by:

N = (d + 1) ∗ g + (g + 1) ∗ C.

Overall time for training is O(N · n · e) where n is the number of

training examples and e is the number of training epochs.

6.6. Evaluation of multi-label classification

The performance evaluation of multi-label classification is completely
different from single-label classification. In both cases, classification
on a validation set is carried out and this is used to evaluate the clas-
sifier. It is also possible to carry out cross-validation on the training
data. The measures used for single-label classification is error and
classification accuracy. The classification accuracy η is given by:
N1
η= ∗ 100,
N
where N1 is the number of correct classifications and N is the total
number of patterns. A classifier giving high η value is preferred. The
error in classification E is given by:
N − N1
E− ∗ 100.
N
In the case of multi-label classification, the evaluation met-
rics should depend on factors like number of labels, the rank-
ing of the labels, the first label, etc. Consider a validation set
{(X1 , Y1 ), . . . , (Xv , Yv )}. Here, h(Xi ) returns a set of proper labels
of Xi ; h(Xi , y) returns a real value which gives the confidence for
y to be a proper label of Xi ; rank h(Xi , y) returns the rank of y
derived from h(Xi , y). Given below are some of the metrics used for
multi-label classification.

1. Hamming loss: This evaluates how many times an instance-label

pair is misclassified. This means that the number of times a label
which is not in the label set for an instance is predicted or the
number of times a label which belongs to a label set of an instance
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 210

210 Introduction to Pattern Recognition and Machine Learning

is not predicted is counted. Hamming loss HL is given by:

1
1
v
HL = |h(Xi )∆Yi ,
i|v| |y|
i=1

∆ denotes the symmetric difference between the actual labels of

Xi (i.e. Yi ) and the predicted set of labels (h(Xi )). The best per-
formance of the algorithm is when HL = 0. The lower the value of
HL , the better is the performance of h.
2. One Error : One Error Eone evaluates how many times there is an
error in predicting the top ranked label. It counts the number of
times the top ranked label is misclassified. It is given by:

1

v
Eone =
[argmax yY h(Xi , y) ∈
/ Yi ]
.
p
i=1

The smaller the value of Eone the better is the performance of h.

3. Coverage: Coverage C evaluates how far we need to go down the
list of labels in order to cover the proper labels of the instance. It
is given by:

1

v
C= maxyYi rank h (Xi , y) − 1.
v
i=1

The performance is perfect when C = 0 and the smaller the value

of C, better is the performance.
4. Ranking Loss: Ranking loss RL evaluates the average fraction of
label pairs which are not in the right order for every instance.

1
1
v
RL = {(y1 , y2 )|h(Xi , y1 )
v |Yi ||Ȳi |
i=1

≤ h(Xi , y2 ), (y1 , y2 )εYi X Yi }|.

Ȳi is the complementary set of Yi .

If RL = 0, the performance is perfect. The smaller the value of
RL , the better is the performance of h.
5. Average Precision: The average precision P evaluates the average
fraction of proper labels ranked above a particular label yεYi and
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 211

Classification using Soft Computing Techniques 211

is given by:
 
1
1
 y  |rank h (Xi , y  ) ≤ rank h (Xi , y), y 
Yi 
v
P = .
v |Yi | rank h (Xi , y)
i=1 y
Yi

If P = 1, the performance is perfect and the larger the value

of P , the better is the performance of h.

Research Ideas
1. Can we apply rough-fuzzy approach and the fuzzy-rough approach to pattern
classification?

Relevant References
(a) N. Verbiest, C. Comelis and F. Herrera, FRPS: A rough-fuzzy approach
for generating classification rules. Pattern Recognition, 46(10):2770–2782,
2013.
(b) S. K. Pal, S. K. Meher and S. Dutta, Class-dependent rough-fuzzy granular
space, dispersion index and classification. Pattern Recognition, 45(7):2690–
2707, 2012.
(c) R. Jensen and C. Comelis, Fuzzy-rough nearest neighbor classification.
Transactions on Rough Sets, LNCS, 6499:56–72, 2011.
(d) Y. Qu et al., Kernal-based fuzzy-rough nearest neighbor classification. Inter-
national Conference on Fuzzy Systems, FUZZ:1523–1529, 2011.
2. A neuro-fuzzy system (or a fuzzy neural network) is a fuzzy system which uses
the neural network to learn the parameters of the fuzzy system. How do we use
the neuro-fuzzy system for classification?

Relevant References
(a) A. Ghosh, B. U. Shankar and S. K. Meher, A novel approach to neuro-fuzzy
classification. Neural Networks, 22:100–109, 2009.
(b) R.-P. Li, M. Mukaidono and I. B. Turksen, A fuzzy neural network for pat-
tern classification and feature selection. Fuzzy Sets and Systems, 130:101–
108, 2002.
3. Instead of GAs, other stochastic search techniques such as simulated annealing
or Tabu search can be used. How do these techniques compare with the GA?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 212

212 Introduction to Pattern Recognition and Machine Learning

Relevant References
(a) D. Glez-Pena, M. Reboiro-Jato, F. Fdez-Riverola and F. Diaz, A simulated
annealing-based algorithm for iterative class discovery using fuzzy logic for
informative gene selection. Journal of Integrated Omics, 1:66–77, 2011.
(b) J. Pacheco, S. Casado and L. Nunez, A variable selection method based in
Tabu search for logistic regression models. European Journal of Operations
Research, 199:506–511, 2009.
4. Hybrid GAs combine GAs with operators from other search algorithms like
simulated annealing, local search etc. Can we improve the performance of GAs
by hybridizing them?

Relevant References
(a) W. Wan and J. B. Birch, An improved hybrid GAs with a new local search
procedure. Journal of Applied Mathematics, 2013.
(b) D. Molina, M. Lozano and F. Herrera, MA-SW-Chains: Memetic algorithm
based on local search chains for large scale continuous global optimization.
Proceedings of the 6th IEEE World Congress on Computational Intelligence
(WCCI’10), 2010.
(c) C. Grosan and A. Abraham, Hybrid evolutionary algorithms: Methodo-
logies, architectures, and reviews. Studies in Computational Intelligence
(SCI), 75:1–17, 2007.
5. A number of algorithms exist which mimic the behavior of a swarm of animals
such as Particle Swarm Optimization, Ant Colony Optimization etc. How do
we adapt these algorithms for pattern classification?

Relevant References
(a) B. Xue, M. Zhang and W. N. Browne, Particle swarm optimization for
feature selection in classification: A multi-objective approach. IEEE Trans-
actions on Cybernetics, 43(6):1656–1671, 2013.
(b) H. Dewan and V. S. Devi, A peer-peer particle swarm optimizer. 6th Interna-
tional Conference on Genetic and Evolutionary Computing, pp. 140–144,
2012.
(c) D. Martens, M. De Backer, R. Haesen, J. Vanthienen, M. Snoeck and
B. Baesens, Classification with ant colony optimization. IEEE Transactions
on Evolutionary Computation, 11(5):651–665, 2007.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch06 page 213

Classification using Soft Computing Techniques 213

(d) S. Hodnefjell and I. C. Junior, Classification rule discovery with ant

colony optimization algorithm. Intelligent Data Engineering and Auto-
mated Learning-IDEAL 2012, LNCS, 7435:678–687, 2012.
6. When the number of attributes and instances are large, the time and space com-
plexity of multi-label classification could go up. Can we use feature selection
to reduce the space and time complexity?

Relevant References
(a) N. Spolaor, E. A. Cherman, M. C. Monard and H. D. Lee, A comparison
of multi-label feature selection methods using the problem transformation
approach. Electronic Notes in Theoretical Computer Science, 292:135–151,
2013.
(b) X. Kong, N. Ng and Z. Zhou, Multi-label feature selection for graph classi-
fication. IEEE 10th International Conference on Data Mining (ICDM),
pp. 274–283, 2010.
(c) M. L. Zhang, J. M. Pena and V. Robles, Feature selection for multi-
label naive Bayes classification. Information Sciences, 179(19):3218–3229,
2009.
May 2, 2013 14:6 BC: 8831 - Probability and Statistical Theory PST˙ws

This page intentionally left blank

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 215

Chapter 7

Data Clustering

Clustering deals with grouping of data where a pair of similar data

points are placed in the same cluster or group. So, the notion of
similarity or matching between data points plays an important role
in clustering. Clustering is either hard or soft. A K-partition, C, of a
set X of n data points is {C1 , C2 , . . . , CK }, where
K


Ci = X , Ci Cj = φ, i
= j and Ci
= φ for 1 ≤ i, j ≤ K.
i=1

The number of K-partitions of the n element set, X , is

K  
1  K
(−1)K−i (i)n . (1)
K! i
i=1

For example, partitioning a set of three data points {X1 , X2 , X3 ) into

two clusters will lead to the following possible 2-partitions:
{{X1 }, {X2 , X3 }}; {{X2 }, {X1 , X3 }}; {{X3 }, {X1 , X2 }}; note that
n = 3 and K = 2 here and hence there are three hard partitions each
having two clusters which agrees with (1).

1. Number of Partitions
It is possible to depict onto functions from the set of data points
X to the set of clusters C as shown in Figure 7.1 where |X | ≥ |C|.
Onto functions are important in the context of counting the number
of partitions of a dataset.

215
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 216

216 Introduction to Pattern Recognition and Machine Learning

X1
C1
X2 C2
C3
X3 ..
.
. CK
.
.
Xn

Figure 7.1. Onto function with |X | = n ≥ K = |C|.

The result in (1) may be derived by using the relation between

onto functions and partitions. Specifically, if Nonto (n, K) is the num-
ber of onto functions from the set of n patterns to the set of K
clusters, then the number of K-partitions, ΠnK is given by
1
ΠnK = Nonto (n, K).
K!
In order to estimate the value of Nonto (n, K), we can start with
the total number of functions from X to C; it is K n and is larger
than the required number. Among these functions, every element of
X is assigned to exactly one cluster. However, it is possible that
none of the elements is assigned to one or more clusters in C which
corresponds to having one or more empty clusters. Because we are
interested in counting the number of K-partitions we need to ensure
that none of the clusters is empty; this means that we need to con-
sider the number of onto functions from X to C. Specifically, we need
to subtract the count of the number of functions violating the onto
property from the total number of functions given by K n . So, from
K n we need to subtract the number of functions that have one or
more of C1 , C2 , . . . , CK missing in their range leading to the violation
of the onto (surjective) property.
Let N (Ci ) be the number of functions in which no element of X
is assigned to cluster Ci which means Ci is not in the range of these
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 217

Data Clustering 217

functions. So, these functions assign each element of X to one of the

remaining K − 1 clusters. So, this count is given by

N (Ci ) = (K − 1)n for all 1 ≤ i ≤ K. (2)

If Nv is the number of functions violating the onto property, then

Nv = N (C1 ) + N (C2 ) + · · · + N (CK ) − N (C1 , C2 ) − N (C1 , C3 )

− · · · − N (CK−1 , CK ) + · · · + (−1)K−1 N (C2 , . . . , CK )

by the principle of inclusion and exclusion where N (Ci , Cj ) is the

number of functions with Ci and Cj , 1 ≤ i, j ≤ K, i
= j, being empty
or equivalently missing in the range of the respective functions. Note
that

N (Ci , Cj ) = (K − 2)n for all 1 ≤ i, j ≤ K. (3)

So, the value of Nv can be written in a compact form using (2) and
(3) as
     
K n K n K−1 K
Nv = (K − 1) − (K − 2) + · · · + (−1) 1n .
1 2 K −1

So, the number of onto functions is

Nonto (n, K) = K n − Nv
   
n K n K
=K − (K − 1) − (K − 2)n
1 2
 
K−1 K
+ · · · + (−1) 1n
K−1
K
  
K−i K
= (−1) (i)n .
i
i=1

Note that unlike in the case of onto functions, ordering of clusters is

not important in partitioning as a K-partition is a set of K clusters.
So, we get the number of K-partitions by dividing the number of
onto functions, Nonto (n, K), by K! giving us the result in (1).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 218

218 Introduction to Pattern Recognition and Machine Learning

This result means that exhaustive enumeration of all possible

partitions of a dataset could be prohibitively expensive. For exam-
ple, even for a small dataset of 19 patterns to be partitioned into
4 groups, we may have to consider around 11,259,666,000 partitions.
So, typically each of the clustering algorithms restricts these possi-
bilities by selecting an appropriate subset of the set of all possible
K-partitions characterized by (1).

2. Clustering Algorithms
Conventionally clustering algorithms are either partitional or hier-
archical. Partitional algorithms generate a partition of the set of
data points and represent or abstract each cluster using one or more
patterns or representatives of the cluster. Consider the data points
shown in Figure 7.2. There is a singleton cluster and two other dense
clusters. Here, centroid of a cluster of patterns is used to represent
the cluster as depicted in the figure.
On the other hand, hierarchical clustering algorithms generate
a hierarchy of partitions. Such a hierarchy is typically generated by
either splitting bigger clusters into smaller ones (divisive clustering)
or by merging smaller clusters to form bigger clusters (agglomera-
tive clustering). Figure 7.3 shows a hierarchy which is also called
as dendrogram. There are two clusters at the top level; these are

Centroid
y
xxx
b4 xxxxx
xxx

Centroid
b3 x
xxx
x x Ox x x
b2 xxxxxxx
xxxx x
b1 xx
Outlier (singleton cluster)

a1 a2 a3 x

Figure 7.2. An example dataset.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 219

Data Clustering 219

1 Cluster

2 Clusters

3 Clusters

4 Clusters

5 Clusters

6 Clusters

7 Clusters
8 Clusters

A B C D E F G H

Figure 7.3. Hierarchical clustering.

given by {A, B, C, D} and {E, F, G, H}. The cluster {E, F, G, H} is

split into two clusters {E, F, G} and {H} and {E, F, G} is further
divided into {E, F } and {G}. Similarly, the cluster {A, B, C, D} is
divided into {A, B} and {C, D} and so on.
A hierarchy of partitions of sizes varying from one to eight is
depicted in Figure 7.3. Further, the same hierarchy depicts both the
agglomerative (bottom up) and divisive (top down) clustering. In the
agglomerative case, we start with eight singleton clusters and keep
merging clusters successively based on similarity between clusters
till finally all the patterns are put in a single cluster. The divisive
algorithms start with a single cluster and keep splitting clusters suc-
cessively; a cluster is selected for splitting based on the spread of
points in the cluster.
We consider some variants of an important algorithm and its
behavior using the dataset shown in Table 7.1.

2.1. K-means algorithm

K-means algorithm is the most popular partitional clustering algo-
rithm. It generates a K-partition of the dataset and the clusters
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 220

220 Introduction to Pattern Recognition and Machine Learning

Table 7.1. A dataset of 10 patterns.

Pattern
numbers Feature1 Feature2 Feature3

1 10 3.5 2.0
2 63 5.4 1.3
3 10.4 3.5 2.1
4 10.3 3.3 2.0
5 73.5 5.8 1.2
6 81 6.1 1.3
7 10.4 3.3 2.3
8 71 6.4 1.0
9 10.4 3.5 2.3
10 10.5 3.3 2.1

are represented by their respective centroids. The algorithm is given

below:

K-means Algorithm

Input: Dataset, X ; Number of Clusters, K

Output: A K-partition of X , ΠnK
1. Select K initial centroids corresponding to the K clusters.
2. Assign each of the n points in X to the cluster whose centroid is
closest to the data point. Update the centroids of the clusters
based on the current assignment of points to the clusters.
3. Stop if there is no change in the cluster assignments during two
successive iterations. Otherwise goto 2.

Some of the important features of K-means algorithm are:

• Optimization of Squared Error:
Squared error or the within-group-error-sum-of squares is the cri-
terion function associated with the K-means algorithm. The basic
idea behind K-means algorithm is to minimize this criterion func-
tion. Formally, the function may be specified as
K 

X − centroidi 2 . (4)
i=1 X∈Ci
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 221

Data Clustering 221

Note that the squared error will be maximum when K = 1 and is

minimum (zero) when K = n. So, we consider the minimization of
the criterion function for a given K. The K-means algorithm does
not guarantee global minimum value of the squared error criterion
shown in (4). Further, the squared error minimization corresponds
to minimizing the variance of points in each cluster. So, naturally
this algorithm has a tendency to generate spherical clusters.
• Selection of initial centroids:
1. Select K out of the n data points as the initial centroids. Various
options are:
(a) Select the first K out of n data points as the initial
centroids.
Considering the first three (K = 3) patterns (10, 3.5, 2.0),
(63, 5.4, 1.3), (10.4, 3.5, 2.1) in Table 7.1 as the centroids of
three clusters respectively, the algorithm stops after two
iterations. The three clusters obtained and their centroids
respectively are:
Cluster 1: {(10, 3.5, 2.0)}
Cluster 2: {(63, 5.4, 1.3), (73.5, 5.8, 1.2), (81, 6.1, 1.3),
(71, 6.4, 1.0)}
Cluster 3: {(10.4, 3.5, 2.1), (10.3, 3.3, 2.0), (10.4, 3.3, 2.3),
(10.4, 3.5, 2.3), (10.5, 3.3, 2.1)}
Cluster Centroids: (10, 3.5, 2.0), (72.1, 5.9, 1.2),
(10.4, 3.4, 2.2)
(b) Select K out of n data points randomly as the initial
centroids.
Selecting the points (10.4, 3.5, 2.3), (10.3, 3.3, 2.0),
(10, 3.5, 2.0) as the randomly selected initial centroids, the
algorithm stops after three iterations. The three clusters
and their centroids respectively are:
Cluster 1: {(63, 5.4, 1.3), (73.5, 5.8, 1.2), (81, 6.1, 1.3),
(71, 6.4, 1.0)}
Cluster 2: {(10.4, 3.5, 2.1), (10.3, 3.3, 2.0), (10.4, 3.3, 2.3),
(10.4, 3.5, 2.3), (10.5, 3.3, 2.1)}
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 222

222 Introduction to Pattern Recognition and Machine Learning

Cluster 3: {(10, 3.5, 2.0)}

Cluster Centroids: (72.1, 5.9, 1.2), (10.2, 3.4, 2.2),
(10, 3.5, 2.0)
(c) Select K out of n points as initial centroids such that the K
points selected are as far away from each other as possible.
This scheme has a better chance of reaching the globally
optimal solution of the criterion function in (4). Selection
of the K initial centroids based on this scheme could be
achieved as follows:
(i) Select the most dissimilar points in X as two centroids.
Let them be X 1 and X 2 . Set q = 2.
(ii) If q = K stop. Otherwise select X q+1 , the q + 1th
centroid from the remaining n − q points, where

argmax
X q+1 = (d(X 1 , X) + · · · + d(X q , X))
X
X ∈ X − {X 1 , X 2 , . . . , X q }.

Repeat this step till K centroids are selected.

In the dataset shown in Table 7.1 the two extreme points
are (10, 3.5, 2.0) and (81, 6.1, 1.3); these are selected as the
first two centroids. The third centroid is (63, 5.4, 1.3) as it
is away from the already selected centroids significantly.
Using these three initial centroids, we get the three clusters
and their respective centroids, in two iterations, as:
Cluster 1: {(10, 3.5, 2.0), (10.4, 3.5, 2.1), (10.3, 3.3, 2.0),
(10.4, 3.3, 2.3), (10.4, 3.5, 2.3), (10.5, 3.3, 2.1)}.
Cluster 2: {(73.5, 5.8, 1.2), (81, 6.1, 1.3)}.
Cluster 3: {(63, 5.4, 1.3), (71, 6.4, 1.0)}.
Cluster Centroids: (10.3, 3.4, 2.1), (77.3, 6, 1.2),
(67, 5.9, 1.2).
Note that option 1(c) gives the minimum squared error.
2. Select K random points in the pattern space as the initial
centroids. These points need not be elements of X . Some of
the practical implementations of K-means algorithm use this
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 223

Data Clustering 223

scheme for initial centroid selection. One problem with this ini-
tialization is that it can lead to empty clusters. For example,
consider the dataset shown in Table 7.1. Let the three initial
centroids be (10, 3.5, 2.0), (81, 6.1, 1.3), and (40, 4.8, 1.7) where
the first two are the two extreme points in the dataset and the
third one is approximately at the middle of the line joining the
other two.
Using these three centroids, the clusters obtained are:
Cluster 1: {(10, 3.5, 2.0), (10.4, 3.5, 2.1), (10.3, 3.3, 2.0),
(10.4, 3.3, 2.3), (10.4, 3.5, 2.3), (10.5, 3.3, 2.1)}
Cluster 2: {(63, 5.4, 1.3), (71, 6.4, 1.0), (73.5, 5.8, 1.2),
(81, 6.1, 1.3)}
Cluster 3: { }
Note that based on the patterns, the minimum and maximum
values of feature1 are 10 and 81. So, the range is 10 · · · 81. Sim-
ilarly, for feature2 the range is 3.3 · · · 6.1 and for feature3 it is
1.0 · · · 2.3. The range box in this example is a hypercube based
on these three range values. So, even though all the three initial
centroids are legal and fall in the range box, one of the clusters
obtained using the K-means algorithms is empty in this exam-
ple. In general, one or more clusters could be empty when such
a scheme is used.
• Time and Space Requirements:
Each iteration of the K-means algorithm requires computation of
distance between every data point and each of the K centroids. So,
the number of distance computations per iteration is O(nK). If the
algorithm takes l iterations to converge then it is O(nKl). Further
if each data point and the centroid are p-dimensional, then it is
O(nklp). Also, it needs to store the K centroids in the memory;
so, the space requirement is of O(Kp).

2.2. Leader algorithm

Leader is the simplest of the clustering algorithms in terms of
time and space complexity. It is the earliest reported incremental
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 224

224 Introduction to Pattern Recognition and Machine Learning

algorithm in the literature on clustering. So, it can be naturally used

in stream data mining. It is also a partitional clustering algorithm.

Leader Algorithm
Input: Dataset, X ; Distance threshold, T
Output: A Partition of X , ΠnK
1. Select the first point as the leader of the first cluster. Set K = 1.
2. Consider the next point in X and assign it to the cluster whose
leader has a distance less than the user specified threshold T . Else
increment the value of K and start the Kth cluster with the current
point as its leader.
3. Repeat step 2 till all the points in X are considered for clustering.
Note that
• Threshold Size: The value of T decides the number of clusters
generated; for a given X and a small value of T the algorithm
generates a large number of small size clusters and for a larger
value of T , the algorithm generates a small number of large size
clusters.
• Order Dependence: The order in which points in X are con-
sidered plays an important role; for different orders the resulting
partitions could be different.
We illustrate with a two-dimensional example using the dataset
shown in Table 7.2.

Table 7.2. A two-dimensional dataset.

Pattern
number Feature1 Feature2

1 1 1
2 2 1
3 2 2
4 3 3
5 6 6
6 7 6
7 7 7
8 8 8
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 225

Data Clustering 225

Now consider a value of 0 for the threshold T . Then we get eight

clusters with each cluster having one point; each point in the table
is assigned to a cluster. On the other hand if the threshold value is
4, then we get two clusters; where

• the first cluster is {(1, 1)t , (2, 1)t , (2, 2)t , (3, 3)t } and
• the second cluster is {(6, 6)t , (7, 6)t , (7, 7)t , (8, 8)t }.

In order to consider the effect of the order in which the patterns

are processed, let us consider patterns in the order in which they are
present in the table. Further, let us assume that the threshold value
T is 2. Then we get 4 clusters where:
• the leaders are: (1, 1)t , (3, 3)t , (6, 6)t , (8, 8)t and
• the clusters are: {(1, 1)t , (2, 1)t , (2, 2)t }; {(3, 3)t }; {(6, 6)t , (7, 6)t ,
(7, 7)t }; {(8, 8)t }.

2.3. BIRCH: Balanced Iterative Reducing

and Clustering using Hierarchies
BIRCH offers a hierarchical clustering framework. It constructs a
data structure called Clustering Feature tree (CF tree) which repre-
sents each cluster as a vector called CF vector.
• CF: Let us consider a cluster of n points, in a d-dimensional
space given by, {(x11 , x12 , . . . , x1d )t , (x21 , x22 , . . . , x2d )t , . . . , (xn1 ,
xn2 , . . . , xnd )t }. The CF vector is three-dimensional and is
  n n n

  
n, xj1, xj2 , . . . , xjd ,
j=1 j=1 j=1
 n n n

  
x2j1 , x2j2 , . . . , x2jd ,
j=1 j=1 j=1

where the three components of the vector are:

1. The first component is the number of elements in the cluster
which is n here,
2. The second component is the linear sum of all the points (vec-
tors) in the cluster which is ls which is l-dimensional here, and
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 226

226 Introduction to Pattern Recognition and Machine Learning

3. The third component, squared sum, another d-dimensional vec-

tor, is sum of squares of the components of the points in the
cluster; here it is (9, 6) (=(12 + 22 + 22 , 12 + 12 + 22 )).

• Merging Clusters: A major flexibility offered by representing

clusters using CF vectors is that it is very easy to merge two or
more clusters. For example, if ni is the number of elements in Ci ,
lsi is the linear sum and ssi is the squared sum then

CF vector of cluster Ci is ni , lsi , ssi and

CF vector of cluster Cj is nj , lsj , ssj , then
CF vector of the cluster obtained by merging Ci and Cj is

ni + nj , lsi + lsj , ssi + ssj .

• Computing Cluster Parameters: Another important property

of the CF representation is that several statistics associated with
the corresponding cluster can be obtained easily using it. A statis-
tic is a function of the samples in the cluster. For example, if a
cluster

C = {X1 , X2 , . . . , Xp }, then

p
ls j=1 Xj
Centroid of C = CentroidC = , =
p p

p 2 2
1

i=1 (Xi − CentroidC )

Radius of C = R =
p
 1
ls2 ls2 2
ssi − 2 pi + p2i
=  ,
p

n
n 1
2
j=1 (Xi − Xj )
2
i=1
Diameter of C = D = .
n(n − 1)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 227

Data Clustering 227

• Construction of the CF-tree

The CF-tree may be viewed as a kind of B Tree. Some of its impor-
tant features are:
– Each leaf node has at most L entries where each entry is a CF-
vector and corresponds to a cluster. Each cluster at the leaf
node has a collection of points such that each of the points is
within a user specified threshold distance T from the center of
the cluster. This view means
∗ Each leaf-level cluster may be viewed as a sphere of radius T .
∗ The data points are all numerical vectors.
– Each non-leaf node has upto B children. B is large in most of
the applications where BIRCH is used. Each non-leaf node is
represented by a CF-vector that is the sum of the CF-vectors of
all its child nodes.
– It is possible to insert the data points incrementally into the tree;
so, it requires a single dataset scan to construct the CF-tree.
We illustrate the construction of the CF-tree using a two-
dimensional example.
Example 1. Consider the following collection of two-dimensional
patterns: {(2, 2), (6, 3), (1, 2), (2, 1), (6, 4), (7, 3), (1, 1), (14, 2),
(14, 3), (2, 2)}. By default each pattern is a two-dimensional col-
umn vector; however, we have not explicitly shown the transpose
symbol for the sake of simplicity. These vectors should be treated as
column vectors. The corresponding CF-tree is shown in Figure 7.4;

(7, (50,19)) (3, (5,4))

(3, (35,8)) (4, (15,11)) (2, (3,2)) (1, (2,2))

(1, (7,3)) (2, (28,5)) (2, (3,4)) (2, (12,7)) (2, (3,2)) (1, (2,2))

Figure 7.4. CF-tree for the two-dimensional data.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 228

228 Introduction to Pattern Recognition and Machine Learning

this is based on a value of 1 for T and B = L = 2. In the figure

only a part of the CF-vector for each cluster is shown. Observe that
only the number of elements in the cluster (first component) and
the linear sum of the elements in the cluster (second component)
are shown. The squared sum (third component) is not shown for
the sake of making the figure simple. These details are adequate
to explain the CF-tree construction.
The incremental construction of the CF-tree may be explained
using the details shown in Figure 7.5. Figure 7.5(a) is obtained
after inserting the first pattern; similarly Figures 7.5(b) and 7.5(c)
are obtained after inserting the second and third points respec-
tively. Figure 7.5(d) is obtained after inserting the first seven pat-
terns. In order to insert a point, we need to find the nearest child
of each non-leaf node including the root node. Ultimately, at the
leaf node we either insert the new pattern into one of the clusters at
the leaf node if the distance between the point and the CF-vector
of the cluster is less than T or start a new cluster. We explain
insertion of each point into the CF-tree, in the example, using

(1, (2,2)) (2, (8,5)) (3, (9,7))

(1, (2,2)) (1, (2,2)) (1, (6,3)) (2, (3,4)) (1, (6,3))

(a) (b) (c)

(5, (22,14)) (2, (3,2))

(1, (7,3)) (4, (15,11)) (2, (3,2)) −−−−

(1, (7,3)) (2, (3,4)) (2, (12,7)) (2, (3,2)) −−−−

(d)

Figure 7.5. Incremental construction of CF-tree for the data.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 229

Data Clustering 229

the following:
1. We consider the first data point (2, 2). It forms a cluster and
the corresponding part of the CF-vector is (1, (2, 2)) as shown
in Figure 7.5(a).
2. Now we consider the next pattern (6, 3); the only neighbor is
(2, 2) (centroid of the cluster represented by (1, (2, 2)) at a dis-
tance of approximately 4.1 units which is greater than T (=1).
So, a new cluster has to be initiated; further, the leaf node can
accommodate one more CF entry (cluster) as L = 2. So, we
create a new cluster and the corresponding partial CF-vector
(1, (6, 3)) is inserted into the leaf node as shown in Figure 7.5(b).
3. Now we consider the point (1, 2); the nearest centroid (2, 2) is at
a distance of 1 unit. So, we insert (1, 2) into the cluster with the
centroid (2, 2); the updated part of the CF-vector is (2, (3, 4))
as shown in Figure 7.5(c). Note that after the updation, the
current centroid is (1.5, 2).
4. We consider the pattern (2, 1) next; it is at a distance of approx-
imately 1.1 units from (1.5, 2) (one centroid) and at a distance
of 4.5 units from the other centroid, (6, 3). So, we need to start
a new cluster; it cannot be accommodated in the existing leaf
as L = 2 and already 2 CF entries (clusters) are present in the
leaf node. So, a new leaf node is added. Next, we consider (6, 4)
which is inserted into the same cluster as (6, 3) leading to the
updated CF-vector (2, (12, 7)) as shown in Figure 7.5(d). Next
insert (7, 3) into a new cluster as none of the three existing
centroids is at a distance of less than or equal to T from (7, 3);
the new CF-vector is (1, (7, 3)) which is shown in Figure 7.5(d).
Next, we consider (1, 1) which is assigned to the same cluster
as (2, 1) and the corresponding CF-vector becomes (2, (3, 2));
this is also depicted in Figure 7.5(d).
5. Now by adding the remaining three points in the order (14, 2),
(14, 3), (2, 2), we get the final tree shown in Figure 7.4.
• Order Dependence
Like the Leader algorithm, BIRCH also suffers from order depen-
dence. Note that two copies of the point (2, 2) are assigned to
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 230

230 Introduction to Pattern Recognition and Machine Learning

different clusters as shown in Figure 7.4. This problem is typically

associated with a majority of the incremental algorithms.

2.4. Clustering based on graphs

There are several applications where the data is available in the form
of a graph. Even when the data is in the form of a matrix of size n×d
where there are n patterns each described by d features, it is possible
to convert this data matrix into a similarity graph. In this section,
we deal with algorithms that work on such graph data. First, we
examine an algorithm that works on the conventional data, that is
data represented in the form of n × d matrix.

2.4.1. Single-link Algorithm (SLA)

A graph-based method of clustering is to use the Minimal Spanning
Tree (MST) corresponding to the data. In such a tree each node
corresponds to a data point; so there are n nodes or vertices. In order
to realize it the distance, dij , between the pair of nodes Xi , and Xj
is calculated; typically Euclidean distance is used. This computation
is performed between all possible pairs of patterns. This leads to the
formation of a complete dissimilarity graph G = V, E, D where V
is the set of n nodes; E is the set of edges; and D is the associated
set of distances.
There are several well-known algorithms to compute the MST of
the data from G. An MST is a tree that spans all the n nodes with
the sum of the weights (distances) of the n − 1 edges in the spanning
tree is the minimum. We show in Figure 7.6 the MST of eight two-
dimensional points. Note that there are seven edges and one can use
these edges to obtain the clusters. One can obtain two clusters by
breaking the edge with the maximum weight (distance). In this case
edge BF will be removed to produce two clusters {A, B, C, D} and
{E, F, G, H}. Further breaking can be done to produce more number
of clusters as exemplified in Figure 7.3.
Typically, SLA is used to generate the clusters based on given
data points. It is an agglomerative algorithm. It starts with n sin-
gleton clusters if there are n data points to be clustered. It uses
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 231

Data Clustering 231

F2
H
E F
G

B
A D
C

F1

Figure 7.6. MST of eight points in a two-dimensional space.

the interpoint distance set D to merge two clusters if the distance

between them is less than or equal to the distance between any other
pair of clusters. Here, distance between a pair of clusters Ci and
Cj is the minimum over all possible pairs of points X and Y where
X ∈ Ci and Y ∈ Cj . A major difficulty with the usage of SLA is that
it requires the dissimilarity set D to be computed and stored which
requires O(n2 ) time and space.

2.4.2. Spectral clustering

Partitional clustering algorithms like the K-means produce good
clusters when the data has isotropic or spherical clusters. K-means
algorithm is not suited when the clusters are non-isotropic; specif-
ically when the clusters are chain-like (elongated in a direction) or
concentric (where the clusters have roughly the same centroid; imag-
ine two circles or spheres with the same center and different radii and
the points in each cluster on the respective circle or sphere). Spectral
clustering algorithms are well suited to deal with such datasets.
Spectral clustering algorithms work on datasets represented in
the form of a graph. Here, a graph is viewed as a triple V, E, S
where S is the matrix of similarity values between pairs of nodes in
the graph. Here, the sets V , E, and S are:

• V = {X1 , X2 , . . . , Xn }. Each node/vertex in V corresponds to a

data point in the collection.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 232

232 Introduction to Pattern Recognition and Machine Learning

• E = {Xi , Xj : Xi ∈ V , and Xj ∈ V } for i, j = 1, 2, . . . , n. So, each

element of E characterizes an edge between a pair of vertices.
• S = {sij : Xi , Xj ∈ V }. Each element of S characterizes similarity
between a pair of nodes. sij = 0 if Xi and Xj are not similar (or not
connected); and sij = 1 if Xi and Xj are similar (or connected).
In our treatment the graph is undirected; so sij = sji . However,
there could be applications where the graph is directed. Further,
we are assuming that the similarity values are binary, either 0 or 1;
in general these could be non-negative real numbers.

• Weightmatrix , W : is a diagonal matrix and Wii = sij and

j∈V
Wij = 0 if i
= j. That is ith diagonal element in W is the sum of
the elements in the ith row of S. This could be called the degree
matrix when sij ’s are binary as the entry Wii corresponds to the
degree of node Xi .
We illustrate these ideas using the graph shown in Figure 7.7. The
corresponding S matrix is given by
 
1 1 1 0 0 0
 1 1 1 
 1 0 0 
 
 1 1 1 0 0 0 
S= ,
 0 1 0 1 1 1 
 
 0 0 0 1 1 1 
0 0 0 1 1 1
where we are assuming that a node is similar to itself and so the
diagonal entries are all 1. The weight matrix W (or degree matrix in

A
E

B D

C F

Figure 7.7. A graph with six vertices.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 233

Data Clustering 233

this case) is
 
3 0 0 0 0 0
 
 0 4 0 0 0 0 
 
 0 0 3 0 0 0 
W = .
 0 0 0 4 0 0 
 
 0 0 0 0 3 0 
0 0 0 0 0 3

Let C1 be a subset of V and C2 , the complement of C1 , be V − C1 .

Based on this notation we can generate a two-partition of V using
the notion of mincut.

cut(C1 , C2 ) = sij (5)
Xi ∈C1 ,Xj ∈C2

and mincut is defined as

mincut(C1∗ , C2∗ ) = minimumC1 ,C2 cut(C1 , C2 ), (6)

where C1∗ and C2∗ (= V − C1∗ ) are the optimal values of C1 and C2 .
Such a C1∗ and its complement C2∗ correspond to the two required
clusters of the partition.
It is possible to abstract the mincut expression in a form suitable
for optimization by considering the following.

• Let C1 and C2 be the two possible clusters being considered. Let

these two clusters be viewed as negative (C1 ) and positive (C2 )
clusters. Based on this one can abstract the index vector I of size
n where there are n vertices in the graph; let Ii be −1 if Xi ∈ C1
and +1 if Xi ∈ C2 for i = 1, 2, . . . , n.
• Note that (Ii − Ij ) is 0 if both Xi and Xj are either in C1 or in
C2 . Further, (Ii − Ij )2 is 1 if Xi belongs to one cluster and Xj is
in the other cluster.
• Note that Cut(C1 , C2 ) considers addition of similarities sij where
Xi ∈ C1 and Xj ∈ C2 . We can select such sij ’s by consider-
ing sij (Ii − Ij )2 in the place of sij in the summation (5). So,
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 234

234 Introduction to Pattern Recognition and Machine Learning

Cut(C1 , C2 ) can be equivalently written as


Cut(C1 , C2 ) = sij (Ii − Ij )2 (7)
Xi ∈C1 , Xj ∈C2

1 
= sij (Ii − Ij )2 . (8)
2
Ii =Ij

• It is possible to simplify equation in (8) to show that

Cut(C1 , C2 ) = I t W I − I t SI = I t DI, (9)

where D = W − S. So, minimizing the Cut amounts to finding
the index vector I such that I t DI is minimized; once I is known
it is possible to obtain the clusters based on the polarity of the
entries in I.
• So, the problem of obtaining the mincut amounts to
minI I t DI such that Ii ∈ {−1, 1} f or all i ∈ {1, 2, . . . , n}.
Because this is a combinatorially difficult problem to solve, we
relax the selection of elements in I to real numbers which leads to
minI I t DI such that I t I = n. (10)
• It is possible to see that D is symmetric as S and W are symmetric.
The smallest eigenvalue of D is 0 and the corresponding eigenvec-
tor is 1 = (1, 1, . . . , 1)t because D1 = 0 = 01. By choosing the
value of I as the eigenvector 1, it is possible to show that I t DI is
equal to 0 as DI = D1 = 0. This value of I does not generate a
two-partition as there is only a positive cluster.
• Instead of the smallest eigenvalue, select the next smallest eigen-
value so that I t DI is still small where I is the eigenvector corre-
sponding to the second smallest eigenvalue. Further, because D is
symmetric eigenvectors of D are orthogonal and the eigenvalues
are all real. So, by choosing I to be the eigenvector corresponding
to the second smallest eigenvalue, we get an I that is orthogonal to
1; this means that there will be both negative and positive entries
in I. So, I is chosen to be the eigenvector corresponding to the
second smallest eigenvalue.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 235

Data Clustering 235

We illustrate this algorithm using the example shown in

Figure 7.7. The matrix D is given by
 
2 −1 −1 0 0 0
 −1 3 −1 −1 0
 0 
 
 −1 −1 2 0 0 0
D= .
 0 −1 0 3 −1 −1 
 
 0 0 0 −1 2 −1 
0 0 0 −1 −1 2
√ √
The eigenvalues of D are 0, 5−2 17 , 3, 3, 3, 5+ 2 17 . The first two eigen-
 √ √ √ √ t
vectors are 1 and 1, −3+2 17 , 1, 3−2 17 , −7+4 17 , 3−2 17 . Note that
in the second eigenvector, the first three entries are positive and the
remaining three are negative. So, the clusters are C1 = {D, E, F }
and C2 = {A, B, C} where C1 is the negative cluster and C2 is the
positive cluster. Also note that this clustering is intuitively appealing
as points in each cluster are completely connected.
In the example shown in Figure 7.7, we have considered the pos-
sibility of a two-partition. It is possible in general that the number
of clusters K is greater than 2. In such a case we consider the K
eigenvectors corresponding to the K smallest eigenvalues. Note that
each eigenvector is n-dimensional. So, the K eigenvectors provide a
K-dimensional representation of the n patterns by viewing the K
eigenvectors as K columns in a matrix. This matrix will be of size
n × K. Also these K eigenvectors are orthogonal to each other. So,
we can cluster the n rows (data points) into K clusters. In the above
example, by considering the first two eigenvectors as two columns in
a matrix we get the n two-dimensional patterns shown in Table 7.3.
By employing K-means algorithm on this data with a value of 2 for
K will give us the same clusters as we got earlier using only the
polarity of entries in the second eigenvector.
Spectral clustering gets its name from the word spectrum. The set
of all eigenvalues of a matrix is called its spectrum. The magnitude of
the maximum eigenvalue of the matrix is called the spectral radius.
Here, we have examined how clustering can be performed by using
the eigenvalues and eigenvectors of the matrix D which is obtained
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 236

236 Introduction to Pattern Recognition and Machine Learning

Table 7.3. Two-dimensional rep-

resentation of the six points.

(1, 1)
„ √ «
−3 + 17
1,
2

(1, 1)
„ √ «
3− 17
1,
2
„ √ «
−7 − 17
1,
2
„ √ «
3− 17
1,
2

from the weight matrix W and the similarity matrix S. It is possi-

ble to consider other variants of D to realize several other spectral
clustering algorithms.

2.4.3. Clustering based on frequent itemsets

Another recently studied direction for clustering is based on frequent
itemsets. The viability of this paradigm is because of efficient algo-
rithms available for mining frequent itemsets. Specifically we discuss
in this section a data structure called Frequent Pattern Tree (FP-
tree) which can be built using two dataset scans and this framework
is ideally suited for clustering large datasets. Further, frequencies of
different itemsets naturally link with the probability estimates based
on the popular maximum likelihood approach.
Typically frequent itemset mining is associated with transaction
datasets. It is possible to view binary patterns routinely as transac-
tions. For example, consider the dataset shown in Table 7.4. There are
six binary patterns labeled P1 to P6 ; each pattern is of size 3 × 3 and
is an example of character one. Some of them are affected by noise in
one out of nine bits. In a more practical setting such characters will
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 237

Data Clustering 237

Table 7.4. Six examples

of character 1.
001 001 001
001 101 001
001 001 101
P1 P2 P3

101 001 001

001 011 001
001 001 011
P4 P5 P6

Table 7.5. Transactions corresponding to the

six patterns.

TID i1 i2 i3 i4 i5 i6 i7 i8 i9

T1 0 0 1 0 0 1 0 0 1
T2 0 0 1 1 0 1 0 0 1
T3 0 0 1 0 0 1 1 0 1
T4 1 0 1 0 0 1 0 0 1
T5 0 0 1 0 1 1 0 0 1
T6 0 0 1 0 0 1 0 1 1

be of bigger sizes; they could be matrices of size 64 × 64 or 200 × 200.

Here, 3 × 3 size characters are used for illustrating the idea. Each
3 × 3 character may be viewed as a transaction based on 9 items by
considering row-major order; for example pattern P1 is represented
by the transaction: 0 0 1 0 0 1 0 0 1. It is possible to set up a
correspondence between the location of an element in the ith row
and jth column of the 3 × 3 matrix and the item number mij using
the row-major order as

mij = (i − 1) ∗ 3 + j for i, j = 1, 2, 3.

The corresponding representation of the six patterns as transac-

tions is given in Table 7.5. Here, T ID stands for Transaction Iden-
tifier and i1 to i9 stand for the nine items. Note that pattern Pi
is represented as transaction Ti for i = 1, . . . , 6. By scanning the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 238

238 Introduction to Pattern Recognition and Machine Learning

dataset once we can observe that the frequencies of the nine items in
the collection of patterns are

f (i1 ) = 1; f (i2 ) = 0; f (i3 ) = 6; f (i4 ) = 1; f (i5 ) = 1; f (i6 ) = 6;

f (i7 ) = 1; f (i8 ) = 1; f (i9 ) = 6.

If we consider items which have frequency more than 2, then we

get the items

i3 (f (i3 ) = 6); i6 (f (i6 ) = 6); i9 (f (i9 ) = 6);

so these are the frequent itemsets.

In order to see how the frequent itemsets are useful in clustering
we consider another character set shown in Table 7.6 corresponding
to character 7. Representing these six patterns in the form of trans-
actions we have the data shown in Table 7.7. The frequencies of the

Table 7.6. Six exam-

ples of character 7.

111 111 111

001 101 001
001 001 101
P7 P8 P9

111 111 111

101 011 001
011 001 011
P10 P11 P12

Table 7.7. Transactions corresponding to the

six patterns of 7.

TID i1 i2 i3 i4 i5 i6 i7 i8 i9

T7 1 1 1 0 0 1 0 0 1
T8 1 1 1 1 0 1 0 0 1
T9 1 1 1 0 0 1 1 0 1
T10 1 1 1 1 0 1 0 1 1
T11 1 1 1 0 1 1 0 0 1
T12 1 1 1 0 0 1 0 1 1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 239

Data Clustering 239

items corresponding to the transactions in Table 7.7 are given by:

f (i1 ) = 6; f (i2 ) = 6; f (i3 ) = 6; f (i4 ) = 2; f (i5 ) = 1; f (i6 ) = 6;
f (i7 ) = 1; f (i8 ) = 2; f (i9 ) = 6.
So, the frequent itemsets, based on a threshold of more than 2, are
i1 (f (i1 ) = 6); i2 (f (i2 ) = 6); i3 (f (i3 ) = 6); i6 (f (i6 ) = 6);
i9 (f (i9 ) = 6).
Using these frequent 1-itemsets which are obtained using a dataset
scan we can build the F P -tree shown in Figure 7.8. Note that in
Figure 7.8 the items are arranged in non-decreasing order; if two or
more items have the same frequency, then such items are placed in
a lexicographic order.
The F P -tree is constructed as follows:
• Scan the dataset once to obtain the frequent 1-itemsets. For
example, from the set of transactions shown in Table 7.5 the
frequent 1-itemsets are {i3 }, {i6 }, {i9 } and similarly in case of the

Root

i3
i 3: 12

i6 i 6: 12

i9
i 9: 12 C1

i2 i1 : 6

i1
i2 : 6 C2

Figure 7.8. FP-tree for the 12 transactions.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 240

240 Introduction to Pattern Recognition and Machine Learning

Table 7.8. Transactions based on

frequent items.

FTID Frequent itemset

F IT1 i3 , i6 , i9 
F IT2 i3 , i6 , i9 
F IT3 i3 , i6 , i9 
F IT4 i3 , i6 , i9 
F IT5 i3 , i6 , i9 
F IT6 i3 , i6 , i9 
F IT7 i3 , i6 , i9 , i1 , i2 
F IT8 i3 , i6 , i9 , i1 , i2 
F IT9 i3 , i6 , i9 , i1 , i2 
F IT10 i3 , i6 , i9 , i1 , i2 
F IT11 i3 , i6 , i9 , i1 , i2 
F IT12 i3 , i6 , i9 , i1 , i2 

transactions in Table 7.7 the frequent 1-itemsets are {i3 }, {i6 },

{i9 }, {i1 }, {i2 }.
• Arrange each transaction in the frequency order of the items in
it; ignore any infrequent items. For example, consider transaction
T4 in Table 7.5; it is {i1 , i3 , i6 , i9 }. Note that i1 is infrequent and
i3 , i6 , i9 are frequent with the same support value of 6; so, we
arrange these items in a lexicographic order. So, arranging the
transactions in Tables 7.5 and 7.7 we get the frequent itemset-
based transactions (FITs) shown in Table 7.8.
• The FITs are used as branches of the F P -tree. If the prefixes
of two or more F IT s are equal then they are used to represent
the same branch of the F P -tree and the frequencies of the items
in the prefix are appropriately incremented. For example in the
F IT s shown in Table 7.8 the prefix i3 , i6 , i9 has a frequency of
12 and the suffix i1 , i2 is present in 6 F IT s. Figure 7.9 shows the
intermediate F P -trees after inserting the first 6 F IT s.
Once the F P -tree is constructed, the two clusters, for the example
data shown in Figure 7.8, correspond to:
1. Cluster1 (C1 ): Items i3 , i6 , and i9 are frequent in patterns of
this cluster; this is indicated by a pointer from i9 to C1 in a
rectangular box.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 241

Data Clustering 241

Figure 7.9. FP-trees after adding 1 and 6 transactions.

2. Cluster2 (C2 ): Items i3 , i6 , i9 , i1 , i2 are frequent in the trans-

actions corresponding to this cluster; it is indicated by a pointer
from item i2 to C2 in a rectangular box.

Note that in this example C1 corresponds to class of ones and C2

corresponds to class of sevens.

3. Why Clustering?
Clustering is useful in several machine learning and data mining
tasks including data compression, outlier detection, and pat-
tern synthesis.

3.1. Data compression

In most of the real-world applications, clustering does not produce
the end product. Clustering may be viewed as a data compression
tool that generates a partition of the dataset and represents the clus-
ters using appropriate representatives/descriptions leading to data
compression. Compression is achieved because the set X of n data
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 242

242 Introduction to Pattern Recognition and Machine Learning

‘n’ Patterns Feature Pattern Grouping Cluster Descriptions

Selection/ based on
Extraction Representations Similarity
‘K’ Representatives

Decision
Making Decision
Input Data Point

Figure 7.10. Clustering in decision making.

points is reduced to a set of O(K) cluster representatives where

K < n and in big data applications we can even say that K n.
So, instead of using n data points, one can use O(K) cluster repre-
sentatives in decision making as shown in Figure 7.10. Classification,
regression, summarization and retrieval of documents are popular
decision making tasks that employ clustering.

3.2. Outlier detection

An outlier is a data point or a pattern that differs from a majority of
the patterns in the collection. For example, consider the set of two-
dimensional points shown in Figure 7.2. One of the points is away
from the rest; it has a value of a3 on the x coordinate and b2 on
the y coordinate. Even though it matches with several points in the
collection in terms of x or y values, it differs from the rest due to its
specific location in terms of the combination of its x and y values.
In general, an outlier can differ from each of the other points in one
or more feature values significantly. Density around a point is useful
to classify whether a point is an outlier or not; typically an outlier is
located in a sparse/low-density region.
Outliers could be either out-of-range or with-in range. In the out-
of-range case, the outlier will have value outside the legal domain of
one or more of the variables. Identifying out-of-range outliers could
be achieved easily by checking whether a feature value falls within the
range of the domain of the feature or not. On the contrary, outliers
falling within the range are not easily detected as they vary relative to
the other values in the respective group or cluster. For example, the
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 243

Data Clustering 243

outlier shown in Figure 7.2 is within the range. Typical data mining
schemes for outlier detection are based on clustering or density of
the data in the vicinity. Once the data is clustered, one needs to
examine small size clusters for possible outliers; typically singleton
clusters are highly likely to be containing outliers.

3.3. Pattern synthesis

One of the important problems in machine learning is to deal with
missing data. For example, consider the data shown in Table 7.1.
There are 10 patterns and each pattern is described using three fea-
tures. Note that the value of feature2 of pattern number 3 is 3.5.
Let us assume that this value is missing and we need to estimate
the missing value. We can use clustering to solve this problem as
follows. Cluster the 10 points using the remaining two features, that
is, feature1 and feature3 with the help of K-means algorithm.
In order to visualize the clusters, consider Figure 7.11 which
shows the 10 points in the two-dimensional space corresponding to
feature1 and feature3. By selecting patterns 1 and 2 in the two-
dimensional space, that is (10.0, 2.0)t and (63, 1.3)t , as the initial cen-
troids, K-means algorithm gives two clusters as shown in Figure 7.11

Cluster 1

X
XXX Cluster 2
XX
feature3

X
X X
X

feature1

Figure 7.11. Clustering points in the two-dimensional space.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 244

244 Introduction to Pattern Recognition and Machine Learning

with (10.3, 2.1)t and (72.1, 1.2)t as the centroids. One can see two
clusters in the collection: Cluster 1 and Cluster 2. Note that pattern
3 is in Cluster 1. Now a simple and good estimate for the missing
value of feature2 of pattern 3 is the sample mean of the values of fea-
ture2 of the remaining patterns in Cluster 1. The values of feature2
of patterns falling in Cluster 1 are: 3.5, 3.3, 3.3, 3.5, 3.3; the average
value is approximately 3.4. Even though the value of 3.4 is different
from the actual value 3.5; the mean value is closer to 3.5 and so this
simple scheme gives acceptable estimates.
In the case of missing values, we are synthesizing a part or esti-
mating the feature value of a pattern. However, there could be appli-
cations where the entire pattern has to be synthesized. Classification
based on a small set of training patterns requires such synthesis. This
is because the number of training patterns required increases with
the dimensionality of the dataset for a good classification.
Clustering could be used to synthesize; specifically cluster repre-
sentatives could be used to synthesize patterns as follows:

1. Obtain the clusters and their representatives; let the centroids of

the clusters be representatives as a special case.
2. Generate new patterns by perturbing one or more of the feature
values of the centroid by adding a fraction of a random value. This
value could be chosen randomly from the domain of the feature.

For example, consider the two clusters shown in Figure 7.11. The cen-
troid of Cluster 1 is (10.33, 2.15)t and that of Cluster 2 is (72.1, 1.2)t .
Now we can perturb these centroids to generate new patterns. For
example, by adding a small value of 0.2 to the value of feature1 and
−0.1 to the value of feature3, we obtain a new pattern (10.53, 2.05)t
from the first centroid. In a similar manner, we can generate patterns
of Cluster 2 by randomly perturbing the values of its centroid.
It is possible to combine patterns in a cluster to generate addi-
tional samples. An algorithm for this is:

1. Cluster the given dataset into K clusters.

2. Combine a pair of patterns in a cluster to generate a pair of
possible novel patterns; this combination could be done using
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 245

Data Clustering 245

crossover like operator. A simple crossover operator is the single-

point crossover. It may be explained as follows:
Let the crossover be applied on two vectors U = (u1 , u2 , . . . , ud )t
and V = (v1 , v2 , . . . , vd )t . We partition each vector into two parts;
the prefix part corresponding to the first p components and the
suffix part corresponding to the remaining d − p components. We
combine U and V to generate two possibly novel patterns X and
Y , where X is the d-dimensional vector obtained by concatenating
the prefix of U with the suffix of V . Similarly Y is obtained by
concatenating the prefix of V with the suffix of U .

We illustrate the pattern synthesis process using the follow-

ing example. Consider the patterns (10.0, 2.0)t and (10.4, 2.1)t from
Cluster 1 shown in Figure 7.11. Here, the value of d is 2. We can gen-
erate possible novel patterns as follows: take the value of feature1,
that is 10.0 (prefix) of the first pattern and the value of feature3, that
is 2.1 (suffix) of the second pattern to form a new pattern (10.0, 2.1)t
and similarly by taking the value of feature1 in the second pattern
and value of feature3 of the first pattern give us another novel pattern
(10.4, 2.0)t . A more illustrative example is provided using the data
given in Table 7.9. Let patterns numbered 1, 2, 3, 4 be given and let
patterns 1 and 2 belong to Cluster 1 and patterns 3 and 4 belong to
Cluster 2. Now by combining patterns 1 and 2 using features 1 and 2
to form the prefix and features 3 and 4 as the suffix, we get two more

Table 7.9. Pattern synthesis based on clustering.

Pattern Cluster
number Feature1 Feature2 Feature3 Feature4 number

1 1 1 1 1 1
2 2 2 2 2 1
3 6 6 6 6 2
4 7 7 7 7 2
5 1 1 2 2 1
6 2 2 1 1 1
7 6 6 7 7 2
8 7 7 6 6 2
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 246

246 Introduction to Pattern Recognition and Machine Learning

patterns in Cluster 1 which are shown in the table as patterns 5 and

6. Similarly patterns 7 and 8 belonging to Cluster 2 are obtained by
combining patterns 3 and 4, in Cluster 2, as shown in the table.

4. Clustering Labeled Data

Clustering is typically associated with grouping unlabeled patterns.
However, it is more practical to consider partitioning sets of labeled
samples. Also, typically clustering is not an end product in itself.
Clusters and their representatives are useful in further decision mak-
ing as depicted in Figure 7.10. Specifically labels associated with
the data points are useful in clustering for classification or in
knowledge-based clustering.

4.1. Clustering for classification

Classification is an important decision making activity that can ben-
efit from clustering. Specifically, clustering the training dataset can
improve the efficiency of classification algorithms.

4.1.1. Efficient nearest neighbor classifier

We illustrate how the NNC can be made more efficient by using
the cluster representatives rather than the original data using an
example. For this we consider a modified version of the dataset shown
in Table 7.1; the modification is in terms of providing the class label
of each pattern. The modified dataset is shown in Table 7.10. Note
that there are six patterns from one class (labeled 1) and four from
the other (labeled 2).
Let us consider a test pattern T = (70, 6.5, 1)t . Using the NNC we
need to compute the distance between T and each of the 10 training
patterns given in Table 7.10; so, we need to compute 10 distances
and find the nearest neighbor of T by locating the training pattern
that has the least distance value. In this example, pattern numbered
8 is the nearest neighbor with a squared Euclidean distance of 1.01.
So, T is assigned to class 2 as the class label of pattern 8 is 2. In this
almost trivial example, we require to compute 10 distances; however,
in large-scale applications involving billions or trillions of training
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 247

Data Clustering 247

Table 7.10. A training dataset.

Pattern Class
number feature1 feature2 feature3 label

1 10 3.5 2.0 1
2 63 5.4 1.3 2
3 10.4 3.5 2.1 1
4 10.3 3.3 2.0 1
5 73.5 5.8 1.2 2
6 81 6.1 1.3 2
7 10.4 3.3 2.3 1
8 71 6.4 1.0 2
9 10.4 3.5 2.3 1
10 10.5 3.3 2.1 1

patterns, the time taken to compute all the distances will be large.
We can reduce this effort by clustering the datasets of each class sep-
arately and use the prototypes or representatives of clusters instead
of the entire training data.
In this example, by clustering patterns in each class separately
using the K-means algorithm with K = 2, we get the following
clusters:
• Class 1:
– Cluster11 = {1}; Centroid11 = (10, 3.5, 2.0)t
– Cluster12 = {3, 4, 7, 9, 10}; Centroid12 = (10.4, 3.4, 2.2)t
• Class 2:
– Cluster1 = {2, 8}; Centroid21 = (67, 5.9, 1.2)t
– Cluster2 = {5, 6}; Centroid22 = (77.2, 6, 1.2)t
Now the cluster centroid nearest to T is Centroid21 which is at a
squared Euclidean distance of 9.4. So, we assign T to Class 2 as the
nearest cluster centroid is from class 2. Here, we need to compute
only four distances from the test pattern T as there are only four
centroids. Note that clustering of the training data needs to be done
only once and it can be done beforehand (offline). Also note that
clustering is done once and centroids of the clusters are obtained. The
same centroids could be used to classify any number of test patterns.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 248

248 Introduction to Pattern Recognition and Machine Learning

4.1.2. Cluster-based support vector machine

Clustering is useful not only in building efficient nearest neighbor
classifiers, but also in building efficient algorithms using a variety of
classifiers. Here, we consider how clustering can be useful in building
an efficient support vector machine based classifier; we consider a
two-class classification problem for the sake of simplicity. The specific
scheme employed is given below:

• Input: Set of positive examples, X+ and set of negative exam-

ples, X− .
• Construct CF-trees CF+ from X+ and CF− from X− .
• Use the centroids in the root nodes of the CF-trees to train an
SVM. Obtain the Support Centroids of this SVM.
• Expand these support centroids; expansion implies considering all
the patterns in the corresponding clusters. Add to this set patterns
obtained by expanding low margin clusters. By low margin clus-
ter Ci we mean a cluster which satisfies the following:

Di − Ri < Ds , (11)

where
Di = Distance from the centroid of Ci to the boundary of the SVM
obtained,
Ri = Radius of cluster Ci ,
Ds = Distance from support centroid to the boundary of the SVM.
• Obtain the SVM using these patterns that are accumulated.
• Repeat this expansion and obtaining SVM till no additional pat-
terns are accumulated.

We illustrate the role of clustering in training Linear SVM classifier

using the 11 two-dimensional patterns shown in Table 7.11.
The data is processed as follows:

• Patterns in each class are clustered separately. The clusters are:

N egative (−ve) Class: C1− = {(−1, 3)t , (1, 3)t }; C2− = {(2, 1)t };
C3− = {(−3, −2)t , (−1, −2)t }.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 249

Data Clustering 249

Table 7.11. Patterns from negative and posi-

tive classes.
Pattern no. Feature1 Feature2 Class

1 −1 3 −
2 1 3 −
3 2 1 −
4 4 8 +
5 4 10 +
6 5 5 +
7 6 3 +
8 −3 −2 −
9 −1 −2 −
10 10 4 +
11 10 6 +

The centroids of these clusters are: (0, 3)t , (2, 1)t , and (−2, −2)t
respectively.
P ositive (+ve) Class: C1+ = {(4, 8)t , (4, 10)t }; C2+ = {(5, 5)t ;
C3+ = {(6, 3)t }; C4+ = {(10, 4)t , (10, 6)t }.
The corresponding centroids respectively are: (4, 9)t , (5, 5)t , (6, 3)t ;
and (10, 5)t .
• Obtain the Linear SVM using the seven centroids. The support
centroids are (2, 1)t , (5, 5)t , (6, 3)t . Expanding them will not add
any more patterns as in this simple case each of these clusters is a
singleton cluster. The corresponding W and b of the SVM are:
 
2 1 t
W = , and b = −2.
5 5
• The distance of a point X = (x1 , x2 )t from the decision boundary
t X+b
is given by WW 
. So, the distance of support centroids (2, 1)t
√
from the decision boundary is 5 and similarly √ for the remaining
two support centroids also the distances are 5 each.
• For the remaining cluster centroids the distances are:
1. (0, 3)t : Distance is √7
5
2. (4, 9)t : Distance is √7
5
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 250

250 Introduction to Pattern Recognition and Machine Learning

16
3. (−2, −2)t : Distance is √ 5
√
4. (10, 5)t : Distance is 3 5.

Also note that the radius of each of these clusters is 1. By noting

that in the first
√ and second cases the inequality in (11) is satisfied
as √75 − 1 < 5, we need to expand both the clusters and add
the corresponding patterns to the expanded set of patterns. In the
remaining two cases (3 and 4) the inequality (11) is not satisfied.
So, these clusters are ignored.
• Now using the SVM on all these expanded set of patterns, that is
patterns numbered 1 to 7, we get the same W and b as obtained
earlier.

In this example there is no significant reduction in computation;

however in large-scale datasets several clusters that do not satisfy
the inequality (11) are typically present. Presence of a large number
of such clusters means a large number of patterns will be ignored in
obtaining the SVM and as a consequence there will be reduction in
the time required to train a linear SVM.

4.2. Knowledge-based clustering

One of the important results in clustering is that Unsupervised learn-
ing is not possible; it is argued through the Theorem of the ugly duck-
ling. It is based on the observation that the number of predicates
(features in binary form) shared by any two patterns is the same.
So, any two patterns are equally similar. So, similarity between a
duckling and a swan is the same as the similarity between a pair of
swans.
The theorem of the ugly duckling may be illustrated using the
two-dimensional dataset shown in Table 7.12. Here, we considered a
two-dimensional binary data; so there are four possible object types
as shown in the table. The two binary features (or equivalently pred-
icates) are f1 and f2 . Note that any data can be represented in a
binary form; in fact a digital computer processes data in a binary
form at the basic hardware level.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 251

Data Clustering 251

Table 7.12. Four object types

in two-dimensional space.

Pattern f1 f2

P1 0 0
P2 0 1
P3 1 1
P4 1 0

Table 7.13. Four objects using eight boolean functions.

f1 ∧ f2 f1
Pattern f1 f2 f 1 ∧ f2 f2 (g2 ) (g1 ) f1 ∨ f2 f1 ∨ f 2

P1 0 0 0 1 0 1 1 1
P2 0 1 0 0 1 1 0 1
P3 1 1 0 0 0 0 1 1

For the sake of simplicity we consider the first three object types.
Now considering all possible boolean functions we have the data
shown in Table 7.13. If we had considered all the four object types
we would have got 16 boolean functions; instead we considered only
three types to have only eight boolean functions. We have chosen the
first three types; it will lead to a similar argument ultimately even if
we consider any other three types.
In Table 7.13, we have considered all possible boolean functions
which are eight in this case. In this representation, between any pair
of patterns exactly four predicates (boolean functions) differ. So,
distance or similarity based on this matching between any pair of
patterns is the same. One may argue that f1 and f2 are primitive
because they are given; others are derived. However, it is possible to
argue that g1 = f1 (where f1 is the negation of f1 ) and g2 = f1 ∧ f2
can be considered as primitive and f1 = g1 and f2 = g1 ∨ g2 means
we can derive f1 and f2 from g1 and g2 . So, it is not possible to fix
some as more primitive than others. Further, for the machine it does
not matter which is more primitive. This means we need to consider
all possible predicates (boolean functions) and the discrimination is
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 252

252 Introduction to Pattern Recognition and Machine Learning

lost. Such an argument leads to the conclusion that it requires extra-

logical evidence (or knowledge) to prioritize two or more predicates
over the others.
So, it can be that clustering cannot be done without using some
knowledge. Such knowledge can be used in one or more of (a) Rep-
resentation, (b) Similarity computation, or (c) Grouping phases. We
consider each of these below:

• Representation: If labeled data is clustered, then the class labels

of the patterns can be used to select a subset of features for repre-
sentation using mutual information. For example if the class labels
are tall and short then height is more important than weight. So,
height will have a higher mutual information with the class labels.
Similarly when the patterns are not labeled, then it is possible
to represent them based on the domain knowledge. For example,
Yahoo! uses ontologies built either manually or automatically to
group documents. Similarly documents can be represented using
Wikipedia article titles and categories. Specifically knowledge from
large document collections is used to select appropriate terms in
representing the documents.
For example by ranking terms based on their frequency of
occurrence in the collection it is observed that

C
fi = , (12)
ri

where fi is the frequency of a term in the collection; C is a con-

stant and ri is the rank of the term. Note that high frequency
terms have low ranks. Typically terms that are either too frequent
or too infrequent are not good for discrimination. So, terms are
weighed based on tf − idf values. A term like the or is occurs in
almost every document in the collection which makes the inverse
document frequency (idf ) very small or close to zero. On the other
hand terms that are infrequent (or have term frequency (tf ) close
to zero) will also have a low tf − idf value. In some sense this
amounts to representing documents using terms that are neither
frequent nor rare.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 253

Data Clustering 253

• Similarity Computation: Conventionally proximity between

two patterns Xi and Xj is defined as

Proximity(Xi , Xj ) = f (Xi , Xj ),

where f is a function of the two patterns Xi and Xj . Popular

distance measures like Euclidean distance, City-block distance and
similarity functions like the cosine of the angle between Xi and Xj
are examples of this kind of functions. A more generic class of
functions is the contextual similarity functions given by

Contextual − Proximity(Xi , Xj ) = g(Xi , Xj , Kcontext ).

Here, the similarity/distance between Xi and Xj is not only a

function of these two patterns but it depends on the contextual
knowledge Kcontext . Such contextual knowledge could be expressed
in terms of other points in the vicinity of Xi and Xj . Let us con-
sider the following example using the terms:
– N (Xi ) = Set of neighbors of Xi = {Xi1 , Xi2 , . . . , Xili }. Neighbor-
hood here could be characterized in different ways. For example,
1. We may have li = K for all i which means the K nearest
neighbors are considered.
2. We may consider neighbors of Xi falling within a sphere of
radius ri around Xi .
3. In a graph or a social network N (Xi ) could be the set of all
the nodes connected to node Xi .
– CN (Xi , Xj ) = |N (Xi ) ∩ N (Xj )| where CN indicates Common
Neighbors. This is a similarity value and it is used in link pre-
diction in social networks.
One of the most generalized similarity functions is based on both
the context and also domain knowledge. An instantiation of this
kind of similarity is Knowledge-Based Proximity (KB Proximity)
given by

KB − Proximity(Xi , Xj ) = h(Xi , Xj , Kcontext , Kdomain ),

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 254

254 Introduction to Pattern Recognition and Machine Learning

where Kdomain is the domain knowledge. We can illustrate such

a similarity function using a simple example using common salt
(sodium chloride), pepper, and potassium cyanide. For these three
objects:
– In a restaurant we group these objects into two clusters:
C1 = {salt , pepper }, C2 = {potassium cyanide}.
Such a partitioning is based on using knowledge in the form of
concepts edible (C1 ) and inedible (C2 ) which are relevant in the
context of a restaurant.
– In a chemistry laboratory the partitioning will be:
C1 = {salt , potassium cyanide}, C2 = {pepper }.
This partitioning is based on using concepts inorganic (C1 ) and
organic (C2 ).
One of the applications in which such knowledge, both contextual
and domain knowledge, is used is text mining. For example, in
finding similarity between two books the context might be pro-
vided by Amazon in terms of a set of books bought along with
the given books. In addition domain knowledge in the form of
Dewey Decimal Classification or Wikipedia is useful in estimating
the similarity better by looking at the semantic or topic matching
between books.
• Grouping Phase: Knowledge can be used in the grouping phase
in different ways. Some of them are:
1. It is possible to fix the number of clusters K in the K-means
type of algorithms.
2. It is possible to fix a threshold value to examine whether a
pattern belongs to a cluster or not based on domain knowledge;
this scheme is useful in algorithms like Leader and BIRCH.
3. In the single link algorithm a MST of the data is formed and
edges with large weights are deleted from the MST to from clus-
ters. Here also one can use the knowledge in terms of the edge
weights to automatically stop partitioning further or equiva-
lently in deleting edges further.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 255

Data Clustering 255

4. It is possible to use knowledge in a variety of forms: in fixing

the prior probabilities; in characterizing entropy, etc.

5. Combination of Clusterings
One of the more recent trends in clustering is to obtain a single
partition by using multiple partitions of the data. Here, a generic
framework is:

1. Obtain l(>1) partitions of the dataset X by using different algo-

rithms; it may be possible to run the same algorithm with dif-
ferent parameter/initial partition settings. Let these partitions be
π1 , π2 , . . . , πl .
2. For each pair of patterns Xi and Xj in X count in how many,
of the l partitions, this pair is assigned to the same cluster; let
this count be sij standing for the similarity between Xi and Xj .
Store the similarity values, for all possible pairs, in a matrix of
size n × n where the n is the size of X and sij is the ijth entry in
the matrix S.
3. Use the SLA along with the similarity matrix S to cluster the
n points into the required number of clusters.

We illustrate this approach using a two-dimensional dataset. Con-

sider the dataset shown in Table 7.14 and in Figure 7.12.

Table 7.14. Two chain-like

clusters.
Pattern f1 f2

P1 1 1
P2 2 2
P3 3 3
P4 4 4
P5 5 5
P6 5 1
P7 6 2
P8 7 3
P9 8 4
P10 9 5
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 256

256 Introduction to Pattern Recognition and Machine Learning

X X

X X

X X

X X

X X

Figure 7.12. An example dataset having two clusters.

Table 7.15. Seven partitions of the dataset.

K Initial centroids Partition

2 (1,1),(9,5) {{(1,1),(2,2),(3,3),(4,4),(5,1)}
{(5,5),(6,2),(7,3),(8,4),(9,5)}}
2 (3,3),(7,3) {{(1,1),(2,2),(3,3),(4,4),(5,5)}
{(5,1),(6,2),(7,3),(8,4),(9,5)}}
2 (4,4),(6,2) {{(1,1),(2,2),(3,3),(4,4),(5,5)}
{(5,1),(6,2),(7,3),(8,4),(9,5)}}
2 (1,1),(5,1) {{(1,1),(2,2),(3,3)},
{(4,4),(5,5),(5,1),(6,2),(7,3),(8,4),(9,5)}}
4 (1,1),(5,1),(5,5),(9,5) {{(1,1),(2,2),(3,3)},{(4,4),(5,5)}
{(5,1),(6,2),(7,3)},{(8,4),(9,5)}}
3 (1,1),(6,2),(9,5) {{(1,1),(2,2),(3,3)},{(5,1),(6,2),(4,4),(5,5),(7,3)},
{(8,4),(9,5)}}
3 (3,3),(6,2),(9,5) {{(1,1),(2,2),(3,3),(4,4),(5,5)},{(5,1),(6,2),(7,3)},
{(8,4),(9,5)}}

Let us consider K-means algorithm with different values of K and

different initial centroids for the same value of K. We consider seven
such partitions shown in Table 7.15. Once we have obtained these
partitions we can form the similarity matrix based on the number of
times a pair of patterns are in the same cluster in these partitions.
Note that (1,1) and (2,2) are in the same cluster in all the seven
partitions; however (1,1) and (5,5) are in the same cluster in only
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 257

Data Clustering 257

Table 7.16. The similarity matrix.

Pattern (1,1) (2,2) (3,3) (4,4) (5,5) (5,1) (6,2) (7,3) (8,4) (9,5)

(1,1) 7 7 7 4 3 1 0 0 0 0
(2,2) 7 7 7 4 3 1 0 0 0 0
(3,3) 7 7 7 4 3 1 0 0 0 0
(4,4) 4 4 4 7 6 3 2 2 1 1
(5,5) 3 3 3 6 7 2 2 2 1 1
(5,1) 1 1 1 3 2 7 6 6 4 4
(6,2) 0 0 0 2 2 6 7 7 5 5
(7,3) 0 0 0 2 2 6 7 7 4 4
(8,4) 0 0 0 1 1 4 5 4 7 7
(9,5) 0 0 0 1 1 4 5 4 7 7

three out of the seven partitions. Using these counts we compute Sij
for all possible pairs and show the resultant matrix S in Table 7.16.
Using the SLA we get a two-partition based on the following steps:

• Merge pairs of points and form clusters based on the largest simi-
larity value of 7 between each pair of points. The clusters are:

{(1, 1), (2, 2), (3, 3)}, {(6, 2), (7, 3)}, {(8, 4), (9, 5)}, {(5, 1)},
{(4, 4)}, {(5, 5)}.

• Now merge pairs of clusters based on the next largest similarity

value which is 6 here. The clusters are:

{(1, 1), (2, 2), (3, 3)}, {(5, 1), (6, 2), (7, 3)}, {(8, 4), (9, 5)},
{(4, 4), (5, 5)}.

• Now consider the similarity value of 5 to merge clusters further.

The resulting clusters are:

{(1, 1), (2, 2), (3, 3)}, {(5, 1), (6, 2), (7, 3), (8, 4), (9, 5)},
{(4, 4), (5, 5)}.

• Finally by considering the similarity value of 4 to merge clusters

we get:

{(1, 1), (2, 2), (3, 3), (4, 4), (5, 5)}, {(5, 1), (6, 2), (7, 3), (8, 4), (9, 5)}.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 258

258 Introduction to Pattern Recognition and Machine Learning

Now we have two clusters. So, we stop here. Note that the base
algorithm used here in the form of K-means cannot generate the
resulting clusters using a single application because clusters in
Figure 7.12 are chain like clusters.

Research Ideas
1. In Section 1, the number of hard partitions of a set of n patterns into K clusters
is discussed. How do we control the number of such partitions? Do divide-
and-conquer based algorithms help?

Relevant References
(a) M. R. Anderberg, Cluster Analysis for Applications. New York: Academic
Press, 1973.
(b) M. N. Murty and G. Krishna, A computationally efficient technique for
data-clustering. Pattern Recognition, 12(3):153–158, 1980.
(c) S. Guha, A. Meyerson, N. Mishra, R. Motwani and L. O. Callaghan, Clus-
tering data streams: Theory and practice. IEEE Transactions on Knowledge
and Data Engineering, 15(3):515–528, 2003.
(d) C.-J. Hseieh, S. Si and I. Dhillon, A divide-and-conquer solver for Kernel
support vector machines. In Proceedings of ICML, 2014.
2. In addition to divide-and-conquer which other approaches help in reducing
the number of partitions being considered? Is it good to consider incremental
algorithms?

Relevant References
(a) H. Spath, Cluster Analysis Algorithms for Data Reduction and Classifica-
tion of Objects. London: E. Horwood, 1980.
(b) T. Zhang, R. Ramakrishnan and M. Livny, BIRCH: An efficient data clus-
tering method for very large databases. Proceedings of SIGMOD, 1996.
(c) V. Garg, Y. Narahari and M. N. Murty, Novel biobjective clustering (BIGC)
based on cooperative game theory. IEEE Transactions on Knowledge and
Data Engineering, 25(5):1070–1082, 2013.
3. Incremental algorithms are order-dependent. Which properties does an incre-
mental algorithm needs to satisfy so as to be order-independent?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 259

Data Clustering 259

Relevant References
(a) B. Shekar, M. N. Murty and G. Krishna, Structural aspects of semantic-
directed clusters. Pattern Recognition, 22(1):65–74, 1989.
(b) L. Rokach and O. Maimon, Clustering methods. In Data Mining and
Knowledge Discovery Handbook, O. Z. Maimon and L. Rokach (eds.).
New York: Springer, 2006.
4. Is it possible to characterize the order-dependence property of the Leader
algorithm as follows?

Conjecture 1. The Leader algorithm is order-dependent if and only

if there exist three points Xi , Xj , and Xk such that d(Xi , Xj ) < T ,
d(Xj , Xk ) < T , and d(Xi , Xk ) > T where T is the distance threshold
specified.

5. By specifying an axiomatic framework there are results on impossibility of

clustering. How do we suitably modify such axiomatic frameworks to make
clustering possible but by considering a small number of partitions?

Relevant References
(a) B. Shekar, M. N. Murty and G. Krishna, A knowledge-based clustering
scheme. Pattern Recognition Letters, 5(4):253–259, 1987.
(b) J. Kleinberg, An impossibility theorem for clustering. Proceedings of
NIPS, 2002.
(c) S. Ben-David and M. Ackerman, Measures of clustering quality: A work-
ing set of axioms for clustering. Proceedings of NIPS, 2008.
6. In Section 2, both partitional and hierarchical clustering algorithms are con-
sidered. How does one hybridize these approaches?

Relevant References
(a) M. N. Murty and G. Krishna, A hybrid clustering procedure for concentric
and chain-like clusters. International Journal of Parallel Programming,
10(6):397–412, 1981.
(b) S. Zhong and J. Ghosh, A unified framework for model-based clustering.
Journal of Machine Learning Research, 4:1001–1037, 2003.
(c) L. Kankanala and M. N. Murty, Hybrid approaches for clustering. Pro-
ceedings of PREMI, LNCS 4815:25–32, 2007.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 260

260 Introduction to Pattern Recognition and Machine Learning

7. In spectral clustering it is assumed that the data is presented in the form of

a graph. Is there a major difference between converting high-dimensional
datasets into similarity/distance weighted graphs and using graphs like the
Facebook social network or www?

Relevant References
(a) N. Mishra, R. Schreiber, I. Stanton and R. E. Tarjan, Clustering social
networks. Proceedings of WAW, LNCS 4863:56–67, 2007.
(b) U. Luxburg, A tutorial on spectral clustering. Statistics and Computing,
17(4):395–416, 2007.
(c) M. C. V. Nascimento and A. C. P. L. F. de Carvalho, Spectral methods
for graph clustering A survey. European Journal of Operations Research,
211:221–231, 2011.
8. Frequent itemsets have been used successfully in both classification and clus-
tering. What is the reason for frequent itemsets to be useful in clustering?

Relevant References
(a) S. Mimaroglu and D. A. Simovic, Clustering and approximate identifica-
tion of frequent item sets. Proceedings of FLAIRS, 2007.
(b) H. Cheng, X. Yan, J. Han and P. S. Yu, Direct discriminative pattern mining
for effective classification. Proceedings of ICDE, 2008.
(c) G. V. R. Kiran and V. Pudi, Frequent itemset based hierarchical document
clustering using Wikipedia as external knowledge. LNCS, 6277:11–20,
2010.
(d) A. Kiraly, A. Gyenesei and J. Abonyi, Bit-table based biclustering and
frequent closed itemset mining in high-dimensional binary data. The Sci-
entific World Journal, 2014. http://dx.doi.org/10.1155/2014/870406.
9. Clustering is a data compression tool. How to exploit this feature? Are there
better schemes for compression? Can we cluster compressed data?

Relevant References
(a) R. Cilibrasi and P. M. B. Vitányi, Clustering by compression. IEEE Trans.
on Information Theory, 51(4):1523–1545, 2005.
(b) T. R. Babu, M. N. Murty and S. V. Subrahmanya, Compression Schemes
for Mining Large Datasets: A Machine Learning Perspective. New York:
Springer, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 261

Data Clustering 261

(c) A. Schmieder, H. Cheng and X. Li, A study of clustering algorithms and

validity for lossy image set compression. Proceedings of IPCV, 2009.
(d) J. Han, M. Kamber and J. Pei, Data Mining: Concepts and Techniques.
New York: Morgan Kaufmann, 2011.
10. How do we use clustering in synthesizing patterns?

Relevant References
(a) P. Viswanath, M. N. Murty and S. Bhatnagar, Pattern synthesis for non-
parametric pattern recognition. Encyclopedia of Data Warehousing and
Mining: 1511–1516, 2009.
(b) M. Agrawal, N. Gupta, R. Shreelekshmi and M. N. Murty, Efficient
pattern synthesis for nearest neighbour classifier. Pattern Recognition,
38(11):2200–2203, 2005.
(c) H. Seetha, R. Saravanan and M. N. Murty, Pattern synthesis using multiple
Kernel learning for efficient SVM classification. Cybernetics and Infor-
mation Technologies, 12:77–94, 2012.
11. Clustering is usually associated with grouping unlabeled patterns. What is the
advantage in clustering labeled patterns?

Relevant References
(a) V. Sridhar and M. N. Murty, A knowledge-based clustering algorithm.
Pattern Recognition Letters, 12(8):511–517, 1991.
(b) M. Grbovic, N. Djuric, S. Guo and S. Vucetic, Supervised clustering of
label ranking data using label preference information. Machine Learning,
93(2–3):191–225, 2013.
12. How to incorporate knowledge from multiple sources to perform clustering
better?

Relevant References
(a) M. N. Murty and A. K. Jain, Knowledge-based clustering scheme for
collection management and retrieval of library books. Pattern Recognition,
28(8):949–963, 1995.
(b) X. Hu, X. Zhang, C. Lu, E. K. Park and X. Zhou, Exploiting Wikipedia as
external knowledge for document clustering. Proceedings of KDD, 2009.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch07 page 262

262 Introduction to Pattern Recognition and Machine Learning

13. How do you perform ensemble clustering in an optimal way?

Relevant References
(a) A. K. Jain, Data clustering: 50 years beyond K-means. Pattern recognition
Letters, 31(8):651–666, 2010.
(b) S. Vega-Pons and J. Ruiz-Shulcloper, A survey of clustering ensemble
algorithms. International Journal of Pattern Recognition and Artificial
Intelligence, 25(3):337–372, 2011.
(c) T. R. Babu, M. N. Murty and V. K. Agrawal, Adaptive boosting with leader
based learners for classification of large handwritten data. Proceedings of
HIS, 2004.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 263

Chapter 8

Soft Clustering

Clustering has been one of the most popular tools in data mining.
Even though hard clustering has been popularly studied and tra-
ditionally used, there are several important applications where soft
partitioning is essential. Some of these applications include text min-
ing and social networks. Soft partitioning is concerned with assigning
a document in text mining or a person in a social network to more
than one cluster. For example, the same document may belong to
both sports and politics. In several cases softness has to be appro-
priately interpreted to make a hard decision; such a process permits
us to delay the decision making. So, one of the major characteristics
of softness is in delaying decision making as far as possible. Another
notion is to acknowledge the assignment of a pattern to more than
one category.
We can work out the number of soft partitions of n patterns into
K clusters as follows:

• Let X1 , X2 , . . . , Xn be the n patterns and C1 , C2 , . . . , CK be the

K clusters. In soft clustering, a pattern may belong to one or more
clusters. So, if pattern Xi belongs to cluster Cj then we record a
1 as the ijth element of cluster indicator matrix (CIM ) otherwise
we store a 0.
• Note that the number of 1s in the ith row of CIM can vary between
1 to K. This can be chosen in 2K − 1 ways by ruling out the all
zero vector because Xi has to be assigned to at least one of the
K clusters.

263
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 264

264 Introduction to Pattern Recognition and Machine Learning

• So for each row in CIM we have 2K − 1 choices; for all the n rows
the number of possibilities is (2K − 1)n which is of O(2Kn ). This
is an upper bound because no column in CIM can be empty.
• Instead of storing a 1 or a 0 if we store one of P possible val-
ues to indicate the extent to which Xi belongs to Cj , then the
number of possibilities is bounded by (P K − 1)n as the number
of distinct values each entry in CIM can assume is P . In some of
the soft clustering algorithms the value of P could be very large;
theoretically it could be infinite.

1. Soft Clustering Paradigms

There are a variety of soft clustering paradigms. Each of them has a
distinct flavor. Some of the important ones are:

1. Fuzzy Clustering: Fuzzy clustering and related possibilistic

clustering is the most popularly established and studied soft clus-
tering paradigm. It is so popular that routinely soft clustering
is interpreted as fuzzy clustering; again fuzzy c-means clustering
algorithm is the most frequently considered algorithm here. Each
pattern Xi may be assigned to a cluster Cj with a membership
value µij which indicates the degree to which Xi belongs to Cj .


It is assumed that µij ∈ [0, 1] and K j=1 µij = 1.
2. Rough Clustering: When there is uncertainty in assigning pat-
terns to clusters which means it is difficult to specify clusters
as sets, then we can use rough sets to characterize the uncer-
tainty. There could be patterns that clearly belong to only one
cluster; other patterns may belong to two or more clusters. This
is abstracted by a rough set which is represented using two sets;
these sets are called lower approximation and upper approxima-
tion. A pattern can belong to at most one lower approximation.
If a pattern does not belong to any lower approximation then it
belongs to two or more upper approximations.
3. Evolutionary Algorithms for Clustering: Evolutionary
approaches including genetic algorithms (GAs), evolutionary
schemes, and evolutionary programming are three popular tools
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 265

Soft Clustering 265

used in both hard and soft clusterings. Unlike the other

approaches, here a collection of possible solutions (population) is
simultaneously processed leading to a more effective scheme for
pursuing important solution directions. GAs have been shown to
lead to optimal hard clustering based on squared error criterion
under some acceptable conditions.
4. Statistical Clustering: Here each pattern is viewed as a vector
in the multi-dimensional space. So clusters are viewed as regions of
points in the space. Hence, this approach is also based on geomet-
ric or vector space models. Typically geometric models are popular
in computational geometry, graphics and visualization. Similarly
vector space model is the most popular in information retrieval.
Because regions can overlap or vectors can be associated with
multiple regions, statistical models can be potentially soft.
Popular hard clustering algorithms like the K-means algo-
rithm can be modified to be soft. There are several soft versions of
the K-means algorithm which will be discussed in later sections.
Leader algorithm is one of the most efficient hard clustering algo-
rithms and it can be modified to perform soft clustering. Because
of the similarity between leader and BIRCH it is possible to have
a soft version BIRCH also.
5. Neural Networks for Clustering: artificial neural networks
(ANNs) have been popularly used in clustering; self organizing
map (SOM) is the most popular of them. Typically ANNs employ
sigmoid type of activation functions which return positive real
values in the range 0 to 1. So, based on the output values returned
ANNs also can be used for soft clustering. ANNs may be viewed
as statistical clustering tools. A major restriction of the ANNs
is that they can deal with only numerical vectors. It is possible
to view support vector machine (SVM) as the most popular and
state-of-the-art ANN.
6. Probabilistic Clustering: Conventionally vector space mod-
els and probabilistic models are treated as different in the area
of information retrieval. Further, probabilistic models are typ-
ically viewed as generative models; once we have the model
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 266

266 Introduction to Pattern Recognition and Machine Learning

we can generate patterns of the corresponding cluster. In fact

probabilistic models may be viewed as theoretical and their imple-
mentations are based on statistics. For example, the Bayesian
models are inherently probabilistic and without knowing the
prior probabilities and likelihood values it is not possible to
use them in practice; here statistics provides appropriate imple-
mentable approximations. This is achieved by employing esti-
mation schemes. Probabilistic models are inherently soft. For
example, while using a Bayesian model in a two class problem, it
is possible that none of the two posterior probabilities is zero; in
such a case the pattern belongs to both the classes but to different
degrees based on the posteriors.
Expectation maximization (EM) is a well-known probabilistic
clustering algorithm. It may be viewed as a probabilistic variant
of the K-means algorithm. It is the most popular soft cluster-
ing algorithm in the history of pattern recognition and machine
learning. Some of the other examples include probabilistic latent
semantic indexing (PLSI ) and latent dirichlet allocation (LDA)
which are popular in document clustering.

Among these, clustering based on fuzzy sets, rough sets, evolution-

ary algorithms, and neural networks are traditionally identified with
the soft computing paradigm. Even though there is a significant
amount of work done in these areas and reported in a variety of
important journals and conference proceedings, statistical and prob-
abilistic paradigms have gained a lot of prominence recently; they
are fashionable currently. Because of the increasing interest in statis-
tical learning theory, soft clustering based on EM and its variants is
arguably the most popular now. We discuss each of the soft clustering
paradigms in detail in the upcoming sections.

2. Fuzzy Clustering
Typically each cluster is abstracted using one or more representa-
tive patterns. For example, centroid and leader are popularly used
representatives of a cluster. Let Ri be the set of representatives of
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 267

Soft Clustering 267

a cluster Ci . Then a pattern X is assigned to one or more clusters

based on the following:

• In hard clustering we assign X to cluster Ci if Similarity(X, Ri ) >

Similarity(X, Rj ) for all j = i or equivalently distance(X, Ri ) <
distance(X, Rj ) for all j = i. It requires some appropriate def-
inition of similarity and distance. In case of having a single
representative for each cluster, for example the centroid of the
cluster, Euclidean distance between X and Ri is popularly used to
characterize proximity where Ri is the centroid of the ith cluster.
Here the winner-take-all strategy is used to realize the hard clus-
tering; this is based on assigning X to only one cluster (winner
cluster gets the pattern) based on minimum distance or maximum
similarity between X and the cluster representative.
• An alternative to winner-take-all is to assign X to more than one
cluster based on the proximity between X and each Ri . Similarity
between X and Ri is used to characterize the membership of X
in Ci . This is typically used in possibilistic clustering; the similarity
values are suitably normalized so that they add up to one in the
case of fuzzy clustering.

Both hard K-means and Leader clustering algorithms can be suitably

modified to realize their fuzzy counterparts; among them the Fuzzy
K-means algorithm is more popular which we describe next.

2.1. Fuzzy K-means algorithm

It goes through an iterative scheme like the K-means algorithm.
Starting with some initial assignment of points, it goes through the
following steps:

1. Obtain Membership Values:

Xj −Ri
−2/(M −1)
µij = PK −2/(M −1)
.
l=1 (
Xj −Rl
)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 268

268 Introduction to Pattern Recognition and Machine Learning

2. Compute the Fuzzy Cluster Centroids:

Pn M
j=1 (µij ) Xj
P
Centroidi = Ri = n M ,
j=1 (µij )

where

• µij = Membership value of pattern Xj in cluster Ci for j =

1, 2, . . . , n and i = 1, 2, . . . , K.
• Xj − Ri  typically is the Euclidean distance between Xj
and Ri .
• Ri = Centroid of the ith cluster.
• M is the fuzzifying constant. If M = 1 then the Fuzzy algo-
rithm works like the hard K-means algorithm. This is because
µij = 1 when Xj is assigned to Ci ; note that in the expression
for µij , the exponent of the distance between Xj and Rl in the
denominator will be tending to −∞ for each l. So, the small-
est of them will be selected which is equal to the numerator
(Xj − Ri −2/(M −1) ) and hence µij = 1. If the value of M is
large or tends to ∞ then µij = K1 as the exponents in both the
numerator and denominator tend to 0.

3. The above two terms are updated iteratively by the Fuzzy

K-means algorithm; assign each pattern to its nearest fuzzy cen-
troid and update the µij and Ri values. The algorithm stops when
there is no significant change in these values computed over two
successive iterations.

It is possible to show that the Fuzzy K-means algorithm minimizes

a variant of the squared error given by


K 
n
(µij )M Xj − Ri 2 .
i=1 i=1

Both the hard and fuzzy K-means algorithms can be guaranteed

to reach the locally optimal solution of their respective objective
functions. A similar form of softness can be extended to realize fuzzy
versions of other hard clustering algorithms.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 269

Soft Clustering 269

Figure 8.1. Examples of patterns from two classes. (a) A Collection of Ones.
(b) A Collection of Sevens.

3. Rough Clustering
A rough set is characterized using the notion of the indiscernabi-
lity relation that is defined based on equivalence classes of objects.
Here, some patterns are known to definitely belong to a cluster.
For example consider the patterns shown in Figure 8.1. There are
two classes of character patterns in the figure; some are of charac-
ter 1 (shown in Figure 8.1(a)) and others are of character 7 (shown
in Figure 8.1(b)). There are four patterns that definitely belong to
1 and 4 patterns that definitely belong to 7. In the case of ones there
are three equivalence classes bounded by circles; characters in each
circle are similar to each other. Elements in the first two equivalence
classes (considered from left to right) shown in Figure 8.1(a) defi-
nitely belong to class of 1 s. Similarly in Figure 8.1(b) there are three
equivalence classes of sevens and these are bounded by rectangles
in the figure; all of them definitely belong to class of 7 s. However,
there is one equivalence class (third) in Figure 8.1(a) where the pat-
terns are either ones or sevens; these two patterns are possibly ones
or sevens.
Patterns in these equivalence classes are indiscernible; note that
1 s class consists of union of the two indiscernible equivalence classes
and similarly all the three equivalence classes in Figure 8.1(b) are also
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 270

270 Introduction to Pattern Recognition and Machine Learning

indiscernible and their union is a subset of class of 7 s. The remaining

equivalence class which has patterns that are possibly ones or sevens
can be subset of both the classes. In a formal setting indiscernability
is defined as follows:

Two patterns Xi and Xj are related or indiscernible if Xil = Xjl for

all l in a subset of features where Xil is the lth feature value of Xi .

It is possible to show that such a relation is an equivalence relation

and it partitions the set of patterns X into equivalence classes. In
this set up two patterns in any equivalence class are identical on a
chosen subset of features. It is possible to relax by insisting that the
two patterns have values falling in an interval, for a feature, than the
values being identical. In this setting the notion of similarity based
on overlapping intervals instead of values could be exploited to form
the equivalence classes.
Once we have the equivalence classes based on similarity on the
set of features F we can define rough set using the lower approxi-
mation (F S) and the upper approximation (F S) where S ⊆ X as
follows:

F S = ∪ i Ei , where Ei ⊆ S, (1)
F S = ∪i Ei , where Ei ∩ S = φ. (2)

Note that in the lower approximation we insist that each equivalence

be completely (or definitely) in S whereas in the upper approximation
we consider equivalence classes that partially overlap with S. Hence,
the lower approximation contains patterns that definitely belong to S
and the upper approximation has patterns that may possibly belong
to S. Additionally, we have:

• Boundary Region of S is given by F S −F S. If the boundary region

has no elements then there is no roughness in S with respect to F .
• Lower approximation may be viewed as the core part of a cluster.
Lower approximations of two clusters do not overlap. So, if a pat-
tern X (definitely) belongs to the lower approximation of a rough
cluster then it will not be present in any other cluster.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 271

Soft Clustering 271

• A lower approximation of a cluster is a subset of its upper approx-

imation.
• If a pattern X belongs to the boundary region of a cluster, then
it belongs to at least one more upper approximations.

We use these properties in designing rough clustering algorithms; a

popular algorithm for rough clustering is the Rough K-means algo-
rithm which we discuss in the next subsection.

3.1. Rough K-means algorithm

Rough K-means algorithm is the most popular among the rough
clustering algorithms. Basically we have to characterize assignment
of a pattern to a rough cluster and computation of the centroid of a
rough cluster. We discuss these steps below:

1. Assigning a Pattern: A pattern X is assigned to one or more

clusters. This may be done as follows:
(a) Initialize: Let C1 , C2 , . . . , CK be the K clusters with
centroids R1 , R2 , . . . , RK respectively. These centroids are
obtained based on some initial assignment of patterns to
clusters.
(b) Obtain the Nearest Centroid: Obtain the nearest centroid
Rj of pattern X based on

min
d(X, Rj ) = d(X, Ri ).
1≤i≤K

(c) Obtain Other Nearer Centroids: Choose an  and obtain

all the clusters i (= j) such that d(X, Ri ) ≤ (1 + )d(X, Rj ).
(d) Assign to Multiple Clusters: If there are q(2 ≤ q ≤
K − 1) such i’s given by i1 , i2 , . . . , iq , then assign X to
F Ci1 , F Ci2 , . . . , F Ciq .
(e) Assign to a Single Cluster j : If there is no such i then
assign X to the lower approximation of the rough cluster Cj ,
that is F Cj . Note that by the property of rough sets, X is
also assigned automatically to F Cj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 272

272 Introduction to Pattern Recognition and Machine Learning

2. Updating Centroids: Unlike the conventional K-means

algorithm, here centroids are updated by assigning different
weightages to patterns based on whether they are in a lower
approximation or in multiple upper approximations. Specifically
centroids are updated as follows:
(a) If F Cj = F Cj and F Cj = φ then its centroid Rj is


X∈F Cj X
Rj = .
|F Cj |
P
X∈(F Cj ) X
(b) If F Cj = φ and F Cj = φ then Rj = |F Cj |
.

(c) If F Cj = φ and F Cj = F Cj then

X∈F Cj X X∈(F Cj −F Cj ) X
Rj = wl + wu .
|F Cj | |F Cj − F Cj |
Here, wl and wu correspond to the weightages or importance
associated with the lower and upper approximations respec-
tively. They add upto 1; that is wl + wu = 1.

The above two steps are iteratively used to realize the Rough
K-means algorithm.

4. Clustering Based on Evolutionary Algorithms

Evolutionary algorithms are inspired by some of the important prop-
erties of natural evolution; survival of the fittest and exchange of
genetic material using crossover or recombination are two such func-
tions captured by evolutionary operators. An evolutionary algorithm
typically has the following steps:

(1) Initialization: A collection of possible solutions, called popula-

tion, is chosen.
(2) Evaluation: The fitness of each element of the population is
computed based on a fitness function.
(3) Selection: A pair of solutions is selected at a time based on the
fitness value assigned to each of the possible solutions in Step 2.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 273

Soft Clustering 273

It is common to view selection as an exploitation operator where

domain-knowledge-based fitness values are exploited to select fit
individuals for further processing. Recombination and mutation
are helpful in exploring the search space and are described next.
Because of this judicial combination of exploitation and explo-
ration, evolutionary algorithms may be considered as focused
(exploitation) random-search (exploration) algorithms.
(4) Recombination: Using the recombination operation on the two
selected solutions, two offspring/children are generated.
(5) Mutation: The two children generated using recombination are
modified based on randomly chosen values in the string and by
mutating these values. The resulting solutions are placed in the
next population. The process of selecting a pair of strings, recom-
bination and mutation are repeated till the required number of
solutions to fill the next population is generated.
(6) Termination: Steps 2 to 5 are repeated till some termination
condition is satisfied. Going through Steps 2 to 5 once generates
from the population at the current instance (Pt ) a new popu-
lation corresponding to the next time instance (Pt+1 ). Such a
process of generating Pt+1 from Pt is called a generation. In
a practical setting the evolutionary algorithm terminates after
a specified number of generations.

We briefly explain below how various steps listed above are realized
in clustering a collection of n patterns. For the sake of illustration
we consider a two-dimensional dataset: X = A : (1, 1)t ; B : (1, 2)t ;
C : (2, 2)t ; D : (6, 2)t ; E : (7, 2)t ; F : (6, 6)t ; G : (7, 6)t .

1. Initialization:

• Let N be the size of a population; N is typically chosen to be

even. The size of the population is chosen based on the size of
the solution space and computational resources.
• The N initial solutions are typically randomly chosen; it is the
right thing to do in the absence of any other information. If
additional knowledge about the problem is available, then it
can be used to choose the initial solutions.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 274

274 Introduction to Pattern Recognition and Machine Learning

• Each solution corresponds to a partition of the dataset and may

be viewed as a string. Some of the possibilities are:

(a) In the case of GAs, each solution string is convention-

ally a binary string. We can represent the partition: C1 =
{A, B, C}; C2 = {D, E}; C3 = {F, G} using the binary
string

1 1 1 0 0 0 0; 0 0 0 1 1 0 0; 0 0 0 0 0 1 1

which has 3 substrings where each cluster is represented by

a substring of size 7; if a pattern is in a cluster then we
place a 1 in that location, otherwise a 0. There will be K
substrings if there are K clusters and each substring is of
length n. This view permits soft clustering. For example,
the binary string

1 1 1 0 0 0 0; 1 0 0 1 1 0 0; 0 0 0 0 1 1 1

indicates overlapping clusters where A belongs to both C1

and C2 and E is in both C2 and C3 .
(b) Another popular representation employs string-of-group-
numbers. Each character in the string represents a cluster
(or group) number. For example:

1 1 1 2 2 3 3 represents a hard partition which indicates that

A, B, and C belong to cluster 1; D and E belong to clus-
ter 2; patterns F, G are in cluster 3. This characterizes the
hard partition C1 = {A, B, C}; C2 = {D, E}; C3 = {F, G}.
This representation does not permit soft clustering using
cluster numbers 1 to K.
(c) String-of-centroids representation employs a real vector to
represent each location in the string. Specifically each string
represents K cluster centers; each cluster center is a string
of size d where d is the dimensionality of the vectors. For
example consider the representation:

1.33, 1.66; 6.5, 2.0; 6.5, 6.0

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 275

Soft Clustering 275

which consists of three centroids each of them in a two-

dimensional space which corresponds to the partition: C1 =
{A, B, C}; C2 = {D, E}; C3 = {F, G} because A, B and
C are closer to the centroid (1.33, 1.66)t ; D and E are
nearer to the centroid (6.5, 2.0)t ; and F and G are closer
to the centroid (6.5, 6.0)t . Note that the string-of-centroids
representation corresponds to the string-of-group-numbers
representation given by 1 1 1 2 2 3 3; in fact there is a
correspondence between these two representations in gen-
eral. There is a bijection between sets of equivalence classes
of the two types defined as follows:
— Two string-of-group-numbers Sgi and Sgj are related
by a binary relation Rg if they both correspond to
the same partition of X . For example, consider the
strings 1 1 1 2 2 3 3 and 2 2 2 1 1 3 3; both
of them lead to the same partition of X given by
{{X1 , X2 , X3 }, {X4 , X5 }, {X6 , X7 }}. Note that Rg is
an equivalence relation on the set of string-of-group-
numbers representations; Rg partitions the set into
equivalence classes where all the strings in each equiva-
lence class are related. Let ECRg be the set of equiva-
lence classes.
— Two strings-of-centroids Sci and Scj are related by
a binary relation Rc if they both correspond to the
same partition of X . For example, the strings-of-
centroids (1, 1); (1.5, 2); (6.5, 4), and (1, 1); (1.5, 2);
(6, 4) correspond to the same partition of X given
by {{(1, 1)}, {(1, 2), (2, 2)}, {(6, 2), (7, 2), (6, 6), (7, 6)}}.
Again note that Rc is an equivalence relation on the set
of strings-of-centroids and partitions it into equivalence
classes. Let ECRc be the set of these equivalence classes.
— There is a bijection between ECRg and ECRc . Asso-
ciated with each partition of X there is an element of
ECRg and an element of ECRc which map to each other
by the bijection.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 276

276 Introduction to Pattern Recognition and Machine Learning

2. Fitness Computation: It is essential to associate a fitness value

with each solution string; this association could be based on one
or more criterion functions. This is required to select highly fit
strings for further exploration. Let us consider for each type of
representation how fitness value is computed.
(a) Because there is a correspondence between string-of-group-
numbers and string-of-centroids we consider them together.
Consider the partition given by
String-of-group-numbers: 1 1 1 3 3 2 2 and equivalently
String-of-centroids: 1.33,1.66; 6.5,2.0; 6.5,6.0
it is the optimal partition and the with-in-group-error-sum-
of-squares value is
1.33 + 0.5 + 0.5 = 2.33, where the left-hand side indicates the
contribution of each cluster.
Next consider the partition given by
string-of-group-numbers: 1 2 2 3 3 3 3 and equivalently
string-of-centroids: 1.0,1.0; 1.5,2.0; 6.5,4.0.
Here, the squared error value given by the sum over the three
clusters is

0 + 0.5 + 17.0 = 17.5.

(b) Consider the binary representation for a hard partition

given by
1 1 1 0 0 0 0; 0 0 0 1 1 0 0; 0 0 0 0 0 1 1.
In this case also the squared error is given by 2.33 which cor-
responds to the optimal partition. It is possible to consider
the soft partition given by
1 1 1 0 0 0 0; 1 0 0 1 1 0 0; 0 0 0 0 1 1 1.
In this case there are some patterns that belong to more than
one cluster. In such a situation we can compute a modified
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 277

Soft Clustering 277

version of the squared error. This modification is achieved by

adding weighted contributions from different patterns in com-
puting the squared error. The weight for a pattern is 1 if it
belongs to a single cluster. If a pattern belongs to more than
one cluster, then a suitable weight is assigned to a pattern for
each cluster based on its distance to centroid of each cluster
to which the pattern belongs.
For example consider the soft partition given by

1 1 1 0 0 0 0; 1 0 0 1 1 0 0; 0 0 0 0 1 1 1,

where A ∈ C1 and C2 and E ∈ C2 and C3 . By noting that

the cluster centers are: (1.33,1.66); (4.66,1.66); (6.66,4.66) we
can compute the distances between A (1, 1)t and the centroids
of C1 and C2 ; the squared distances are 59 and 125
9 which can be
125 5
used to assign the weights as 130 and 130 . In a similar manner
one can compute the weights for E (7, 2)t based on the ratio of
squared distances to centroids of C2 and C3 which will be 50 9
and 65 50 65
9 . So, the weights are 115 and 115 . Using these weights
the squared error computed is 1.3 + 5.6 + 8.5 = 15.4. We have
suggested a simple scheme here; a modified version of this
scheme is used in characterizing the fuzzy criterion function.
(c) Multi-objective Optimization Problem (MOOP): In several
real-world problems the optimization problem might involve
more than one criterion function to be optimized. For exam-
ple, minimizing squared error without any constraint on the
number of clusters K will mean that the minimum squared
error is zero when K = n; in this case each pattern is a clus-
ter. However, in real-world applications, we want a much
smaller K. So, we need to minimize both the squared error
and the number of clusters. In addition to the objective func-
tion, we may have a set of constraints. The two directions for
dealing with MOOP are:

(i) Take a weighted combination of the multiple criterion

functions and treat the resulting function as a single-
objective criterion function.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 278

278 Introduction to Pattern Recognition and Machine Learning

(ii) Treat the multi-objective criterion as a vector of indi-

vidual objectives. Here we consider a feasible solution
dominating another based on how the two solutions are
related on each criterion function value; based on the
non-dominated set of solutions we have pareto-optimal
set and we consider such solutions.
3. Selection: Selection is carried out by using the survival of
the fittest strategy. We have to select a solution so that the
squared error is minimized. However, it is convenient to imple-
ment selection based on maximizing some criterion function. So,
a simple mapping that we implement is to take the reciprocal
of the squared error value and maximize it assuming that the
squared error is not 0 for any solution string. Let us illustrate
using the two-dimensional example considered in computing the
squared error values. We use the string-of-centroids represen-
tation. We give in Table 8.1 a string-of-centroids; the squared
error; its reciprocal; and the probability of selection propor-
tional to the reciprocal. Note that selection probability shown
in the last column is obtained by normalizing the reciprocal of
the fitness value. For example, for string 1 the selection prob-
0.43
ability is (0.43+0.057+0.046) = 0.807. The other probability values
are obtained similarly. Assuming for a while that the popula-
tion has three strings shown in Table 8.1 we generate a uni-
formly distributed random number r in the range 0 to 1. We
select the first string if r ≤ 0.807; the second string if 0.807 <
r ≤ 0.914(0.807 + 0.107); a copy of the third string is selected
if 0.914 < r ≤ 1. Note that the first string has very high

Table 8.1. Three different solution strings.

Fitness =
String Squared 1/squared Selection
number Centroid1 Centroid2 Centroid3 error error probability

1 (1.33,1.66) (6.5,2.0) (6.5,6.0) 2.33 0.43 0.807

2 (1.00,1.00) (1.5,2.0) (6.5,4.0) 17.5 0.057 0.107
3 (1.00,1.00) (1.0,2.0) (5.6,3.6) 21.88 0.046 0.086
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 279

Soft Clustering 279

Table 8.2. Crossover on two strings.

Fitness =
Squared 1/squared
String Centroid1 Centroid2 Centroid3 error error

Parent 1 (1.0,1.0) (6.5,2.0) (1.5,2.0) 33.5 0.03

Parent 2 (1.0,1.0) (1.0,2.0) (6.0,6.0) 34 0.03
Child 1 (1.0,1.0) (6.5,2.0) (6.0,6.0) 3.5 0.29
Child 2 (1.0,1.0) (1.0,2.0) (1.5,2.0) 134 0.007

probability of selection. Selection is the popularly used term

in GAs; reproduction is the term used in other evolutionary
algorithms.
4. Crossover: Crossover is a binary operator; it takes two strings
as input and combines them to output two children strings. This
may be illustrated using the data in Table 8.2. By considering
two parent strings that are not highly fit crossover results a child
(child 1) that is highly fit (squared error = 3.5).
• Here we have used a single-point crossover operator where the
genetic material is exchanged across the crossover point.
• The crossover point is randomly selected. In the dataset shown
in Table 8.2, the crossover point is selected between centroid2
and centroid3.
• Based on this crossover point the prefix (centroid1 and cen-
troid2) of parent 1 is combined with the suffix (centroid3) of
parent 2 to generate child 1. Similarly prefix of parent 2 and
suffix of parent 1 are combined to form child 2.
• Note that in the string-of-centroids representation considered
here the crossover point can be selected between two successive
centroids.
• Typically crossover is the popularly used term in GAs whereas
recombination is used in Evolutionary search.
5. Mutation: Mutation is a unary operator; it takes a string as input
and outputs a string by mutating some randomly selected values.
If the value at position i is vi and δ ∈ [0, 1]. After mutation vi
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 280

280 Introduction to Pattern Recognition and Machine Learning

Table 8.3. A string before and after mutation.

Fitness =
Squared 1/squared
String Centroid1 Centroid2 Centroid3 error error

4
Input (1.5,1.5) (6.5,3.0) (6.5,6.0) 4.25
17
4
Output (1.5,1.5) (6.5,2.0) (6.5,6.0) 2.25
9

becomes vi + 2δvi or vi − 2δvi . We show how it works using the

input and output strings shown in Table 8.3.

• By mutating the value 3.0 to 2.0 in centroid2 the reciprocal of

the squared error almost doubles.
• Mutation working on a binary string randomly selects a bit
position and replaces 0 by 1 and 1 by 0.
• It is possible to show that selection (or reproduction) and muta-
tion are adequate for simulating evolution.
• Evolutionary Programming based algorithms use only reproduc-
tion and mutation; they do not use recombination.

6. Steady-state genetic algorithm (SSGA): Typically cross-

over and mutation are performed with probabilities Pc and Pµ
respectively; further larger Pc values are common, but the value
of Pµ is typically very small. Increasing the mutation rate Pµ can
lead to random search. It is possible to avoid random search and
still increase the value of Pµ by copying some highly fit (elitist)
strings to the next population. This is exploited by SSGA. It can
help us in exploring the search space better by using larger values
for Pµ , however it requires a longer time to converge based on gen-
eration gap. Generation gap between two successive populations
Pt and Pt+1 is characterized by the percentage of elitist strings
copied from Pt to Pt+1 .
It is possible to show that using an elitist strategy the GA
converges to the globally optimal solution of the criterion function
under some acceptable conditions.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 281

Soft Clustering 281

5. Clustering Based on Neural Networks

Some of the important properties of neural networks are competi-
tive learning and self-organization. Competitive learning is concerned
with assigning an input pattern to one or more of the competing
output neurons. Typically it is implemented using a two-layer neural
network where each input node is connected to all the output nodes.
Competitive learning is abstracted with the help of minimization of
the error function given by:

n 
K
µij Xi − Rj 2 ,
i=1 j=1

where µij is the membership of Xi in Cj and Rj is the prototype from

cluster j that is closer to Xi . Self organization is achieved by using
lateral inhibition. The most influential network is the SOM where
the output layer is typically two-dimensional. Some of the important
features of SOM are:

• It can be viewed as a feature extractor; it can be viewed as repre-

senting a high-dimensional input vector as a member of a cluster
present in the two-dimensional output layer.
• It is a topological map where transitions in the input patterns are
captured in the output layer with topological properties preserved.
For example, if we look at transitions from character 1 to charac-
ter 7 as shown in Figure 8.1 it is captured in the output layer of
SOM so that these transitions are captured topologically.
• It employs a strategy called winner-take-most WTM which is a soft
version of winner-take-all strategy. The soft version is implemented
by assigning a pattern Xi to more than one node in the output
layer by using appropriate values of µij . A neighborhood function
is used to achieve it.
• Training a SOM works as follows:
1. If the patterns are d-dimensional, then the input layer has d
nodes. Each of the input nodes is connected initially using some
random weights to all the nodes in the output layer. The number
of nodes in the output layer is chosen based on the resolution at
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 282

282 Introduction to Pattern Recognition and Machine Learning

which the clustering is required. So, each output node is viewed

as a d-dimensional vector based on the weights with which the
d input nodes are connected to it.
2. For each pattern Xi (i = 1, 2, . . . , n), the nearest output node is
selected based on the Euclidean distance between the pattern
and the node which are both d-dimensional vectors. The winner
node and its neighbors are updated by

Rl (k + 1) = Rl (k) + η(k)glm (k)(Xi − Rl (k)),

where the nearest node to Xi is m. η(k) is a monotonically

decreasing learning rate. In addition to the winning node m,
the other nodes (l) in its vicinity are also updated. But the
extent to which a neighbor is updated is based on its distance
to m and is reflected in glm . One popularly used function for glm
is the Mexican hat function; another is the Gaussian function
given by
Rl −Rm 2
−
glm (k) = c1 exp exp−c2 k ,

where c1 and c2 are constants. Note that if we want to use

winner-take-all strategy then we have to update only Rm .
3. It is possible to show that because of the WTM strategy the
learning is not trapped by local minima easily and also based
on the choice of values of η and other constants like c1 and
c2 in the update equation it is possible to show some kind of
probabilistic convergence to the global optimum.

6. Statistical Clustering
K-means algorithm is a well-known hard partitional clustering algo-
rithm where we use the winner-take-all strategy. It is possible to have
an overlapping version of the K-means algorithm that generates a
covering instead of a partition. In a covering we have the clusters
C1 , C2 , . . . , CK of the given dataset X satisfying the following:

i=1 Ci = X
∪K and Ci = φ∀i
∀Xi ∈ X ∃Cj s.t. Xi ∈ Cj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 283

Soft Clustering 283

The difference between a covering and a partition is that Ci ∩ Cj = φ

for i = j in a partition whereas the intersection need not be empty
in a covering. Next we describe the Overlapping K-means (OKM)
algorithm with the help of a two-dimensional dataset.

6.1. OKM algorithm

1. Let the dataset X be {(1, 1), (1, 2), (2, 2), (6, 1), (6, 3), (8, 1),
(8, 3)}. Choose K (= 3) clusters C1 , C2 , and C3 with the initial
(0) (0)
centers randomly chosen from X , say M1 = (1, 2), M2 = (6, 1),
(0)
M3 = (8, 3).
2. For each Xi ∈ X obtain Li , the list of centroids of clusters to
which Xi is assigned. Li is obtained as follows:

(a) Compute the distance between Xi and each of the K centroids

and rank them in increasing order of distance. For example,
the Euclidean
√ distances between (1, 1) and the centroids are
1, 5, 53.
(b) Assign the nearest centroid to Li . So, in the example
L1 = ((1, 2).
(c) Keep on including the next centroid in Li , based on the rank
order, if the error in representing Xi using the centroids in Li
is non-increasing. Let |Li | be the size of Li ; then the error is
defined as
 
 |L

i | 
 
 M j
 
error(Xi , Li ) =  .
j=1
X −

i | 
i
|L
 
 1 
 j=1 

In the case of (1,1) the error in using (1,2) is 1. If we consider

adding  the next centroid, in the rank order, (6,1) then the
error is (1, 1) − 12 [(1, 2) + (6, 1)] = 2.55 which is larger than

1; so, the error increases and we do not add (6,1) to L1. So,
L1 = ((1, 2)).
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 284

284 Introduction to Pattern Recognition and Machine Learning

(d) Similarly for the other patterns we have the lists as follows:

(1, 2) : L2 = ((1, 2)); (2, 2) : L3 = ((1, 2)).

(6, 1) : L4 = ((6, 1)); (6, 3) : L5 = ((6, 1), (8, 3)).
(8, 1) : L6 = ((6, 1), (8, 3)); (8, 3) : L7 = ((8, 3)).
3. Obtain the clusters based on the assignments and Li ’s:

C1 = {(1, 1), (1, 2), (2, 2)},

C2 = {(6, 1), (6, 3), (8, 3)},
C3 = {(6, 3), (8, 1), (8, 3)}.

4. Update the centroids using the weighted average given below:

1 
Mk∗ =
wi · Cki ,
wi
Xi ∈Ck
Xi ∈Ck

1

where wi = |Li |2 and Cki = |Li |Xi − Mj ∈Li −{Mk } Mj .
5. Now the updated centroids are
     
(1) 4 5 (1) 19 4 (1) 23 8
M1 = , , M2 = , , M3 = , .
3 3 3 3 3 3

6. For the other patterns we have the lists as follows:

   
4 5 4 5
(1, 1) : L1 = , ; (1, 2) : L2 = , ;
3 3 3 3
 
4 5
(2, 2) : L3 = ,
3 3
     
19 4 19 4 23 8
(6, 1) : L4 = ,
; (6, 3) : L5 = , , ,
3 3 3 3 3 3
     
19 4 23 8 23 8
(8, 1) : L6 = , , , ; (8, 3) : L7 = , .
3 3 3 3 3 3
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 285

Soft Clustering 285

7. Using the details given in Step 4, the updated centroids are:

     
(1) 4 5 (1) 19 4 (1) 23 8
M1 = , , M2 = , , M3 = , .
3 3 3 3 3 3

8. There is no change in the clusters and their centroids during two

successive iterations. So, the algorithm terminates and the final
clusters obtained are:

{(1, 1), (1, 2), (2, 2)}; {(6, 1), (6, 3), (8, 1)}; {(8, 3), (6, 3), (8, 1)}.

Note that (6,3) and (8,1) belong to two clusters leading to a soft
partition.
9. However, initial centroid selection is important here. If the initial
centroids chosen are (1, 2); (6, 1); (8, 3) then the clusters obtained
using this algorithm are:

{(1, 1), (1, 2), (2, 2)}; {(6, 1), (6, 3)}; {(8, 3), (8, 1)} which is a hard
partition.

6.2. EM-based clustering

K-means algorithm is the most popular hard clustering algorithm
in the history of clustering. However several important applications
of current interest require soft clustering; these applications include
document clustering, customer churn prediction, and social networks.
EM algorithm plays an important role in soft clustering. Each of
the clusters is viewed as being generated by a hidden or latent
variable.
EM has been used in learning hidden markov models (HMMs) in
the form of the well-known Baum–Welch algorithm. It is also used
in learning the PLSI model. In a nutshell it has revolutionized learn-
ing based on probabilistic models including Bayes belief nets and
gaussian mixture models (GMMs). It can be viewed as a soft clus-
tering paradigm; more specifically it is a probabilistic version of the
popular K-means algorithm.
From a theoretical perspective it could be viewed as data likeli-
hood maximizer. It can effectively deal with incomplete data; such
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 286

286 Introduction to Pattern Recognition and Machine Learning

an incompleteness could result because of different reasons:

1. In supervised classification we are given training data X of n

patterns where the ith pattern Xi ∈ X , i = 1, . . . , n, is a
(d + 1)-dimensional vector. Typically the first d features are inde-
pendent and the (d + 1)th feature is the class label of the pat-
tern. In such a classification context, it is possible that values of
one or more features are missing. Here, EM can be used to learn
the parameter values using a single or a mixture of distributions
to characterize each class. This is where EM is used in GMMs
and HMMs.
2. In clustering, the data given has no class labels; so, each pattern
is a d-dimensional vector. Here, EM could be used to obtain the
clusters. Note that in this context, the actual clusters are latent or
hidden. More specifically we would like each d-dimensional pattern
here to be associated with an additional feature; this (d + 1)th
feature will have as its value one or more cluster labels resulting
in a soft clustering of patterns in X . Note that one can use a
mixture distribution here also; for example one can learn a mixture
of Gaussians using the EM where each cluster is represented by a
Gaussian component.

We concentrate on the application of EM to clustering here. We

assume that the data given in the form of X is incomplete because
the cluster labels associated with the patterns are not available. The
log likelihood function is

l(θ) = ln p(X |θ), (3)

where θ is the vector of parameters, corresponding to the distribu-

tion/s to be learnt. Let z1 , z2 , . . . , zK be the K hidden/latent vari-
ables corresponding to the K unknown clusters. Let the joint density
of X and z conditioned on θ be p(X , z|θ). Then we can write the log
likelihood as the marginal of the conditioned joint likelihood given by

l(θ) = ln p(X , z|θ). (4)
z
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 287

Soft Clustering 287

This equality is difficult to deal with as it involves logarithm of a

sum. A form of the Jensen’s inequality is useful in simplifying such
an expression. We discuss this inequality next.

6.2.1. Jensen’s inequality

It is important to examine the notion of convex function before we
discuss Jensen’s inequality.

Definition 1. A function f (t) is convex if

f (αt0 + (1 − α)t1 ) ≤ αf (t0 ) + (1 − α)f (t1 ) for 0 ≤ α ≤ 1. (5)

For example, f (t) = t2 is a convex function because

f (αt0 + (1 − α)t1 ) = (αt0 + (1 − α)t1 )2

= α2 t20 + (1 − α)2 t21 + 2α(1 − α)t0 t1 (6)
and αf (t0 ) + (1 − α)f (t1 ) = αt20 + (1 − α)t21 . (7)

(7)–(6) gives us

α(1 − α)[(t0 − t1 )2 ] (8)

which is equal to 0 if either α = 0 or α = 1; otherwise it is proportional

to (t0 −t1 )2 which is non-negative as it is the square of a real number.
So, we can infer that

left-hand side of (6) is ≤ left-hand side of (7) which shows that f (t)
is convex.

Theorem 1. If f (t) is a convex function then

 

K 
K
f αi ti ≤ αi f (ti) where αi ≥ 0
i=1 i=1


K
for 1 ≤ i ≤ K and αi = 1. (9)
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 288

288 Introduction to Pattern Recognition and Machine Learning

Proof. We show the result using mathematical induction.

• Base case: The property holds for K = 2 as

f (α1 t1 + α2 t2 ) ≤ α1 f (t1 ) + α2 f (t2 )

for α1 , α2 ≥ 0 and α1 + α2 = 1. (10)

This is because f is a convex function.

• Induction Hypothesis: Let the property hold for K = l. That is
 l 
 l
f αi t i ≤ αi f (ti ), where αi ≥ 0
i=1 i=1

l
for 1 ≤ i ≤ l and αi = 1. (11)
i=1

• Induction Step: Using the Induction Hypothesis (inequality in

(11)) we need to show that
 l+1  l+1
 
f αi t i ≤ αi f (ti ), where αi ≥ 0
i=1 i=1

l+1
for 1 ≤ i ≤ l + 1 and αi = 1. (12)
i=1

Consider the left-hand side of the inequality (12) which could be

simplified as
 l+1   l 
 
f αi ti = f αi ti + αl+1 tl+1 .
i=1 i=1

We can write it in the required form by multiplying and dividing

by (1 − αl+1 ) as

l
ti
f (1 − αl+1 ) αi + αl+1 tl+1 . (13)
1 − αl+1
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 289

Soft Clustering 289

Using the base case we can show that the expression in (13) is

 l
 αi
≤ (1 − αl+1 )f ti + αl+1 f (tl+1 )
(1 − αl+1 )
i=1

l
αi
≤ (1 − αl+1 ) f (ti ) + αl+1 f (tl+1 )
(1 − αl+1 )
i=1

l+1
= αi f (ti ) = right-hand side of (12) thus proving the result.
i=1

We can use Jensen’s inequality to simplify the expression of loga-

rithm of the sum seen in (4); it is transformed to a sum of logarithm
form which is easier to deal with. In order to achieve it we need to
have appropriate αs which are non-negative and add upto 1. Prob-
ability mass function (discrete random variable) is ideally suited for
this and we know from (4) that z is a discrete random variable. This
prompts us to find a suitable form for the mass function on z. Further,
we need to recognize that the function under consideration in (4) is
natural logarithm (ln) which is a concave function or equivalently
−ln(x) is convex. In order to show that ln is concave we can use the
following definition.

Definition 2. A function f is concave on an interval if its second

derivative f  (x) is negative in the interval.

Note that if f (x) = ln(x) then f  (x) = − x12 which is strictly decreas-
ing if x > 0. So, ln(x) is concave and −ln(x) is convex.
We can rewrite Eq. (4) by multiplying and dividing by p(z|X , θi ),
where θi is the estimate of θ at the ith step of the iterative scheme
that we need to use, as

 p(X , z|θ)
l(θ) = ln p(z|X , θi ) . (14)
z
p(z|X , θi )
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 290

290 Introduction to Pattern Recognition and Machine Learning

By using the Jensen’s inequality seen earlier, we can show that the
above is
 p(X , z|θ)
≥ p(z|X , θi ) ln . (15)
z
p(z|X , θi )

It is possible to show that θ that maximizes the right-hand side of (15)

is the same as the θ that maximizes (14). Further as θ appears only
in the numerator of the argument of ln, the same θ that maximizes
(15) also maximizes

p(z|X , θi ) ln p(X , z|θ) = Expectation (ln p(X , z|θ)), (16)
z

where the expectation is over z conditioned on X and the value

of θ in the form of θi . So, the two important steps in the EM algo-
rithm are:
1. Expectation or E Step: Compute Ez|X ,θi (ln p(X , z|θ)).
2. Maximization or M Step:

θi+1 = argmax Ez|X ,θi (ln p(X , z|θ)).

θ
EM algorithm also can reach only a locally optimal solution.

6.2.2. An example
Let us consider a one-dimensional example with two clusters. Let the
dataset be

X = {2.1, 5.1, 1.9, 4.9}.

It is easy to see the two intuitively appealing clusters in the form

of {2.1, 1.9} {5.1, 4.9}. Now let us examine how EM can be used
to cluster the set X . Let the clusters be c1 and c2 . Let us assume
that these data points are drawn from two normals each with the
same variance (σ 2 = 1) of value 1. The parameter vector θ = (µ1 , µ2 )t
where µ1 , µ2 are the means of the normals is to be estimated. The
log likelihood function is given by
4

ln p(X , z|θ) = ln Π4i=1 p[Xi , Pi |θ] = ln p[Xi , Pi |θ], (17)
i=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 291

Soft Clustering 291

where Pi = (Pi1 , Pi2 ); Pij = Probability that Xi belongs to cluster cj .

  
 4 2
ln √ 1 1
E[ln p(X , z|θ)] = − 2 E(Pij )(xi − µj )2 .
2πσ 2 2σ
i=1 j=1

(18)

Here, E(Pij ) = Pij as i and j are fixed. Looking

 at the right hand
√ 1
side of the above equation, the first term ln in the summa-
2πσ2
tion does not play any role as we are interested in µs only and σ is
known. By equating the gradient, with respect to θ, of the resulting


expression we get 4i=1 Pij (Xi − µj ) = 0 which implies

4
Pij Xi
i=1
µj = . (19)


4
Pij
i=1

So, in order to estimate the parameter vector θ or equivalently,

the values of µ1 , µ2 we need to get the values of Pij s; note that Pij
is the probability that Xi belongs to cluster Cj or equivalently Xi is
generated by the corresponding (jth) normal density. So,
 
exp − 2σ1 2 (Xi − µj )2
Pij = 2 . (20)

 1 
exp − 2σ2 (Xi − µl ) 2
l=1

So, in this case, Eqs. (20) and (19) characterize the Expectation
and M aximization steps and they are repeated iteratively till
convergence.
Let θ0 = (2, 4)t be the initial selection of the µs. Let us consider
computation of P11 . By using (20) and σ = 1, it is given by
 
exp − 12 (2.1 − 2)2
P11 =     = 0.728.
exp − 12 (2.1 − 2)2 + exp − 12 (2.1 − 4)2

In a similar manner P21 , P31 , P41 can be computed. Once we have

these values, we can estimate µ1 using Eq. (19). In a similar manner
one can estimate Pi2 , for i = 1, . . . , 4 and µ2 . These values are shown
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 292

292 Introduction to Pattern Recognition and Machine Learning

Table 8.4. Values during the first iteration.

j P 1j P 2j P 3j P 4j µj

1 0.728 0.018 0.9 0.015 2.76

2 0.272 0.982 0.1 0.985 4.53

Table 8.5. Values during the second iteration.

j P 1j P 2j P 3j P 4j µj

1 0.94 0.09 0.96 0.1 2.26

2 0.06 0.91 0.04 0.9 4.6

Table 8.6. Values during the third iteration.

j P 1j P 2j P 3j P 4j µj

1 0.946 0.214 0.97 0.03 2.345

2 0.04 0.786 0.03 0.97 4.88
0.5
0.4
0.3
0.2
0.1
0.0

−2 0 2 4 6 8 10

Figure 8.2. The densities corresponding to the two clusters.

in Table 8.4. So, at the end of the first iteration we get µ1 = 2.76 and
µ2 = 4.53. So, θ1 = (2.76, 4.53)t . Using this value of θ, the correspond-
ing values of the parameters are given in Table 8.5. Table 8.6 shows
the parameters in the third iteration. After some more iterations
we expect the value of θ = (2, 5) to be reached. The corresponding
densities are shown in Figure 8.2.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 293

Soft Clustering 293

7. Topic Models
One of the important areas of current interest is large-scale document
processing. It has a variety of applications including:

• Document retrieval: Here, the documents in a collection are

represented, typically, using the bag-of-words (BOW) model in the
case of text. In the BOW model, the order of occurrence of words
is not important; their frequency of occurrence is important. Here
for a given input query, a collection of documents is presented in
a ranked order as output. Classification, clustering and ranking of
documents are important machine learning tasks that are useful
in this application.
• Spam filtering: Spam electronic mails are unsolicited mails which
can be wasting resources of the recipient. So, spam mails need
to be identified and isolated appropriately. This activity requires
clustering and classification of email messages.
• Verification (biometrics): There are several applications con-
cerned with forensics and cyber security where biometrics plays
an important role. These include identification based on biomet-
rics like fingerprint, speech, face, and iris. So, classification and
clustering are important again in this application.
• Pin-Code recognition: Automatic pin-code recognition is
required to route surface mail to its destination mechanically. This
involves recognition of handwritten numerals.
• Multimedia and multi-lingual document classification:
Classification and summarization of reports in legal and medical
domains.

It is possible to view most of the data stored and processed by a

machine as comprising of documents. In this sense, the following are
some of the popularly considered document categories:

• Web pages: These are easily the most popular type of semi-
structured documents. Every search engine crawls and indexes web
pages for possible information retrieval; typically search engines
return, for a query input by the user, a collection of documents
in a ranked manner as output. Here, in addition to the content
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 294

294 Introduction to Pattern Recognition and Machine Learning

of the webpages, the link structure in terms of hyperlinks is also

exploited.
• Academic publications: This kind of documents are also prominent
to the extent that their content and links in terms of co-citations
are used by search engines in document retrieval.
• Research grant applications: It is typically a semi-structured text
document containing charts, graphs, and some images.
• Company reports: It may also be semistructured text document.
• Newspaper articles: This could be typically a short text document
and it can have images at times.
• Bank transaction slips: This is another semi-structured short text
document.
• Manual pages: This is another short size text document with pos-
sible figures.
• Tweets: Tweets are short text documents and they may employ
several non-standard words.
• Encyclopedia: This could be a very lengthy and comprehensive
text document with figures and images combined. Wikipedia is a
well-known example where search is facilitated.
• Images (video): It is an example of a non-textual document.
• Speech records: These are also examples of non-textual documents.
• Fingerprints: It is also a non-textual type of document used as a
biometric.
• Electronic mails: It is one of the most popularly used type of doc-
uments for communication between two or more individuals; it is
typically a short text.
• Health records: It could be a multimedia document. The prescrip-
tion and diagnosis in handwritten textual form, ultrasound and
X-rays in the form of images form a part of the document.
• Legal records: It is another textual document and words here may
have to be precise.
• Software code: Analysis of software code is gaining a lot of impor-
tance to identify bugs and inefficient parts of the code.
• Bug reports: These are reports used to indicate various bugs
encountered in the software package and how some of them are
fixed periodically.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 295

Soft Clustering 295

There could be several other kinds of documents. In order to analyze

document collections, a variety of topic based schemes have been
proposed. It is possible to view these methods to be performing fac-
torization of the document term matrix. So, we consider these matrix
factorization based methods next.

7.1. Matrix factorization-based methods

Let X be the document term matrix of size n × l which means that
there are n documents and l terms. Typically l could be very large
in most of the practical settings. A possible generic factorization is:

Xn×l = Bn×K DK×K CK×l . (21)

The basic idea behind such a factorization is that it permits

us to work with documents represented in a lower-dimensional
(K-dimensional) space. The value of K corresponds to the number
of non-zero eigenvalues of XX t or the rank of the matrix X. In
other words we can work with the B matrix instead of the X matrix.
Further, it is possible to view this as working with K topics/clusters
instead of l terms or words.

7.1.1. Why should this work?

It is well-known that conventional distance-based methods fail to
characterize neighbors in high-dimensional spaces. This is because
the ratio of d(x, N N (x)) and d(x, F N (x)) tends to 1 as the dimen-
sionality gets larger and larger; here d(x, N N (x)) is the distance
between a point x and its nearest neighbor, N N (x), and d(x, F N (x))
is the distance between x and its farthest neighbor, F N (x). Two
different ways of solving this problem are:

1. Explicit Dimensionality Reduction: Here the data in the

high-dimensional space is projected into a low-dimensional space
by using either a linear or a nonlinear projection tool. For example
Random Projections are useful in unsupervised feature extraction;
similarly mutual information has been successfully used in fea-
ture selection. Once the dimensionality is reduced, it is possible
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 296

296 Introduction to Pattern Recognition and Machine Learning

to successfully use classifiers based on matching. In such a case

the data matrix X becomes a n × K matrix instead of n × l
where K is the number of features extracted from the given l
features.
2. Implicit Dimensionality Reduction: Instead of explicit selec-
tion it is possible to use matrix factorization approaches to implic-
itly get the features. For example, if we assume that D is an
identity matrix of size K × K, then the resulting equation is
X = BC; note that even though X is a collection of n data points
(rows) in a possibly high-dimensional space of l terms (columns),
the resulting B matrix represents the n patterns in a reduced-
dimensional (K) space. This amounts to dimensionality reduction;
specifically each of the K columns of B may be viewed as a linear
combination of the l columns in A. Further it is important that
entries of C are well-behaved; popularly the objective function of
the optimization problem deals with some error between A and
its approximation in the form of BC. In addition C is constrained
to be sparse, for example. We deal with different factorization
approaches in this chapter.

7.2. Divide-and-conquer approach

The simplest scheme to deal with high-dimensional spaces is to parti-
tion the set of features F into some P blocks F1 , F2 , . . . , FP such that

F = Pi=1 Fi ; cluster subpatterns in each of these P subspaces and
use the cluster representatives to realize a compact set of prototypes.
We can use them in large data classification. We illustrate it with a
simple example.
Let us consider the dataset shown in Table 8.7; there are two
classes and each pattern is four dimensional as indicated by the
features f1 , f2 , f3 and f4 . Now given a test pattern (2, 2, 2, 2), it is
classified as belonging to class 1 using the nearest neighbor classi-
fier as its nearest neighbor is pattern number 4, given by (1, 2, 2, 2),
which is in class 1. Let this set of features be partitioned into
F1 = {f1 , f2 } and F2 = {f3 , f4 }. Now we cluster data in each class
separately and while clustering we consider each subset of features
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 297

Soft Clustering 297

Table 8.7. Patterns corresponding to two classes.

Pattern no. f1 f2 f3 f4 Class

1 1 1 1 1 1
2 1 2 1 1 1
3 1 1 2 2 1
4 1 2 2 2 1
5 6 6 6 6 2
6 6 7 6 6 2
7 6 6 7 7 2
8 6 7 7 7 2

separately. We explain the specific steps as:

• Consider patterns 1 to 4 which are from class 1. Cluster these

patterns based on F1 and F2 separately.
— The subpatterns based on F1 are (1, 1), (1, 2), (1, 1), and (1, 2).
If we seek two clusters using any conventional algorithm, we get
the two clusters: {(1, 1), (1, 1)}, and {(1, 2), (1, 2)}. The cluster
centroids are (1, 1) and (1, 2) respectively.
— The subpatterns based on F2 are (1, 1), (1, 1), (2, 2), (2, 2)
which form two clusters with (1, 1) and (2, 2) as representatives.
• Similarly consider patterns 5 to 8 that belong to class 2. Again by
clustering the subpatterns based on F1 and F2 we have
— Cluster representatives based on F1 are (6, 6) and (6, 7).
— Cluster representatives based on F2 are (6, 6) and (7, 7).

We collect these cluster representatives corresponding to different

classes based on different feature subsets. In this example, the rel-
evant details are shown in Table 8.8. It is possible to use data in
Table 8.8 to classify a test pattern. For example, consider the pat-
tern (2, 2, 2, 2); we consider its subpatterns corresponding to F1 and
F2 which are (2, 2) and (2, 2). Now we obtain the nearest neighbors
of these subpatterns in each class. They are given by

• The nearest subpatterns of (2, 2) (based on F1 ) are (1, 2) from

class 1 and (6, 6) from class 2; of these class 1 subpattern (1, 2) is
nearer.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 298

298 Introduction to Pattern Recognition and Machine Learning

Table 8.8. Cluster representatives of the two classes.

Representative Based Based

number on F 1 on F 2 Class

1 1 1 1 1 1
2 1 2 2 2 1
1 6 6 6 6 2
2 6 7 7 7 2

• The nearest neighbors of (2, 2) (based on F2 ) are (2, 2) from class 1

and (6, 6) from class 2; of these two (2, 2) is nearer.

Observing that the nearest subpatterns are from class 1 based on

both F1 and F2 , we assign (2, 2, 2, 2) to class 1. If we concatenate
the nearest subpatterns (1, 2) and (2, 2) then we get (1, 2, 2, 2) as the
nearest neighbor pattern. We can abstract the whole scheme using
the following steps.
1. Let there be C classes with the ith class having ni patterns. Par-
tition patterns of the ith class into Pi subsets of features. In the
above example C = 2 and P1 = P2 = 2 as in both the classes we
have used the same F1 and F2 .
2. For the test pattern we obtain the nearest neighbors from each of
the C classes.
• For class i consider the Pi subsets of features. Find the nearest
subpattern of each test subpattern in each of the corresponding
subsets from the training data.
• Concatenate the Pi nearest subpatterns of the test subpatterns.
Compute the distance between the test pattern and this con-
catenated pattern.
3. Assign the test pattern to class j if the nearest pattern from the
jth class is nearer than the other classes.
It is possible to consider several variants of this simple divide-and-
conquer approach. Some of them are:
• Instead of using a single nearest neighbor one can use K nearest
neighbors from each class to realize a robust classifier.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 299

Soft Clustering 299

• Instead of concatenating the Pi nearest subpatterns one can con-

sider K nearest subpatterns based on each of the Pi feature sub-
sets. Then generate neighbors by concatenating Pi subpatterns
selecting one from each of the K subpatterns selected based on
each of the Pi feature sets. This leads to selection of PiK concate-
nated patterns. Select K of these patterns based on nearness to the
test pattern. If there are C classes then we get CK neighbors, K
from each class. Select K nearest of these and use majority voting
to decide the class label.
• Computation of distance between the test pattern and a neighbor
can be obtained by summing partial distances across the Pi sub-
patterns in some cases. This is possible when squared Euclidean
distance or city-block distance is used to compute partial distances.

7.3. Latent Semantic Analysis (LSA)

In this factorization D is a diagonal matrix. LSA employs this fac-
torization directly. In such a case, the above factorization exploits
the Singular Value Decomposition (SVD) where the diagonal entries
in D are singular values and they correspond to the eigenvalues of
the matrices XX t and X t X where X t is the transpose of X. This
may be explained as follows:
Consider XX t ; it is a square matrix of size n × n. If β is an
eigenvector and λ the corresponding eigenvalue then we have

XX t β = λβ

by premultiplying both sides by X t we have

X t (XX t β) = X t (λβ).

By using the associativity of matrix multiplication and noting that

λ is a scalar, we have

(X t X)X t β = λ(X t β).

Note that X t X is a square matrix of size l × l; β is a n × 1 vector

and X t β is a l × 1 vector. By assuming that X t β = γ we can write
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 300

300 Introduction to Pattern Recognition and Machine Learning

the above equation as

X t Xγ = λγ,

which means that γ is the eigenvector of X t X with the associated

eigenvalue being λ. This means that λ is an eigenvalue of both XX t
and X t X but with the corresponding eigenvectors being β and γ (=
X t β) respectively.
Given a matrix X of size m × n the SVD is given by

X = B D C,

where size of B is m × K; D is a K × K diagonal matrix; and C is of

size K × n. Here, B consists of K eigenvectors of the matrix XX t ;
these are eigenvectors corresponding to the K non-zero eigenvalues
of the matrix XX t . Similarly C is composed of K eigenvectors of
X t X as the K rows of C. We now illustrate these ideas using an
example.

7.3.1. Illustration of SVD

Consider a 3 × 2 size matrix X given by
 
1 −1
X = 1 0.
0 1

Note that X t X is given by

 
t 2 −1
X X= .
−1 2

The characteristic equation of X t X is λ2 − 4λ + 3 = 0 and so the

roots of the equation or eigenvalues of X t X are 3 and 1.
The corresponding eigenvectors are
 
1
−1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 301

Soft Clustering 301

and
 
1
.
1
The corresponding normalized vectors are:
 
1
√

 2
 1 
−√
2
and
 
1
√
 2
 
 1 .
√
2
In a similar manner it is possible to observe that the matrix
XX t is
 
2 1 −1
 1 1 0.
−1 0 1
The corresponding eigenvalues are 3, 1, and 0 and the eigenvectors
corresponding to the non-zero eigenvalues after normalization are
 2 
√
 6
 
 1 
 √ 
 6
 
 1 
−√
6
and
 
0
 
 1 
√ 
 2 .
 
 1 
√
2
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 302

302 Introduction to Pattern Recognition and Machine Learning

The entries of the diagonal matrix Σ are the square roots of the
eigenvalues (singular values) 3 and 1. So, the various matrices are
 
2
√ 0
 6 
 
 1 1 
 √ √ 
B= ,
 6 2
 
 1 1 
−√ √
6 2
√ 
3 0
D= ,
0 1
 1 1 
√ −√
 2 2
C= 1
.
1 
√ √
2 2

So, finally we have the SVD of X given by

 
2
√  1 1 
0
   6 
1 −1  √  √ −√
 1 1  2
 0 
√   2
1 3 .
0 =  √ 
 6 2  0 1  1 1 
0 1   √ √
 1 1  2 2
−√ √
6 2

7.4. SVD and PCA

Principal component analysis (PCA) is a well-known tool for dimen-
sionality reduction. Here, linear combinations of the original features
are obtained which are uncorrelated. Mathematically it amounts to
computing the eigenvectors of the covariance matrix. The covariance
matrix is symmetric and the entries are real numbers; so, the eigen-
vectors are orthogonal. The eigenvector corresponding to the largest
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 303

Soft Clustering 303

eigenvalue is the first principal component; the one corresponding to

the next largest eigenvalue is the second principal component, and so
on. We illustrate the computation of PCA using a two-dimensional
example.

7.4.1. Example to illustrate PCA

Consider the four two-dimensional patterns given by
    
1 2 6 7
.
2 1 7 6

 
4
The sample mean of the 4 data points is . The zero mean nor-
4
malized set of points is
    
−3 −2 2 3
.
−2 −3 3 2

The sample covariance matrix of the data is given by

1
(−3, −2)t (−3, −2) + (−2, −3)t (−2, −3)
4

+ (2, 3)t (2, 3) + (3, 1)t (3, 1)
   
1 26 24 6.5 6
= = .
4 24 26 6 6.5

The eigenvalues of
 the
 matrix
 are 12.5 and 0.5 and the respective
1 1
eigenvectors are . After normalization we get the unit-
1 −1
 1   1 
√ √
2 2
norm orthogonal eigenvectors given by 1
. The four
√1 − 2
√
2
data points and the corresponding principal components are shown
in Figure 8.3.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 304

304 Introduction to Pattern Recognition and Machine Learning

X
PC1

PC2

Figure 8.3. The two principal components.

7.4.2. Computing PCs using SVD

Consider the four data points after zero-mean normalization. It is
given by the matrix X of size 4 × 2:
 
−3 −2
−2 −3
X=  2
.
3
3 2
The SVD of X is given by
 
1 1
− −
 2 2
   1

−3 −2
 
− 1 1 √ 1 
  √ √
−2 −3  2 50 √0  2
 = 
2
  2 .
 2 3   1 1 0 2  1 1 
  √ −√
3 2  2 −2 2 2
 
 1 1 
2 2
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 305

Soft Clustering 305

Note that the rows of the matrix C correspond to the principal com-
ponents of the data. Further, the covariance matrix is proportional
to X t X given by
 1 1   1 1 
√ √   √ √
 2 2  2 
X tX =   50 0  2 .
 1 1  0 2  1 1 
√ −√ √ −√
2 2 2 2
In most of the document analysis applications, it is not uncommon
to view a document collection as a document-term matrix, X as
specified earlier. Typically such a matrix is large in size; in a majority
of the practical applications the number of documents is large and
the number of terms in each document is relatively small. However,
such data is high-dimensional and so the matrix can be very sparse.
This is because even though the number of terms in a document is
small, the total number of distinct terms in the collection could be
very large; out of which a small fraction of terms appear in each of the
documents which leads to sparsity. This means that dimensionality
reduction is essential for applying various classifiers effectively.
Observe that the eigenvectors of the covariance matrix are the
principal components. Each of these is a linear combination of the
terms in the given collection. So, instead of considering all possible
eigenvectors, only a small number of the principal components are
considered to achieve dimensionality reduction. Typically the num-
ber of terms could be varying between 10,000 to 1 million whereas
the number of principal components considered could be between
10 to 100. One justification is that people use a small number of
topics in a given application context; they do not use all the terms
in the given collection.
Latent semantic analysis involves obtaining topics that are latent
and possibly semantic. It is based on obtaining latent variables in
the form of linear combinations of the original terms. Note that the
terms are observed in the given documents; however the topics are
latent which means topics are not observed. Principal components are
such linear combinations. The eigenvalues of the covariance matrix
represent variances in the directions of the eigenvectors. The first
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 306

306 Introduction to Pattern Recognition and Machine Learning

principal component is in the direction of maximum variance. In a

general setting dimensionality reduction is achieved by considering
only the top K principal components where K is smaller than the
rank, r, of the matrix X. Specifically dimensionality reduction is
useful when the data points in X are high-dimensional.
It is possible to use the SVD to obtain a low-rank representation
XK which is an approximation of X. This is obtained by considering
the largest K singular values and forcing the remaining r−K singular
values in D to be zero. So, XK is given by

XK = BDK C,

where DK is a r × r diagonal matrix with the top K eigenvalues in

D and the remaining r − K diagonal values to be zero. Note that
XK is an approximation of X. It is an optimal approximation in the
sense that among all possible K rank matrices XK has the minimal
Frobenius norm with X. It is possible to show that this Frobenius
norm is

X − XK F = λK+1 ,

where λK+1 is the largest singular value ignored among the least
r − K singular values.
There are claims that the resulting reduced dimensional represen-
tation is semantic and can handle both synonymy and polysemy in
information retrieval and text mining. Here, by synonymy we mean
two words having the same meaning are synonymous. Similarly we
have polysemy when the same word has multiple meanings. This
will have impact on the similarity between two documents. Because
of synonymy the similarity value computed using dot product type
of functions will be less than the intended. For example, if car and
automobile are used interchangeably (synonymously) in a document
and only car is used in the query document then the similarity mea-
sure will fail to take into account the occurrences of automobile. In
a similar manner because of polysemy it is possible that similarity
between a pair of documents is larger than what it should be. For
example if tiger is used in a document both to mean an animal and
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 307

Soft Clustering 307

say airlines and the query document has tiger in only one sense then
the dot product could be larger than the intended value.
An important feature of the SVD is that it is a deterministic
factorization scheme and the factorization is unique. Here each row
of the matrix C is a topic and it is an assignment of weights to each
of the terms. The entry Cij is the weight or importance of the jth
term (j = 1, . . . , l) to the ith topic. The entry Dii in the diagonal
matrix indicates some kind of weight assigned to the entire ith topic.

7.5. Probabilistic Latent Semantic Analysis (PLSA)

Latent semantic analysis employs a deterministic topic model that
is reasonably popular in analyzing high-dimensional text document
collections. However it does not offer a framework to synthesize or
generate documents. PLSA is a generative model; it is also a topic
model where each topic is a mapping from set of terms to [0, 1].
Specifically a topic assigns a probability to each term.
It is possible to view the probability, P (t), of a term t as


K 
K
P (t) = P (Ci )P (t|Ci ) where P (Ci ) = 1,
i=1 i=1

where Ci is a soft cluster or topic. It is possible to use the Bayes rule

to write the joint probability P (d, tj ) as


K
P (d, tj ) = P (d)P (tj |d) where P (tj |d) = P (tj |Ci )P (Ci |d).
i=1

It is possible to view the PLSA as a matrix factorization scheme

as follows:

K
P (d, tj ) = P (d|Ck )P (Ck )P (tj |Ck ).
i=1

This can be viewed as a matrix factorization of the form X = BDK E,

where the diagonal matrix DK is made up of probabilities of the
K topics. Specifically the ith element on the diagonal is given by
P (Ci ). The element Bij is given by P (di |Ck ) where di is the ith
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 308

308 Introduction to Pattern Recognition and Machine Learning

document and Ck is the kth topic. Similarly, Ejk corresponds to

P (tj |Ck ).
Learning the model involves, given the document topic matrix X
to get the B, DK and E matrices or equivalently getting the values of
P (di |Ck ), P (Ci ), and P (tj |Ck ). Alternatively, one may consider the
parameters to be P (di ), P (tj |Ck ) and P (Ck |di ). These are obtained
using the EM algorithm. The EM iteratively goes through the expec-
tation and maximization steps. In the expectation step the expected
value of the log-likelihood function is obtained and in the maximiza-
tion step, parameter values are obtained based on the maximization
of the expected value. Let us assume the following:

• Let the input X be a document collection of n documents and each

is a vector in an l dimensional space; this means the vocabulary
size is l.
• Let n(di , tj ) be the number of times term tj occurred in document
di and P (di , tj ) be the probability of term tj in document di .
• Let us assume the bag of terms model which means each term
occurs independent of the others and it may occur more than once
in a document.
• Let θ be the parameter vector with components P (di ), P (tj |Ck )
and P (Ck |di ); learning involves estimating the values of these
parameters from X.

The likelihood function, L(θ), is given by


n 
l
L(θ) = P (di , tj )n(di ,tj ) .
i=1 j=1

The log-likelihood function, l(θ) is given by


n 
l
l(θ) = n(di , tj )log P (di , tj ),
i=1 j=1


n 
l
Hence, l(θ) = n(di , tj )log[P (di )P (tj |Ck )P (Ck |di )].
i=1 j=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 309

Soft Clustering 309

So, expected value of l(θ) with respect to the latent cluster Ck con-
ditioned on di and tj is given by


n 
l 
K
ECk |di ,tj (l(θ)) = n(di , tj ) P (Ck |tj , di )
i=1 j=1 k=1

× [log P (di ) + log P (tj |Ck ) + log P (Ck |di )].

P (tj |Ck )P (Ck |di )
Here, P (Ck |tj , di ) = PK .
k=1 P (tj |Ck )P (Ck |di )

In addition to the expected value of l(θ), we need to consider the

constraints on the probabilities which are


n 
l 
l
P (di ) = 1; P (tj |Ck ) = 1; P (Ck |di ) = 1.
i=1 j=1 j=1

The corresponding Lagrangian is given by

 n   
 
K l
ECk |di ,tj (l(θ)) + α1 P (di ) − 1 + α2  P (tj |Ck ) − 1
i=1 k=1 j=1
 

n 
K
+ α3 P (Ck |di ) − 1 ,
i=1 k=1

where α1 , α2 , and α3 are the Lagrange variables.

By taking the partial derivatives of this Lagrange function with
respect to the three parameters P (di ), P (tJ |Ck ), and P (Ck |di ) and
equating them to zero we get the following estimates:

l
K
j=1 k=1 n(di , tj )P (Ck |tj , di )
• P (di ) =
n
l
K ,
i=1 j=1 k=1 n(di , tj )P (Ck |tj , di )

n
n(di , tj )P (Ck |tj , di )
• P (tj |Ck ) =
l
n
i=1
,
j=1 i=1 n(di , tj )P (Ck |tj , di )

l
j=1 n(di , tj )P (Ck |tj , di )
• P (Ck |di ) =
l
K .
j=1 k=1 n(di , tj )P (Ck |tj , di )
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 310

310 Introduction to Pattern Recognition and Machine Learning

7.6. Non-negative Matrix Factorization (NMF)

NMF is another popular soft clustering tool. Here, in the factoriza-
tion shown in (21), D is the Identity matrix I of size K × K and
so X = BC. Further, given that X has non-negative entries, we
insist that B and C also have only non-negative entries. Hence this
factorization is called NMF. Typically we seek an approximate fac-
torization of X into the product BC. There are several possibilities
for realizing the approximate factors B and C; some of them are:

• Obtain B and C such that X − BC2 is minimized. This corre-

sponds to the minimization of squared Euclidean distance between
the matrices X and BC; such a kind of expression stands for the
squared Frobenius Norm or element-wise distance given by
 2
 
K
X − BC2 = Xij − Bik Ckj .
ij k=1

It is easy to observe that the minimum value of such a distance is

zero when X = BC.
• Minimize a generalized Kullback–Leibler divergence between X and
BC given by

 Xij 
k
Xij log
K − Xij + (Bik Ckj ) .
ij k=1 (Bik Ckj ) k=1

When entries of X and BC are normalized or probabilities such

that ij Xij = ij k (Bik Ckj ) = 1. It could be seen that the
minimum value of this divergence is zero.

So, the problem of the minimization of distance is

M inimize X − BC2 with respect to B and C

such that B and C have non-negative entries. Equivalently, B, C ≥ 0.

There is no guarantee that we can get the globally optimal solution.

However, it is possible to get the local optimum using the following
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 311

Soft Clustering 311

update rules.
(XC t )ij
Bij ← Bij ,
(BCC t )ij
(B t X)ij
Cij ← Cij .
(B t BC)ij
It is possible to show some kind of equivalence between N M F and
the K-means algorithm; also between N M F and PLSA.

7.7. LDA
Even though PLSA offers an excellent probabilistic model in charac-
terizing the latent topics in a given collection of documents, it is not
a fully generative model. It cannot explain documents which are not
part of the given collection. Some of the important features of the
PLSA are:

• A document di and a term tj are assumed to be independent con-

ditioned on a cluster Ck . Because of this we could write

K
P (di , tj ) = P (di ) P (tj , Ck |di ).
k=1

By using Bayes rule we can write P (tj , Ck |di ) as P (tj , di |Ck ) PP(Ck)
(di )
.
This will mean that

K
P (Ck )
P (di , tj ) = P (di ) P (tj , di |Ck )
P (di )
k=1


K
P (Ck )
= P (di ) P (tj |Ck )P (di |Ck ) .
P (di )
k=1

The independence between di and tj is used to simplify the previ-

ous expression which leads to

K
P (di , tj ) = P (di ) P (tj |Ck )P (Ck |di ).
k=1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 312

312 Introduction to Pattern Recognition and Machine Learning

• A document may contain multiple topics or equivalently belong

to multiple clusters; P (Ck |di ) is learnt only on those documents
on which it is trained. It is not possible to make probabilistic
assignment to an unseen document.
• P (tj |Ck ) and P (Ck |di ) are important parameters of the model.
The number of parameters is Kl + Kn; so there will be a linear
growth with corpus size and also can overfit.
• Each cluster Ck is characterized by l probabilities P (tj |Ck ) for
j = 1, 2, . . . , l. So, a cluster Ck associates different probabilities to
different terms. Specifically, a topic or a description of the cluster
is explained by the l probabilities.

A more appropriate probabilistic topic model is the LDA; it

exploits the conjugate prior property between Dirichlet and multino-
mial distributions. LDA is a truly generative probabilistic model that
can assign probabilities to documents not present in the training set
unlike the PLSA. Some of the important features of the LDA are

• Each document is a random mixture over latent topics or clusters;

each cluster is characterized by a distribution over terms.
• It is assumed that there are a fixed number of topics or clusters
that are latent and using them one can generate documents.
• Each topic is a multinomial over the l terms.
• It is assumed that the prior density is Dirichlet which is charac-
terized by a K-dimensional vector θ such that the ith component


of θ, given by θi is non-negative for all j and K k=1 θk = 1. The
functional form of the density is given by

K
p(θ|α) = C(α) θkαk −1 ,
k=1

where


K
Γ i=1 αi
B(α) = K ,
i=1 Γ(α i )
where Γ stands for the Gamma function and α is the input param-
eter vector.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 313

Soft Clustering 313

• Here Dirichlet is chosen because it is the conjugate prior of

the multinomial distribution. Multinomial is useful here because
topics/clusters are drawn using the multinomial distribution.
Specifically description of the cluster Ck is viewed as a multinomial
based on the parameter vector θ.
• Documents are given; terms in the document collection form the
observed random variables. Clusters or topics are hidden or latent.
The idea of learning or inference in this model is to estimate the
cluster structure latent in the collection; typically these clusters
are soft.
• The hidden variables characterizing the model are α, a
K-dimensional vector and β, a matrix of size K × l when there are
K soft clusters/topics and l distinct terms in the entire document
corpus.
• In order to generate a document having N terms, it is assumed that
each term is generated based on a topic. So, in order to generate a
term, a topic is sampled and using the topic and β matrix a term
is generated. This procedure is repeated N times to generate the
document.
• The generative process may be characterized by a joint probability
distribution over both the observed and hidden random variables.
The posterior distribution of the latent cluster structure condi-
tioned on the observed distribution of the terms in the document
collection is derived out of this.
• The entire document generation process is formally characterized
as follows:

— The vector φ is sampled from a Dirichlet distribution p(φ|β)

where φ is a vector of size K where the kth component corre-
sponds to the kth cluster in the document collection.
— The vector θ is sampled from a Dirichlet distribution p(θ|α)
where θ is the vector of K cluster distributions at the document
level.
— For each term in the document, sample a topic/cluster using
a multinomial distribution p(Ck |θ); use a multinomial p(tj |φk )
to sample a term tj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 314

314 Introduction to Pattern Recognition and Machine Learning

— So, the probability of the document with N terms drawn inde-

pendently is given by

p(d, θ, C, φ|α, β) = p(d|φ)p(φ|β)p(C|θ)p(θ|α).

The document generation scheme is explained using this equation.
It explains how a collection of documents is generated by using the
parameters α and β and assuming independence between documents.
However the learning or inference process works by using the docu-
ments as input and learning the parameter vector θ, φ and the set of
clusters C. The corresponding posterior distribution obtained using
Bayes rule is given by

p(θ, φ, C, d|α, β)
p(θ, φ, C|d, α, β) = .
p(d|α, β)

Unfortunately exact computation of this quantity is not possible;

the basic difficulty is associated with the denominator which has
a coupling between θ and β. There are several approximations sug-
gested in the literature; we consider a popular scheme based on Gibbs
sampling.

7.7.1. Gibbs Sampling-based LDA

It is based on the observation that the complete cluster structure
can be derived if one knows the assignment of a cluster/topic given
a word. Specifically it needs to compute the probability of a cluster
Ci being relevant for a term ti given all other words being relevant
to all other clusters. Formally, this probability is

p(Ci |C−i , α, β, d),

where C−i means all cluster associations other than Ci . We can show
using Bayes Rule

p(Ci , C−i , d|α, β)

p(Ci |C−i , α, β, d) = ,
p(C−i , d|α, β)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 315

Soft Clustering 315

where the denominator is independent of Ci ; so, it can be ignored

but for a scaling factor and the expression may be written as

p(Ci |C−i , α, β, d) ∝ p(Ci , C−i , d|α, β) = p(C, d|α, β).

Note that Ci and C−i together correspond to C which is used in

simplifying the expression to get p(C, d|α, β).
We can get the required marginal as

p(d, C|α.β) = p(d, C, θ, φ|α, β)dθ dφ.

The joint distribution can be simplified as


p(d, C|α, β) = p(φ|β)p(θ|α)p(C|θ)p(d|φC )dθ dφ.

We could separate terms having dependent variables to write the

above equation as a product of two integrals.
 
p(d, C|α, β) = p(θ|α)p(C|θ)dθ p(d|φC )p(φ|β)dφ.

Both the integrands have Dirichlet priors in the form of p(θ|α) and
p(φ|β); also there are multinomial terms, one in each integrand. The
conjugacy of Dirichlet to the multinomial helps us here to simplify
the product to a Dirichlet with appropriate parameter setting. So, it
is possible to get the result as


n
B(α) 
K
B(β)
p(D, C|α, β) = ,
B(ndi ,. + α) B(n.,k + β)
i=1 k=1

where ndi ,k is the number of terms in document di that are associated

with cluster Ck . There are n documents in the set D. A ‘.’ indicates
summing over the corresponding index; for example, ndi ,. is the total
number of terms in di and n.,k is the number of terms in the entire
collection that are associated with Ck . It is possible to use the above
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 316

316 Introduction to Pattern Recognition and Machine Learning

equation to get the estimate for p(Ck |C−k , D, α, β) as

p(D, C|α, β)
p(Cm |C−m , D, α, β) =
p(D, C−m |α, β)
  n−(m) + βt
(−m) t,k
∝ nd,k + αk
(−m) .
 n
t t ,k  + βt

Here the superscript (−m) corresponds to not using the mth token in
counting; note that nd,k and nt,k are the counts where k is the topic,
d is the document, and t and t are terms. So, Gibbs sampling based
LDA essentially maintains various counters to store these count val-
ues. Basically, it randomly initializes K clusters/topics and iterates
in updating the probabilities specified by the above equation which
employs various counters and also in every iteration the counters are
suitably updated.

7.8. Concept and topic

Another related notion that is used in clustering is concept. A frame-
work called conceptual clustering is a prominent soft clustering
paradigm that employs classical logic and its variants to describe
clusters. For example, a cricket ball could be described using (color =
red ∨ white)∧(make = leather)∧(dia = medium)∧(shape = sphere)
which is a conjunction of internal disjunctions (or predicates). Each of
the conjuncts is viewed as a concept that describes a set of objects;
for example, (color = red) describes a set of red colored objects.
This framework is called conjunctive conceptual clustering. This is
also related to frequent itemset based clustering and classification.
For example, consider patterns of 7 and 1 shown in Table 8.9
wherein the left part is character 7 and in the right part character 1
is depicted; each is in the form of a 3 × 3 binary matrix (image).
In a realistic scenario the size of the matrix could be larger and the
number of 1 pixels will also be larger; possibly there could be noise
affecting the pixel values mutating a zero by a one and a one by
a zero. If there are n such character patterns with half of them from
class 7 and the other half from class 1, then it is possible to describe
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 317

Soft Clustering 317

Table 8.9. Printed characters of

7 and 1.
1 1 1 1 0 0
0 0 1 1 0 0
0 0 1 1 0 0
7 1

Table 8.10. Description of the concepts.

Concept i1 i2 i3 i4 i5 i6 i7 i8 i9
Class 1 1 0 0 1 0 0 1 0 0
Class 7 1 1 1 0 0 1 0 0 1

the classes as follows. Consider the 9 pixels in the 3 × 3 matrix and

label them as i1 , i2 , . . . , i9 in a row-major fashion. Then the frequent
itemset corresponding to 7 is i1 ∧ i2 ∧ i3 ∧ i6 ∧ i9 because this pattern
is present in half the number of characters given and the frequent
itemset in class 1 is i1 ∧ i4 ∧ i7 . So, it is possible to have the following
classification rules where the antecedent of the rule is the frequent
itemset and the consequent is the Class Label:

• Character 1: i1 ∧ i4 ∧ i7 → Class1
• Character 7: i1 ∧ i2 ∧ i3 ∧ i6 ∧ i9 → Class7

Equivalently one can describe these conjunctive concepts as

shown in Table 8.10. Note the similarity between concept which
describes a cluster or a class by selecting a subset of items or features
and topic which assigns a probability to each item or feature; both
describe clusters. In the example shown in Table 8.10 both the class
descriptions (concepts) select item i1 indicating softness.

Research Ideas
1. Derive an expression for the number of soft clusterings of n patterns into K soft
clusters.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 318

318 Introduction to Pattern Recognition and Machine Learning

2. Discuss Rough Fuzzy Clustering in the case of leader and what can happen to
other algorithms.

Relevant References

(a) S. Asharaf and M. N. Murty, An adaptive rough fuzzy single pass algorithm
for clustering large data sets. Pattern Recognition, 36:3015–3018, 2003.
(b) V. S. Babu and P. Viswanath, Rough-fuzzy weighted k-nearest leader clas-
sifier for large data sets. Pattern Recognition, 42:1719–1731, 2009.
(c) P. Maji and S. Paul, Rough-fuzzy clustering for grouping functionally simi-
lar genes from microarray data. IEEE/ACM Transactions on Computational
Biology Bioinformatics, 10:286–299, 2013.

3. A difficulty with the use of GAs is that they are not scalable. The problem gets
complicated further when one considers MOOP. How to design scalable GAs?

Relevant References

(a) A. Kink, D. Coit and A. Smith, Multi-objective optimization using

genetic algorithms: A tutorial. Reliability Engineering and System Safety,
91(9):992–1007, 2006.
(b) K. Amours, Multi-objective optimization using genetic algorithms.
Master’s thesis, Jinking University, 2012.
(c) D. Yumin, X. Shufen, J. Fanghua and L. Jinhai, Research and application
on a novel clustering algorithm of quantum optimization in server load
balancing. Mathematical Problems in Engineering, 2014.

4. Discuss the possibilities of abstracting soft clustering using the string-of-

centroids. Can we extend the string-of-group-representation to realize a soft
partition? How?

Relevant References

(a) U. Maulik and S. Bandyopadhyay: Genetic algorithm-based clustering tech-

nique: Pattern Recognition, 33:1455–1465, 2000.
(b) M. N. Murty, Clustering large data sets. In Soft Computing Approach to
Pattern Recognition and Image Processing, A. Ghosh and S. K. Pal (eds.),
pp. 41–63. New Jersey: World-Scientific, 2002.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 319

Soft Clustering 319

(c) L. Zhu, L. Cao and J. Yang, Multiobjective evolutionary algorithm-based

soft subspace clustering. Proceedings of IEEE Congress on Evolutionary
Computation, 2012.
5. EM is a probabilistic version of the K -Means algorithm. Why did EM became
so popular in Machine Learning?

Relevant References
(a) T. Hoffman, Latent semantic models for collaborative filtering. ACM Trans-
actions on Information Systems, 22(1):89–115, 2004.
(b) D. Sontag and D. M. Roy, Complexity of inference in latent Dirichlet allo-
cation. Proceedings of NIPS, 2011.
(c) S.-K. Ng, Recent developments in expectation-maximization methods for
analyzing complex data. Wiley Interdisciplinary Reviews: Computational
Statistics, 5:415–431, 2013.
6. Matrix factorization is useful in clustering. It is possible to show equivalence
between PLSA and NMF; similarly between K -Means and NMF. Is it possible
to unify clustering algorithms through matrix factorization?

Relevant References
(a) A. Roy Chaudhuri and M. N. Murty, On the relation between K-Means and
PLSA. Proceedings of ICPR, 2012.
(b) C. Ding, T. Li and W. Peng, On the equivalence between non-negative
matrix factorization and probabilistic latent semantic indexing. Computa-
tional Statistics and Data Analysis, 52:3913–3927, 2008.
(c) J. Kim and H. Park, Sparse nonnegative matrix factorization for clustering.
Technical Report, Georgia Technical, GT-CSE-08-01.pdf, 2008.
7. Is it possible to view clustering based on frequent itemsets as a matrix factor-
ization problem?

Relevant References
(a) B. C. M. Fung, K. Wang and M. Ester, Hierarchical document clustering
using frequent itemsets. Proceedings of SDM, 2003.
(b) G. V. R. Kiran, R. Shankar and V. Pudi, Frequent itemset-based hierarchical
document clustering using Wikipedia as external knowledge. Proceedings
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch08 page 320

320 Introduction to Pattern Recognition and Machine Learning

of Knowledge-Based and Intelligent Information and Engineering Systems,

Lecture Notes in Computer Science, Vol. 6277, 2010.
(c) J. Leskovec, A. Rajaraman and J. D. Ullman, Mining massive datasets,
http://infolab.stanford.edu/∼ullman/mmds/book.pdf, 2014.
8. LDA employs Dirichlet and multinomial conjugate pair of distributions. Is it
possible to use other distributions?

Relevant References
(a) D. M. Blei, Probabilistic topic models. Communications of the ACM, 55:77–
84, 2012.
(b) D. Newman, E. V. Bonilla and W. L. Buntine, Improving topic coherence
with regularized topic models. Proceedings of NIPS, 2011.
(c) H. M. Wallach, D. M. Mimno and A. McCallum: Rethinking LDA: Why
priors matter. Proceedings of NIPS, 2009.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 321

Chapter 9

Application — Social and Information Networks

1. Introduction
Social networks characterize different kinds of interactions among
individuals. Typically, a social network is abstracted using a
network/graph. The individuals are represented as nodes in a net-
work and interaction between a pair of individuals is represented
using an edge between the corresponding pair of nodes. Usually a
social network is represented as a graph. The nodes represent indi-
viduals or entities with attributes such as interests, profile, etc. The
interactions among the entities could be one of friendship, business
relationship, communication, etc.
These graphs could be either directed or undirected. Typically
friendship between two individuals is mutual; so, edges in a friendship
network are undirected. However, in influence networks the relation
may not be symmetric; a person A may influence B, but B may not
be able to influence A. So, in influence networks, the edges could
be directed. Note that author–co-author relation is symmetric and
network of authors is undirected whereas the citation network/graph
is directed.
Such a graphical representation can help in analyzing not only
the social networks but also other kinds of networks including
document/term networks. For example, in information retrieval typ-
ically a bag of words paradigm is used to represent document col-
lections. Here, each document is viewed as a vector of terms in the
collection; it only captures the frequency of occurrence of terms not
the co-occurrence of terms in the document collection. It is possible

321
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 322

322 Introduction to Pattern Recognition and Machine Learning

to extract a collection of co-occurring terms by viewing each term

as a node in a graph and co-occurrence of a pair of terms using an
edge between the corresponding nodes. Typically a clique of size l
in the graph characterizes a collection of l co-occurring terms; it is
possible to perform clustering of nodes in the graph to obtain such
co-occurring term collections. The corresponding operation is called
community detection in social networks.
Another important problem in social networks is link prediction.
In link prediction, a similarity function is used to predict whether two
nodes which are not currently linked can have a link between them
in the future. A popular subset, of such functions which compute
similarity between a pair of nodes, is based on the set of adjacent
nodes.

2. Patterns in Graphs
It is found that networks have some distinguishing features. The main
features associated with them are

• Follow power law distributions

• Have small diameters
• Exhibit community structure

According to the power law, two variables x and y are related by

a power law when

y(x) = Cx−γ ,

where C and γ are constants.

A random value is distributed according to a power law when the
probability distribution function is given by:

p(x) = Ax−γ , γ > 1, x ≥ xmin .

In the case of internet graphs, a number of parameters follow the

power law. Firstly, if we find the degree di of every node i and arrange
them in decreasing order of di , then if ri is the index of node i when
arranged in decreasing order (called the rank), then for every node i
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 323

Application — Social and Information Networks 323

1000

100

Out-degree 10

1 10 100 1000 10000

Rank

Figure 9.1. Log–log plot of out-degree versus rank.

we get

di ∝ riR .

R is called the rank exponent and is the slope of the plot of

out-degree of the nodes versus the rank drawn in log–log scale. If the
nodes are sorted in decreasing order of out-degree di and the log–log
plot of the out-degree versus the rank is plotted, the plot is as shown
in Figure 9.1.
Secondly, for every out-degree d we find the number of nodes
with that out-degree. This is called the frequency fd of the degree d.
If fd versus d is plotted for every degree d, we get

fd ∝ dO .

The constant O is the slope of the plot of frequency of

the out-degree versus the out-degree drawn in the log–log scale.
Figure 9.2 shows this plot.
Thirdly, if we find the eigenvalues of the adjacency matrix A
and arrange them in decreasing order then an eigenvalue λi has an
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 324

324 Introduction to Pattern Recognition and Machine Learning

10000

1000

100
Out-degree
10

1 10 100
Frequency

Figure 9.2. Log–log plot of out-degree versus frequency.

100

Eigenvalue
10

1
1 10
Order

Figure 9.3. Log–log plot of eigenvalue versus order.

order i. Then we get

λi ∝ iE .

E is called the eigenvalue exponent and is the slope of the eigen-

values versus the order in log–log scale. Figure 9.3 shows this plot.
It shows the log–log plot of the eigenvalues in decreasing order.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 325

Application — Social and Information Networks 325

1e+09

1e+08
N(h)
1e+07

1e+06 N(h): No. of pairs of

nodes within h hops
100000

10000

1000
1 10
h

Figure 9.4. Log–log plot of number of pairs of nodes within h hops versus h.

Fourthly, the total number of pairs of nodes N (h) which are

within h hops is proportional to the number of hops to the power of
a constant H. It can be represented as:

N (h) = hH .

H is called the hop-plot exponent. This is shown in Figure 9.4.

The effective diameter of a graph is the minimum number of
hops required for some fraction (say 90%) of all connected pairs of
nodes to reach each other.
A social network gives more importance to the topology and
ignores information pertaining to each node. Information networks
give importance to this information also. It is found that a group
of nodes interact strongly among themselves than with the outside
world. These groups of nodes are the communities or groups in the
network. It can be seen that the intra-group connectivity is much
more than the inter-group connectivity. A measure commonly used
for the notion of community is the conductance or the normalized
cut metric. The conductance of a community is the ratio of the num-
ber of cut edges between the set of nodes in the community and its
complement divided by the number of internal edges inside the set
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 326

326 Introduction to Pattern Recognition and Machine Learning

of nodes. It can thus be seen that if the conductance is small, then

it is a good community.

3. Identification of Communities in Networks

If we have a graph G = V, E, with |V |(=n) nodes and |E|(=e)
edges, a subset of the nodes S where S ⊂ V forms a commu-
nity if some score f (S) is optimized. This scoring function f (S)
depends on the internal connectivity between nodes in S and external
connectivity of nodes in S to other nodes.
Identifying communities in social and information networks is
useful as it helps in summarization, visualization, classification, and
link prediction. Some criteria which are used on the basis of which
the clustering is done are as follows:
Clustering Coefficient:
One of the related issues dealing with some notion of connectivity or
transitivity is the clustering coefficient. It may be viewed as the prob-
ability that two randomly selected neighbors of a node are neighbors
of each other. It is defined for node i as
2ei
Ci = ,
ki (ki − 1)
where ei is the number of links among the neighbors of i, ki is the
number of neighbors of i and ki (k2i −1) is the maximum number of
links possible among the ki neighbors of i. We illustrate it with the
help of the network shown in Figure 9.5. There are eight nodes in

5 8

3 4 1 6

2 7

Figure 9.5. An example network to illustrate clustering coefficient.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 327

Application — Social and Information Networks 327

Table 9.1. Clustering coefficients for the nodes.

Node number Set of neighbors Clustering coefficient

1 {2,4,5,7,8} 0.3
2 {1,3,4} 0.66
3 {2,4,5} 0.66
4 {1,2,3,5} 0.75
5 {1,3,4} 0.66
6 {7,8} 1
7 {1,6,8} 0.66
8 {1,6,7} 0.66

the network and the clustering coefficient for each of the nodes in
the network is given in Table 9.1. Note that the average clustering
coefficient is 0.67; this gives a measure of how the nodes can form
clusters.

Modularity:

Another important notion associated with clustering is Modularity

of the network. A network is said to be modular if it has a higher
clustering coefficient compared to any other network with the same
number of nodes assuming the appropriate degree of distribution.
It is possible to define modularity using the following notation:

• Let |E| be the total number of edges in the network.

• Let di and dj be the degrees of the vertices i and j.
• Let A be the adjacency matrix where Aij = 1 if there is a link
between nodes i and j and Aij = 0 otherwise.
• Let there be two communities in the network, C1 and C2 .
• Let Q be the modularity and it is given by
 
1
di dj
Q= Aij − .
2|E| 2|E|
i,j∈Cl

In some sense it characterizes how clustered some nodes are; the

summand for each ij pair consists of the difference between the Aij
value and the expected value based on the degrees di and dj .
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 328

328 Introduction to Pattern Recognition and Machine Learning

Graph clustering algorithms can be used to identify communities.

Some approaches for graph clustering are:

1. Graph partitioning
2. Spectral clustering
3. Linkage-based clustering
4. Hierarchical clustering
5. Random walks

3.1. Graph partitioning

Partitioning of the graph into smaller components is carried out such
that the number of edges connecting vertices from different compo-
nents is minimized. One method for doing this is Multi-level recursive
bisection where the actual graph is coarsened into a much smaller
graph. The communities are computed in this smaller graph and
projected back to the original graph.
Some graph partitioning algorithms which are used for commu-
nity detection are flow-based methods, spectral methods and hierar-
chical methods.
One way of partitioning the graph is to use bisection of the graph.
If more clusters are needed, bisection can be done iteratively.
One of the earlier methods starts with an initial partition which is
improved over a number of iterations. The parameter to be optimized
is a benefit function Q which is the difference between the number of
edges in the clusters and the number of edges between the clusters.
Nodes are swapped between clusters in such a way that Q increases.
To avoid the local maxima of Q, swaps giving lower Q values are also
accepted. Finally, the partition giving largest Q is accepted. This
procedure is used to get a bisection of the graph. Then a cluster is
chosen which is divided into two smaller clusters and this goes on till
there are a predefined number of clusters.
According to the max-flow min-cut theorem, the minimal subset
of edges whose deletion would separate two nodes s and t carries
the maximum flow that can be transported from s to t across the
graph. An artificial sink t is added to the graph and the maximum
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 329

Application — Social and Information Networks 329

flow from a source vertex s to t is calculated which helps to identify

the minimum cut separating the graph into communities.
Graphs can also be partitioned by minimizing the conductance.
In a graph G, consider a subgraph G1 . Then the conductance C of
G1 is
no(G1 , G\G1 )
C(G1 ) = ,
min(dG1 , dG\G1 )
where no(G1 , G\G1 ) is the cut size of G1 and dG1 and dG\G1 are
the total degrees of G1 and the total degree of the rest of the graph
respectively.

3.2. Spectral clustering

The similarity matrix for the graph is computed. The eigenvectors of
this matrix are used to partition the graph into clusters. The unnor-
malized and normalized graph Laplacian is used for the partitioning.
Unnormalized graph Laplacian L is given by
L = D − W,
where W is the weight matrix such that for some i and j, wij = wji
gives the weight between two nodes. D is the degree matrix which is
a diagonal matrix with the degrees d1 , . . . , dn on the diagonal.
The matrix L is symmetric and positive semi-definite. L has n
non-negative, real eigenvalues and the smallest eigenvalue of L is 0.
This means
0 = λ 1 ≤ λ2 ≤ · · · ≤ λ n .
Also for any vector vεRn
1

n
T
v Lv = wij (vi − vj )2 .
2
i,j=1

Spectral clustering is carried out by finding the k eigenvectors of

L where k is the number of clusters. The n data points are trans-
formed using the k eigenvectors and clustered using the k-means
algorithm.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 330

330 Introduction to Pattern Recognition and Machine Learning

The normalized graph Laplacian can be got in two ways. The

first matrix is Lsym which is a symmetric matrix and Lrw which
corresponds to a random walk. These can be computed as:
1 1 1 1
Lsym = D − 2 LD− 2 = I − D− 2 W D− 2 ,
Lrw = D −1 L = I − D −1 W.

Some properties of the normalized graph Laplacian are to be

noted. λ is an eigenvalue of Lrw with eigenvector u if and only if
1
λ is an eigenvalue of Lsym with eigenvector w = D 2 u. Also λ is an
eigenvalue of Lrw with eigenvector u if and only if λ and u solve
Lu = λDu.
0 is an eigenvalue of Lrw and Lsym . Both Lrw and Lsym are pos-
itive semidefinite and have n non-negative real valued eigenvalues
0 = λ1 ≤ · · · ≤ λn .
Additionally, if we take a vector vεRn , we get
 2
1

n
vi vi
T
v Lsym v = wij  − .
2 (d i ) (d j )
i,j=1

Normalized spectral clustering is done in two ways.

Method 1:
The unnormalized Laplacian L is computed using the weight matrix
W and the degree matrix D. If there are k clusters, find the first
k eigenvectors of the generalized eigen problem Lu = λDu. The
n data points are transformed using the k eigenvectors and these
points are clustered using k-means algorithm as discussed in the
chapter on clustering.
Method 2:
The normalized Laplacian Lsym is computed. Using Lsym , the first
k eigenvectors u1 , . . . , uk are computed. If U εRn×k is the matrix con-
taining u1 , . . . , uk as the columns, a new matrix P εRn×k is formed
 1
from U by normalizing the rows, i.e. pij = uij /( k u2ik ) 2 . The data-
points are transformed using the P matrix. These points are clustered
using k-means algorithm.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 331

Application — Social and Information Networks 331

3.3. Linkage-based clustering

The similarity between two objects is recursively defined as the aver-
age similarity between objects linked with them. The links between
different objects with the similarities are used for clustering the
graph.
If the edges that connect vertices of different communities
are identified, then by removing those edges, communities can be
identified.
One of the earliest methods use the edge betweenness. This is
called the Girvan–Newman algorithm. The edge centrality is mea-
sured according to some property of the graph. The edge with the
largest centrality is removed. The edge centralities are again mea-
sured and the edge with the largest centrality is again removed. Edges
are found which have highest “betweenness measure” for dividing the
network. One measure of betweenness is the shortest-path between-
ness. Here the shortest paths between all pairs of vertices is found
and it is necessary to count how many run along each edge. The other
measure of betweenness is the current-flow betweenness. A circuit is
created by placing a unit resistance on each edge of the network and
unit current source and sink at the pair of vertices being considered.
The resulting current flow in the network will travel along the various
paths where the edges with least resistance carry more current. The
current-flow betweenness for an edge is the absolute value of the
current aggregated over all source/sink pairs. After the edge with
highest betweenness measure is removed, the betweenness measure
is recalculated to find the next edge to remove.

3.4. Hierarchical clustering

Social networks generally have a hierarchical structure where smaller
communities are embedded within larger communities. This will be
reflected in the dendrogram. Depending on the level in which the
dendrogram is considered, different partitions will result.
One hierarchical clustering algorithm that employs agglomera-
tive clustering based on random walk to find the similarity measures
between vertices is considered next.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 332

332 Introduction to Pattern Recognition and Machine Learning

3.4.1. Random walks

In this algorithm, a measure of similarity between vertices is found
based on random walks. This measure can be used in agglomerative
clustering to determine the communities. A discrete random walk
process on a graph is one where at each time step a walker is on a
vertex and moves to a vertex chosen randomly and uniformly among
its neighbors. The sequence of vertices visited is a Markov Chain.
At each step, the transition probability from vertex i to vertex j is
A
Pij = Dijii . By doing this, we can get the transition matrix P which
can be written as P = D−1 A where A is the adjacency matrix and D
is a diagonal matrix with the degree of the nodes along the diagonal.
To find the distance r between nodes, note the following:
• If two vertices i and j belong to the same community, then Pijt will
be high where t gives the length of the random walk.
• Pijt is influenced by the degree Djj as the walker has a higher
probability to go to the node of higher degree.
• For two vertices i and j belonging to the same community, prob-
ably ∀k, Pik
t
 Pjk t
.
The distance rij between two vertices will be

n  

(Pik t − P t )2
 1 1 
rij =
= D− 2 Pi.t − D − 2 Pj.t .
jk
Dkk
k=1

The distance between two communities C1 and C2 can be defined

as

n

(PCt 1 k − PCt 2 k )2  1 1


rC1 C2 =
= D − 2 PCt 1 . − D − 2 PCt 2 . .
Dkk
k=1

The probability to go from community C to vertex j is given by

t 1
t
PCj = Pij .
|C|
iεC

The algorithm used is the agglomerative clustering. At first, each

vertex belongs to a different clusters. At each step k two communities
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 333

Application — Social and Information Networks 333

C1 and C2 are chosen to be merged together. The two communities

to be chosen has to minimize the mean σk of the squared distances
between each vertex k and its community. This means

1

2
σk = riC .
n
CεPk iεC

This is done by computing the variation σ(C1 , C2 ) for every

pair of adjacent communities {C1 , C2 }. This is given by
 
1



σ(C1 , C2 ) =  2
riC3 − 2
riC1
− 2 
riC 2
.
n
iεC3 iεC1 iεC2

Two communities which give the lowest value of σ are merged

at each step.

3.4.2. Divisive hierarchical clustering

This method finds edges which have highest “betweenness” measure
for dividing the network. One measure of betweenness is the shortest-
path betweenness. Here, the shortest paths between all pairs of ver-
tices is found and it is necessary to count how many runs along each
edge. The other measure of betweenness is the current-flow between-
ness. A circuit is created by placing a unit resistance on each edge
of the network and unit current source and sink at the pair of ver-
tices being considered. The resulting current flow in the network will
travel along the various paths where the edges with least resistance
carry more current. The current-flow betweenness for an edge is the
absolute value of the current aggregated over all source/sink pairs.
After the edge with the highest betweenness measure is removed, the
betweenness measure is recalculated to find the next edge to remove.

3.5. Modularity optimization for partitioning graphs

High values of modularity indicate good partitions. An optimization
of the modularity Q of a graph is intractable as there are a very
large number of ways in which a graph can be partitioned. Some
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 334

334 Introduction to Pattern Recognition and Machine Learning

approximation algorithms are used instead to get the partition with

the maximum modularity. Some of them are:

3.5.1. Iterative greedy methods

In one technique, an agglomerative clustering technique is used. At
first there are n clusters if there are n vertices, i.e. each cluster con-
sists of a single vertex. First one edge is connected making the num-
ber of clusters from n to n − 1, giving a new partition of the graph.
The edge is chosen such that the partition formed gives the max-
imum increase in modularity. At each iteration, an edge is added
so that the modularity increase is maximum. It is to be noted that
if it is an internal edge of a cluster, then the modularity does not
change.
Since calculation of ∆Q for each edge requires a constant time,
this calculation requires O(m) where m is the number of edges. After
merging the chosen communities, the matrix which indicates the frac-
tion of edges between two clusters i and j of the running partition
is updated in O(n). Since the algorithm runs for n − 1 iterations,
its complexity is O((n + n)n) which is O(n2 ) for sparse graphs. An
improvement on the time is obtained when the ∆Q values are stored
in a max-heap and the matrix which expresses the fraction of edges
between clusters is replaced by an array which contains the sums
of the elements of each row. The complexity of the algorithm is
O(md log n) where d is the depth of the dendrogram which shows
the successive partitions found as the iterations progress. For graphs
with a strong hierarchical structure, the complexity is O(n log2 n).
The above algorithm tends to form large communities rather than
smaller communities which leads to poor values of Q. One method to
address this problem is to normalize the increase in modularity ∆Q
by dividing it with the fraction of edges incident on one of the com-
munities. Another method is to choose the communities to be merged
at each step by finding the largest value of ∆Q times a factor. This
factor known as the consolidation ratio is larger for communities of
equal size. Another variation to take care of the problem of larger
communities is to allow more than one community pair to be merged
in each iteration. It also allows the movement of single nodes from a
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 335

Application — Social and Information Networks 335

community to a neighboring community. Another method of greedy

optimization is to start from some promising intermediate configu-
ration rather than from individual nodes which is measured by the
topological similarity between clusters. Using an algorithm which
alternates between greedy optimization and stochastic perturbation
of the partitions is known to be a good strategy.
Another greedy approach initially puts all the vertices of the
graph in different communities and conducts a sequential sweep over
the vertices. For a vertex i, the gain in modularity from putting i in
the neighboring communities is found. The community giving largest
increase in Q is selected to move i. At the end of the sweep, a first
level partition is obtained. The communities are then replaced by
supervertices, and supervertices are connected if there is at least an
edge between the nodes of the two communities. The weight given
to the edge is the sum of the weights of the edges between the com-
munities at the lower level. The two steps of sequential sweep and
forming supervertices is repeated forming new hierarchical levels and
different supergraphs. When modularity does not increase any fur-
ther, the iterations are stopped. This method is very fast and the
time complexity is of the order of O(m) where m is the number of
edges.

3.5.2. Simulated annealing (SA)

In SA, the solution is found by the method of search. A string Sc
represents the trial solution. This is called the current solution which
is evaluated to get Ec . A new trial solution St is found which is a
neighbor of the current solution. Its evaluation Et is found. If Et > Ec
then St is accepted as the new current solution. If Et ≤ Ec , St is
accepted as the new current solution with a probability p given by
 
−∆E
p = exp ,
T
where ∆E = Ec − Et and T is the temperature which is high at the
beginning so that p is high. As the iteration progress, T is reduced
using the cooling rate α which has a typical value of 0.9 or 0.95. Every
few iterations, T = α ∗ T is carried out. As T reduces the probability
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 336

336 Introduction to Pattern Recognition and Machine Learning

p of accepting a solution which is worse than the current solution

decreases. The procedure is stopped when T becomes very small.
SA has been used for modularity optimization where the current
solution is a clustering of the data. The moves carried out are the
local moves where a single node is moved from one cluster to another
and global moves which consist of mergers and splits of clusters. The
evaluation of a solution is done by finding the modularity increase.
It is to be noted that the method is slow and can be used only for
small graphs of upto 104 nodes.

3.5.3. Extremal optimization

Extremal Optimization is a heuristic technique where the contribu-
tion of local variables to the overall modularity of the system is stud-
ied and optimized. For every vertex the local modularity is divided by
the degree of the vertex to get a fitness measure. Initially the graph is
randomly partitioned into two groups, each having equal number of
nodes. At each iteration, the fitness measure of every node is calcu-
lated. The node with the lowest fitness is shifted to the other cluster.
The local fitness values are again recalculated and new partitions are
formed by moving a node. This process is repeated till there is no
improvement in the global modularity Q. This process is likely to
sometimes give local optima. This can be improved if probabilistic
selection is carried out. The nodes are ranked according to their
fitness value and the vertex of rank r is selected with probability
P (r) = r−α ,
where α is a constant. Sorting the fitness values is O(n log n). Choos-
ing the node to be moved is O(log n). Checking whether the mod-
ularity can be improved is O(n). The complexity of this method is
O(n2 log n). This method gives accurate results in reasonable time.

3.5.4. Spectral optimization

In this method, a modularity matrix M is formed whose elements
are:
di dj
Mij = Aij − ,
2m
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 337

Application — Social and Information Networks 337

where A is the adjacency matrix, m is the number of edges in the

graph and di and dj are the degrees of the nodes i and j.
The modularity Q can be written as:
 
1
di dj
Q= Aij − δCi ,Cj , (1)
2m 2m
ij

where δCi ,Cj is the Kronecker delta. It takes the value 1 if Ci = Cj

i.e. if the two points belong to the same community.
In the case of a partition of the graph into two clusters, if ui = +1
when vertex i belongs to cluster C1 and si = −1 when vertex i belongs
to cluster C2 , the Eq. (1) can be written as
 
1
di d j
Q= Aij − (ui uj + 1)
4m 2m
ij

1
1 T
= Bij ui uj = u Bu.
4m 4m
ij

The vector u can be expressed using the eigenvectors vi of the

modularity matrix B. We get


u= (viT · u)vi . Q then becomes
i

1

1
T
n
Q= (vi · u)viT B (vj · u)uj = (vi · u)2 βi ,
4m 4m
i j i=1

where βi is the eigenvalue of B corresponding to the eigenvector vi .

It is to be noted that B has the trivial eigenvector (1, 1 . . . , 1)
with eigenvalue zero. Hence if B has no positive eigenvalues, only
the trivial solution exists which consists of a single cluster with Q0 .
Otherwise, the eigenvector of B corresponding to the largest positive
eigenvalue, v1 and group the nodes according to the signs of the
components of v1 . This result can be further improved if nodes are
shifted from one community such that modularity increases. The
drawback of the spectral bisection is that it gives good results for
two communities but is not so accurate for larger number of com-
munities. To take care of this, more than one eigenvector can be
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 338

338 Introduction to Pattern Recognition and Machine Learning

used. Using the first p eigenvectors, n p-dimensional vectors can be

constructed, each corresponding to a node. The components of the
vector of node i is proportional to the p values of the eigenvectors in
position i. Community vectors can be found by summing the vectors
of the nodes in a community. Then if two community vectors form
an angle larger than π2 , keeping the two communities separate yields
larger modularity.
Spectral optimization of modularities to find communities is
O(n(m + n)) which is O(n2 ) on a sparse graph.
The above methods can be extended to graphs with weighted
edges where the adjacency matrix A is replaced by the weight
matrix W . We get the equation:
1
 ui uj 
Q= Wij − δ(Ci , Cj ).
2W 2W
ij

It can also be extended to directed graphs where di dj is replaced

by dout in in
i dj where d is the indegree and d
out
is the out-degree. The
expression becomes:
 
1
dout in
i dj
Q= Aij − δ(C1 , C2 ).
m m
ij

Another way of spectral optimization uses the weighted adja-

cency matrix W and an assignment matrix X which is n × k where k
is the number of clusters. In X, xij = 1 if node i belongs to cluster j
and zero otherwise. The modularity can be written as:

Q ∝ tr[X T (W − D)X] = −tr[X T LQ X].

Here entries of D are Dij = di dj . LQ = D − W is called the

Q-Laplacian. Setting the first derivative of Q with respect to X to
zero, we get

LQ X = X· C, (2)

where C is a diagonal matrix. Equation (2) uses the Q-Laplacian

matrix to carry out modularity maximization. For larger graphs, the
Q-Laplacian can be approximated by the transition matrix W  which
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 339

Application — Social and Information Networks 339

is obtained by normalizing W such that the sum of the elements of

each row is one. The first k eigenvectors of the transition matrix W 
can be computed where k is the number of clusters. This can be used
to find the co-ordinates of the graph nodes which can be clustered
using k-means clustering.

3.5.5. Mathematical programming

Modularity optimization can be formulated as a linear or quadratic
program. The linear program is defined on the links and the modu-
larity matrix B. It can be written as:


Q∝ Bij (1 − xij ), (3)
ij

where xij = 1 if i and j belong to the cluster and zero otherwise.

Equation (3) is an integer programming problem and is NP-hard.
However, if x is made real-valued, it can be solved in polynomial
time. Since the solution does not correspond to an actual partition
as the values of x are fractional, a rounding step is required. The
values of the x variables is used as a sort of distance and those close
to each other in the x values are put in the same cluster.
Quadratic programming can also be used. The equation used is:

1

Q= Bij (1 + ui uj ),
4m
ij

where ui = 1 if vertex i belongs to cluster 1 and ui = 0 if vertex i

belongs to cluster 2.
This problem is NP-complete, and can be solved in polynomial
time only if u is allowed to be real. Each scalar u is transformed into
a n-dimensional vector u and a product such as ui uj is a scalar prod-
uct between vectors. The vectors are normalized so that their tips are
on a unit-sphere of the n-dimensional space. An (n − 1)-dimensional
hyperplane centered at the origin cuts the space in two giving two
subsets for the vectors. The hyperplane which gives the highest mod-
ularity is picked.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 340

340 Introduction to Pattern Recognition and Machine Learning

3.5.6. Mean field annealing

This method is an alternative to SA. Gibbs probabilities are used
to compute the conditional mean value for a node. This is the com-
munity membership of the node. Mean field approximation is made
on all the nodes to find a consistent set of nonlinear equations in an
iterative way. This requires a time of O[n(m + n)].

3.5.7. Genetic algorithms

The chromosome in the genetic algorithm gives the partition to which
each node belongs. At each step, a bipartition of the graph is car-
ried out. For the original two partitions and the successive biparti-
tions, a genetic algorithm is applied on the graph/subgraphs formed.
The fitness function of each chromosome is the modularity of the
partition.

4. Link Prediction
There are several applications associated with networks/graphs
where predicting whether a pair of nodes X and Y which are not
currently connected will get connected (or have a link) in the future
is important. Based on the type of objects the nodes represent in
a network, we may have either a homogeneous or a heterogeneous
network.

• Homogeneous network: Here, all the nodes represent the same

type of objects. For example, in a social network each node
represents an individual and the links characterize the type of
interaction; so, such a network is homogeneous. Another example
of a homogeneous network is the citation network where each node
represents a publication and a directed link from node X to node Y
indicates that publication represented by X is citing the publica-
tion corresponding to Y .
• Heterogeneous network: In a heterogeneous network nodes may
correspond to different types of objects. For example, in an author-
paper network some nodes may correspond to authors and others
correspond to papers. There could be a co-author link between two
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 341

Application — Social and Information Networks 341

authors. Also there could be a link between an author and a paper.

So, not only nodes, even the edges could be of different types.

Even though these two types could be different in terms of nodes

and edges, generically a proximity function is used to predict the
possibility of a link between a pair of nodes. In the current chapter
we consider only homogeneous networks. Also we consider simple
graphs where it is not possible to have multiple edges between a pair
of nodes, even though the graphs need not be simple in general. For
example, a pair of individuals could have co-authored more than one
paper together; if there is one link for each co-authored paper, then
the graph could be a multi-graph. Further in the discussion here we
consider undirected graphs only.
Some of the specific applications of link prediction are:

• Suggesting friends in a social network. Two individuals have a

better possibility of becoming friends even if they are not currently
so provided they have a larger number of common friends.
• Recommending collaborators: It involves recommending a col-
laborator Y to an employee X where X and Y are from the same
or different organizations.
• Monitoring Terrorist Networks: It identifies possible links
between terrorists; this may involve predicting links that are not
explicit currently.
• Recommending a movie or a book to a user based on a pos-
sible link or association between a person and a movie or a book
which requires link prediction in heterogeneous networks; such a
network will have as nodes people, movies, and books.

4.1. Proximity functions

Predicting the link between a pair of nodes is inherently characterized
by some notion of proximity between them. Such proximity functions
should be based on the knowledge of how the network evolves. By
proximity we mean either similarity or distance/dissimilarity. Two
currently unconnected nodes X and Y are highly likely to have a link
between them if similarity between them is high or correspondingly
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 342

342 Introduction to Pattern Recognition and Machine Learning

distance/dissimilarity between them is low. Some of the popular local

similarity measures, between a pair of nodes X and Y , are based on
the nodes that are directly linked to both X and Y ; such nodes are
typically called common neighbors of X and Y . Another intuitively
appealing proximity function is based on some weighted distance
between the two nodes. We discuss below some of the popularly used
proximity functions.
Two important types of functions for characterizing proximity
between a pair of nodes are based on similarity or those based on
distance. We explain some of these functions using the example net-
work shown in Figure 9.6.

4.1.1. Similarity functions

Typically the similarity functions are based on the common neighbors
as they depend on information corresponding to the two nodes; so,
they are called local similarity functions as they use the information
local to the nodes. Let X and Y be two nodes in the network. Let

• N (X) = set of neighbors of X = {t|there is a link between X and t}.

• Z = N (X) ∩ N (Y ) = {z|z ∈ N (X) and z ∈ N (Y )}.
Note that Z is the set of common neighbors. The related similarity
functions, where S(X, Y ) is the similarity between X and Y , are

6 7

5 8

3 4 1 9

2 10

Figure 9.6. An example network.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 343

Application — Social and Information Networks 343

1. Preferential Attachment: Here, the similarity Spa (X, Y ) is pro-

portional to the product of the degrees of X and Y . It is given by

Spa (X, Y ) = |N (X)| × |N (Y )|.

This function encourages links between high degree nodes. The

similarity values for the missing links in the example graph are:

• Spa (1, 3) = 6 × 4 = 24
• Spa (1, 6) = Spa (1, 9) = 12
• Spa (3, 7) = Spa (3, 8) = Spa (3, 9) = Spa (3, 10) = 8
• Spa (2, 4) = Spa (2, 5) = Spa (4, 5) = Spa (2, 6) = 4
• Spa (2, 7) = Spa (2, 8) = Spa (2, 9) = Spa (2, 10) = 4
• Spa (4, 6) = Spa (4, 7) = Spa (4, 8) = Spa (4, 9) = Spa (4, 10) = 4
• Spa (5, 6) = Spa (5, 7) = Spa (5, 8) = Spa (5, 9) = Spa (5, 10) = 4
• Spa (6, 8) = Spa (6, 9) = Spa (6, 10) = 4
• Spa (7, 8) = Spa (7, 9) = Spa (7, 10) = Spa (8, 10) = 4

So, the link (1, 3) has the largest similarity value; this is followed
by (1, 6) and (1, 9). Also note that this function considers possible
links between every pair of nodes that are not linked currently.

2. Common Neighbors: The similarity Scn (X, Y ) is

Scn (X, Y ) = |N (X) ∩ N (Y )|.

This captures the notion that two people will become friends if
they share a large number of friends or they have a large num-
ber of common friends. Note that for the missing links (currently
unconnected nodes) in the example graph

• Scn (1, 3) = |{2, 4, 5}| = 3

• Scn (1, 9) = Scn (2, 4) = Scn (2, 5) = Scn (4, 5) = Scn (8, 10) = 2
• Scn (1, 6) = Scn (2, 6) = Scn (2, 7) = Scn (2, 8) = Scn (2, 10) =
Scn (3, 7) = 1
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 344

344 Introduction to Pattern Recognition and Machine Learning

• Scn (5, 8) = Scn (5, 10) = Scn (4, 6) = Scn (4, 7) = Scn (4, 8) =
Scn (4, 10) = 1
• Scn (5, 6) = Scn (5, 7) = Scn (5, 8) = Scn (5, 10) = Scn (7, 8) =
Scn (7, 10) = 1
Observe that the node pairs with zero similarity value are not
shown; for example, Scn (6, 8) = 0. Based on these similar-
ity values, we can make out that the pair of nodes 1 and 3
has the largest similarity value of 3; this is followed by pairs
(1, 9), (2, 4), (2, 5), (2, 8), (2, 10), and so on. Here, the link (1, 9)
has a larger similarity value compared to the link (1, 6). Note
that the similarity function is symmetric because the graph is
undirected.

3. Jaccard’s Coefficient: The similarity function Sjc may be

viewed as a normalized version of Scn . Specifically, Sjc (X, Y ) is
given by

|N (X) ∩ N (Y )| Scn (X, Y )

Sjc (X, Y ) = = .
|N (X) ∪ N (Y )| |N (X) ∪ N (Y )|

So, this similarity function gives importance to those pairs of

nodes with smaller degree. If two pairs (X1 , Y1 ) and (X2 , Y2 ) have
the same number of common neighbors but if |N (X1 ) ∪ N (Y1 )| <
|N (X2 ) ∪ N (Y2 )| then the pair (X1 , Y1 ) has larger similarity than
the pair (X2 , Y2 ). Note that the scores for the missing links in the
example graph are:
2
• Sjc (8, 10) = Sjc (2, 5) = Sjc (2, 4) = Sjc (4, 5) = 2 =1
|{2,4,5}| 3
• Sjc (1, 3) = |{2,4,5,6,7,8,10}
= 6 = 0.5
1
• Sjc (1, 9) = Sjc (2, 6) = Sjc (2, 7) = Sjc(2, 8) = Sjc(2, 10) = 3
1
• Sjc (4, 6) = Sjc (4, 7) = Sjc (4, 8) = Sjc(4, 10) = Sjc (5, 6) = 3
1
• Sjc (5, 7) = Sjc (5, 8) = Sjc (5, 10) = Sjc (7, 10) = Sjc(7, 8) = 3
1
• Sjc (3, 7) = 5
1
• Sjc (1, 6) = 7
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 345

Application — Social and Information Networks 345

Note that unlike the common neighbors function which ranks the
link (1, 3) above the others the Jaccard coefficient ranks the links
(2, 4), (2, 5), (4, 5), (8, 10) (with similarity 1) above the pair (1, 3)
(similarity value is 0.5). Also links between 1 and 9 and 1 and 6
have lesser similarity values.
4. Adamic–Adar: This similarity function Saa may be viewed as
a weighted version of the common neighbors similarity function.
It gives less importance to high degree common neighbors and
more importance to low degree common neighbors. The similarity
function is given by

1
Saa (X, Y ) = .
log|N (z)|
z∈N (X)∩N (Y )

For the given example graph, the similarity values are:

1 1 1 3
• Saa (1, 3) = log2 + log2 + log2 = 0.3 = 10
• Saa (1, 6) = 3.3; Saa (1, 9) = 6.6
• Saa (2, 4) = Saa (2, 5) = Saa (4, 5) = 2.9; Saa (2, 6) = 1.65
• Saa (2, 7) = Saa (2, 8) = Saa (2, 10) = 1.29
• Saa (3, 7) = 3.3; Saa (4, 6) = Saa (5, 6) = 1.65
• Saa (4, 7) = Saa (4, 8) = Saa (4, 10) = Saa (5, 7) = 1.29
• Saa (5, 8) = Saa (5, 10) = Saa (7, 8) = Saa (7, 10) = 1.29
• Saa (8, 10) = 4.59
Note that Scn (1, 9) = Scn (2, 4) = 2 whereas Saa (1, 9) = 6.6 and
Saa (2, 4) = 2.9. Also the similarity between 1 and 6 is 3.3. So, the
similarity functions can differ; different similarity functions can
give rise to different ranks.
5. Resource Allocation Index: This also is a weighted version of
the common neighbors function; here the contribution of a com-
mon neighbor is the reciprocal of its degree. The function is

1
Sra(X, Y ) = .
|N (z)|
z∈N (X)∩N (Y )
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 346

346 Introduction to Pattern Recognition and Machine Learning

The similarity values of links in the example graph are:

1 1 1 3
• Sra (1, 3) = 2 + 2 + 2 = 2 = 1.5
• Sra (1, 6) = 0.5; Sra (1, 9) = 1
5
• Sra (2, 4) = Sra (2, 5) = Sra (4, 5) = 12 ; Sra (2, 6) = 0.25
• Sra (2, 7) = Sra (2, 8) = Sra (2, 10) = 0.17
• Sra (3, 7) = 0.5; Sra (4, 6) = Sra (5, 6) = 0.25
• Sra (4, 7) = Sra (4, 8) = Sra (4, 10) = Sra(5, 7) = 0.17
• Sra (5, 8) = Sra (5, 10) = Sra(7, 8) = Sra(7, 10) = 0.17
• Sra (8, 10) = 0.67
Observe that the ranking is similar to that provided by the
Adamic–Adar function in this example; the major difference is
that the weights come from a larger range in the case of Adamic–
Adar compared to that of Resource Allocation Index as the
denominators corresponding to each common neighbor are smaller
as they are compressed logarithmically in the case Saa .

4.1.2. Distance functions

One may use some kind of distance/dissimilarity between nodes X
and Y to explore the possibility of having a link between them. Based
on the pair of nodes being considered it may be required to con-
sider several nodes in the network to compute the distance; so these
distance functions may be characterized as looking for some global
dissimilarity. Also it is not uncommon to convert the distance values
to the corresponding similarity values and use the similarity values.
Larger the distance between the two nodes X and Y smaller the
similarity S(X, Y ) and vice versa. Some of the distance functions
that are popular are:

1. Graph Distance: Here the distance is the length of the shortest

path between the two nodes; typically the negated value of the
distance is used to characterize the similarity. Specifically,

Sgd (X, Y ) = −length of the shortest path(X, Y ).

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 347

Application — Social and Information Networks 347

The similarity values between some of the unconnected links in

the example graph are:
• Sgd (1, 3) = Sgd (1, 6) = Sgd (1, 9) = −2
• Sgd (2, 9) = Sgd (3, 8) = Sgd (4, 9) = Sgd (5, 9) = −3
• Sgd (3, 9) = Sgd (6, 9) = −4
Note that in this case all the three links (1, 3), (1, 6), (1, 9) are
equally ranked unlike any of the other functions.
2. Katz Similarity: It accumulates all the paths of the same length
between the two nodes and weighs appropriately the number of
such paths to get the distance. Specifically, it is
∞

 (l) 
Skd (X, Y ) = − β l · pathX,Y ,
l=1

(l)
where pathX,Y is the set of all paths of length l between X and
Y . The similarity for some of the pairs in the example graph are:
• Sks (1, 3) = 2β 2 + β 3 = 0.02 + 0.001 = 0.021
• Sks (1, 9) = 2β 2 = 0.02
• Sks (1, 6) = β 2 + 3β 3 = 0.013
• Sks (7, 8) = β 2 + β 3 = 0.011
Note that the similarity values are computed using a value of 0.1
for β; the similarity between 1 and 6 is different from that between
1 and 9.
3. SimRank: It may be viewed as a recursive version of simple sim-
ilarity. It is given by
 
Ssr (P, Q)
P ∈N (X) Q∈N (Y )
Ssr (X, Y ) = .
|N (X)| · |N (Y )|

5. Information Diffusion
Any information communicated in a social network spreads in the
network. The study of how this information gets disseminated in the
network is called information diffusion. The shape of the network is
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 348

348 Introduction to Pattern Recognition and Machine Learning

influenced by the way the information diffusion occurs. It is neces-

sary to model the process of information diffusion. The movement
of information from one individual to another, from one location to
another, from one node in the network to another either in the same
community or another community is called information diffusion or
information propagation. Information diffusion helps in several areas
such as marketing, security, search etc. In marketing, for example, it
is useful to know in what way to advertise a product so as to have
better information diffusion. In security, criminal investigators need
to understand how and where the criminal information is spreading
in a community. It can thus be seen that we need to answer questions
such as what type of information or topics are popular and diffuse
the most, which are the paths through which the diffusion is taking
place and will take place in the future and which are the users of
the social network who play important roles in the diffusion. It is
necessary to predict the temporal dynamics of information diffusion.
In information diffusion, a closed world assumption is made
which means that the information only propagates from node to node
along the edges in the network and nodes are not influenced by exter-
nal sources. This is not always true. Though most of the influence
is from internal nodes, there is some influence by external nodes.
Another assumption generally made is that the diffusion processes
are independent. However, this may not be true and there is a model
proposed where different diffusion processes interact with each other.
The emphasis of information diffusion can be such that it is sender-
centric or receiver-centric.
In diffusion, the activation sequence gives an ordered set of
nodes which gives the order in which the nodes of the network adopt
a piece of information. A spreading cascade is a directed tree with
the first node of the activation sequence as the root. The tree shows
the influence between nodes and captures the same order as the acti-
vation sequence.

5.1. Graph-based approaches

In the graph-based approach the topology of the process in a graph
structure is used. The two models are (i) Independent Cascades (IC)
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 349

Application — Social and Information Networks 349

and (ii) linear threshold (LT). Both these models are synchronous
along the time-axis. IC is sender-centric whereas LT is receiver-
centric.

5.1.1. Independent cascade model

In IC a directed graph is used where each node is either activated
(or informed) or not. Monotonicity is assumed which means that
activated nodes cannot be deactivated. Each edge in the graph is
associated with a diffusion probability. The diffusion process proceeds
along a discrete time-axis. In each iteration, the nodes which are
newly activated try to activate their neighbors with the diffusion
probability defined on the edge. The activations made are effective
at the next iteration. This process is continued till there are no new
nodes which can be activated.

5.1.2. Linear threshold model

In LT also a directed graph is used where the nodes are either acti-
vated or not. Each edge has an influence degree defined and a influ-
ence threshold for each node. At each iteration, inactive nodes are
activated by their active neighbors if the sum of the influence degrees
is higher than its influence threshold. The new activations are consid-
ered from the next iteration. When no new activations are possible,
the process ends.
Both IC and LT proceed in a synchronous manner which is not
true in a social network. Asynchronous versions of IC and LT are
AsIC and AsLT; they use a continuous time line and use a time-
delay parameter for each edge.

5.2. Non-graph approaches

5.2.1. SIR and SIS
In non-graph approaches there are two models called SIR and SIS
where S stands for “susceptible”, I stands for “infected” and R stands
for “recovered”. Nodes in S class move to I class with a fixed prob-
ability β. Then in SIS, nodes in I class move to S class with a fixed
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 350

350 Introduction to Pattern Recognition and Machine Learning

probability γ and in the case of SIR, they switch permanently to the

R class. Connections are made at random as every node as the same
probability to be connected to another node.

5.2.2. Using influence of nodes

Another approach uses the influence of nodes to control the diffusion.
A time series is used to describe the rate of diffusion of a piece of infor-
mation. The volume of nodes that adopt the information over time is
measured. A linear influence model (LIM) is developed. The influence
functions are non-parametric and a non-negative least squares prob-
lem is solved using reflective newton method to find the value of the
functions.

5.2.3. Using partial differential equation (PDE)

The diffusion of a piece of information in the network by a node can
also be modeled using a PDE. A logistic equation is used to model
the density of influenced users at a given distance from the source
and at a given time. The parameters of the model are estimated using
the Cubic Spline Interpolation method.
Another approach is the time-based asynchronous independent
cascades (T-BAsIC) model. The parameters of this model are not
fixed values but functions which depend on time.

5.2.4. A predictive model

A predictive model for temporal dynamics of information diffu-
sion, considers the following three types of attributes for calculat-
ing diffusion probability: (i) semantics, (ii) social and (iii) time. The
propagation is modeled as an asynchronous process like the asyn-
chronous independent cascades (AsIC). The attributes given above
are used to define the diffusion probability on each edge of the
graph.
The social attributes are activity (Ac), Homogeneity (H), ratio
of directed tweets for each user, whether the user has had active
interactions in the past, and the mention rate of the users.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 351

Application — Social and Information Networks 351

The activity is the average amount of tweets emitted per hour

which varies from 0 to 1. For a user u, it is given by


 Vu if Vu < 
Ac(u) =  ,

1 Otherwise

where Vu is the volume of tweets of user u and  = 30.4 × 24.

The Homogeneity of two users u1 and u2 is the overlap in the
users with whom the pair of users interact with. It is given by

|Vu1 Vu2 |
H(u1 , u2 ) =  .
|Vu1 Vu2 |

The ratio of directed tweets T (u) of a user u is computed as:


 |Du |

if |Vu | > 0
T (u) = |Vu | ,


0 Otherwise

where Du is the directed tweets including the retweets.

Between pairs of users a boolean relationship exists (F (u1, u2))
which shows if they have had active interaction in the past. This
shows if they are “friends” of each other. It is determined by using
the following equation:

1 if u2εVu1
F (u1, u2) = .
0 Otherwise

The mention rate of a user u is R(u) and is the volume of directed

tweets received by her. It is given by
 u
 |V |

if |V u | < k
R(u) = k .


1 Otherwise
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 352

352 Introduction to Pattern Recognition and Machine Learning

The semantic attribute S(u, i) states whether a particular user u

has tweeted about a topic ci in the past and is given by:

1 if ci εKu
S(u, i) = ,
0 Otherwise

where Ku is the set of keywords in the messages of u.

The temporal attribute looks at the activity of the user at dif-
ferent times of the day. A receptivity function F (u, t) for a user u
considers an interval of time [t1 : t2 ] which is a 4 hour interval. This
is because the day is divided into 6 intervals of 4 hours each. So
t1 < t < t2 . The receptivity will be:

|Vut |
P (u, t) = ,
|Vu |

where Vu is the messages of user u and Vut gives the messages of user
u during time t.
For every pair of nodes u and v, the above parameters are used
to find 13 features. These features are Ac(u), Ac(v), S(u, i), S(v, i),
F (u, v), F (v, u), R(u), R(v), T (u), T (v), P (u, t), P (v, t) and H(u, v).
For a month, the data of the social network is used and according
to the spreading cascades available, each link with its 13 features is
classified as “diffusion” or “non-diffusion”. This data is then used to
predict the diffusion for the next month.

5.2.5. Using core index

In a diffusion process, it is also critical to find the most influential
spreaders in a network. There are a number of ways of finding the
most important users in a network. However, many a time the best
spreaders are not necessarily the most important users in a network.
The most efficient spreaders are those located within the core of the
network. A core index is assigned to each node in the network. Those
nodes which have the lowest values are located at the periphery
of the network and the nodes with the highest values are located
in the center of the network. These nodes form the core of the
network.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 353

Application — Social and Information Networks 353

6. Identifying Specific Nodes in a Social Network

It is essential to identify central or important nodes in a network.
It maybe necessary to find the influential users in a network. The
influence measure can be used for doing this. One way of doing this
is to say that the influence of a node is the number of in-network
votes his posts generate. An influence model needs to be used here.
The network is a graph G(V, E) where there are n nodes and m edges.
There is an edge between two nodes i and j, if actor i is linked to
actor j. It can also be a weighted graph where the edges have a weight
which reflects the strength of the link. Centrality measures help to
find the important nodes. Some of the centrality measures include
Closeness Centrality, Graph Centrality and Betweenness Centrality.
These measures depend on network flow.

• Closeness Centrality
Closeness centrality of a node i is the inverse of the shortest total
distance from i to every connected node j. It can be written as:
1
Cc (i) = n ,
j=1 dij

where dij gives the distance between nodes i and j. In case the
network is not strongly connected, the closeness centrality depends
also on the number of nodes, Ri , reachable from i. It can be writ-
ten as:
Ri
n−1
Cc (i) = Pn .
j=1 d(i,j)
Ri

• Graph Centrality
For this measure, it is necessary to find the node k which is far
away in terms of distance from the node i. Graph centrality is the
inverse of the distance from i to k. It can be written as
1
Cg (i) = ,
maxj∈V (i) d(i, j)
where V (i) is the set of nodes reachable from i.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 354

354 Introduction to Pattern Recognition and Machine Learning

• Betweenness Centrality
Considering all paths between pairs of nodes in the graph G,
betweenness centrality counts the number of times the path crosses
the node i. It can be written as

Njk (i)
Cb (i) = ,
Njk
i=j=k

where Njk is the number of paths between j and k and Njk (i) gives
the number of shortest paths between j and k passing through i.
• Page rank Centrality
The Page rank is given by

Cpr (j)
Cpr (i) = (1 − α) + α
outdegj
jεout(i)

where α is the damping factor and out(i) gives the out incident
nodes of i. In matrix form, this can be written as
Cpr = (1 − α)e + αCpr P,
where e is the (1 × n) unit vector, Cpr is the (1 × n) Page Rank
vector and P is the (n × n) transition matrix.
• Degree Centrality
This measure uses the topology of the graph. The in-degree cen-
trality of a node i is the in-degree of node i and can be written as
Cd (i) = indegi .
Similarly, the out-degree centrality of a node i is the out-degree of
the node i and can be written as
Cd (i) = outdegi .
• α-Centrality
This is a path-based measure of centrality. It can be defined as
k→∞ 


Cα = v αt At .
t=0
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 355

Application — Social and Information Networks 355

1
This converges only if α < |λ1 |
. In this method, α is called the
attenuation factor.
• Katz Score
The α-Centrality when v = αeA yielding:

Ckatz = αA(1 − αA)−1 .

So far we have considered analysis of social networks based only

on their link structure. It is possible to exploit the content associ-
ated with the nodes in the network also to understand its behavior
better. Typically, content is analyzed using Topic Models which we
discuss next.

7. Topic Models
Topic modeling is a method used to analyze large documents. A topic
is a probability distribution over a collection of words and a topic
model is a statistical relationship between a group of observed and
unknown(latent) random variables which specifies a generative model
for generating the topics. Generative models for documents are used
to model topic-based content representation. Each document is mod-
eled as a mixture of probabilistic topics.

7.1. Probabilistic latent semantic analysis (pLSA)

In pLSA, the latent or hidden variables which are the topics and are
associated with the observed variables such as documents and words.
For any document, a document-term matrix can be formed which
is also called the co-occurrence matrix. This co-occurrence matrix can
be used to extract the topics occurring in the document.
Consider N documents d = {d1 , . . . , dN } consisting of M
words w = {w1 , . . . , wM }. The latent variables are the K topics
z = {z1 , . . . , zK }.
A generative process is carried out for the documents as follows:

1. Select a document dj with probability P (d).

2. For each word wi in the document dj :
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 356

356 Introduction to Pattern Recognition and Machine Learning

(a) Select a topic zi from a multinomial conditioned on the given

document with probability P (z|dj ).
(b) Select a word wi from a multinomial conditioned on the topic
chosen in the previous step with probability P (w|zi ).

The model can be specified as

P (d, w) = P (d)P (w|d).

Considering the set of topics

P (w|d) = P (w, z|d)
zεZ


= P (w|d, z)P (z|d).
zεZ

Due to conditional independence P (w|d, z) = P (w|z) and we get

P (w|d) = P (w|z)P (z|d).
zεZ

Then


P (w, d) = P (z)P (d|z)P (w|z).
zεZ

Since the predictive model of the pLSA mixture model is P (w|d),

the objective function of the maximization is given by
  
P (w|d) = P (w|d)n(d,w) .
d,w dεD wεW

Using Expectation-Maximization (EM) algorithm, we find the

log-likelihood as




n(d, w) · log P (w|z)P (z|d).
dεD wεW zεZ

It is also possible to take the document-word matrix or the co-

occurrence matrix which is a sparse matrix and split it into three
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 357

Application — Social and Information Networks 357

matrices. The document-word matrix A can be written as:

A = L · U · R.

Here, L and R are low ranked matrices and U is a diagonal matrix.

The first matrix contains the document probabilities P (d|z), the sec-
ond diagonal matrix U contains the prior probabilities of the topics
P (z) and the third matrix R contains the word probabilities P (w|z).

7.2. Latent dirichlet allocation (LDA)

Consider a dataset with a number of documents D where each doc-
ument has T topics each being a multinomial containing K elements
where each element is a term in the corpus. The LDA generative
process randomly chooses a distribution over topics for each docu-
ment, for every word in the document one of the T topics is drawn
probabilistically from the distribution over topics and one of the K
words is drawn probabilistically. The process is given below:

1. For each document

(a) Draw a topic distribution, θd ∼ Dir(α) which is drawn from
a uniform Dirichlet distribution with scaling parameter α.
(b) For each word in the document
i. Draw a specific topic zd,n ∼ multi(θd ) which is a
multinomial.
ii. Draw a word wd,n ∼ βzd,n

The posterior distribution of the latent variable in a document is

p(θ, z, w|α, β)
p(θ, z|w, α, β) = , (4)
p(w|α, β)

where w represents a word and w represents a document of N words.

i.e. w = (w1 , w2 , . . . , wN ).
α = (α1 , α2 , . . . , αN ) where α is the Dirichlet distribution param-
eter. z is a vector of topics.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 358

358 Introduction to Pattern Recognition and Machine Learning

Taking the right hand side of Eq. (4), the numerator can be
written as
   N k V
Γ( ki=1 αi )  αi −1   
k
j i
p(θ, z, w|α, β) = k θi (θi βi,j )wn zn .
i=1 Γ(αi ) i=1 n=1 i=1 j=1
(5)

The denominator is
    N k V 
Γ k
α  k   
i=1 i j
p(w|α, β) = k θiαi  (θi βij )wn dθ.
i=1 Γ(αi ) i=1 n=1 i=1 j=1

(6)

A simpler, convex distribution is used to find the lower bound

on the log likelihood and are optimized. In Figure 9.7, due to the
edges between θ, z and w, the coupling between θ and β makes this
inference intractable. By dropping the edges between θ, z and w, we
get the simplified model as shown in Figure 9.8. We get a family of
distributions for φ and γ on the latent variables θ and z.
Using this distribution, the posterior is


N
p(θ, z|γ, φ) = p(θ|γ) p(zn |φn ). (7)
n=1

β
1:k

α θd zd,n ωd,n N
M

Figure 9.7. LDA model.

April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 359

Application — Social and Information Networks 359

γ φd,n
d

θd zd,n
N
M

Figure 9.8. Simplified LDA model.

To find the optimal values of γ and φ, we need to solve the

optimization problem

(γ∗, φ∗) = argmin(γ,φ) D(p(θ, z|γ, φ)  p(θ, z|w, α, β)). (8)

This is the minimization of the Kullback–Liebler divergence

between the variational distribution and the actual posterior dis-
tribution. The EM algorithm can be used to estimate β and α.
In the EM algorithm, given a set of observed variables, a set of
latent variables and the model parameters, the log probability of the
observed data is optimized. In the E-step of the EM algorithm, the
log likelihood of the data is found. In the M-step, the lower bound
on the log likelihood with respect to α and β is maximized.
The algorithm is as follows:
1. E-step: Find the optimal values of γd∗ and φ∗d for every document
in the corpus. This is used to compute the expectation of the log
likelihood of the data.
2. M-step: Maximize the lower bound on the log likelihood of


D
L(α, β) = log p(wd |α, β).
d=1

7.3. Author–topic model

The author–topic model simultaneously models the contents of doc-
uments and authors. Each document is represented as a mixture of
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 360

360 Introduction to Pattern Recognition and Machine Learning

topics and author modeling allows a mixture of weights for different

topics according to the authors of the document. The set of topics
that are in a corpus are obtained and which topics are used by which
authors is identified.
The model represents documents with a mixture of topics like in
LDA and the mixture weights of different topics are determined by
the authors of the document. In this model, the contents of docu-
ments and the interest of authors are simultaneously modeled.
The LDA model has two sets of unknown parameters — the
document distribution and the topic distribution. Also unknown are
the latent variables corresponding to the assignment of individual
words to topics. For LDA, we get

Pmj + β Q +α
P (zi = j|wi = m, z−i , w−i )α   dj ,
m
Pm
j + V β j
Qdj
+ T α

where zi = j represents the assignment of the ith word in the doc-

ument to topic j, wi = m represents the fact that the ith word is
the mth word in the lexicon, z−i represents all topics not including
the ith word. Pmj is the number of times word m is assigned to
topic j, not including the current instance and Qdj is the number of
times topic j has occurred in document d, not including the current
instance.
Pmj + β
φmj =  ,
m
Pm
j + V β
Qdj + α
θdj =  ,
j
Qdj
+ T α

where φmj is the probability of using word m in topic j, and θdj is

the probability of topic j in document d.
In the author–topic model,

P (zi = j, xi = k|wi = m, z−i , x−i w−i , ad )

Pmj + β R +α
× α  kj ,
m
Pm
j + V β j
Rkj
+ T α
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 361

Application — Social and Information Networks 361

where zi = j represents the assignment of the ith word in a document

to topic j, xi = k represents the assignment of the ith word to author
k, wi = m represents the fact that the ith word is the mth word
in the lexicon, z−i , x−i represents all topics and author assignment
not including the ith word. Rkj is the number of times author k is
assigned to topic j not including the current instance. The random
variables φ and θ are estimated as:
Pmj + β
φmj =  ,
m
Pm
j + V β
Rkj + α
θkj =  .
j
Rkj
+ T α

Research Ideas
1. The similarity measure SimRank may be viewed as a recursive version of simple
similarity. It is given by
 
Ssr (P, Q)
P ∈N (X) Q∈N (Y )
Ssr (X, Y ) = .
|N (X)| · |N (Y )|
Note that it is a global measure of similarity. How do you justify its need against
its computational cost?

Relevant References
(a) G. Jeh and J. Widom, SimRank: A measure of structural-context similarity.
Proceedings of the ACM SIGKDD International Conference on KDD, July
2002.
(b) D. Liben-Nowell and J. Kleinberg, The link prediction problem for social
networks. Proceedings of CIKM, 2003.
(c) L. Lu and T. Zhou, Link prediction in complex networks: A survey. Phys-
ica A, 390:1150–1170, 2011.
(d) M. A. Hasan and M. J. Zaki, A survey of link prediction in social networks.
Social Network Data Analysis:243–275, 2011.
2. It was observed that Adamic–Adar and Resource Allocation Index are found to
perform better among the local similarity measures. Why?
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 362

362 Introduction to Pattern Recognition and Machine Learning

Relevant References
(a) L. Adamic, and E. Adar, Friends and neighbours on the web. Journal of
Social Networks, 25:211–230, 2003.
(b) T. Zhou, L. Lu, and Y.-C. Zhang, Predicting missing links via local infor-
mation. Journal of European Physics B, 71:623–630, 2009.
(c) Z. Liu, W. Dong and Y. Fu, Local degree blocking model for missing link
prediction in complex networks, arXiv:1406.2203 [accessed on 29 October
2014].
(d) N. Rosenfeld, O. Meshi, D. Tarlow and A. Globerson, Learning structured
models with the AUC loss and its generalizations. Proceedings of AISTATS,
2014.
3. What is the relevance of the power-law degree distribution in link prediction?

Relevant References
(a) Y. Dong, J. Tang, S. Wu, J. Tian, N. V. Chawla, J. Rao and H. Cao, Link
prediction and recommendation across heterogeneous social networks. Pro-
ceedings of ICDM, 2012.
(b) S. Virinchi and P. Mitra, Similarity measures for link prediction using power
law degree distribution. Proceedings of ICONIP, 2013.
4. How can one exploit community detection in link prediction?

Relevant References
(a) S. Soundararajan and J. E. Hopcroft, Using community information to
improve the precision of link prediction methods. Proceedings of WWW
(Companion Volume), 2012.
(b) B. Yan and S. Gregory, Detecting community structure in networks using
edge prediction methods, Journal of Statistical Mechanics: Theory and
Experiment, P09008, 2012.
5. In link prediction we deal with adding links to the existing network. However,
in a dynamically changing network it makes sense to delete some of the links.
How to handle such deletions?

Relevant References
(a) J. Preusse, J. Kunegis, M. Thimm, S. Staab and T. Gottron, Structural
dynamics of knowledge networks. Proceedings of ICWSM, 2013.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 363

Application — Social and Information Networks 363

(b) K. Hu, J. Xiang, W. Yang, X. Xu and Y. Tang, Link prediction in com-

plex networks by multi degree preferential-attachment indices, CoRR
abs/1211.1790 [accessed on 29 October 2014].
6. How can we link the Modularity of the network with spectral clustering?

Relevant References
(a) M. E. J. Newman, Community detection and graph partitioning. CoRR
abs/1305.4974 [accessed on 29 October 2014].
(b) M. W. Mahoney, Community structure in large social and information net-
works. Workshop on Algorithms for Modern Massive Data Sets (MMDS),
2008.
(c) U. von Luxburg, A tutorial on spectral clustering. Statistics and Computing,
17(4):395–416, 2007.
(d) S. White and P. Smyth, A spectral clustering approach to finding commu-
nities in graph. SDM, 5:76–84, 2005.
(e) M. E. J. Newman and M. Girvan, Finding and evaluating community struc-
ture in networks. Physical Review E, 69(2):56–68, 2004.
7. Can we use the Modularity matrix to design better classifiers?

Relevant References
(a) P. Schuetz and A. Caflisch, Efficient modularity optimization: Multi-step
greedy algorithm and vertex mover refinement. CoRR abs/0712.1163,
2007.
(b) P. Schuetz and A. Caflisch, Multi-step greedy algorithm identifies com-
munity structure in real-world and computer-generated networks. CoRR
abs/0809.4398, 2008.
8. How does diffusion help in community detection?

Relevant References
(a) A. Guille, H. Hacid and C. Favre, Predicting the temporal dynamics of
information diffusion in social networks. Social and Information Networks,
2013.
(b) F. Wang, H. Wang and K. Xu, Diffusive logistic model towards predict-
ing information diffusion in online social networks. ICDCS’ Workshops,
pp. 133–139, 2012.
April 8, 2015 12:57 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-ch09 page 364

364 Introduction to Pattern Recognition and Machine Learning

(c) K. Saito, M. Kimura, K. Ohara and H. Motoda, Selecting information dif-

fusion models over social networks for behavioral analysis. PKDD ’10,
2010.
(d) A. Guille and H. Hacid, A predictive model for the temporal dynamics of
information diffusion in online social networks. WWW 2012, 2012.
(e) J. Yang and J. Leskovec, Modeling information diffusion in implicit net-
works. ICDM ’10, pp. 599–608, 2010.
9. Most of the community detection methods are computationally expensive. How
do we realize scalable versions?

Relevant References
(a) V. D. Blondel, J.-L. Guillaume, R. Lambiotte and E. Lefebvre, Fast unfold-
ing of communities in large networks. Journal of Statistical Mechanics,
10:P10008, 2008.
(b) K. Wakita and T. Tsurumi, Finding community structure in mega-scale social
networks, eprint arXiv:cs/070248, 2007.
(c) L. Danon, A. Diaz-Guilera and A. Arenas, The effect of size heterogeneity
on community identification in complex networks. Journal of Statistical
Mechanics, P11010, 2006.
(d) P. Pons and M. Latapy, Computing communities in large networks using
random walks. Computer and Information Sciences, ISCIS 2005, Springer
Berlin Heidelberg, pp. 284–293, 2005.
(e) F. Radicchi, C. Castellano, F. Cecconi, V. Loreto and D. Parisi, Defining and
identifying communities in networks. Proceedings of the National Academy
of Science USA, 101:2658–2663, 2004.
(f) A. Clauset, M. E. J. Newman and C. Moore, Finding community structure
in very large networks. Physical Review E, 70:066111, 2004.
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 365

Index

α-Centrality, 354 application, 178, 254

2-level decision tree, 193 applications of link prediction, 341
2-partition, 215 approximate factorization, 310
approximate nearest neighbor, 172
a unified framework, 259 approximate search, 88
Academic publication, 294 approximation algorithms, 334
accuracy, 187, 194 approximation error, 84, 85
accuracy of the classifier, 3 approximation space, 179
activation function, 197–199, 208, 265 Area Under the Curve, 102
activation sequence, 348 arithmetic mean, 102
active neighbor, 349 array, 192
activity, 350, 351 Artificial Neural Network (ANN), 76,
actual class label, 198, 200 168, 171, 195, 196, 265
actual fitness, 195 artificial sink, 328
actual output, 207 assignment matrix, 338
Adamic–Adar, 361 Asynchronous Independent Cascades,
Adamic–Adar function, 346 350
adaptive, 318 asynchronous process, 350
adaptive boosting, 262 attenuation factor, 355
adjacency matrix, 323, 327, 332, 337, attribute, 148, 180, 182, 183, 185,
338 191, 193, 350
adjacent node, 322 attribute index, 191, 192
agglomerative, 219 attribute reduction, 180
agglomerative algorithm, 16, 230 attribute value, 192
agglomerative clustering, 332, 334 AUC loss, 362
aggregation, 196 author modeling, 360
alphabet, 171 author–topic model, 359, 360
Amazon, 254 author-paper network, 340
anomaly detection, 167 average precision, 210
ant colony optimization, 212, 213 average probability of error, 112
antecedent of the rule, 317 average-link algorithm, 17

365
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 366

366 Index

axiomatic framework, 259 binary feature, 250

axis-parallel decision, 10 binary matrix, 316
axis-parallel decision tree, 191 binary operator, 279
binary pattern, 236
B Tree, 227 binary prediction, 206
backpropagation, 195, 197, 206 binary random variable, 119
backpropagation algorithm, 198–200 binary relation, 275
backward filter, 79 Binary Relevance, 205
bag of terms model, 308 binary representation, 276
bag-of-words, 44, 128, 321, 293 binary string, 274, 280
bagging, 172 binary tournament selection, 94
bagging predictor, 172 binary trees, 192
Bank transaction slip, 294 binomial distribution, 125, 128
base case, 288 biobjective clustering, 258
basis matrix, 156 biomedical, 108
basis vectors, 83 biometric, 293, 294
Baum–Welch algorithm, 285 bipartition of the graph, 340
Bayes belief nets, 285 BIRCH, 15, 22, 225, 227, 229, 254,
Bayes classifier, 13, 112, 119 258, 265
Bayes error rate, 136 bisection, 328
Bayes rule, 111, 112, 117, 119, 121, BLAST, 168
126, 161, 204, 307, 311, 314 boolean, 192
Bayesian analysis, 133 boolean function, 251
Bayesian approach, 111 boolean relationship, 351
Bayesian estimation, 116, 121, 130 bootstrapping, 8, 9
Bayesian learning, 29, 111, 117 bottom up, 219
Bayesian model, 266 boundary condition, 66
behavioral analysis, 364 boundary pattern, 19
benefit function, 328 boundary region, 270, 271
Bernoulli random variable, 119, 123, boundary set, 180
124 branch and bound, 76
best fitness, 185 bug report, 294
best weights, 200 building a classifier, 165
Beta density, 126, 128
betweenness centrality, 353, 354 caltrop, 192
bias, 207–209 candidate decision tree, 191
biased mutation, 201 candidate solution, 182, 191
biclustering, 260 categorical data, 79
big data, 131, 242 categorical feature, 7, 72
bijection, 275 centrality measure, 353
binary classification, 186 centroid, 5, 17, 220, 222, 226, 223,
binary classifier, 12, 32 244, 248, 249, 266, 268, 271, 272,
binary data, 250, 260 275, 277, 283, 284
binary dataset, 184 centroid of a rough cluster, 271
binary equivalence relation, 179 CF vector, 225–229
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 367

Index 367

CF-tree, 225, 227, 228, 248 classification rule, 211, 213, 317
CF-tree construction, 228 classification task, 180
chain-like clusters, 258, 259 classification technique, 177
character, 236, 238, 317 classification time, 9, 13, 20, 196
character pattern, 269 classification using ANN, 196
character set, 238 classification using GAs, 187
characteristic equation, 300 classification/test time, 9
chi-square statistic, 79, 102 classifier, 2, 165, 204, 205, 206, 209,
child chromosome, 201 296
children nodes, 192 classifiers in the compressed domain,
children strings, 279 175
Choquet distance, 188 clique, 322
Choquet Hyperplane, 188, 191 closed world assumption, 348
Choquet Hyperplane H, 187 closeness centrality, 353
Choquet integral, 188 cluster, 217, 219, 225, 227–229,
chromosome, 182, 183, 187, 189–191, 244–247, 249, 250, 252, 256, 258,
200, 340 263, 266, 270, 271, 281, 284, 286,
circuit, 331, 333 296, 297, 313–316
citation network, 321, 340 cluster analysis, 258
city-block distance, 55, 253, 299 cluster assignment, 220
class, 1, 185, 207, 246, 247, 298 cluster assumption, 160, 161
class descriptions, 317 cluster based support vector machine,
class feature vector, 185 248
class imbalance, 8, 12, 30 cluster centers, 274, 277
class label, 2, 37, 99, 102, 109, 112, cluster centroid, 247, 249
135, 136, 147, 149, 161, 168, 171, cluster compressed data, 260
177–179, 184, 186, 192, 194, cluster ensemble, 172
196–198, 205, 207, 246, 252, 286, Cluster Indicator Matrix, 263
298, 299, 317 cluster labels, 286
class separability, 90 cluster numbers, 274
class-conditional independence, 113, cluster representative, 5, 17, 242, 246,
114, 132 267, 296–298
classical logic, 316 cluster structure, 313, 314
classification, 1, 135, 139, 159, 160, cluster validity, 25
167, 168, 173–178, 180, 182, 183, clustering, 3, 148, 172, 174, 215, 224,
185, 187, 188, 196, 197, 200, 209, 230, 236, 238, 241–244, 247, 248,
211, 212, 242, 244, 260, 262, 293, 252, 259–261, 263, 265, 266, 272,
316, 326 273, 281, 282, 285, 286, 293, 316,
classification accuracy, 75, 139, 142, 318, 319, 326, 330, 336
160, 185, 191, 193, 194, 209 clustering algorithms, 160, 218, 223,
classification algorithm, 76, 136, 140, 261, 267, 318, 319
168, 169, 175, 178 clustering by compression, 260
classification model, 136 clustering coefficient, 326, 327
classification of the time series, 170 clustering ensemble algorithms, 262
classification performance, 173 Clustering Feature tree, 225
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 368

368 Index

clustering for classification, 246 condensation, 138

clustering labeled data, 20, 34, 246 condensation algorithm, 171
clustering large datasets, 21, 236 Condensed Nearest Neighbor, 138
clustering of nodes, 322 condensed set, 138, 139
clustering quality, 259 conditional independence, 356
clustering the graph, 331 conductance, 325, 326, 329
clusters, 257, 312, 313, 334 conjugate pair, 320
CNN, 138, 139, 171 conjugate prior, 29, 125, 126,
co-occurrence matrix, 355 130–132, 312, 313
co-occurrence of terms, 321 conjunct, 316
co-training, 165 conjunctive conceptual clustering, 316
collaborative filtering, 108, 319 consequent, 317
collection management, 261 consistent rule set, 186
collection of documents, 314 consolidation ratio, 334
column vector, 227 constraint, 152, 174, 309
combination of clusterings, 255 construction of the CF-tree, 227
combining predictions, 174 contents of document, 360
common neighbors, 253, 342–345 context, 253
common neighbors function, 345 contextual knowledge, 253
communities, 325, 327, 328, 331, 364 contextual similarity, 253
communities in networks, 326 continuity condition, 66
community, 332, 335, 348 conventional clustering, 34
community detection, 30, 322, convergence, 199, 291
362–364 convergence criterion, 94
community information, 362 convex function, 287
community structure, 322, 362–364 convex loss function, 162
compact data structure, 22 convex non-negative matrix
company report, 294 factorization, 107
competitive learning, 281 convex optimization problem, 162
complete-link algorithm, 17 cooling rate, 335
complex networks, 361–364 cooperative game theory, 258
complexity of the algorithm, 334 core, 270
component, 308, 312 core index, 352
compress data, 21 core of the network, 352
compressed frequent pattern, 175 corpus size, 312
compression, 176 correlation coefficient, 64
compression scheme, 176, 260 cosine of the angle, 59, 73, 253
computation time, 16 cosine similarity, 61, 63
computational cost, 205, 361 cost function, 94
computational geometry, 265 cost-sensitive measure, 194
computational resources, 273 covariance, 27
concave function, 289 covariance matrix, 86, 106, 132, 133,
concentric, 231, 259 157, 167, 302, 305
concept, 24, 316 coverage of the rules, 186
conceptual clustering, 25, 34, 316 covering, 282, 283
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 369

Index 369

crawl, 293 decision tree, 10, 27, 144, 145, 148,

crisp classification, 178 166, 167, 169, 173, 191–194
criterion function, 16, 220, 222, Decision Tree Classifier, 5
276–278, 280 Decision tree induction, 191
cross-validation, 93, 187, 209 definable set, 180
crossover, 94, 182, 190, 191, 195, 200, degree, 322, 332, 345
201, 245, 272, 279 degree centrality, 354
crossover point, 279 degree distribution, 125
crowding index, 95 degree matrix, 232, 329, 330
Cubic Spline Interpolation method, degree of freedom, 80
350 degrees of the vertices, 327
current solution, 335 dendrogram, 218, 334
current source, 331 dense cluster, 218
current-flow, 333 density based clustering, 19
current-flow betweenness, 331 density estimation, 119
curse of dimensionality, 140 dependent variable, 315
curve fitting, 25 depth of the decision tree, 192
cut, 234 design time, 136, 196
cutting plane algorithm, 150, 154 desired output, 207
cyber security, 293 detection of outlier, 19
Dewey Decimal Classification, 254
d-dimensional vector, 188, 245, 282, diagonal matrix, 232, 299, 300, 302,
286 306, 307, 329, 332, 338, 357
damping factor, 354 diameter, 226
data, 254, 260, 295 diffusion kernel, 169
data analysis, 110 diffusion probability, 349, 350
data clustering, 29, 107, 215 diffusion process, 349, 352
data compression, 5, 20, 84, 241, 260 diffusive logistic model, 363
data matrix, 37, 107, 230, 296 digit recognition problem, 199
data mining, 21, 30, 37, 174, 241, 243, dimensionality reduction, 25, 29, 70,
259, 261, 263 84, 103, 104, 108, 169, 173, 174,
data point, 215, 221, 223, 227, 229, 295, 296, 305, 306
230, 242, 246, 290, 296, 303 directed graph, 338, 349
data reduction, 258 directed link, 340
data sample, 190 Dirichlet, 312, 313, 315, 320
data streams, 258 Dirichlet allocation, 130
data structure, 9, 225, 236 Dirichlet prior, 129, 315
dataset, 205, 215, 218–220, 222–224, Discrete Fourier Transform, 104, 169
239, 241, 246, 247, 250, 255, 274, Discrete Haar Wavelet Transform,
282, 283, 357 104
dataset scan, 236, 239 discrete random variable, 289
decision boundary, 8, 152, 160, 161, discrete random walk, 332
249 Discrete Wavelet Transform, 169
decision forest, 145, 172 discriminative feature analysis, 106
decision making, 242, 246, 263 discriminative information, 159
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 370

370 Index

discriminative models, 11 domain knowledge, 111, 121, 124,

discriminative pattern mining, 175, 252–254
260 domain of the feature, 244
discriminative phrase, 132 domain-knowledge, 273
disjoint labelsets, 205 dot product, 89, 306, 307
disjunction, 181 dual form, 156
dispersion index, 211 dynamic clustering, 18, 19
dissimilarity, 341, 342, 346 dynamic programming, 65, 168
distance, 188, 231, 247–249, 251, 267, dynamic time warping, 64, 73
277, 283, 295, 332, 341, 342, 346,
353 E-step, 359
distance function, 15, 51, 71, 72, 153, edge, 232, 321, 326, 331, 341, 348,
171, 172, 346 349, 353, 358
distance measure, 53, 72, 253 edge centrality, 331
distance metric, 71, 139, 140, 171 edge prediction methods, 362
distance threshold, 224 edge weights, 254
distance-based, 171 edges in the cluster, 328
distance-based classification, 168 edible, 254
edit distance, 172
distributed estimation, 175
editing, 138
distribution over topics, 357
effective classification, 175, 260
divergence, 310
effective diameter, 325
divergence threshold, 144
efficient algorithm, 248
divide-and-conquer, 258, 296, 298
efficient classification, 4
divide-and-conquer clustering, 21
efficient modularity optimization, 363
dividing hyperplane, 178
eigenvalue, 235, 299–303, 305, 306,
divisive algorithm, 16, 219 323, 324, 330, 337
divisive clustering, 72, 218 eigenvector, 234, 235, 299, 300–303,
divisive hierarchical clustering, 333 305, 329, 330, 337–339
document, 2, 242, 252, 263, 295, 305, electronic mail, 294
306, 308, 311–313, 315, 316, 321, elementary sets, 180
355–357, 359 elitist strategy, 280
document analysis, 305 elitist string, 280
document categories, 293 EM algorithm, 161, 285, 290, 308, 359
document classification, 70, 106, 111 email message, 293
document clustering, 34, 134, 261, embedded method, 27, 76
266, 285 empirical study, 171
document collection, 39, 308, 313, 321 empty cluster, 216, 223
document generation, 313, 314 encoding, 191
document probabilities, 357 Encyclopedia, 294
document retrieval, 293, 294 ensemble, 109, 174
document topic matrix, 308 ensemble classifier, 144
document-term matrix, 39, 295, 305, ensemble clustering, 262
355 ensemble feature ranking, 102
documents, 293, 313, 355 entity extraction, 108
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 371

Index 371

entropy, 44, 45, 167, 186, 255 external knowledge, 134, 260, 261,
equivalence class, 179, 269, 270, 275 319
equivalence relation, 270, 275 external node, 348
error coding length, 194 extremal optimization, 336
error function, 281
error in classification, 209 F -score, 99, 100, 109
error rate, 92, 112, 136, 186 factorization, 295, 299, 310
estimate, 116, 244, 291, 309, 316, 350, false positives, 195
359 farthest neighbor, 295
estimation of parameters, 111 feasible solution, 278
estimation of probabilities, 115 feature, 183, 192, 230, 243, 296
estimation scheme, 266 feature elimination, 94, 96
Euclidean distance, 55, 64, 86, 119, feature extraction, 27–29, 75, 86, 105,
142, 168, 230, 246, 253, 267, 268, 108, 109, 169
282, 283 feature ranking, 99–103, 109
evaluating community structure, 363 feature selection, 26, 29, 75–78 80, 83,
evaluation function, 182, 200 91, 92, 96, 97, 103, 105, 108–110,
131, 139, 169, 172, 173, 175, 187,
evaluation metric, 209
211–213
evaluation of a solution, 336
feature set, 299
evaluation of the strings, 184
feature subset, 91, 92, 297
evolutionary algorithm, 23, 91, 102,
feature subspace selection, 172
108, 191, 193, 266, 272, 273, 279
feature vector, 168, 170, 187, 207
evolutionary algorithms for
feature weights, 131
clustering, 264
feature-based classification, 168, 169
evolutionary computation, 108
feedforward neural network, 199
evolutionary operator, 272
filter method, 26, 76, 77
evolutionary programming, 264, 280
final solution, 182
evolutionary schemes, 264
finding communities, 363
evolutionary search, 279 fingerprint, 293, 294
exhaustive enumeration, 76, 218 first principal component, 303
exhaustive search, 99 first-order feature, 170
expectation, 141, 290 Fisher’s kernel, 169
expectation maximization, 161, 266 Fisher’s linear discriminant, 139
expectation step, 308 fitness calculation, 184
expectation-maximization, 319, 356 fitness evaluation, 183
expected frequency, 80 fitness function, 91, 102, 183, 185,
expected label, 161 190, 191, 200, 272, 340
expected value, 308, 309 fitness value, 193, 272, 273, 276, 278,
explanation ability, 196 336
exploitation operator, 273 Fixed length encoding, 192
exploration, 273 fixed length representation, 195
exponential family of distributions, flow-based methods, 328
117 forensics, 293
external connectivity, 326 forest size, 147
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 372

372 Index

forward filter, 79 Gaussian edge weight function, 162

Fourier coefficient, 104 Gaussian mixture model, 105, 285
F P -tree, 239, 240 gene, 189
fractional distance, 141 gene selection, 212
fractional metric, 142 generalization error, 108, 145, 148,
fractional norms, 56, 171 160
frequency, 115, 323 generalized Choquet integral, 187
frequency distribution, 133 generalized eigenproblem, 330
frequency domain, 104, 169 generalized Kullback–Leibler
frequency of occurrence, 252, 293, 321 divergence, 310
frequent 1-itemsets, 239, 240 generating new chromosomes, 190
frequent closed itemset mining, 260
generation, 190, 273
frequent itemset, 30, 132, 175, 236,
generation gap, 280
238–240, 260, 316, 317, 319
Generation of new chromosomes, 190
Frequent Pattern Tree, 22, 236
Frobenius norm, 306, 310 generative model, 11, 31, 160, 161,
function learning, 28 265, 307, 311, 355
function node, 192 generative probabilistic model, 312
functional form, 312 generative process, 313, 355, 357
fuzzy, 179 genetic algorithm, 76, 91, 108, 177,
fuzzy K-means algorithm, 267, 268 178, 264, 318, 340
fuzzy classification, 178 genetic approach, 106
fuzzy classifier, 177 genetic operators, 94, 195
fuzzy clustering, 22, 173, 264, 266, genetic programming, 187
267 geometric, 265
fuzzy criterion function, 277 Gibbs sampling, 314, 316
fuzzy decision tree, 148 gini impurity, 147
fuzzy kNN classification, 179 gini index, 145
fuzzy logic, 212 Girvan–Newman algorithm, 331
fuzzy membership, 177, 179 global dissimilarity, 346
fuzzy neural network, 211 global error, 208
fuzzy random forest, 148, 172 global measure, 361
fuzzy sets, 266
global minimum, 221
fuzzy tree, 148, 149
global modularity, 336
fuzzy-rough approach, 211
global optimization, 212
gamma distribution, 127 global optimum, 282
gamma function, 312 globally optimal solution, 222, 280,
GA, 178, 182–185, 187, 189, 199, 200, 310
211, 212, 264, 265, 274, 279, 280, GMMs, 285, 286
318 Goodman–Kruskal measure, 81
GAs for classification, 178 goodness of fit, 79
Gaussian, 157 gradient, 200, 291
Gaussian component, 286 gradient of the Lagrangian, 152
Gaussian distribution, 89 granular space, 211
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 373

Index 373

graph, 230, 231, 253, 260, 321, 325, hierarchical clustering, 14, 15, 218,
326, 328, 331–333, 340, 343–346, 225, 328
353, 354, 363 hierarchical document clustering, 260,
graph centrality, 353 319
graph classification, 175, 213 hierarchy of partitions, 218, 219
graph clustering algorithms, 328 high degree common neighbor, 345
graph data, 230 high degree nodes, 343
graph distance, 346 high density region, 160, 161
graph energy, 162 high frequency, 252
graph Laplacian, 329, 330 high-dimensional, 139, 142, 151, 172,
graph partitioning, 363 260, 281, 305–307
graph partitioning algorithms, 328 high-dimensional data, 86
Graph-based approach, 348 high-dimensional space, 7, 29, 89,
graph-based method, 162 171, 173, 295, 296
graphical representation, 321 hinge loss, 162
greedy algorithm, 363 HMM, 171, 286
greedy optimization, 335 Hole ratio, 186
homogeneity, 350, 351
greedy search algorithm, 99
homogeneous network, 340, 341
grouping, 318
hop-plot exponent, 325
grouping of data, 215
hybrid clustering, 21, 259
grouping phase, 252, 254
hybrid evolutionary algorithm, 212
grouping unlabeled patterns, 246, 261
hybrid feature selection, 110, 131
groups, 325
hybrid GA, 212
grow method, 192
hyper-parameter, 128
hyper-rectangular region, 145
Hamming loss, 209, 210
hyperlinks, 294
handwritten data, 262
hyperplane, 150, 189, 339
handwritten text, 294 hypothesis, 81
hard clustering, 263, 265, 267
hard clustering algorithm, 268, 285 identifying communities, 326, 364
hard decision, 263 identity matrix, 296, 310
hard partition, 215, 258, 274, 276, 285 implicit networks, 364
hashing algorithm, 172 impossibility of clustering, 259
health record, 294 impurity, 45, 145, 146
Hessian, 155 in-degree of node, 354
heterogeneous network, 340, 341 incident nodes, 354
heterogeneous social networks, 362 incomplete data, 285
heuristic technique, 336 incomplete knowledge, 71
hidden layer, 178, 197–199, 207 incremental algorithm, 224, 230, 258
Hidden Markov Model, 20, 168, 171, incremental clustering, 18, 21
285 independent attributes, 181
hidden node, 201 independent cascade model, 349
hidden unit, 197, 199, 200, 207, 208 index, 293
hidden variables, 313, 355 index vector, 233, 234
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 374

374 Index

indexing, 106 intra-group connectivity, 325

indiscernibility relation, 179 intractable, 358
indiscernible, 179, 269, 270 inverse document frequency, 252
indiscernible equivalence classes, 269 inverted file, 22
induction hypothesis, 288 inverted index, 39
inequality, 119, 250, 288 isotropic, 231
inference, 313, 319, 358 Itakura Parallelogram, 66, 67
inference process, 314 iterative, 173, 291, 328
influence measure, 353 Itukara Parallelogram, 68
influence model, 353
influence network, 321 J-measure, 193
influence of node, 350 Jaccard coefficient, 345
influence threshold, 349 Jeffrey’s Prior, 133
influential user, 353 Jensen’s inequality, 287, 289, 290
information diffusion, 347, 348, 363, joint density, 286
364 joint distribution, 315
information function, 180
information gain, 102, 106, 145, 148, k-class problem, 193
166, 167 k-nearest neighbor algorithm, 203
information networks, 325 K-dimensional vector, 312, 313
information retrieval, 22, 39, 47, 59, K-fold cross-validation, 3
70, 128, 131, 265, 306, 321 k-gram, 169
infrequent items, 240 k-labelset, 205, 206
initial assignment, 267, 271 K-means algorithm, 15, 16, 219–223,
initial centroid, 220–223, 243, 256 231, 235, 243, 247, 256, 265–267,
initial partition, 16, 328 282, 285, 311, 319, 329, 330
initial population, 193 k-nearest neighbor, 136, 179, 203
initial solutions, 273 k-nearest neighbor classifier, 5
initial weights, 200 K-partition, 15, 215, 217, 219, 220
initialization, 193, 200, 272 k-spectrum kernel, 169
initialization of population, 192 Katz Score, 355
inner product, 88, 169 Katz Similarity, 347
input layer, 178, 199, 207, 281 Kendal’s similarity score, 97
input node, 201, 281, 282 kernal-based fuzzy-rough nearest
input unit, 197, 200, 207 neighbor classification, 211
instance, 193, 202, 205, 208, 209 kernel function, 101, 169
instance-label pair, 209 kernel matrix, 153, 156
integer programming, 99 kernel support vector machines, 258
integer programming problem, 339 kernel SVM, 173
inter-group connectivity, 325 kernel trick, 173
internal connectivity, 326 keywords, 352
internal disjunctions, 316 KL distance, 63
internal edge, 325, 334 kNN classifier, 179
internal node, 13, 348 kNN edge weight function, 162
interval-valued variable, 48 knowledge discovery, 259
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 375

Index 375

knowledge networks, 362 leader, 223, 224, 254, 262, 265–267,

knowledge-based clustering, 24, 246, 318
250, 259, 261 leader algorithm, 15, 16, 223, 224,
knowledge-based proximity, 253 229, 259, 265
Kronecker delta, 337 leaf node, 147, 194, 195, 227, 229
Kullback–Leibler measure, 131 leaf-level cluster, 227
Kullback–Liebler divergence, 359 learning, 136, 314
kurtosis, 170 learning by example, 196
learning phase, 136
L1 metric, 144 learning rate, 198, 199, 208, 282
L1 norm, 142 least resistance, 331
L2 norm, 142 least squares learning algorithm, 159
Lk norm, 141, 142 least squares problem, 350
l-labelsets, 206 legal record, 294
L1-loss, 101 length of the random walk, 332
L2-loss, 101 length of the shortest path, 346
lexicographic order, 239, 240
label, 192, 193, 204, 208, 209
lexicon, 360
labeled clustering, 30
likelihood function, 121, 126, 129, 308
labeled data, 135, 159–161, 165, 166,
likelihood ratio, 157, 158
252
linear classification, 173
labeled pattern, 1, 4, 20
linear classifier, 7
labelset, 205, 206, 208
linear combination, 83, 296, 305
Lagrange function, 309
linear decision boundary, 173
Lagrange variable, 109, 152, 309
linear program, 339
large datasets, 318
linear sum, 228
large networks, 364 linear SVM, 101, 109, 150, 153, 154,
large-scale, 173, 212 173, 249, 250
large-scale application, 246 linear threshold model, 349
largest centrality, 331 linear time, 173, 174
largest margin, 166 linearly separable, 153
largest positive eigenvalue, 337 link prediction, 20, 30, 253, 322, 326,
largest singular value, 306 340, 341, 361–363
latent cluster, 309 linkage-based clustering, 328, 331
latent Dirichlet allocation, 24, 129, local alignment algorithm, 168
266, 357 local fitness value, 336
Latent Gaussian models, 133 local information, 362
latent semantic analysis, 28, 299, 305, local maxima, 328
307 local minimum, 16, 107, 199
latent semantic indexing, 28, 106 local modularity, 336
latent topic, 311, 312 local search, 212
latent variable, 285, 286, 305, 357–360 local similarity measure, 20, 342, 361
lazy learning, 203 locality sensitive hashing, 88, 90
LDA, 266, 311, 312, 316, 320, 357, log-likelihood, 286, 358, 356, 359
360 log-likelihood function, 286, 290, 308
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 376

376 Index

logical description, 24 maximum variance, 106, 306

logical operator, 195 maximum-margin hyperplane, 169
logistic equation, 350 MCNN, 138, 139, 171
logistic regression, 7, 29, 156, 157 MDL, 194
longest common subsequence, 168 mean, 48, 49, 108, 118, 125, 127, 157,
loss function, 101 161, 170, 290, 333
lossy and non-lossy compression, 21 mean field annealing, 340
lossy compression, 38 median, 5, 46
low density region, 160–162 medoid, 5, 18
low dimensional subspace, 89 membership function, 179
low-dimensional space, 295 membership value, 177–179, 264, 267
low-rank representation, 306 Memetic algorithm, 212
lower approximation, 22, 177, 179, Mexican hat function, 282
180, 264, 270–272 microarray data, 318
lower bound, 358, 359 Min–max, 76
lp-norms, 172 min-cut, 328
LSA, 299 mincut, 233, 234
minimal reduct, 181
M-step, 359 Minimal Spanning Tree, 17, 230
machine learning, 1, 25, 29, 30, 75, minimum cut, 329
129, 150, 176, 241, 243, 293, 319 Minimum Description Length, 194
majority class, 160 minimum distance, 267
majority voting, 299 Minimum Distance Classifier, 118
Manhattan distance, 65 minimum squared error, 222, 277
manual page, 294 mining frequent itemsets, 236
MAP, 203, 204 mining large dataset, 176, 260
Map-Reduce, 34 Minkowski distance, 55, 63, 71
Markov chain, 332 minority class, 12
matching, 215, 251, 296 misclassification cost, 194
mathematical induction, 288 missing data, 243
mathematical programming, 339 missing link prediction, 362
matrix, 27, 230, 235, 257, 296, 300, missing value, 19, 244
303, 305, 354 mixture model, 161, 356
matrix factorization, 295, 296, 307, mixture of distributions, 286
319 mixture of probabilistic topics, 355
max-flow, 328 mkNN, 137
max-heap, 334 ML estimate, 122, 123
maximize the margin, 151 MLP, 171
maximum depth, 192 mode, 44, 45, 125
maximum eigenvalue, 27, 235 model parameters, 359
maximum fitness, 190 model-based classification, 168, 170
maximum likelihood estimate, 161 model-based clustering, 259
maximum modularity, 334 modified condensed nearest neighbor,
maximum number of links, 326 138
maximum similarity, 267 modified k-nearest neighbor, 136
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 377

Index 377

modularity matrix, 336, 337, 339, 363 nearest neighbor classification, 171,
modularity maximization, 338 211
modularity optimization, 336, 339 nearest neighbor classifier, 5, 53, 132,
momentum term, 199 248, 296
MST, 230, 254 Needleman–Wunsch algorithm, 168
multi-class classification, 14, 178, 206 negative class, 151, 188
multi-class problem, 11, 150, 153 negative example, 146
multi-graph, 341 neighboring communities, 335
multi-label classification, 202, 206, neighboring community, 335
209, 213 neighbors, 326
multi-label kNN, 203 nervous system, 196
multi-label naive Bayes classification, network, 200, 207, 326, 327, 331, 340,
213 342, 347, 348, 352, 353, 355,
multi-label problem, 14 362–364
multi-label ranking, 202 neural network, 23, 169, 174, 177,
multi-layer feed forward network, 195, 178, 196, 199, 206, 208, 211, 266,
197 281
Multi-level recursive bisection, 328 neural networks for classification, 195
multi-lingual document classification, neural networks for clustering, 265
293 neuro-fuzzy classification, 211
multi-media document, 294 newspaper article, 294
multi-objective approach, 212 NMF, 310, 311, 319
multi-objective fitness function, 194, NN classifier, 140, 168
195 NNC, 246
Multi-objective Optimization, 277 node, 207, 230, 321, 326, 327, 341, 348
multinomial, 129, 312, 313, 315, 320, nominal feature, 41
356, 357 non-differentiable, 178
multinomial distribution, 313 non-dominated, 278
multinomial random variable, 129 non-isotropic, 231
multinomial term, 315 non-metric distance function, 171
multiobjective evolutionary non-negative matrix factorization, 24,
algorithm, 319 84, 310, 107, 319
multiple kernel learning, 261 non-parametric, 79, 135, 350
multivariate data, 167 non-spherical data, 173
multivariate split, 145 non-terminal node, 191, 192
mutation, 94, 182, 190, 191, 195, 200, non-zero eigenvalue, 295, 300
273, 279, 280 nonlinear classifier, 6
mutual information, 26, 78, 105, 106, nonlinear dimensionality reduction,
131, 252, 295 174
nonlinear SVM, 173, 174
Naive Bayes classifier, 7, 29, 113, 131 nonmetric similarity, 72
natural evolution, 272 normal distance, 151
natural selection, 91, 182 normalization, 301, 303
nearest neighbor, 19, 76, 135, 172, normalized cut metric, 325
183, 246, 253, 295, 297, 298 normalized gradient, 202
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 378

378 Index

normalized graph Laplacian, 330 online social network, 363, 364

normalized spectral clustering, 330 ontologies, 252
NP-complete, 339 optimal condensed set, 138
NP-hard, 339 optimal hard clustering, 265
number of K-partitions, 215, 217 optimal partition, 276
number of attributes, 194 optimal prediction, 204
number of branches, 167 Optimization of Squared Error, 220
number of chromosomes, 190, 193 optimization problem, 107, 151,
number of classes, 12, 13, 187, 195, 153–155, 159, 166, 187, 277, 296,
199 359
number of clusters, 25, 220, 230, 235, order dependence, 18, 224, 229, 259
254, 255, 277, 338, 339 order-independent, 138, 139, 258
number of distance computations, 223 ordinal data, 45
number of edges, 327, 334, 335, 337 orientation, 152
number of generations, 273 orthogonal, 152, 234, 235, 302
number of hidden layers, 200 out-degree centrality, 354
number of hidden nodes, 200
outlier, 18, 148, 242, 243
number of labels, 209
outlier detection, 34, 241–243
number of leaf nodes, 195
output layer, 178, 197, 198, 281
number of links, 326
output unit, 197, 198, 200, 207
number of neighbors, 326
overall entropy, 186
number of nodes, 194, 201
overall error, 207
number of onto functions, 217
overall modularity, 336
number of partitions, 14, 215
number of paths, 354 overfitting, 145
number of rules, 185, 186 overlapping K-means, 283
number of terms, 305, 315 overlapping clusters, 274
number of topics, 312 overlapping intervals, 270
number of training epochs, 209
number of training patterns, 186, 244 page rank, 354
numerical feature, 7, 72 Page rank Centrality, 354
Nystrom approximated SVM, 155, pairwise distance, 153
174 parallelism, 184
Nystrom approximation, 156 parameter vector, 290, 291, 308, 313,
314
object type, 250, 251 parametric estimation, 119
objective fitness function, 194 pareto-optimal set, 278
objective function, 85, 268, 296, 356 Particle Swarm Optimization, 212
oblique decision tree, 172 partition, 4, 181, 187, 218, 224, 241,
oblique split, 10, 145 255, 256, 270, 274–276, 282, 283,
observed frequency, 80 328, 333, 334, 337, 340
offspring, 95, 273 partitional clustering, 14, 219, 224,
one versus one, 12 231
one versus the rest, 11 partitioning, 215, 217, 246, 254, 328
one-dimensional, 290 path, 193, 194, 348, 354
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 379

Index 379

pattern, 2, 149, 181, 186, 206, 207, precision, 97, 195, 362
223, 228–230, 242–248, 250, 252, predicate, 250, 316
253, 263, 265, 269, 271, 276, 281, predict, 167, 203, 204, 209, 210
282, 284, 286, 297 predictive model, 350, 356, 364
pattern classification, 70, 184, 195, preferential-attachment, 343, 363
211, 212 preprocessing, 8
pattern matrix, 2, 37 primal form, 154, 155
pattern recognition, 1, 19, 30, 261 primitive, 251
pattern synthesis, 8, 243, 245, 261 principal component analysis, 88,
patterns in graphs, 322 107, 302
PCA, 302 principle of inclusion and exclusion,
peak dissimilarity, 63 217
peaking phenomenon, 75 prior density, 121, 122, 124–126, 129,
penalty coefficient, 189 312
penalty function, 91 prior knowledge, 111, 115, 116
performance evaluation, 209 prior probabilities, 111, 113, 115, 119,
performance of GA, 212 158, 204, 255, 266, 357
performance of the classifier, 79 probabilistic assignment, 312
piecewise aggregate approximation,
probabilistic clustering, 265
103
probabilistic clustering algorithm, 266
Pin-Code recognition, 293
probabilistic convergence, 282
PLSA, 307, 311, 319
probabilistic latent semantic analysis,
PLSI, 266
307, 355
polynomial kernel, 169
probabilistic latent semantic
polynomial time, 339
indexing, 24, 28, 266, 319
polysemy, 306
probabilistic model, 311, 265, 266,
population, 91, 190, 265, 272, 273,
285
278
probabilistic selection, 336
population of chromosomes, 178, 191
probabilistic topic model, 312
population of strings, 182, 184
probability density function, 119
positive class, 12, 146, 151, 188
positive eigenvalue, 337 probability distribution, 23, 170, 201,
313, 322, 355
positive reflexivity, 51, 57
positive semi-definite, 329 probability emission matrix, 171
possibilistic clustering, 264, 267 probability mass function, 289
posterior distribution, 313, 314, 357, probability of error, 13
359 probability of selection, 278
posterior probabilities, 13, 111, 114, probability transition matrix, 171
116, 158, 266 product rule, 204
posterior probability, 112, 117, 147, protein sequence, 168
204 prototype, 8, 247, 281, 296
power law, 124, 133, 322 prototype selection, 138, 171
power law distribution, 125, 322 proximity, 61, 62, 253, 341
power law prior, 130 proximity function, 341, 342
power-law degree distribution, 362 Proximity measures, 50
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 380

380 Index

quadratic program, 339 representation, 191, 275, 200, 252

quasi-Newton, 174 representatives of the cluster, 218
query document, 307 reproduction, 184, 279
research grant application, 294
R tree, 103 resource allocation index, 345, 346,
radial basis function, 101 361
radius, 226, 248, 250 RNN, 171
random feature selection, 90 robust classification, 70
random field, 175 robust classifier, 298
random forest, 13, 29, 144, 147, 148, robust clustering, 18
166, 172, 173 robust optimization, 70
random projections, 28, 86, 89, 108, rotation invariant, 56
172, 295 rough K-means algorithm, 271, 272
random sampling, 156 rough classification, 179
random sampling without rough clustering, 22, 264
replacement, 206 rough decision forest, 173
random subspace method, 145, 172 rough fuzzy clustering, 318
random variable, 119, 141, 313, 355 rough set, 180, 264, 266, 269–271
random walk, 328, 332, 364 roulette wheel selection, 190, 195
random weight, 197, 199, 281 row in CIM , 264
random-search, 273 row-major order, 237
range box, 223 rule, 182, 184, 185
rank, 283, 306, 322, 323, 345 rule consistency, 186
rank exponent, 323 rule-based classification, 185
ranking, 25, 29, 96, 252, 293, 346
ranking features, 109 SA, 336, 340
ranking loss, 210 Sakoe–Chiba band, 66, 67
ratio variable, 41, 49 sample covariance matrix, 303
reachable, 353 sample mean, 17, 244, 303
real valued eigenvalue, 330 scalable algorithm, 150
recall, 195 scalable GAs, 318
receiver operator characteristics, 102 scale free networks, 125
recombination, 272, 273, 279, 280 scale-invariant, 56
recombination operation, 273 scoring function, 204, 326
recommendation, 362 search algorithm, 212
recommending a movie, 341 search engine, 293, 294
recommending collaborators, 341 search space, 273, 280
recurrent neural network, 168 second principal component, 303
reduced support vector machine, 156 second smallest eigenvalue, 234
reduct of least cardinality, 181 selection, 94, 182, 190, 191, 195, 272,
reduct set, 181 273, 278, 280
reflective Newton method, 350 Selection of initial centroids, 221
regression, 25, 147, 242 selection probability, 278
regularization, 174 self organizing map, 265
regularized topic model, 133, 320 semantic class labels, 3
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 381

Index 381

semi-structured document, 293 single pass algorithm, 318

semi-supervised classification, 148, single-link algorithm, 17, 230
159, 166, 174 single-point crossover operator, 279
semi-supervised learning, 135, 160, singleton cluster, 218, 230, 243, 249
165–167 singular value, 83, 84, 299, 302, 306
semi-supervised learning algorithms, Singular Value Decomposition, 83,
160 104, 169, 299
semi-supervised SVM, 166 SLA, 230, 231, 255, 257
semisupervised feature selection, 175 slack variable, 154
sentiment analysis, 202 smallest eigenvalue, 234, 235, 329
separating hyperplane, 100 Smith–Waterman algorithm, 168
sequence classification, 175 social and information networks, 321,
sequential floating backward 326, 363
selection, 76 social network, 20, 30, 33, 253, 260,
sequential floating forward selection, 263, 285, 321, 322, 325, 340, 341,
76 347–349, 352, 353, 355, 361, 363,
sequential selection, 76 364
sequential sweep, 335 soft classification, 29, 177
set of features, 180, 270, 296 soft clustering, 4, 22, 130, 263, 265,
set of labels, 180, 202, 203, 205, 206, 266, 274, 285, 286, 310, 316–318
210 soft computing, 172, 177
set of support vectors, 109 soft partition, 263, 276, 277, 285, 318
set of topics, 356, 360 software code, 294
set of transactions, 239 solution string, 276
Shannon’s entropy, 45 SOM, 265, 281
short text document, 294 source vertex, 329
shortest path, 333, 354 space complexity, 223
shortest-path betweenness, 331, 333 spam mail, 293
shrinkage threshold, 144 spanning tree, 230
shrinkage–divergence proximity, 143 Sparse nonnegative matrix
sigmoid function, 159 factorization, 319
signature verification, 167 sparse random projections, 87, 108
signed Choquet distance, 189, 190 spatial data, 41, 50
similarity, 215, 219, 232, 251, 255, Spatio-temporal data, 49, 50
257, 267, 341–343 spectral bisection, 337
similarity computation, 252, 253 spectral clustering, 231, 235, 260,
similarity function, 253, 254, 322, 328, 329, 363
342, 344, 345 spectral decomposition, 104
similarity graph, 230 spectral optimization, 336, 338
similarity matrix, 236, 255, 256, 329 spectral radius, 235
similarity measure, 108, 168, 306, 362 spherical cluster, 25, 221, 231
simplified LDA model, 359 split, 148, 167, 356
SimRank, 347 squared error, 220, 221, 268, 276–278,
simulated annealing, 211, 212, 335 280
single link algorithm, 254 squared error criterion, 25, 221, 265
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 382

382 Index

squared Euclidean distance, 299, 310 tabu search, 211, 212

squared loss, 162 Tanimoto similarity, 61
squared sum, 226, 228 target label, 2
SSGA, 280 Taxonomy, 171
stacked ensemble, 108 temporal attribute, 352
standard deviation, 48, 49, 157, 170 temporal data, 41, 50, 175
statistical clustering, 265, 282 temporal data mining, 175
statistical model, 171, 265 temporal dynamics, 348, 350, 363, 364
statistical optimization, 107 temporal pattern, 62
steady-state genetic algorithm, 280 term, 295, 296, 311, 313, 316
stochastic search techniques, 211 term frequency, 252
stopping criterion, 155 terminal node, 192
stratified sampling, 172 termination condition, 273
stream data mining, 18, 224 terrorist network, 341
string, 182, 200, 273, 274, 278–280 test pattern, 2, 135, 136, 149, 166,
string kernel, 169 177, 179, 202, 246, 247, 296–299
string-of-centroid, 275, 276, 278, 318 text categorization, 132
string-of-centroids representation, text classification, 106, 131
274, 278, 279 text document, 294, 307
string-of-group-numbers, 274–276 text mining, 59, 254, 263, 306
string-of-group-representation, 318 theorem of the ugly duckling, 250
strings, 275 threshold, 15, 152, 192, 197, 208, 209,
subspace clustering, 319 225, 239
summarization, 25, 29, 242, 293, 326 threshold-based feature selection, 102
supervised classification, 105, 148, 286 time complexity, 99, 139, 155
supervised clustering, 261 time series, 64, 103–105, 110,
supervised learning, 135 167–170, 187, 350
support lines, 151, 152 time series classification, 168, 187
support plane, 151 topic, 305, 307, 308, 313, 314, 316,
support vector, 109, 152, 153 317, 352, 355–357, 360
support vector machine, 6, 109, 150, topic coherence, 133, 320
174, 175, 187, 248, 265 topic distribution, 357, 360
surjective, 216 topic model, 23, 30, 130, 132, 133,
survival of the fittest strategy, 278 293, 307, 320, 355
SVD, 299, 300, 302, 304, 306, 307 topic modeling, 133, 134, 355
SVM, 27, 166, 187, 248, 250 topological map, 281
SVM classification, 261 topological similarity, 335
SVM formulation, 154 topology, 325, 348, 354
symbolic data, 169 tournament selection, 195
symbolic sequences, 168 trained network, 196
symmetric difference, 210 training, 9, 199, 208
synonymy, 306 training data, 2, 5, 8, 135, 136, 138,
syntactic labels, 3 153, 159, 160, 171, 183, 186, 196,
synthesizing patterns, 261 198, 209, 247, 286, 298
April 8, 2015 12:58 Introduction to Pattern Recognition and Machine Learning - 9in x 6in b1904-index page 383

Index 383

training example, 148, 171, 186, 205, validation set, 3, 91, 92, 102, 191,
209 193, 209
training loss, 155 variance, 108, 120, 125, 127, 129, 141,
training pattern, 2, 13, 132, 152, 159, 221, 290
177, 178, 185, 186, 199, 206, 207, variance impurity, 146
246 variational inference, 133
training phase, 9 vector, 265, 274
training time, 9, 75, 136, 196 vector space, 132, 265
transaction, 175, 237, 239, 240 vector space model, 265
verification, 293
transaction dataset, 236
visualization, 265, 326
transition matrix, 332, 338, 339, 354
vocabulary, 114, 308
translation invariant, 55
tree, 191, 227 warping window, 66
tree-based encoding, 192 wavelet transform, 104
tree-based representation, 195 web, 362
triangle inequality, 52, 53 web page, 293
tweet, 294, 352 weight, 183, 196, 200, 207–209, 277
two-class classification, 248 weight functions, 162
two-class problem, 11, 147, 188 weight matrix, 232, 236, 329, 330, 338
two-partition, 233, 234, 257 weight vector, 100, 109, 152
types of data, 29 weighted average, 284
weighted edge, 338
typical pattern, 139
weighted Euclidean distance, 183
weighted graph, 353
unary operator, 279
weighted kNN classifier, 182
unconstrained optimization, 100 Wikipedia, 134, 252, 254, 260, 261,
undirected, 232, 321, 344 294, 319
undirected graph, 341 Wilcoxon statistic, 102
uniform density, 124 winner node, 282
uniform Dirichlet distribution, 357 winner-take-all, 267, 281
uniform distribution, 141, 142 winner-take-most, 281
uniform prior, 116, 124 winning node, 282
unit resistance, 331, 333 with-in-group-error-sum-of-squares,
unit step function, 197, 198 276
unit vector, 354 Wolfe dual, 155
word probabilities, 357
unit-norm, 303
working set, 155
unit-sphere, 339
wrapper method, 76, 26
univariate, 157
WTM, 282
unlabeled data, 135, 159–161,
165–167 X-rays, 294
unsupervised feature extraction, 295
unsupervised learning, 24, 27, 135 zero-mean normalization, 304
upper approximation, 22, 177, 180, Zipf’s curve, 133
264, 270–272 Zipf’s law, 47, 124