100% found this document useful (2 votes)

533 views

Modelbased Clustering Classification Data Science PDF

Uploaded by

Williamette Core

Available Formats

Download as PDF, TXT or read online on Scribd

100% found this document useful (2 votes)

533 views

Modelbased Clustering Classification Data Science PDF

Uploaded by

Williamette Core

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 447

Model-based Clustering and Classiﬁcation

for Data Science

Cluster analysis consists of methods for finding groups in data automatically.
Most methods have been heuristic and leave open such central questions as: How
many clusters are there? Which clustering method should I use? How should
I handle outliers? Classification involves assigning new observations to groups given
previously classified observations, and also has open questions about parameter
tuning, robustness and uncertainty assessment. This book frames cluster analysis
and classification in terms of statistical models, thus yielding principled estima-
tion, testing and prediction methods, and soundly-based answers to the central
questions. It develops the basic ideas of model-based clustering and classification
in an accessible but rigorous way, using extensive real-world data examples and
providing R code for many methods, and describes modern developments for high-
dimensional data and for networks. It explains recent methodological advances,
such as Bayesian regularization methods, non-Gaussian model-based clustering,
cluster merging, variable selection, semi-supervised classification, robust classifica-
tion, clustering of functional data, text and images, and co-clustering. Written for
advanced undergraduates and beginning graduate students in data science, as well as
researchers and practitioners, it assumes basic knowledge of multivariate calculus,
linear algebra, probability and statistics.

c h a r l e s b o u v e y r o n is Professor of Statistics at Université Côte d’Azur

and the Chair of Excellence in Data Science at Inria Sophia-Antipolis. He has
published extensively on model-based clustering, particularly for networks and high-
dimensional data.
g i l l e s c e l e u x is Director of Research Emeritus at Inria Saclay Île-de-France.
He is one of the founding researchers in model-based clustering, having published
extensively in the area for 35 years.
t. b r e n d a n m u r p h y is Professor of Statistics at University College Dublin. His
research interests include model-based clustering, classiﬁcation, network modeling
and latent variable modeling.
a d r i a n e . r a f t e r y is Professor of Statistics and Sociology at University
of Washington, Seattle. He was one of the founding researchers in model-based
clustering, having published in the area since 1984.
C A M B R I D G E S E R I E S I N S TAT I S T I C A L A N D
P R O B A B I L I S T I C M AT H E M AT I C S

Editorial Board
Z. Ghahramani (Department of Engineering, University of Cambridge)
R. Gill (Mathematical Institute, Leiden University)
F. P. Kelly (Department of Pure Mathematics and Mathematical Statistics,
University of Cambridge)
B. D. Ripley (Department of Statistics, University of Oxford)
S. Ross (Department of Industrial and Systems Engineering,
University of Southern California)
M. Stein (Department of Statistics, University of Chicago)

This series of high-quality upper-division textbooks and expository monographs covers all
aspects of stochastic applicable mathematics. The topics range from pure and applied statistics
to probability theory, operations research, optimization and mathematical programming. The
books contain clear presentations of new developments in the ﬁeld and also of the state of
the art in classical methods. While emphasizing rigorous treatment of theoretical methods, the
books also contain applications and discussions of new techniques made possible by advances
in computational practice.
A complete list of books in the series can be found at www.cambridge.org/statistics.
Recent titles include the following:

30. Brownian Motion, by Peter Mörters and Yuval Peres

31. Probability (Fourth Edition), by Rick Durrett
33. Stochastic Processes, by Richard F. Bass
34. Regression for Categorical Data, by Gerhard Tutz
35. Exercises in Probability (Second Edition), by Loı̈c Chaumont and Marc Yor
36. Statistical Principles for the Design of Experiments, by R. Mead, S. G. Gilmour and
A. Mead
37. Quantum Stochastics, by Mou-Hsiung Chang
38. Nonparametric Estimation under Shape Constraints, by Piet Groeneboom and Geurt
Jongbloed
39. Large Sample Covariance Matrices and High-Dimensional Data Analysis, by Jianfeng
Yao, Shurong Zheng and Zhidong Bai
40. Mathematical Foundations of Inﬁnite-Dimensional Statistical Models, by Evarist Giné
and Richard Nickl
41. Conﬁdence, Likelihood, Probability, by Tore Schweder and Nils Lid Hjort
42. Probability on Trees and Networks, by Russell Lyons and Yuval Peres
43. Random Graphs and Complex Networks (Volume 1), by Remco van der Hofstad
44. Fundamentals of Nonparametric Bayesian Inference, by Subhashis Ghosal and Aad van
der Vaart
45. Long-Range Dependence and Self-Similarity, by Vladas Pipiras and Murad S. Taqqu
46. Predictive Statistics, by Bertrand S. Clarke and Jennifer L. Clarke
47. High-Dimensional Probability, by Roman Vershynin
48. High-Dimensional Statistics, by Martin J. Wainwright
49. Probability: Theory and Examples (Fifth Edition), by Rick Durrett
Model-Based Clustering and
Classification for Data Science
With Applications in R

Charles Bouveyron
Université Côte d’Azur

Gilles Celeux
Inria Saclay Île-de-France

T. Brendan Murphy
University College Dublin

Adrian E. Raftery
University of Washington
University Printing House, Cambridge CB2 8BS, United Kingdom

One Liberty Plaza, 20th Floor, New York, NY 10006, USA

477 Williamstown Road, Port Melbourne, VIC 3207, Australia

314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India

79 Anson Road, #06–04/06, Singapore 079906

Cambridge University Press is part of the University of Cambridge.

It furthers the University’s mission by disseminating knowledge in the pursuit of

education, learning, and research at the highest international levels of excellence.

www.cambridge.org
Information on this title: www.cambridge.org/9781108494205
DOI: 10.1017/9781108644181

This publication is in copyright. Subject to statutory exception

and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.

First published 2019

Printed in Singapore by Markono Print Media Pte Ltd

A catalogue record for this publication is available from the British Library.

Library of Congress Cataloging-in-Publication Data

Names: Bouveyron, Charles, 1979– author. | Celeux, Gilles, author. |
Murphy, T. Brendan, 1972– author. | Raftery, Adrian E., author.
Title: Model-based clustering and classification for data science : with applications in R /
Charles Bouveyron, Université Côte d’Azur, Gilles Celeux, Inria Saclay Île-de-France,
T. Brendan Murphy, University College Dublin, Adrian E. Raftery, University of Washington.
Description: Cambridge ; New York, NY : Cambridge University Press, 2019. |
Series: Cambridge series in statistical and probabilistic mathematics |
Includes bibliographical references and index.
Identifiers: LCCN 2019014257 | ISBN 9781108494205 (hardback)
Subjects: LCSH: Cluster analysis. | Mathematical statistics. |
Statistics–Classification. | R (Computer program language)
Classification: LCC QA278.55 .M63 2019 | DDC 519.5/3–dc23
LC record available at https://lccn.loc.gov/2019014257

ISBN 978-1-108-49420-5 Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of

URLs for external or third-party internet websites referred to in this publication
and does not guarantee that any content on such websites is, or will remain,
accurate or appropriate.
To Nathalie, Alexis, Romain and Nathan
Charles

To Maı̈lys and Maya

Gilles

To Trish, Áine and Emer

Brendan

To Hana, Isolde and Finn

Adrian
Contents

Page
Preface xv

1 Introduction 1
1.1 Cluster Analysis 1
1.2 Classiﬁcation 4
1.3 Examples 7
1.4 Software 12
1.5 Organization of the Book 13
1.6 Bibliographic Notes 14

2 Model-based Clustering: Basic Ideas 15

2.1 Finite Mixture Models 15
2.2 Geometrically Constrained Multivariate Normal Mixture Models 18
2.3 Estimation by Maximum Likelihood 23
2.4 Initializing the EM Algorithm 31
2.5 Examples with Known Number of Clusters 39
2.6 Choosing the Number of Clusters and the Clustering Model 46
2.7 Illustrative Analyses 60
2.8 Who Invented Model-based Clustering? 71
2.9 Bibliographic Notes 75

3 Dealing with Diﬃculties 79

3.1 Outliers 79
3.2 Dealing with Degeneracies: Bayesian Regularization 92
3.3 Non-Gaussian Mixture Components and Merging 97
3.4 Bibliographic Notes 105

4 Model-based Classiﬁcation 109

4.1 Classification in the Probabilistic Framework 109
4.2 Parameter Estimation 113
4.3 Parsimonious Classification Models 114
4.4 Multinomial Classification 119
4.5 Variable Selection 124
4.6 Mixture Discriminant Analysis 126
4.7 Model Assessment and Selection 127

vii
viii Contents

5 Semi-supervised Clustering and Classiﬁcation 134

5.1 Semi-supervised Classification 134
5.2 Semi-supervised Clustering 141
5.3 Supervised Classification with Uncertain Labels 144
5.4 Novelty Detection: Supervised Classification with Unobserved Classes 154
5.5 Bibliographic Notes 160

6 Discrete Data Clustering 163

6.1 Example Data 163
6.2 The Latent Class Model for Categorical Data 165
6.3 Model-based Clustering for Ordinal and Mixed Type Data 185
6.4 Model-based Clustering of Count Data 190
6.5 Bibliographic Notes 197

7 Variable Selection 199

7.1 Continuous Variable Selection for Model-based Clustering 199
7.2 Continuous Variable Regularization for Model-based Clustering 208
7.3 Continuous Variable Selection for Model-based Classiﬁcation 210
7.4 Categorical Variable Selection Methods for Model-based Clustering 211
7.5 Bibliographic Notes 215

8 High-dimensional Data 217

8.1 From Multivariate to High-dimensional Data 217
8.2 The Curse of Dimensionality 221
8.3 Earlier Approaches for Dealing with High-dimensional Data 227
8.4 Subspace Methods for Clustering and Classiﬁcation 238
8.5 Bibliographic Notes 257

9 Non-Gaussian Model-based Clustering 259

9.1 Multivariate t-Distribution 259
9.2 Skew-normal Distribution 267
9.3 Skew-t Distribution 270
9.4 Box–Cox Transformed Mixtures 278
9.5 Generalized Hyperbolic Distribution 282
9.6 Example: Old Faithful Data 285
9.7 Example: Flow Cytometry 287
9.8 Bibliographic Notes 288

11 Model-based Clustering with Covariates 331

11.1 Examples 331
11.2 Mixture of Experts Model 333
11.3 Model Assessment 339
11.4 Software 339
11.5 Results 340
11.6 Discussion 348
11.7 Bibliographic Notes 349

12 Other Topics 351

12.1 Model-based Clustering of Functional Data 351
12.2 Model-based Clustering of Texts 363
12.3 Model-based Clustering for Image Analysis 368
12.4 Model-based Co-clustering 373
12.5 Bibliographic Notes 382

List of R Packages 384

Bibliography 386
Author Index 415
Subject Index 423
Expanded Contents

Page
Preface xv

1 Introduction 1
1.1 Cluster Analysis 1
1.1.1 From Grouping to Clustering 1
1.1.2 Model-based Clustering 3
1.2 Classiﬁcation 4
1.2.1 From Taxonomy to Machine Learning 4
1.2.2 Model-based Discriminant Analysis 6
1.3 Examples 7
1.4 Software 12
1.5 Organization of the Book 13
1.6 Bibliographic Notes 14

2 Model-based Clustering: Basic Ideas 15

2.1 Finite Mixture Models 15
2.2 Geometrically Constrained Multivariate Normal Mixture Models 18
2.3 Estimation by Maximum Likelihood 23
2.4 Initializing the EM Algorithm 31
2.4.1 Initialization by Hierarchical Model-based Clustering 33
2.4.2 Initialization Using the smallEM Strategy 36
2.5 Examples with Known Number of Clusters 39
2.6 Choosing the Number of Clusters and the Clustering Model 46
2.7 Illustrative Analyses 60
2.7.1 Wine Varieties 60
2.7.2 Craniometric Analysis 65
2.8 Who Invented Model-based Clustering? 71
2.9 Bibliographic Notes 75

3 Dealing with Diﬃculties 79

3.1 Outliers 79
3.1.1 Outliers in Model-based Clustering 79
3.1.2 Mixture Modeling with a Uniform Component for Outliers 81
3.1.3 Trimming Data with tclust 88
3.2 Dealing with Degeneracies: Bayesian Regularization 92
3.3 Non-Gaussian Mixture Components and Merging 97
3.4 Bibliographic Notes 105
x
Expanded Contents xi

4 Model-based Classiﬁcation 109

4.1 Classification in the Probabilistic Framework 109
4.1.1 Generative or Predictive Approach 110
4.1.2 An Introductory Example 111
4.2 Parameter Estimation 113
4.3 Parsimonious Classification Models 114
4.3.1 Gaussian Classification with EDDA 114
4.3.2 Regularized Discriminant Analysis 115
4.4 Multinomial Classification 119
4.4.1 The Conditional Independence Model 119
4.4.2 An Illustration 123
4.5 Variable Selection 124
4.6 Mixture Discriminant Analysis 126
4.7 Model Assessment and Selection 127
4.7.1 The Cross-validated Error Rate 129
4.7.2 Model Selection and Assessing the Error Rate 131
4.7.3 Penalized Log-likelihood Criteria 133

5 Semi-supervised Clustering and Classiﬁcation 134

5.1 Semi-supervised Classification 134
5.1.1 Estimating the Model Parameters through the EM Algorithm 135
5.1.2 A First Experimental Comparison 136
5.1.3 Model Selection Criteria for Semi-supervised Classification 138
5.2 Semi-supervised Clustering 141
5.2.1 Incorporating Must-link Constraints 143
5.2.2 Incorporating Cannot-link Constraints 144
5.3 Supervised Classification with Uncertain Labels 144
5.3.1 The Label Noise Problem 145
5.3.2 A Model-based Approach for the Binary Case 146
5.3.3 A Model-based Approach for the Multi-class Case 150
5.4 Novelty Detection: Supervised Classification with Unobserved Classes 154
5.4.1 A Transductive Model-based Approach 155
5.4.2 An Inductive Model-based Approach 157
5.5 Bibliographic Notes 160

6 Discrete Data Clustering 163

6.1 Example Data 163
6.2 The Latent Class Model for Categorical Data 165
6.2.1 Maximum Likelihood Estimation 167
6.2.2 Parsimonious Latent Class Models 169
6.2.3 The Latent Class Model as a Cluster Analysis Tool 171
6.2.4 Model Selection 172
6.2.5 Illustration on the Carcinoma Data Set 174
6.2.6 Illustration on the Credit Data Set 178
6.2.7 Bayesian Inference 180
6.3 Model-based Clustering for Ordinal and Mixed Type Data 185
6.3.1 Ordinal Data 185
6.3.2 Mixed Data 186
6.3.3 The ClustMD Model 186
xii Expanded Contents

6.3.4 Illustration of ClustMD: Prostate Cancer Data 187

6.4 Model-based Clustering of Count Data 190
6.4.1 Poisson Mixture Model 191
6.4.2 Illustration: Vélib Data Set 194
6.5 Bibliographic Notes 197

7 Variable Selection 199

7.1 Continuous Variable Selection for Model-based Clustering 199
7.1.1 Clustering and Noisy Variables Approach 200
7.1.2 Clustering, Redundant and Noisy Variables Approach 200
7.1.3 Numerical Experiments 203
7.2 Continuous Variable Regularization for Model-based Clustering 208
7.2.1 Combining Regularization and Variable Selection 209
7.3 Continuous Variable Selection for Model-based Classiﬁcation 210
7.4 Categorical Variable Selection Methods for Model-based Clustering 211
7.4.1 Stepwise Procedures 212
7.4.2 A Bayesian Procedure 212
7.4.3 An Illustration 214
7.5 Bibliographic Notes 215

8 High-dimensional Data 217

8.1 From Multivariate to High-dimensional Data 217
8.2 The Curse of Dimensionality 221
8.2.1 The Curse of Dimensionality in Model-based Clustering and Classification 223
8.2.2 The Blessing of Dimensionality in Model-based Clustering and Classification 225
8.3 Earlier Approaches for Dealing with High-dimensional Data 227
8.3.1 Unsupervised Dimension Reduction 228
8.3.2 The Dangers of Unsupervised Dimension Reduction 230
8.3.3 Supervised Dimension Reduction for Classification 231
8.3.4 Regularization 236
8.3.5 Constrained Models 237
8.4 Subspace Methods for Clustering and Classification 238
8.4.1 Mixture of Factor Analyzers (MFA) 238
8.4.2 Extensions of the MFA Model 241
8.4.3 Parsimonious Gaussian Mixture Models (PGMM) 244
8.4.4 Mixture of High-dimensional GMMs (HD-GMM) 247
8.4.5 The Discriminative Latent Mixture (DLM) Models 251
8.4.6 Variable Selection by Penalization of the Loadings 254
8.5 Bibliographic Notes 257

9 Non-Gaussian Model-based Clustering 259

9.1 Multivariate t-Distribution 259
9.2 Skew-normal Distribution 267
9.3 Skew-t Distribution 270
9.3.1 Restricted Skew-t Distribution 273
9.3.2 Unrestricted Skew-t Distribution 275
9.4 Box–Cox Transformed Mixtures 278
9.5 Generalized Hyperbolic Distribution 282
9.6 Example: Old Faithful Data 285
Expanded Contents xiii

9.7 Example: Flow Cytometry 287

9.8 Bibliographic Notes 288

10 Network Data 292

10.1 Introduction 292
10.2 Example Data 294
10.2.1 Sampson’s Monk Data 295
10.2.2 Zachary’s Karate Club 296
10.2.3 AIDS Blogs 296
10.2.4 French Political Blogs 297
10.2.5 Lazega Lawyers 298
10.3 Stochastic Block Model 298
10.3.1 Inference 299
10.3.2 Application 301
10.4 Mixed Membership Stochastic Block Model 304
10.4.1 Inference 305
10.4.2 Application 306
10.5 Latent Space Models 312
10.5.1 The Distance Model and the Projection Model 313
10.5.2 The Latent Position Cluster Model 314
10.5.3 The Sender and Receiver Random Eﬀects 315
10.5.4 The Mixture of Experts Latent Position Cluster Model 315
10.5.5 Inference 316
10.5.6 Application 317
10.6 Stochastic Topic Block Model 320
10.6.1 Context and Notation 322
10.6.2 The STBM Model 323
10.6.3 Links with Other Models and Inference 326
10.6.4 Application: Enron E-mail Network 326
10.7 Bibliographic Notes 329

11 Model-based Clustering with Covariates 331

11.1 Examples 331
11.1.1 CO2 and Gross National Product 331
11.1.2 Australian Institute of Sport (AIS) 331
11.1.3 Italian Wine 332
11.2 Mixture of Experts Model 333
11.2.1 Inference 337
11.3 Model Assessment 339
11.4 Software 339
11.4.1 ﬂexmix 339
11.4.2 mixtools 340
11.4.3 MoEClust 340
11.4.4 Other 340
11.5 Results 340
11.5.1 CO2 and GNP Data 340
11.5.2 Australian Institute of Sport 341
11.5.3 Italian Wine 343
11.6 Discussion 348
xiv Expanded Contents

11.7 Bibliographic Notes 349

12 Other Topics 351

12.1 Model-based Clustering of Functional Data 351
12.1.1 Model-based Approaches for Functional Clustering 353
12.1.2 The fclust Method 354
12.1.3 The funFEM Method 356
12.1.4 The funHDDC Method for Multivariate Functional Data 358
12.2 Model-based Clustering of Texts 363
12.2.1 Statistical Models for Texts 364
12.2.2 Latent Dirichlet Allocation 365
12.2.3 Application to Text Clustering 366
12.3 Model-based Clustering for Image Analysis 368
12.3.1 Image Segmentation 368
12.3.2 Image Denoising 370
12.3.3 Inpainting Damaged Images 372
12.4 Model-based Co-clustering 373
12.4.1 The Latent Block Model 375
12.4.2 Estimating LBM Parameters 376
12.4.3 Model Selection 379
12.4.4 An Illustration 380
12.5 Bibliographic Notes 382

List of R Packages 384

Bibliography 386
Author Index 415
Subject Index 423
Preface

About this book

The century that is ours is shaping up to be the century of the data revolution. Our
numerical world is creating masses of data every day and the volume of generated
data is estimated to be doubling every two years. This wealth of available data
offers hope for exploitation that may lead to great advances in areas such as
health, science, transportation and defense. However, manipulating, analyzing
and extracting information from those data is made difficult by the volume and
nature (high-dimensional data, networks, time series, etc) of modern data.
Within the broad field of statistical and machine learning, model-based tech-
niques for clustering and classification have a central position for anyone interested
in exploiting those data. This textbook focuses on the recent developments in
model-based clustering and classification while providing a comprehensive in-
troduction to the field. It is aimed at advanced undergraduates, graduates or
first-year Ph.D. students in data science, as well as researchers and practitioners.
It assumes no previous knowledge of clustering and classification concepts. A basic
knowledge of multivariate calculus, linear algebra and probability and statistics is
needed.
The book is supported by extensive examples on data, with 72 listings of code
mobilizing more than 30 software packages, that can be run by the reader. The
chosen language for codes is the R software, which is one of the most popular
languages for data science. It is an excellent tool for data science since the most
recent statistical learning techniques are provided on the R platform (named
CRAN). Using R is probably the best way to be directly connected to current
research in statistics and data science through the packages provided by researchers.
The book is accompanied by a dedicated R package (the MBCbook package) that
can be directly downloaded from CRAN within the R software or at the following
address: https://cran.r-project.org/package=MBCbook. We also encourage
the reader to visit the book website for the latest information: http://math.
unice.fr/~cbouveyr/MBCbook/.
This book could be used as one of the texts for a graduate or advanced
undergraduate course in multivariate analysis or machine learning. Chapters 1
and 2, and optionally a selection of later chapters, could be used for this purpose.
The book as a whole could also be used as the main text for a one-quarter or
xv
xvi Preface

one-semester course in cluster analysis or unsupervised learning, focusing on the

model-based approach.

Acknowledgements
This book is a truly collaborative effort, and the four authors have contributed
equally. Each of us has contributed to each of the chapters.
We would like to thank Chris Fraley for initially developing the mclust software
and later R package, starting in 1991. This software was of extraordinary quality
from the beginning, and without it this whole field would never have developed as
it did. Luca Scrucca took over the package in 2007, and has enhanced it in many
ways, so we also owe a lot to his work. We would also like to thank the developers
and maintainers of Rmixmod software: Florent Langrognet, Rémi Lebret, Christian
Poli, Serge Iovleff, Anwuli Echenim and Benjamin Auder.
The authors would also like to thank the participants in the Working Group on
Model-based Clustering, which has been gathering every year in the third week of
July since 1994, first in Seattle and then since 2007 in different venues around
Europe and North America. This is an extraordinary group of people from many
countries, whose energy, interactions and intellectual generosity have inspired
us every year and driven the field forward. The book owes a great deal to their
insights.
Charles Bouveyron would like to thank in particular Stéphane Girard, Julien
Jacques and Pierre Latouche, for very fruitful and friendly collaborations. Charles
Bouveyron also thanks his coauthors on this topic for all the enjoyable collabora-
tions: Laurent Bergé, Camille Brunet-Saumard, Etienne Côme, Marco Corneli,
Julie Delon, Mathieu Fauvel, Antoine Houdard, Pierre-Alexandre Mattei, Cordelia
Schmid, Amandine Schmutz and Rawya Zreik. He would like also to warmly thank
his family, Nathalie, Alexis, Romain and Nathan, for their love and everyday
support in the writing of this book.
Gilles Celeux would like to thank his old and dear friends Jean Diebolt and
Gérard Govaert for the long and intensive collaborations. He also thanks his
coauthors in the area Jean-Patrick Baudry, Halima Bensmail, Christophe Biernacki,
Guillaume Bouchard, Vincent Brault, Stéphane Chrétien, Florence Forbes, Raphaël
Gottardo, Christine Keribin, Jean-Michel Marin, Marie-Laure Martin-Magniette,
Cathy Maugis-Rabusseau, Abdallah Mkhadri, Nathalie Peyrard, Andrea Rau,
Christian P. Robert, Gilda Soromenho and Vincent Vandewalle for nice and
fruitful collaborations. Finally, he would like to thank Maı̈lys and Maya for their
love.
Brendan Murphy would like to thank John Hartigan for introducing him to
clustering. He would like to thank Claire Gormley, Paul McNicholas, Luca Scrucca
and Michael Fop with whom he has collaborated extensively on model-based
clustering projects over a number of years. He also would like to thank his
students and coauthors for enjoyable collaborations on a wide range of model-
based clustering and classification projects: Marco Alfò, Francesco Bartolucci,
Nema Dean, Silvia D’Angelo, Gerard Downey, Bailey Fosdick, Nial Friel, Marie
Preface xvii

Galligan, Isabella Gollini, Sen Hu, Neil Hurley, Dimitris Karlis, Donal Martin,
Tyler McCormick, Aaron McDaid, Damien McParland, Keefe Murphy, Tin Lok
James Ng, Adrian O’Hagan, Niamh Russell, Michael Salter-Townshend, Lucy
Small, Deirdre Toher, Ted Westling, Arthur White and Jason Wyse. Finally, he
would like to thank his family, Trish, Áine and Emer for their love and support.
Adrian Raftery thanks Fionn Murtagh, with whom he first encountered model-
based clustering and wrote his first paper in the area in 1984, Chris Fraley for a
long and very fruitful collaboration, and Luca Scrucca for another very successful
collaboration. He warmly thanks his Ph.D. students who have worked with him
on model-based clustering, namely Jeff Banfield, Russ Steele, Raphael Gottardo,
Nema Dean, Derek Stanford and William Chad Young for their collaboration
and all that he learned from them. He also thanks his other coauthors in the
area, Jogesh Babu, Jean-Patrick Baudry, Halima Bensmail, Roger Bumgarner,
Simon Byers, Jon Campbell, Abhijit Dasgupta, Mary Emond, Eric Feigelson,
Florence Forbes, Diane Georgian-Smith, Ken Lo, Alejandro Murua, Nathalie
Peyrard, Christian Robert, Larry Ruzzo, Jean-Luc Starck, Ka Yee Yeung, Naisyin
Wang and Ron Wehrens for excellent collaborations.
Raftery would like to thank the Office of Naval Research and the Eunice
Kennedy Shriver National Institute of Child Health and Human Development
(NICHD grants R01 HD054511 and R01 HD070936) for sustained research support
without which this work could not have been carried out. He wrote part of the
book during a fellowship year at the Center for Advanced Study in the Behavioral
Sciences (CASBS) at Stanford University in 2017–2018, which provided an ideal
environment for the sustained thinking needed to complete a project of this kind.
Finally he would like to thank his wife, Hana Ševčı́ková, for her love and support
through this project.
1
Introduction

Cluster analysis and classification are two important tasks which occur daily
in everyday life. As humans, our brain naturally clusters and classifies animals,
objects or even ideas thousands of times a day, without fatigue. The emergence
of science has led to many data sets with clustering structure that cannot be
easily detected by the human brain, and so require automated algorithms. Also,
with the advent of the “Data Age,” clustering and classification tasks are often
repeated large numbers of times, and so need to be automated even if the human
brain could carry them out.
This has led to a range of clustering and classification algorithms over the
past century. Initially these were mostly heuristic, and developed without much
reference to the statistical theory that was emerging in parallel. In the 1960s, it
was realized that cluster analysis could be put on a principled statistical basis by
framing the clustering task as one of inference for a finite mixture model. This
has allowed cluster analysis to benefit from the inferential framework of statistics,
and provide principled and reproducible answers to questions such as: how many
clusters are there? what is the best clustering algorithm? how should we deal with
outliers?
In this book, we describe and review the model-based approach to cluster
analysis which has emerged in the past half-century, and is now an active research
field. We describe the basic ideas, and aim to show the advantages of thinking in
this way, as well as to review recent developments, particularly for newer types of
data such as high-dimensional data, network data, textual data and image data.

1.1 Cluster Analysis

The goal of cluster analysis is to ﬁnd meaningful groups in data. Typically, in the
data these groups will be internally cohesive and separated from one another. The
purpose is to ﬁnd groups whose members have something in common that they
do not share with members of other groups.

1.1.1 From Grouping to Clustering

The grouping of objects according to things they have in common goes back at
least to the beginning of language. A noun (such as “hammer”) refers to any one
of a set of diﬀerent individual objects that have characteristics in common. As
1
2 Introduction

Greene (1909) remarked, “naming is classifying.” Plato was among the first to
formalize this with his Theory of Forms, defining a Form as an abstract unchanging
object or idea, of which there may be many instances in practice. For example, in
Plato’s Cratylus dialogue, he has Socrates giving the example of a blacksmith’s
tool, such as a hammer. There are many hammers in the world, but just one
Platonic Form of “hammerness” which is the essence of all of them.
Aristotle, in his History of Animals, classified animals into groups based on
their characteristics. Unlike Plato, he drew heavily on empirical observations. His
student Theophrastus did something similar for plants in his Enquiry Into Plants.

An early and highly inﬂuential example of the systematic grouping of objects

based on measured empirical characteristics is the system of biological classifica-
tion or taxonomy of Linnaeus (1735), applied to plants by Linnaeus (1753) and to
animals by Linnaeus (1758). For example, he divided plants into 24 classes, includ-
ing flowers with one stamen (Monandria), flowers with two stamens (Diandria)
and flowerless plants (Cryptogamia). Linnaeus’ methods were based on data but
were subjective. Adanson (1757, 1763) developed less subjective methods using
multiple characteristics of organisms.
Cluster analysis is something more: the search for groups in quantitative data
using systematic numerical methods. Perhaps the earliest methods that satisfy
this description were developed in anthropology, and mainly consisted of defining
quantitative measures of difference and similarity between objects (Czekanowski,
1909, 1911; Driver and Kroeber, 1932). Most of the early clustering methods were
based on measures of similarity between objects, and Czekanowski (1909) seems
to have been the first to define such a measure for clustering purposes.
Then development shifted to psychology, where Zubin (1938) proposed a method
for rearranging a correlation matrix to yield clusters. Stephenson (1936) proposed
the use of factor analysis to identify clusters of people, while, in what seems to
have been the first book on cluster analysis, Tryon (1939) proposed a method
for clustering variables similar to what is now called multiple group factor analy-
sis. Cattell (1944) also introduced several algorithmic and graphical clustering
methods.
In the 1950s, development shifted again to biological taxonomy, the original
problem addressed by the ancient Greeks and the eighteenth-century scientists
interested in classification. It was in this context that the single link hierarchical
agglomerative clustering method (Sneath, 1957), the average link method and
the complete link method (Sokal and Michener, 1958) were proposed. These are
sometimes thought of as marking the beginning of cluster analysis, but in fact
they came after a half-century of previous, though relatively slow development.
They did mark the takeoff of the area, though. The development of computational
power and the publication of the important book of Sokal and Sneath (1963) led
to a rapid expansion of the use and methodology of cluster analysis, which has
not stopped in the past 60 years.
Many of the ensuing developments in the 1970s and 1980s were driven by
applications in market research as well as biological taxonomy. From the 1990s
1.1 Cluster Analysis 3

there was a further explosion of interest fueled by new types of data and questions,
often involving much larger data sets than before. These include finding groups
of genes or people using genetic microarray data, finding groups and patterns in
retail barcode data, finding groups of users and websites from Internet use data,
and automatic document clustering for technical documents and websites.
Another major area of application has been image analysis. This includes medical
image segmentation, for example for finding tumors in digital medical images such
as X-rays, CAT scans, MRI scans and PET scans. In these applications, a cluster
is typically a set of pixels in the image. Another application is image compression,
using methods such as color image quantization, where a cluster would correspond
to a set of color levels. For a history of cluster analysis to 1988, see Blashfeld and
Aldenderfer (1988).

1.1.2 Model-based Clustering

Most of the earlier clustering methods were algorithmic and heuristic. The majority
were based on a matrix of measures of similarity between objects, which were
in turn derived from the objects’ measured characteristics. The purpose was to
divide or partition the data into groups such that objects in the same group were
similar, and were dissimilar from objects in other groups. A range of automatic
algorithms for doing this was proposed, starting in the 1930s.
These developments took place largely in isolation from mainstream statistics,
much of which was based on a probability distribution for the data. At the same
time, they left several practical questions unresolved, such as which of the many
available clustering methods to use? How many clusters should we use? How
should we treat objects that do not fall into any group, or outliers? How sure are
we of a clustering partition, and how should we assess uncertainty about it?
The mainstream statistical approach of specifying a probability model for the
full data set has the potential to answer these questions. The main statistical
model for clustering is a finite mixture model, in which each group is modeled by
its own probability distribution.
The first successful method of this kind was developed in sociology in the early
1950s for multivariate discrete data, where multiple characteristics are measured
for each object, and each characteristic can take one of several values. Data of this
kind are common in the social sciences, and are typical, for example, of surveys.
The model proposed was called the latent class model, and it specified that within
each group the characteristics were statistically independent (Lazarsfeld, 1950a,c).
We discuss this model and its development in Chapter 6.
The dominant model for clustering continuous-valued data is the mixture
of multivariate normal distributions. This seems to have been first mentioned
by Wolfe (1963) in his Master’s thesis at Berkeley. John Wolfe subsequently
developed the first real software for estimating this model, called NORMIX, and
also developed related theory (Wolfe, 1965, 1967, 1970), so he has a real claim
to be called the inventor of model-based clustering for continuous data. Wolfe
proposed estimating the model by maximum likelihood using the EM algorithm,
4 Introduction

which is striking since he did so ten years before the article of Dempster et al.
(1977) that popularized the EM algorithm. This remains the most used estimation
approach in model-based clustering. We outline the early history of model-based
clustering in Section 2.9, after we have introduced the main ideas.
Basing cluster analysis on a probability model has several advantages. In essence,
this brings cluster analysis within the range of standard statistical methodology
and makes it possible to carry out inference in a principled way. It turns out that
many of the previous heuristic methods correspond approximately to particular
clustering models, and so model-based clustering can provide a way of choosing
between clustering methods, and encompasses many of them in its framework. In
our experience, when a clustering method does not correspond to any probability
model, it tends not to work very well. Conversely, understanding what probability
model a clustering method corresponds to can give one an idea of when and why
it will work well or badly.
It also provides a principled way to choose the number of clusters. In fact, the
choice of clustering model and of number of clusters can be reduced to a single
model selection problem. It turns out that there is a trade-oﬀ between these
choices. Often, if a simpler clustering model is chosen, more clusters are needed
to represent the data adequately.
Basing cluster analysis on a probability model also leads to a way of assessing
uncertainty about the clustering. In addition, it provides a systematic way of
dealing with outliers by expanding the model to account for them.

1.2 Classification
The problem of classification (also called discriminant analysis) involves classifying
objects into classes when there is already information about the nature of the
classes. This information often comes from a data set of objects that have already
been classified by experts or by other means. Classification aims to determine
which class new objects belong to, and develops automatic algorithms for doing
so. Typically this involves assigning new observations to the class whose objects
they most closely resemble in some sense.
Classification is said to be a “supervised” problem in the sense that it requires
the supervision of experts to provide some examples of the classes. Clustering, in
contrast, aims to divide a set of objects into groups without any examples of the
“true” classes, and so is said to be an “unsupervised” problem.

1.2.1 From Taxonomy to Machine Learning

The history of classification is closely related to that of clustering. Indeed, the
practical interest of taxonomies of animals or plants is to use them to recognize
samples on the field. For centuries, the task of classification was carried out
by humans, such as biologists, botanists or doctors, who learned to assign new
observations to specific species or diseases. Until the twentieth century, this was
done without automatic algorithms.
1.2 Classification 5

The first statistical method for classification is due to Ronald Fisher in his
famous work on discriminant analysis (Fisher, 1936). Fisher asked what linear
combination of several features best discriminates between two or more populations.
He applied his methodology, known nowadays as Fisher’s discriminant analysis or
linear discriminant analysis, to a data set on irises that he had obtained from the
botanist Edgar Anderson (Anderson, 1935).
In a following article (Fisher, 1938), he established the links between his
discriminant analysis method and several existing methods, in particular analysis
of variance (ANOVA), Hotelling’s T-squared distribution (Hotelling, 1931) and
the Mahalanobis generalized distance (Mahalanobis, 1930). In his 1936 paper,
Fisher also acknowledged the use of a similar approach, without formalization, in
craniometry for quantifying sex differences in measurements of the mandible.
Discriminant analysis rapidly expanded to other application fields, including
medical diagnosis, fault detection, fraud detection, handwriting recognition, spam
detection and computer vision. Fisher’s linear discriminant analysis provided good
solutions for many applications, but other applications required the development
of specific methods.
Among the key methods for classification, logistic regression (Cox, 1958) ex-
tended the usual linear regression model to the case of a categorical dependent
variable and thus made it possible to do binary classification. Logistic regression
had a great success in medicine, marketing, political science and economics. It
remains a routine method in many companies, for instance for mortgage de-
fault prediction within banks or for click-through rate prediction in marketing
companies.
Another key early classification method was the perceptron (Rosenblatt, 1958).
Originally designed as a machine for image recognition, the perceptron algorithm
is supposed to mimic the behavior of neurons for making a decision. Although
the first attempts were promising, the perceptron appeared not to be able to
recognize many classes without adding several layers. The perceptron is recognized
as one of the first artificial neural networks which recently revolutionized the
classification field, partly because of the massive increase in computing capabilities.
In particular, convolutional neural networks (LeCun et al., 1998) use a variation
of multilayer perceptrons and display impressive results in specific cases.
Before the emergence of convolutional neural networks and deep learning,
support vector machines also pushed forward the performances of classification
at the end of the 1990s. The original support vector machine algorithm or SVM
(Cortes and Vapnik, 1995), was invented in 1963 and it was not to see its first
implementation until 1992, thanks to the “kernel trick” (Boser et al., 1992). SVM
is a family of classifiers, defined by the choice of a kernel, which transform the
original data in a high-dimensional space, through a nonlinear projection, where
they are linearly separable with a hyperplane. One of the reasons for the popularity
of SVMs was their ability to handle data of various types thanks to the notion of
kernel.
As we will see in this book, statistical methods were able to follow the different
revolutions in the performance of supervised classification. In addition, some of
6 Introduction

the older methods remain reference methods because they perform well with low
complexity.

1.2.2 Model-based Discriminant Analysis

Fisher discriminant analysis (FDA, Fisher (1936)) was the first classification
method. Although Fisher did not describe his methodology within a statistical
modeling framework, it is possible to recast FDA as a model-based method.
Assuming normal distributions for the classes with a common covariance matrix
yields a classification rule which is based on Fisher’s discriminant function. This
classification method, named linear discriminant analysis (LDA), also provides a
way to calculate the optimal threshold to discriminate between the classes within
Fisher’s discriminant subspace (Fukunaga, 1999).
An early work considering class-conditional distributions in the case of discrimi-
nant analysis is due to Welch (1939). He gave the first version of a classification rule
in the case of two classes with normal distributions, using either Bayes’ theorem
(if the prior probabilities of the classes are known) or the Neyman–Pearson lemma
(if these prior probabilities have to be estimated). Wald (1939, 1949) developed
the theory of decision functions which offers a sound statistical framework for
further work in classification. Wald (1944) considered the problem of assigning
an individual into one of two groups under normal distributions with a common
covariance matrix, the solution of which involves Fisher’s discriminant function.
Von Mises (1945) addressed the problem of minimizing the classification error
in the case of several classes and proposed a general solution to it. Rao (1948,
1952, 1954) extended this to consider the estimation of a classification rule from
samples. See Das Gupta (1973) and McLachlan (1992) for reviews of the earlier
development of this area.
Once the theory of statistical classification had been well established, researchers
had to face new characteristics of the data, such as high-dimensional data, low sam-
ple sizes, partially supervised data and non-normality. Regarding high-dimensional
data, McLachlan (1976) realized the importance of variable selection to avoid the
curse of dimensionality in discriminant analysis. Banfield and Raftery (1993) and
Bensmail and Celeux (1996) proposed alternative approaches using constrained
Gaussian models. About partially supervised classification, McLachlan and Gane-
salingam (1982) considered the use of unlabeled data to update a classification
rule in order to reduce the classification error. Regarding non-normality, Celeux
and Mkhadri (1992) proposed a regularized discriminant analysis technique for
high-dimensional discrete data, while Hastie and Tibshirani (1996) considered
the classification of non-normal data using mixtures of Gaussians. In Chapter
4 specific methods for classification with categorical data are presented. These
topics will be developed in this book, in Chapters 4, 5 and 8.
1.3 Examples 7

1.3 Examples
We now brieﬂy describe some examples of cluster analysis and discriminant
analysis.

Example 1: Old Faithful geyser data

The Old Faithful geyser in Yellowstone National Park, Wyoming erupts every
35–120 minutes for about one to ﬁve minutes. It is useful for rangers to be able
to predict the time to the next eruption. The time to the next eruption and its
duration are related, in that the longer an eruption lasts, the longer the time until
the next one. The data we will consider in this book consist of observations on
272 eruptions Azzalini and Bowman (1990). Data on two variables were measured:
the time from one eruption to the next one, and the duration of the eruption.
These data are often used to illustrate clustering methods.

Example 2: Diagnosing type of diabetes

Figure 1.1 shows measurements made on 145 subjects with the goal of diagnosing
diabetes and, for diabetic patients, the type of diabetes present. The data consist
of the area under a plasma glucose curve (glucose area), the area under a plasma in-
sulin curve (insulin area) and the steady-state plasma glucose response (SSPG) for
145 subjects. The subjects were subsequently clinically classiﬁed into three groups:
chemical diabetes (Type 1), overt diabetes (Type 2), and normal (non-diabetic).
The goal of our analysis is either to develop a method for grouping patients into
clusters corresponding to diagnostic categories, or to learn a classiﬁcation rule able
to predict the status of a new patient. These data were described and analyzed
by Reaven and Miller (1979).

Example 3: Breast cancer diagnosis

In order to diagnose breast cancer, a fine needle aspirate of a breast mass was
collected, and a digitized image of it was produced. The cells present in the
image were identified, and for each cell nucleus the following characteristics were
measured from the digital image: (a) radius (mean of distances from center to
points on the perimeter); (b) texture (standard deviation of gray-scale values);
(c) perimeter; (d) area; (e) smoothness (local variation in radius lengths); (f)
compactness (perimeter2 / area − 1); (g) concavity (severity of concave portions
of the contour); (h) concave points (number of concave portions of the contour);
(i) symmetry; (j) fractal dimension. The mean, standard deviation and extreme
values of the 10 characteristics across cell nuclei were then calculated, yielding 30
features of the image (Street et al., 1993; Mangasarian et al., 1995).
A pairs plot of three of the 30 variables is shown in Figure 1.2. These were
selected as representing a substantial amount of the variability in the data, and in
fact are the variables with the highest loadings on each of the first three principal
components of the data, based on the correlation matrix.
This looks like a more challenging clustering/classification problem than the first
two examples (Old Faithful data and diabetes diagnosis data), where clustering
8 Introduction

0 500 1000 1500

350
● ●
● ●
● ●
● ● ● ●

300
● ●● ●
●●

● ● ●●
● ● ●●

250
● ●

glucose ●● ● ●● ●

200
● ●
●● ●● ●
●
● ●
● ●

150
●
●
● ● ● ●●
● ● ● ●
● ●
●● ● ● ● ● ● ●
●
●● ● ● ●
●● ●● ● ● ●●●● ● ●
●● ●
● ●

100
●● ●
● ●●● ●●●● ●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●●●
● ●●● ●
●
●
●
● ●
●●●
●
●
●●
●●
●
●●
●
●●
●
●●●●● ●●● ●
●●●
●
● ● ●● ●
●●
●●
●
●●
●●●● ●● ●● ●
● ● ●
●
●
●
●
●●
●
●●
●●
●●
●
●
● ●
●●
●●●● ●
●●●●
●
●
●
●
●
●
●●
● ● ● ●●●● ● ●
●●●
●●
● ●● ● ● ● ●● ●
● ●
1500

● ●
● ●
●
● ● ●●●
● ●
● ● ● ●●●
● ● ● ●
● ●
● ● ● ●
1000

●
● ●●
●●
● ●● ●
● ● ● ●
●
●● ●●● ●●

●●
●
insulin ●●
●● ● ● ●
●● ●
● ●● ● ● ●
●
●●● ● ●
500

● ●
0

● ●

600
● ●
● ●

●● ● ●
● ●
● ●

400
●
●
●
●
●
●
●
●
sspg
●● ● ●● ●●●● ●
● ●●●● ● ●●
●
● ● ●●● ●
● ●● ●● ●
●● ● ● ●● ●● ● ●
● ●●

0
100 150 200 250 300 350 0 200 400 600

Figure 1.1 Diabetes data pairs plot: three measurements on 145 patients.
Source: Reaven and Miller (1979).

was apparent visually from the plots of the data. Here it is hard to discern
clustering in Figure 1.2. However, we will see in Chapter 2 that in this higher
dimensional setting, model-based clustering can detect clusters that agree well
with clinical criteria.

Example 4: Wine varieties

Classifying food and drink on the basis of characteristics is an important use of
cluster analysis. We will illustrate this with a data set giving up to 27 physical
and chemical measurements on 178 wine samples (Forina et al., 1986). The goal of
an analysis like this is to partition the samples into types of wine, and potentially
also by year of production. In this case we know the right answer: there are three
types of wine, and the year in which each sample was produced is also known.
1.3 Examples 9

0.00 0.05 0.10 0.15 0.20

● ● ● ●
● ●

0.4
● ●
● ●
● ● ●●
●● ● ●
● ● ● ● ● ● ●● ● ●

0.3
● ● ● ●
● ● ●●
● ● ●
● ●●
● ● ● ● ●
● ●
● ● ●● ● ● ● ●●● ●●●●
●
●● ●●● ● ● ● ●● ●
● ● ●●●●
concavity1 ●
●● ● ●● ●
● ●●
●●●●●●● ●●

0.2
● ●
● ● ● ●●
● ● ●● ●● ● ●
●●● ●●● ●
●●●●●
● ● ●
● ● ● ●● ●
● ● ●●
●●●●● ● ●●
●● ●●●●●●
●●●●●● ●●
●●●
●
●●
●●●●● ● ● ●●● ●
●
●
●●
●●
●
●● ● ●●● ● ● ●
● ● ● ●●
● ●●
●
●●
●●
●
●●
●
●●●
● ●
●●● ●
●● ● ● ●●●●●●● ● ●
● ●● ●●● ● ●●● ●● ● ●● ● ●●● ●●● ●●● ● ●●●
●●●● ●●
●● ●● ●●
●●● ● ●● ●●●
●● ● ●●●● ● ●
● ●● ●● ●●
●● ●●● ●●●● ●●
● ●●● ●
●● ●● ● ● ●● ● ●●●●● ● ●

0.1
● ●● ●
●● ●●
● ●●
● ●●●●●●● ●●
●● ●●●
●●●●●● ●
●● ●
●●●● ●●●
●●●
●●
●● ●● ●● ●●
●●
●
●
●●
● ●●● ●
● ●
● ●●
● ●● ●●
●
●
● ●
●●
●●●●● ●●●●●●
●●
● ● ●●●● ●
●●●●
● ●
●
●●●●
●● ●●●
●●●●
●●● ● ●●●● ●
●●● ●● ●
●●● ●
●●
●
●●● ●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
● ●
●
●●●
● ● ● ●●●
●●●●
● ●
●●● ●
●●
●
●
●●
●
●
●● ●●
●● ●●●● ●
● ●●
●
●● ●● ●● ● ● ●
●●● ●
● ●
● ● ● ●
● ● ●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●
● ●
●●
●●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●●●●
●●● ●● ● ●
●●●
●●
●
●●
●●●
●
●
●●
●
●●
●●
●●●● ●
●
●● ●
●
●●●●
●●
●●
● ●●
●
●●
●●●
●
●●● ●
●● ●
●● ●●●
●● ● ●● ●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●●
●●
●
●●●
●
●● ●● ●
●●
●●●●
●
●●
●●
●● ●●●
●●
●●
●
●
●●●
●● ●●
●
●● ●●
● ●
●●●● ●●●● ●●●
●● ●●
●●● ● ●●

0.0
●
●
●●
●
●●
●
●● ●●●●●
●●●●● ●●
●●● ●● ●●
●
●●● ●●●
● ●● ● ● ●
0.20

● ●
● ● ● ●
● ● ●
●
● ●
● ● ●● ●
● ●
0.15

●
●● ● ●●●● ●
● ●● ●● ● ●
●●● ● ●● ● ●
● ●●
● ● ● ● ●● ● ● ●
● ●● ● ● ● ●
● ● ●● ● ● ● ●
● ● ●●● ● ●● ● ●
●●●●●● ● ●●● ● ●
0.10

●
●●
●●●
●●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
● ● ●●●
●●
●●
●

●●●●
●
●
● ●
●● ●●● ●● ●
●●●
points1 ●
●●
●●
●●
● ●● ● ●●
●●●
●
●●
●
●
●
●●
●
●
●
●
●●● ●
●●
●●●
●●●●
●
●●
●●
●●● ●●
●
●
● ● ●
● ●
●●●
● ●● ● ●●● ● ●● ●●
● ●●
●●●
●●●●●●●●●● ● ● ● ●●●●●
●●● ●
● ●● ●●● ● ●
●
●● ●● ●
● ● ● ● ●●● ●
● ●●●●● ●
● ●●●●
●●●●
● ● ●● ● ●●●●●● ●
●●
● ● ●●● ●
●●
● ●
●● ●
●●
● ●● ● ● ●
●●●
●
● ●● ●● ●● ●●●
●●●
0.05

●
●●
●● ● ●●●●
● ●●● ●
● ● ●
●
●
●
● ● ● ●●
●● ●●●●●
●●●●
● ●
● ●● ●● ● ● ●

5
● ●

4
● ●
● ● ● ●

● ●

3
● ● ● ●
●
● ●● ●
● ● ●
●
● ●●
● ●● ● ●
●● ● ●
●

●
●
●
●
●
● ●●●
●
●●
●● ● ● ●●
● ●
●
●
●
● texture2
● ●● ● ● ●● ●
● ●
●●
●● ● ●●
● ●● ●●●● ●
● ●

2
●●● ●
●●● ●●● ● ● ● ●
●●● ●● ● ●● ●●●
●● ● ●● ●
● ● ● ●●
●●● ● ●
●●●●●●● ● ●●
●●● ●●
● ● ● ●● ●●●● ● ●
●●● ●●●
● ● ● ● ●
●
●● ●
● ●●●●●●●●●●
●●● ● ●●● ●● ● ●● ● ●●● ●●● ● ● ● ● ●●
●●● ●
●●● ●● ● ● ● ● ●
●●● ●●●●●●● ● ● ●
●
●●●
●● ●●
● ●●● ●●●●
● ● ●●
●●●
●
● ●
●● ●●● ● ● ●● ●● ● ●●
● ●
●●●● ● ●
●
● ●●
●●● ●● ●
●●●●●●●●
●
● ●●● ● ● ●●
●
● ●
●
●●●
●●●
●
●
●●●
●●
● ●
●●●● ●
●
●●
● ●
●
● ● ●● ●
● ●
●
●●● ● ● ● ●
●●
●●●● ●●
●
●●●
● ●
●
●
●●●
●
● ●
●
●●●●●●●●●
● ● ● ● ● ●● ●●
●
●
●
●●
● ●
●● ● ● ● ●● ● ●
● ● ● ● ●
● ● ● ●● ● ● ●
● ●
●●●
● ●
●●●●●
●●
● ●● ● ●●
● ● ● ●●
● ● ●●●●
●●●●●●
●●●●●● ●●●●●●●● ● ●
●●●● ● ●●●
●●●
●●●●
●
● ●
●
●●●
● ●
●
●● ●
●●
●●
●
●●● ●
●●●●● ●● ●
● ●● ●● ●●● ●
●●
●
●●●
●
●●●
●
●●
●●● ●
●●● ●● ●●
●● ●●● ●
● ● ●
●●
●●
●
●●●
● ●●●
● ●● ●● ●●●●
●● ●●●●●●● ●● ●● ● ● ●● ●
● ●●●
●
●●●
●
●● ●● ●
●●●● ●●●●● ●●●● ●
●●●
●
● ●●● ● ●●

1
●
●●●
●●
●
● ●
●●
●●
●
●●●● ●● ●
●●
● ●●
●● ● ●●●●
● ●
●
●●●
● ●● ●
●● ●
●
●●● ● ●● ●
●
● ● ●
● ●● ● ● ● ● ● ●● ●●
●● ●
●●
●●
●●
●●
●●
●
● ●●●●●●●
● ●●
●
●●●
●● ●● ● ●● ● ● ● ●● ●● ● ●●
●●●●●
●●
● ● ● ●● ● ●
● ●●● ● ●●● ● ● ●●●● ● ●
● ●
●
●
●
●●
●●
●●
●
●
●●●
●
●
●●
●
●
●
●
● ●
●
●●●
● ● ●
●● ●●●● ●●●
● ●
●●
●●
●
●●●
●●
●●
●
●●
●●
●
●
●
● ●● ●
●●● ● ●●● ●● ●
●
●
●●
●
●●●
●●
●
●
●
●
●
●●
● ●
●●●●
●
●
● ●● ● ●
●
●●●
●
●
●●●
● ●
●●
● ●
●
●●
●
●●●●
●
●
●
●
● ●●● ●
●
●
●●●●
●●● ●
●●
●
●● ● ●●●
●●●●

0.0 0.1 0.2 0.3 0.4 1 2 3 4 5

Figure 1.2 Pairs plot of three of the 30 measurements on breast cancer

diagnosis images.

Thus we can assess how well various clustering methods perform. We will see in
Chapter 2 that model-based clustering is successful at this task.

Example 5: Craniometric analysis

Here the task is to classify skulls according to the populations from which they
came, using cranial measurements. We will analyze data with 57 cranial measure-
ments on 2,524 skulls. As we will see in Chapter 2, a major issue is determining
the number of clusters, or populations.

Example 6: Identifying mineﬁelds

We consider the problem of detecting surface-laid mineﬁelds on the basis of an
image from a reconnaissance aircraft. After processing, such an image is reduced to
a list of objects, some of which may be mines and some of which may be “clutter”
10 Introduction
Minefield data True classification

Figure 1.3 Mineﬁeld data. Left: observed data. Right: true classiﬁcation
into mines and clutter.

or noise, such as other metal objects or rocks. The objects are small and can be
represented by points without losing much information. The analyst’s task is to
determine whether or not mineﬁelds are present, and where they are. A typical
data set is shown in Figure 1.3.1 The true classiﬁcation of the data between mines
and clutter is shown in the right panel of Figure 1.3. These data are available as
the chevron data set in the mclust R package.
This problem is challenging because the clutter form over two-thirds of the data
points and are not separated from the mines spatially, but rather by their density.

Example 7: Vélib data

This data set has been extracted from the Vélib large-scale bike sharing system
of Paris, through the open-data API provided by the operator JCDecaux. The
real time data are available at https://developer.jcdecaux.com/ (with an api
key) . The data set consists of information (occupancy, number of broken docks,
...) about bike stations collected on the Paris bike sharing system over five weeks,
between February 24 and March 30, 2014. Figure 1.4 presents a map of the Vélib
stations in Paris (left panel) and loading profiles of some Vélib stations (right
panel). The red dots correspond to the stations for which the loading profiles are
displayed on the right panel.
The information can be analyzed in different ways, depending on the objective
or the data type. For instance, the data were first used in Bouveyron et al. (2015)
in the context of functional time series clustering, in order to recover the temporal
pattern of use of the bike stations. This data set will be analyzed in this book
1 Actual minefield data were not available, but the data in Figure 1.3 were simulated according to
specifications developed at the Naval Coastal Systems Station, Panama City, Florida, to represent
minefield data encountered in practice (Muise and Smith, 1992).
1.3 Examples 11
Station NATION Station CHAMPS ELYSEES Station MONTMARTRE

1.0

0.8

0.8

48.90

0.6

0.4

0.2

48.87

0.0

Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15

lat

Station TOUR EIFFEL Station GARE DE LYON Station HOTEL DE VILLE

1.0

1.0

48.84

0.8

0.6

0.4

0.4

48.81

0.2

0.2
0.0

0.0

0.0
Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15

Lun−03

Lun−15

Mar−03

Mar−15

Mer−03

Mer−15

Jeu−03

Jeu−15

Ven−03

Ven−15

Sam−03

Sam−15

Dim−03

Dim−15
2.25 2.30 2.35 2.40 2.45
lon

Figure 1.4 Map of the Vélib stations in Paris (left panel) and loading
proﬁles of some Vélib stations (right panel). The red dots correspond to the
stations for which the loading proﬁles are displayed on the right panel.

alternatively as count data (6), as functional data and as multivariate functional

data (12). These data are available as the velib data set in the funFEM package.

Example 8: Chemometrics data

High-dimensional data are more and more frequent in scientific fields. The NIR
data set (Devos et al., 2009) is one example, from a problem in chemometrics
of discriminating between three types of textiles. The 202 textile samples are
analyzed here with a near-infrared spectrometer that produces spectra with
2,800 wavelengths. Figure 1.5 presents the textile samples as 2,800-dimensional
spectra, where the colors indicate the textile types. This is a typical situation
in chemometrics, biology and medical imaging, where the number of samples is
lower than the number of recorded variables. This situation is sometimes called
“ultra-high dimensionality”. This is a difficult situation for most clustering and
classification techniques, as will be discussed in Chapter 8. Nevertheless, when
using appropriate model-based techniques, it is possible to exploit the blessing of
high-dimensional spaces to efficiently discriminate between the different textile
types. The data set is available in the MBCbook package.

Example 9: Zachary’s karate club network data

The Zachary’s karate club data set (Zachary, 1977) consists of the friendship
network of 34 members of a university-based karate club. It was collected following
observations over a three-year period in which a factional division caused the
members of the club to formally separate into two organizations. While friendships
within the club arose organically, a ﬁnancial dispute regarding the pay of part-time
instructor Mr. Hi tested these ties, with two political factions developing. Key to
the dispute were two members of the network, Mr. Hi and club president John A.
The dispute eventually led to the dismissal of Mr. Hi by John A., and Mr. Hi’s
12 Introduction

Figure 1.5 Some textile samples of the three-class NIR data set.

supporters resigned from the karate club and established a new club, headed by
Mr. Hi. The data set exhibits many of the phenomena observed in social networks,
in particular clustering, or community structure. The data are shown in Figure 1.6
where the friendship network is shown and the locations of Mr. Hi and John A.
within the network are highlighted.

1.4 Software
We will give examples of software code to implement our analyses throughout
the book. Fortunately, model-based clustering is well served by good software,
mostly in the form of R packages. We will primarily use the R packages mclust
(Scrucca et al., 2016) and Rmixmod (Langrognet et al., 2016), each of which carries
out general model-based clustering and classification and has a rich array of
capabilities. The capabilities of these two packages overlap to some extent, but
not completely. An advantage of using R is that it allows one to easily use several
different packages in the same analysis.
We will also give examples using several other R packages that provide additional
model-based clustering and classification capabilities. These include FlexMix
(Leisch, 2004), fpc (Hennig, 2015a), prabclus (Hennig and Hausdorf, 2015), pgmm
(McNicholas et al., 2018), tclust (Iscar et al., 2017), clustMD (McParland and
Gormley, 2017) and HDclassif (Bergé et al., 2016).
1.5 Organization of the Book 13

Mr. Hi

John A.

Figure 1.6 The friendship network of Zachary’s karate club. The two key
members in the dispute within the club, Mr. Hi and John A., are labeled
and colored diﬀerently.

1.5 Organization of the Book

In Chapter 2 we introduce the basic ideas of model-based clustering. In Chapter
3 we discuss some common difficulties with the framework, namely outliers,
degeneracies and non-Gaussian mixture components, and we describe some initial,
relatively simple strategies for overcoming them.
In Chapter 4 we describe model-based approaches to classification. This differs
from clustering in that a training set with known labels or cluster memberships is
available. In Chapter 5 we extend this to discuss semi-supervised classification, in
which unlabeled data are used as part of the training data.
Chapter 6 is devoted to model-based clustering involving discrete data, starting
with the latent class model, which was the earliest instance of model-based
clustering of any kind. We also consider ordinal data, and data of mixed type, i.e.
that include both discrete and continuous variables.
In Chapter 7 we consider the selection of variables for clustering, for both
continuous and discrete data. This is important because if variables are used that
are not useful for clustering, the performance of algorithms can be degraded.
In Chapter 8 we describe methods for model-based clustering for high-dimensio-
nal data. Standard model-based clustering methods can be used in principle
regardless of data dimension, but if the dimension is high their performance can
14 Introduction

decline. We describe a range of dimension reduction, regularization and subspace

methods.
Chapter 9 describes ways of clustering data where the component distributions
are non-Gaussian by modeling them explicitly, in contrast with the component
merging methods of Chapter 3. In Chapter 10 we describe model-based approaches
to clustering nodes in networks, with a focus on social network data. Chapter 11
treats methods for model-based clustering with covariates, with a focus on the
mixture of experts model.
Finally, in Chapter 12, we describe model-based clustering methods for a range
of less standard kinds of data. These include functional data, textual data and
images. They also include data in which both the rows and the columns of the data
matrix may have clusters and we want to detect them both in the same analysis.
The methods we describe fall under the heading of model-based co-clustering.

1.6 Bibliographic Notes

A number of books have been written on the topic of cluster analysis and mixture
modeling and these books are of particular importance in the area of model-based
clustering.
Hartigan (1975) wrote an early monograph on cluster analysis, which included
detailed chapters on k-means clustering and Gaussian model-based clustering.
Everitt and Hand (1981) wrote a short monograph on finite mixture models
including Gaussian, exponential, Poisson and binomial mixture models. Titter-
ington et al. (1985) is a detailed monograph on mixture models and related
topics. McLachlan and Basford (1988) is a detailed monograph on finite mixture
modeling and clustering. Everitt (1993) is a textbook on cluster analysis which
covers a broad range of approaches. Lindsay (1995) gives an in-depth overview of
theoretical aspects of finite mixture modeling. Gordon (1999) gives a thorough
overview of clustering methods, including model-based approaches. McLachlan
and Peel (2000) is an extensive book on finite mixture modeling and clustering.
Frühwirth-Schnatter (2006) is a detailed monograph on finite mixtures and Markov
models with a particular focus on Bayesian modeling. McNicholas (2016a) gives a
thorough overview of model-based clustering approaches developed from a range
of different finite mixtures.
Hennig et al. (2015) is an edited volume on the topic of cluster analysis including
chapters on model-based clustering. Mengersen et al. (2011) and Celeux et al.
(2018a) are edited volumes on the topic of mixture modeling which include chapters
on model-based clustering and applications.
2
Model-based Clustering: Basic Ideas

In this chapter we review the basic ideas of model-based clustering. Model-based

clustering is a principled approach to cluster analysis, based on a probability
model and using standard methods of statistical inference. The probability model
on which it is based is a ﬁnite mixture of multivariate distributions, and we
describe this in Section 2.1, with particular emphasis on the most used family of
models, namely mixtures of multivariate normal distributions.
In Section 2.3 we describe maximum likelihood estimation for this model, with
an emphasis on the Expectation-Maximization (EM) algorithm. The EM algorithm
is guaranteed to converge to a local optimum of the likelihood function, but not
to a global maximum, and so the choice of starting point can be important. We
describe some approaches to this in Section 2.4. In Section 2.5, examples with
known numbers of clusters are presented.
Two persistent questions in the practical use of cluster analysis are: how many
clusters are there? and, which clustering method should be used? It turns out that
determining the number of clusters can be viewed as a model choice problem, and
the choice of clustering method is often at least approximately related to the choice
of probability model. Thus both of these questions are answered simultaneously in
the model-based clustering framework by the choice of an appropriate probability
model, and we discuss model selection in Section 2.6. We will work through some
examples in Section 2.7.

2.1 Finite Mixture Models

Suppose we have data that consist of n multivariate observations, y1 , . . . , yn , each
of dimension d, so that yi = (yi,1 , . . . , yi,d ). A ﬁnite mixture model represents the
probability distribution or density function of one multivariate observation, yi ,
as a ﬁnite mixture or weighted average of G probability density functions, called
mixture components:

G
p(yi ) = τg fg (yi | θg ). (2.1)
g=1

In Equation (2.1), τg is the probability that an observation was generated by the

G
gth component, with τg ≥ 0 for g = 1, . . . , G, and g=1 τg = 1, while fg (· | θg ) is
the density of the gth component given the values of its parameters θg .
15
16 Model-based Clustering: Basic Ideas

Density for 1−dim 2−component normal mixture model

Mixture
0.04 Component 1
Component 2
0.03
Probability Density

0.02
0.01
0.00

50 60 70 80 90

Figure 2.1 Probability density function for a one-dimensional univariate

ﬁnite normal mixture with two mixture components. The individual
component densities multiplied by their mixture probabilities are shown in
red and blue, respectively, and the resulting overall mixture density (which
is the sum of the red and blue curves) is the black curve. The dots show a
sample of size 272 simulated from the density, with the colors indicating the
mixture component from which they were generated.

Most commonly, fg is a multivariate normal distribution. In the univariate

case, when yi is one-dimensional, fg (yi |θg ) is a N (μg , σg2 ) density function, and
θg = (μg , σg ), consisting of the mean and standard deviation for the gth mixture
component.
Figure 2.1 shows an example of the density function for a univariate ﬁnite
normal mixture model with two mixture components, together with a sample
simulated from it. The model parameters were selected by estimating such a model
for the Waiting Time data from the Old Faithful data set in Example 1, yielding
μ1 = 54.7, μ2 = 80.1, σ1 = 5.91, σ2 = 5.87, τ1 = 0.362 and τ2 = 0.638. The R code
to produce Figure 2.1 is shown in Listing 2.1.
The density is clearly bimodal, and the density is lower in the middle between
the two humps, so there is some separation between the two mixture components.
2.1 Finite Mixture Models 17

Listing 2.1: R code for Figure 2.1

# Estimate parameters of 1 d finite normal mixture model
# with 2 components for Waiting time observations from
# the Old Faithful data set :
library ( mclust )
waiting <- faithful $ waiting
n <- length ( waiting )
waiting.Mclust <- Mclust ( waiting , model = " V " ,G =2)

# Plot densities :
x <- seq ( from = min ( waiting ) , to = max ( waiting ) , length
=1000)
den1 <- dnorm (x , mean = waiting.Mclust $ parameters $ mean
[1] ,
sd = sqrt ( waiting.Mclust $ parameters $ variance $ sigmasq
[1]) )
den2 <- dnorm (x , mean = waiting.Mclust $ parameters $ mean
[2] ,
sd = sqrt ( waiting.Mclust $ parameters $ variance $ sigmasq
[2]) )
tau1 <- waiting.Mclust $ parameters $ pro [1]
tau2 <- waiting.Mclust $ parameters $ pro [2]
dens <- tau1 * den1 + tau2 * den2
plot (x , dens , type = " l " , xlab = " y " , ylab = " Probability
Density " ,
ylim = c ( -max ( dens ) / 10 ,1 .05 * max ( dens ) ) ,
main = " Density for 1 -dim 2 -component normal mixture
model " , lwd =2)
lines (x , tau1 * den1 , col = " red " )
lines (x , tau2 * den2 , col = " blue " )
legend ( x = min ( x ) ,y = max ( dens ) , legend = c ( " Mixture " ,"
Component 1 " ," Component 2 " ) ,
col = c ( " black " ," red " ," blue " ) , lty = c (1 ,1 ,1) , lwd = c (2 ,1 ,1)
)

# Simulate and plot a sample from the distribution :

sim.results <- simV ( waiting.Mclust $ parameters ,n , seed
=0)
ysim <- sim.results [ ,2]
groupsim <- sim.results [ , " group " ]
ysim1 <- ysim [ groupsim ==1]
ysim2 <- ysim [ groupsim ==2]
points ( ysim1 , rep ( -max ( dens ) / 20 , length ( ysim1 ) ) , col = " red
" , pch =19)
points ( ysim2 , rep ( -max ( dens ) / 20 , length ( ysim2 ) ) , col = "
blue " , pch =19)
18 Model-based Clustering: Basic Ideas

However, the separation is not total. Also, note that there is some overlap between
the points from the two mixture components. In particular, one of the blue points
is to the left of many of the red points. So even in this fairly clear situation we
would be uncertain about which components the points in the middle belong to,
if they were not conveniently colored. Assessing this kind of uncertainty is one of
the things that model-based clustering allows us to do.
When the data are multivariate, fg is often the multivariate normal or Gaussian
density φg , parameterized by its mean vector, μg and by its covariance matrix Σg ,
and has the form
1
φg (yi |μg , Σg ) = |2πΣg |− 2 exp − 12 (yi − μg )T Σ−1
g (yi − μg ) . (2.2)
Data generated by mixtures of multivariate normal densities are characterized
by groups or clusters centered at the means μg , with increased density for points
nearer the mean. The corresponding surfaces of constant density are ellipsoidal.
Figure 2.2 shows the density contours for a two-dimensional ﬁnite bivariate
normal mixture model with two mixture components. The parameters were those
estimated from the two-dimensional Old Faithful data (Example 1), namely
μ1 = (4.29, 79.97), μ2 = (2.04, 54.48), τ1 = 0.644, τ2 = 0.356,

0.170 0.938 0.069 0.437
Σ1 = , and Σ2 = .
0.938 36.017 0.437 33.708
We can see the ellipsoidal nature of the contours. The values simulated from
the two mixture components do not overlap in this bivariate setting. The six
contours shown were chosen so as to contain 5%, 25%, 50%, 75%, 95% and 99%
of the probability, respectively. The R code used to produce the plot is shown in
Listing 2.2.

2.2 Geometrically Constrained Multivariate Normal Mixture Models

The full multivariate normal finite mixture model specified by Equations (2.1) and
(2.2) has been found useful in many applications. However, it has (G − 1) + Gd +
G{d(d + 1)/2} parameters, and this can be quite a large number. In our simplest
example, the Old Faithful data (Example 1), with d = 2 and G = 2, this is 11
parameters, which is manageable. However, for the more complicated Italian wine
data, with d = 27 and G = 3, this is 1,217 parameters, which is very large. Such
large numbers of parameters can lead to difficulties in estimation, including lack
of precision or even degeneracies. They can also lead to difficulties in interpreting
the results.
In order to alleviate this problem, it is common to specify more parsimonious
versions of the model. One way to do this is via the eigenvalue decomposition of
the group covariance matrices Σg , in the form
Σg = λg Dg Ag DgT . (2.3)
In Equation (2.3), Dg is the matrix of eigenvectors of Σg , Ag = diag{A1,g , . . . , Ad,g }
2.2 Geometrically Constrained Multivariate Normal Mixture Models 19

100
●

Density contours
● ●
● Component 1
● Component 2 ●
● ●
●
90

● ● ● 1
0.01
●
● ●●
●●● ● ● ●
0.021 ●●●● ● ●
●
●● ● ●
● ● ●● ● ● ●● ● ●●
● ●● ● ● ●● ● ● ● ●
0.032
●●●●
●
● ●
●●
● ●● ● ●
● ●● ●● ●
●
●● ●●● ● ●
●● ●0.042 ●
80

● ● ● ●●● ●
● ●● ● ●
● ●● ●●● ● ● ● ●
● ●●● ● ● ●●
●● ●
● ●● ● ● ● ●●
Dimension 2

●
● ●
● ● ●● ● ●●●
● ●● ● ●
● ● ●●● ● ●
● ● ● ● ● ●
● ●
● ● ●
70

● ●
● ● ●●
0.002 ● ●
● ●
● ● ●
● ● ● ● ●
● ● 0.002
●● ●
●●
●● ● ●
60

●● ●
●●● ●
● ● ● ● ● ●
●
●0.032 ●● ●
● ●●
●●● ● ● ● ● ●
●●●● ●●
●
● ● ●●
●
●
● ●● ● ●
● ● ● ●●●
50

● ● ●
● ● ● ●
0.02●
1 ●
● ●● ●
● ●
● 0.011
●
● ● ● ●
●

2 3 4 5

Dimension 1

Figure 2.2 Contours of the density function for a two-dimensional ﬁnite

bivariate normal mixture model with two mixture components (see text for
parameter values). The dots show a sample of size 272 simulated from the
density, with the colors indicating the mixture component from which they
were generated.

is a diagonal matrix whose elements are proportional to the eigenvalues of Σg in

descending order, and λg is the associated constant of proportionality.
Each factor in this decomposition corresponds to a particular geometric prop-
erty of the gth mixture component. The matrix of eigenvectors Dg governs its
orientation in Rd . The diagonal matrix of scaled eigenvalues Ag determines its
shape. For example, if A1,g A2,g , then the gth mixture component is tightly
concentrated around a line in Rd . If A1,g ≈ A2,g A3,g , then the gth mixture
component is tightly concentrated around a two-dimensional plane in Rd . If all
the values of Aj,g are approximately equal, then the gth mixture component is
roughly spherical.
The constant of proportionality λg determines the volume occupied by the
gth mixture component in Rd , which is proportional to λdg |Ag |. It is common to
constrain the matrix Ag so that its determinant is equal to 1, in which case λg
directly determines the volume of the gth component, which is then proportional
20 Model-based Clustering: Basic Ideas

Listing 2.2: R code for Figure 2.2

# Estimate parameters of 2 -component mixture model for the
# 2 d Old Faithful data and plot the density
library ( mclust )
n <- length ( faithful [ ,1])
faithful.Mclust2den <- densityMclust ( faithful , model = " VVV " ,G =2)

# Simulate a sample from the density

sim.results <- simVVV ( faithful.Mclust2den $ parameters ,n , seed =0)
ysim <- sim.results [ , c (2 ,3) ]
groupsim <- sim.results [ , " group " ]
ysim1 <- ysim [ groupsim ==1 ,]
ysim2 <- ysim [ groupsim ==2 ,]

# Plot the density and simulated points

plot ( faithful.Mclust2den , col = " black " , xlab = " Dimension 1 " , ylab = "
Dimension 2 " , xlim = c ( min ( ysim [ ,1]) , max ( ysim [ ,1]) ) , ylim = c ( min (
ysim [ ,2]) , max ( ysim [ ,2]) ) , levels = round ( quantile (
faithful.Mclust2den $ density , probs = c (0 .05 , 0 .25 , 0 .5 , 0 .75 , 0
.95 , 0 .99 ) ) ,3) )
points ( ysim1 , col = " red " , pch =19)
points ( ysim2 , col = " blue " , pch =19)
legend ( x =1 .5 , y =100 , legend = c ( " Density contours " ," Component 1 " ,"
Component 2 " ) , col = c ( " black " ," red " ," blue " ) , lty = c (1 ,0 ,0) , lwd = c
(2 ,0 ,0) , pch = c ( NA ,19 ,19) )

to λdg .1 As a result of these geometric interpretations, the decomposition (2.3)

is sometimes called the geometric decomposition or Volume-Shape-Orientation
(VSO) decomposition.
Parsimony of the finite mixture model given by (2.1) and (2.2) can be enforced in
various ways using the decomposition (2.3). Any or all of the geometric properties,
volume, shape or orientation, can be constrained to be constant across mixture
components. Also, the covariance matrix for any mixture component can be forced
to be spherical (i.e. proportional to the identity matrix I), or diagonal.
This allows for two possible univariate models, and for 14 possible models in
the multivariate case. These are shown in Table 2.1. Each of the multivariate
models is denoted by a three-letter identifier, where the first letter is “E” if the
volumes of the clusters are constrained to be equal, and “V” if not (“E” stands
for “equal” and “V” stands for “varying”). Similarly, the second letter is “E” if
the shape matrices Ag are constrained to be equal across clusters, so that Ag ≡ A
for g = 1, . . . , G, “V” if they are not so constrained and “I” if the clusters are
spherical, in which case Ag = I for g = 1, . . . , G. Finally, the third letter is equal
to “E” if the matrices Dg of eigenvectors specifying the cluster orientations are
constrained to be equal, so that Dg ≡ D for g = 1, . . . , G, to “V” if they are not
so constrained, and to “I” if the clusters are spherical, in which case Dg = I for
g = 1, . . . , G.

1 In earlier work (Banﬁeld and Raftery, 1993), λg was taken to be equal to the ﬁrst eigenvalue of Σg .
2.2 Geometrically Constrained Multivariate Normal Mixture Models 21

Table 2.1 Parameterizations of the covariance matrix Σg in model-based clustering. A denotes

a diagonal matrix.

Identiﬁer Model Distribution Volume Shape Orientation

E Univariate Equal
V Univariate Variable

EII λI Spherical Equal Equal NA

VII λg I Spherical Variable Equal NA

EEI λA Diagonal Equal Equal Axis-aligned

VEI λg A Diagonal Variable Equal Axis-aligned
EVI λAg Diagonal Equal Variable Axis-aligned
VVI λg Ag Diagonal Variable Variable Axis-aligned

EEE Σ Ellipsoidal Equal Equal Equal

VEE λg DADT Ellipsoidal Variable Equal Equal
EVE λDAg DT Ellipsoidal Equal Variable Equal
EEV λDg ADgT Ellipsoidal Equal Equal Variable

VVE λg DAg DT Ellipsoidal Variable Variable Equal

EVV λDg Ag DgT Ellipsoidal Equal Variable Variable
VEV λg Dg ADgT Ellipsoidal Variable Equal Variable
VVV Σg Ellipsoidal Variable Variable Variable

Figure 2.3 Models used in model-based clustering: examples of contours of

the bivariate normal component densities for the 14 parameterizations of the
covariance matrix used in model-based clustering.

Figure 2.3 shows examples of contours of the component densities for the various
models in the two-dimensional case with two mixture components.
Sometimes these constrained models can have far fewer parameters that need
to be independently estimated than the unconstrained model, while ﬁtting the
22 Model-based Clustering: Basic Ideas

Table 2.2 Numbers of parameters needed to specify the covariance matrix for models used in
model-based clustering.

Model # Covariance Parameters

General d = 2, G = 2 d = 27, G = 3
EII 1 1 1
VII G 2 3

EEI d 2 27
VEI G + (d − 1) 3 29
EVI 1 + G(d − 1) 3 79
VVI Gd 4 81

EEE d(d + 1)/2 3 378

VEE G + (d + 2)(d − 1)/2 4 380
EVE 1 + (d + 2G)(d − 1)/2 4 430
EEV 1 + (d − 1) + G[d(d − 1)/2] 4 1080

VVE G + (d + 2G)(d − 1)/2 5 432

EVV 1 + G(d + 2)(d − 1)/2 5 1132
VEV G + (d − 1) + G[d(d − 1)/2] 5 1082
VVV G[d(d + 1)/2] 6 1134

sample data almost as well. When this is the case, they can yield more precise
estimates of model parameters, better out-of-sample predictions, and more easily
interpretable parameter estimates. All the models have Gd parameters for the
component means μg , and (G − 1) parameters for the mixture proportions τg .
Table 2.2 shows the numbers of parameters needed to specify the covariance
matrix for each model in the two-dimensional two-component case, d = 2, G = 2,
and the 27-dimensional three-component case, d = 27, G = 3. These results are
obtained by noting that for one mixture component, the volume is speciﬁed by 1
parameter, the shape by (d − 1) parameters, and the orientation by d(d − 1)/2
parameters.
It is clear that the potential gain in parsimony as measured by number of
parameters is small for the two-dimensional case. But for higher-dimensional cases,
the gain can be large. In the most extreme case in Table 2.2, in the 27-dimensional
case with 3 mixture components, the VVV model requires 1,134 parameters to
represent the covariance matrices, whereas the EII model requires only one.
Also, more parameters are required to specify the shape than the volume, and
far more again to specify the orientation. Thus big gains in parsimony are achieved
by the models that require the orientations to be equal across mixture components,
and the largest gains come from requiring the component densities to be diagonal.
However, as we will see, these most parsimonious models do not always ﬁt the
data adequately.
2.3 Estimation by Maximum Likelihood 23

2.3 Estimation by Maximum Likelihood

We will now describe how the models in Section 2.1 can be estimated by maximum
likelihood. The most common way to do this is via the Expectation-Maximization,
or EM, algorithm (Dempster et al., 1977; McLachlan and Krishnan, 1997). This
is a general approach to maximum likelihood for problems in which the data
can be viewed as consisting of n multivariate observations (yi , zi ), in which yi
is observed and zi is unobserved. If the (yi , zi ) are independent and identically
distributed (iid) according to a probability distribution f with parameters θ, then
the complete-data likelihood is

n
LC (y, z | θ) = f (yi , zi | θ), (2.4)
i=1

where y = (y1 , . . . , yn ) and z = (z1 , . . . , zn ). The observed data likelihood, LO (y | θ),

can be obtained by integrating the unobserved data z out of the complete-data
likelihood,

LO (y | θ) = LC (y, z | θ) dz. (2.5)

The maximum likelihood estimator (MLE) of θ based on the observed data

maximizes LO (y | θ) with respect to θ.
The observed data likelihood, also called the mixture likelihood, or just the
likelihood, can also be written as

n
G
LO (y | θ) = τg fg (yi |θg )
i=1 g=1

n
G
= τg φg (yi |μg , Σg ). (2.6)
i=1 g=1

The EM algorithm alternates between two steps. The ﬁrst is an “E-step,” or

expectation step, in which the conditional expectation of the complete data
log-likelihood given the observed data and the current parameter estimates is
computed. The second is an “M-step,” or maximization step, in which parameters
that maximize the expected log-likelihood from the E-step are determined. In its
application to MLE for mixture models, the unobserved portion of the data, z,
involves quantities that are introduced in order to reformulate the problem for
EM. Under fairly mild regularity conditions, EM can be shown to converge to a
local maximum of the observed-data likelihood (Dempster et al., 1977; Boyles,
1983; Wu, 1983; McLachlan and Krishnan, 1997). The EM algorithm has been
widely used for maximum likelihood estimation of mixture models, with good
results.
In EM for mixture models, the “complete data” are considered to be (yi , zi ),
where zi = (zi,1 , . . . , zi,G ) is the unobserved portion of the data, with

1 if yi belongs to group g
zi,g = (2.7)
0 otherwise.
24 Model-based Clustering: Basic Ideas

τ Z

μ Y Σ

Figure 2.4 Graphical model representation of a Gaussian mixture model

(GMM). Random variables and vectors are nodes represented by circles.
Observed random variables are shown in green, whereas latent (unobserved)
variables are shown in white. The plate, or red rectangle, indicates that the
nodes within it consist of n conditionally independent observations or
instances. Thus Y = (y1 , . . . , yn ) and Z = (z1 , . . . , zn ). Unknown model
parameters are nodes shown only by their names.

We assume that the zi are independent and identically distributed, each accord-
ing to a multinomial distribution of one draw from G categories with probabilities
τ1 , .
. . , τG . We also assume that the density of an observation yi given zi is given
by g=1 fg (yi | θg )zi,g . Then the resulting complete-data log-likelihood is
G

n
G
C (θg , τg , zi,g | y) = zi,g log [τg fg (yi | θg )] . (2.8)
i=1 g=1

It can be helpful in understanding this to represent the Gaussian mixture

model as a graphical model, using a directed acyclic graph (DAG), as shown in
Figure 2.4. In a DAG, the absence of an edge between two nodes indicates that
the corresponding variables are conditionally independent given the intervening
variables.
The E-step of the sth iteration of the EM algorithm for mixture models is given
by
(s) τ̂g(s−1) fg (yi | θ̂g(s−1) )
ẑi,g = G , (2.9)
fh (yi | θ̂h )
(s−1) (s−1)
h=1 τ̂h

where τ̂g(s) is the value of τg after the sth EM iteration. The M-step involves
maximizing (2.8) in terms of τg and θg with zi,g ﬁxed at the values computed in
the E-step, namely ẑi,g .
The quantity ẑi,g = E[zi,g |yi , θ1 , . . . , θG ] for the model (2.1) is the conditional
(s)

expectation of zi,g given the parameter values at the (s − 1)th iteration and the
observed data y. The value ẑi,g of zi,g at a maximum of (2.6) is the estimated
conditional probability that observation i belongs to group g. The maximum
likelihood classiﬁcation of observation i is {h | ẑih = maxg ẑi,g }. As a result,
(1 − maxg ẑi,g ) is a measure of the uncertainty in the classiﬁcation of observation
i (Bensmail et al., 1997).
For multivariate normal mixtures, the E-step is given by (2.9) with fg replaced
2.3 Estimation by Maximum Likelihood 25

by φg as deﬁned in (2.2), regardless of the parameterization. For the M-step,

estimates of the means and probabilities have simple closed-form expressions
involving the data and ẑi,g from the E-step, namely:
n
n̂(s−1)
g
(s−1)
i=1 ẑi,g yi n
(s−1)
(s) (s) (s−1)
τ̂g = ; μ̂g = (s−1)
; n̂g = ẑi,g . (2.10)
n n̂g i=1

Computation of the covariance estimate Σ̂(s) g depends on its parameterization.

For details of the M-step for Σk parameterized by the eigenvalue decomposition
(2.3) see Celeux and Govaert (1995). For instance, for the more general model
“VVV,” where no constraints are placed on the volumes, shapes and orientations of
the mixture covariance matrices, it is
n
ẑi,g (yi − μ̂(s)
g )(yi − μ̂g )
(s−1) (s) T
Σ̂g = i=1
(s)
(s−1)
. (2.11)
n̂g
Convergence of the EM algorithm to a local maximum of the log-likelihood
function can be assessed in several ways. In essence, these consist of seeing whether
the algorithm has been moving slowly in the latest iterations. One possible criterion
is that the log-likelihood has changed very little between the last two iterations; a
typical threshold is a change of less than 10−5 . Another possible criterion is that
the model parameters have changed little between the last two iterations. The R
package mclust uses the log-likelihood criterion by default.
As we will see in Section 3.2, the EM algorithm can also be used in a Bayesian
context to ﬁnd the posterior mode when there is a prior distribution on the
parameters.

Example 1 (ctd.)
We illustrate the EM algorithm for multivariate normal mixture models using the
Old Faithful data. We used the EM algorithm to carry out maximum likelihood
estimation for the model with two mixture components and unconstrained covari-
ance matrices (model “VVV”). We initialized the EM algorithm using a random
partition (other initialization strategies will be discussed in Section 2.4).
A trace of the values of the component means, μ1 and μ2 , at each iteration
of the EM algorithm is shown in Figure 2.5. The algorithm converged within 50
iterations, and the converged estimates are shown by solid circles. Because the
algorithm was initialized with a random partition, the starting estimates of μ1
and μ2 were very similar. R code to produce Figure 2.5 is shown in Listings 2.3
and 2.4;; the other figures in this example can be produced in a similar way.
Figure 2.6 (left) shows the log-likelihood values for each iteration of the EM
algorithm. As the theory predicts, the log-likelihood increases at each iteration.
For the first 30 iterations, the log-likelihood remains relatively constant as the
algorithm struggles to escape from the initial random partition, which gives little
information about the two groups in the data. Then it detects the separation
between the groups, and the log-likelihood increases rapidly between iteration 30
and iteration 40 as the algorithm refines the partition. From iteration 40 onwards
26 Model-based Clustering: Basic Ideas

Mean Parameters EM Trace

Mean 1 ●
● ●
●
Mean 2 ●

90
● ●● ●●●
● ● ●
● ● ● ● ●●
● ●
● ●● ● ●
● ●● ●●●
●● ●● ●●● ●● ●
● ● ● ● ● ●●● ● ●
● ● ●●●● ●● ●
● ● ●
● ● ● ●● ● ● ● ● ●● ●
80

● ● ●● ● ●● ●
● ● ● ● ●● ● ● ●●
● ●
● ● ● ●● ● ● ●● ●● ●●
●● ●● ● ● ● ● ● ●
●● ●● ● ● ● ● ●
● ● ●
● ● ●
●●
● ● ● ● ●●
● ● ● ●● ●
●
Waiting

● ● ● ● ●
70

● ● ●●
● ●
●
●
● ●
● ● ●
● ● ● ●
● ● ●
● ● ● ●
60

● ● ● ●● ●
● ●● ● ● ●●
● ●● ●
● ●
●● ●
● ●● ● ● ●
●●●
● ●● ● ● ●
●● ●● ● ●
● ● ● ● ●
●●●● ● ●
50

● ● ● ●
●●● ● ●
● ● ●
● ● ●
●●
● ●
●● ●
●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Eruptions

Figure 2.5 Estimates of the component mean parameter vectors μ1 and μ2

for the two-component bivariate normal mixture model with unconstrained
variance matrix (VVV) for the Old Faithful data at each iteration of the EM
algorithm. The red and blue solid curves show the estimates of the means of
the first and second mixture components, respectively. The solid circles show
the final estimates. The open circles show the data points, with colors
indicating the final classification from the EM algorithm.

it stabilizes and remains essentially constant, increasing by tiny amounts at each

iteration. By iteration 50 it has fairly clearly converged.
Figure 2.6 (right) shows the estimates of the mixing proportion for the first
(s)
component, τ̂1 , at each iteration. A rapid change between iterations 30 and 40
is again apparent. Similar plots for the elements of the component covariance
matrices, Σg , are shown in Figure 2.7. The parameters for component 1 show
the same rapid change between iterations 30 and 40, while the parameters for
component 2 show a more gradual evolution.
Figure 2.8 shows the evolution of the classification and the fitted model over the
course of the EM iterations. The top left panel shows the random initial partition,
which gives no indication of grouping in the data. The fit after one iteration is
shown in the top right panel. This shows the bivariate normal distributions for the
2.3 Estimation by Maximum Likelihood 27

Listing 2.3: R code for Figure 2.5 (part 1)

# Loading the mclust library
library ( mclust )

# Random initial partition

n <- length ( faithful [ ,1])
set.seed (0)
z.init <- rep (1 , n )
n2 <- rbinom ( n =1 , size =n , prob =0 .5 )
s2 <- sample (1: n , size = n2 )
z.init [ s2 ] <- 2

# EM algorithm step-by-step
# Initialize EM algorithm
itmax <- 50
G <- 2
zmat <- matrix ( rep (0 , n * G ) , ncol = G )
for ( g in (1: G ) ) zmat [ , g ] <- z.init == g
mstep.out <- vector ( " list " , itmax )
estep.out <- vector ( " list " , itmax )

# EM iterations
for ( iter in 1: itmax ) {
# M-step
mstep.tmp <- mstep ( modelName = " VVV " , data = faithful , z = zmat )
mstep.out [[ iter ]] <- mstep.tmp
# E-step
estep.tmp <- estep ( modelName = " VVV " , data = faithful , parameters =
mstep.tmp $ parameters )
estep.out [[ iter ]] <- estep.tmp
zmat <- estep.tmp $ z
}

# Extract classification
EMclass <- rep ( NA , n )
for ( i in 1: n ) {
zmati <- zmat [i ,]
zmat.max <- zmati == max ( zmati )
zmat.argmax <- (1: G ) [ zmat.max ]
EMclass [ i ] <- min ( zmat.argmax )
}

two components (shown by ellipses that represent constant density contours) to

overlap almost completely. Almost all the observations are classified into cluster 2,
1 1
reflecting the fact that ẑi,2 > ẑi,1 for most i. This means that after one iteration,
the estimated posterior probability of having been generated by component 2 is
higher than that of coming from component 1 for almost all observations. In fact,
however, the difference is small and the classification is far from decisive. The
slight preference for component 2 is due largely to the fact that τ̂21 > τ̂11 , i.e. that
after one iteration the estimated mixture probability is higher for component 2
than for component 1.
The bottom left panel of Figure 2.8 shows the situation after the 33rd iteration,
28 Model-based Clustering: Basic Ideas

Listing 2.4: R code for Figure 2.5 (part 2)

# Two-dimensional trace plot of component means
# First , extract the EM iteration estimates of mu_ {g , d }
d <- length ( faithful [1 ,]) # data dimension
mu1 <- matrix ( rep ( NA , itmax * d ) , ncol =2)
mu2 <- matrix ( rep ( NA , itmax * d ) , ncol =2)
for ( iter in (1: itmax ) ) {
mu1 [ iter ,] <- estep.out [[ iter ]] $ parameters $ mean [ ,1]
mu2 [ iter ,] <- estep.out [[ iter ]] $ parameters $ mean [ ,2]
}

# Plot the traces of mu_ {g , d }

plot ( faithful , type = " n " , xlab = " Eruptions " , ylab = " Waiting " , main = "
Mean Parameters EM Trace " )
y1 <- faithful [ EMclass ==2 ,]
y2 <- faithful [ EMclass ==1 ,]
lines ( mu1 [ ,1] , mu1 [ ,2] , type = " l " , lwd =3 , col = " red " )
lines ( mu2 [ ,1] , mu2 [ ,2] , type = " l " , lwd =3 , col = " blue " )
points ( y1 , col = " red " , pch =1)
points ( y2 , col = " blue " , pch =1)
points ( mu1 [ itmax ,1] , mu1 [ itmax ,2] , col = " red " , pch =19)
points ( mu2 [ itmax ,1] , mu2 [ itmax ,2] , col = " blue " , pch =19)
legend ( x = min ( faithful [ ,1]) ,y = max ( faithful [ ,2]) , legend = c ( " Mean 1 " ,
" Mean 2 " ) , col = c ( " red " ," blue " ) , lty = c (1 ,1) , lwd = c (3 ,3) )

Log−Likelihood for EM Iterations EM Iterations for Mixing Proportion 1

●●●●●●●●● ●
●
●
● ●
●
●
−1150

0.46

●
●
● ●
●
● ●●
●● ●●●
●●●●●●●●●●●●●●●●●●●
●
0.44

●
●
Mixing Proportion 1
Log−Likelihood

−1200

0.42

●
0.40
−1250

●
●
0.38

●
●

●
0.36

● ●
●●
●●●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●

0 10 20 30 40 50 0 10 20 30 40 50

Iteration Iteration

Figure 2.6 Left: log-likelihood, and Right: mixing proportion, τ1 , for

component 1, at each iteration of the EM algorithm for the two-component
bivariate normal mixture model with unconstrained variance matrix (VVV)
for the Old Faithful data.

when the algorithm has started to detect the separation between the two clusters.
It is apparent that the classiﬁcation is still inadequate. The bottom right panel
shows the result of the 50th iteration, at which stage the algorithm has converged.
The two clusters have been satisfactorily separated.
2.3 Estimation by Maximum Likelihood 29

EM Iterations for Variance 1 EM Iterations for Variance 2

●●●●● ●
●●●
1.4
● ●● ●●●●●
●● ● ●●
●●●●●●●● ●●
●●●●
●●●●●
●● ●●●●●
●●● ● ●● ●
●●● ● ●● ●
● ●●●●
●●
● ●●
●●●● ●●●●● ● ●
● ●
●●●●●
● ●●●● ●
●●● ●
●● ●
1.2

●● ●
● ● ●
●
●

150
● ●
● ●
● ●
●
1.0

●
● ●
●
● ●

●
0.8
Variance 1

Variance 2
●

100
●
●
● ●
0.6

●
0.4

● ●

50
●

●
● ●
●● ●●●●●●●●
●●
● ●●●●●●●
● ●●●
0.2

●
●●●●●●●●●●
●●●●
●
● ●●●●●●●●● ●
Component 1 Component 1
0.0

● Component 2 ● Component 2

0
0 10 20 30 40 50 0 10 20 30 40 50

Iteration Iteration

EM Iterations for Covariance

●●●
●●●
15

●● ●
●● ●
● ●●●
●●●
●●●●●●●●●●●●● ●
●●●●●●●●●●●●●
●●●
● ●● ●
●●
●
●
●
●
● ●
●
●
●
●
10

●
●
Covariance

●
5

●
●

● ●
● ●●●●●●●●●●
●
Component 1 ●●●● ●●●●●●●●●
● Component 2
0

0 10 20 30 40 50

Iteration

Figure 2.7 Estimates of the variance matrices Σ1 and Σ2 for the

two-component bivariate normal mixture model with unconstrained variance
matrix (VVV) for the Old Faithful data at each iteration of the EM algorithm.
Top left: variance of variable 1. Top right: variance of variable 2. Bottom:
covariance of variables 1 and 2.

The probability density function of the data evaluated at the parameter es-
timates is a ﬁnite mixture distribution, and this is depicted in several ways in
Figure 2.9, using the R code in Listing 2.5.
The uncertainty associated with the allocation of the ith observation’s cluster
membership is measured by
Unceri = 1 − max ẑi,g . (2.12)
g=1,...,G

The uncertainty for data point i, Unceri , will be largest for data points i for which
the ẑi,1 , . . . , ẑi,G are equal to one another, and thus Unceri = 1/G for those data
points. It will be smallest when one of the ẑi,1 , . . . , ẑi,G is close to 1, in which case
30 Model-based Clustering: Basic Ideas

Random Initial Partition

● ●
● Component 1 ● ●
● ● ● ●
● ● ●
90 Component 2 ● ●

90
● ●● ●●● ● ●● ●●●
● ● ● ● ● ●
● ● ● ● ●● ● ● ● ● ●●
● ● ● ●
● ●● ● ● ● ●● ● ●
● ● ● ●●● ● ●● ●●●
●● ●
●
●●● ●● ● ●● ●
●
●●● ●● ●
● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ● ●
● ● ●●
●● ●● ●
● ● ● ● ● ●●●● ●● ●
● ● ●
● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ●● ●
80

80
● ● ● ● ●● ● ● ● ● ● ●● ●
● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ●●
● ●
● ● ● ●● ● ● ●● ●● ●● ● ●
● ● ● ●● ● ● ●● ●● ●●
●● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ●
●● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●
● ● ●
● ● ●●
● ● ● ●
● ● ●
●●
● ● ● ● ●● ● ● ● ● ●●
● ● ● ●● ● ● ● ● ●● ●
● ●
Waiting

waiting
● ● ● ● ● ● ● ● ● ●
70

70
● ● ●● ● ● ●●
● ● ● ●
● ●
● ●
● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●
60

60
● ● ● ●● ● ● ● ● ●● ●
● ●● ● ● ●● ● ●● ● ● ●●
● ●● ● ● ●● ●
● ● ● ●
●● ● ●● ●
● ●● ● ● ● ● ●● ● ● ●
●●●
● ●● ● ● ●●●
● ●● ● ●
●● ●● ● ● ●● ●● ● ●
● ● ● ● ● ● ● ● ● ●
●●●● ● ● ●●●● ● ●
50

50
● ● ● ● ● ● ● ●
●●● ● ● ●●● ● ●
● ● ● ● ● ●
● ● ● ● ● ●
●
●● ● ●
●● ●
●● ● ●● ●
● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Eruptions eruptions

● ●
● ●
● ● ● ●
● ●
● ●
90

90
● ●● ●●● ● ●● ●●●
● ● ● ● ● ●
● ● ● ● ●● ● ● ● ● ●●
● ● ● ●
● ●● ● ● ● ●● ● ●
● ●● ●●● ● ●● ●●●
●● ●●● ●● ●
●
● ●● ●●● ●● ●
●
●
● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ● ●
● ● ●●●● ●● ●
● ● ● ● ● ●●●● ●● ●
● ● ●
● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ●
80

● ● ● ● ●● ● ● ● ● ● ●● ●
● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ●●
● ●
● ● ● ●● ● ● ●● ●● ●● ● ●
● ● ● ●● ● ● ●● ●● ●●
●● ●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ●
●● ●● ● ● ● ● ● ●● ●● ● ● ● ● ●
● ● ●
● ● ●●
● ● ● ●
● ● ●●
●
● ● ● ● ●● ● ● ● ● ●●
● ● ● ●● ● ● ● ● ●● ●
● ●
waiting

waiting

● ● ● ● ● ● ● ● ● ●
70

● ● ●● ● ● ●●
● ● ● ●
● ●
● ●
● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●
60

● ● ● ●● ● ● ● ● ●● ●
● ●● ● ● ●● ● ●● ● ● ●●
● ●● ● ● ●● ●
● ● ● ●
●● ● ●● ●
● ●● ● ● ● ● ●● ● ● ●
●
●●●●● ● ● ●●●
● ●● ● ●
●● ●● ● ● ●● ●● ● ●
● ● ● ● ● ● ● ● ● ●
●●●● ● ● ●●●● ● ●
50

● ● ● ● ● ● ● ●
●●● ● ● ●●● ● ●
● ● ● ● ● ●
● ● ● ● ● ●
●●
● ● ●●
● ●
●● ● ●● ●
● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

Figure 2.8 Classiﬁcations and ﬁtted models for the two-component

bivariate normal mixture model with unconstrained variance matrix (VVV)
for the Old Faithful data during the EM algorithm. Top left: random initial
partition. Top right: iteration 1. Bottom left: iteration 33. Bottom right:
iteration 50 (converged).

it will be close to zero. The uncertainty values for the Old Faithful data are shown
in Figure 2.10, produced using the R code in Listing 2.6.
In the left panel of Figure 2.10, the 5% of the observations with the greatest
uncertainty are shown by large black circles; these are the ones that are roughly
equidistant between the two clusters. The next 20% of the observations in terms
of uncertainty are shown by smaller gray circles, and these are closer to the
cluster centers than the most uncertain points. The least uncertain 75% of the
observations are shown by small gray dots, and these are the ones that are closest
to the cluster centers. In this data set the clusters are well separated, and so there
is little uncertainty about the allocation of most of the data points.
2.4 Initializing the EM Algorithm 31

Listing 2.5: Code for density plots

# Loading the mclust library
library ( mclust )
data ( faithful )

# Clustering with Mclust

faithful.Mclust2 <- Mclust ( faithful , model = " VVV " ,G =2)

# Plotting the densities as contour

surfacePlot ( faithful , faithful.Mclust2 $ parameters , type = " contour " ,
what = " density " )
points ( faithful )

# Alternative ways of plotting the densities

surfacePlot ( faithful , faithful.Mclust2 $ parameters , type = " image " ,
what = " density " )
surfacePlot ( faithful , faithful.Mclust2 $ parameters , type = " persp " ,
what = " density " )

Listing 2.6: Code for Old Faithful uncertainty plot

# Loading the mclust library
library ( mclust )
data ( faithful )

# Clustering with Mclust

faithful.Mclust2 <- Mclust ( faithful , model = " VVV " ,G =2)

# Display uncertainty on cluster assignments

plot ( faithful.Mclust2 , what = " uncertainty " )
surfacePlot ( faithful , faithful.Mclust2 $ parameters , type = " contour " ,
what = " uncertainty " )
points ( faithful )

The right panel of Figure 2.10 shows contours of uncertainty level. These are
slightly nonlinear, reflecting the fact that the two clusters have different covariance
matrices. They also fall away quickly from the contour of highest uncertainty,
reflecting the fact that most of the data points are assigned to clusters with little
uncertainty.

2.4 Initializing the EM Algorithm

The likelihood for a mixture model is in general not convex. As a result there can
be local maxima; an example is given in Biernacki et al. (2003, Section 1). One
consequence is that the converged estimate can depend on the initial value chosen.

For normal mixture models things are even trickier, as there are typically
multiple paths in parameter space along which the likelihood can tend to inﬁnity
at parameter values on the edge of the parameter space (Titterington et al.,
32 Model-based Clustering: Basic Ideas

●
● ● ●
0.005 ● ●● 0.005
90

90
● ● ● ●● ●●● ●
● ●0. ●● ●●
0.02
● ●●● ●● ● ●
● 02 ●●
● ● ●● ●●●●● ●
●
●●●
●
● ●●
●●● ● ●●●●●
●
●
● ●●
● ●●●●●

25
●●● ● ● ● ●
●●●
80

80
● ● ● ●● ●● ●●
● ●
●
● ●

0.0

0.0
● ●●●●●●
● ●
●●
●●●●●●●
●●
● ●●●
●
●● ●● ● ● ● ●●
● ● ●● ● ●● ● ●●5 ●●●
Waiting

Waiting
15 ● ●● ● 1 ●
0.0 ● ●● ● ●●
●
●0.0●
70

70
0.01 ● ●● ●
0.01
●
● ●
● ● ●●
0.005 ● ● 0.00 5● ● ● ●
● ● ●● ● ●
60

60
0.015 ●
●●●
● ●●● ●
●
0.015
●● ●● ●●
● ●
●
●
● ●●
0.02

0.02
●
●●
●●
● ●
● ●●
● ●●● ● ● ●
● ●●●
●●
●●●
●● ● ●●
50

50
●● ●●●●●●●
●●
●●
●
●● ●
0.01 ●● ●●
0.01
●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Eruptions Eruptions
90

Densit
80
Waiting

y
70

Eru
60

ptio

iti ng
ns
50

2.0 2.5 3.0 3.5 4.0 4.5 5.0

Eruptions

Figure 2.9 Two-dimensional probability density function of the data at

the parameter estimates for the Old Faithful data: contour plot, contour plot
with observations, image plot, and perspective plot.

1985). For example, for a one-dimensional mixture of two normal distributions,

the likelihood tends to inﬁnity as σ1 −→ 0 when μ1 = yi for any i. These inﬁnite
solutions are in general degenerate and do not yield estimates with good properties.
The solutions with good properties are local maxima of the likelihood function
that are internal to the parameter space (Redner and Walker, 1984). Thus in
general we are searching for the largest internal local maximum of the likelihood
function.
Because of these properties of the likelihood function, the result can depend
critically on the initial values chosen. In Example 1 with the Old Faithful data,
2.4 Initializing the EM Algorithm 33

Classification Uncertainty

● ●

0.4
●

0.1

0.3
0.0
●

● ●

0.1
● ●

5
● ●

0.3
● ●
90

0.0
● ●● ● ● ● ● ●● ●●●
● ● ●

5
● ● ●

0.2
0.1
● ● ● ● ● ● ● ● ● ● ●●
● ● ● ●

0.2
● ●● ● ●

0.2
● ●● ● ●
● ● ● ● ●● ● ● ● ●●●

5
● ● ●● ●●● ●● ● ●● ●
●
●●● ●● ●
● ● ● ● ● ●● ● ● ● ● ● ●● ● ●●● ● ●

0.1
● ● ● ●●●● ●● ● ● ● ● ● ●●●● ●● ●
● ● ●
● ● ● ●● ● ● ● ● ●● ●

5
● ● ● ● ● ● ●● ● ●● ●
80

0.2
● ● ● ● ● ● ● ● ● ● ● ●● ●
● ● ● ● ●● ● ● ●●

5
● ● ● ● ●● ●● ● ●

● ●● ● ● ●● ● ● ●● ●● ● ● ● ●
● ● ● ●● ● ● ●● ●● ●●
●● ●● ● ● ●● ● ● ●● ●● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ●
● ● ●● ● ●● ● ● ● ●
● ● ●
●●
● ● ● ● ● ● ● ● ● ● ●●
● ● ● ●● ● ● ● ● ●● ●
●

Waiting
waiting

0.35
● ● ● ● ● ● ● ● ●
70

70
● ● ●● ● ● ●●
● ● ● ●
● ●

0.35
● ●
● ● ● ●
● ● ● ● ● ●

0.35
● ● ● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●

0.35
60

60
● ● ● ●● ● ● ● ● ●● ●
● ● ● ● ● ●● ● ●● ● ● ●●
● ●● ● ● ●● ●

0.35
● ● ● ●
●● ● ●● ●
● ●● ● ●
● ●
● ●●●
●
● ●●
●●
● ●
● ●
●

0.35
●● ● ●● ●

● ●
● ●
● ●
●●
●
● ● ●
●●
● ● ● ●
●● ● ●
● ●● ● ● ● ●●●● ● ●
50

50
● ● ● ● ● ● ● ●

0.35
● ●● ● ● ●●● ● ●
● ● ● ● ● ●
● ● ● ● ● ●
●
●● ●

0.35
● ●● ●
● ● ● ●● ●
● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions Eruptions

Figure 2.10 Uncertainty values for the Old Faithful data with two clusters.
Left: uncertainty values for individual observations. The large black circles
indicate the top 5% of the observations in terms of uncertainty, the smaller
gray circles indicate the next 20%, and the light gray dots indicate the
lowest 75%. Right: contours of uncertainty level.

we saw in Section 2.3 that the EM algorithm gave satisfactory performance when
initialized with a random partition. This often does not work well, however.
Here we will describe two methods for selecting initial values that have performed
well in a variety of settings: hierarchical model-based clustering, and the so-called
smallEM method.

2.4.1 Initialization by Hierarchical Model-based Clustering

In its first decades, cluster analysis was done mainly using hierarchical clustering
methods, also called agglomerative hierarchical clustering methods. The starting
point for these methods is a set of pairwise dissimilarities between all objects.
If these are not given or measured directly, they must be computed before the
cluster analysis is carried out. The methods are generally initialized with the trivial
partition where each object is its own cluster. Then, at each stage two clusters are
merged. The pair of clusters to be merged is chosen so as to minimize a criterion
over all possible pairs. Different merging criteria define different hierarchical
clustering methods. If there are n observations, then there are n − 1 stages in the
algorithm, yielding n different clusterings, with n, n − 1, . . . , 1 clusters respectively.
Popular criteria include the single link criterion, defined as the smallest dissimi-
larity between a member of one cluster and a member of the other cluster (Sneath,
1957), the average link criterion, defined as the average between-cluster dissimilar-
ity (Sokal and Michener, 1958), the complete link criterion, defined as the largest
between-cluster dissimilarity (Defays, 1978), and the sum of squares criterion,
defined as the total within-group sum of squares about the group means (Ward,
34 Model-based Clustering: Basic Ideas

1963). These criteria are all special cases of a more general, computationally
efficient algorithm proposed by Lance and Williams (1967).
In hierarchical model-based clustering (Banfield and Raftery, 1993), the criterion
used for deciding which clusters to merge at each stage is no longer based on
dissimilarities. Instead the criterion used is the classification likelihood,

n
LCL (θ, z|y) = fzi (yi |θzi ). (2.13)
i=1

Note that this is different from the observed data (or mixture) likelihood given
by Equation (2.5) in that the set of cluster memberships z is an argument of the
classification likelihood, whereas these are integrated out in the observed data
likelihood. It is also different from the complete-data likelihood given by Equation
(2.4), in that it does not include the mixing probabilities τg corresponding to the
cluster memberships zi .
Methods that attempt to maximize the classification likelihood (2.13) attempt
to estimate the cluster memberships and the model parameters simultaneously
and are not in general asymptotically consistent in the sense of being guaranteed
to give estimates that tend to the true values as sample size increases (Marriott,
1975). However, they are often much more computationally efficient and so can
work well for initialization. Clustering methods using the classification and mixture
likelihoods were compared by Celeux and Govaert (1993).
Another method aimed at maximizing the classification likelihood is the Clas-
sification EM (CEM) algorithm (Celeux and Govaert, 1992).This CEM algo-
rithm incorporates a classification step (C-step) between the E-step and the
M-step of the EM algorithm. The C-step consists of replacing the unobserved
labels hi , i = 1, . . . , n, defining a partition into G clusters of the observations
by {hi | ẑih = maxg ẑi,g }. Thus, the M-step reduces to an estimation of the
(s) (s) (s)

proportions, means and covariance matrices of the resulting clusters. An advantage

of this CEM algorithm, which is a k-means-like algorithm, is that it converges in
a finite number of iterations. In most situations, this number of iterations is small.
Thus, it is recommended and not expensive to run the CEM algorithm many
times from random initial positions to get a good maximizer of the classification
likelihood.
Hierarchical model-based clustering proceeds by successively merging pairs of
clusters corresponding to the greatest increase in the classification likelihood (2.13)
among all possible pairs. In the absence of any information about groupings, the
procedure starts by treating each observation as a singleton cluster. When the
probability model in (2.13) is multivariate normal with the equal-volume spherical
covariance λI (the EI model), model-based hierarchical clustering is equivalent to
hierarchical clustering with the sum of squares criterion.
An advantage of initialization by hierarchical model-based clustering is that it
is often computationally efficient, in that a single pass over the data gives initial
clusterings for all models and numbers of clusters. As we will see, model-based
2.4 Initializing the EM Algorithm 35

Listing 2.7: Initialization with hierarchical model-based clustering

# Calling hc for initializing mclust
faithful.hmc <- hc ( modelName = " VVV " , faithful )
faithful.hmclass <- c ( hclass ( faithful.hmc ,2) )

Listing 2.8: Initialization using a subset in hierarchical model-based clustering

# Initialization by sub-sampling
S <- sample (1: nrow ( faithful ) , size =100)
Mclust ( faithful , initialization = list ( subset = S ) )

clustering can involve ﬁtting large numbers of diﬀerent models, so obtaining initial
partitions for all of them from a single run is appealing.
By default, in the mclust R package, the EM algorithm is initialized by running
the hierarchical model-based clustering with the VVV model. This gives a partition
for each number of clusters. These partitions are used as the initial points for the
algorithm.

Example 1 (ctd.)
In the Old Faithful data, a hierarchical model-based clustering of the data is
carried out using the R commands shown in Listing 2.7.
The results are shown in Figure 2.11. This is close, but not identical, to the
converged maximum likelihood solution shown in the bottom left panel of Figure
2.8. It is close enough to provide a good starting value for the EM algorithm. This
is the default initialization in the mclust package.
A potential disadvantage of hierarchical model-based clustering is that, in its
most basic form, it can take a long time to converge when the data set is large.
This is because it involves computing and maintaining a matrix of the classification
likelihood increase for all pairs of clusters that could potentially be merged. Since
initially there is one cluster per observation, this matrix is of size O(n2 ). If n is
large enough, this matrix does not fit in memory, and manipulating it involves
swapping in and out of memory. When that threshold is passed, computing time
can increase hugely.
A simple fix is to base the initialization by hierarchical model-based clustering
on a random subset of the data. Experience suggests that there are few gains to
be made by increasing the size of the subset beyond about 2,000 (Wehrens et al.,
2004). Code to do this for the Old Faithful data with a subset size of 100 is shown
in Listing 2.8.
36 Model-based Clustering: Basic Ideas

Initial Partition from Hierarchical Model−Based Clustering

●
● Component 1 ●
● ●
● ●
Component 2 ●

90
● ●● ●●●
● ● ●
● ● ● ● ●●
● ●
● ●● ● ●
● ● ● ●●●
●● ●● ●●● ●● ●
● ● ● ● ● ●●● ● ●
● ● ●●
●● ●● ●
● ● ●
● ● ● ●● ● ●● ● ●● ●
80

● ● ● ● ●● ●
● ● ● ● ●● ● ● ●●
● ●
● ● ● ●● ● ● ●● ●● ●●
●● ●● ● ● ● ● ● ●
●● ●● ● ● ● ● ●
● ● ●
● ● ●
●●
● ● ● ● ●●
● ● ● ●● ●
●
Waiting

● ● ● ● ●
70

● ● ●●
● ●
●
●
● ●
● ● ●
● ● ● ●
● ● ●
● ● ● ●
60

● ● ● ●● ●
● ●● ● ● ●●
● ●● ●
● ●
●● ●
● ●● ● ● ●
●●●
● ●● ● ●
●● ●● ● ●
● ● ● ● ●
●●●● ● ●
50

● ● ● ●
●●● ● ●
● ● ●
● ● ●
●●
● ●
●● ●
●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

Eruptions

Figure 2.11 Partition of the Old Faithful data into two clusters using
hierarchical model-based clustering with the unconstrained variance (VVV)
model.

2.4.2 Initialization Using the smallEM Strategy

The “smallEM” strategy (Biernacki et al., 2003) is implemented separately for each
clustering model and number of clusters considered. It consists of the following
steps:
1 Repeat the following M times:
1 Simulate a random set of parameter values θ.
2 Carry out a short run of the EM algorithm starting at the simulated set of
parameter values. This is not run to convergence but instead is terminated
at the ﬁrst iteration s where
O (y|θ̂(s) ) − O (y|θ̂(s−1) )
< ε,
O (y|θ̂(s) ) − O (y|θ̂(1) )
where O (y|θ̂(s) ) = log LO (y|θ̂(s) ) is the mixture (or observed data) log-
likelihood after the sth EM iteration.
2.4 Initializing the EM Algorithm 37

Listing 2.9: Initialization with smallEM

# Load the Rmixmod library
library ( Rmixmod )

# Run six smallEMs ( here limited to two iterations )

nb = 6
par ( mfrow = c (2 ,3) )
res = list () ; likelihood = c ()
myModel = m ixmodGaussianModel ( listModels = ' Gaussian_pk_Lk_Ck ')
for ( i in 1: nb ) {
res [[ i ]] = mixmodCluster ( faithful , nbCluster =2 , model = myModel ,
strategy = mixmodStrategy ( initMethod = ' random ' , nbTryInInit =1 ,
nbIterationInInit =1 , nbIterationInAlgo =1) )
likelihood [i] = res [[ i ]] @bestResult@likelihood
plot ( faithful , col = res [[ i ]] @b e s t R e s u l t @ p a r t i t i o n +1 , pch = c (17 ,19) [
res [[ i ]] @ b e s t R e s u l t @ p a r t i t i o n ])
title ( main = paste ( ' Likelihood = ' , round ( likelihood [ i ]) ) ) }

# Run EM initialized with the best smallEM solution

par ( mfrow = c (1 ,1) )
final.res = mixmodCluster ( faithful , nbCluster =2 , model = myModel )
plot ( faithful , col = f i n a l . r e s @ b e s t R e s u l t @ p a r t i t i o n +1 , pch = c (17 ,19) [
f i n a l . r e s @ b e s t R e s u l t @ p a r t i t i o n ])
title ( main = paste ( ' Likelihood = ' , round (
final.res@bestResult@likelihood )))

3 Record the mixture log-likelihood at the stopping point of the short EM run.
2 Among the stopping points of the short EM runs, choose the one with the
highest mixture likelihood. Using it as a starting point, run the EM algorithm
to convergence.

Biernacki et al. (2003) recommended using ε = 0.01. They compared the

smallEM strategy with several other possible initialization strategies and found it
to work at least as well as the other ones they considered. The smallEM strategy
with M = 5 is the default initialization method in the Rmixmod package.
Note that smallEM starts with random sets of parameter values. By contrast,
for our illustration of the EM algorithm in Section 2.3 we used a random partition
of the data points. This was a bad initial value, although the EM algorithm was
able to recover from it and ﬁnd a good estimate eventually. Random parameter
values tend to provide better starting values than random partitions of the data
points.

Example 1 (ctd.)
Results from the smallEM initialization method for the Old Faithful data are
shown in Figures 2.12 and 2.13. They were produced using the R commands in
Listing 2.9.
The smallEM solutions shown in Figure 2.12 are quite similar, even though
38 Model-based Clustering: Basic Ideas

Likelihood = −1174 Likelihood = −1192 Likelihood = −1243

●
●
● ●
●
●
90

90
● ●● ●●●
● ● ●
● ● ●● ●●
● ●
● ●● ●●
● ● ● ●● ●
●● ● ● ●●● ●● ●
● ● ● ● ● ●●● ● ● ●
● ● ●●●
●● ●● ● ● ●
●● ●●●● ● ● ● ●● ●
80

80
● ● ● ● ●● ●
● ● ● ● ●● ● ●●●
● ●● ● ● ●●● ● ●●●●●● ●
●● ●● ● ● ●● ● ●
●● ●● ● ● ● ● ●
●● ●
● ● ●●●
● ● ● ●● ● ●
waiting

waiting

waiting
● ● ●● ● ●
●
● ●● ● ● ● ●
70

70
●●● ●
● ● ●
● ●
● ●
● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●
60

60
● ● ● ●●● ● ● ● ●●●
● ●● ● ● ●● ● ●● ● ● ●●
● ●● ● ● ●● ●
● ● ● ●
●● ● ●● ●
● ●● ● ● ● ● ●● ● ● ●
●
●●●●● ● ● ●●●
● ●● ● ●
●● ●● ● ● ●● ●● ● ●
● ● ●● ● ● ● ●● ●
●●●● ● ● ●●●● ● ●
50

50
● ●●● ● ●●●
●●
●●● ●●
●●●
● ● ● ● ● ●
● ● ● ● ● ●
●●
● ● ●●
● ●
●● ● ●● ●
● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
eruptions eruptions eruptions

Likelihood = −1258 Likelihood = −1250 Likelihood = −1200

90
80

80
●

●
waiting

waiting

waiting
●
● ● ● ● ●
70

70
● ● ●
● ● ●
● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ● ● ● ●
60

60
● ● ● ●●● ● ● ● ●●● ● ● ● ●●●
● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●●
● ●● ● ● ●● ● ● ●● ●
● ● ● ● ● ●
●● ● ●● ● ●● ●
● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ●
●●●●●● ● ● ●●●●●● ● ● ●●●●●● ● ●
●● ●● ● ● ●● ●● ● ● ●● ●● ● ●
● ● ●● ● ● ● ●● ● ● ● ●● ●
●●●● ● ● ●●●● ● ● ●●●● ● ●
50

50
● ●●● ● ●●● ● ●●●
●●●
●● ●●
●●● ●●
●●●
● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
●●
● ● ●●
● ● ●●
● ●
●● ● ●● ● ●● ●
● ● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0
eruptions eruptions eruptions

Figure 2.12 Initialization with smallEM on the Old Faithful Geyser data:
six runs of two-iteration EM to select the best initialization. Here, the best
initialization is the top-left panel with log-likelihood equal to −1, 174.
“Likelihood” refers to the maximized mixture log-likelihood.

each one is the result of only two EM iterations. However, the differences in
log-likelihood between the solutions are quite substantial.
The final estimate from smallEM shown in Figure 2.13 is the same classification
as that from the random partition initial value shown in Figure 2.5. This is in
spite of the fact that the initial values are dramatically different.
Finally, especially in the clustering context, the CEM algorithm could be a
good alternative to the smallEM strategy since the CEM algorithm is expected to
converge rapidly and can be repeated many times at small cost (Biernacki et al.,
2003).
We also illustrate the possible differences between the partitions derived from
the EM and the CEM algorithms with the Old Faithful geyser data. If the number
of clusters is fixed at G = 2, the CEM algorithm with the VVV model from a
random initial solution gives exactly the same partition, displayed in Figure 2.13,
that we get from EM initialized with the smallEM strategy. This is not surprising
since with G = 2 the two mixture components are well separated. But with G = 3
and the VVV model, the EM and the CEM algorithms yield different partitions;
see Figure 2.14.
2.5 Examples with Known Number of Clusters 39

Likelihood = −1130

90
80
waiting

●
70

● ● ●
● ●
● ● ●
● ● ● ●
60

● ● ● ● ●●
● ●● ●● ● ●
● ●● ●
● ●
●● ●
● ●● ● ● ●
●● ● ●● ● ● ●
● ● ● ● ● ●
● ● ● ● ●
● ●● ● ● ●
50

● ● ● ●
● ●● ● ●
● ● ●
● ● ●
● ●● ●
● ● ●

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions

Figure 2.13 Initialization with smallEM on Old Faithful geyser data: ﬁnal
result of the EM algorithm from the best two-iteration run. “Likelihood”
refers to the maximized mixture log-likelihood.

2.5 Examples with Known Number of Clusters

In this section, we will analyze several of the data sets introduced in Chapter 1. In
each case, we will take the number of clusters to be known in advance. This is often
not the case in practice, and in Section 2.6 we will describe methods for inferring
the number of clusters from the data. We will analyze the data using model-based
clustering, and also using some other popular methods for comparison.

Example 2 (ctd.): Diabetes diagnosis

We will now analyze the diabetes measurements shown in Figure 1.1. To carry out
model-based clustering, we will take the number of clusters to be equal to 3, the
number in the clinical classification. We will also use the unconstrained covariance
model (VVV), in the absence of any information about likely constraints. In Section
2.6, we will see how to relax the assumption that the number of clusters is known,
and instead select the clustering model using the data.
Figure 2.15 shows the result of model-based clustering for these data. The
clusters clearly have quite different volumes, shapes and orientations. The compact
blue cluster captures the non-diabetic (normal) subjects, and corresponds to
relatively low values of all three measurements. The red cluster largely captures
the chemical diabetics, who can have a wide range of levels of SSPG, up to very
high ones. The green cluster corresponds to the subjects suffering from overt
diabetes, who have a wide range of both glucose and insulin measurements, again
up to very high levels. Insulin and glucose levels are highly correlated with one
another for the overt diabetics, but less so for the other groups.
40 Model-based Clustering: Basic Ideas

Log−Likelihood = −1114 Log−Likelihood = −1120

90
80

80
waiting

waiting
70

70
60

60
50

50
1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

Figure 2.14 Classiﬁcation plots for the Old Faithful data with the VVV
model and G = 3. Left: EM results. Right: CEM results.

The model-based clustering classiﬁcation corresponds quite well to the clinical

classification, as can be seen from Figure 2.16. Seventeen of the 145 subjects
were misclassified, an error rate of 11.7%. Perhaps more importantly, the general
configuration of all three groups was well captured; this is important for classifying
future subjects.
We compared model-based clustering to several other popular clustering meth-
ods. The results are shown in Figure 2.17. Single link clustering gave a very
poor result, clustering all the points except two into a single cluster. Average link
clustering, complete link clustering, and k-means all combined the normal subjects
with those suffering from chemical diabetes. They also split the overt diabetes
subjects into two clusters. Thus these other four methods failed to capture the
general nature of the clustering. In the case of the last three methods, this is due
to the fact that they are designed to find tight clusters, and as a result tend to
find clusters that are roughly spherical. Model-based clustering seeks instead to
find clusters with distributions that fit the data well.
Table 2.3 shows a quantitative comparison between the different clustering
methods. We compared model-based clustering with single link clustering, average
link clustering, complete link clustering and k-means clustering. We show two
measures of performance. The first is the classification error rate (CER), which is
the proportion of points that are misclassified. The minimum over all permutations
of cluster labels is taken. The smaller the CER, the better.
The second measure is the adjusted Rand index (ARI). The Rand index (Rand,
2.5 Examples with Known Number of Clusters 41

0 500 1000 1500

350
300
250
glucose

200
150
●
● ●●● ●●

100
●●●●
● ●
●●
●●
●●
●
●●
●●
●● ● ●
●●●●
● ●●
●●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●●●
● ●●●
● ● ●
●
●●
●
●●
●
●●
●●
●●●
●●
●●●●●
●●
●
●●
●
●●●
●
●●●
●● ●●●● ●●
● ●
● ●
●●●●
●
● ●● ●
●●
● ● ●
●
● ● ● ●
1500
1000

insulin
500

● ●
●●
●
●●
●●● ● ●●●●
●● ● ●
● ●●
●●
●● ●
●●●● ● ●●●● ●●
●● ●●●
●●●
●
●
● ●
●●
●●
●
●●●
●●
●● ● ●●
●
●●
●●
●
●
●
●
●●
●
●
●
●
●
● ●
●● ●
●
●
●●●●●●
●●●
●●●
●
●
●● ●● ●●
●●●
●●
● ●
●●● ●
● ●
●●●●● ● ●
0

1.0
0.5
sspg

0.0
●●
● ● ●
●●●
● ● ●●
●● ● ●●
●

−0.5
●●
●
●
●●●
●●
●
●●●
●
●●
●
●
●●
●●●
● ●●●
●● ● ●●●
●●
●● ● ●● ●● ●●
●
●● ●●
●●
●●
●●
●
●
●●
●
●
●●
●
●●
●
● ●
●
●●●●
●
●
● ●●
●
●● ● ● ● ● ●●
●
● ●●
●●● ● ●●
●●●●●●
● ●●
●●●● ●●
● ● ●

100 150 200 250 300 350 −1.0 −0.5 0.0 0.5 1.0 −1.0

Figure 2.15 Model-based clustering of the diabetes data with G = 3

groups using the unconstrained covariance (VVV) model. The colors indicate
the classiﬁcations of the points into the three clusters. The ellipses show
contours of the ﬁtted multivariate component densities corresponding to one
standard deviation along each principal component. The dashed black lines
show the principal component directions of each mixture component.

1971) of the similarity of two partitions of a data set is the proportion of pairs of
data points that are in the same group in both partitions. The adjusted Rand
index (ARI) (Hubert and Arabie, 1985) is the chance-adjusted value of the Rand
index. When the two partitions are statistically independent, its expected value is
zero, and when the two partitions are identical its value is 1. The higher the ARI,
the better.
The classiﬁcation error rate for model-based clustering for this data set was
42 Model-based Clustering: Basic Ideas

600

●
400
sspg

●
●
200

● ●
● ●
● ● ●
●
●● ●
● ● ●
● ● ●
●
● ● ●● ●
● ●●
● ● ●
0

0 500 1000 1500

insulin

Figure 2.16 Model-based classiﬁcation of diabetes data, with errors

marked. The errors are shown by black symbols, and the clinically
determined class is shown by the symbol (triangle, square or circle).

11.7%, and the ARI was 0.719. Single link clustering performed poorly for this
data set, putting all the data points except two into the same cluster. Average
link clustering, complete link clustering and k-means performed similarly, as one
might expect from Figure 2.17, with classification error rates between 27.6% and
29.7%, and ARI values between 0.318 and 0.375. These are all substantially worse
than model-based clustering, as they fail to capture the strongly non-spherical
nature of the clusters.
Model-based clustering also provides a quantification of the uncertainty about
the cluster allocation of each data point, Unceri , defined by Equation (2.12). These
2.5 Examples with Known Number of Clusters 43

600

600
●
400

400
sspg

sspg
●

● ●
●● ● ●● ●
● ● ● ●
●● ●●
● ●● ●●● ● ● ●● ●●● ● ●
●
200

200
●●● ●● ● ●●● ●● ● ●●
●● ●●
●
●●● ●
● ●
●●● ●
●
●●● ●●●
● ●● ●●● ● ●● ●●● ●
● ●● ●●● ● ● ●● ●●● ●●●
●● ● ●● ●● ● ●●
● ●●
●● ● ● ●● ● ●●
●● ●
● ●● ●
●●● ●● ●●● ● ●
● ●
●●● ● ●● ● ●
0

0
0 500 1000 1500 0 500 1000 1500

insulin insulin
(a) Clinical classification (b) Model−Based Clustering

●
600

600

●
●
●
●
400

400

●
sspg

sspg

● ●
●
●
● ●●
● ●
● ●●
●● ● ● ●
● ● ●
●● ● ● ●
● ●● ● ●
● ●●
● ●
200

200

●●● ●● ● ●●
●● ●
●
●●● ●
● ●
●●●
● ●● ●●● ● ●
● ●● ●●● ●●● ● ●
● ● ●●
●
● ●●
●● ● ●●● ● ●●● ●
●●● ● ●●
●
●● ● ●
●
0

0 500 1000 1500 0 500 1000 1500

insulin insulin
(c) Single link (d) Average link

● ●

● ●
600

600

● ●

● ●
● ●
● ●
● ●
400

400

● ●
sspg

sspg

● ●
● ●
● ●
● ●● ● ●●
● ● ● ●
● ●● ● ●●
●● ● ●● ●● ● ● ●
● ● ● ● ● ●
●● ● ● ● ●● ● ●
● ●● ●●● ● ● ● ● ●● ●●● ● ●
● ●
200

200

●●● ●● ● ●● ●●● ●● ● ●●
●● ● ●● ●
●
●●● ●
● ● ●
●●● ●
● ●
●●● ●●●
● ●● ●●● ● ● ● ● ●● ●●● ● ●
● ●● ●●● ● ●
● ● ● ●● ●●● ●●● ● ●
●● ● ●● ●● ● ●● ● ●● ●● ●
● ●●
●● ● ● ●● ● ● ●●
● ●●●● ●●● ●
●
●●● ● ●● ●●
● ●●
● ●
●●● ● ●●● ●
●
0

0 500 1000 1500 0 500 1000 1500

insulin insulin
(e) Complete link (f) k means

Figure 2.17 Clustering results for diabetes data, shown for two of the
three variables, insulin and SSPG. (a) Clinical classiﬁcation; (b) model-based
clustering; (c) single link; (d) average link; (e) complete link; (f) k-means.
44 Model-based Clustering: Basic Ideas

Table 2.3 Performance of diﬀerent clustering methods on the diabetes data: classiﬁcation error
rate (CER: smaller is better) and Adjusted Rand Index (ARI: larger is better).

Clustering Method CER % ARI

Model-based G = 3 11.7 .719
Single link 46.2 .013
Average link 29.7 .318
Complete link 29.7 .321
k-means 27.6 .375

are summarized in the uncertainty plot in Figure 2.18, which plots the uncertainty
values in increasing order, and shows the classification errors by vertical lines.
Most of the classification errors correspond to higher levels of uncertainty.
The uncertainty values are shown in a different way in Figure 2.19, where the
top 5% of points by uncertainty are shown by solid black circles, and the next
20% by solid gray circles. The points with the largest uncertainty are between
clusters; the points close to cluster centers have low uncertainty.
R commands to analyze the diabetes diagnosis data using model-based clustering
are shown in Listing 2.10. R commands to analyze the same data using the non-
model-based clustering methods are shown in Listing 2.11.

Example 3 (ctd.): Breast cancer diagnosis

We carried out model-based clustering of the breast cancer diagnosis data, which
has 569 subjects and 30 variables. We know the number of known classes to be
G = 2, and so we will restrict attention to models with two clusters. We will
also refrain, for now, from imposing any constraints on the component covariance
matrices, and so we will use the VVV model. This is done using the command:
Mclust (diagnostic.data, G=2, modelNames="VVV")
We compared the results with the clinical classification, which we will take here
as being correct. Of the 569 subjects, 541 were correctly classified, a 95.1% correct
classification rate, or, equivalently, a 4.9% classification error rate. To depict this,
we computed the first two principal components of the data, using the correlation
matrix (i.e. after first scaling the variables by their standard deviations). The
model-based clustering classification and the classification errors are shown in
Figure 2.20. The errors are all on the boundary between the two components in
the plot of the first two principal components.
Figure 2.21 shows a comparison of the model-based clustering with the clinical
classification and with other widely used clustering methods, again in terms of the
first two principal components. As for the diabetes diagnosis data set, single link
clustering performed very poorly, putting almost all the subjects into one cluster.
Average link clustering and complete link clustering gave results that were very
similar to each other, classifying too few subjects into the Malignant category.
2.5 Examples with Known Number of Clusters 45

0.5
0.4
0.3
uncertainty

0.2
0.1
0.0

observations in order of increasing uncertainty

Figure 2.18 Uncertainty plot for model-based clustering of the diabetes

data. The vertical lines indicate errors in the model-based classiﬁcation.

The k-means method also classified too few subjects into the Malignant category,
but more than average link and complete link.
Table 2.4 shows a quantitative comparison between the different clustering
methods in terms of classification error rate (CER) and adjusted Rand index
(ARI). Single link clustering had an ARI that was essentially zero, indicating
that its classification was not better than random chance, while average link and
complete link clustering had slightly better performances. The k-means method
performed better, but it had a classification error rate that was three times greater
than model-based clustering, which performed best for this data set.
The uncertainty plot is shown in Figure 2.22. It can be seen that the majority
46 Model-based Clustering: Basic Ideas

●
600

●
●

●
400

●
sspg

●
● ●
●
● ●

● ●●
● ●● ● ●
● ● ●

●●
●● ● ● ●
● ●● ●●
●●
● ●
200

●
● ●● ● ●
●●
●
●
●
● ●
●●●
● ● ●
●
● ●
● ●
● ●
● ● ●●
●●
●
●● ●
● ●
●
●

●● ● ●
●● ●
●●
● ●● ● ●

● ●
●
● ●
● ●
●●
●
●
● ●
●

● ●
●
● ●● ●
●
● ●
● ● ●
●● ● ● ●

●
● ● ●● ●
● ●
●
● ●
●
0

0 500 1000 1500

insulin
Model−Based Clustering Uncertainty

Figure 2.19 Uncertainty for model-based clustering of the diabetes data.

The solid black circles show the top 5% of data points by uncertainty, the
solid gray circles show the next 20%, and the least uncertain 75% of data
points are shown by gray dots.

of the errors corresponded to the subjects with the highest uncertainty, so in this
case the assessment of uncertainty captured the likelihood of error fairly well, at
least in a qualitative sense.

2.6 Choosing the Number of Clusters and the Clustering Model

Two basic issues arising in applied cluster analysis are selection of the clustering
method and determination of the number of clusters. In the mixture modeling
approach, these questions can be reduced to a single problem, that of model
2.6 Choosing the Number of Clusters and the Clustering Model 47

Listing 2.10: R code for model-based cluster analysis of the diabetes data
# Initial setup
library ( mclust )
data ( diabetes )
diabetes.data <- diabetes [2:4]
diabetes.class <- unlist ( diabetes [1])

# Model-based clustering fitting and plotting

diabetes.Mclust <- Mclust ( diabetes.data , modelNames = " VVV " ,G =3)
plot ( diabetes.Mclust , what = " classification " )

# Classification error measures

classError ( diabetes.Mclust $ classification , truth = diabetes.class )
adjustedRandIndex ( diabetes.Mclust $ classification , diabetes.class )

# Plot clinical classification

coordProj ( diabetes.data , dimens = c (2 ,3) , what = " classification " ,
classification = diabetes.class ,
col = c ( " red2 " ," dodgerblue2 " ," green3 " ) , symbols = c (0 ,16 ,17) ,
sub = " ( a ) Clinical classification ")

# Plot mclust classification

coordProj ( data = diabetes.data , dimens = c (2 ,3) , what = " classification "
,
classification = diabetes.Mclust $ classification ,
sub = " ( b ) M odel-Based Clustering " )

# Uncertainty plot
uncerPlot ( z = diabetes.Mclust $z , truth = diabetes.class )

# Plot mclust uncertainty

coordProj ( data = diabetes.data , dimens = c (2 ,3) , what = " uncertainty " ,
parameters = diabetes.Mclust $ parameters , z = diabetes.Mclust $z ,
truth = diabetes.class )

selection. Recognizing that each combination of a number of groups and a clustering

model corresponds to a different statistical model for the data reduces the problem
to comparison among the members of a set of possible models.
There are trade-offs between the choice of the number of clusters and that of
the clustering model. If a simpler model is used, more clusters may be needed
to provide a good representation of the data. If a more complex model is used,
fewer clusters may suffice. As a simple example, consider the situation where
there is a single Gaussian cluster whose covariance matrix corresponds to a long,
thin ellipsoid. If a model with equal-volume spherical components (the model
underlying Ward’s method and k-means) were used to fit these data, more than
one hyperspherical cluster would be needed to approximate the single elongated
ellipsoid.
One approach to the problem of model section in clustering is based on Bayesian
model selection, via Bayes factors and posterior model probabilities (Kass and
48 Model-based Clustering: Basic Ideas

Listing 2.11: R code for cluster analysis of diabetes data using non-model-based
methods
# Loading libraries and data
library ( MBCbook )
data ( diabetes )
diabetes.data = diabetes [ , -1 ]
diabetes.class = diabetes $ class

# Single link clustering

tmp <- hclust ( dist ( diabetes.data ) , method = " single " )
diabetes.single <- rect.hclust ( tmp , k =3)
n.diabetes <- length ( diabetes.data [ ,1])
diabetes.single.class <- rep (NA , n.diabetes )
for ( g in 1:3) d iabetes.single.class [ diabetes.single [[ g ]]] <- g
coordProj ( diabetes.data , dimens = c (2 ,3) , what = " classification " ,
col = c ( " d o d g erblue2 " ," green3 " ," red2 " ) , symbols = c (16 ,17 ,0) ,
classification = diabetes.single.cla s s , sub = " ( c ) Single link " )
classError ( diabetes.single.class , truth = diabetes.class )
adjustedRandIndex ( diabetes.single.class , diabetes.class )

# Average link clustering

tmp <- hclust ( dist ( diabetes.data ) , method = " average " )
diabetes.average <- rect.hclust ( tmp , k =3)
n.diabetes <- length ( diabetes.data [ ,1])
diabetes.average.class <- rep (NA , n.diabetes )
for ( g in 1:3) di abe te s . a v e r a g e . c l a s s [ diabetes.average [[ g ]]] <- g
coordProj ( diabetes.data , dimens = c (2 ,3) , what = " classification " ,
classification = diabetes.average.cl a s s , sub = " ( d ) Average link " )
coordProj ( diabetes.data , dimens = c (2 ,3) , what = " errors " ,
classification = diabetes.average.class , truth = diabetes.class )
classError ( diabetes.average.class , truth = diabetes.class )
adjustedRandIndex ( diabetes.average.class , diabetes.class )

# Complete link clustering

tmp <- hclust ( dist ( diabetes.data ) , method = " complete " )
diabetes.complete <- rect.hclust ( tmp , k =3)
n.diabetes <- length ( diabetes.data [ ,1])
d i a b e t e s . c o m p l e t e . c l a s s <- rep ( NA , n. diabetes )
for ( g in 1:3) di a be t es . c o m p l e t e . c l a s s [ d iabetes.complete [[ g ]]] <- g
coordProj ( diabetes.data , dimens = c (2 ,3) , what = " classification " ,
classification = diabetes.complete.class ,
col = c ( " d o d g erblue2 " ," green3 " ," red2 " ) , symbols = c (16 ,17 ,0) ,
sub = " ( e ) Complete link " )
classError ( diabetes.complete.class , truth = diabetes.class )
adjustedRandIndex ( diabetes.complete.class , diabetes.class )

# k-means clustering
diabetes.kmeans <- kmeans ( diabetes.data , centers =3 , nstart =20)
coordProj ( diabetes.data , dimens = c (2 ,3) , what = " classification " ,
classification = diabetes.kmeans $ cluster ,
col = c ( " green3 " ," red2 " ," dodgerblue2 " ) , symbols = c (17 ,0 ,16) ,
sub = " ( f ) k means " )
classError ( diabetes.kmeans $ cluster , truth = diabetes.class )
adjustedRandIndex ( diabetes.kmeans $ cluster , diabetes.class )
2.6 Choosing the Number of Clusters and the Clustering Model 49

10
5
Comp.2

0
−5

−5 0 5 10 15

Comp.1

Figure 2.20 Model-based classiﬁcation of breast cancer diagnosis data,

with errors marked. The ﬁrst two principal components of the data (with
variables scaled) are shown. Benign = red squares, Malignant = blue
triangles. The errors are shown by black symbols, and for them the clinically
determined class is shown by the symbol (square or triangle).

Raftery, 1995). The basic idea is that if several statistical models, M1 , . . . , MK ,

are considered, with prior probabilities p(Mk ), k = 1, . . . , K (often taken to be
equal), then by Bayes’ theorem the posterior probability of model Mk given data
D is proportional to the probability of the data given model Mk , times the model’s
prior probability, namely
p(Mk |D) ∝ p(D|Mk )p(Mk ). (2.14)
When there are unknown parameters, then, by the law of total probability, p(D|Mk )
50 Model-based Clustering: Basic Ideas

10 ● ●

10
● ●
● ●

● ●
● ●
5

5
● ● ● ● ● ●
● ●● ● ● ●● ●
●● ● ●● ●
Comp.2

Comp.2
●● ● ●● ●
● ● ● ● ● ●● ● ● ● ● ●●
●
●
● ●● ● ● ●● ● ● ●● ●●● ●● ● ●
● ● ● ●
● ● ● ● ● ● ●
●
● ●● ● ● ●● ●
● ● ● ●● ● ● ● ● ● ●● ●
●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●
● ● ●
●● ● ●● ●●
● ● ● ● ●● ● ●● ●● ● ● ●
● ● ● ●
● ● ●
●
0

−5
● ● ● ●
● ● ● ●
● ●
● ●
● ●
● ●

−5 0 5 10 15 −5 0 5 10 15

Comp.1 Comp.1
(a) Clinical classification (b) Model−based clustering
10

10
5

5
Comp.2

Comp.2

●
0

●
●
● ●
●
● ● ●
●
−5

−5

● ●
● ●
●
●
●
● ●

−5 0 5 10 15 −5 0 5 10 15

Comp.1 Comp.1
(c) Single link clustering (d) Average link clustering
10

10
5

●
Comp.2

Comp.2

●
●
● ● ●
● ● ●
●
●
● ●
● ● ●● ● ●
● ●
● ● ●
● ● ●
●
0

● ●
● ● ● ●●
● ●● ● ●
● ●●●● ● ●
● ● ●
● ●●
●● ●●
● ●● ● ● ●●
●
●●● ● ●●●●● ● ● ● ●● ●
● ●●●● ● ● ● ●
● ● ● ●
● ●
● ●●● ●●
● ● ●
● ● ●● ● ●● ● ●
● ● ●● ● ●
● ● ● ● ●● ● ● ● ●
● ●
● ● ●
−5

−5

● ● ● ●
● ● ● ●
● ●
● ●
● ●
● ●

−5 0 5 10 15 −5 0 5 10 15

Comp.1 Comp.1
(e) Complete link clustering (f) k means clustering

Figure 2.21 Clustering results for breast cancer diagnosis data, shown for
the ﬁrst two principal components of the data. (a) Clinical classiﬁcation
(Benign = red squares, Malignant = blue circles); (b) model-based
clustering; (c) single link; (d) average link; (e) complete link; (f) k-means.
2.6 Choosing the Number of Clusters and the Clustering Model 51

Table 2.4 Performance of diﬀerent clustering methods on the breast cancer diagnosis data:
classiﬁcation error rate (CER: smaller is better) and Adjusted Rand Index (ARI: larger is
better). All methods take the number of clusters (G = 2) as known.

Clustering Method CER % ARI

Model-based 4.9 .811
Single link 37.1 .002
Average link 33.7 .052
Complete link 33.7 .052
k-means 14.6 .491

is obtained by integrating (not maximizing) over the parameters, i.e.

p(D|Mk ) = p(D|θMk , Mk )p(θMk |Mk )dθMk , (2.15)

where p(θMk |Mk ) is the prior distribution of θMk , the parameter vector for model
Mk . The quantity p(D|Mk ) is known as the integrated likelihood, or marginal
likelihood, of model Mk .
A natural Bayesian approach to model selection is then to choose the model that
is most likely a posteriori. If the prior model probabilities, p(Mk ), are the same,
this amounts to choosing the model with the highest integrated likelihood. For
comparing two models, M1 and M2 , the Bayes factor is defined as the ratio of the
two integrated likelihoods, B12 = p(D|M1 )/p(D|M2 ), with the comparison favoring
M1 if B12 > 1, and conventionally being viewed as providing very strong evidence
for M1 if B12 > 100 (Jeffreys, 1961). Often, values of 2 log(B12 ) rather than B12
are reported, and on this scale, rounding, very strong evidence corresponds to a
threshold of 10 (Kass and Raftery, 1995).
This approach is appropriate in the present context because it applies when
there are more than two models, and can be used for comparing non-nested
models. In addition to being a Bayesian solution to the problem, it has some
desirable frequentist properties. For example, if one has just two models and they
are nested, then basing model choice on the Bayes factor minimizes the total error
rate, which is the sum of the Type I and Type II error rates (Jeffreys, 1961).
The main difficulty in the use of Bayes factors is the evaluation of the integral
that defines the integrated likelihood. For regular models, the integrated likelihood
can be approximated simply by the Bayesian Information Criterion or BIC:
2 log p(D|Mk ) ≈ 2 log p(D|θ̂Mk , Mk ) − νMk log(n) = BICMk , (2.16)
where νMk is the number of independent parameters to be estimated in model
Mk (Schwarz, 1978; Haughton, 1988). This approximation is particularly good
when a unit information prior is used for the parameters, that is, a multivariate
normal prior that contains the amount of information provided on average by one
observation (Kass and Wasserman, 1995; Raftery, 1995).
52 Model-based Clustering: Basic Ideas

0.4
0.3
uncertainty

0.2
0.1
0.0

observations in order of increasing uncertainty

Figure 2.22 Uncertainty plot for model-based clustering of the breast

cancer diagnosis data. The vertical lines indicate errors in the model-based
classiﬁcation.

The reasonableness of the unit information prior is discussed by Raftery (1999).

For example, for a mean parameter it is a prior that spreads most of its weight over
the range of the data but not beyond, and thus can be viewed as corresponding
to the prior information of someone who knows something, but relatively little,
about the problem at hand. Such a person would be likely to know the range of
the data to be expected, at least roughly, and would likely be comfortable saying
that the mean is likely to lie within this range, but would probably not have
enough information to reﬁne this range greatly. In this sense, it is as spread out
as is reasonable: an analyst is unlikely to want to use a more spread out prior,
2.6 Choosing the Number of Clusters and the Clustering Model 53

but may well have information that would allow them to refine the prior. It thus
seems a reasonable candidate to be used as a default prior.
Note that an informed analyst may well have a more precise prior or external
information, leading to a more concentrated prior distribution. More concentrated
prior distributions often tend to lead to Bayes factors that favor larger models
(and the alternative hypothesis in two-hypothesis testing situations). Thus the
unit information prior can be viewed as conservative, in the sense that informative
priors are likely to favor the alternative hypothesis (or larger models) more. If
external information tends to provide evidence against the null hypothesis, then its
use is legitimate, but the fact that the conclusion is based on external information
as well as the data at hand should be revealed to readers and users.
Finite mixture models do not satisfy the regularity conditions that underlie the
published proofs of (2.16), but several results suggest its appropriateness and good
performance in the model-based clustering context. Leroux (1992) showed that
basing model selection on a comparison of BIC values will not underestimate the
number of groups asymptotically. Keribin (1998) showed that BIC is consistent for
the number of groups, subject to the assumption that the likelihood is bounded.
The likelihood is not in fact bounded in general for Gaussian mixture models
because of the degenerate paths of the likelihood to infinity as the variance matrix
becomes singular. However, a very mild restriction, such as a very small lower
bound on the smallest eigenvalue of the covariance matrices, is enough to ensure
validity of the assumption. The main software used in model-based clustering
incorporates such restrictions, so the assumption is valid in practice for all the
models used.
Roeder and Wasserman (1997) showed that if a mixture of (univariate) normal
distributions is used for one-dimensional nonparametric density estimation, then
using BIC to choose the number of components yields a consistent estimator of
the density. Finally, in a range of applications of model-based clustering, model
choice based on BIC has given good results (Campbell et al., 1997, 1999; Dasgupta
and Raftery, 1998; Fraley and Raftery, 1998; Stanford and Raftery, 2000).
For multivariate normal mixture models, the models to be compared correspond
to different numbers of clusters and different sets of constraints on the covariance
matrices, or clustering models. Each combination of a number of clusters and
a clustering model defines a distinct model to be compared. Typically, models
with G = 1, . . . , Gmax are considered, for some reasonable choice of the maximum
number of clusters, Gmax . In the mclust R package, the default value of Gmax is
9, but in specific applications the appropriate number might be much greater. For
example, in the craniometric data of Example 5, there are known to be at least
28 populations represented, so in examples of this kind it would be wise to take
Gmax to be 30 or greater. By default, the mclust software considers 14 covariance
models, leading to a default number of Gmax × 14 = 126 models considered. The
Rmixmod software considers 28 covariance models, leading to a larger total number
of models considered.
For a finite mixture model, the usual likelihood function is also called the
mixture likelihood, or, particularly in the context of the EM algorithm, the
54 Model-based Clustering: Basic Ideas

observed data likelihood. From (2.1), it has the form

⎛ ⎞

n
G Mk

p(y|θMk , Mk ) = ⎝ τg fg (yi |θg )⎠ , (2.17)

i=1 g=1

where y = (y1 , . . . , yn ), GMk is the number of mixture components in model Mk ,

and θMk consists of the parameters for model Mk . For a multivariate normal
mixture model, the likelihood takes the form (2.17), with fg (yi !θg ) = φg (yi |μg , Σg ),
where φg (·|·) is the multivariate normal density deﬁned by (2.2). The integrated
likelihood is then equal to

p(y|Mk ) = p(y|θMk , Mk )p(θMk |Mk )dθMk . (2.18)

If there are no cross-component constraints on the covariance matrices (as for

example in models VII or VVV), then θMk = {(τg , θg ) : g = 1, . . . , GMk }, where θg is
the parameter vector for the gth mixture component. In model VVV, θg = (μg , Σg ).
For a model with cross-cluster constraints on the covariance matrix, the parameter
set θMk will reflect the constraints. For example, if Mk is a VEV model, then
θMk = {(τg , λg , Dg ) : g = 1, . . . , GMk ; A}, where λg , Dg and A are defined as in
(2.3).
The BIC for model Mk is then defined by (2.16), where the data D consist
of the observations to be clustered y, θ̂Mk is the MLE of the model parameters,
typically found by the EM algorithm, and νMk is the number of free parameters
in Mk , found for example from Table 2.2.
The BIC is designed to choose the number of components in a mixture model
rather than the number of clusters in the data set per se. The difference is subtle
but important. Model-based clustering was initially grounded in the hope that the
number of mixture components is the same as the number of clusters, but this may
not always be true. For example, one cluster may be strongly non-Gaussian, and
may itself be best represented by a mixture of normal distributions. In this case,
the number of mixture components would be greater than the number of clusters.
In Section 3.3 we will describe how this problem can be addressed by merging
mixture components, but for now we focus on a different solution that uses a
different model selection criterion, designed for clustering rather than selecting
the model that best fits the data.
When interest is primarily focused on clustering rather than finding the best
mixture model to fit the data, one solution to this problem is to use, instead of
BIC, the Integrated Completed Likelihood (ICL) (Biernacki et al., 2000). This is
based on the joint likelihood of both the data y and the clustering z, defined by

n G Mk
zi,g
p(y, z|θMk , Mk ) = {τg fg (yi |θg )} . (2.19)
i=1 g=1
2.6 Choosing the Number of Clusters and the Clustering Model 55

Integrating over the model parameter θMk then gives

p(y, z|Mk ) = p(y, z|θMk , Mk )p(θMk |Mk )dθMk . (2.20)

We use a BIC-like approximation to the integral and take twice the logarithm of
it, yielding
2 log p(y, z|Mk ) ≈ 2 log p(y, z|θ̂Mk , Mk ) − νMk log(n). (2.21)

The Integrated Completed Likelihood of model Mk is then deﬁned as the right-

hand side of (2.21) with z replaced by its maximum a posteriori (MAP) estimate.
The MAP estimate of z, denoted by z ∗ , satisﬁes zi,g
∗
= 1 if ẑi,g = arg maxh ẑi,h
and 0 otherwise, where
τˆg fg (yi |θ̂g )
ẑi,g = GM . (2.22)
h=1 τˆh fh (yi |θ̂h )
k

Thus
ICL = 2 log p(y, z ∗ |θ̂Mk , Mk ) − νMk log(n). (2.23)
It turns out that
ICL = BIC − E(Mk ), (2.24)
where

n G Mk

E(Mk ) = − ẑi,g log(ẑi,g ) (2.25)

i=1 g=1

is the expected entropy of the classification from model Mk (Biernacki et al., 2000).
Thus ICL is equal to the BIC penalized by the expected entropy of the classification.
Entropy is high when there is large uncertainty about the classification, and it
is highest when all the ẑi,g are equal (i.e. all equal to 1/GMk ), at which point it
attains the value n log(GMk ). It is lowest when all the ẑi,g are equal to either 0 or
1, at which point it is equal to zero. As a result, ICL tends to favor models that
produce more clearly separated clusters more than BIC does, and so in practice
ICL usually chooses the same or smaller numbers of clusters than BIC.

Example 1 (ctd.): Old Faithful data

Figure 2.23 shows the BIC and ICL plots for the Old Faithful data, using the
R commands in Listing 2.12. Values are shown for up to Gmax = 9 mixture
components and for the 14 covariance models estimated in mclust, i.e. for 9×14 =
126 diﬀerent competing models in all. A zoomed version of Figure 2.23 is shown
in Figure 2.24.
BIC selects the model with three mixture components and the EEE covariance
speciﬁcation, in which all the covariance matrices are the same, so that Σg ≡ Σ in
(2.2). ICL selects the model with two mixture components and the equal-orientation
56 Model-based Clustering: Basic Ideas

Listing 2.12: BIC and ICL plots

# Loading the mclust library
library ( mclust )

# Clustering with Mclust and displaying BIC selection

faithful.Mclust <- Mclust ( faithful )
plot ( faithful.Mclust , what = " BIC " )

# Clustering with Mclust and displaying ICL selection

faithful.mclustICL <- mclustICL ( faithful )
plot ( fai th fu l. mc lu st IC L )

●
● ●
● ● ●
● ● ● ●
● ● ● ● ●
● ● ● ● ●
● ●
● ● ●
● ● ●
● ●
● ●
●

−2500
●
−2500

● ● ● ●
● ● ●
● ● ●
●

−3000
−3000

● ●
BIC

ICL

−3500
−3500

EII EVE EII EVE

VII VEE VII VEE
● EEI VVE ● EEI VVE
● VEI EEV ● VEI EEV
● EVI VEV ● EVI VEV
● VVI EVV ● VVI EVV
−4000

−4000

EEE VVV EEE VVV

1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

Number of components Number of components

Figure 2.23 Model selection for the Old Faithful data: Left: BIC plot.
Right: ICL plot.

VVE covariance model. The model selected by BIC uses 11 free parameters to
model the covariances, while the model selected by ICL has one fewer, at 10.
Figure 2.25 shows the classiﬁcations and the component density ellipses for
the two selected models. Figure 2.26 shows the density contours for the overall
estimated mixture density from the two selected models. It seems that the model
selected by BIC may give a better estimate of the overall density of the data, but
ICL yields a clustering that agrees better with the visual impression.
One possible conclusion from this analysis is that there are likely two clusters
in the data, but that the cluster at the top right in the right panel of Figure 2.25
may well be non-Gaussian and could be better represented as a mixture of two
Gaussian components than as by a single multivariate normal distribution. This
suggests using BIC to select the number of mixture components, but then merging
the two top right components into a single non-Gaussian cluster. We will discuss
how to do this in Chapter 3.3. For the Old Faithful data, it can be seen from
2.6 Choosing the Number of Clusters and the Clustering Model 57

−2320

−2350
●
●
● ●
●
●
●
−2330

●
●
● ● ●
●

● ●
BIC

ICL
●

−2400
−2340

●
●

●
●
−2350

● EEI VEE ● EEI VEE

●
● VEI VVE ● VEI VVE
●

−2450
● EVI EEV ● EVI EEV
● ● ●
VVI VEV VVI VEV
EEE EVV EEE EVV
−2360

EVE VVV EVE VVV

2 3 4 2 3 4

Number of components Number of components

Figure 2.24 Zoomed version of Figure 2.23: BIC and ICL plots for Old
Faithful data.

BIC−best classification ICL−best classification

● ●
● ●
● ● ● ●
● ●
● ●
90

● ●● ●●● ● ●● ●●●
● ● ● ● ●
● ● ● ● ●● ● ● ● ● ●●
● ● ●
●● ● ● ● ●● ● ●
● ●●● ● ●● ●●●
●
● ●●● ●● ● ●● ●●● ●● ●
●
●
● ● ●●● ● ● ● ● ● ● ● ●●● ● ●
●●●● ●● ●
● ● ● ● ● ●●●● ●● ●
● ● ●
● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ●● ●
80

● ● ●● ● ● ● ● ● ●● ●
● ●● ● ● ●● ● ● ● ● ●● ● ● ●●
● ●● ● ● ●● ●● ●● ● ● ● ● ●● ●
● ● ●● ●● ●●
●● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ●
●● ● ● ● ● ● ●● ●● ● ● ● ● ●
●● ● ●●
● ● ● ●
● ● ●
●●
● ● ●● ● ● ● ● ●●
● ● ●● ● ● ● ● ●● ●
● ●
waiting

waiting

● ● ● ●
70

●● ● ● ●●
● ● ●
●
● ●
● ●
60

60
50

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

Figure 2.25 Classiﬁcation plots for the Old Faithful data using selected
models. Left: model selected by BIC with G = 3, model EEE. Right: model
selected by ICL with G = 2, model VVV.

Figure 2.25 that this approach gives the same classiﬁcation as the two-cluster
Gaussian model selected by ICL.

Example 2 (ctd): Diabetes diagnosis

Figure 2.27 shows the BIC and ICL plots for the diabetes diagnosis data. These
two criteria both select the model with three components and the unconstrained
(VVV) covariance model, in both cases fairly decisively. The resulting classiﬁcation
is shown in the left panel of Figure 2.28.
58 Model-based Clustering: Basic Ideas

BIC−best density ICL−best density

90 0.005

90
0.02 0.015

0.03 0.025

0.04 0.035
4
0.045 0.0
80

80
0.035
0.03
0.025
0.02
0.015
waiting

waiting
0.01
70

70
0.01
0.005

0.005 0.005

0.01
0.015
60

60
0.02
0.025
3
0.0

5
35

0.03
0.0

3
0.0
50

50
0.02
5
0.02
0.015
0.01

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

Figure 2.26 Density contour plots for the Old Faithful data using selected
models. Left: Model selected by BIC with G = 3, model EEE. Right: Model
selected by ICL with G = 2, model VVV. −4800
−4800

● ● ● ●
● ● ● ● ●
● ● ● ●
●
● ●
● ● ● ● ● ● ●
● ● ●
● ● ●
● ● ●
● ●
● ●
−5000
−5000

● ● ●
● ● ●
● ●
−5200

●
−5200

●
BIC

ICL

−5400
−5400

● ●
−5600
−5600

EII EVE EII EVE

VII VEE VII VEE
● EEI VVE ● EEI VVE
● VEI EEV ● VEI EEV
● EVI VEV ● EVI VEV
−5800
−5800

● VVI EVV ● VVI EVV

EEE VVV EEE VVV

1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

Number of components Number of components

Figure 2.27 Model selection for the diabetes diagnosis data. Left: BIC.
Right: ICL.

This is a satisfactory choice of number of clusters and of clustering model. The

number of clusters selected corresponds to the clinical classification. Also, in the
clinical classification, the volumes of the clusters differ: the cluster of non-diabetic
or “normal” subjects is much more compact and has a much smaller volume than
the other two clusters, and so the volume should differ between clusters. The
shapes are also very different, with the normal cluster being compact and close
to spherical, while the other two clusters are elongated. Finally, the orientations
are different, in that the overt diabetes and chemical diabetes clusters point in
2.6 Choosing the Number of Clusters and the Clustering Model 59

600

600
400

400
sspg

sspg
●
●● ●
● ●
●● ●
● ●● ●●● ● ● ● ●●
●
200

200
●●● ●● ● ●● ●●● ●● ●
●● ●● ●
●
●●● ●
● ●
●●● ●●
●●● ●● ●
● ●● ●●● ● ● ●● ●●● ● ●
● ●● ●●● ●●● ● ●● ●●● ●●● ● ●
●● ● ●● ● ●● ● ●● ●● ●
●●
● ●●●● ●●
● ●● ●●
● ●●●● ●
●●●
●● ● ●
●● ●●
● ●
●●● ● ●●● ●
●
0

0
0 500 1000 1500 0 500 1000 1500

insulin insulin

Figure 2.28 Classiﬁcation plots for selected models for the diabetes
diagnosis data. Left: three-component model with unconstrained covariance
selected by BIC and ICL. Right: ten-component equal volume spherical
model (EII).

very different directions. Thus, the cluster volume, shape and orientation all vary
between clusters, so the VVV model is the most appropriate one, and it is the one
selected by our two criteria.
One interesting aspect of model choice in model-based clustering is that there
is a trade-off between complexity of clustering model and number of clusters. For
example, a single elongated cluster could also be represented approximately by a
larger number of spherical clusters. If we think of a pea plant, the single elongated
cluster could be like a pod, and the approximating spherical clusters could be like
the peas, so the problem could be viewed as one of distinguishing between the
peas and the pod.
The best constant variance spherical model for these data has ten mixture
components, with the EII model. The classification from this is shown in the right
panel of Figure 2.28. The “peas versus pod” dynamic is apparent. For example, the
long thin cluster in the left panel of Figure 2.28 is replaced in the ten-component
spherical clustering by three small components in the right panel. In this case,
BIC and ICL both choose the “pod” model over the “peas” model, selecting the
three-component model.
For some purposes, such as grouping data into compact groups for data com-
pression purposes, a model such as EII with a large number of clusters might be
better suited than a selected more complex covariance model. In such a case, the
spherical model may well be preferred, even if its fit is not as good according to
model selection criteria. It is always important to bear the purpose of the data
analysis in mind when making modeling decisions.
60 Model-based Clustering: Basic Ideas

2.7 Illustrative Analyses

We now carry out more complete analyses of two substantial data sets.

2.7.1 Wine Varieties

We will analyze a data set giving 27 physical and chemical measurements on
178 wine samples (Forina et al., 1986). These data are available in the pgmm R
package. There are three types of wine, Barolo, Grignolino and Barbera, and the
type of wine is known for each of the samples. The year in which each sample was
produced is also known. For now, we will use only the 27 measurements to cluster
the samples, and assess to what extent they reproduced the known types.
Figure 2.29 shows a pairs plot for five of the 27 measurements. These were
chosen because they were the variables with the highest loadings on each of the
first five principal components, suggesting the possibility that they might be
variables carrying information about clustering. In fact, only the plot of the first
two variables (Flavanoids and Color Intensity) gives a strong visual suggestion of
clustering.
Figure 2.30 shows the same pairs plot, but with the wine types shown in different
colors. Note that the clustering method does not have access to this information.
It shows that, while there is no clear separation between the wine types, the
samples do tend to cluster, particularly again in the first two variables shown.
This suggests that clustering could be successful.
Figure 2.31 shows the BIC and ICL plots for these data, with a zoomed version
showing the most likely models and numbers of clusters in Figure 2.32. It can be
seen from Figure 2.32 that there are two competing and quite different descriptions
of the data between which the evidence is inconclusive. One description consists
of the spherical models with scales in each dimension that vary between clusters,
EVI and VVI, and three clusters. The other description consists of the spherical
model with volumes that vary between clusters but scales that do not, VEI, and
sevan clusters.
The top three covariance-model/number-of-cluster combinations are the same
for the two criteria, but in different orders, as shown in Table 2.5. We will focus
initially on the ICL solution, which gives the same number of clusters as wine
types, and so gives what at first sight appears to be the “correct” number of
clusters. However, we will see later that the BIC seven-cluster solution recovers
some information about year as well as type that the three-cluster solution cannot
do, and so which solution is better can be debated.

Figure 2.33 (left) shows the correct classification and the confusion matrix
is shown in Table 2.6. It can be seen that the red group (Grignolino) is for
the most part fairly compact, but also has about nine samples that are quite
spread out and are separated from the core Grignolino samples. The right panel
of Figure 2.33 shows the same information, but with classification errors from
model-based clustering shown by filled black symbols. It is clear that all but one
2.7 Illustrative Analyses 61

2 4 6 8 12 3.0 3.2 3.4 3.6

● ● ● ●

5
4
● ● ● ●
● ● ●●● ●● ●● ●●● ● ●● ● ● ●● ● ●●
● ● ● ● ● ●●● ● ● ●●●
● ●
●●
● ●● ● ● ●●●● ●●●● ●● ● ● ●
● ● ●●●● ●●
●●● ●●
● ●●●
● ●
●● ●● ●●●●●● ●●●●
●●
● ●
●●
● ●●●
●● ●●●●● ● ● ● ● ●●
●●●●● ●●●
●● ●●●
●
●●●●● ● ● ● ● ●● ●● ●●●●●●●● ● ●
●

3
● ●
Flavanoids ●●● ● ●●●●● ● ● ●● ●
●●● ●●● ● ● ● ●●●● ●
●
●● ●
● ● ● ● ●● ● ●●●●● ●
●
●
●
●●
●
● ●
● ●
●●
● ●
●● ●●
● ●
● ●●● ● ●●●●● ● ●●
● ●● ● ●
●●●
● ●●●● ● ●● ●●
● ●●
● ●
●● ● ●●●●
●●● ●
●●●
● ●
●●●
●● ●● ●●● ●●
●
●● ● ●●● ●
●●●●● ● ●●● ●● ●
●
● ●●●
●
●●
●
●● ● ●●●●● ● ● ● ●● ● ● ●●●●● ●
●● ●
●●●
●
●●●●● ●●●
●●
●●

1
● ●● ●●
●●●●● ●●●●●● ● ●● ● ● ●●●●●●●● ●
●●●● ●●●●●●● ●
●● ● ● ● ●
● ●●●●●
● ●●
●
●
● ● ●●●● ●● ● ● ●●●●
●●●●
●
● ●
● ●●●
●●●
●●● ●●
●●●
●●●● ● ●●
●● ●
●●
●●
●
●
●
● ●●
●
●●● ●
● ●
●●
●
● ● ●

● ● ● ●
12

● ● ● ●
●● ● ●●● ●● ● ● ● ● ●
●● ●● ● ●
● ●● ●●● ●
●
● ● ●● ● ●●● ●● ● ●●
●
● ● ● ●●
● ●● ●● ● ● ●●● ● ● ● ● ● ●● ●
● ●● ●
● ● ● ● ● ● ●
●●●●●● ●●● ● ●● ●● ●
2 4 6 8

●
●●●● ● ● ●● ●●● ● ● ●
● ● ●●●●●
●●● Color_Intensity ●●
● ●●●● ●● ●● ● ● ● ●●● ● ● ● ●●●● ●●
●●●●
● ●● ●●
● ●● ●● ● ● ● ● ● ● ● ● ●
●● ●●● ● ●● ● ●● ●●● ●●● ●●● ● ●●
●●
●● ●● ●● ●●● ●●●● ●●
●●●●● ●● ● ●● ● ● ● ● ●●●●●● ● ●
●● ●
●●●● ● ● ● ● ●●
● ●●
●
●●
●
●●
● ● ●●●●●●
●●
●● ●●
●● ●●●●● ●●
● ●●● ●
● ●● ●●●
●●●●●
●● ●
●
●● ● ●●●● ●●●●
●●●
●
●●
●●
●
● ●● ●●
● ● ● ●●●●● ●● ● ● ●● ●●● ●● ● ●
●●
●● ● ●
●● ●●●●● ●
●●●
●●● ● ● ●●●●
●●●●● ● ●
● ●●●
●●● ●●●●●
●● ●●●● ● ●●●
●●● ●
● ●
● ● ●● ● ● ●● ● ●●●
●●●● ●● ●● ● ● ● ●●● ● ●
● ●● ●● ●
●
●● ● ●●
●● ●●●● ●●
● ● ●●● ●● ● ● ● ●●●● ● ● ●●
●● ● ●●●
● ●●●●●●● ●●
● ●●●●
●● ●
●●●●● ●●●●●
●● ●●●
●● ●● ● ● ●● ● ●● ●
● ●●● ● ● ●
●●●●●●●
●●●
●●●
●●
●●●● ● ●
●● ●
●●● ● ●●
●
●●● ●
●●
●● ● ● ●● ●●
● ● ●
●● ●●
● ●●
● ●● ● ● ●●
●
●●●●● ●
● ●●
●● ● ●
●● ● ● ●
● ●
●
● ●
●● ● ●●●●●● ● ●
●● ●● ●● ● ●
● ●●
● ● ● ●

30
● ● ● ●
● ● ● ● ● ● ●
● ● ● ●
● ● ● ●●
● ●
●
● ●● ● ● ● ●

25
●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●
●●● ● ● ● ●●● ●●● ● ● ●
●● ●● ● ●● ●
● ● ● ● ● ●● ●
●
●●●●● ● ●●● ●● ● ●● ●● ● ● ● ● ● ●
● ● ● ● ● ●●
●
●
●●● ●●
●●●● ●●
● ●● ● ●
●●●● ● ●
●● ●● ● ●
● ● ●
●● ●●
●●●
● ●●
●● ● ● ●●●
●
●
●●
●●● ● ●
●● ● ●● ● ●● ●● ●● ●●
●●
● ●●
● ●●●●
● ● ●●●●●●●●
●
●●
● ● ● ●●
●
●●
●● ●
●●●
● ●● ●●●
●●● ●
● ●● ● ● ● ● AlNalinity_of_ash ●● ● ●● ●●●●●

20
●●
●●●
● ●
●● ●● ● ●●●● ● ●●●●
●●●●
● ● ● ●●
●●●●●● ●
●●●
●●
●●●● ● ●●●●●
● ● ●
● ●●
●●●● ●● ●
●● ●
●●
● ●● ● ●●
●● ●●
● ●●
● ● ●● ●
●●●●● ● ●●●●●●
●
● ●● ●● ●●●● ● ●●●● ●
● ●
● ●● ●● ●
●●●●● ●● ●
●
●● ● ●
●●
●● ●
● ●
●● ●● ●●●
● ● ●●
●● ●● ●●●●●
●● ●
●●●
●●●●
● ● ●● ● ● ● ● ●●
●●●
●●
● ●●●● ●
●●
● ●
●● ●
●●
●● ● ●●● ● ●
●●
●
●● ●●●
●●● ●●●●●●● ●
●
●●● ● ●●●● ●●● ●●●●●●●● ● ●●●●●●
●●●
●● ● ● ●●● ●
●●●● ●● ● ●●● ● ● ●●
● ●●● ● ●● ●●●● ●●●●
●● ● ● ● ●
●
● ●●● ●●●●
●● ●
●●●●●●● ●● ● ●● ●●●
●
●● ●●● ●

15
● ● ●●●● ● ● ●
●
● ● ●●● ● ● ●
● ● ● ● ●● ● ●
● ●
● ● ● ●
●● ●● ● ● ●
● ● ● ● ● ● ● ●
● ● ● ● ●

10
● ● ● ● ● ● ●
● ● ● ● ● ●● ●
3.6

●
● ● ● ●● ● ● ●● ●
● ● ●● ●●● ● ● ● ● ●●
●
● ● ●● ● ● ●●● ● ● ●●● ●●● ● ●●● ●●● ●● ● ● ● ●● ●
● ●● ● ●
●
●●●
● ● ●● ●● ● ● ● ●
●●● ●●● ● ●● ●● ●
● ●
● ●● ●●● ● ● ●● ●
●● ● ● ● ●●●● ●●
● ●●● ●● ● ● ●●●● ●●●
●
3.4

●●●● ● ● ● ● ● ●● ●●● ● ●
● ● ● ●
● ● ● ●

160
● ● ● ●
● ● ● ●

● ● ● ● ●
● ● ●● ●
●
● ● ●● ●
●● ● ● ●● ●
●● ●● ● ● ●

120
● ●● ●● ● ● ●● ● ●
●●● ●
●● ●●●●● ●
●● ●●●●
●●● ● ●●● ●
●
●●
●● ●● ●●●● ● ● ●● ●●
● ●
● ● ● Magnesium
●●
●● ● ●● ●● ● ● ●●● ● ●●●● ● ● ● ●● ●●●●● ●●● ● ●
● ● ● ● ●●●●● ●
● ●
●
●●●●● ● ● ● ●●●●●● ● ●● ●● ●●● ●● ●●● ●●● ●
●●●
● ●● ●● ● ● ●● ●●●●●●●●●● ●
●
●● ●● ● ●● ●● ●●
● ● ●
● ● ●● ●
● ●
●●●● ● ● ●●
●● ●
● ●● ●●●● ●●
●● ●
●
● ●●● ●●● ● ●●●●●●●
●●● ● ● ●
● ●
●●●● ● ● ● ●●●●●●
●●● ● ●
●●●●●●
●●
●● ● ●
● ●
● ●● ● ● ●●● ●● ●●●●● ●●●●● ●
●
● ●●●●
●●
●●
● ●●●●
● ●●
●●
●●
●●
●
● ●●●●●●
●●
●● ●●● ● ●●●●●
● ●
●●
●●
●●
● ● ●●●● ●● ●●●●
●
●
●●●●
● ●●●●●●
● ● ●●●● ●
●
●●● ●
●●● ●
●●●●●●●●● ●●●●● ●●●
● ●● ●
● ●● ● ● ●●
● ● ● ●●● ● ●● ● ● ● ●● ● ●●● ●
●●●●●
●●
●●
●
●●
● ●
●
●●●
●
●●●
●●
● ● ●
●
●● ● ● ●
●
●●
●●
●
●●
●
●●●
● ●●
● ●
●● ●● ●
● ● ● ● ●●● ●●
● ●●●●
●
● ● ● ●●
● ●●●●
● ●●● ●
●●●
●●
● ●●● ●
●● ●
●●● ●● ●● ● ●●

80
● ●●●
●●●● ● ●●● ● ● ●● ●● ●●●●● ● ● ● ●●● ●
● ● ● ●

1 2 3 4 5 10 15 20 25 30 80 120 160

Figure 2.29 Pairs plot for ﬁve of the 27 physical and chemical
measurements of wine samples

of the ten misclassiﬁed samples are among these Grignolino samples whose values
are separated from the main Grignolino group, and so are classiﬁed as being of
one of the other types. It would be hard for any clustering method to classify
these samples correctly.

Table 2.7 shows the performance of diﬀerent clustering methods for the wine
data, when all are given the “correct” number of clusters, G = 3. The model-based
method performs best by a wide margin, while the single link method again
performs very poorly.
62 Model-based Clustering: Basic Ideas

2 4 6 8 12 3.0 3.2 3.4 3.6

5
4
● ● ● ●
●
● ● ●●● ● ●● ●● ●● ● ● ●●
● ● ● ● ●● ●
●
●●
●●
●
● ●●
●
●● ●●●● ● ● ●● ●
●
● ●●●●
●●
●●
●● ●
●●●● ●● ● ● ● ●
● ●●
●●●● ●● ● ● ●
●●●●
●●● ●● ●● ●●●●●
●
● ●●● ● ● ● ●●●
● ●● ●●●●●●●● ● ●
● ●

3
● ● ●
Flavanoids ●●
●●
●
● ●●● ●●
● ● ● ● ●●
● ●
●● ●●● ● ●
● ● ●●● ●
● ●●
● ●
● ●
●●●●●●●●●●● ● ●
●
●●● ●
●
● ● ●● ●●●●● ●
● ●
●
● ●● ●● ● ●●● ● ●
●● ●●● ● ●●

2
1
12

●● ●● ● ● ●
●
2 4 6 8

● ● ● ●
●●●●
● ● Color_Intensity ●●
● ●●●● ● ● ● ●
●●● ● ● ●●●●● ●
●● ●● ●● ●● ●● ● ●● ●●●
●●
● ● ●● ●● ● ●● ●
● ●
●
●●
●
●
●
●●
●
● ●●
●●●●●● ● ●●
● ●
●
●
● ●●● ●● ●●●●
●● ●● ●
●
●
●●●
●● ●● ●● ● ●●●●●● ●● ●●● ● ● ●● ● ●●●● ●● ● ●
●●
● ● ●● ●●
●
●
● ●
●
●
●●●●
●
● ● ●●●●●● ●● ● ●● ● ●
● ● ●●
● ●●
●● ●●
●●
●●● ● ●
●● ●●
●
● ●●● ●● ●● ● ●●●●●● ● ●
●●●● ●

30
25
● ● ● ●
● ● ● ●
● ● ● ●●●
●●●● ● ●●● ●●● AlNalinity_of_ash ● ●● ●● ●● ● ●

15
● ● ● ●● ● ●● ●● ●● ●
● ● ●● ● ●
● ● ● ●
●● ●● ● ● ●
● ● ● ● ● ● ● ●●

10
3.6

● ● ● ● ●●
●● ● ● ●●
● ● ● ● ●●● ● ●
●●● ●●●
●
● ● ●●●●
● ●
●● ●
●●● ●
●●● ● ●
●●●● ●● ●● ●● ●●●
3.4

●
●
●●
●
● ●● ●● ●●
●
● ●● ●● ●● ●
● ● ●● ● ● ●●●●● ●●● ●●● ● ●●
●●●●●
● ●● ● ●● ●
●● ●●
●●●
● ●
●●●●●
● ●
●
●
●●
●● ●●
●● ● ●●
●● ●●●●
● ●● ● ● pH ●● ●●
● ●●●●
● ●
● ●● ●● ● ●● ● ●
● ●● ●●
●●●
● ●
3.2

●●●●●
● ● ●
●●●●●● ● ● ●●●● ●● ●● ●
● ●
● ●
●● ● ● ●
●
● ● ● ●
● ● ● ●
3.0

● ● ● ●

160
● ● ● ●
●● ● ● ●● ●
● ●● ● ● ●

120
●● ● ● ●●● ●
●
● ●●
●●●●
●●● ● ● ●●●● ●● ● ● ● ●● Magnesium
●●● ●●● ● ●
● ●●
●● ●● ●●● ●
●
●●
●●
●●●●●●
●●
●● ●● ●● ●●● ●●●
● ● ● ●●●●
●●
● ●●
●
●●●●● ●
● ● ●● ●● ● ●●
●●
●● ●●●● ●●
●●● ●●●●●●
●●● ●
●●●● ●
● ●
●●●
●●
● ●
●● ● ●● ● ●● ● ●
●●● ● ●●
●●
●
●●●● ●● ● ● ●
● ●●● ●●
●● ● ● ● ●● ●
●
●
● ● ● ● ● ●● ● ● ●● ●●

80
1 2 3 4 5 10 15 20 25 30 80 120 160

Figure 2.30 The same pairs plot for ﬁve of the 27 physical and chemical
measurements of wine samples, with true wine types shown by color.

It remains the fact that with BIC, although not with ICL, model-based clustering
has chosen seven clusters, even though there are in fact only three wine varieties.
The confusion matrix for comparing the seven-cluster solution with the partition
into three wine types is shown in Table 2.8. It can be seen that each of the three
wine types has itself been divided into either two or three clusters by the clustering
algorithm. If the clusters were merged correctly, the number of errors would be
very similar to the three-cluster solution (11 versus 10). Not surprisingly, though,
the classiﬁcation error rate (52%) and the Adjusted Rand Index (0.418) are much
worse than for the three-cluster solution. Note, however, that the ARI is still
2.7 Illustrative Analyses 63

−25000
−25000
●
● ● ● ●
● ●
● ●
● ● ●
● ● ● ●
● ●
● ●
● ●
● ● ● ● ● ●
● ●
● ●

−30000
−30000
−35000

−35000
−40000

−40000
BIC

ICL
−45000

−45000
EII EVE EII EVE
VII VEE VII VEE
−50000

−50000
● EEI VVE ● EEI VVE
● VEI EEV ● VEI EEV
● EVI VEV ● EVI VEV
● VVI EVV ● VVI EVV
−55000

−55000
EEE VVV EEE VVV

1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

Number of components Number of components

Figure 2.31 BIC (left) and ICL (right) for the wine data.

● ● ●
−24000

●
−24000

● ● ● ●
● ● ● ●
● ● ●
●
●
● ● ● ● ●
● ● ●
● ● ●
●
● ●
−24200

● ●
−24200

● ●
● ●
● ●
● ● ● ●
● ●
−24400
−24400

● ●
● ●
BIC

ICL

−24600
−24600

● ●
● ●
● ●

● ●
−24800

● EEI EEE ● EEI EEE

−24800

● ●
● VEI EVE ● VEI EVE
● EVI VEE ● EVI VEE
● VVI VVE ● VVI VVE
● ●

2 3 4 5 6 7 8 9 2 3 4 5 6 7 8 9

Number of components Number of components

Figure 2.32 Zoomed version of the BIC (left) and ICL (right) plots for the
wine data.

better than for the three heuristic methods, even though the latter were “told”
the correct number of clusters.

There are 14 wine type/year combinations in the data set. The confusion matrix
for the seven-cluster solution against those 14 type/year categories is shown in
Table 2.9. Within each type, several of the clusters tend to separate out particular
years. For example, cluster 2 corresponds to Barolo wines that are not from 1973,
cluster 5 corresponds to later Grignolino wines, particularly from 1976, while
cluster 7 is mostly made up of Barbera wines from 1978. Thus it seems that the
64 Model-based Clustering: Basic Ideas

Table 2.5 Wine data: top models by BIC and ICL. The number of clusters is denoted by G. The
best model by each criterion is shown in bold font. 23,950 has been added to all BIC and ICL
values for readability.

G Model BIC ICL

7 VEI −1.91 −8.99
3 EVI −3.87 −5.05
3 VVI −4.37 −6.46

● ●
12

12
● ●

● ● ● ●
● ●
●● ●●
10

10
● ●
●● ●●
●● ● ●● ●
● ●
● ● ● ●
● ● ● ●
Color_Intensity

Color_Intensity
● ●
8

8
● ●
●●
● ● ●●
● ●
● ●
● ●
● ● ● ●
● ●
6

6
● ●
●
●● ● ●
●● ●
● ● ● ● ● ●
● ●
● ●
● ● ● ●
● ●
● ●
● ● ● ●
● ● ● ●
4

● ●
2

1 2 3 4 5 1 2 3 4 5

Flavanoids Flavanoids

Figure 2.33 Wine data: true classification (left) and classification errors
from model-based clustering (right), in clustering samples by wine type. The
errors are shown as solid black symbols, with the shape corresponding to the
true classification. Two of the 27 measurements, Flavanoids and Color
Intensity, are shown.

more detailed seven-cluster clustering selected by BIC is at least partly capturing

diﬀerences between years.

Table 2.6 Wine data: confusion matrix for three-cluster model and wine types. For each cluster,
the most represented wine type is marked by a box.

Cluster Wine type

Barolo Grignolino Barbera

1 58 7 0
2 1 62 0
3 0 2 48
2.7 Illustrative Analyses 65

Table 2.7 Wine data: performance of diﬀerent clustering methods for classifying wine types.
classiﬁcation error rate (CER – smaller is better) and Adjusted Rand Index (ARI – larger is
better). All methods are based on G = 3 clusters.

Method CER % ARI

Model-based (EVI,3) 5.6 0.830
Single link 59.0 0.001
Average link 32.6 0.400
Complete link 32.3 0.323
k-means 25.8 0.353

Table 2.8 Wine data: confusion matrix for seven-cluster model chosen by BIC and wine types.
For each cluster, the most represented wine type is marked by a box.

Cluster Wine type

Barolo Grignolino Barbera

1 31 0 0
2 25 0 0
3 3 17 0
4 0 28 0
5 0 18 0
6 0 8 21
7 0 0 27

2.7.2 Craniometric Analysis

We now consider a data set consisting of 57 different cranial measurements from
2,524 skulls. The goal is to classify the skulls according to the populations from
which they came. The idea of using quantitative data and cluster analysis to
classify skulls comes from Howells (1973, 1989, 1995). Howells (1995) used a sum of
squares hierarchical clustering method, and was unable to find strong evidence for
clustering of populations. Here we reanalyze the data using model-based clustering,
drawing on the earlier analysis of Pontikos (2004, 2010). The skulls come from 30
different populations.
The first version of the data set we analyze here was documented by Howells
(1996). The version we analyze here is the “test” data set assembled by Pontikos
(2010), slightly modifying an earlier version considered by Pontikos (2004).
The data are roughly balanced between male and female skulls. Exploratory
analysis indicated that the marginal distribution of the male skull measurements is
statistically indistinguishable from that of the female skull measurements increased
by 5%. We therefore multiplied the female measurements by 1.05 prior to analysis,
and analyzed the male and female skulls together.
The BIC values for the model-based clustering of the resulting data are shown
in Figure 2.34. The best model for the covariance structure is the VEE model,
which says that the covariance matrices for the different clusters are equal up to
Table 2.9 Wine data: confusion matrix for the seven-cluster model chosen by BIC against wine type and year. The horizontal lines group clusters
that correspond largely to the same type.

Cluster Barolo Grignolino Barbera

1971 1973 1974 1970 1971 1972 1973 1974 1975 1976 1974 1976 1978 1979
1 6 17 8 0 0 0 0 0 0 0 0 0 0 0
2 13 1 11 0 0 0 0 0 0 0 0 0 0 0
3 0 2 1 2 4 1 0 5 4 1 0 0 0 0
4 0 0 0 4 3 3 9 8 1 0 0 0 0 0
5 0 0 0 0 0 1 0 3 4 10 0 0 0 0
6 0 0 0 3 2 2 0 0 0 1 9 5 5 2
7 0 0 0 0 0 0 0 0 0 0 0 0 24 3
2.7 Illustrative Analyses 67

−650000
−750000

● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
● ●
●
●
●
●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
● ●
● ●
● ●
●
●
BIC

●
−850000

EII EVE
VII VEE
● EEI VVE
● VEI EEV
● EVI VEV
● VVI EVV
−950000

EEE VVV

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Number of components

Figure 2.34 BIC plot for the Howells craniometric data.

a cluster-speciﬁc scale factor. The BIC values for diﬀerent numbers of clusters
for the VEE model are shown in Figure 2.35. The maximum is attained at G = 15
clusters.
The correspondence between the 15 clusters inferred by model-based clustering
and the 30 populations is shown in Table 2.10. The number of clusters is smaller
than the number of true populations, so there are several clusters that include
the majority of skulls from more than one population. Apart from this, however,
the correspondence between the clusters and the populations is close.
Seven of the 15 clusters correspond closely to a single population: Andaman,
Berg, Buriat, Bushman, Eskimo, Santa Cruz and Tolai. Cluster 8 groups the Ainu
(aboriginal Japanese) and Guam skulls. Cluster 9 groups most of the Arikara and
68 Model-based Clustering: Basic Ideas

−641000
−642000
−643000
BIC

−644000
−645000

VEE

1 2 3 4 5 6 7 8 9 11 13 15 17 19 21

Number of components

Figure 2.35 BIC plot for the VEE model only for the Howells
craniometric data.

Peruvian skulls; these are both Native American populations. Cluster 10 groups
(aboriginal) Australian and Tasmanian skulls; these are geographically proximate.
Cluster 11 groups Easter Island and Mokapu skulls; again these are relatively
geographically proximate. Cluster 12 groups Dogon from Mali, Teita from Kenya
and Zulu from South Africa; these are all sub-Saharan African populations.
Cluster 13 groups (medieval) Norse, (Hungarian) Zalavar and ancient Egyptian
skulls; these are all viewed as Caucasoid populations. Cluster 14 groups most of
the North and South Maori and Moriori skulls; these are all from New Zealand.
Finally, cluster 15 groups the Anyang and Hainan (both from China), Atayal
2.7 Illustrative Analyses 69

Table 2.10 Craniometric data: confusion matrix for the 15 clusters inferred by model-based
clustering and the 30 populations from which the skulls come.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
ANDAMAN 59 0 0 1 0 0 0 1 0 0 0 6 2 0 1
BERG 0 98 0 1 0 1 0 0 1 0 0 0 8 0 0
BURIAT 0 2 105 0 0 0 0 1 0 0 0 0 0 0 1
BUSHMAN 0 0 0 80 0 0 0 0 0 1 0 8 0 0 1
ESKIMO 0 0 0 0 108 0 0 0 0 0 0 0 0 0 0
SANTA CRUZ 0 1 0 0 0 95 0 0 3 0 0 0 3 0 0
TOLAI 0 0 0 0 0 0 100 1 0 3 2 2 0 1 1
AINU 0 1 0 0 0 0 0 71 0 0 2 3 3 0 6
GUAM 0 0 0 0 0 0 0 45 1 0 0 0 0 0 11
ARIKARA 0 5 0 0 0 4 0 3 43 0 1 1 1 4 7
PERU 1 0 0 0 0 3 0 3 102 0 0 0 0 0 1
AUSTRALIA 0 0 0 0 0 0 2 0 0 98 0 0 1 0 0
TASMANIA 0 1 0 0 0 0 10 0 0 70 3 2 1 0 0
EASTER I 0 0 0 0 0 0 0 1 0 0 84 1 0 0 0
MOKAPU 0 0 0 0 0 0 0 4 0 0 92 0 0 3 1
DOGON 0 0 0 0 0 0 0 0 0 0 0 99 0 0 0
TEITA 0 0 0 0 0 0 0 0 0 1 0 80 2 0 0
ZULU 0 0 0 3 0 0 1 0 0 1 0 91 2 0 3
EGYPT 0 0 0 0 0 0 0 0 0 0 0 7 104 0 0
NORSE 0 15 0 0 0 2 0 5 0 0 0 1 85 1 1
ZALAVAR 0 32 0 0 1 0 0 4 1 1 0 1 57 0 1
MORIORI 0 0 0 0 0 0 0 2 0 0 5 0 0 100 1
N MAORI 0 0 0 0 0 0 0 0 0 0 4 0 0 6 0
S MAORI 0 0 0 0 0 0 0 2 0 0 3 0 0 5 0
ANYANG 0 0 0 0 0 0 0 5 1 0 0 0 0 0 36
ATAYAL 0 0 0 0 0 0 1 2 0 0 0 2 0 0 42
HAINAN 0 1 1 0 1 0 0 9 1 0 0 1 0 0 69
N JAPAN 1 0 0 0 0 0 0 12 1 0 0 3 0 0 70
S JAPAN 0 0 0 0 0 0 0 5 1 0 0 0 0 0 85
PHILIPPINES 1 1 0 0 0 0 0 6 1 0 1 4 2 0 34

(Taiwan aboriginal), North and South Japanese, and Philippine populations. These
are all East Asian populations.
If we match each cluster with the population most represented within it, we
calculate a misclassification rate of 44.7%, so that about 56% of the skulls were
correctly classified. However, it seems more reasonable to match each cluster with
the set of geographically proximate populations most represented within it, as
discussed above. If we do this, the misclassification rate is only 12.3%. Overall,
it seems that the clustering has largely kept skulls from the same populations
together, but in some cases has been unable to distinguish between populations
that are relatively close geographically or historically.
Since it is known that there are in fact 30 populations, it is of interest to run
the analysis with the true number of groups, G = 30. This time the preferred
model for the covariance structure is the equal-covariance model, EEE, which is
closely related to the VEE model found best for G = 15 clusters. The confusion
matrix is shown in Table 2.11.
When the algorithm is told to use the correct number of clusters, 30, the
Table 2.11 Craniometric data: Confusion matrix for model-based clustering when the true number of groups, 30, is assumed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
AINU 69 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 1 2 3 2 1 3 0
ANDAMAN 0 63 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 1 0 0 0 0 1 1 0 2
BERG 0 0 73 0 1 0 2 0 0 0 1 0 1 0 0 0 0 0 2 0 28 0 0 0 0 0 0 0 1 0
BURIAT 0 0 0 106 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 2 0 0 0 0 0
BUSHMAN 0 0 0 0 81 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 1 5 0 0 0 1 0 0
ESKIMO 0 0 0 0 0 108 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
SANTA CRUZ 0 0 0 0 0 0 93 0 0 0 3 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 2
TOLAI 0 0 0 0 0 0 0 82 19 0 0 0 0 3 0 1 0 1 0 0 0 0 0 1 0 0 1 2 0 0
ARIKARA 0 0 1 0 0 0 5 0 0 45 4 0 4 0 0 0 2 1 0 0 2 0 0 0 2 3 0 0 0 0
PERU 0 1 0 0 0 0 3 0 0 1 37 27 37 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 2
AUSTRALIA 0 0 0 0 0 0 0 1 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
TASMANIA 0 0 0 0 0 0 0 8 4 0 0 0 0 3 65 2 1 0 0 0 2 1 0 0 0 0 1 0 0 0
EASTER I 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 84 2 0 0 0 0 0 0 0 0 0 0 0 0 0
MOKAPU 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 94 2 0 0 0 0 0 0 0 0 1 0 0 0
MORIORI 0 0 0 0 0 0 1 0 0 2 0 0 0 0 0 1 5 98 0 0 0 0 0 0 1 0 0 0 0 0
N MAORI 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 1 5 0 0 0 0 0 0 0 0 0 0 0 0
S MAORI 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 7 0 0 0 0 0 0 0 0 0 0 0 0
EGYPT 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 68 38 2 0 0 1 0 0 0 1 0 0
NORSE 0 0 1 0 0 0 2 0 0 0 0 0 0 0 0 0 0 1 67 4 34 0 1 0 0 0 0 0 0 0
ZALAVAR 1 0 4 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 12 5 71 0 0 1 0 1 0 0 0 0
DOGON 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 87 1 6 0 1 0 1 0 1
TEITA 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 3 66 9 0 0 0 0 0 1
ZULU 0 0 0 0 3 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 5 1 90 0 0 0 0 0 0
ANYANG 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 27 11 2 1 0 0
ATAYAL 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 28 0 16 2 0
GUAM 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 1 51 1 0 0
HAINAN 1 0 0 1 0 0 0 0 0 1 0 0 2 0 0 0 1 0 0 0 0 0 0 0 30 29 2 15 1 0
N JAPAN 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 26 15 2 1 40 0
PHILIPPINES 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 1 1 0 1 0 1 5 7 3 28 0 1
S JAPAN 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 55 25 1 3 6 0
2.8 Who Invented Model-based Clustering? 71

Table 2.12 Performance of diﬀerent clustering methods on the craniometric data: classiﬁcation
error rate (CER – smaller is better) and Adjusted Rand Index (ARI – larger is better). Except
for model-based clustering, the other methods are run with the true number of clusters, G = 30.

Clustering Method CER % ARI

Model-based (Ĝ = 15) 44.7 .463

Model-based (G = 30) 24.7 .612
Single link 94.5 0
Average link 80.2 .048
Complete link 73.1 .093
k-means 65.5 .160

misclassiﬁcation rate is 24.7%, as compared to the 44.3% misclassiﬁcation rate of

the 15-cluster solution preferred by BIC. However, the misclassiﬁcation rate of the
15-cluster solution is only 12.3% when compared with the aggregated populations
it detects.
In the 30-cluster solution, several populations are split between multiple clusters,
which does not happen in the 15-cluster solution. If one wishes to avoid splitting
populations, one might therefore prefer the 15-cluster solution. On the other hand,
in the 30-cluster solution, 17 of the 30 populations are largely concentrated in
one cluster each, which might lead to a preference for the 30-cluster solution.
In any event, this analysis suggests that, even when the “true” number of groups
is known, it is worth paying attention to the solution preferred by BIC or another
statistical criterion if the estimated number of groups is lower than the “true”
number. The BIC solution may be indicating the maximum number of clusters
that can be reliably detected; increasing the number of clusters increases the risk
of splitting up subpopulations in the data.
Diﬀerent clustering methods are compared for these data in Table 2.12. The
model-based approach gives the best results, even when it is not told the correct
number of clusters.

2.8 Who Invented Model-based Clustering?

Who invented model-based clustering? And when did it happen? For the purposes
of answering this question, we will deﬁne model-based clustering as involving the
following three elements: (1) being based on a ﬁnite mixture probability model;
(2) a systematic statistical method for estimating the parameters of the model;
and (3) a systematic method for classifying the observations conditionally on the
model. As it unfolded, there were two quite separate inventions, one for discrete
data and one for continuous data.
The American and Austrian sociologist Paul F. Lazarsfeld (1901–1976) (Fig-
ure 2.36 (a)) invented model-based clustering for discrete data in a 110-page opus
published in 1950. It appeared as two chapters in Volume IV (Measurement and
72 Model-based Clustering: Basic Ideas

Figure 2.36 Left: Paul Lazarsfeld; Right: John H. Wolfe.

Prediction) of the massive series Studies in Social Psychology in World War II

which described a range of social psychological studies carried out by interdis-
ciplinary teams of researchers employed by the US War Department to study
military personnel in World War II. (Lazarsfeld, 1950a,c). He called it latent class
analysis, a special case of the broader latent structure analysis that he proposed
in the same work. It was motivated by the analysis of a large survey aimed at
understanding American soldiers who had fought in World War II.
It has been suggested that Lazarsfeld’s fondness for the word “latent,” which
he was among the first to use in a statistical context, was due to his having spent
his early adulthood in Vienna at the time when Sigmund Freud was active there,
developing his theory of the latent content of dreams (Henry, 1999). Lazarsfeld
also referred to observed data as “manifest data,” again echoing Freud’s theory
of dreams, in which the manifest content of the dream is the dream as it is
remembered, while the latent content of the dream is its hidden meaning.
This work is remarkable in several ways. It created latent class analysis from
scratch, and included fully worked out versions of the three essential elements
of model-based clustering we have identified, namely model, estimation and
classification. There seem to be no real precursors, and the chapters cited almost
no previous publications. It acknowledges conversations with sociologist Samuel
Stouffer and statistician Frederick Mosteller, but their contributions seem to have
been minor.
The probability model he used was a finite mixture of distributions, each
one of which was a product of independent multinomial distributions, one for
2.8 Who Invented Model-based Clustering? 73

each variable (Lazarsfeld, 1950a, e.g. p.384). He proposed a method of moments

estimation method for the model, where observed marginal counts of the multiway
table are equated with their expected values under the model, and the resulting
system of nonlinear equations is solved. Finally, in one of the earliest uses of the
term, he provided what he called the “a posteriori probability” that someone with
a particular response pattern came from a given latent class (Lazarsfeld, 1950c,
p.430).
It took a long time after that for these ideas to be translated to model-based
clustering for continuous data. Tiedeman (1955) described verbally the mixture of
multivariate normal distributions as a model for the analysis of “types,” classes
or groups in psychology, and came very close to writing it down explicitly. He was
considering the situation where the covariance matrix for each group is different,
or what we would call the VVV model. However, he did not provide either a
systematic method for estimating the parameters when the type labels were not
available, or a systematic classification method. He correctly identified these as
key requirements for an applicable method and encouraged research to develop
them. So, Tiedeman can be viewed as a precursor of model-based clustering for
continuous data, but not its inventor.
John H. Wolfe (Figure 2.36(b)), an American born in Buenos Aires in 1933,
graduated with a B.A. in mathematics from Caltech in 1955, and then went to
graduate school in Psychology at University of California, Berkeley. He worked
with Robert Tryon, who as we have seen wrote the first book on cluster analysis
in 1939. Tryon encouraged Wolfe to develop hierarchical clustering, but Wolfe
recounts that he couldn’t see a rationale for it without arbitrary assumptions at
several stages. Around 1959, Paul Lazarsfeld visited Berkeley and gave a lecture
on his latent class analysis, which fascinated Wolfe, and led him to start thinking
about how one could do the same thing for continuous data.
Wolfe’s 1963 M.A. thesis is an attempt to do this. He started by defining a
“type” as one of the component distributions of a mixture of distributions. Then
he said:
“The above definition makes it clear that the problem of identifying and describing types is
merely an application of the statistical theory of estimation. The population from which a
sample of objects is drawn is assumed to have a particular form of distribution (e.g. a mixture
of r normal distributions with unknown parameters, such as means and variances for each type
and percentage of objects of each type). The task is to devise a procedure for estimating the
population parameters.” (Wolfe, 1963, p.3)

This is eﬀectively the model that underlies continuous model-based clustering to

this day. However, the estimation method he used was an eﬀort to use Mahalanobis
distances, which was not very eﬀective (Wolfe, 1963, chap. 2). Wolfe acknowledged
this, saying that the methods in the thesis were bad. So his thesis can be viewed
as a heroic failure. However, he did say at the end of the thesis that he had
developed a much better method called “maximum likelihood analysis of types.”
He later said that his professor didn’t believe him, but grudgingly approved the
thesis anyway (Wolfe, 2018).
After graduating from Berkeley, Wolfe took a job with the US Navy in San
74 Model-based Clustering: Basic Ideas

Figure 2.37 Cover page of Wolfe (1965), the paper in which model-based
clustering for continuous data was invented.

Diego ﬁrst as a computer programmer and then as an operations research analyst.

He continued his research on clustering and in 1965 he published the paper that
really did solve the problem (Wolfe, 1965) (see Figure 2.37). He used the same
mixture of multivariate normals model as in his thesis, which we would call the VVV
model. He estimated it by maximum likelihood using a Newton–Raphson method.
A method of successive substitution is proposed as an alternative. Finally, he gives
the correct explicit expression for the (posterior) probabilities of membership in
each “type” (or cluster). Thus this paper for the first time proposes a method
for model-based clustering of continuous data that includes all our key elements:
model, estimation method and a method for classification.
Wolfe (1965) is an unusual paper, because it took the form of a Technical
Bulletin meant to document a FORTRAN computer program, the listing of which
is given in the paper, covering 22 pages (Figure 2.37). It is, however, a regular
publication, submitted in 1963, revised in 1964, and approved by a sequence of
scientific reviewers and supervisors. The fact that it was accompanied by the
computer program to implement it was rare then, but of course is commonplace
now, giving it in some ways a modern feel. It also may have led Wolfe to consider
some issues which have subsequently been the subject of substantial threads of
work in the model-based clustering literature.
One of these is initialization. Wolfe recognized the importance of this for his
optimization methods, and proposed specific, and quite sensible, initialization
methods. He also recommended using “any good prior information from theoretical
2.9 Bibliographic Notes 75

hypotheses or any other classiﬁcation-clustering procedures to provide initial

estimates” (Wolfe, 1965, p.12). Initialization methods have been a signiﬁcant
consideration in the ensuing literature.
Secondly, he recognized the importance of variable selection, saying:
“By intuition, cluster analysis of variables, factor analysis, or any other means, reduce the
number of variables to those that are really important for discriminating types.” (Wolfe, 1965,
p.12)

This is highly prescient, preﬁguring the signiﬁcant literature on variable selection

in model-based clustering that got started almost 40 years later. His conjecture
that reducing the number of variables to those that are important for clustering
was borne out by subsequent research.
Thirdly, he recognized that computer time may become excessive if the sample
is large. He gives the advice:
“If you have a large sample, say 10,000, don’t run it all at once. Take a small sample, say 100
to 200 cases at random and run an experiment using a weak convergence criterion. Get an idea
of how many types there are and some initial estimates from the small sample. Run the entire
set of data using these initial estimates on speciﬁed numbers of types.” (Wolfe, 1965, p.12)

What to do with large samples also emerged as a concern, and Wolfe’s advice
prefigured later findings remarkably accurately.
Fourthly, he recognized that with model-based clustering the choice of the
number of clusters (or “types”) is reduced to one of statistical hypothesis testing.
He proposed a likelihood-ratio test of the hypothesis that there are r types against
the alternative that there are r − 1 types. He wrote that the test statistic has a
χ2 distribution, which we now know not to be the case, but this was a key insight
for subsequent work.
Finally, he evaluated his methods using a simulated example, which became a
preferred approach in this literature.
On the basis of this one paper, we can conclude that John Wolfe was the inventor
of model-based clustering for continuous data. He developed the ideas further in
Wolfe (1967), and in Wolfe (1970) produced the first journal article describing his
results, but the key insights were already present in his 1965 Technical Bulletin.

2.9 Bibliographic Notes

Ward (1963) proposed hierarchical agglomerative clustering using the sum of
squares method. This involves choosing the cluster memberships zi,g so as to
G
minimize the criterion tr(W ) with respect to the zi,g , where W = g=1 Wg , and
n
Wg = i=1 zi,g (yi − μ̂g )(yi − μ̂g )T . This amounts to minimizing the sum of within-
cluster sums of squared distances from the cluster means. MacQueen (1967) later
proposed an iterative relocation algorithm for the same minimization problem,
which became known as k-means.
Although Ward was not working in a model-based context, his clustering crite-
rion later turned out to correspond to a speciﬁc probability model. Minimizing
76 Model-based Clustering: Basic Ideas

Ward’s sum of squares is equivalent to likelihood estimation for the multivari-

ate normal mixture model with Σg = λI, i.e. the EII model. The likelihood
being maximized, however, was not the mixture likelihood (2.6), but instead
the
n likelihood
G of the parameters and the cluster memberships jointly, namely
i=1 g=1 φ(y i |μ zi,g , Σzi,g )
zi,g
, called the classification likelihood. Note that the
mixture proportions τg do not appear in this likelihood. Edwards and Cavalli-
Sforza (1965) described a similar method. The authors of these papers did not
put their methods in a model-based context, however, and so are not candidates
to be considered as inventors of model-based clustering for continuous data.
Friedman and Rubin (1967) proposed a clustering method that corresponds to
maximizing the classification likelihood for the Σg = Σ, or EEE model. Scott and
Symons (1971) proposed a CEM algorithm for model-based clustering based on
the model where the component-specific covariances, Σg , can vary freely between
components, namely the VVV model.
Although John Wolfe deserves credit as the inventor of model-based clustering
for continuous data, Day (1969), working independently, was the first to publish
a journal article describing the approach. He proposed estimating a multivariate
normal mixture model by maximum likelihood and clustering by assigning each
object to the most likely cluster.
Binder (1978) developed a Bayesian estimation method for the VVV model. In
this framework, the difference between the classification likelihood and mixture
likelihood approaches disappears if the prior distribution for zi,g is multinomial
with probabilities τg . In that case, the classification maximum likelihood is an
approximation to the posterior mode of z under both approaches, as shown by
Banfield and Raftery (1993).
Murtagh and Raftery (1984) appear to have been the first to develop a model-
based clustering method based on the eigenvalue decomposition of the component
covariance matrices. They developed a hierarchical agglomerative model-based
clustering method based on the variable orientation, or EEV model, Σg = λDg ADgT ,
and applied it to problems in character recognition.
McLachlan and Basford (1988) was the first book on model-based clustering.
This was a major advance, showing the potential of the approach, and spurred a
great deal of interest.
Banfield and Raftery (1989, 1993) coined the term “model-based clustering”
and, building on Murtagh and Raftery (1984), proposed the general eigenvalue
decomposition-based geometric formulation of Σg and the corresponding family of
model-based clustering models. They also proposed the normal-uniform mixture
model for outliers in clustering. They introduced the idea of using approximate
Bayes factors to choose the number of clusters using the approximate weight of
evidence (AWE) criterion based on the classification likelihood. They also described
the first version of the mclust software, written by Chris Fraley in FORTRAN and
also available as an S-PLUS function, and published on the StatLib repository
in 1991 (Fraley and Raftery, 1999, 2007b). Celeux and Govaert (1995) introduced
maximum mixture likelihood for the Banfield–Raftery family of models using the
EM algorithm.
2.9 Bibliographic Notes 77

Lavine and West (1992) and Bensmail et al. (1997) proposed methods for
Bayesian estimation of the multivariate normal mixture model for clustering using
Markov chain Monte Carlo (MCMC). They each did this for a different subset of
the Banfield–Raftery family of models. Cheeseman and Stutz (1995) introduced
the AUTOCLASS software, which carried out model-based clustering in a partly
Bayesian way.
Alternative parsimonious covariance matrix structures have been proposed for
model-based clustering. McNicholas and Murphy (2010a) developed a model-based
clustering approach for longitudinal data based on the Cholesky decomposition of
the covariance and Fop et al. (2018) proposed a sparse covariance structure for
model-based clustering.
Celeux and Soromenho (1996) proposed a normalized entropy criterion (NEC)
to select the number of clusters and the clustering model. The NEC criterion
has been improved by Biernacki et al. (1999) to decide between one and more
than one clusters. NEC is available in the MIXMOD software. There have been
many comparisons of the various available model selection criteria in the context
of model-based clustering, including, more recently, Steele and Raftery (2010).
Alternatives to model selection have also been explored. These include Bayesian
model averaging, in which, instead of choosing a single model, predictive distribu-
tions are averaged over all the models considered, with weights that reflect their
predictive performance over the data (Russell et al., 2015). They also include
regularization methods (Witten and Tibshirani, 2010), which have been compared
with model selection methods by Celeux et al. (2014).
Finding the best way to initialize the EM algorithm remains an open ques-
tion. We have described the hierarchical model-based clustering and smallEM
approaches, but improvements are possible. Improved versions of the initialization
strategy have been described by Scrucca and Raftery (2015) and are implemented
in the mclust R package. O’Hagan et al. (2012) developed a variant of the smallEM
approach that discarded unpromising starting values incrementally. Michael and
Melnykov (2016) and O’Hagan and White (2018) proposed an alternative approach
to initialization based on model averaging, and reported good results.
Most of the heuristic clustering methods that predated model-based clustering
were based on a matrix of similarities between objects, often derived from a
matrix of measured characteristics of the objects, as discussed in Section 1.1.
Model-based clustering, on the other hand, has for the most part modeled the full
data on the measured characteristics directly. Sometimes, however, the full set of
measured characteristics isn’t available, or the similarities are what are measured
directly, or it is desirable to model the similarities rather than the characteristics
for computational reasons. For these situations, Oh and Raftery (2007) developed
a model-based clustering method based on the similarity matrix rather than
the measured characteristics, building on the Bayesian multidimensional scaling
method of Oh and Raftery (2001).
Model-based clustering has been used in a wide range of application areas,
several of which we illustrate in this book. Others include astronomy (Mukherjee
et al., 1998), chemistry (Fraley and Raftery, 2006), political science (Ahlquist and
78 Model-based Clustering: Basic Ideas

Breunig, 2012), education (Howard et al., 2018) and actuarial science (O’Hagan
and Ferrari, 2017). It has been used in a variety of ways in the analysis of gene
microarray data. For example, Yeung et al. (2001) used it to cluster genes according
to their expression levels in a range of experiments, while Young et al. (2017) used
it to remove artifacts caused by incorrect swapping of genes in paired experiments.
McLachlan and Peel (2000) and Fraley and Raftery (1998, 2002) gave overviews
of the area to that point. More recent overviews have been provided by Ahlquist
and Breunig (2012) and McNicholas (2016b,a).
3
Dealing with Diﬃculties

In this chapter, we will discuss some difficulties that the basic modeling strategy
can have in specific contexts, and describe some relatively simple strategies for
overcoming them, which are readily available and implemented in current software.
More elaborate strategies will be described in later chapters.
One difficulty is that data sets often contain outliers, that is data points that
do not belong to any cluster. We will discuss some ways of dealing with this issue
in Section 3.1.
Another issue is that maximum likelihood estimates of the model parameters
can be degenerate. Bayesian estimation can be useful for regularizing the inference
in such situations, and avoiding these degeneracies. We discuss one Bayesian
approach to this problem in Section 3.2. This is a simple approach designed
to solve the degeneracy problem; when degeneracy is not an issue, it produces
solutions similar to those produced by maximum likelihood.
It often happens that some clusters are not Gaussian, so the most common
probability model in model-based clustering, the multivariate Gaussian, does not
apply. Using it tends to lead to the representation of one cluster by a mixture of
Gaussian components rather than just one. This still gives good estimates of the
underlying overall probability density function, but can lead to overestimation
of the number of clusters. In Section 3.3 we describe one way to overcome this
difficulty, by merging components after an initial fit.
Finally, we briefly mention some other approaches to these issues in Section 3.4.
Other approaches will be described in more detail later in the book.

3.1 Outliers
3.1.1 Outliers in Model-based Clustering
In general, in statistical analysis, an outlier is deﬁned as one of a relatively small
number of points that do not follow the same pattern as the main bulk of the data.
In model-based clustering, outliers are points that do not belong to any of the
clusters. In many statistical analyses, outliers are often data points that lie outside
the main bulk of the data, and this can be true in cluster analysis too. However,
they may also be “inliers” in the sense that they may be between clusters, and
thus in the interior of the data, particularly if clusters are well-separated. Outliers
can increase the estimated number of clusters beyond what is present in the data,
79
80 Dealing with Diﬃculties

Simulated data with outliers

●
●
●
●
●

2
●●
●
● ● ●●
●
●●● ●●
●●●
●●
●
●● ●
●● ●
● ● ●● ● ● ●●
●
●
●● ●
●● ●●●●●●
●
●●
●
●●●●
●
●●●●●
● ●
●
●
● ●●
● ●● ●●●●●●
● ●
●●●●●● ●
●
● ●
●●
● ●●●
●
●●●● ● ●●
●●●
●
● ●● ●●●● ●
●
● ● ●
●
●
0

● ●
● ●
● ● ●
●● ●● ●
●
Y

● ●●
● ● ● ●
● ● ● ●●●● ●
● ●● ● ●
●●●●●●●●●● ● ●
●● ●●● ●●●
● ●● ●
●
●
● ●
−2

●
●

●
−4

−4 −2 0 2 4 6

Figure 3.1 Simulated data set with two well-separated clusters and
outliers.

because if not identiﬁed they can themselves be identiﬁed as separate clusters.

Also, if not separated out or down-weighted, they can create error in the estimates
of the cluster means, and inﬂate the estimated cluster variances.
A simple example with some simulated data is shown in Figure 3.1. In this
data set, 180 two-dimensional data points were simulated from two well-separated
clusters using the mixture model 23 N ((1, 1)T , 0.32 I) + 13 N ((−1, −1)T , 0.32 I). Then
20 noise points were added, simulated from a N (0, 4I) distribution, but with
points constrained to not overlap with the clusters.
Nineteen of the 20 outliers are fairly clear visually in Figure 3.1. However, there
are several points in the vicinity of the clusters that are more visually ambiguous.
Figure 3.2 shows which points were simulated as outliers; the outlier closest to
the left cluster is hard to detect visually.
Sometimes model-based clustering can handle outliers without any special
precautions, and indeed this turns out to be the case in our simulated data set.
We will soon see that this does not always work, though, and it can be necessary
to take precautions against outliers. Figure 3.3 shows the results from a routine
application of model-based clustering. BIC chooses the unequal-variance spherical
model, VII, quite decisively. It recovers the two actual clusters, and represents
the outliers by a third, highly dispersed, mixture component. It is thus able to
3.1 Outliers 81

Simulated data with outliers: outliers shown

●
●
●
●
●

● ●
● ●
● ● ●
●● ●● ●
●
Y

● ●●
● ● ● ●
● ● ● ●●●● ●
● ●● ● ●
●●●●●●●●●● ● ●
●● ●●● ●●●
● ●● ●
●
●
● ●
−2

●
●

●
−4

−4 −2 0 2 4 6

Figure 3.2 Simulated data set with outliers shown.

identify all the simulated outliers correctly, except for the one close to the left
cluster, which is hard to identify visually anyway.
We will now describe two relatively straightforward ways to deal with outliers.
One is to add an additional component to the mixture model to represent the out-
liers, typically a uniform component. A second approach is to remove observations
identiﬁed as outliers at the estimation stage.

3.1.2 Mixture Modeling with a Uniform Component for Outliers

A straightforward but more general way to handle outliers, or “noise,” in model-
based clustering is simply to add an additional component to the mixture model
to represent the outliers. Ideally, this should have outlier-like properties, being
dispersed over the data region. One form that has been used successfully is a
uniform distribution on the data region. This corresponds to the assumption that
outliers are generated approximately by a homogeneous Poisson process on the
data region. The Gaussian mixture model deﬁned by (2.1) and (2.2) then becomes

τ0
G
p(yi ) = + τg φg (yi |μg , Σg ), (3.1)
V g=1
82 Dealing with Diﬃculties

−800
●
● ●
●
●
●
● ●
●
● ●
●

−900
●

● ●
● ●
●
●
−1000
●
●

●
●
●
BIC

−1100

●
−1200

●
●
EII EVE
VII VEE
● EEI VVE
−1300

● VEI EEV
● EVI VEV
● VVI EVV
● EEE VVV

1 2 3 4 5 6 7 8 9

Number of components

Figure 3.3 Simulated data with outliers: BIC plot for standard
model-based clustering.

where τ0 is the expected proportion of outliers in the data and V is the volume of
the data region. Estimation and model selection proceed as before.
Operationally, V can be defined as the volume of the smallest axis-aligned
hypercube containing the data. Another definition is the volume of the smallest
hypercube aligned with the principal components of the data set that contains the
data. The mclust software uses the minimum of these two values as its default
value.
It might seem at first sight that the result would be sensitive to the value of
the volume of the data region. However, in practice the sensitivity typically does
not seem to be great. This is probably because with the model (3.1), a point
is classified as an outlier approximately if the uniform density is greater than
the density under the nearest mixture component, and so the cutoff point will
usually be in the tail of one of the mixture densities. Because the Gaussian density
declines fast as one moves away from its mode, the precise location of this cutoff
point is not too sensitive to the height of the uniform density.
The estimation results can be quite sensitive to the initialization of the EM
algorithm, however, particularly the initial assignment of noise points. One auto-
matic way to initialize the outlier assignment is by the nearest neighbor cleaning
method of Byers and Raftery (1998). This essentially designates points as outliers
3.1 Outliers 83

if they are far from other points. It does this by approximating the non-outlier
distribution of the distance of a point to its Kth nearest neighbor by a gamma
distribution with shape parameter K (this is exact if the data are generated by
a homogeneous Poisson distribution). It then represents the distribution of Kth
nearest neighbor distances by a mixture of two components, one corresponding
to non-outliers and one to outliers, estimating the resulting mixture model, and
designating as outliers the points in the cluster with larger values of the Kth
nearest neighbor distance. The method is implemented in the NNclean function
in the prabclus R library (Hennig and Hausdorf, 2015).
If the variables are measured on the same scale, such as spatial coordinates or
scores on similar tests, it is not necessary to scale the data before applying the
nearest neighbor cleaning method. Otherwise it is usually a good idea to scale the
data first. This can be done, for example, by dividing each variable by its standard
deviation, or by a robust measure of scale. Otherwise, the result is not invariant
to rescaling of particular variables. A variable measured on a scale that gives it a
large variance may dominate the calculations, which would be undesirable.
For our simulated data, this analysis proceeds as shown in Listing 3.1, with
the results shown in Figure 3.4. The upper left panel shows the initialization
using nearest neighbor cleaning. This slightly overestimates the number of outliers,
identifying 25 outliers instead of the 20 that were simulated. However, this does
not matter much, as this is only the initial assignment, which is then refined by
the model-based clustering.
Figure 3.4(b) shows the BIC plot. This allows for the possibility of there being
no clusters, i.e. that the data set consists only of Poisson noise, shown in the plot
as “0 components.” This possibility is strongly rejected by BIC for the simulated
data. BIC chooses the equal-variance spherical model, EII, with two clusters and
noise. This is indeed the model from which the data were simulated. Figure 3.4(c)
shows the resulting classification. This correctly identifies the 20 outliers that were
simulated, including the one that was visually ambiguous, with no false positives.
Finally Figure 3.4(d) shows the uncertainty plot. Uncertainty was relatively
high on the periphery of the clusters, but 17 of the 20 simulated outliers were
identified with little uncertainty.
The nearest neighbor cleaning method tends to identify more points as outliers
than are actually present. This happened in the simulated data, where nearest
neighbor cleaning identified 25 outliers compared with 20 that were actually
simulated; the overestimation is often more severe. This usually does not matter
too much because it is being used only for initialization; the mixture model
estimation typically corrects things.
A modification of the nearest neighbor cleaning method that tends to identify
fewer false positives is the nearest neighbor variance estimation method (NNVE).
This adds simulated noise to the data before applying the nearest neighbor
cleaning method; perhaps surprisingly this tends to reduce the number of false
positives identified in the original data. It is implemented in the covRobust R
package, and its application to our simulated data is shown in Listing 3.2, with
the results shown in Figure 3.5.
84 Dealing with Difficulties

Listing 3.1: Outlier modeling for simulated data

# Loading libraries and data
library ( prabclus )

# Simulating data with noise

simdata1 <- sim ( modelName = " EII " ,n =180 ,
parameters = list ( pro = c ( .67 , .33 ) , mean = matrix ( c (1 ,1 , -1 , -1 ) ,
ncol =2) , variance = list ( sigmasq =0 .09 ) ) )
x1N <- rnorm (60 , sd =2) ; x2N <- rnorm (60 , sd =2)

# Remove noise points that are close to the cluster centers

dist1 <- rep ( NA ,60) ; dist2 <- rep ( NA ,60)
for ( k in 1:60) dist1 [ k ] <- sqrt (( x1N [ k ] -1 ) ^2 + ( x1N [ k ] -1 ) ^2)
for ( k in 1:60) dist2 [ k ] <- sqrt (( x1N [ k ]+1) ^2 + ( x1N [ k ]+1) ^2)
close <- dist1<1.4 | dist2<1.4
noise <- cbind ( x1N , x2N )
noise <- noise [! close ,]
Nnoise <- dim ( noise ) [[1]]
simdata <- rbind ( simdata1 [ , c (2 ,3) ] , noise )
n <- dim ( simdata ) [[1]]

# Plotting noisy data

plot ( simdata , type = " n " )
points ( simdata [1:180 ,])
points ( simdata [181: n ,] , col = " red " )

# Noise detection and clustering with Mclust

NNclean.out <- NNclean ( scale ( simdata ) , k =12)
MclustN <- Mclust ( simdata , initialization = list ( noise = (
NNclean.out $ z ==0) ) )

# Plotting results
plot ( simdata , col =1+ NNclean.out $ z )
mclust2Dplot ( simdata , parameters = MclustN $ parameters , z = MclustN $z ,
what = " classification " , PCH =20)
plot ( MclustN , what = " uncertainty " )

NNVE identifies fewer outliers than nearest neighbor cleaning, in this case 23
compared with 25. While this does not seem like a big difference, the number of
false positives is reduced from 5 to 3, i.e. by 40%. The final classification from the
mixture model is the same as when it is initialized with nearest neighbor cleaning.
The initial sets of outliers identified by nearest neighbor cleaning and NNVE both
include all the simulated outliers.

Example 6 (ctd.): Mineﬁeld identiﬁcation

We now return to the problem of minefield identification from an image with a
great deal of clutter. We showed the raw data and the true classification in Figure
1.3. This is a challenging problem for a number of reasons. The minefield follows
a highly nonlinear pattern, and so could not be represented by a single Gaussian
component, since bivariate normal distributions are concentrated around a line in
3.1 Outliers 85

Nearest Neighbor Cleaning

−800
● ● ●
● ● ●
● ● ●
● ● ● ● ●
● ● ● ● ● ●
●
2

●● ● ●
●
●●
●●●●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●●
●●●
●
●
●●
●●
●●
●
●
●●
● ●
●●
●
●●
●●
●
●
●●
●
●●
●●
●
●●●
●●●●
●●
●●
● ●
●●●●●
●●
●

−1200
● EII EVE
0

●●●●

BIC
●●●●
● ●● ● VII VEE
●
Y

●●
● ●●
●●●
●●●
●●●
●
●●
●
●
●
●
●
● ●
●
●●●●●
●
● ● ●
●●●●
● ●● ●
● EEI VVE
● ● ●
−2

● VEI EEV
●
● ● EVI VEV

−1600
●
● VVI EVV
●
−4

EEE VVV
● ●

−4 −2 0 2 4 6 0 1 2 3 4 5 6 7 8 9

X Number of components

Classification Uncertainty

●
●
●
●
●

●●
2

●●●●●●●●●
●
x2

● ●●
●● ●●
●
●●
●
● ●●●●
●● ●● ●●
●●●●
● ●
●●
● ●●●
● ●●●
●●
●● ● ●
●
●
●
●
●
−2

−2

●
●

●
−4

−4

−4 −2 0 2 4 6 −4 −2 0 2 4 6

x1 x1

Figure 3.4 Simulated data: Analysis using model-based clustering with

uniform noise component. (a) Initialization using nearest neighbor cleaning.
The initially identified outliers are shown as black circles. (b) BIC plot. (c)
Classification using model-based clustering. The finally identified outliers are
shown as green triangles. (d) Uncertainty plot.

two-dimensional space. Also, if one thinks of the clutter as outliers, their number
is large, consisting of over two-thirds of the points.
When we apply model-based clustering with no noise component to the data,
the best model according to BIC is the varying covariance model VVV with eight
clusters. The BIC plot is shown in Figure 3.6, with the BIC values for all models
in the left panel, and the BIC values for the VVV model with different numbers of
clusters in the right panel.
The resulting classification is shown in Figure 3.7. While the minefield appears
86 Dealing with Difficulties

Listing 3.2: Outlier modeling for simulated data using nearest neighbor variance
estimation
# Loading libraries
library ( covRobust )

# Robust covariance estimation and clustering ( using the simulated

data of the previous listing )
nnve.out <- cov.nnve ( scale ( simdata ) )
s i m d a t a . M c l u s t N . N N V E <- Mclust ( simdata , initialization = list ( noise
=( nnve.out $ classification ==0) ) )

# Plotting results
plot ( simdata , col =1+ nnve.out $ classification )
mclust2Dplot ( simdata , parameters = simdata.MclustN.NNVE $ parameters ,
z = s i m d a t a . M c l u s t N . N N V E $z ,
what = " classification " , PCH =20)

NNVE

●
●
●
●
●
2

● ●
● ●
● ●●
●● ●● ●
x2

●
Y

● ●
● ●●
● ● ●●●●●● ●●
● ●● ● ●
●●● ●
● ●●● ● ●
●●
● ● ●● ●
● ●●●● ●●
●
●
● ●
−2

−2

●
●

●
−4

−4

−4 −2 0 2 4 6 −4 −2 0 2 4 6

X x1

Figure 3.5 Outlier modeling for simulated data with nearest neighbor
variance estimation (NNVE): Left: initialization of assignment of outliers via
NNVE. Right: classiﬁcation by model-based clustering initialized with
NNVE.

as two clusters, the outliers are represented by six clusters that do not have any
reality. Thus, without a noise component, model-based clustering does not give
an interpretable result for these data.
The identification of outliers by nearest neighbor cleaning is shown in Figure
3.8. This does identify most of the outliers correctly, but it erroneously identifies
some clumps of outliers as part of the minefield. This is because, by chance, their
12th nearest neighbor distances are relatively small. Visually, it is fairly clear from
looking at the data as a whole that they do not belong to the main minefield, but
nearest neighbor cleaning takes a local rather than a global approach to the data
3.1 Outliers 87

●
●
−21600

−21600
● ●
● ● ● ●
● ●
● ● ●
● ●
● ●
● ● ●
● ● ● ●
● ●
●
● ● ● ●
● ● ● ●
● ● ●
−21700

−21700
● ● ● ● ●
● ● ●
●
● ● ●
●
● ●
● ● ● ●
●
● ●
BIC

BIC
● ● ●
●
●
● ●
−21800

−21800
●
●
●

EII EVE
−21900

−21900
VII VEE
● EEI VVE
● VEI EEV
● EVI VEV
● VVI EVV
● EEE VVV VVV

1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Number of components Number of components

Figure 3.6 BIC values for mineﬁeld data for model-based clustering with
no noise component. Left: all 14 models. Right: VVV model only.

and so cannot make this kind of inference. It provides a good initialization for
model-based clustering, which does take a global view of the data.
The BIC plot for model-based clustering with noise for the minefield data is
shown in the left panel of Figure 3.9. The model chosen is the variable-orientation
model, EEV, with two clusters, in which the volumes and shapes of the clusters
are the same. This means that there are two Gaussian clusters in addition to the
noise. The resulting classification is shown in the right panel of Figure 3.9.
The minefield here has a chevron shape, being concentrated around a piecewise
linear curve consisting of two lines that connect. Thus it cannot be well represented
by any one bivariate Gaussian distribution. This is because the bivariate Gaussian
distribution is in general concentrated around a line in two-dimensional space.
(When the covariance matrix is proportional to the identity matrix, the distribution
is distributed in a circular manner around a point in two-dimensional space.) The
model approximates the minefield by two clusters rather than one, each of which
is concentrated about a line. This illustrates the fact that a cluster concentrated
about a nonlinear curve can be approximated by a piecewise linear curve, which
can in turn be represented by several mixture components rather than one. The
model was extended to the situation where a single cluster is concentrated around
a smooth curve, represented by a principal curve, by Stanford and Raftery (2000).

The noise model gives 63 errors out of 1,104 data points, an error rate of 5.7%.
The errors are shown in the left panel of Figure 3.10, and it can be seen that
they are concentrated on the edge of the true minefield, which is where one would
expect any algorithm to have difficulties. The uncertainties are plotted in the
right panel of Figure 3.10, which indicates that the actual errors occurred where
the model said that the uncertainty was greatest.
88 Dealing with Difficulties

●
● ●● ● ●
● ● ●● ●
● ● ●●
● ●●
● ●● ● ● ● ● ●●● ● ●
● ● ● ●●
●
●
● ●
● ● ●● ● ● ●●
●● ●● ●
●● ● ●● ● ● ● ● ● ●
●
● ● ● ● ● ●
● ● ● ● ●● ● ●●
● ●
● ●●● ● ●
●●
● ●● ● ●●●
● ● ● ●● ● ●●
●

Figure 3.7 Classiﬁcation of the mineﬁeld data by model-based clustering

with no noise component.

It is possible to compare the best mixture models without and with noise using
BIC to provide an approximate Bayesian test of the presence of uniform noise.
For the mineﬁeld data, the BIC value for the best model without noise (the
unconstrained covariance VVV model with eight clusters) is −21, 538.79, while for
the best model with noise (the varying orientation model EEV with two clusters
and noise) it is −21, 127.80. Thus the model with noise has a BIC value that
improves on the model without noise by 411 points, a rather decisive diﬀerence.
This analysis also illustrates the fact that with the noise model, one can have a
high proportion of outliers, more than 50%, without the analysis breaking down.

3.1.3 Trimming Data with tclust

Another popular technique to deal with noisy data in clustering is the so-called
“tclust” method (Gallegos and Ritter, 2005; Garcı́a-Escudero et al., 2008). tclust
can be viewed as a robust version of the EM algorithm for mixture models.
The idea of this approach is to exclude observations identiﬁed as outliers when
estimating the model parameters.
The tclust model assumes that the data at hand contains nα “spurious” ob-
servations, α ∈ [0, 1], that one wants not to include when estimating the mixture
3.1 Outliers 89

NNclean classification

●
●● ●● ●
●
●● ●
●
●
●

Figure 3.8 Identiﬁcation of outliers in the mineﬁeld data by nearest

neighbor cleaning, for use as initialization in model-based clustering with
noise. The black dots represent the identiﬁed outliers.

● ● ● ●
● ●
● ● ● ●
● ● ● ●
● ● ●
● ●
● ● ●
−21150

● ● ● ● ● ● ●
● ●
● ● ●
● ●
● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●● ●
●
● ●
●
● ● ● ● ●
● ●
● ● ●
● ● ●
●
● ● ●
● ●
● ●

● ●
● ●
● ●
● ● ● ● ●
● ● ● ●
● ● ● ● ●
● ●
● ● ● ● ●
● ● ● ●
● ● ● ● ● ●
● ●
● ● ●
● ● ● ●
●
● ● ●
● ● ● ● ●
● ● ● ●
● ● ●
● ● ● ●
● ● ●
● ● ●
● ●
● ● ● ● ● ●
● ● ● ● ●
● ● ●
●● ● ●
●
● ●
● ● ● ● ● ● ●
● ● ●
● ●
● ● ● ● ● ● ●
● ● ●
−21200

● ● ● ●
● ● ●
● ● ● ● ● ●
●● ● ● ●
● ● ●
● ● ● ●
● ● ● ● ●
● ●

●
● ● ●
● ● ● ●
● ● ●
●
● ● ● ●
● ●
● ● ● ● ●
● ● ● ●
● ●
●
● ●
● ● ●

●
● ● ●
● ● ●

●
● ● ●
●

●
● ●
● ● ●
● ● ●● ● ● ●
● ●
● ● ● ●
● ●

●
● ● ●
● ● ● ● ●● ●
●
● ●
● ● ●

●
●● ●
●
●

●
● ● ● ●
● ●
● ●
● ● ●
●

●
● ● ● ● ● ●
●
● ● ●
● ● ● ● ● ●
● ● ●
● ● ●
● ● ● ●
● ● ● ● ●
● ● ●
●

●
−21250

●
● ● ● ●
●
● ● ● ● ●
● ● ●
● ●
● ● ● ●

● ● ● ●
●

●
●

● ●
● ● ● ●
● ●
BIC

● ●

●
● ●

●
● ● ●
● ● ● ● ● ● ●
●

●
●

●
● ● ●

● ● ●
● ● ●
● ● ● ● ●

● ●
● ● ● ● ●

●
● ● ●

●
● ● ●●
● ● ● ● ●

●
● ● ● ●
● ●

●
●

●
● ● ● ● ●
● ● ● ●
● ● ● ● ● ● ● ● ●

●
●
● ● ● ●

●
● ● ● ● ●

● ●
●

●● ●
● ● ●

●
● ●
● ● ● ● ●
●

●
● ● ● ●

●● ●
● ●
● ● ●
● ●
●
● ● ● ●

● ●
● ● ●
● ●

●● ●
●
● ● ● ● ●

●
● ●
● ● ● ● ● ●
−21300

●●● ●●
● ●

● ●●●
●
● ● ●

● ● ●●
● ● ●
● ● ● ●
●

●
● ●

●
● ●
●
●

● ● ●●
● ●

●● ●● ●
● ● ● ● ● ●
● ● ●

●● ●
●

●
●
● ● ● ●
●

●
● ●

●● ●●● ●●● ● ●●
● ● ●
● ● ● ● ● ●

●
●

●● ●●
●

●
● ●

●
●

● ●● ● ●● ●●●
● ● ●
●
● ●

●●
● ●

EII EVE
● ●

●●
●

● ●● ●● ●
●
● ● ●

●●●● ● ●
●
● ● ●
● ● ●
●

●
●

VII VEE ● ●●●● ● ●

● ●
● ●
●

●
●● ● ● ●
●
● ●

●●
● ●

●● ●●●● ●●
●
● ● ● ●

●
●

●●● ●● ● ●
●

● EEI VVE
●
−21350

● ● ●
●

● ●●
● ●
● ● ● ●

● ●●● ● ●● ●
●
● ● ● ● ● ●
●

●● ●●
●
● ●
● ● ● ●
● ●

● VEI EEV
● ● ● ●

●
● ● ● ● ●

●●
●

●●
● ● ●
●

●
● ● ●
●

● ● ●
● ● ●
● ● ● ●
● ●

●● ●●
● ● ●

● ●
●

● EVI VEV ● ● ● ● ● ●
● ●

●●●●
● ● ●
●
● ●
●
● ●
● ●
● ● ●
● ● ● ● ●
● ● ●

●●
● ● ● ●

●
●

VVI EVV
● ● ●
● ● ●
● ● ● ● ●
● ●● ●
● ● ● ● ● ● ●
● ● ●
● ●
● ● ● ● ● ● ● ●
● ● ●
● ● ●
● ● ● ● ●
● ● ● ● ● ●
● ●

EEE VVV
● ● ● ●● ●
● ● ● ●
● ● ●

●
●
● ●
● ● ● ● ●
● ● ●
●

0 1 2 3 4 5 6 7 8 9

Number of components

Figure 3.9 Analysis of mineﬁeld data by model-based clustering with

noise, initialized by nearest neighbor cleaning. Left: BIC plot. Right:
classiﬁcation. The outliers are shown by black dots.
90 Dealing with Diﬃculties

Classification Errors

● ● ● ●
● ●
● ● ● ●
● ● ● ●
● ● ●
● ●
● ● ● ● ● ●
● ● ● ● ●
●
● ● ●
● ●
● ● ●
● ● ● ● ● ●
● ● ● ● ● ● ● ●● ●
●
● ●
●
● ● ● ● ●
● ●
● ● ●
● ● ●
●
● ● ●
● ●
● ●

●●
● ● ●
● ● ● ● ●
● ● ● ● ●
● ● ● ●

●●
● ● ● ● ● ●●
●

●
● ● ● ●

●
● ●
● ● ●
● ● ●
● ● ● ●

● ●
● ● ● ● ● ● ● ● ●

●
●
● ● ● ●

●
● ● ● ● ●

●
●

●● ●
● ● ●

●
● ●
● ● ●

●●
● ●
●
● ● ● ●

●● ●
● ●
● ● ●

● ● ●
●
● ● ● ●

● ●
● ● ●
● ●

●● ●
●
● ● ● ● ●
● ●
● ● ● ● ● ● ●

●●● ●●
●

●●
●

●
●●
●
● ● ●

● ● ●●
● ● ●
● ● ● ●
●

●
● ●

●
● ●
●
●

● ●●
● ●

●● ●● ●
● ● ● ● ● ●
● ● ●

● ●
●

● ●
●
● ● ● ●
●

●●● ● ●●
● ●

● ●● ● ●●●
●●●
● ● ●
● ● ● ● ● ●

● ●● ●●●
●

●●●●●
● ●
●
● ●

●
● ● ●

● ● ● ●● ●●
●

●●
● ●

●●
● ● ● ●

● ● ●● ●●●● ●
●
●
● ● ●

● ●
●

● ● ●●●●
● ● ●
● ● ●
●
● ●

●
●

●● ●
● ●

● ● ●●
● ●
●

●●
● ●

●
●● ● ●
● ●

●●
● ●

●● ●●●● ●●
●
● ● ● ●

●
●

● ●
●

●●● ●● ●
●
● ● ● ●

●●
●

● ●●
● ●

●
● ● ● ●

● ● ●●● ● ●● ●
●
● ●

●● ●
● ● ● ●
●

●● ●●
●
● ●

●
● ● ● ●
● ● ● ●
● ●

●
●

●
● ● ● ●

●●
●

●
●

●
● ●

●
● ● ●
●

● ● ●
●

●
● ●
● ● ● ●
● ●

●● ●●
● ● ●

● ● ●
●●
●

●
● ● ● ● ● ●
● ●

●●●●
● ● ●
●
● ●

●●
●
● ●

● ●
● ●
●

●
● ● ● ●
● ● ● ●
● ●

●●
● ● ● ●
● ● ● ●
● ● ●

●
● ● ●

●●
● ●● ● ●
● ● ● ● ● ●
● ● ● ●
● ● ●

● ●● ●●
● ● ● ● ● ● ● ●
● ● ●
● ● ●
● ● ● ● ●

●● ●●
● ● ● ● ● ●
● ●
● ● ● ●● ●
● ● ● ●
● ● ●

●●
●
● ●
● ● ● ● ●
● ● ●
●

Figure 3.10 Errors and uncertainty in model-based clustering of mineﬁeld

data with noise. Left: errors; right: uncertainty.

model. To this end, let us introduce a set S of “spurious” observations which is of

cardinality nα. Then, the spurious mixture model is deﬁned through its likelihood:
⎡ ⎤
G
L(Y ; Θ) = ⎣ τg φ(yi ; μg , Σg )⎦ × fi (yi ) ,
yi ∈S̄ g=1 yi ∈S

where the densities fi are not necessarily Gaussian probability density functions.
Under these assumptions, Gallegos and Ritter (2005) and Garcı́a-Escudero et al.
(2008) show that mixture parameters and group memberships can be inferred by
just maximizing the “trimmed” log-likelihood:

G
(Y ; Θ, S) = log (τg φ(yi ; μg , Σg )) .
yi ∈S g=1

To do so, Garcı́a-Escudero et al. (2008) proposed the tclust algorithm for a

Gaussian mixture model which can be viewed as an EM-like algorithm with
additional trimming step. The tclust algorithm operates as follows at iteration s:
• E-step: for all observations, compute the expectation of the complete data
likelihood as usual, and compute and store the quantities:
di = max {D1 (yi ; Θ), ..., DG (yi ; Θ)} ,
where Dg is a discriminant function deﬁned as Dg (yi ; Θ) = τg φ(yi ; μg , Σg );
• T-step: the nα observations with the smallest di values are assigned to the set
S of spurious observations;
• M-step: the mixture parameters are updated based only on the non-spurious
observations, i.e. the observations which do not belong to S.
3.1 Outliers 91

Data with actual labels Tclust clustering

k = 2, α = 0.1

● ●

● ●
4

4
● ●

● ●

● ● ● ●
2

2
● ●
●
●

● ●
0

0
●
● ●
●
● ●
−2

−2
● ●
● ●
● ● ●

● ●
−4

−4
● ●

● ●
● ● ● ●

−4 −2 0 2 4 −4 −2 0 2 4

Figure 3.11 tclust for robust clustering. The left panel shows the simulated
data with the actual group label (outliers are the black circles). The right
panel displays the tclust solution, which gives results similar to the actual
group labels. Notice that the true values of G and α were provided to tclust
here.

Listing 3.3 shows a simple use of tclust on simulated data, using the tclust
package for R. Here the data are simulated from a mixture of two Gaussian
distributions and a uniform component. The percentage of data in the uniform
component (assumed to be the outlier group) is α = 0.1. The results are shown in
Figure 3.11.
As always, the actual number of groups and the percentage of spurious data are
not known in practical situations. It is therefore desirable to be able to automati-
cally choose appropriate values for G and α for the data at hand. Unfortunately,
as discussed by Garcı́a-Escudero et al. (2008), it is not valid to use classical model
selection criteria in this context. Instead, Garcı́a-Escudero et al. (2008) proposed
a heuristic which consists of examining the trimmed classification log-likelihood
curves and searching for the value of α for which the curves G and G+1 are similar
for larger values of α. Listing 3.4 presents code which allows the simultaneous
choice of G and α with tclust. The plot produced by the last commands is shown
in Figure 3.12.
The left panel of Figure 3.12 shows the curves of the trimmed classification
log-likelihood for G = 2, 3 and 4 against the value of α. The curve for G = 2
quickly joins the curves associated with larger numbers of groups. Following the
heuristic of Garcı́a-Escudero et al. (2008), one can conclude that an appropriate
value for G is 2, and that α should be between 0.07 and 0.1. The right panel gives
a better view of the relative differences between the curves for G = 2 and 3. It
92 Dealing with Difficulties

Listing 3.3: tclust for robust clustering

# Loading libraries
library ( tclust )
library ( MBCbook )

# Data simulation ( class 1 is for outliers )

n = 200; alpha = 0 .1
m1 = c ( -1 ,2) ; m2 = 3* c (1 , -1 )
S1 = 0 .25 * diag (2) ; S2 = 0 .25 * diag (2)
X = rbind ( mvrnorm ( n / 2 , m1 , S1 ) , mvrnorm ( n / 2 , m2 , S2 ) ,
matrix ( runif ( alpha * n *2 , -5 ,5) , ncol =2) )
cls = rep (3:1 , c ( n / 2 , n / 2 , alpha * n ) )

# tclust with the actual G and percentage of outliers

out = tclust (X , k =2 , alpha = alpha )

# Plotting results
par ( mfrow = c (1 ,2) )
plot (X , col = cls , pch = cls , xlab = ' ' , ylab = ' ' ,
main = " Data with actual labels " )
plot ( out , xlab = ' ' , ylab = ' ' , main = ' tclust clustering ')

Listing 3.4: Choice of G and α within tclust

# Clustering for various G and alpha ( using data of the previous
example )
out = ctlcurves (X , k =2:4 , alpha = seq (0 ,0 .2 , len =21) )

# Plotting results
par ( mfrow = c (1 ,2) )
plot ( out )
plot ( seq (0 , 0 .2 , len = 21) , out $ obj [1 ,] - out $ obj [2 ,] , type = 'b ' ,
xlab = ' alpha ' , ylab = ' ' , main = ' Difference between G =2 and G =3 ')
abline ( h =0 , lty =2)

appears that the curves join at around α = 0.09, which is a reasonable estimate
of the actual percentage of outliers (the actual value of α is 0.1).

3.2 Dealing with Degeneracies: Bayesian Regularization

A general problem with estimating Gaussian mixture models is that maximization
of the mixture likelihood (2.6) without any constraints could lead to a degenerate
solution. This is because as μg −→ yi and |Σg | −→ 0 for any observation i and
mixture component g, i.e. as the component mean approaches the observation
and the component covariance becomes singular, then the likelihood for that
observation becomes infinite and hence so does the whole mixture likelihood (2.6).
Thus in general there is at least one path in parameter space per observation
along which the likelihood tends to infinity (Day, 1969; Titterington et al., 1985).
The parameter values corresponding to these infinite likelihoods are not inter-
3.2 Dealing with Degeneracies: Bayesian Regularization 93

CTL−Curves Difference between G=2 and G=3

Restriction Factor = 50
−1500

3 ● ● ●

0
4
2 ● ● ● ● ● ● ● ● ● ●
●
3
4
2 ●
3
4
2
3
4
2
3
4
2
3
4
2 ●
4
3
2

−100
4
3
2
−2000

3
4
2
4
3
2
3
4
2 ●
4
3
2
Objective Function Value

4
3
2

−200
4
3
2
4
3
2 ●
−2500

4
3
3 2
4

−300
4
3
4 2 ●
3
4
3
2
4
−3000

−400
2 ●

−500
−3500

2 ●

0.00 0.05 0.10 0.15 0.20 0.00 0.05 0.10 0.15 0.20

α alpha

Figure 3.12 Choice of G and α with tclust. The left panel shows the curves
of the trimmed classiﬁcation log-likelihood for G = 2, 3 and 4 against the
value of α. On the right panel, the diﬀerence between the curves for G = 2
and 3 is displayed. Following the heuristic of Garcı́a-Escudero et al. (2008),
one can conclude that appropriate values for G and α are respectively 2 and
0.09.

esting in general, and are often spurious or degenerate solutions. They do not have
good properties as estimators. Redner and Walker (1984) showed that there is in
general a sequence of interior maximum likelihood estimates that is consistent, i.e.
that approaches the true values in the limit with high probability as the sample
size increases. But the spurious estimators do not have this good property.
In practice for estimation, this is not a big problem, because if the model is well
specified and the starting values are reasonable, it is unusual for the EM algorithm
to get trapped at a spurious solution. However, for model selection it is a bigger
problem. This is because model selection inevitably involves fitting models that
are bigger than the best fitting model, and so include mixture components that
are not well supported by the data. This is precisely the situation where spurious
solutions are most likely to be found.
Several solutions have been proposed. Hathaway (1985, 1986b) proposed maxi-
mizing the mixture likelihood (2.6) subject to the constraint there exists a constant
c such that all the eigenvalues of Σg Σ−1
h are greater than or equal to c for all pairs
of mixture components g, h. However, the resulting constrained maximization is
difficult.
The geometric constraints described in Section 2.2 provide a partial solution,
in that models that impose equal-volume constraints, such as EEE or EVV, do not
suffer from this problem. This is not a complete solution, however, because one
94 Dealing with Difficulties

might be interested in solutions involving models that do have spurious solutions

when the number of components is too large, such as the unconstrained covariance
model, VVV.
The default solution in mclust is to compute the ratio of the smallest to
the largest eigenvalue of the estimated covariance matrix for all the mixture
components, and not return any solution or BIC value for models for which this
value is below a user-specified threshold for any threshold (Fraley and Raftery,
2003). By default this threshold is taken to be the relative machine precision, which
is about 2 × 10−16 on IEEE-compliant computers. This avoids spurious solutions,
but leaves the problem that log-likelihoods or BIC values are not provided for
some models, leaving open the possibility that they might fit the data better than
the selected model. This is not likely in practice, because spurious solutions tend
to correspond to the presence of components whose separate existence is not well
supported by the data.
Nevertheless, any component with fewer than d points will tend to have a
singular covariance matrix, and hence produce an infinite likelihood, even if there
is a true cluster with fewer than d points, where d is the dimension of the data.
Geometric constraints on the component covariance matrices could resolve this,
but they may not be justified by the data. Thus the singularities might lead to
incorrect model specification if, for example, there are several clusters of which
one has fewer than d points, and geometric constraints on the covariance matrices
are not justified by the data.
A more general solution that avoids most of these difficulties is to adopt
a Bayesian approach, and instead of maximum likelihood, use a maximum a
posteriori (MAP) estimate (Fraley and Raftery, 2007a). Unlike what happens
with the likelihood, there are generally not paths in parameter space along which
the posterior density tends to infinity, with reasonably specified proper prior
distributions. This is because the prior distribution is multiplied by the likelihood
and tends to smooth or regularize these features, usually eliminating them.
Fraley and Raftery (2007a) used the standard normal-inverse Wishart prior
distribution for the component means and covariance matrices. Conditionally
on the cluster assignments, this is a conjugate prior, i.e. it yields a posterior
distribution with the same functional form (Gelman et al., 2013). They then
used the EM algorithm to find the posterior mode rather than the maximum
likelihood estimator (Dempster et al., 1977). We modify the definition of BIC in
(2.16) so that the maximized likelihood p(D|θ̂Mk , Mk ) is replaced by the likelihood
evaluated at the posterior mode.
For the mean μ and covariance matrix Σ of one component, the normal-inverse
Wishart prior distribution takes the following form. The prior distribution of the
mean given the covariance matrix is normal:

μ|Σ ∼ M V N (μP , Σ/κP ) (3.2)

1
κ
P
∝ |Σ|− 2 exp − tr (μ − μP )T Σ−1 (μ − μP ) . (3.3)
2
3.2 Dealing with Degeneracies: Bayesian Regularization 95

The prior distribution of the covariance matrix is inverse Wishart, namely

Σ ∼ inverseWishart(νP , ΛP ) (3.4)
νP +d+1
∝ |Σ| 2 exp − 12 tr Σ−1 Λ−1P . (3.5)
The parameters of the prior distribution, or hyperparameters, are denoted by
a subscript P. The hyperparameter κP is called the prior sample size, νP is
the number of degrees of freedom, and ΛP is the scale matrix of the Wishart
distribution. Explicit expressions for the different models were given by Fraley
and Raftery (2007a), while detailed derivations of the posterior distributions were
provided by Fraley and Raftery (2005).
Default hyperparameters were selected to have two desirable properties. One is
that the prior be spread out enough for inference to be almost unchanged from the
maximum likelihood inference when the EM algorithm converges to a non-spurious
solution. The second property we want is that the prior be concentrated enough
for spurious solutions to be avoided in most cases and for the BIC curve to
transition smoothly from models with non-spurious solutions, to models where
the EM algorithm converges to spurious solutions. The prior mean μP is taken to
be equal to the mean of the data. The prior sample size κP is taken to be 0.01.
The prior degrees of freedom νP is taken to be d + 2, which is the smallest integer
value that gives a prior distribution with finite covariance matrix. The prior scale
matrix ΛP is taken to be the empirical covariance matrix of the data divided by
G2/d .
The resulting posterior mode is usually such that when the local maximum of
the likelihood found by the EM algorithm is not spurious, the parameter values,
maximized likelihood and BIC value are relatively unchanged. When a spurious
local maximum is found, however, usually because the number of components is
larger than what is supported by the data, the sequence of BIC values tends to
decline smoothly, rather than jumping spuriously to infinity (Fraley and Raftery,
2007a).1

Example 2 (ctd.): Diabetes diagnosis

We saw in Section 2.5 that this data set is well fit by the VVV model with three
components; this is the choice made by BIC. A closer look, however, reveals that
for the VVV model, no BIC values were returned by mclust for G ≥ 5, i.e. for five
or more mixture components. This is because for these cases the EM algorithm
diverged to solutions with infinite likelihoods and so, by default, the results were
not reported. It therefore remains possible that a solution with 5 or more groups
would fit the data better.
To investigate this further, we removed the constraint forbidding the reporting
of results from spurious singular solutions, as shown in Listing 3.5. The control
parameter eps, which is the reporting threshold on the ratio of smallest to largest
eigenvalue of the covariance matrix of any component, was set to zero, so that
1 Currently, as of version 5.4, the mclust software implements the Bayesian method with a prior for
only 10 of the 14 geometrically constrained models.
96 Dealing with Difficulties

Listing 3.5: BIC for diabetes data with spurious singular solutions and with
prior
# Loading libraries and data
library ( MBCbook )
data ( diabetes )
X = diabetes [ , -1 ]

# Including spurious singular solutions :

Mclust (X , modelNames = " VVV " , control = emControl ( eps =0 , tol = c (1 e-3 ,1
e-3 ) ) )

# Including prior :
Mclust (X , prior = priorControl () , modelNames = " VVV " )

spurious singular solutions are reported. The tolerance for convergence of the EM
algorithm, specified by the control parameter tol, was relaxed to 10−3 instead of
the default 10−5 .
The result is shown in Figure 3.13. The log-likelihoods found by the EM
algorithm for five or more components diverged to infinity without regularization,
but finite values were found here because the algorithm stopped. These BIC
values are shown by dashed red lines and open circles in Figure 3.13; by default
they would not be shown. Non-spurious results are found only for four or fewer
components, and the corresponding results are shown by solid red circles. Using
the full red BIC curve in Figure 3.13 would lead to the erroneous conclusion that
there are at least nine components.
When the default prior and posterior mode are used, the result is shown in Figure
3.13 by the solid blue circles and lines. The BIC values are very close to those
for the default solution for four or fewer components, when the default solution
is non-spurious, as we would wish. For five or more components, the spurious
singular solutions are no longer found because of the Bayesian regularization, and
the BIC continues to decline in a smooth way. Thus using the prior confirms our
earlier conclusion that using three mixture components is best.
To illustrate the issues, we now consider the five-component solution. This is the
smallest number of components for which the EM algorithm yields a degenerate
solution, shown in the left panel of Figure 3.14. Note that the data do not support
five components, so this would not be a good scientific solution, and is shown
only for illustration. Cluster 4 has only three points, shown in black in Figure
3.14. The dimension of the data is d = 3, and so Cluster 4 has the same number
of points as there are dimensions, not enough to estimate the covariance matrix.
Thus the estimated covariance matrix for Cluster 4 is singular, with the ratio of
the smallest to the largest eigenvalue equal to zero, and determinant equal to zero
also. There is zero uncertainty about membership of cluster 4. This is unrealistic,
and is another consequence of the degeneracy. In the default mclust solution, this
result would not be reported.
The right panel of Figure 3.14 shows the classification using the maximum
3.3 Non-Gaussian Mixture Components and Merging 97

BIC ●

−4500
Singular BIC ●
●
BIC with prior

−4600
●
−4700

●
●
−4800
BIC

●
●
●
● ●
●
−4900

●
−5000
−5100

2 4 6 8

Number of components

Figure 3.13 Diabetes diagnosis data: BIC values for the unconstrained
covariance VVV model calculated in diﬀerent ways. The red ﬁlled circles show
the BIC values from the usual EM algorithm for 1 to 4 mixture components;
these correspond to non-spurious interior solutions. The red open circles and
dashed lines show the spurious computed BIC values for 5 through 9
components; these were obtained by relaxing the convergence tolerance. The
blue circles show the BIC values when a prior is used.

a posteriori estimates of the model parameters. Cluster 4 now has only two
points (again shown in black). However, the solution is no longer degenerate: the
covariance matrix for Cluster 4 has the ratio of smallest to largest eigenvalue
equal to 1.6 × 10−4 . While small, this is much larger than the reciprocal machine
infinity computed for the unregularized solution (2 × 10−16 ). Note that even with
the Bayesian regularization, the BIC strongly prefers three components to five, so
the five-component solution would not be used in practice.

3.3 Non-Gaussian Mixture Components and Merging

Model-based clustering is based on the idea of a ﬁnite mixture of multivariate
normal or Gaussian distributions, with each mixture component distribution
corresponding to one cluster. Sometimes, however, some of the clusters do not
have a Gaussian distribution. In this case, BIC will tend to choose a model that
represents a non-Gaussian cluster by a mixture of Gaussian mixture components.
98 Dealing with Diﬃculties

Diabetes data: Degenerate 5−cluster VVV solution Diabetes data: 5−cluster VVV solution with prior

● Cluster 1 ● Cluster 1
Cluster 2 Cluster 2
Cluster 3 Cluster 3
● Cluster 4 (singular) ● Cluster 4 (2 points)
Cluster 5 Cluster 5
600

600
400

400
sspg

sspg
● ●

● ●
●● ● ●● ●
● ● ● ●
●● ●●
● ●● ●●● ● ● ●● ●● ●●
200

200
● ● ●
●● ●●●● ●● ●●●● ●
●
●●● ●
● ●
●●● ●
●
●●● ●●●
● ●● ●●● ● ●● ●
● ●● ●● ● ● ●● ●●●●
●● ● ●● ●●● ● ●
●●
● ●●●● ● ●●
● ●●● ●● ●
●● ●●
● ●
●●● ●●● ●
0

0
0 500 1000 1500 0 500 1000 1500

insulin insulin

Figure 3.14 Five-cluster solutions for diabetes data using the

unconstrained covariance VVV model. Left: spurious degenerate solution from
EM algorithm. Cluster 4 (shown in black) has only three points and is
concentrated on a two-dimensional plane, and so is degenerate. Right:
maximum a posteriori solution. Cluster 4 has only two points, but its
estimated covariance matrix is not singular because of the Bayesian
regularization.

This will give a good estimate of the density, but will overestimate the number of
clusters.
On the other hand, ICL will tend to select a number of mixture components
that corresponds to the number of clusters. However, in so doing it typically
represents a non-Gaussian cluster by a single Gaussian mixture component, which
may not be a good fit to the data. The difficulty is illustrated by the Old Faithful
data, as shown in Figure 3.15.
As we saw in Section 2.6, BIC chooses three mixture components, as illustrated
in the top panels of Figure 3.15. The top left panel suggests that, while the mixture
component in red at the bottom left of the plot is cohesive and well-separated
from the rest of the data, the two components in blue and green at the top right
part of the plot are not very separate from one another, and may in fact make
up a single cluster. The density estimate in the top right panel reinforces this. It
looks as if there are really two clusters, not three, but the one at the top right
has a non-Gaussian distribution and is itself being modeled by a mixture of two
Gaussian distributions.
As seen in the bottom left panel of Figure 3.15, ICL chooses two mixture
components. This is both good and bad. Good because there do indeed seem to
be two clusters, and hence the number of components chosen corresponds to the
number of clusters. But bad because the cluster at the top right of the plot is
represented by a single Gaussian in the ICL solution, which may not be a very
good fit to the data.
3.3 Non-Gaussian Mixture Components and Merging 99

BIC−best classification BIC−best density

●
●
● ●
●
●
90

90
● ●● ●●●
● ●
● ● ● ● ●● 0.02
●
●● ● ● 0.03
● ●●●
●
● ●●● ●● ● 0.04
● ● ●●● ● ●
●●●● ●● ●
● ● ● 0.045
● ●● ● ●● ● ●● ●
80

80
● ● ●● ●
● ●● ● ● ●●
● ●● ● ● ●● ●● ●●
●● ● ● ● ● ● ● 0.035
●● ● ● ● ● ●
●● ● ●
●●
● ● ●● 0.025
● ● ●● ●
● 0.015
waiting

waiting
0.01
70

70
●●
●
0.005

0.005

0.01
60

60
0.02

3
0.0

35
0.0
50

50
0.02
5

0.015

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

ICL−best classification ICL−best density

●
●
● ● 0.005
●
●
90

90
● ●● ●●●
● ● ● 0.015
● ● ● ● ●●
● ●
● ●● ● ● 0.025
● ● ● ●●●
●● ●● ●●● ●● ● 0.035
● ● ● ● ● ●●● ● ● 4
● ● ●●●● ●● ●
● ● ● 0.0
● ● ● ●● ● ● ● ● ●● ●
80

● ● ● ● ●● ●
● ●● ● ●● ● ● ●●
● ● ● ● ●● ●
● ● ●● ●● ●●
●● ●● ● ● ● ● ● ●
●● ●● ● ● ● ● ●
● ● ●
● ● ●●
● 0.03
● ● ● ● ●●
● ● ● ●● ● 0.02
●
waiting

waiting

● ● ● ●
70

● ● ●● 0.01
● ●
●
● ● 0.005
● ●
0.015
60

0.025
5
0.03

3
0.0
50

0.02

0.01

1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0

eruptions eruptions

Figure 3.15 Old Faithful data: Top left: classiﬁcation by the BIC-best
model with three components. Top right: corresponding density estimate.
Bottom left: classiﬁcation by the ICL-best model with two components.
Bottom right: corresponding density estimate.

A widely used solution to this dilemma is to keep the BIC solution as a model for
the density of the data, but to merge mixture components that are close together
before clustering. This has the potential to give the best of both worlds: the right
number of clusters, but a ﬂexible model for each of them that ﬁts the data well.
As summarized by Hennig (2010), methods for merging Gaussian components
proceed as follows:

1 Start with all components of the initially estimated Gaussian mixture as current
clusters.
2 Find the pair of current clusters most promising to merge.
100 Dealing with Diﬃculties

3 Apply a stopping criterion to decide whether to merge them to form a new

current cluster, or to use the current clustering as the final one.
4 If merged, go to 2.
Different choices of initially estimated Gaussian mixture, different methods
for finding the most promising pair of components to merge at each stage, and
different stopping criteria lead to different merging methods. The model selected
by BIC is often used as the basis for the initially estimated Gaussian mixture.
Methods proposed differ mainly in the selection of pairs for merging and the
stopping criterion.
One approach is to base these on the entropy of the resulting classification
[G]
(Baudry et al., 2010). Let ẑi,g denote the conditional probability that observation
i arises from cluster g = 1, . . . , G, based on the G-cluster solution. The corre-
sponding estimated soft or probabilistic classification, defined by (2.9) at the
converged solution for G clusters, has entropy

G
n
Ent(G) = − ẑi,g log(ẑi,g ) ≥ 0.
[G] [G]
(3.6)
g=1 i=1

[G]
For a classification with no uncertainty, i.e. if ẑi,g is equal to 0 or 1 for all i and g,
[G]
Ent(G) = 0. The entropy is maximized when ẑi,g = 1/G for all i and g, i.e. when
the classification is effectively uninformative.
The basic idea is to choose the pair of clusters to merge at each stage that
minimize the increase in entropy over all pairs. The calculations are facilitated
by the fact that there is a simple relationship between the posterior cluster
membership probabilities of a point for any two clusters, and the corresponding
value for the merged cluster: the latter is just the sum of the two former. Merging
continues as long as the resulting increase in entropy is small, and stops when
a merge leads to a large increase in entropy. Using an elbow or change point in
the plot of the entropy against the number of observations has worked well for
deciding when to stop. This approach is implemented in the clustCombi function
in the mclust package.
If clusters g and g from the G-cluster solution are combined, the values of ẑi,h
remain the same for every group h except g and g . The new cluster g ∪ g then
has the following conditional probability:
[G] [G] [G]
ẑi,g∪g = ẑi,g + ẑi,g . (3.7)
Then the resulting entropy is

n [G]
−
[G] [G] [G] [G] [G]
ẑi,g + ẑi,g log ẑi,g + ẑi,g + ẑi,h log(ẑi,h ) . (3.8)
i=1 h=g,g

Thus, the two clusters g and g to be combined are those that maximize the
criterion
n n
−
[G] [G] [G] [G] [G] [G]
ẑi,g log(ẑi,g ) + ẑi,g log(ẑi,g ) + ẑi,g∪g log(ẑi,g∪g ) (3.9)
i=1 i=1
3.3 Non-Gaussian Mixture Components and Merging 101

among all pairs of clusters (g, g ). Then the conditional classiﬁcation probabilities,
ẑi,g , i = 1, . . . , n; g = 1, . . . , G − 1, can be updated.
[G−1]

At the ﬁrst step of the combining procedure, G is the number of components

[G]
selected by BIC, and ẑi,g is the conditional probability that observation i arose
from the gth mixture component. But as soon as at least two components are
[G]
merged into a cluster g, so that G is smaller than the value selected by BIC, ẑi,g is
the conditional probability that the ith observation belongs to one of the merged
components in cluster g.
Hennig (2010) has proposed several other criteria within the same general frame-
work. These include the Bhattacharyya distance between two cluster densities,
directly estimated misclassification probabilities (DEMP), prediction strength
(Tibshirani and Walther, 2005), two versions of the ridgeline method of Ray and
Lindsay (2005) to prohibit merging if the resulting merged cluster has a density
that is not unimodal, and the method of Tantrum et al. (2003) that is based on
the dip test of Hartigan and Hartigan (1985) for unimodality. In a simulation
study comparing these methods, Hennig (2010) found the Bhattacharyya distance
method to have good performance across a wide range of settings. It is also com-
putationally cheap. The prediction strength method also had good performance,
but was by far the most computationally expensive of the methods considered.
These criteria are implemented in the fpc R package (Hennig, 2015a).
We illustrate the methods by applying them to a simulated data set shown in
Figure 3.16. These data were used in Baudry et al. (2010, Section 4.1) and are
available in the mclust package. Visually, this data set appears to contain four
well-separated clusters. However, the clusters in the top left and bottom right of
the plot are far from Gaussian, and instead appear to follow a cross-like pattern.
The BIC solution chooses six mixture components. It divides each of the two
cross-like clusters into two mixture components, representing each cluster by a
mixture of two Gaussian distributions. The ICL solution (not shown here) chooses
four clusters, which is the right number. However, it represents each of the two
cross-like clusters by a nearly spherical Gaussian, which clearly is not a very good
fit to the data.
The entropy values for different numbers of clusters from the entropy merging
method of Baudry et al. (2010) are shown in Figure 3.17. The left panel shows
how the entropy changes with the number of clusters, starting with one cluster
on the left of the plot (with zero entropy by definition), and going to six clusters
on the right of the plot. The entropy is plotted against the cumulative sum of the
number of observations merged at each step. On this scale, the plot is expected
to increase slowly up to the true number of clusters, and then to increase more
quickly and roughly linearly beyond that. The red dotted line comes from a fitted
linear change-point model. The change-point clearly occurs at G = 4 clusters,
which is the number indicated by visual inspection.
The right panel of Figure 3.17 shows the same information in a different form,
namely the differences in entropy in going from G to G + 1 clusters, normalized
by the number of observations in the merged clusters. We see that the difference
102 Dealing with Difficulties

BIC solution (6 clusters)

●
● ●
●
●
●●
● ●
●●
●● ●● ●●
● ●●●● ●
● ● ● ●●●
●
● ● ●
●● ● ●
6

6
● ●● ●
● ●●●●●●
● ●●●● ●
●● ● ●●●● ● ● ●
● ●●
● ● ●●
●●● ● ● ●
● ● ●●●●●● ●
● ● ● ●●● ● ● ● ●● ●
● ●●
●● ●●● ●● ●●
● ● ● ●● ●
●●
●●● ●●
●●●●
● ●
●●●●● ● ● ● ●●●●
●● ●●
●●●
● ●●●● ●
●●●● ● ●●
● ●●● ● ● ●● ● ● ● ●
●●●
●
● ● ● ●●●● ● ● ● ●● ●●
● ●● ●●● ●●●
● ●● ●●
●●●●● ●●●●● ●●
● ● ●●●●●●
●●
● ● ●●●●●●
● ● ● ●●● ●● ●
● ● ●
●
●● ● ●●
●● ● ●● ●
●
●●
●●●
● ● ●
● ●● ● ●
● ● ●●●● ● ● ● ●●●
● ●●●
● ●●
● ● ●●●●●
●
● ●●●
4

4
●
●● ● ● ●
●
● ●● ● ● ●
● ● ●●
●
● ● ● ●
X2

X2
●
● ●●
● ●
● ● ● ●
2

0
●● ● ●
● ● ● ● ●●● ●● ● ●●● ●● ● ● ●●● ●● ● ●
●● ● ●
● ●● ● ● ●● ● ● ●●●
● ● ●●
●●● ●●●●
●● ● ●
●●
● ●● ●
● ●●
●● ●●● ● ● ● ●
●●
●● ● ●● ●● ● ●● ●
●● ●●●● ● ●
●
● ● ●
● ●● ● ●
● ●● ●●●●● ●
● ● ●● ● ●
● ● ●
● ●
●
● ●● ●
−2

−2
● ●
●

−2 0 2 4 6 8 10 −2 0 2 4 6 8 10

X1 X1

Figure 3.16 Simulated data (left) and BIC-best solution with six mixture
components (right).

Listing 3.6: Cluster merging by entropy method for simulated data

# Loading libraries and data
library ( mclust )
data ( B a u d r y _ e t a l _ 2 0 1 0 _ J C G S _ e x a m p l e s )

# Clustering and merging

output <- clustCombi ( data = ex4.4.1 )

# plots the hierarchy of combined solutions , then some " entropy

plots " which may help one to select the number of classes
plot ( output )

in entropy is small in going from one to two clusters, from two to three clusters
and from three to four clusters. However, going from four to ﬁve clusters involves
a big increase in entropy, again suggesting that merging down to four clusters
improves the entropy substantially, but merging beyond that does not bring big
improvements. Overall, this suggests that four clusters is the best choice.
Figure 3.18 shows the results of merging mixture components, as one goes from
the BIC solution of six clusters down to two clusters. The selected four-cluster
solution combines the components that make up the cross-like clusters, as desired.
The number of clusters is the same as that selected by ICL, but the resulting
densities are diﬀerent. In the ICL solution, each of the two cross-like clusters is
represented by a single, nearly spherical Gaussian distribution, which is clearly not
fully satisfactory. In the merged four-cluster solution, each of the two cross-like
clusters is itself represented by a mixture of two Gaussian distributions, retaining
the true, cross-like nature of the resulting clusters. Figures 3.17 and 3.18 were
produced by the code in Listing 3.6.
3.3 Non-Gaussian Mixture Components and Merging 103

Normalized entropy plot

● ●
●
120

0.3
100

Normalized difference in entropy

0.2
Entropy

●
40

0.1
20

● ●
0.0

● ● ● ● ●
0

0 500 1000 1500 1 2 3 4 5

Cumul. count of merged obs. Number of clusters

Figure 3.17 Simulated data. Left: entropy values for the G-cluster
combined solution, using the entropy merging method of Baudry et al.
(2010), plotted against the cumulative sum of the number of observations
merged at each step. The dashed line shows the best piecewise linear ﬁt,
with a breakpoint at G = 44 clusters. Right: rescaled diﬀerences between
successive entropy values, equal to (Ent(G + 1)−Ent(G))/ Number of
merged observations.

We now apply the method of Hennig (2010) where clusters are merged until the
Bhattacharyya distance between all remaining pairs of clusters is greater than a
user-specified threshold. It is implemented using the code in Listing 3.7 and gives
the results shown there. It also selects four clusters, as do the DEMP method, and
the dip test method. The ridgeline methods, however, select five clusters, which
seems less satisfactory.
We now return to the Old Faithful data, where BIC selected three mixture
components but visual inspection suggests two clusters. All the merging methods
we have discussed here point to merging the two mixture components at the top
right of the top left panel of Figure 3.15, as desired. ICL selects three clusters,
each represented by a single Gaussian, as shown in the bottom right panel of
Figure 3.15, while the merging solution has the density shown in the top right
104 Dealing with Difficulties

Combined solution with 5 clusters Combined solution with 4 clusters

6
4

4
X2

X2
●●
●
●
2

2
●
●●
●
● ● ●
●●
●● ●●●●
● ● ● ● ● ● ● ● ● ● ●● ●
● ● ● ● ●● ● ●●●
●● ●●●
● ● ● ● ● ●● ● ●
●●●●●●
●●●
● ● ● ● ● ● ●●●● ● ● ● ● ● ● ●●●
● ●● ● ● ●● ●● ●●
● ● ● ●● ● ● ●● ●●● ●● ●
0

0
● ● ●●● ●● ● ● ● ● ●●● ●● ● ●●
● ● ●● ●●●
● ● ●● ● ● ● ●●
● ● ●●
● ●●
● ● ●● ●
●● ●●● ● ● ● ● ●● ●
●● ● ●● ●● ●
●
●
● ●
●●●●● ●

●
●
−2

−2
●

−2 0 2 4 6 8 10 −2 0 2 4 6 8 10

X1 X1

Combined solution with 3 clusters Combined solution with 2 clusters

●
●
●

●●
● ● ●●●
● ●
6

6
● ●● ●
●●●● ●
●
● ●●
● ●●●●●● ●
●
●
●●●●
● ● ●● ●
● ●●●●
●●
●● ●●
● ● ●● ●●
●●
●●●●
●●●●●●●●●
●●●
●●●
● ● ●
● ● ●●●
● ●●●
●●●●
●●
● ●●●
4

● ●
● ●
●●
● ● ●
X2

●● ●●
● ●
● ●
2

● ●
●● ●●
● ●
● ● ● ● ● ●
●● ●●
●● ●●●● ●● ●●●●
● ● ● ● ● ●● ● ● ● ● ● ● ●● ●
● ● ● ● ●●● ●●●●●
●● ● ● ● ● ●●● ●●●●●
●●
● ● ● ● ●●
●● ● ●● ● ● ● ● ●●
●● ● ●●
● ●● ● ● ● ● ●●●●
● ● ● ●● ● ● ● ● ●●●●
● ●
●● ●● ●● ●●
0

● ● ●
●●● ●● ●● ●●●
● ●●
●
● ● ● ●
●●● ●● ●● ●●●
● ●●
●
●
●● ●● ●●● ●● ● ●
●● ●● ●●● ●● ● ●
●● ●●
● ●
● ●
● ● ● ●
●●●●● ● ●●●●● ●

● ●
● ●
−2

−2

● ●

−2 0 2 4 6 8 10 −2 0 2 4 6 8 10

X1 X1

Figure 3.18 Simulated data: results from merging for 5, 4, 3 and 2 clusters.

panel. Thus the merging solution seems to provide a better representation of the
apparently non-Gaussian cluster at the top right of the data.
We have found that, even if ICL selects a model that may not be fully satisfactory
because it represents each cluster by a single Gaussian distribution, even when
the cluster is non-Gaussian, it tends to do well in selecting the right number of
clusters (as opposed to the right number of mixture components). Thus a hybrid
approach in the case where BIC selects more mixture components than ICL would
be as follows: use ICL to select the number of clusters, use BIC to select the
number of mixture components and the covariance model, and select the ﬁnal
clustering model by merging the components as described here, until the number
of clusters selected by ICL is reached.
3.4 Bibliographic Notes 105

Listing 3.7: Cluster merging by Hennig–Bhattacharyya method for simulated data

# Loading libraries and data
library ( mclust )
library ( fpc )
data ( B a u d r y _ e t a l _ 2 0 1 0 _ J C G S _ e x a m p l e s )

# Clustering and merging

ex4.1.Mclust <- Mclust ( ex4.1 )
ex4.1.bhat <- mergenormals ( ex4.1 , method = " bhat " , clustering =
ex4.1.Mclust $ classification , probs = ex4.1.Mclust $ parameters $ pro
, muarray = ex4.1.Mclust $ parameters $ mean , Sigmaarray =
ex4.1.Mclust $ parameters $ variance $ sigma , z = ex4.1.Mclust $ z )

# Displaying results
summary ( ex4.1.bhat )

Method : bhat , cutoff value : 0 .1

Original number of components : 6
Number of clusters after merging : 4
Values at which clusters were merged :
[1 ,] 5 0 .55978081
[2 ,] 4 0 .55617411
[3 ,] 3 0 .01132068
Components assigned to clusters :
[1 ,] 1
[2 ,] 2
[3 ,] 3
[4 ,] 3
[5 ,] 4
[6 ,] 1

3.4 Bibliographic Notes

Outliers
The model consisting of a mixture of a multivariate normal distribution with
a uniform distribution was mentioned and used by Titterington et al. (1985)
to model a data set, but not for identifying outliers. The model consisting of a
mixture of several multivariate normal distributions with a uniform distribution
was proposed by Banﬁeld and Raftery (1993) and Dasgupta and Raftery (1998)
for identifying and removing or down-weighting outliers, and further developed
by Fraley and Raftery (1998, 2002). The robustness properties of the model were
studied by Hennig (2004), in particular its breakdown point (Hampel, 1971).
Hennig (2004) showed that the method is theoretically not breakdown-robust,
although in practice very extreme outliers are needed to spoil it. It has been
used in many applications, including in a large-scale image processing context by
Horaud et al. (2011). It was extended to accommodate clusters centered around
curves rather than lines in the bivariate case by Stanford and Raftery (2000),
using the concept of principal curves.
The nearest neighbor cleaning method for identifying outliers, used here to
106 Dealing with Diﬃculties

initialize model-based clustering with noise, was introduced by Byers and Raftery
(1998). It was extended to robust covariance estimation by Wang and Raftery
(2002), introducing simulated noise points to reduce bias.
Several alternatives to the uniform noise model for outliers have been proposed.
Hennig (2004) proposed replacing the uniform distribution here by an improper
uniform component, an idea further developed by Hennig and Coretto (2008);
Coretto and Hennig (2010, 2011). An approach based on iteratively trimming
potential outliers was proposed by Garcı́a-Escudero et al. (2008, 2010, 2011); it
is implemented in the tclust R package (Fritz et al., 2012). Evans et al. (2015)
have proposed a method for outlier detection in model-based clustering based on
leave-one-out re-estimation of cluster parameters.
The normal-uniform mixture model for outliers has also been used in other
contexts. For example, Dean and Raftery (2005) used it as the basis for a method for
detecting differentially expressed genes in microarray data. There the differentially
expressed genes were analogous to the outliers and were modeled with the uniform
distribution, while the normal distribution was used to model the non-differentially
expressed genes.
An alternative approach is based on replacing the Gaussian mixture components
by t-distributions (Peel and McLachlan, 2000). This does not aim to identify
and remove outliers and then identify compact clusters, but rather to model
mixture components that may include outlying data points, and so its goal is
rather different from the other methods discussed in this chapter. We discuss it in
more detail along with more recent extensions in Chapter 9.

Degeneracy
The fact that the likelihood for normal mixture models has multiple infinite modes
that are not interesting was first pointed out by Day (1969) in the one-dimensional
case and further elucidated by Titterington et al. (1985). Redner and Walker
(1984) showed that interior local maxima of the likelihood can define estimators
that are consistent, but that the spurious infinite maxima on the edge of the
parameter space do not.
Methods for dealing with this problem can be classified as constraint methods,
Bayesian methods, penalty methods, and others. Hathaway (1985) was the first
to propose a constraint method. He proposed maximizing the likelihood subject
to the constraints that the eigenvalues of Σg Σ−1
h be greater than or equal to some
minimum value c > 0, and developed an algorithm for the univariate case (Hath-
away, 1986b). Ingrassia and Rocci (2007) showed how the Hathaway constraints
can be implemented directly at each iteration of the EM algorithm. Gallegos and
Ritter (2009b,a) proposed different, but conceptually similar, constraints based
on the Löwner matrix ordering, essentially that no group covariance matrix differs
from any other by more than a specified factor.
Fraley and Raftery (2003) proposed computing the condition number for each
estimated covariance matrix and not reporting the results for any covariance
3.4 Bibliographic Notes 107

model and number of components for which any of the condition numbers is zero
(or below relative machine precision); this is the default approach in mclust.
Garcı́a-Escudero et al. (2015) proposed a constraint on the ratio of the maximum
to the minimum of all the eigenvalues of all the mixture component covariance
matrices. This controls differences between groups and singularity of individual
components at the same time. They described an algorithm for implementing
it. Cerioli et al. (2018) proposed a refinement of this method that modifies the
likelihood penalty for model complexity to take account of the higher model
complexity that a larger constraint entails.
In addition to Fraley and Raftery (2007a), several other Bayesian approaches
have used the EM algorithm to estimate the posterior mode for mixture models.
Roberts et al. (1998) used a Dirichlet prior on the mixing proportions and a non-
informative prior on the elements of the means and covariances, while Figueiredo
and Jain (2002) used non-informative priors for all the parameters, and Brand
(1999) proposed an entropic prior on the mixing proportions. These methods work
by starting with more components than necessary, and then pruning those for
which the mixing proportions are considered negligible.
Chi and Lange (2014) proposed a prior on the covariance matrices that dis-
courages their nuclear norm (sum of their eigenvalues) from being too large or
too small. It penalizes large values of a weighted average of the nuclear norm of
Σg and the nuclear norm of Σ−1 g , leading to a method for covariance regularized
model-based clustering. Zhao et al. (2015) proposed a hierarchical BIC using the
posterior mode based on an inverse Wishart prior for the covariance matrices and
non-informative priors for the component means and mixing proportions.
Bayesian estimation for mixture models can also be done via Markov chain
Monte Carlo (MCMC) simulation, typically using conjugate priors on the means
and covariances similar to those of Fraley and Raftery (2007a) (Lavine and West,
1992; Bensmail et al., 1997; Dellaportas, 1998; Bensmail and Meulman, 2003; Zhang
et al., 2004; Bensmail et al., 2005). These can suffer from label-switching problems
(Celeux et al., 2000; Stephens, 2000b) and can be computationally demanding.
In our experience, use of the posterior mode is simpler and less computationally
expensive, provides most of the benefits of the Bayesian approach, and avoids
most of the difficulties.
Ciuperca et al. (2003) and Chen and Tan (2009) proposed maximum penalized
likelihood methods to overcome the unboundedness of the likelihood, with purpose-
built penalty functions. Ruan et al. (2011) and Bhattacharya and McNicholas
(2014) proposed using 1 penality functions. Halbe et al. (2013) proposed an EM
algorithm with Ledoit–Wolf-type shrinkage estimation of the covariance matrices
(Ledoit and Wolf, 2003, 2004, 2012).
Rather than dealing with the problem algorithmically, McLachlan and Peel
(2000) proposed monitoring the local maximizers of the likelihood function and
carefully evaluating them. Seo and Kim (2012) and Kim and Seo (2014) proposed
a systematic algorithmic approach to selection of the best local maximizer to
avoid spurious solutions.
108 Dealing with Difficulties

Non-Gaussian Clusters and Merging

One of the earliest authors to propose merging mixture components for clustering
was Li (2005), who assumed that the true number of clusters is known in advance,
unlike the other methods discussed here. She used k-means clustering to merge
components, which works well when the mixture components are spherical, but
may have problems when they are not. This method was compared with the
entropy merging method by Baudry et al. (2010). A related method based on
finding modes was proposed by Li et al. (2007a).
All the methods discussed here are effectively multilevel clustering methods,
in which each cluster can itself be represented by a mixture model. Jörnsten
and Keleş (2008) proposed a multilevel mixture model that provides parsimony
in modeling both the clusters and the relationships between them, allowing a
different number of clusters at each level of an experimental factor of interest.
Various criteria for finding clusters from mixture models have been proposed,
other than the ones described here. Longford and Bartošová (2014) proposed a
confusion index, Zeng and Cheung (2014) used a minimum message length criterion,
while Scrucca (2016b) proposed a method for finding connected components based
on the density estimate from the model-based clustering itself. Naim et al. (2014)
proposed a merging method based on a Kullback–Leibler divergence criterion,
while Mosmann et al. (2014) applied it to flow cytometry data.
The problem of merging mixture components has also arisen in the context
of Hidden Markov Models (Ackerson and Fu, 1970; Harrison and Stevens, 1971;
West and Harrison, 1989; Volant et al., 2014). There sequential updating using,
for example, the multiprocess Kalman filter, leads to a posterior distribution that
is a finite mixture, but with a number of components that explodes exponentially
as time proceeds. There is a substantial literature on how to approximate the
resulting mixture with a more manageable form. One approach is to reduce the
number of mixture components, approximating a mixture with many components
by one with fewer components. Recent contributions include the merging algorithm
of Runnals (2007), the joining algorithm of Salmond (2009), and the variational
Bayes approach of Bruneau et al. (2010). Note that this is very different from the
goal of clustering that we are discussing here.
A more direct approach to model-based clustering with non-Gaussian clusters
would be to use mixture models with non-Gaussian component distributions.
The merging approach relies on the fact that one can approximate a continuous
non-Gaussian distribution arbitrarily closely by a finite mixture of Gaussian dis-
tribution, and that model-based clustering with Gaussian distributions is fairly
mature. Model-based clustering with non-Gaussian distributions is developing
rapidly, and we discuss it in more detail in Chapter 9. The choice raises philosoph-
ical as well as technical issues, and some of these have been discussed by Hennig
(2013, 2015b).
4
Model-based Classification

Supervised classiﬁcation, also known as discriminant analysis, is an important

domain of supervised learning which consists of predicting the values of categorical
outputs from the values of a set of input variables. In order to construct a predic-
tion rule, a training data set (y1 , z1 ), . . . , (yn , zn ) of n iid labeled observations is
available, where yi ∈ Rd are the inputs and the labels zi ∈ {0, 1}G for i = 1, . . . , n
with G known are the outputs. In particular, zig = 1 if observation belongs to
group g, where g = 1, . . . , G, and zig = 0 otherwise. This training data set is used
to classify new observations y into one of the G classes.
Thus, supervised classification is the set of methods aiming to predict the output
of an individual based on its values for the d predictors. The precise purpose of
supervised classification depends on the context. In most cases, good prediction
is of the greatest interest, and the aim is to minimize the misclassification error
rate. But in some cases, providing a relevant interpretation of the G classes from
the predictors is of specific interest. These two goals could be in conflict in some
cases.
Examples of popular and successful classification methods not based on a
probabilistic model are kernel methods. But here naturally, we concentrate on
model-based methods.

4.1 Classiﬁcation in the Probabilistic Framework

In the probabilistic framework, assumptions are made on the joint distribution
p(yi , zi ) or on the conditional distribution p(zi |yi ) of the label zi knowing the d
predictors yi = (yi1 , . . . , yid ). Then, assuming equal misclassification costs, the
optimal Bayes classifier consists of assigning a unit i to the class g by maximizing
the conditional probability P (zig = 1 | yi ) that this unit belongs to class g knowing
its observed vector yi .
Different approaches are possible to approximate P (zig = 1|yi ). Predictive
or discriminative approaches estimate these conditional probabilities directly
using a parametric model such as logistic regression (Hosmer et al., 2013), or a
nonparametric approach such as classification trees (Breiman et al., 1984), and
related methods such as random forests (Breiman, 2001). Generative methods
derive the conditional probabilities P (zig = 1|yi ) from the class densities fg (yi )
109
110 Model-based Classification

τ Z

Y μ, Σ

Figure 4.1 Graphical representation of the LDA and QDA models.

through the Bayes formula:

P (zig = 1|yi ) ∝ τg fg (yi ),
where τg is the prior probability of class g. The class densities can be estimated by
a nonparametric method such as the k-nearest neighbors method (Ripley, 1996,
Chapter 6), or with a parametric model.
The first documented methods in discriminant analysis were model-based clas-
sification methods. These methods use parametric models for which the marginal
density of the observations can be written as a finite mixture of parameterized
densities, namely

G
p(yi |τ, θ) = τg fg (yi |θg ), (4.1)
g=1

where θg is the parameter of the gth class conditional density, τ = (τ1 , . . . , τG )

denotes the mixing proportions, and θ = (θ1 , . . . , θG ). Thus, this marginal distri-
bution is a finite mixture of densities with parameters (τ, θ).
The first classification models assumed that the class conditional densities are
Gaussian (Anderson, 2003, Chapter 6). When the class covariance matrices are
assumed to be equal, it leads to the popular linear discriminant analysis (LDA)
which was proposed and studied first by Wald (1944) for G = 2 and by Von Mises
(1945) for G ≥ 2. When the class covariance matrices are unequal, it leads to
quadratic discriminant analysis (QDA) and the boundaries between the classes
are no longer linear, but quadratic.
In this chapter, we restrict attention to model-based classification methods.

4.1.1 Generative or Predictive Approach

The model-based classification methods presented in this chapter are instances of
the generative approach supervised classification. This consists of modeling the
joint distribution p(yi , zi ), while the predictive approach directly evaluates the
conditional distribution p(zi | yi ) in order to predict zi for new values of yi .
The predictive point of view has been much developed recently and has given
rise to many popular methods. Most of them, such as k-nearest neighbors, neural
networks, support vector machine, random forests are nonparametric and do not
4.1 Classification in the Probabilistic Framework 111

assume any model for the joint distribution p(yi , zi ) (Vapnik, 1998; Bishop, 2006;
Hastie et al., 2009). New methods such as deep learning for neural networks are
taking advantage of rapidly increasing computer power to address increasingly
complex problems involving larger data sets and more complex models.
In this landscape, model-based classification methods provide often slightly
higher misclassification error rates than the best predictive methods due to
differences between the models and the true distribution of the data. But model-
based classification has some advantages: interpretability and simplicity combined
with good if not optimal performance, and the ability to deal with partially
labeled data. Often, it provides almost optimal misclassification error rates with
parsimonious and understandable models. The decision rules of model-based
classifiers are also easier to interpret. Hybrid approaches aiming to achieve the
best of both worlds have also been proposed (Bouchard and Triggs, 2004; Lasserre
et al., 2006).
Moreover, a model-based approach is useful in the semi-supervised classification
context where many data are unlabeled (see Chapter 5). By using the information
provided by cheap and widespread unlabeled data, semi-supervised classification
aims to improve the classifiers’ performance. Most predictive approaches are
unable to take unlabeled data into account without additional assumptions on
the marginal distribution p(y) of the predictors.
In a semi-supervised setting it is natural to adopt the generative point of view,
since it leads to modeling the joint distribution
p(yi , zi ) = p(zi )p(yi |zi ).
This in turn leads to writing the marginal distribution p(yi ) as a mixture model

p(yi ) = p(zi )p(yi |zi ).
zi

Thus, the maximum likelihood estimate of the model parameters can be simply
computed using the EM algorithm. The straightforward formulae of EM in the
semi-supervised context are given in Chapter 5 and details can be found in Miller
and Browning (2003).

4.1.2 An Introductory Example

We illustrate the use of a nonparametric method and a parametric model-based
classification method on a simple data set, originally analyzed by Habbema et al.
(1974). The objective is to find a procedure for detecting potential haemophilia
A carriers on the basis of two measured variables, Y1 = log10 (AHF activity) and
Y2 = log10 (AHF-like antigen). The first class of n1 = 30 women consists of known
non-carriers (normal class) and the second class of n2 = 45 women is selected
from known haemophilia A carriers (obligatory carriers). This data set is depicted
in Figures 4.2 and 4.3.
The nonparametric classification method we consider is a k-nearest neighbors
(NN) method which allocates an observation to the majority class among its
112 Model-based Classification

Learning data set Decision boundaries of k−NN

0.3

0.3
● ●

● ●

● ●
0.2

0.2
● ● ● ● ● ●
● ●
● ●
● ● ● ●
0.1

0.1
●
● ● ● ●
●
● ●
● ●
AHFantigen

AHFantigen
● ●● ●
0.0

0.0
● ● ●
● ● ● ●
● ●
● ● ● ● ●
● ● ● ●
● ● ●
−0.1

−0.1
● ●
● ● ● ● ●
● ● ●
●

●
● ● ● ● ●
−0.2

−0.2
● ●
●
● ●
−0.3

−0.3
● ●

−0.6 −0.4 −0.2 0.0 −0.6 −0.4 −0.2 0.0

AHFactivity AHFactivity

Classification of new data with k−NN Actual labels of new data

0.3

0.3
●
0.2

0.2
● ●

● ●
● ●
0.1

0.1
● ● ● ●
●
● ●
AHFantigen

AHFantigen
● ● ●
● ●
0.0

0.0
● ●

● ●
●
−0.1

−0.1
● ●
● ●
●
−0.2

−0.2
● ● ●
●
● ●
● ●
−0.3

−0.3

● ●
●

−0.6 −0.4 −0.2 0.0 −0.6 −0.4 −0.2 0.0

● AHFactivity AHFactivity

Figure 4.2 The panel on the top left displays the training data set for the
haemophilia data set (no carriers in blue and carriers in red); the panel on
the top right displays the boundary of the 1NN decision rule on the training
data set; the bottom left panel displays the test data set and the bottom
right panel its assignment.

k-nearest neighbors in the training set. In this illustration, we took k = 1.

The parametric model-based classification method is the quadratic discriminant
analysis (QDA) which consists of assuming that each class follows a Gaussian
distribution with no restrictions on its parameters.
In order to assess the predictive ability of the classifiers, the whole data set was
divided at random into a training data set (75%) and a test data set (25%). The
training data set was used to design the classifiers while the test data set was used
to estimate their misclassification rates (see Section 4.7). Figure 4.2 displays the
classification with the 1NN method. It appears that the 1NN classifier overfitted
the training data set. It produced no error on the training data set while it led to
5 misclassified observations on the test data set (4 for the non carrier class on 8
observations and 1 for the carrier class on 16 observations).

Figure 4.3 displays the classification with the QDA method. Contrary to the 1NN
classifier, there is no sensitive difference between the misclassification error rates
on the training and the test data sets with QDA. On the test data set, it leads to
4 misclassified observations (3 for the noncarrier class on 8 observations and 1 for
the carrier class on 16 observations).
4.2 Parameter Estimation 113

0.3 Learning data set Learned model and decision boundaries of QDA

0.3
● ●

● ●

● ●
0.2

0.2
● ● ● ● ● ●
● ●
● ●
● ● ● ●
0.1

0.1
●
● ● ● ●
●
● ●
● ●
AHFantigen

AHFantigen
● ●● ●
0.0

0.0
● ● ●
● ● ● ●
● ●
● ● ● ● ●
● ● ● ●
● ● ●
−0.1

−0.1
● ●
● ● ● ● ●
● ● ●
●

●
● ● ● ● ●
−0.2

−0.2
● ●
●
● ●
−0.3

−0.3
● ●

−0.6 −0.4 −0.2 0.0 −0.6 −0.4 −0.2 0.0

AHFactivity AHFactivity

Classification of new data with QDA Actual labels of new data

0.3

0.3
●
0.2

0.2
● ●

● ●
● ●
0.1

0.1
● ● ● ●
●
● ●
AHFantigen

AHFantigen
● ● ●
● ●
0.0

0.0
● ●

● ●
●
−0.1

●
●
−0.1 ●
●
●
−0.2

−0.2

● ● ●
●
● ●
● ●
−0.3

−0.3

● ●
●

−0.6 −0.4 −0.2 0.0 −0.6 −0.4 −0.2 0.0

● AHFactivity AHFactivity

Figure 4.3 The panel on the top left displays the training data set for the
haemophilia data set (non carriers in blue and carriers in red); the panel on
the top right displays the boundary of the QDA decision rule on the training
data set; the bottom left panel displays the test data set and the bottom
right panel its assignment.

4.2 Parameter Estimation

In the model-based classiﬁcation context, the estimation task is easier than in the
clustering context because both the number G of classes and the labels z1 , . . . , zn
are known. In this context, the n labeled observations (y1 , z1 ),..., (yn , zn ) are
identically and independently distributed random observations with distribution

G
p(yi , zi |τ, θ) = τgzig [fg (yi |θg )]zig . (4.2)
g=1

The way the whole vector parameter (τ, θ) is estimated depends on the way the
training data set is sampled.

Sampling Schemes
In the mixture sampling, the training data set arises from a random sample from
the whole population under study, while in retrospective sampling, the training
data set is the concatenation of G independent sub-samples with ﬁxed sizes ng
and distribution fg (yi |θg ) for k = 1, . . . , G.
114 Model-based Classiﬁcation

With mixture sampling, the yi are sampled from the distribution

G
p(yi |τ, θ) = τg fg (yi |θg ).
g=1

Mixture sampling is more natural and the maximum likelihood estimate of τg is

the proportion ng /n for g = 1, . . . , G, where ng denotes the number of objects
arising from the gth class in the training data set. However, when rare classes are
present, retrospective sampling is preferred. But it requires the class proportions
to be known since they cannot be estimated from the ﬁxed proportions ng and, in
such cases, the vector parameter to be estimated reduces to θ = (θ1 , . . . , θG ).
An estimate θ̂ of θ is obtained by the maximum likelihood method
θ̂ = arg max L(τ, θ|y, z), (4.3)
θ

where the log-likelihood function L(τ, θ|y, z) is deﬁned by

n
G
L(τ, θ|y, z) = zig log[τg fg (yi |θg )]. (4.4)
i=1 g=1

The maximum likelihood estimate of θ depends on the parameterized model being

used. For some models, θ̂ is not available in closed form. The computation needed
to get θ̂ is equivalent to a single M-step of the EM algorithm in a model-based
clustering context.
As soon as θ̂ is derived, it is possible to assign any new point yn+1 with unknown
label to one of the G classes by the maximum a posteriori (MAP) procedure. The
estimated conditional probability, ẑ(n+1)g , that yn+1 arises from the gth class is
τ̂g fg (yn+1 |θ̂g )
ẑ(n+1)g = G . (4.5)
=1 τ̂ fg (yn+1 |θ̂ )

The MAP procedure consists of assigning yn+1 to the class maximizing this
conditional probability, namely

∗ 1 if g = arg max=1...,G ẑ(n+1)g

z(n+1) g = (4.6)
0 otherwise.

4.3 Parsimonious Classiﬁcation Models

4.3.1 Gaussian Classification with EDDA
When the variables are continuous, it is possible to take advantage of the numerous
Gaussian models proposed in the model-based clustering context to get a collection
of classification models of different complexity and with simple geometric interpre-
tation. This leads to the so-called EDDA (Eigenvalue Decomposition Discriminant
Analysis) method which makes use of the eigenvector decomposition of the class
variance matrices
Σg = λg Dg Ag DgT , (4.7)
4.3 Parsimonious Classification Models 115

where λg = |Σg |1/d , Dg is the eigenvector matrix of Σg and Ag is a diagonal matrix

with determinant 1, whose diagonal contains the normalized eigenvalues of Σg in
decreasing order (see Bensmail and Celeux, 1996).
Recall that the parameter λg determines the volume of class g, Dg its orientation
and Ag its shape. By allowing some of these elements to vary or not between classes,
we get different, more or less parsimonious, and easy to interpret models. Moreover,
by considering particular situations (diagonal or proportional to identity variance
matrices), we get other interesting and highly parsimonious models. Overall,
there are 14 competing models, and EDDA chooses the model that minimizes the
cross-validated error rate, the parameters being estimated by maximum likelihood.
In many cases, EDDA provides relevant parsimonious classifiers with simple
geometric interpretation and the interest of EDDA increases when the number
of classes is large. For instance, the standard linear (LDA) and quadratic (QDA)
classifiers (McLachlan, 1992) are special cases of the EDDA methodology. The
linear classifier is obtained by assuming equal volumes, shapes and orientations
for the covariance matrices Σg and the quadratic classifier is obtained when the
Σg are allowed to vary freely.
As an illustration, some models available with EDDA are applied in Listing 4.1
on the diabetes data set, using the Rmixmod package. The classes to be recovered
are the three classes of the clinical classification. For this classification problem, we
assumed that the data have been drawn according to the mixture sampling and thus
the class proportions are free. Figure 4.4 displays the classification boundaries of
the quadratic classifier (pg Lg Cg in the terminology of the mixmodLearn function),
the classifier allowing free class volumes (pg Lg C) and the linear classifier with
free proportions (pg LC) in the plane of the first two variables.
In the standard version of EDDA (Bensmail and Celeux, 1996), one of the
available models is selected by minimizing the cross-validated error rate (see
Section 4.7). This criterion provides usually good results especially in a small
sample setting, but other model selection criteria such as BIC could also be used.
For the diabetes data set, we compare the behavior of BIC and the cross-validated
error rate (CV). Note that in Figure 4.5 and Listing 4.2 we consider the criterion
−BIC because BIC is to be maximized in Rmixmod. For this data set, there is
little difference between the two criteria (see Figure 4.5) and both of them select
the most complex model QDA.

4.3.2 Regularized Discriminant Analysis

A standard way to design classiﬁer estimates with reduced variances, when the
size of the training data set is small, consists of replacing plug-in estimates
with shrunk estimates. In the Gaussian framework, Friedman (1989) proposed
regularized discriminant analysis (RDA) based on the following regularization of
the class covariance matrices

tr(Σ̂g (λ))
Σ̃g (λ, γ) = (1 − γ)Σ̂g (λ) + γ Id ,
d
116 Model-based Classiﬁcation

Listing 4.1: Classiﬁcation boundaries for EDDA models

# Loading libraries and data
library ( MBCbook )
data ( diabetes )
diabetes.data = diabetes [2:4]
diabetes.class = unlist ( diabetes [1])

# Learning the classifier with Mixmod

# LDA = ' Gaussian_pk_L_C ' , QDA = ' Gaussian_pk_Lk_Ck '
res = mixmodLearn ( diabetes.data [ ,2:3] , diabetes.class , models =
mixmodGaussianModel ( listModels = ' Gaussian_pk_Lk_C ') )

# Drawing the classification boundaries

prec = 150; Z = diabetes.data [ ,2:3]
x1 = seq ( -200 ,1800 , length = prec )
x2 = seq ( -200 ,900 , length = prec )
s = expand.grid ( x1 , x2 ) ; s <- as.data.frame ( s )
P = mixmodPredict ( as.data.frame ( s ) , res@bestResult ) @proba
plot ( diabetes.data [ ,2:3] , type = 'n ' , xlim = c ( -100 ,1700) , ylim = c ( -100
,800) )
pastel <- .9
points (s , type = 'p ' , pch =16 , col = c ( rgb (1 , pastel , pastel ) , rgb ( pastel ,1 ,
pastel ) , rgb ( pastel , pastel ,1) ) [ max.col ( P ) ])
points ( diabetes.data [ ,2:3] , col = as.numeric ( diabetes.class ) +1 , pch =19 ,
xlim = c ( -100 ,1700) , ylim = c ( -100 ,800) )
for ( i in 1:3) {
T = P [ , i ] - apply ( P [ , -i , drop = FALSE ] ,1 , max )
contour ( x1 , x2 , matrix (T , prec , prec ) , level =0 , add =1 , col = " black " , lwd
=3 , drawlabels =0) ;
}
title ( main = " EDDA with model ' pk_Lk_Ck ' " ) ; box ()

where
(1 − λ)(ng − 1)Σ̂g + λ(n − G)Σ̂
Σ̂g (λ) = ,
(1 − λ)(ng − 1) + λ(n − G)
Σ̂g are the estimates of the covariance matrices for the QDA model for g = 1, . . . , G,
that is n
zig (yi − μ̂g )(yi − μ̂g )T
Σ̂g = i=1 ,
ng
where n
i=1 zig yi
μ̂g = ,
ng
and Σ̂ is the common covariance matrix estimate for the LDA model, namely
G n
i=1 zig (yi − μ̂g )(yi − μ̂g )
T
g=1
Σ̂ = .
n
The complexity parameter λ (0 ≤ λ ≤ 1) controls the contribution of QDA and
LDA, while the regularization parameter γ (0 ≤ γ ≤ 1) controls the amount of
4.3 Parsimonious Classiﬁcation Models 117

EDDA with model 'pk_Lk_Ck' EDDA with model 'pk_Lk_C'

800

800
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
600

600
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
400

400
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
sspg

sspg
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●
●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●
●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●
●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
200

200
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●
●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●
●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●
●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●
●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●● ●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●● ●●● ●● ● ●●●●●●●●●●●●●●●●●●●
●●●●●
● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●● ●●● ●● ● ●●●●●●●●●●●●●●●●●●●
●●●●●
● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●●●● ●●●●● ●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●● ●●●●● ●●●●● ●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0

0
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

0 500 1000 1500 0 500 1000 1500

insulin insulin
800
600

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0

0 500 1000 1500

insulin

Figure 4.4 C ass ﬁcat on boundar es for EDDA mode s pg Lg Cg pg Lg C and

pg LC on the d abetes data set

shrinkage of the eigenvalues towards equality, since tr(Σ̂g (λ))/d is the mean of
eigenvalues of Σ̂g (λ).
Thus, with fixed γ = 0, varying λ leads to an intermediate method between
LDA and QDA Now w th fixed λ = 0 ncreas ng γ eads to more b ased est mates
of the e genva ues of the covar ance matr ces For λ = 1 and γ = 1 we get a qu te
s mp e c ass fier wh ch cons sts of ass gn ng any vector to the c ass whose center s
the nearest.
Figure 4.6 illustrates the role of the parameters λ and γ on simulated data.
Finally, note that it is also possible to use the Moore–Penrose pseudo-inverse of
Σ̂ instead of the usual inverse Σ̂−1 . The reader can also refer to Mkhadri et al.
(1997) who provides a comprehensive overview of regularization techniques in
discriminant analysis.
118 Model-based Classification

Listing 4.2: Model selection with BIC and CV

# List of models and learning with mixmodLearn
models = mixmodGaussianModel ( family = ' all ')
res = mixmodLearn ( diabetes.data , diabetes.class , models = models ,
criterion = c ( ' CV ' , ' BIC ') )

# Collecting the BIC and CV values

BIC = CV = rep ( NA , length ( models@ listMo dels ) )
for ( i in 1: length ( res@results ) ) {
ind = which ( res@results [[ i ]] @model == models@l istMod els )
CV [ ind ] = res@results [[ i ]] @criterionValue [1]
BIC [ ind ] = - res@results [[ i ]] @criterionValue [2]
}

# Plot of the results

par ( mfrow = c (2 ,1) )
plot ( BIC , type = 'b ' , xlab = ' ' , xaxt = 'n ' , col =2) ; axis (1 , at =1: length (
res@results ) , labels = substr ( models@listModels ,10 ,30) , cex.axis =0 .8
, las =2)
abline ( v = which.max ( -BIC ) , col =1 , lty =2)
plot ( CV , type = 'b ' , xlab = ' ' , xaxt = 'n ' , col =3) ; axis (1 , at =1: length (
res@results ) , labels = substr ( models@listModels ,10 ,30) , cex.axis =0 .8
, las =2)
abline ( v = which.max ( CV ) , col =1 , lty =2)

The solution based on regularization does not have the same drawbacks as
dimension reduction and can be used more widely (see Chapter 8).
The tuning parameters λ and γ are computed by minimizing the cross-validated
error rate. In practice, this method performs well compared with LDA and QDA
for small sample sizes. The drawbacks of RDA are that it provides classifiers
that can be difficult to interpret and the methods are not very sensitive to the
parameters λ and γ; Figure 4.7 provides an illustration of the similar performance
for large ranges of values of the parameters.
The next illustration allows us to compare the EDDA and RDA behaviors on the
haemophilia data. The data set presented in Section 4.1.2 consists of a population
of 75 women (45 were haemophilia A carriers and 30 were not) described by two
variables y1 = 100 log(AHF activity) and y2 = 100 log(AHF like antigen). We
use this data set to illustrate the ability of EDDA to suggest a plausible geometric
model. We applied RDA (see Listing 4.3), LDA, QDA and EDDA to this data set.
The prior probabilities of the two classes were assumed to be equal.
The cross-validated misclassification risks were .14, .16, .17 and .13 for the four
methods. For RDA, the complexity parameter was λ = 0.15 and the shrinkage
parameter was γ = 0.8. Thus, RDA proposed a shrunk version of a quadratic
classifier. But, as shown in Figure 4.7, the two selected tuning parameters vary
greatly with the cross-validated samples and so are difficult to interpret. Conversely,
the selected model with EDDA was [λg DAg D ] and the boundaries of this classifier
are depicted in Figure 4.8.
Thus EDDA suggests that the best model for this data set assumes different
4.4 Multinomial Classification 119

−5200

−BIC

−5800

pk_L_I

pk_Lk_I

pk_L_B

pk_Lk_B

pk_L_Bk

pk_Lk_Bk

pk_L_C

pk_Lk_C

pk_L_D_Ak_D

pk_Lk_D_Ak_D

pk_L_Dk_A_Dk

pk_Lk_Dk_A_Dk

pk_L_Ck

pk_Lk_Ck
0.94

0.90

0.86

0.82

pk_L_I

pk_Lk_I

pk_L_B

pk_Lk_B

pk_L_Bk

pk_Lk_Bk

pk_L_C

pk_Lk_C

pk_L_D_Ak_D

pk_Lk_D_Ak_D

pk_L_Dk_A_Dk

pk_Lk_Dk_A_Dk

pk_L_Ck

pk_Lk_Ck
Figure 4.5 Model selection with BIC and CV on the diabetes data.

volumes and shapes and the same orientations for the two classes. Its boundaries
are depicted in Figure 4.8 and show that this proposal seems quite reasonable.
EDDA performs well and provides a clear representation of the class distributions.
For this example, we do not claim that EDDA performs better than RDA or LDA.
The .632 bootstrap, (Efron and Tibshirani, 1997), with 100 bootstrap replications
gives the same estimated misclassiﬁcation risk (.08) for RDA, LDA and EDDA.
But, we think that, for this data set, EDDA provides a clearer and more relevant
representation of the class distributions than either RDA or LDA.

4.4 Multinomial Classiﬁcation

When the variables are categorical, the empirical model is simple and natural.
It consists of assuming that the variables follow a multivariate multinomial
distribution within each class. Unfortunately the number of parameters to be
estimated grows rapidly with data dimension. For instance, with d binary variables,
the number of parameters of the empirical model is 2d per class. Clearly, this
model suﬀers from the curse of dimensionality (see Chapter 8) and there is the
need to deﬁne far simpler models with fewer parameters.

4.4.1 The Conditional Independence Model

A simple way to address the dimensionality problem is to assume that the
variables within an observation are independent conditional on knowing the class
membership of the observation. This leads to the conditional or local independence
120 Model-based Classiﬁcation

λ = 0, γ = 0 λ = 0, γ = 0.5 λ = 0, γ = 1

λ = 0.5, γ = 0 λ = 0.5, γ = 0.5 λ = 0.5, γ = 1

λ = 1, γ = 0 λ = 1, γ = 0.5 λ = 1, γ = 1

Figure 4.6 Eﬀect of the parameters λ and γ in regularized discriminant

analysis (RDA) on simulated data. The parameter λ allows the classiﬁer to
vary from QDA to LDA whereas γ controls the ridge regularization.

model which can be regarded as a reference generative classiﬁcation model for

categorical data. This model assumes that the d descriptors are independent
conditionally on knowing the classes and that the class densities are multivariate
multinomial distributions. This is similar to the latent class model described in
Chapter 6 for model-based clustering.
Assuming that variable j has mj response levels, the data are (y1 , . . . , yn ) where
yi is coded by the vector (yij ; j = 1, . . . , d) of response level h for each variable j
or, equivalently, by the binary vector (yijh ; j = 1, . . . , d; h = 1, . . . , mj ) with

yijh = 1 if yij = h
yijh = 0 otherwise.

Then, the class conditional densities are multivariate multinomial distributions,

4.4 Multinomial Classiﬁcation 121

Listing 4.3: Regularized discriminant analysis on the hemophilia data

# Loading libraries and data
library ( klaR ) ; library ( rrcov )
library ( MBCbook )
data ( hemophilia )
X = hemophilia [ , -3 ]
cls = as.factor ( hemophilia [ ,3])

# Learning the classifier with RDA

out = rda (X , cls , prior = 1 , crossval = TRUE , fold = 10)
out $ regularization
out $ error.rate [2]

# Analysis of the parameter sensitivity

OUT = matrix ( NA ,25 ,3)
for ( i in 1:25) {
out = rda (X , cls , prior = 1 , crossval = TRUE , fold = 10)
OUT [i ,1:2] = out $ regularization
OUT [i ,3] = out $ error.rate [2]
}
boxplot ( OUT , names = c ( ' Gamma ' , ' Lambda ' , ' CV error rate ') , col =2:4)

# Learning the classifier with EDDA

# ( LDA = ' Gaussian_p_L_C ' , QDA = ' Gaussian_pk_Lk_Ck ')
res = mixmodLearn (X , cls , models = mixmodGaussianModel ( listModels = '
G a u s s i a n _ p _ L k _ D _ A k _ D ') )
1.0
0.8
0.6
0.4
0.2

●
●
●
●

●
0.0

Gamma Lambda CV error rate

Figure 4.7 Boxplot of the tuning parameters γ and λ of RDA found using
repeated cross-validated samples.

for g = 1, . . . , G,
jh
fg (yi |αg ) = (αgjh )yi , (4.8)
j,h
122 Model-based Classiﬁcation

EDDA with model 'pk_Lk_D_Ak' RDA (γ=0.8, λ=0.15)

● ●

● ● ● ●

0.2

0.2
● ●

● ● ● ●
● ● ● ●
● ●
● ●
● ●
● ● ● ●
● ●
● ● ● ●
● ● ● ●
● ● ● ●
● ● ● ●
● ● ● ●
● ● ● ●
● ● ●● ● ● ●●
0.0

0.0
● ● ● ●
● ●
● ●
● ●
● ● ● ●
AHFantigen

● ●
● ●

AHFantigen
● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ●
● ●
● ● ● ●
● ●
● ● ● ● ● ● ● ●
● ●
● ●

● ●
● ● ● ●
● ● ● ●
−0.2

● ● ● ●

−0.2
● ●
● ●
● ●
● ●
● ●
● ●

● ●
● ●

● ●
−0.4

−0.4
● ●

−0.6 −0.4 −0.2 0.0 −0.6 −0.4 −0.2 0.0

AHFactivity AHFactivity

Figure 4.8 Classiﬁcation boundaries for the EDDA model [λg DAg DT ]
(left) and for the RDA method with tuning parameters λ = 0.15 and
γ = 0.80 on the haemophilia data set (right).

where αgjh denotes the probability that variable j has level h in class g, and
αg = (αgjh ; j = 1, . . . , d; h = 1, . . . , mj ). The maximum likelihood estimates of the
αg parameters are
(ujh
g )
(αgjh ) =
, j = 1, . . . , d; h = 1, . . . , mj ; g = 1, . . . , G,
ng
n n jh
where ng = i=1 zig and (ujh g ) = i=1 zig yi .
But maximum likelihood estimation with multivariate categorical data remains
problematic because of the curse of dimensionality. This conditional independence
model requires (G − 1) + G j (mj − 1) parameters to be estimated. The model
is more parsimonious than the empirical model which requires j mj parameters,
but it can still be too complex with regard to the sample size n and it can even
involve numerical difficulties (“divide by zero” occurrences or parameters αgjh
estimated to zero).
One way of avoiding such drawbacks is to use the regularized maximum likeli-
hood estimator:
g )+c−1
(ujh
(αgjh ) = (r) ,
ng + mj (c − 1)
where c is a fixed positive number, for updating the estimation of the αgjh . The
estimates can be sensitive to the value of c and considering this regularization
issue in a Bayesian setting could be beneficial (see Section 6.2.7).
With this in mind, the parsimonious models presented in Chapter 6 in the
clustering context could be thought of as desirable. These models use a repa-
rameterization of αg in two parts. For any variable j, let ajg denote the most
frequent response level in class g, the parameter αg can be replaced by (ag , εg )
4.4 Multinomial Classification 123

with ag = (a1g , . . . , adg ) and εg = (ε11 dmd

g , . . . , εg ) where

1 − αgjh if h = ajg
ajg = arg max αgjh and εjh =
h
g
αgjh otherwise.
Vector ag provides the modal levels in class g for the variables and the elements
of vector εg can be regarded as scattering values.
Using this form, it is possible to impose various constraints on the scattering
parameters εjh
g . The models we consider are the following.

• the standard conditional independence model [εjh g ]: the scattering depends upon
classes, variables and levels.
• [εjg ]: the scattering depends upon classes and variables but not upon levels.
• [εg ]: the scattering depends upon classes, but not upon variables.
• [εj ]: the scattering depends upon variables, but not upon classes and levels.
• [ε]: the scattering is constant over variables and classes.
We do not detail here the maximum likelihood estimation of these models in
the classiﬁcation context. This can be straightforwardly derived from the M-step
formulae given in Chapter 6.
As an example, the maximum likelihood formulae for the model [εjg ], are
(ajg ) = arg max(ujh
g )
h

and
ng − vgj
(εjg ) = ,
ng
∗
∗
for g = 1, . . . , G and j = 1, . . . , d, where vgj = ujh
g , h is the modal level ag
j
j
(Aitchison and Aitken, 1976). Notice that εg is simply the percentage of times the
level of an object in class g is different from its modal level for variable j.
By considering this particular representation of the multivariate multinomial
distribution, it is possible to get parsimonious and simple classifiers. The pa-
rameters of these classifiers are estimated using the formulae presented in the
latent class model context. In particular, it could be beneficial to consider regular-
ized maximum likelihood or Bayesian estimates to avoid the numerical problems
occurring when dealing with many categorical variables.

4.4.2 An Illustration
To illustrate this family of models for supervised classification with categorical
data, a data set describing the morphology of birds (puffins) is used. This data
set was first used by Bretagnolle (2007). Each bird is described by five categorical
variables. One variable for the gender and four variables providing a morphological
description of the birds: the eyebrow stripe (4 levels), the collar description (4
levels), the sub-caudal description (4 levels) and the border description (3 levels).
There are 69 puffins from two species: lherminieri (34) and subalaris (35). The
124 Model-based Classification

purpose is to recover the puffin species with the five categorical variables. Assuming
a mixture sampling scheme, the model minimizing the cross-validated error rate
(leave-one-out) is the standard model [εjhg ] and the estimated error rate is 1.5%.
Listing 4.4 presents the code for learning the classifier on the puffin data set
with Rmixmod.
**************************************************************
* Number of samples = 69
* Problem dimension = 5
**************************************************************
* Number of cluster = 2
* Model Type = Binary_pk_Ekjh
* Criterion = CV(0.9855)
* Parameters = list by cluster
* Cluster 1 :
Proportion = 0.4928
Center = 1.0000 2.0000 3.0000 1.0000 1.0000
Scatter = | 0.4714 0.4714 |
| 0.1786 0.3929 0.2071 0.0071 |
| 0.1500 0.3786 0.5929 0.0643 |
| 0.4500 0.3214 0.0929 0.0357 |
| 0.0762 0.0667 0.0095 |
* Cluster 2 :
Proportion = 0.5072
Center = 2.0000 3.0000 1.0000 1.0000 1.0000
Scatter = | 0.4306 0.4306 |
| 0.0069 0.0069 0.1319 0.1181 |
| 0.0208 0.0069 0.0069 0.0069 |
| 0.0486 0.0069 0.0069 0.0347 |
| 0.0741 0.0370 0.0370 |
* Log-likelihood = -237.3621
**************************************************************

Figure 4.9 displays the distribution of the two classes for the ﬁve descriptors.
The discriminative variables are eyebrow stripe, collar description and sub-caudal
description, while gender is (not surprisingly) useless as border description.

4.5 Variable Selection

In the model-based classification setting, selecting the relevant variables is often
of primary interest to get an efficient classifier. Each of the d predictive variables
brings discriminant information (its own ability to separate the classes), and
noise (its sampling variance). It is important to select the variables bringing more
discriminant information than noise. In practice there are three kinds of variables:
4.5 Variable Selection 125

Listing 4.4: Multinomial classiﬁcation for the puffin data set

# Loading libraries and data
library ( MBCbook )
data ( puffin )
X = puffin [ , -1 ]
cls = puffin $ class

# Learning the classifier with Mixmod

out = mixmodLearn (X , as.factor ( cls ) , dataType = " qualitative " )
barplot ( out )

# Computing and displaying the alphas

p = ncol ( X )
prms = o u t @ b e s t R e s u l t @ p a r a m e t e r s
M = ncol ( prms@scatter [[1]])
alpha = array ( NA , c ( max ( cls ) ,p , M ) )
for ( c in 1: max ( cls ) ) {
center = prms@center [c ,]
a = matrix (0 ,p , M )
for ( j in 1: M ) a [ center == j , j ] = 1
alpha [c , ,] = abs ( a - prms@scatter [[1]])
}
par ( mfrow = c (2 ,3) )
for ( j in 1: p ) {
alph = alpha [ ,j ,]
alph = alph [ , colSums ( alph ) != 0]
barplot ( alph , beside = TRUE , names.arg = levels ( X [ , j ]) , le gend.text = c ( "
Borealis " ," Diomedea " ) , col = c (1 ,10) , ylab = ' Alpha ')
}
summary ( out )

• Useful variables bringing strong discriminant information. For instance, to

classify men and women, the height is clearly a useful variable.
• Redundant variables for which the discriminant information is essentially pro-
vided by other variables: for the same example, weight does not give information
on the gender knowing the height of a person.
• Irrelevant or noisy variables. Eye color does not contain any information on the
gender of a person.

Thus variable selection is in general a crucial step to get a reliable classiﬁer.

In model-based classification, it is possible to define the roles of the predictive
variables in a proper way, as precisely described in Chapter 7 and to derive
algorithms to assign a relevant status (useful, redundant or independent) to the
predictors. In the model-based context, it is possible to design efficient variable
selection procedures that work well. This possibility could enhance the efficiency of
complex nonlinear generative classification methods such as quadratic discriminant
analysis (QDA) in high dimensional contexts and make them competitive in many
situations with popular classifiers such as LDA, logistic regression, k-nearest
neighbors classifier or support vector machine classifiers.
126 Model-based Classification

0.5

0.8
Borealis Borealis Borealis
Diomedea Diomedea Diomedea

0.4

0.6

0.6
0.3
Alpha

Alpha

Alpha
0.4

0.4
0.2

0.2

0.2
0.1
0.0

0.0

0.0
Male Female None Pronounced None Dashed Continuous
0.5

Borealis Borealis

0.8
Diomedea Diomedea
0.4

0.6
0.3
Alpha

Alpha
0.4
0.2

0.2
0.1
0.0

0.0

White Black Black & white None Few Many

Figure 4.9 The distribution of the two puﬃn species on the ﬁve
descriptors: gender, eyebrow stripe, collar description, sub-caudal description
and border description (from left to right and top to bottom).

4.6 Mixture Discriminant Analysis

Contrary to situations where regularization is desirable, there exist situations,
with a reasonable training sample size, where the Gaussian assumption for the
class-conditional densities is too restrictive. Thus, as proposed by Hastie and
Tibshirani (1996), it is possible to model the class-conditional densities with a
mixture of distributions. In what follows, we restrict attention to mixtures of
Gaussian distributions on continuous predictors. Mixture discriminant analysis
(MDA) assumes that, for g = 1, . . . , G, the class density is a mixture density

rg

fg (yi ) = πg Φ(yi |mg , Σg ),

=1
rg
where the mixture proportions πg satisfy =1 πg = 1.
The parameters of the mixture distributions for each class can be estimated
using the EM algorithm described in Chapter 2. All the models proposed in EDDA
can be used when deﬁning an MDA classiﬁer (see Fraley and Raftery, 2002). But
such a general method would lead to a large number of parameters to be estimated.
Thus, MDA would need a large sample size for each class to be relevant. For
this reason, in Hastie and Tibshirani (1996), the authors restricted attention to a
mixture model with a common covariance matrix for all mixture components of
each class: Σg = Σ, (1 ≤ ≤ rg ; 1 ≤ g ≤ G), the number of mixture components
rg being chosen a priori by the user. This allows for a dramatic reduction in the
number of parameters to be estimated, but the price to be paid is a great loss of
4.7 Model Assessment and Selection 127

0.25
0.20
Classification error
0.15
0.10
0.05
0.00

2 4 6 8 10
G

Figure 4.10 Choice of the common number of components per class for
MDA with the diabetes data set.

the flexibility of MDA. An illustration of MDA with the most general Gaussian
mixture with rg = r for g = 1, . . . , G is given in Figures 4.10 and 4.11 for the
diabetes data set. The best cross-validated error rate is achieved when r = 3. The
more and more complex boundaries of the classifications with r = 1, 3 and 10 are
displayed in Figure 4.11.
2
An alternative solution is to consider covariance matrices of the form Σg = σg I,
I being the identity matrix. Acting in such a way, the class conditional densities
fg (yi ) are modeled by a union of Gaussian balls with free volumes, where a Gaussian
ball is a multivariate Gaussian distribution with covariance matrix proportional to
the identity matrix. It is a flexible model since (numerous) Gaussian balls could be
expected to fit the cloud of the ng points of class g, and it remains parsimonious
since each covariance matrix is parameterized with a single parameter.
As an illustration, we consider in Listing 4.5 MDA with Gaussian balls for
learning a supervised classifier on the diabetes data set. For each class, the number
of Gaussian balls was chosen by the BIC criterion (3, 5 and 3 components for
the Type 1 diabetic, Type 2 diabetic and non-diabetic classes, respectively). The
boundaries of the classification rule are displayed in Figure 4.13. It can be seen
that they are somewhat less complex than the boundaries displayed in Figure 4.11
but remain flexible.

4.7 Model Assessment and Selection

A classifier aims at providing a decision function to predict the unknown labels of
future observations. Thus, it is important to assess the expected misclassification
error rate of a classifier designed on a finite training set, but supposed to be used
on an independent sample from a population of potentially infinite size. The most
128 Model-based Classification

MDA with model "EEE" and G=1

sspg
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●●● ● ●● ● ●●● ●● ●● ●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●
●●●
●●●
●●●
● ●
●
●●
●● ●●
● ●
●
●
●
●
●
●●
●●●
● ●●
●●●
●
●
●●
●●●
● ●●
●●●
●●●
● ●●
●●●
● ●●
● ●●
●●●
● ●●
● ●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●● ●● ● ●● ●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●
●●●●●●
● ●
●●
● ●
●● ●● ●●● ●●●●●●●
●● ●●●● ●●
●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●● ●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●● ●
●●
●●
● ●
●●
● ●
●●●●
●●
●●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●●
●●
●●●● ●●
●● ●●
●●●●
●● ●
●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
● ●
● ● ● ● ●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
● ●●
●●
●●●●●
●● ●
●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●
●●●●
●●●●
●●
●●●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
● ●● ● ● ●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●●
●●
●●
●●
●●●● ●●●●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●
●●
●●
● ●
●
●●
●
● ●●
● ●●
●●●●●
●●●●
●●●●
●● ●●
●●●●
● ●●
●●●●●●
●●●●●●●
●●●●
●●●●●●
●●●●
●●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●● ●●
●● ●
●●●●●●
●●
●●●●
●● ● ●
●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
200
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●
● ● ●●●
●
● ●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●
● ●
●●
●●●
●●
●
● ●
●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
● ●
●●
●●
●●
● ●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●
●●●●●
●
● ●●
● ●●
●●
● ●●
●●●
● ●●
●●●●●
●●●●
●●
●●●●●●●●●
●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●
● ● ●● ●
●● ●● ●● ●●●●●
● ● ● ●● ● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●● ●
●●● ●
●●
●●●●●●●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●● ●●
●● ●●●●
●●●●
●●●●
●●●● ●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
● ●● ● ●
● ●
● ●●● ●●
●●● ● ● ● ●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●
●●
●●
●●
●●
●● ●●
●●
● ●●
● ●●
●
●●
●
●
●
●●●●●●
●●
● ●●
● ●●● ●●
●●●●●●●●
●●● ●
●●●●●●
●● ●●
●●●●●●●●●●●●●●●●●● ●●●● ●●●●●
●●● ●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●● ●●
●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●● ● ●●● ● ●● ● ●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●●●● ●● ● ●●●
● ●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●●●●
●●
●●
●●●●
●●
●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●● ● ●● ●● ●● ●● ●●● ●● ● ●● ● ●●●● ●●●●●●●●●●●●●●●●● ● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
● ●●
●●●●
●●●●
●●
● ●●
●●●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●●
●●
●●
●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●●●
●●●●●
●●
●●
●●
●●
●●
●●●●
●●●●●
●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●●●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
● ●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●
● ●●
●●
●●●●
●●●●●●
●●●●●●
●●
●●
●●●●
●●●●
●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●●●●
●●
●● ●●
●●
●●
●●
●●
●●●
●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●● ●●●●
●●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●● ●
●●
●●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●
●●●●
●●●
●●
●●
●●●●
●●●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●
●●
●
●
●
● ●
●●
● ●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
● ●
●●
●●
●●
●●
●●
●
●
●
● ●
●●
●●
●●
●●
●●
●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●
●●
●
0

0 500 1000 1500

insulin

MDA with model "EEE" and G=3

●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●● ●
● ●
●●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●● ●●● ●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●●
●●●●
●●●●
●●
●●
●●●
●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
● ●●
●●
●●●●
●●●●
●●
●●
●●●
●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
● ●●
● ●●
●●●
● ●●
●●●
● ●●
●●●
● ●●
●●●
● ●●
●●●
●●●
● ●●
●●●
● ●●
● ●●
●●●
● ●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●
●
●●●
●●●
●●●
●●●
●●
●●
●●
●●●
●●●
●●●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ● ●● ●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●
●●
●● ●●
●●
●●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●●
● ●
●●●●
● ●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●●
● ●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●●
●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ● ●●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
600

●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
●●●
●●●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●
●●
●●
●
●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●●●●●●
●●●●
●●●●
●● ●
●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
● ●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
● ●●●
● ● ●
●●
● ●
●●
● ●
●●
● ●
●● ●●● ●● ●●●●
● ●●●●● ●●●●●●●●●●●●●●●●●
●●●
●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●● ●●
●●
● ●●
●●●●
●●
●●●●
●●●●●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●● ●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●●●●●● ●● ● ●● ●● ● ●● ●●● ●● ● ●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●
●●
● ●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●●
● ●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●●●
●●●●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●●
● ●
●●
● ●●
●●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●● ●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●● ●●
●●
●●●●
●● ●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●
●
●●
●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●● ●● ●● ●● ●●● ● ● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●●
●●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
400

●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
● ●
●●
● ●
●●
●●
● ●●
●●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●● ●● ●● ●● ●● ● ●● ●●● ●● ● ●● ●●●●●
●● ●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●
●●●
● ●●
● ●●
●●●
● ●●
●●●
● ●
●●
●●
●●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●
●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●● ●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●● ● ●● ●● ●● ●● ●●● ●●●●●●● ●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●● ●
●●
●●●●
●●
●●
●●
●● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●
●●
● ●
●●
● ●●
●●●●●
●●
●●
● ●●
●●● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●
●●
●●
●●●●
●●●● ●●
●● ●●
●● ●●
●● ●●●●●●
●●●●●● ●●●●●●
●●●
●●●●●●
●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●●
● ●
●●
● ●●
● ●●
● ●
●●
●●●●●
●●
●
●
●●
●●●
● ●●●●
●●
● ●●
●●
● ●●
● ●●
●●
● ●●●●
●●
● ●●●
●●
● ●●
●●
● ●●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●●●●●●●
●●
●●●●
●●
●● ●●
●●●●●●
●●
● ●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
● ● ●
●● ●● ●●● ●● ● ●● ●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●●●
●●●●●● ●●
●●●●
●●●● ● ●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ● ●● ● ●●●●
● ●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
200

●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●
●●
●●
● ●●●
● ● ●●
● ●●●
● ●●
●●●●
●
●
●●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●●
● ●●
●●●●
● ●
●●
●●●● ●●
●●
●●●●
●●●●●●
●●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●●
●
●
● ●
● ●●●
●●
● ●
●
●
●
●●
● ●●
●●● ●
●●
●●●
●●
●●
●●
● ●●
● ●
● ●
●
●●●
●
● ●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●● ●
●●
●●●●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●● ●●
●● ●●●●
●●●●
●●●●
●●●●
● ●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
● ● ●
● ●
● ●● ●●● ● ● ● ●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●● ●●
●
●●●
●●●●
●●
●● ●●●
●● ●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●
●●
●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●
●● ●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
● ●
●● ●●●●● ●●
● ● ●● ● ● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ● ● ●
●● ● ● ●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●● ●●
●● ●●●
●●
●●●
●●
●●●●●
●●
●●●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●● ●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●
●● ● ●●●
● ●● ●●
● ●●
● ●●
●●● ●● ● ●● ●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●
● ●● ●●●●●●●●●● ●
●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●●
●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●●●
●●● ●●
●●●●
●●●●
●●●●
●● ●● ●● ● ●●●●●
●●
●●
●●
●● ●●●●●●●●●●●●●●●●●●
● ●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●
●●
●●●●
●● ●
●●
●●●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●
●●
●●●●
●●●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●●
●●●●
●●●
●●●
●●●
●●
● ●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●
●●●
● ●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●●
● ●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●● ●●
●●●
●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●
●●
●●
●●
●●
●●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●● ●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●●
●
●●
● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●
● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
●●
●●
●●
● ●
●●
●●
●● ● ● ●● ●
●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
0

0 500 1000 1500

insulin

MDA w h mode EEE and G=10

800
600
400

●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
sspg

●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●
●●
●●
●●●●●●●●●●●● ●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●
●●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
● ●●
●●●●
●●●
●
●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●
●●
●●
●●
●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●●●
●●
●●
●●●●
●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●
●●●
●●●
●●●●
●●
●●●
●●●
●●●
●●●
●
●
●●
●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●●●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
●
200

●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
●
●●
●
●
●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●
●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
0

0 500 1000 1500

insulin

Figure 4.11 C ass ﬁcat on boundar es for MDA w th rg = r = 1 3 10

components per c ass for the d abetes data set
4.7 Model Assessment and Selection 129

Diabetes − model selection per class

−21
●
● Class 1
● ● ●
−22 ● ●
●
● Class 2
●
●
● Class●3
−23
−24
BIC value

● ●
● ●
−25

● ●

● ●
●

●
−26

●
● ● ●

● ●
−27

●
●
●
●
−28

2 4 6 8 10
Number of components

Figure 4.12 Choice of the number of components per class for MDA with
the diabetes data set.

widely used method to estimate the misclassiﬁcation error rate of a classiﬁer is

cross-validation.

4.7.1 The Cross-validated Error Rate

Cross-validation in its most common version is known as leave-one-out cross-va-
lidation, and is as follows. For i = 1, . . . , n the classifier is designed from the
training set A without the ith observation and the ith observation is assigned
to one of the classes with this classifier. The error rate is then estimated by the
frequency of misclassified points by this procedure. The resulting estimated error
rate is almost unbiased. But its variability is underestimated for large n since
the different classifiers designed with n − 2 common observations have a strong
tendency to be identical. Moreover, leave-one-out is costly for large n even if it is
often possible to derive the (n − 1)-classifier from the n-classifier by simplified
calculations.
So-called v-fold cross-validation is now often preferred to leave-one-out. It con-
sists of dividing at random the training data set A into v parts of approximatively
equal sizes. For = 1, . . . , v, a classifier is designed from A without the th part,
then the observations of the th part are assigned to one of the classes with
this classifier. Finally, the misclassification error rate is estimated by the mean
of the corresponding v obtained error rates. If v = n, v-fold cross-validation is
leave-one-out. Conversely if v = 2 v-fold cross-validation is known as half sampling
130 Model-based Classification

MDA model "EII" and G=(3,5,3)

●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
● ●
● ●
● ●
●●
●●
● ●
●●●
● ●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
● ● ●
●●●● ●
●●
● ●
●●
● ●
● ●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●●
●●●
●●●
●● ●
●●
●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●●
●●●
●● ●●
●●
● ●●
●●●●
●●●●
● ● ●●● ●● ● ●● ●●●●●●
● ●●
●●●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●● ●● ● ●
●● ●
●●
●●●●
●●
●●●
●●●
●●●●
●●●
●●●●
●●
● ●
●●●●
●●
●●●●●
●●●●
●●●
●●
●●●●
●●●
●● ●
●●●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
● ●●●● ●● ● ●● ●● ●● ●● ●●●● ●●● ●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●● ●
●●
● ●
●●
●●●
●●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●● ●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●
●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●●
● ●
●
●●
● ●
●●
● ● ● ●●● ●● ● ●● ●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●●
● ●●
●● ●●
●●●●●●
●● ●●●●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●●●● ● ●●● ●● ● ●● ●●●●●
● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●● ●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●●
●●●●
●●●●
●●●●●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●
●●●
●●●●
●●●
●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
● ●● ●● ●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●● ●●
●● ●●●●●●
●●
●●●●
●●●●●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
200

●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ●●
● ● ● ●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●
●●
●●
● ●●
●●
●●●●
●
●
●●
●●●●
●● ●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●●
●●●
●●●
●●
●● ●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●●●
●● ●●
●●●●
●●
●●●●● ●
●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
● ● ●
●●
● ●●●●●●
●●
● ●
●●
●●●
●●●●
●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●● ● ● ●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●● ●●
●● ●●●●
●●●●
●●●●
●●
●● ●●●
● ●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
● ● ●
● ●
● ● ● ● ●● ● ● ● ●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●● ●●
●
●●●
●●●●
●●
●● ●●●
●● ●●
●●●
●●●
●●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●
●●
●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●
●● ●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
● ●
●● ●●●●● ●
●● ● ●● ● ● ●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●● ● ● ●
●● ● ● ●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●● ●●
●● ●●●
●●
●●●
●●
●●●●
●●●
●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●
●● ● ●●●
● ●● ●●
● ● ●
● ●●
●●●
●●●●
●●●●●●
●●●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●● ●●
●●●●●●●
●●●●
●●●●
●●●●●●●●
●●
●●●●
●●●●●●
●●●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●●
●●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●● ●● ●●
● ● ● ●●● ●● ● ●● ●●●●● ●●●●●●●●●●●●●●●●●●●
● ●●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●
●●●●
●●●●
●●●●●●●●
●●
●●●●
●●●● ●●
●●
● ●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ● ● ●●● ●● ● ● ●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●●●
● ●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●
●●●●●
●●●●
●●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●
●●
●● ●●
●●
●●
● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●
●●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●● ● ●●● ●● ● ●● ●●● ● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●
●●●●●●●●● ●●●●●●●●●●●●●●●●●● ● ●●●
● ●●
●●●●●●●●●●●●●●●●●●●●●●
●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●●●
● ●●
● ●●
● ●●
●●●
●●●
● ●●
●●●
● ●●
● ●●
●●●
● ●●
●●●
●●
●●
●●
●●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
● ●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●
●●
●●
●
●
●●
0

0 500 1000 1500

insulin

MDA model "EII" and G=(3,5,3)

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●

● ●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●●●
● ●● ●
●● ●●
●●
●● ●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
● ●●●
●● ●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●●●
●●●●●
●●●
●●
●●
●●●●●●
●●●●●●●
●●●
●●
●●
●●
●●
●●
●●
●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●
●●
●●
●●
●●●●
●●
●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
● ●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
● ●
●●
●●
● ●
●●
● ●
●●
● ●
●●
● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●
●●
●●
●●
●●
●●
● ●
●●
●●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●● ●● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
●
●
●
●
●
●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●● ●●
●● ●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
● ●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●
●
600

●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●● ●● ●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●
●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●●
● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●
●●●●
●●
●● ●
●●
●●●●
●●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●●●
●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
● ● ●●● ●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●●●
● ●●
●●●●
●●
● ●
●●
● ●
●●
● ●
●●
● ●●
●●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
● ●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●
●● ●●
● ●●
●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●
●●●●
●●
● ●●
●●
●●
●●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●
●●●
●●
●● ●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●●●
●●
●●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●
●●●●
●●●
●●
●●●●
●●
●●●●
●●●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●●
●●
●●
●●
●●●●
●●●
●●●
●●
●●●●
●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●
●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●● ●● ●● ●●●●●● ●● ●● ●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●● ●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●●●●
●●●●
●●●●
●●●●
●●●●
●●
●●●●
●●●●
●●●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●
●●●●
●●●
400