Objectives of Multivariate Analysis

The objectives of scientific investigations to which multivariate methods

most naturally lend themselves include the following
Data reduction or structural simplification: The phenomenon being
studied is represented as simply as possible without sacrificing
valuable information. This will make interpretation easier. Example:
principal component analysis
Sorting and grouping: Groups of ”similar” objects or variables are
created, based upon measured characteristics. Example: discriminant
analysis
Investigation of the dependence among variables: Studying the
covariance structure will help to determine the nature of the
relationships among variables.
In multivariate study, the interest is on the off-diagonals
(covariances). Example: canonical correlation analysis.

Abiyot. (JU) Multivariate statistical Methods 28

prediction: The relationship between variables can be determined for
the purpose of predicting the values of one or more variables on the
basis of observations on the other variables. Example: multivariate
linear regression, multivariate analysis of variance.
Hypothesis construction and testing :Specific statistical hypotheses
can be tested to validate assumptions or to reinforce prior convictions.

Abiyot. (JU) Multivariate statistical Methods 29

 Characterizingand Displaying MultivariateData Introduction

2. Characterizing and Displaying Multivariate Data

2.1 Organization of Multivariate Data
Most multivariate data sets can be represented in a rectangular format, in
which each row correspond to the variable values and the columns
correspond to the values taken by a particular variable.

Variable1 Variable2 ··· Variablej ··· Variablep

Item1 X11 X12 ··· X1j ··· X1p

Item2 X21 X22 ··· X2j ··· X2p

.. .. .. .. .. .. ..
. . . . . . .

Itemi Xi1 Xi2 ··· Xij ··· Xip

Itemn Xn1 Xn2 ··· Xnj ··· Xnp

A single multivariate observation is the collection of measurements on p

different variables
Abiyot. (JU) Multivariate statistical Methods 31
 Characterizing and Displaying Multivariate Data Random Vectors and Matrices

2.2 Random Vectors and Matrices

Suppose X = [X1 , X2 · · · , Xp ] is a p × 1 random vectors.

Then each element of X is a random variables with its own marginal
probability distribution.
The marginal mean µi = E (Xi ), i = 1, 2, · · · , p.
The marginal variance σi2 = E (Xi − µi )2 , i = 1, 2, · · · , p
The covariance of any pair of random variables, such as Xi and Xk , is
described by their joint probability function,

σik = E (Xi − µi )(Xk − µk )

The means and covariances of p × 1 random vector X can be set out

as matrices named population variance-covariance (matrices)

µ = E (X), Σ = E (X − µ)(X − µ)0

Abiyot. (JU) Multivariate statistical Methods 32

 Characterizing and Displaying Multivariate Data Random Vectors and Matrices

Statistical independent Xi and Xk if

P(Xi ≤ xi and Xk ≤ xk ) = P(Xi ≤ xi )P(Xk ≤ xk )

or
fik (xi , xk ) = fi (xi )fk (xk )
Mutually statistically independent of the p continuous random
variables X1 , X2 , · · · , Xp if

f1,2,··· ,p (x1 , x2 , · · · , xp ) = f1 (x1 )f2 (x2 ) · · · fp (xp )

linear independent of Xi , Xk if

Cov (Xi , Xk ) = 0

Abiyot. (JU) Multivariate statistical Methods 33

 Characterizing and Displaying Multivariate Data Random Vectors and Matrices

The population variance-covariance matrix

Let x be an p × 1 random vector, i.e, x = (x1 ; x2 , · · · xp ). Then the mean
vector is:    
X µ
 1  1
   
X2  µ2 
 ..  =  ..  = µ
E (X) = E    
. .
   
Xp µp

 
σ11 σ12 · · · σ1p 
 
 
21 σ22 · · · σ2p 
σ 
0

E (X − µ)(X − µ) = 
 =Σ
 .. .. ..
.
.. 
 . . . 

 
 
σp1 σp2 · · · σpp

Abiyot. (JU) Multivariate statistical Methods 34

 Characterizing and Displaying Multivariate Data Random Vectors and Matrices

Population correlation coefficient ρik

σik
ρik = √ √
σii σkk

The correlation coefficient measures the amount of linear association

between the random variable Xi and Xk
The population correlation matrix ρ
 
 1 ρ12 · · · ρ1p 
 
 
ρ
 21 1 · · · ρ2p 

ρ=
 
 .. .
.. . .. . 
.. 
 . 
 
 
ρp1 ρp2 · · · 1

Abiyot. (JU) Multivariate statistical Methods 35

 Characterizing and Displaying Multivariate Data Partition

2.3 Partition

Let
   
X1 µ1
 .   . 
 ..   .. 
   
       
X(1) µ(1)
X  µ 
 q   q 
       
X=
 · · ·  =  · · ·  and then E (X) = µ =  · · ·  =  · · · 
      
   
Xq+1 
  X(2) µq+1 
  µ(2)
 .   . 
 ..   .. 
   
Xp µp

Abiyot. (JU) Multivariate statistical Methods 36

 Characterizing and Displaying Multivariate Data Partition

Define
 
(X(1) − µ(1) )(X(1) − µ(1) )0 (X(1) − µ(1) )(X(2) − µ(2) )0
E (X − µ)(X − µ)0 = E  
(X(2) − µ(2) (X(1) − µ(1) )0 (X(2) − µ(2) )(X(2) − µ(2) )0

 
Σ11 Σ12 
=



Σ21 Σ22

It is sometimes convenient to use Cov (X (1) , X (2) ) note where

Cov (X (1) , X (2) ) = Σ12 = Σ021

is a matrix containing all of the covariance between a component of

X (1) and a component of X (2)

Abiyot. (JU) Multivariate statistical Methods 37

 Characterizing and Displaying Multivariate Data Random sampling

2.4 Random sampling

Define
Pn
The mean, xk = n−1 j=1 Xjk , is a measure of center
1 Pn
the variance, Skk = n−1 j=1 (Xjk − xk )2 , is a measure of spread
1 Pn
the covariance, Sk` = n−1 j=1 (Xjk − xk )(Xj` − x` ), is a measure of
linear association. Sk` = S`k )
The correlation, rk` = √sskkk`s`` , is a measure of strength of linear
dependence. (rkk = 1 and rk` = r`k )
All of these summary measures are sensitive to extreme observations

Abiyot. (JU) Multivariate statistical Methods 38

 Characterizing and Displaying Multivariate Data Random sampling

Sample Covariance Matrix

 
s11 s12 ··· s1p
 
 
s21 s22 ··· s2p 
S=
 .. .. .. ..


. . . . 
 
sp1 sp2 ··· spp

Sample Correlation Matrix

   
s s12 s
√ 11 √ √ √ ··· √ 1p √ 1 r12 ··· r1p
 s11 s11 s11 s22 s11 spp   
 s21 s s   
 √s11 √s22 √ 22 √ ··· √ 2p √  r21 1 ··· r2p 
R=
s22 s22 s22 spp
=  = D−1/2 SD−1/2
 .. .. .. ..   .. .. .. .. 
 . . . .  . . . . 
   
sp1 s spp
√ √ √ p2 √ ··· √ √ rp1 rp2 ··· 1
s11 spp s22 spp spp spp

Where D−1/2 = diag ( √1s11 , · · · √1spp )

Abiyot. (JU) Multivariate statistical Methods 39
 Characterizing and Displaying Multivariate Data Random sampling

Distance

Straight-line, or Euclidean distance between points P = (x1 , x2 ) and

O = (0, 0) q
d(O, p) = x12 + x22
In general, if the point P = (x1 , x2 , · · · , xp ) and O = (y1 , y2 , · · · , yp )
q p
d(O, p) = (x1 − y1 )2 + . . . + (xp − yp )2 = (x − y)0 (x − y)

Straight line or Euclidean distance is unsatisfactory for most

statistical purposes. This is because each co-ordinate contributes
equally to the calculation of Euclidean distance.

Abiyot. (JU) Multivariate statistical Methods 40

 Characterizing and Displaying Multivariate Data Random sampling

Statistical distance: a statistical distance of the point P = (x1 ; x2 )

from the origin O = (0, 0) can be defined as
s
x12 x2
d(O, p) = + 2
s11 s22

The statistical distance from an arbitrary point P = (x1 , x2 ) to any

fixed point Q = (y1 , y2 )
s
(x1 − y1 )2 (x2 − y2 )2
d(O, p) = +
s11 s22

The extension of statistical distance to more than two dimensions

P = (x1 , x2 , · · · , xp ) to any fixed point Q = (y1 , y2 , · · · , yp )
s
(x1 − y1 )2 (x2 − y2 )2 (xp − yp )2
d(O, p) = + + ··· +
s11 s22 spp

Abiyot. (JU) Multivariate statistical Methods 41

 Characterizing and Displaying Multivariate Data Random sampling

Ellipsoids
Equation of an Ellipsoid
For A symmetric and positive definite, the equation

(x − b)0 A−1 (x − b) = c 2

defines an ellipsoid centered at b

From spectral decomposition

A = PΛP0

where Λ = diag(λ1 , · · · , λp ) with λ1 ≥ λ2 · · · ≥ λp > 0

P = (e1 , · · · , ep ) and PP0 = P0 P = I
√ p
The axes of the ellipsoid will be c λ1 e1 , · · · , c λp ep (i.e. in the
directions of the eigenvectors e1 , · · · , ep ).
p = 2 is an ellipse
√ √
c λ1 e1 is major axis and c λ2 e2 is minor axis
Abiyot. (JU) Multivariate statistical Methods 42
 Characterizing and Displaying Multivariate Data Random sampling

All point P that are a constant squared distance from Q lie on a

hyperellipsoid centered at Q whose major and minor axes are parallel
to the coordinate axes.
Any distance measure d(P, Q) between two points P and Q is valid
provided that it satisfies the following properties, where R is any
other intermediate point:
(a) d(P, Q) = d(Q, P)
(b) d(P, Q) > 0 if P 6= Q
(c) d(P, Q) = 0 if P = Q
(d) d(P, Q) ≤ d(P, R) + d(R, Q)

Abiyot. (JU) Multivariate statistical Methods 43

 Characterizing and Displaying Multivariate Data Linear Combinations of Random Vectors

2.5 Linear Combinations of Random Vectors

The linear combination c0 X = c1 X1 + · · · + cp Xp has
mean = E (c0 X) = c0 µ
Variance = Var (c0 X) = c0 Σc
Let C be a matrix, then the linear combinations of Z = CX have
µZ = E (Z) = E (CX) = CµX
ΣZ = Cov (Z) = Cov (CX) = CΣX C0
c0

Example: C2×p = c10 . Then the sample mean of Z is
2
c0 x

Cx = c01 x . The sample covariance matrix is
2
 
0 0
c1 Sc1 c1 Sc2 
0
CSC = 
 

c02 Sc1 c02 Sc2

Abiyot. (JU) Multivariate statistical Methods 44

 The Multivariate Normal Distribution

Moments of x and S

Theorem

(i) E (x) = µ

(ii) Cov (x) = n−1 Σ

(iii) E (S) = Σ

Pn
Proof: (i) and (ii) are trivial. For (iii), observe that i=1 (xi − x) = 0
and

n
X n
X
(xi − x)(xi − x)0 = (xi − µ)(xi − µ)0 − n(x − µ)(x − µ)0
i=1 i=1

Abiyot. (JU) Multivariate statistical Methods 45