Tutorial on Multivariate Logistic Regression

Javier R. Movellan
July 23, 2006

1
1 Motivation
The last few years have seen a resurgence of interest in the Machine Learning community for main-
effect models, i.e., models without interaction effects. This is due to the emergence of learning
algorithms, like SVMs and ADABoost that work well with very high dimensional representations,
avoiding the need for modeling interaction effects. The goal of this tutorials is to introduce the most
important main effect regression models, how they relate to each other and common algorithms
used for training them.

2 General Model
Our goal is to train a well defined model based on examples of input-output pairs. In our notation
the inputs will be n-dimensional, the outputs will be c-dimensional and our training sample will
consist of m input output pairs. For convenience we organize the example inputs as an m × n
matrix x. The corresponding example outputs are organized as a c × m matrix y. Our goal is to
predict the rows of y using the rows of x. The models under consideration produce a intermediary
linear transformation of the inputs and make predictions about y based on that transformation. Let
λ be an n × c matrix and let ŷ = xλ represent the intermediary linear transformation of the data.
We will evaluate the optimality of ŷ, and thus of λ, using the following criterion
m c
X αX T
Φ(λ) = − wj ρj (y, ŷ) + λk λk (1)
j=1
2 k=1

where yj· , ŷj· represent the j th rows of y and ŷ, ρj is an error

Pc function that2measures the mismatch
between the j rows of y and ŷ. For example ρj (y, ŷ) = k=1 (yj,k − ŷj,k ) . We let ρ = (ρ1 , ρm )T .
th

The terms w1 , . . . , wm are positive weights that capture the relative importance of each of the m
input-output pairs, and α is a positive constant. Informally, the first term can be seen as a negative
log-likelihood function, capturing the degree of match between the data and the model, the second
term can be interpreted as a negative log prior over λ.
A standard approach for minimizing Φ is the Newton-Raphson algorithm (See Appendix B),
which calls for computation of the gradient and the Hessian matrix (See Appendix A).
Let λi represent the ith row of λ. The gradient can be computed using the chain rule (See
Appendix A)
c
αX T
∇λi λ λk = αλi (2)
2 k=1 k
m
X m
X
∇λi wj ρj (y, ŷ) = ∇λi ŷi ∇ŷi ρ∇ρ wj ρj (y, ŷ) = xT wΨi (3)
j=1 j=1

2
where w is a diagonal matrix with diagonal elements w1 , . . . , wm and
 ∂ρ1 
∂ ŷ1i
Ψi = 
 .. 
(4)
. 
∂ρm
∂ ŷmi

Thus
∇λi Φ = xT wΨi + αλi (5)
The overall gradient vector follows ∇λ Φ = (∇λ1 Φ, . . . , ∇λn Φ)T . The chain rule can be used to
compute the component matrices of the overall Hessian matrix

∇λj ∇λi Φ = ∇λj αλi + ∇λj xT wΨi (6)

∇λj αλi = δij αIn (7)

where δik is the Kronecker delta function (0 valued unless i = k in which case it takes value 1) and
In is the n × n identity matrix.

∇λj xT wΨi = ∇λj ŷj ∇ŷj Ψi ∇Ψi xT wΨi = xT Ψ0ij wx (8)

and  
∂ 2 ρ1
∂ ŷ1j ∂ ŷ1i
0 ··· 0

∂ 2 ρ2 (y,ŷ) .. 
 0 . 0 
Ψ0ij =  ∂ ŷ2j ∂ ŷ2i
(9)
 
.. ... .. 

 . ··· . 

∂ 2 ρm
0 0 ··· ∂ ŷmj ∂ ŷmi

Thus
∇λj ∇λi Φ = xT Ψ0ij wx + δij αIn (10)
The overall Hessian matrix follows
 
∇λ1 ∇λ1 Φ · · · ∇λ1 ∇λ1 Φ
∇λ ∇λ Φ = 
 .. ... .. 
(11)
. . 
∇λn ∇λ1 Φ · · · ∇λn ∇λn Φ

3 Linear Ridge Regression

Without loss of generality we will examine the case with a single response variable, i.e., c = 1.
Our error function will be the sum of squared errors, i.e,
1
ρj (y, ŷ) = (yj − ŷj )2 (12)
2

3
Thus  ∂ρ1   
∂ ŷ1i y1 − ŷ1
Ψi = −  .. ..
=  = y − xλ (13)
   
. .
∂ρm
∂ ŷmi
ym − ŷm
The gradient follows
∇λ Φ = xT w(y − xλ) + αλ (14)
Setting the equation to zero results on a linear equation on λ, that can be solved analytically

−1 T
λ̂ = xT wx + αIn x wy (15)
where λ̂ represents the optimal value of λ.

4 Robust Regression
m
α T X
λ λ+ wj ρ(yj − ŷj ) (16)
2 j=1

where ŷ = xλ. Where ρ typically penalizes extreme deviations between y and ŷ less than the
squared error function does. It is easy to show that in this case
∇λ Φ = −xT wΨ + αλ (17)
where Ψj = ρ0 (yj − ŷj ) the derivative of the error function. This derivative is known as the
influence function for it controls the influence of each example on the final solution. Note if
ρ(yj ŷj ) = (yh − ŷj )2 then Ψ = y − ŷ. If ρ has no second derivative, a common procedure to
find an optimal λ involves the use of the following heuristic. Note that (31) can be expressed as
follows:
∇λ Φ = −xt w̃(y − ŷ) + λ (18)
where w̃ is an m × m diagonal matrix such that wjj = wj Ψj /(yj − ŷj ). Setting this gradient to
zero, and disregarding the fact that w is a function of λ, results on a linear weighted least squares
problem whose solution we have already seen. This iterative version of weighted least squares we
start with an arbitrary value λ and apply the following iteration:
λt+1 = (xt w̃t x + αIn )−1 xt w̃t y (19)
where wt is the matrix of weights obtained using λt .
If ρ has second derivative, then the Hessian matrix exists and has the following form
∇λ ∇λ Φ = xT Ψ0 wx + αIn (20)
00
Where Ψ is an m × m diagonal matrix such that Ψjj = ρ (yj − ŷj ), the second derivative of the
error function. The Newton-Raphson algorithm calls for the following iteration
λt+1 = λt + (xt Ψ0 x)−1 xt Ψ (21)
which can also be casted as weighted least squares solution with weight matrix Ψ0 and desired
response vector z = xλt + Ψ−1 xT Ψ

4
5 Multinomial Logistic Regression
Let c
X
ρj (y, ŷ) = yjk log hk (ŷj ) (22)
k=1

where
eŷjk
hk (ŷj ) = Pc ŷji
(23)
i=1 e

Note for any u = (u1 , . . . , uc )T ∈ Rc

∂hk (u) ∂ log hk (u)

= hk (u) = hk (u)(δik − hi (u)) (24)
∂ui ∂ui
Thus c
∂ρj (y, ŷ) X yjk
= hk (ŷjk )(δik − h(ŷji )) = yji − hi (ŷji ) (25)
∂ ŷji k=1
hk (ŷjk )

Ψi = yi − hi (ŷ) (26)
and
ŷ1k (δki − ŷ1i ) ···
 
0 0
...
0 ŷ2k (δki − ŷ2i ) 0
 
Ψ0ki = (27)
 
.. ... .. 
 . ··· . 
0 0 · · · ŷmk (δki − ŷmi )

5.1 Iteratively Reweighted Least Squares

When there are only two response alternatives and α = 0 the Newton- Raphson algorithm simpli-
fies as follows (See Appendix B)

−1
λ(t + 1) = λ(t) + xT w̃(t)x xT w(y − ŷ(t)) (28)

where λ(t) represents the n-dimensional vector of weights at iteration t, y and ŷ(t) are m-dimensional
vectors containing the desired and predicted probabilities for the first response alternative and w(t)
is an m × m matrix w = ∇xλ ŷ, i.e., wii = ŷi (1 − ŷi ). Some simple algebra shows that this iteration
can be casted as the solution to a weighted linear regression problem

−1
λ(t + 1) = xT w̃(t)x xT w̃(t)z(t) (29)

where z(t) = xλ(t) + ww̃−1 (t)(y − ŷ) is a modified target variable.

It is common practice to start the iterations at the zero vector, i.e., λ0 = 0. It is easy to show
that in such case the matrix w has constant diagonal terms and the solution obtained after the first
iteration is the standard least squares solution. Successive iterations change the weight assigned to
each example.

5
6 Robust Regression
In robust regression the criterion function is of the form
m
α X
Φ(λ) = λT λ + wj ρ(yj − ŷj ) (30)
2 j=1

where ŷ = xλ. Where ρ typically penalizes extreme deviations between y and ŷ less than the
squared error function does. It is easy to show that in this case
∇λ Φ = −xT wΨ + αλ (31)
where Ψj = ρ0 (yj − ŷj ) the derivative of the error function. This derivative is known as the
influence function for it controls the influence of each example on the final solution. Note if
ρ(yj ŷj ) = (yh − ŷj )2 then Ψ = y − ŷ. If ρ has no second derivative, a common procedure to
find an optimal λ involves the use of the following heuristic. Note that (31) can be expressed as
follows:
∇λ Φ = −xt w̃(y − ŷ) + λ (32)
where w̃ is an m × m diagonal matrix such that wjj = wj Ψj /(yj − ŷj ). Setting this gradient to
zero, and disregarding the fact that w is a function of λ, results on a linear weighted least squares
problem whose solution we have already seen. This iterative version of weighted least squares we
start with an arbitrary value λ and apply the following iteration:
λt+1 = (xt w̃t x + αIn )−1 xt w̃t y (33)
where wt is the matrix of weights obtained using λt .
If ρ has second derivative, then the Hessian matrix exists and has the following form
∇λ ∇λ Φ = xT Ψ0 wx + αIn (34)
00
Where Ψ is an m × m diagonal matrix such that Ψjj = ρ (yj − ŷj ), the second derivative of the
error function. The Newton-Raphson algorithm calls for the following iteration
λt+1 = λt + (xt Ψ0 x)−1 xt Ψ (35)
which can also be casted as weighted least squares solution with weight matrix Ψ0 and desired
response vector z = xλt + Ψ−1 xT Ψ

7 Support Vector Machines

In support vector machines (SVMs) there typically is a single response variable c = 1 and the error
function has the following form
ρ(yi , ŷi ) = − max{1 − yi ŷ, 0} (36)
where y ∈ {0, 1}. It is easy to show that as α → 0, the solution is the hyperplane that maximizes
the minimum distance from the training examples to the hyperplane.
It is interesting to note tha tthe SVM error function and the logistic error function are are very
similar, the logistic error function being a smooth version of the SVM error function.

6
8 ADABoost and the Backfitting algorithm
Under construction.

A Vector Calculus
Definition Gradient Matrix: Let y = f (x) where y ∈ Rn , x ∈ Rm . The gradient of y with
respect to x, symbolized ∇x y, is an m × n matrix defined as follows
 ∂y1
· · · ∂y n

∂x1 ∂x1
∇x y =  ... ... ..  (37)

. 
∂y1 ∂yn
∂xm
· · · ∂xm

Gradient of quadratic functions: Let y = xT ax + bx + c where x ∈ Rn . It can be shown that

∇x y = bT + (a + aT )x (38)

where aT is the transpose of a. Note y ∈ Rn . If a is symmetric then a + aT = 2a.

Chain Rule: Let y = f (x), x ∈ Rm , y ∈ Rn and z = h(y), z ∈ Ro . Then

∇x z = ∇x y ∇y z (39)

Example: Let y = (x − µ)T a(x − µ) where x, ∈ Rn and µ is a constant vector. Then

∇y x = ∇x (x − µ) ∇(x−µ) y = In a(x − µ) = a(x − µ). (40)

where In is the n × n identity matrix.

Definition Hessian Matrix: Let y = f (x), y ∈ R, x ∈ Rn . The matrix
∂2y ∂2y
 
∂x1 ∂x1
··· ∂x1 ∂xn
∇x ∇x f (x) = 
 .. .. .. 
(41)
. . . 
∂2y ∂2
∂xn ∂x1
··· ∂xn ∂xn

is called the Hessian matrix of y. Example: Let y = xT ax + bx + c then

∇x ∇x y = ∇x ax + bT = a (42)

B Newton-Raphson Optimization Method

Let y = f (x), for y ∈ R, x ∈ Rn . The Newton-Raphson algorithm is an iterative method for
optimizing y. We start the process at an arbitrary point x0 ∈ Rn . Let xt ∈ Rn represent the state

7
of the algorithm at iteration t. We approximate the function f using the linear and quadratic terms
of the Taylor expansion of f around xt .
1
fˆt (x) = f (xt ) + ∇x f (xt )(x − xt )T + (x − xt )T (∇x ∇x f (xt )) (x − xt ) (43)
2

and then we then find the extremum of fˆt with respect to x and move directly to that extremum. To
do so note that
∇x fˆt (x) = ∇x f (xt ) + (∇x ∇x f (xt )) (x − xt ) (44)
We let x(t + 1) be the value of x for which ∇x fˆt (x) = 0

xt+1 = xt + (∇x ∇x f (xt ))−1 ∇x f (xt ) (45)

It is useful to compare the Newton-Raphson method with the standard method of gradient ascent.
The gradient ascent iteration is defined as follows

xt+1 = xt + In ∇x f (xt ) (46)

where  is a small positive constant. Thus gradient descent can be seen as a Newton-Raphson
method in which the Hessian matrix is approximated by 1 In .

C Gauss-Newton Optimization Method

Let f (x) = ni=1 ri (x)2 for ri : Rn → R. We start the process with an arbitrary point x0 ∈ Rn .
P
Let xt ∈ Rn represent the state of the algorithm at iteration t. We approximate the functions ri
using the linear term of their Taylor expansion around xt .

r̂i (xt ) = ri (xt ) + (∇x ri (xt ))T (x − xt ) (47)

X n n
X
fˆt (x) = (r̂i (xt ))2 = (ri (xt ) − (∇x ri (xt ))T xt + (∇x ri (xt ))T x)2 (48)
i=1 i=1
(49)

Minimizing fˆt (x) is a linear least squares problem of well known solution. If we let yi = (∇x ri (xt ))T xt −
ri (xt ) and ui = ∇x ri (xt ) then
n
X n
X
xt+1 = ( ui uTi )−1 ( ui y i ) (50)
i=1 i=1

History
• The first version of this document was written by Javier R. Movellan, May 2002. It was 7
pages long.

8
 • The document was made open source under GNU FDL license 1.1 as part of the Kolmogorov
project on August 9 2002.

• Javier Movellan added Gauss-Newton Method on March 14, 2003.

• October 9, 2003. Javier R. Movellan changed the license to GFDL 1.2 and included an
endorsement section.

References