Support Vector machines

Arthur Gretton
January 26, 2016

1 Outline
• Review of convex optimization
• Support vector classification. C-SV and ν-SV machines

2 Review of convex optimization

This review covers the material from [1, Sections 5.1-5.5]. We begin with defi-
nitions of convex functions and convex sets.

2.1 Convex sets, convex functions

Definition 1 (Convex set). A set C is convex if for all x1 , x2 ∈ C and any
0 ≤ θ ≤ 1 we have θx1 + (1 − θ)x2 ∈ C, i.e. every point on the line between x1
and x2 lies in C. See Figure 2.1.
In other words, every point in the set can be seen from any other point in
the set, along a straight line that never leaves the set.
We next introduce the notion of a convex function.
Definition 2 (Convex function). A function f is convex if its domain domf is
a convex set and if ∀x, y ∈ domf , and any 0 ≤ θ ≤ 1,
f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y).

Figure 2.1: Examples of convex and non-convex sets (taken from [1, Fig. 2.2]).
The first set is convex, the last two are not.

1
 Figure 2.2: Convex function (taken from [1, Fig. 3.1])

The function is strictly convex if the inequality is strict for x 6= y. See Figure
2.2.

2.2 The Lagrangian

We now consider an optimization problem on x ∈ Rn ,

minimize f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . , m (2.1)
hi (x) = 0 i = 1, . . . p.
Tm Tp
We define by p∗ the optimal value of (2.1), and by D := i=0 domfi ∩ i=1 domhi ,
where we require the domain1 D to be nonempty.
The Lagrangian L : Rn × Rm × Rp → R associated with problem (2.1) is
written
m
X p
X
L(x, λ, ν) := f0 (x) + λi fi (x) + νi hi (x),
i=1 i=1
m p
and has domain domL := D×R ×R . The vectors λ and ν are called lagrange
multipliers or dual variables. The Lagrange dual function (or just “dual
function”) is written
g(λ, ν) = inf L(x, λ, ν).
x∈D

If this is unbounded below, then the dual is −∞. The domain of g, dom(g),
is the set of values (λ, µ) such that g > −∞. The dual function is a pointwise
infimum of affine2 functions of (λ, ν), hence it is concave in (λ, ν) [1, p. 83].
When3 λ  0, then for all ν we have

g(λ, ν) ≤ p∗ . (2.2)
1 The domain is the set on which a function is well defined. Eg the domain of log x is R++ ,
the strictly positive real numbers [1, p. 639].
2 A function f : Rn → Rm is affine if it takes the form f (x) = Ax + b.
3 The notation a  b for vectors a, b means that a ≥ b for all i.
i i

2
Figure 2.3: Example: Lagrangian with one inequality constraint, L(x, λ) =
f0 (x) + λf1 (x), where x here can take one of four values for ease of illustration.
The infimum of the resulting set of four affine functions is concave in λ.

3
 See Figure (2.4) for an illustration on a toy problem with a single inequality
constraint. A dual feasible pair (λ, ν) is a pair for which λ  0 and (λ, ν) ∈
dom(g).
Proof. (of eq. (2.2)) Assume x̃ is feasible for the optimization, i.e. fi (x̃) ≤ 0,
hi (x̃) = 0, x̃ ∈ D, λ  0. Then
m
X p
X
λi fi (x̃) + νi hi (x̃) ≤ 0
i=1 i=1

and so
p
m
!
X X
g(λ, ν) := inf f0 (x) + λi fi (x) + νi hi (x)
x∈D
i=1 i=1
m
X p
X
≤ f0 (x̃) + λi fi (x̃) + νi hi (x̃)
i=1 i=1
≤ f0 (x̃).

This holds for every feasible x̃, hence (2.2) holds.

We now give a lower bound interpretation. Idealy we would write the prob-
lem (2.1) as the unconstrained problem
m
X p
X
minimize f0 (x) + I− (fi (x)) + I0 (hi (x)) ,
i=1 i=1

where (
0 u≤0
I− (u) =
∞ u>0
and I0 (u) is the indicator of 0. This would then give an infinite penalization
when a constraint is violated. Instead of these sharp indicator functions (which
are hard to optimize), we replace the constraints with a set of soft linear con-
straints, as shown in Figure 2.5. It is now clear why λ must be positive for the
inequality constraint: a negative λ would not yield a lower bound. Note also
that as well as being penalized for fi > 0, the linear lower bounds reward us for
achieving fi < 0.

2.3 The dual problem

The dual problem attempts to find the best lower bound g(λ, ν) on the optimal
solution p∗ of (2.1). This results in the Lagrange dual problem

maximize g(λ, ν)
subject to λ  0. (2.3)

4
Figure 2.4: Illustration of the dual function for a simple problem with one
inequality constraint (from [1, Figs. 5.1 and 5.2]). In the right hand plot, the
dashed line corresponds to the optimum p∗ of the original problem, and the
solid line corresponds to the dual as a function of λ. Note that the dual as a
function of λ is concave.

5
Figure 2.5: Linear lower bounds on indicator functions. Blue functions represent
linear lower bounds for different slopes λ and ν, for the inequality and equality
constraints, respectively.

6
We use dual feasible to describe (λ, ν) with λ  0 and g(λ, ν) > −∞. The
solutions to the dual problem are written (λ∗ , ν ∗ ), and are called dual optimal.
Note that (2.3) is a convex optimization problem, since the function being max-
imized is concave and the constraint set is convex. We denote by d∗ the optimal
value of the dual problem. The property of weak duality always holds:
d∗ ≤ p ∗ .
The difference p∗ − d∗ is called the optimal duality gap. If the duality gap is
zero, then strong duality holds:
d∗ = p ∗ .
Conditions under which strong duality holds are called constraint qualifica-
tions. As an important case: strong duality holds if the primal problem is
convex,4 i.e. of the form
minimize f0 (x)
subject to fi (x) ≤ 0 i = 1, . . . , n (2.4)
Ax = b
for convex f0 , . . . , fm , and if Slater’s condition holds: there exists some
strictly feasible point5 x̃ ∈ relint(D) such that
fi (x̃) < 0 i = 1, . . . , m Ax̃ = b.
A weaker version of Slater’s condition is sufficient for strong convexity when
some of the constraint functions f1 , . . . , fk are affine (note the inequality con-
straints are no longer strict):
fi (x̃) ≤ 0 i = 1, . . . , k fi (x̃) < 0 i = k + 1, . . . , m Ax̃ = b.
A proof of this result is given in [1, Section 5.3.2].

2.4 A saddle point/game characterization of weak and

strong duality
In this section, we ignore equality constraints for ease of discussion. We write
the solution to the primal problem as an optimization
m
!
X
sup L(x, λ) = sup f0 (x) + λi fi (x)
λ0 λ0 i=1
(
f0 (x) fi (x) ≤ 0, i = 1, . . . , m
=
∞ otherwise.
4 Strong duality can also hold for non-convex problems: see e.g. [1, p. 229].
5 We denote by relint(D) the relative interior of the set D. This looks like the interior of
the set, but is non-empty even when the set is a subspace of a larger space. See [1, Section
2.1.3] for the formal defintion.

7
In other words, we recover the primal problem when the inequality constraint
holds, and get infinity otherwise. We can therefore write
p∗ = inf sup L(x, λ).
x λ0

We already know
d∗ = sup inf L(x, λ).
λ0 x
Weak duality therefore corresponds to the max-min inequality:
sup inf L(x, λ) ≤ inf sup L(x, λ). (2.5)
λ0 x x λ0

which holds for general functions, and not just L(x, λ). Strong duality occurs
at a saddle point, and the inequality becomes an equality.
There is also a game interpretation: L(x, λ) is a sum that must be paid
by the person ajusting x to the person adjusting λ. On the right hand side of
(2.5), player x plays first. Knowing that player 2 (λ) will maximize their return,
player 1 (x) chooses their setting to give player 2 the worst possible options
over all λ. The max-min inequality says that whoever plays second has the
advantage.

2.5 Optimality conditions

If the primal is equal to the dual, we can make some interesting observations
about the duality constraints. Denote by x∗ the optimum solution of the original
problem (the minimum of f0 under its constraints), and by (λ∗ , ν ∗ ) the solutions
to the dual. Then
f0 (x∗ ) = g(λ∗ , ν ∗ )
p
m
!
X X
= inf f0 (x) + λ∗i fi (x) + νi∗ hi (x)
(a) x∈D
i=1 i=1
m
X p
X
≤ f0 (x∗ ) + λ∗i fi (x∗ ) + νi∗ hi (x∗ )
(b)
i=1 i=1
≤ f0 (x∗ ),
where in (a) we use the definition of g, in (b) we use that inf x∈D of the expression
in the parentheses is necessarily no greater than its value at x∗ , and the last line
we use that at (x∗ , λ∗ , ν ∗ ) we have λ∗  0, fi (x∗ ) ≤ 0, and hi (x∗ ) = 0. From
this chain of reasoning, it follows that
m
X
λ∗i fi (x∗ ) = 0, (2.6)
i=1

which is the condition of complementary slackness. This means

λ∗i > 0 =⇒ fi (x∗ ) = 0,
fi (x∗ ) < 0 =⇒ λ∗i = 0.

8
Consider now the case where the functions fi , hi are differentiable, and the
duality gap is zero. Since x∗ minimizes L(x, λ∗ , ν ∗ ), the derivative at x∗ should
be zero,
m
X p
X
∇f0 (x∗ ) + λ∗i ∇fi (x∗ ) + νi∗ ∇hi (x∗ ) = 0.
i=1 i=1
We now gather the various conditions for optimality we have discussed. The
KKT conditions for the primal and dual variables (x, λ, ν) are
fi (x) ≤ 0, i = 1, . . . , m
hi (x) = 0, i = 1, . . . , p
λi ≥ 0, i = 1, . . . , m
λi fi (x) = 0, i = 1, . . . , m
m
X p
X
∇f0 (x) + λi ∇fi (x) + νi ∇hi (x) = 0
i=1 i=1

If a convex optimization problem with differentiable objective and constraint

functions satisfies Slater’s conditions, then the KKT conditions are necessary
and sufficient for global optimality.

3 The representer theorem

This description comes from Lecture 8 of Peter Bartlett’s course on Statistical
Learning Theory.
We are given a set of paired observations (x1 , y1 ), . . . (xn , yn ) (the setting
could be regression or classification). We consider problems of a very general
type: we want to find the function f in the RKHS H which satisfies
J(f ∗ ) = min J(f ), (3.1)
f ∈H

where  
2
J(f ) = Ly (f (x1 ), . . . , f (xn )) + Ω kf kH ,
Ω is non-decreasing, and y is the vector of yi . Examples of loss functions might
be
Pn
• Classification: Ly (f (x1 ), . . . , f (xn )) = i=1 Iyi f (xi )≤0 (the number of
points for which the sign of y disagrees with that of the prediction f (x)),
Pn
• Regression: Ly (f (x1), . . . , f(xn )) = i=1 (yi −f (xi ))2 , the sum of squared
2 2
errors (eg. when Ω kf kH = kf kH , we are back to the standard ridge
regression setting).
The representer theorem states that a solution to 3.1 takes the form
n
X
f∗ = αi k(xi , ·).
i=1

9
If Ω is strictly increasing, all solutions have this form.
Proof: We write as fs the projection of f onto the subspace
span {k(xi , ·) : 1 ≤ i ≤ n} , (3.2)
such that
f = fs + f⊥ ,
Pn
where fs = i=1 αi k(xi , ·). Consider first the regularizer term. Since
2 2 2 2
kf kH = kfs kH + kf⊥ kH ≥ kfs kH ,
then    
2 2
Ω kf kH ≥ Ω kfs kH ,
so this term is minimized for f = fs . Next, consider the individual terms f (xi )
in the loss. These satisfy
f (xi ) = hf, k(xi , ·)iH = hfs + f⊥ , k(xi , ·)iH = hfs , k(xi , ·)iH ,
so
Ly (f (x1 ), . . . , f (xn )) = Ly (fs (x1 ), . . . , fs (xn )).
Hence the loss L(. . .) only depends on the component of f in the subspace 3.2,
and the regularizer Ω(. . .) is minimized when f = fs . If Ω is strictly non-
decreasing, then kf⊥ kH = 0 is required at the minimum, otherwise this may be
one of several minima.

4 Support vector classification

4.1 The linearly separable case
We consider problem of classifying two clouds of points, where there exists a
hyperplane which linearly separates one cloud from the other without error.
This is illustrated in Figure (4.1). As can be seen, there are infinitely many
possible hyperplanes that solve this problem: the question is then: which one
to choose? We choose the one which has the largest margin: it is the largest
possible distance from both classes, and the smallest distance from each class
to the separating hyperplane is called the margin.
This problem can be expressed as follows:6
 
2
max (margin) = max (4.2)
w,b w,b kwk
6 It’s easy to see why the equation below is the margin (the distance between the positive

and negative classes): consider two points x+ and x− of opposite label, located on the margins.
The width of the margin, dm , is the difference x+ − x− projected onto the unit vector in the
direction w, or
w
dm = (x+ − x− )> (4.1)
kwk
Subtracting the two equations in the constraints (4.3) from each other, we get
w> (x+ − x− ) = 2.
Substituting this into (4.1) proves the result.

10
 11

Figure 4.1: The linearly separable case. There are many linear separating hy-
perplanes, but only one max. margin separating hyperplane.
subject to (
min w> xi + b = 1


i : yi = +1,
(4.3)
max w> xi + b = −1


i : yi = −1.
The resulting classifier is
y = sign(w> x + b),
where sign takes value +1 for a positive argument, and −1 for a negative ar-
gument (its value at zero is not important, since for non-pathological cases we
will not need to evaluate it there). We can rewrite to obtain
1
max or min kwk2
w,b kwk w,b

subject to
yi (w> xi + b) ≥ 1. (4.4)

4.2 When no linear separator exists (or we want a larger

margin)
If the classes are not linearly separable, we may wish to allow a certain number
of errors in the classifier (points within the margin, or even on the wrong side
of the decision boudary). We therefore want to trade off such “margin errors”
vs maximising the margin. Ideally, we would optimise
n
!
1 2
X
>


min kwk + C I[yi w xi + b < 1] ,
w,b 2 i=1

where C controls the tradeoff between maximum margin and loss, and I(A) = 1
if A holds true, and 0 otherwise (the factor of 1/2 is to simplify the algebra later,
and is not important: we can adjust C accordingly). This is a combinatorical
optimization problem, which would be very expensive to solve. Instead, we
replace the indicator function with a convex upper bound,
n
!
1 2
X
>



min kwk + C θ yi w xi + b .
w,b 2 i=1

We use the hinge loss,

(
1−α 1−α>0
θ(α) = (1 − α)+ =
0 otherwise.

although obviously other choices are possible (e.g. a quadratic upper bound).
See Figure 4.2.
Substituting in the hinge loss, we get
n
!
1 X
kwk2 + C θ yi w > xi + b



min .
w,b 2 i=1

12
Figure 4.2: The hinge loss is an upper bound on the step loss.

13
Figure 4.3: The nonseparable case. Note the red point which is a distance ξ/kwk
from the margin.

or equivalently the constrained problem

n
!
1 X
min kwk2 + C ξi (4.5)
w,b,ξ 2 i=1

subject to7
yi w> xi + b ≥ 1 − ξi


ξi ≥ 0
(compare with (4.4)). See Figure 4.3.
Now let’s write the Lagrangian for this problem, and solve it.
n n n
1 X X
X
kwk2 +C αi 1 − yi w> xi + b − ξi +


L(w, b, ξ, α, λ) = ξi + λi (−ξi )
2 i=1 i=1 i=1
(4.6)
7 To see this, we can write it as ξi ≥ 1 − yi w> xi + b . Thus either ξi = 0, and

w> x


yi i + b ≥ 1 as before, or ξi > 0, in which case to minimize (4.5), we’d use the smallest
possible ξi satisfying the inequality, and we’d have ξi = 1 − yi w> xi + b .

14
with dual variable constraints

αi ≥ 0, λi ≥ 0.

We minimize wrt the primal variables w, b, and ξ.

Derivative wrt w:
n n
∂L X X
=w− αi yi xi = 0 w= αi yi xi . (4.7)
∂w i=1 i=1

Derivative wrt b:
∂L X
= yi αi = 0. (4.8)
∂b i

Derivative wrt ξi :
∂L
= C − αi − λi = 0 αi = C − λi . (4.9)
∂ξi
We can replace the final constraint by noting λi ≥ 0, hence

αi ≤ C.

Before writing the dual, we look at what these conditions imply about the scalars
αi that define the solution (4.7).
Non-margin SVs: αi = C:
Remember complementary slackness:
1. We immediately have 1 − ξi = yi w> xi + b .

2. Also, from condition αi = C − λi , we have λi = 0, hence it is possible that

ξi > 0.
Margin SVs: 0 < αi < C:
1. We again have 1 − ξi = yi w> xi + b

2. This time, from αi = C − λi , we have λi 6= 0, hence ξi = 0.

Non-SVs: αi = 0
1. This time we have: yi w> xi + b > 1 − ξi

2. From αi = C − λi , we have λi 6= 0, hence ξi = 0.

This means that the solution is sparse: all the points which are not either on
the margin, or “margin errors”, contribute nothing to the solution. In other
words, only those points on the decision boundary, or which are margin errors,
contribute. Furthermore, the influence of the non-margin SVs is bounded, since
their weight cannot exceed C: thus, severe outliers will not overwhelm the
solution.

15
 We now write the dual function, by substituting equations (4.7), (4.8), and
(4.9) into (4.6), to get

n n n
1 X X
X
kwk2 + C αi 1 − yi w> xi + b − ξi +


g(α, λ) = ξi + λi (−ξi )
2 i=1 i=1 i=1
m m m m X m m
1 XX X X X
= αi αj yi yj x> x
i j + C ξ i − α α y y x >
i j i j i jx − b αi yi
2 i=1 j=1 i=1 i=1 j=1 i=1
| {z }
0
m
X m
X m
X
+ αi − αi ξi − (C − αi )ξi
i=1 i=1 i=1
m m m
X 1 XX
= αi − αi αj yi yj x>
i xj .
i=1
2 i=1 j=1

Thus, our goal is to maximize the dual,

m m m
X 1 XX
g(α) = αi − αi αj yi yj x>
i xj ,
i=1
2 i=1 j=1

subject to the constraints

n
X
0 ≤ αi ≤ C, yi αi = 0.
i=1

So far we have defined the solution for w, but not for the offset
b. This is simple
to compute: for the margin SVs, we have 1 = yi w> xi + b . Thus, we can
obtain b from any of these, or take an average for greater numerical stability.

4.3 Kernelized version

We can straightforwardly define a maximum margin classifier in feature space.
We write the original hinge loss formlation (ignoring the offset b for simplicity):
n
!
1 2
X
min kwkH + C (1 − yi hw, k(xi , ·)iH )+
w 2 i=1

for the RKHS H with kernel k(x, ·). When we kernelize, we use the result of the
representer theorem,
Xn
w= βi k(xi , ·). (4.10)
i=1

In this case, maximizing the margin is equivalent to minimizing kwk2H : as we

have seen, for many RKHSs (e.g. the RKHS corresponding to a Gaussian ker-
nel), this corresponds to enforcing smoothness.

16
 Substituting (4.10) and introducing the ξi variables, get
n
!
1 > X
min β Kβ + C ξi (4.11)
w,b 2 i=1

where the matrix K has i, jth entry Kij = k(xi , xj ), subject to

n
X
ξi ≥ 0 yi βj k(xi , xj ) ≥ 1 − ξi
j=1

Thus, the primal variables w are replaced with β. The problem remains convex
since K is positive definite. With some calculation (exercise!), the dual becomes
m m m
X 1 XX
g(α) = αi − αi αj yi yj k(xi , xj ),
i=1
2 i=1 j=1

subject to the constraints

0 ≤ αi ≤ C,
and the decision function takes the form
n
X
w= yi αi k(x, ·).
i=1

4.4 The ν-SVM

It can be hard to interpret C. Therefore we modify the formulation to get a
more intuitive parameter. Again, we drop b for simplicity. Solve
n
!
1 2 1X
min kwk − νρ + ξi
w,ρ,ξ 2 n i=1

subject to

ρ ≥ 0
ξi ≥ 0
>
yi w xi ≥ ρ − ξi ,

where we see that we now optimize the margin width ρ. Thus, rather than
choosing C, we now choose ν; the meaning of the latter will become clear shortly.
The Lagrangian is
n n n
1 2 1X X
>

X
kwkH + ξi − νρ + α i ρ − yi w xi − ξi + βi (−ξi ) + γ(−ρ)
2 n i=1 i=1 i=1

17
for αi ≥ 0, βi ≥ 0, and γ ≥ 0. Differentiating wrt each of the primal variables
w, ξ, ρ, and setting to zero, we get
n
X
w = αi yi xi
i=1
1
αi + βi = (4.12)
n
Xn
ν = αi − γ (4.13)
i=1

From βi ≥ 0, equation (4.12) implies

0 ≤ αi ≤ n−1 .

From γ ≥ 0 and (4.13), we get

n
X
ν≤ αi .
i=1

Let’s now look at the complementary slackness conditions.

Assume ρ > 0 at the global solution, hence γ = 0, and (4.13) becomes
n
X
αi = ν. (4.14)
i=1

1. Case of ξi > 0: then complementary slackness states βi = 0, hence from

(4.12) we have αi = n−1 for these points. Denote this set as N (α). Then
X 1 X Xn
= αi ≤ αi = ν,
n i=1
i∈N (α) i∈N (α)

so
|N (α)|
≤ ν,
n
and ν is an upper bound on the number of non-margin SVs.
2. Case of ξi = 0. Then αi < n−1 . Denote by M (α) the set of points
n−1 > αi > 0. Then from (4.14),
n X 1
X X X 1
ν= αi = + αi ≤ ,
i=1
n n
i∈N (α) i∈M (α) i∈M (α)∪N (α)

thus
|N (α)| + |M (α)|
ν≤ ,
n
and ν is a lower bound on the number of support vectors with non-zero
weight (both on the margin, and “margin errors”).

18
Substituting into the Lagrangian, we get
m m n m X m n n
1 XX > 1X X
>
X X
αi αj yi yj xi xj + ξi − ρν − αi αj yi yj xi xj + αi ρ − αi ξi
2 i=1 j=1 n i=1 i=1 j=1 i=1 i=1
n   n
!
X 1 X
− − αi ξi − ρ αi − ν
i=1
n i=1
m m
1 XX
=− αi αj yi yj x>
i xj
2 i=1 j=1

Thus, we must maximize

m m
1 XX
g(α) = − αi αj yi yj x>
i xj ,
2 i=1 j=1

subject to
n
X 1
αi ≥ ν 0 ≤ αi ≤ .
i=1
n

References
[1] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge
University Press, Cambridge, England, 2004.

19