ART Onvex Optimization
ART Onvex Optimization
C ONVEX OPTIMIZATION
85
Convex sets and functions
Convex set
A set C Rn is convex if for each x, y C and 0 1, one has
x + (1 )y C.
Example
Set on the left is convex, on the right non-convex
86
Convex functions
Convex function
f : Rn R is convex function, if domain of f is convex and for each
x, y domain(f ) and 0 1 one has
(y, f (y))
(x, f (x))
Example
k k (any norm) is a convex function, since k xk = || kxk and
kx + yk kxk + kyk. Thus kx + (1 )yk kxk + (1 )kyk.
87
Some operations preserving convexity
Lemma 3.1
If f1 , . . . , fn : Rn R are convex functions over the same domain and
w1 , . . . , wn 0 are non-negative weights, then ni=1 wi fi is convex
P
function.
Lemma 3.2
If f : Rn R is convex A Rnm and b Rn , then g : Rm R with
g(x) = f (A x + b) is convex.
88
Reminder
A symmetric matrix Q Rnn is called positive semi-definite if
xT Qx 0 for each x Rn .
Theorem 3.3
Let Q Rnn be a symmetric matrix. The following are equivalent.
i) Q is positive definite.
ii) All Eigenvalues of Q real and non-negative
iii) Q = U T diag(1 , . . . , n )U, where U Rnn is an orthogonal
matrix and i R0 for i = 1, . . . , n.
Lemma 3.4
Let Q Rnn be symmetric and positive semidefinite, then
f (x) = xT Qx is convex.
89
Sublevel sets
Definition C
f : Rn R convex and R, C = {x Rn : f (x) } is -sublevel
set of f .
Lemma 3.5
If f is convex, then C is a convex set for each R.
Epigraph
f : Rn R convex, epi(f ) = {(x, t) : x domain(f ), f (x) t} is
epigraph of f .
Lemma 3.6
f is convex if and only if epi(f ) is convex set.
90
Convex optimization problem
Convex optimization problem
A convex optimization problem is of the form
minimize f0 (x)
subject to fi (x) bi for i = 1, . . . , m,
Corollary 3.7
Q Rnn positive semidefinite, c Rn A Rmn and b Rm . Convex
quadratic program
min xT Qx + cT x
Ax = b
x 0,
is convex optimization problem.
91
Binary search for minimum
Search smallest R such that convex set
C = {x Rn : f0 (x) , f1 (x) b1 , . . . , fm (x) bm } is non-empty.
Keep upper bound U and lower bound L
Test: Whether C(L+U)/2 = ;. If yes, then L := (L + U)/2. If no,
then U := (L + U)/2.
After O(log ((U L)/) many steps, one obtains a value of
distance from the optimum value.
92
Separating hyperplane
Theorem 3.8
If S Rn is closed and convex and x S, then there exists a
hyperplane cT x = such that cT s < for each s S and cT x > .
93
Balls and ellipsoids
Clearly
E(A, b) = {x Rn | kA1 x A1 bk 1}. (14)
Exercise
Consider the mapping t(x) = 12 35 x(1)
x(2) . Draw the ellipsoid which is
defined by t. What are the axes of the ellipsoid?
95
Proof
Let x be contained in the unit ball, i.e., kxk 1 and suppose further
that 0 x(1) holds. We need to show that
2 2
n2 1 X
n
n+1 1
x(1) + x(i)2 1 (16)
n n+1 n2 i=2
Pn 2
holds. Since i=2 x(i) 1 x(1)2 holds we have
2 2
n2 1 Xn
n+1 1
x(1) + x(i)2
n n+1 n2 i=2
n+1 2 1 2 n2 1
x(1) + (1 x(1)2 )
n n+1 n2
(17)
96
Proof cont.
Now consider the right-hand-side of (17) as a function of x(1), i.e.,
consider
n+1 2 1 2 n2 1
f (x(1)) = x(1) + (1 x(1)2 ). (18)
n n+1 n2
We have f (0) < 0 and since both f (0) = 1 and f (1) = 1, we have
f (x(1)) 1 for all 0 x(1) 1 and the assertion follows.
97
Corollary 3.10
98
S Rn convex compact set. Suppose the following:
I) We have an ellipsoid Einit which contains S.
II) We have separation oracle for S
99
Theorem 3.12
The ellipsoid method computes a point in S or asserts that vol(S) < L.
The number of iterations is bounded by 2 n ln(vol(Einit )/L).
100