Convex sets Convex functions

Convex sets and convex functions

Mauro Passacantando

Department of Computer Science, University of Pisa

Largo Pontecorvo 3
mauro.passacantando@unipi.it

Optimization Methods
Master of Science in Embedded Computing Systems – University of Pisa
http://pages.di.unipi.it/passacantando/om/OM.html

M. Passacantando Optimization Methods 1 / 24 – :

Convex sets Convex functions

Subspaces

Given x, y ∈ Rn .

A linear combination of x and y is a point αx + βy , where α, β ∈ R.

A set C ⊆ Rn is a subspace if it contains all the linear combinations of any two

points in C .

Examples:
I {0}
I any line which passes through zero
I the solution set of a homogeneous system of linear equations

C = {x ∈ Rn : Ax = 0},

where A is a m × n matrix.

M. Passacantando Optimization Methods 2 / 24 – :

Convex sets Convex functions

Affine sets

An affine combination of x and y is a point αx + βy , where α + β = 1.

A set C ⊆ Rn is an affine set if it contains all the affine combinations of any two
points in C .

Examples:
I any single point {x}
I any line
I the solution set of a system of linear equations

C = {x ∈ Rn : Ax = b},

where A is a m × n matrix and b ∈ Rm

I any subspace

M. Passacantando Optimization Methods 3 / 24 – :

Convex sets Convex functions

Convex sets
A convex combination of two given points x and y is a point αx + βy , where
α + β = 1, α ≥ 0, β ≥ 0.

A set C ⊆ Rn is convex if it contains all the convex combinations of any two

points in C .

y
x

convex set non-convex set

Exercise. Prove that if C is convex, then for any x 1 , . . . , x k ∈ C and

k k
αi x i ∈ C .
P P
α1 , . . . , αk ∈ (0, 1) s.t. αi = 1, one has
i=1 i=1

M. Passacantando Optimization Methods 4 / 24 – :

Convex sets Convex functions

Convex hull

The convex hull conv(C ) of a set C is the smallest convex set containing C .

C conv(C )

Exercise. Prove that conv(C ) = {all convex combinations of points in C }.

Exercise. Prove that C is convex if and only if C = conv(C ).

M. Passacantando Optimization Methods 5 / 24 – :

Convex sets Convex functions

Convex sets - Examples

Examples:
I subspace
I affine set
I line segment
I halfspace {x ∈ Rn : aT x ≤ b}
I polyhedron P = {x ∈ Rn : Ax ≤ b} solution set of a system of linear
inequalities

M. Passacantando Optimization Methods 6 / 24 – :

Convex sets Convex functions

Convex sets - Examples

I ball B(x, r ) = {y ∈ Rn : ky − xk ≤ r }, where k · k is any norm, e.g.

s
n
xi2 (Euclidean norm)
P
kxk2 =
i=1
n
P
kxk1 = |xi | (Manhattan distance)
i=1
kxk∞ = max |xi | (Chebyshev norm)
i=1,...,n
s
Pn
kxkp = p |xi |p , with 1 ≤ p ≤ ∞
i=1
√
kxkA = x T Ax, where A is a symmetric and positive definite matrix, i.e.,

x T Ax > 0 ∀ x 6= 0.

 
2 0
Exercise. Find B(0, 1) w.r.t. k · k1 , k · k∞ and k · kA where A = .
0 1

M. Passacantando Optimization Methods 7 / 24 – :

Convex sets Convex functions

Operations that preserve convexity

Sum and difference

If C1 and C2 are convex, then C1 + C2 := {x + y : x ∈ C1 , y ∈ C2 } is convex.
If C1 and C2 are convex, then C1 − C2 := {x − y : x ∈ C1 , y ∈ C2 } is convex.

Intersection
If C1 and C2 are convex, then C1 ∩ C2 is convex. T
Exercise. If {Ci }i∈I is a family of convex sets, then Ci is convex.
i∈I

Union
If C1 and C2 are convex, then C1 ∪ C2 is convex?

Closure and interior

If C is convex, then cl(C ) is convex.
If C is convex, then int(C ) is convex.

M. Passacantando Optimization Methods 8 / 24 – :

Convex sets Convex functions

Operations that preserve convexity

Affine functions
Let f : Rn → Rm be affine, i.e. f (x) = Ax + b, with A ∈ Rm×n , b ∈ Rm .
I If C ⊆ Rn is convex, then f (C ) = {f (x) : x ∈ C } is convex
I If C ⊆ Rm is convex, then f −1 (C ) = {x ∈ Rn : f (x) ∈ C } is convex

Examples:
I scaling, e.g. f (x) = α x, with α > 0
I translation, e.g. f (x) = x + b, with b ∈ Rn
 
cos θ − sin θ
I rotation, e.g. f (x) = x, with θ ∈ (0, 2π)
sin θ cos θ

M. Passacantando Optimization Methods 9 / 24 – :

Convex sets Convex functions

Cones

A set C ⊆ Rn is a cone if α x ∈ C for any x ∈ C and α ≥ 0.

Examples:
I Rn+ is a convex cone
I {x ∈ R2 : x1 x2 = 0} is a nonconvex cone
I Given a polyhedron P = {x : Ax ≤ b}, the recession cone of P is defined as

rec(P) := {d : x + α d ∈ P for any x ∈ P, α ≥ 0}.

It is easy to prove rec(P) = {x : Ax ≤ 0}, thus it is a polyhedral cone.

p
I {x ∈ R3 : x3 ≥ x12 + x22 } is a non-polyhedral cone.

M. Passacantando Optimization Methods 10 / 24 – :

Convex sets Convex functions

Exercises

1. Write the vector (1, 1) as the convex combination of the vectors

(0, 0), (3, 0), (0, 2), (3, 2).
2. When does one halfspace contain another? Give conditions under which

{x ∈ Rn : a1T x ≤ b1 } ⊆ {x ∈ Rn : a2T x ≤ b2 },

where ka1 k2 = ka2 k2 = 1. Also find the conditions under which the two
halfspaces are equal.
3. Which of the following sets are polyhedra?
a) {y1 a1 + y2 a2 : −1 ≤ y1 ≤ 1, −1 ≤ y2 ≤ 1}, where a1 , a2 ∈ Rn .
 n n n

b) x ∈ Rn : x ≥ 0, ai2 xi = b2 , where
P P P
xi = 1, ai xi = b1 ,
i=1 i=1 i=1
b1 , b2 , a1 , . . . , an ∈ R.
c) {x ∈ Rn : x ≥ 0, aT x ≤ 1 for all a with kak2 = 1}.
d) {x ∈ Rn : x ≥ 0, aT x ≤ 1 for all a with kak1 = 1}.

M. Passacantando Optimization Methods 11 / 24 – :

Convex sets Convex functions

Convex functions
Given a convex set C ⊆ Rn , a function f : C → R is convex if
f (αy + (1 − α)x) ≤ αf (y ) + (1 − α)f (x) ∀ x, y ∈ C , ∀ α ∈ (0, 1)

f (y )

α f (y ) + (1 − α) f (x)

f (x)

f (αy + (1 − α)x)

x αy + (1 − α)x y

f is said concave if −f is convex.

Exercise. Prove that if f is convex, then for  any x 1 , . .. , x k ∈ C and

k k k
αi x i ≤ αi f (x i ).
P P P
α1 , . . . , αk ∈ (0, 1) s.t. αi = 1, one has f
i=1 i=1 i=1
M. Passacantando Optimization Methods 12 / 24 – :
Convex sets Convex functions

Strictly convex and strongly convex functions

Given a convex set C ⊆ Rn , a function f : C → R is strictly convex if

f (αy + (1 − α)x) < αf (y ) + (1 − α)f (x) ∀ x, y ∈ C , ∀ α ∈ (0, 1)

Given a convex set C ⊆ Rn , a function f : C → R is strongly convex if there exists

τ > 0 s.t.
τ
f (αy + (1 − α)x) ≤ αf (y ) + (1 − α)f (x) − α(1 − α)ky − xk2
2
∀ x, y ∈ C , ∀ α ∈ (0, 1)

τ
Thm. f is strongly convex if and only if ∃ τ > 0 s.t. f (x) − kxk2 is convex
2
Exercise.
I Prove that: strongly convex =⇒ strictly convex =⇒ convex
I convex =⇒ strictly convex ?
I strictly convex =⇒ strongly convex ?
M. Passacantando Optimization Methods 13 / 24 – :
Convex sets Convex functions

First order conditions

Assume that C ⊆ Rn is open convex and f : C → R is continuously differentiable.
Theorem
f is convex if and only if

f (y ) ≥ f (x) + (y − x)T ∇f (x) ∀ x, y ∈ C .

f (y )

f (x) + (y − x)T ∇f (x)

f (x)

x y

First-order approximation of f is a global understimator

M. Passacantando Optimization Methods 14 / 24 – :
Convex sets Convex functions

First order conditions

Theorem
I f is strictly convex if and only if

f (y ) > f (x) + (y − x)T ∇f (x) ∀ x, y ∈ C , with x 6= y .

I f is strongly convex if and only if there exists τ > 0 such that

τ
f (y ) ≥ f (x) + (y − x)T ∇f (x) + ky − xk22 ∀ x, y ∈ C .
2

M. Passacantando Optimization Methods 15 / 24 – :

Convex sets Convex functions

Second order conditions

Assume that C ⊆ Rn is open convex and f : C → R is twice continuously
differentiable.
Theorem
I f is convex if and only if for all x ∈ C the Hessian matrix ∇2 f (x) is positive
semidefinite, i.e.
v T ∇2 f (x)v ≥ 0 ∀ v 6= 0,
or, equivalently, the eigenvalues of ∇2 f (x) are ≥ 0.

I If ∇2 f (x) is positive definite for all x ∈ C , then f is strictly convex.

I f is strongly convex if and only if there exists τ > 0 such that ∇2 f (x) − τ I is
positive semidefinite for all x ∈ C , i.e.

v T ∇2 f (x)v ≥ τ kv k22 ∀ v 6= 0,

or, equivalently, the eigenvalues of ∇2 f (x) are ≥ τ .

M. Passacantando Optimization Methods 16 / 24 – :

Convex sets Convex functions

Examples

f (x) = c T x is both convex and concave

f (x) = 21 x T Qx + c T x is
I convex iff Q is positive semidefinite
I strongly convex iff Q is positive definite
I concave iff Q is negative semidefinite
I strongly concave iff Q is negative definite
f (x) = e ax for any a ∈ R is strictly convex, but not strongly convex
f (x) = log(x) is strictly concave, but not strongly concave
f (x) = x a with x > 0 is strictly convex if a > 1 or a < 0. Is it strongly convex?
f (x) = x a with x > 0 is strictly concave if 0 < a < 1
f (x) = kxk is convex, but not strictly convex
f (x) = max{x1 , . . . , xn } is convex, but not strictly convex

M. Passacantando Optimization Methods 17 / 24 – :

Convex sets Convex functions

Exercises
1. Prove that the function
x1 x2
f (x1 , x2 ) =
x1 − x2
is convex on the set {x ∈ R2 : x1 − x2 > 0}.
1
2. Prove that f (x1 , x2 ) = is convex on the set {x ∈ R2 : x1 , x2 > 0}.
x1 x2
3. Given a convex set C ⊆ Rn , the distance function is defined as follows:

dC (x) = inf ky − xk.

y ∈C

Prove that dC is a convex function.

4. Given C = {x ∈ R2 : x12 + x22 ≤ 1}, write the distance function dC explicitly.
5. Prove that the arithmetic mean of n positive numbers x1 , . . . , xn is greater or
equal to their geometric mean, i.e.,
x1 + x2 + · · · + xn √
≥ n x1 x2 . . . xn .
n
(Hint: exploit the log function.)
M. Passacantando Optimization Methods 18 / 24 – :
Convex sets Convex functions

Operations that preserve convexity

Theorem
I If f is convex and α > 0, then αf is convex
I If f1 and f2 are convex, then f1 + f2 are convex
I If f is convex, then f (Ax + b) is convex

Examples
I Log barrier for linear inequalities:
m
X
f (x) = − log (bi −aiT x) C = {x ∈ Rn : bi −aiT x > 0 ∀ i = 1, . . . , m}
i=1

I Norm of affine function: f (x) = kAx + bk

Exercise. If f1 and f2 are convex, then is the product f1 f2 convex?

M. Passacantando Optimization Methods 19 / 24 – :

Convex sets Convex functions

Pointwise maximum

Theorem
I If f1 , . . . , fm are convex, then f (x) = max{f1 (x), . . . , fm (x)} is convex.
I If {fi }i∈I is a family of convex functions, then f (x) = sup fi (x) is convex.
i∈I

Example. If L(x, λ) : Rn × Rm → R is convex in x and concave in λ, then

p(x) = sup L(x, λ) is convex

λ
d(λ) = inf L(x, λ) is concave
x

M. Passacantando Optimization Methods 20 / 24 – :

Convex sets Convex functions

Composition
f : Rn → R and g : R → R.
Theorem
I If f is convex and g is convex and nondecreasing, then g ◦ f is convex.
I If f is concave and g is convex and nonincreasing, then g ◦ f is convex.

I If f is concave and g is concave and nondecreasing, then g ◦ f is concave.

I If f is convex and g is concave and nonincreasing, then g ◦ f is concave.

Examples
I If f is convex, then e f (x) is convex
I If f is concave and positive, then log f (x) is concave
I If f is convex, then − log(−f (x)) is convex on {x : f (x) < 0}
1
I If f is concave and positive, then is convex
f (x)
I If f is convex and nonnegative, then f (x)p is convex for all p ≥ 1
M. Passacantando Optimization Methods 21 / 24 – :
Convex sets Convex functions

Sublevel sets

Given f : Rn → R and α ∈ R, the set

Sα (f ) = {x ∈ Rn : f (x) ≤ α}

is said the α-sublevel set of f .

Exericise. Prove that if f is convex, then Sα (f ) is a convex set for any α ∈ R.

Is the converse true?

M. Passacantando Optimization Methods 22 / 24 – :

Convex sets Convex functions

Quasiconvex functions

Given a convex set C ⊆ Rn , a function f : C → R is quasiconvex if the α-sublevel

sets are convex for all α ∈ R.

f is said quasiconcave if −f is quasiconvex.

Examples
p
I f (x) = |x| is quasiconvex on R
I f (x1 , x2 ) = x1 x2 is quasiconcave on {x ∈ R2 : x1 > 0, x2 > 0}
I f (x) = log x is quasiconvex and quasiconcave
I f (x) = ceil(x) = inf{z ∈ Z : z ≥ x} is quasiconvex and quasiconcave

M. Passacantando Optimization Methods 23 / 24 – :

Convex sets Convex functions

Exercise

T
Express each convex set defined below in the form {x : fi (x) ≤ 0}, where
i∈I
fi : Rn → R are suitable convex functions:

a) conv{(−1, −1), (1, 0), (0, 2)}

b) conv{(0, 0), (1, 1)}

 

c) conv x ∈ R2 : x12 + (x2 − 1)2 = 1 ∪ x ∈ R2 : x12 + (x2 + 1)2 = 1

d) conv{x ∈ R2 : x1 x2 = 1}

M. Passacantando Optimization Methods 24 / 24 – :