Semidefinite matrices &

Convex functions

Rudi Pendavingh

Eindhoven Technical University

Optimization in Rn , lecture 7

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 1 / 17

Semidefinite matrices
A symmetric matrix A is positive semidefinite (PSD) if and only if
x t Ax ≥ 0 for all x ∈ Rn .
Theorem
Let A be a symmetric matrix. The following are equivalent:
1 A is PSD, i.e. x t Ax ≥ 0 for all x ∈ Rn .
2 all eigenvalues of A are nonnegative.
3 A = Z t Z for some real matrix Z .

Corollary
Let p ∈ R[X1 , . . . , Xn ] be a homogeneous quadratic polynomial. Then
p(x1 , . . . , xn ) ≥ 0 for all x1 , . . . , xn ∈ R if and only if

p = s12 + · · · + sn2

for some si ∈ R[X1 , . . . , Xn ]

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 2 / 17
Recognizing PSD matrices
Definition
symmetric matrix operations on a matrix are:
1 multiplying both the i-th row and i-th column by λ 6= 0
2 swapping the i-th and j-th column; and swapping the i-th and j-th
row
3 adding λ× i-th column to j-th column and adding λ× i-th row to
j-th row
A∼= B :⇐⇒ B is obtained from A by zero or more symmetric matrix
operations.

Lemma
A∼
= B if and only if B = Y t AY for some invertible Y .

Lemma
Let A, B be symmetric. If A ∼
= B, then A is PSD ⇔ B is PSD.
Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 3 / 17
Recognizing PSD matrices

To determine whether A is PSD, compute a diagonal matrix D ∼

= A.
Example
   
1 2 ∼ 1 0
A= = = D.
2 3 0 −1
D is not PSD, hence A is not PSD.

Example
     
1 2 −1 1 0 0 1 0 0
A= 2 5 1 =∼ 0 1 3  ∼
= 0 1 0 = D.
−1 1 12 0 3 11 0 0 2

D is PSD, hence A is PSD.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 4 / 17

Positive definite matrices
A matrix A is positive definite (PD) if x t Ax > 0 for all nonzero x ∈ Rn .

Lemma
A is PD ⇐⇒ A is PSD and det(A) 6= 0.

Theorem
Let A be a symmetric matrix. The following are equivalent:
1 A is PD, i.e. x t Ax > 0 for all nonzero x ∈ Rn .
2 all eigenvalues of A are positive.
3 A = Z t Z for some real matrix Z so that det(Z ) 6= 0.

Lemma
Let A, B be symmetric. If A ∼
= B, then A is PD ⇔ B is PD.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 5 / 17

Further characterizations

Let A be a square matric and let I ⊆ {1, . . . , n}. Then AI denotes the
restriction of A to the rows and columns indexed by I .
Lemma
Let A be an n × n matrix. Then A is PSD ⇐⇒ det(AI ) ≥ 0 for all
I ⊆ {1, . . . , n}.

Lemma
Let A be an n × n matrix. Then A is PD ⇐⇒ det(AI ) > 0 for
I = {1}, . . . , {1, . . . , n}.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 6 / 17

Geršgorin’s Theorem
P
Given a complex n × n matrix A, we put ρp := j6=p |apj | for p = 1, . . . , n.

Theorem (Geršgorin, 1931)

Let A be a complex n × n matrix and let λ be an eigenvalue of A. Then
|λ − app | ≤ ρp for some p ∈ {1, . . . , n}.

Proof.
Let x ∈ Cn be a nonzero vector such that λx = Ax.
Let p be such that |xp | = maxi |xi |.
We have λxp = nj=1 apj xj , hence (λ − app )xp = j6=p apj xj .
P P
P P
Taking norms, |λ − app ||xp | = | j6=p apj xj | ≤ j6=p |apj ||xj | ≤ ρp |xp |.
Dividing by |xp |, the Theorem follows.

For each p, we have the Geršgorin disk {λ ∈ C | |λ − app | ≤ ρp }. The

Theorem states that each eigenvalue is in one of the Geršgorin disks.
Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 7 / 17
Convex and concave functions

Definition
A function f : Rn → R is convex if Dom(f ) is convex, and

λf (x) + (1 − λ)f (y ) ≥ f (λx + (1 − λ)y ),

for all x, y ∈ Dom(f ) and for all λ ∈ [0, 1].

Definition
f is concave if −f is convex.

Lemma
f is convex and concave if and only if f is affine.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 8 / 17

Norms
Definition
A function f : Rn → R is a norm if
f (x) ≥ 0 for all x ∈ Rn
f (x) = 0 if and only if x = 0
f (λx) = |λ|f (x) for all x ∈ Rn , λ ∈ R
f (x + y ) ≤ f (x) + f (y ) for all x, y ∈ Rn

Theorem
Let f : Rn → R be a norm. Then f is convex.

Lemma
Let f : Rn → R be a convex function. Then {x ∈ Rn | f (x) ≤ α} is a
convex set for any α ∈ R.

So the norm ball {x ∈ Rn | kxk ≤ 1} is convex for any norm k.k.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 9 / 17
Definition
The epigraph of a function f : Rn → R is

epi(f ) := {(x, t) ∈ Rn+1 | x ∈ Dom(f ), t ≥ f (x)}.

Lemma
f is a convex function if and only if epi(f ) is a convex set.

Definition
A subgradient of f at x is a row vector w such that

f (y ) − f (x) ≥ w (y − x) for all y .

Theorem
f is convex ⇐⇒ Dom(f ) is convex, and there exists a subgradient of f at
each x in the interior of Dom(f ).

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 10 / 17

Operations that preserve convexity

Lemma
If f : Rn → R is convex, then the following functions are convex as well:
x 7→ αf (x), for any α > 0
x 7→ f (x + t), for any fixed t ∈ Rn
x 7→ f (Ax), for any n × m matrix A

Lemma
If f1 , . . . , fm : Rn → R are convex, then the following functions are convex
as well:
x 7→ max{f1 (x), . . . , fm (x)}
x 7→ f1 (x) + · · · + fm (x)

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 11 / 17

Differentiable convex functions
Definition
∂f ∂f
The gradient of f : Rn → R at x is ∇f (x) := ( ∂x1
(x), . . . , ∂xn
(x)).

Lemma
If w is a subgradient of f at x, and f is differentiable at x, then
w = ∇f (x).

Theorem (First-order condition for convexity)

A differentiable function f is convex if and only if

f (y ) ≥ f (x) + ∇f (x)(y − x)

for all x, y ∈ Dom(f ).

Note: x is a minimizer of f if and only if 0 is a subgradient of f at x.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 12 / 17

Twice differentiable convex functions

Definition
The Hessian of a function f : Rn → R at x is the matrix
 ∂2f 2f 
∂x12
(x) · · · ∂x∂1 ∂x n
(x)
∇2 f (x) := 
 .. .. 
.
 . . 
2
∂ f 2
∂ f
∂xn ∂x1 (x) · · · ∂x 2
(x)
n

Theorem (Second-order condition for convexity)

Let f be a twice differentiable function. Then f is convex ⇐⇒ Dom(f ) is
convex and
∇2 f (x) is positive semidefinite
for all x in the interior of Dom(f ).

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 13 / 17

Quadratic functions

Example
x t Qx
Let f : x 7→ 2 + px, where Q is a symmetric matrix, p a row vector.
∇f (x) = xtQ + p for all x
∇2 f (x) = Q for all x
f is convex if and only if Q is PSD
if f is convex, then f (y ) = min{f (x) | x ∈ Rn } if and only if

0 = ∇f (y ) = y t Q + p

Note: f is convex if and only if the second-order approximation of f is

convex everywhere.

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 14 / 17

Some convex functions

Example (Functions of one variable)

7 e ax is convex on R, for any a ∈ R
x→
x→7 x a is
I convex on {x ∈ R | x > 0} when a > 1 or a < 0
I concave on {x ∈ R | x > 0} when 0 ≤ a ≤ 1
x 7→ log(x) is concave on {x ∈ R | x > 0}
x 7→ x log(x) is convex on {x ∈ R | x > 0}

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 15 / 17

More convex functions

Example (Functions of several variables)

(x, y ) 7→ x 2 /y is convex on {(x, y ) ∈ R2 | y > 0}
(x1 , . . . , xn ) 7→ max{x1 , . . . , xn } is convex on Rn
(x1 , . . . , xn ) 7→ log(e x1 + · · · + e xn ) is convex on Rn
1
(x1 , . . . , xn ) 7→ ( ni=1 xi ) n is concave on {x ∈ Rn | x > 0}
Q

X 7→ log(det(X )) is concave on

{X | X is PSD, det(X ) 6= 0}

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 16 / 17

The function f : X 7→ − log(det(X )) is convex
Proof.
It suffices to show that for any PD X and symmetric Y , the function

g : t 7→ − log(det(X + tY ))

is convex. As X is PD, there exists a PD matrix Z so that Z 2 = X . Hence

g (t) = − log(det(ZZ )) − log(det(Z −1 (X + tY )Z −1 )) =

= − log(det(X )) − log(det(I + tY 0 ))
where Y 0 = Z −1 YZ −1Q. Let λ1 , . . . , λn be the eigenvalues of Y 0 .
Then det(I + tY 0 ) = i (1 + tλi ), hence
X
g (t) = − log(det(X )) − log(1 + tλi ).
i

This is a sum of a constant and convex funtions t 7→ − log(1 + tλ).

Rudi Pendavingh (TUE) Semidefinite matrices & Convex functions ORN7 17 / 17