Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Scribd
Upload
85 views

Linear Algebra - Part II: Projection, Eigendecomposition, SVD

This document provides an overview of linear algebra concepts including projection, eigendecomposition, and singular value decomposition (SVD). It defines key terms such as orthogonal matrices, L2 norm, angle between vectors, vector projection, diagonal matrices, determinants, eigenvalues and eigenvectors, characteristic polynomials, symmetric matrices, positive definite matrices, and the SVD definition. The document uses examples to illustrate these concepts and their applications in matrix decomposition.

Uploaded by

john bianco
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
85 views

Linear Algebra - Part II: Projection, Eigendecomposition, SVD

This document provides an overview of linear algebra concepts including projection, eigendecomposition, and singular value decomposition (SVD). It defines key terms such as orthogonal matrices, L2 norm, angle between vectors, vector projection, diagonal matrices, determinants, eigenvalues and eigenvectors, characteristic polynomials, symmetric matrices, positive definite matrices, and the SVD definition. The document uses examples to illustrate these concepts and their applications in matrix decomposition.

Uploaded by

john bianco
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Linear Algebra - Part II

Projection, Eigendecomposition, SVD

Punit Shah

(Adapted from Sargur Srihari’s slides)


Brief Review from Part 1

◮ Symmetric Matrix:
A = AT

◮ Orthogonal Matrix:

AT A = AAT = I and A−1 = AT

◮ L2 Norm: 󰁶󰁛
||x||2 = xi2
i

Linear Algebra, Part II 2/20


Angle Between Vectors
◮ Dot product of two vectors can be written in terms of
their L2 norms and the angle θ between them.

aT b = ||a||2 ||b||2 cos(θ)

Linear Algebra, Part II 3/20


Cosine Similarity

◮ Cosine between two vectors is a measure of their


similarity:
a·b
cos(θ) =
||a|| ||b||
◮ Orthogonal Vectors: Two vectors a and b are
orthogonal to each other if a · b = 0.

Linear Algebra, Part II 4/20


Vector Projection
◮ Given two vectors a and b, let b̂ = b
||b||
be the unit vector
in the direction of b.
◮ Then a1 = a1 b̂ is the orthogonal projection of a onto a
straight line parallel to b, where
b
a1 = ||a|| cos(θ) = a · b̂ = a ·
||b||

Image taken from wikipedia.

Linear Algebra, Part II 5/20


Diagonal Matrix
◮ Diagonal matrix has mostly zeros with non-zero entries
only in the diagonal, e.g. identity matrix.

◮ A square diagonal matrix with diagonal elements given by


entries of vector v is denoted:

diag(v)
◮ Multiplying vector x by a diagonal matrix is efficient:

diag(v)x = v ⊙ x
◮ Inverting a square diagonal matrix is efficient:
󰀓 1 1 󰀔
diag(v)−1 = diag [ , . . . , ]T
v1 vn

Linear Algebra, Part II 6/20


Determinant
◮ Determinant of a square matrix is a mapping to a scalar.

det(A) or |A|
◮ Measures how much multiplication by the matrix expands
or contracts the space.

◮ Determinant of product is the product of determinants:

det(AB) = det(A)det(B)

Image taken from wikipedia.

Linear Algebra, Part II 7/20


List of Equivalencies

The following are all equivalent:

◮ A is invertible, i.e. A−1 exists.


◮ Ax = b has a unique solution.
◮ Columns of A are linearly independent.
◮ det(A) ∕= 0
◮ Ax = 0 has a unique, trivial solution: x = 0.

Linear Algebra, Part II 8/20


Zero Determinant

If det(A) = 0, then:

◮ A is linearly dependent.

◮ Ax = b has no solution or infinitely many solutions.

◮ Ax = 0 has a non-zero solution.

Linear Algebra, Part II 9/20


Matrix Decomposition

◮ We can decompose an integer into its prime factors, e.g.


12 = 2 × 2 × 3.

◮ Similarly, matrices can be decomposed into factors to


learn universal properties:

A = Vdiag(λ)V−1

Linear Algebra, Part II 10/20


Eigenvectors

◮ An eigenvector of a square matrix A is a nonzero vector v


such that multiplication by A only changes the scale of v.

Av = λv

◮ The scalar λ is known as the eigenvalue.

◮ If v is an eigenvector of A, so is any rescaled vector sv.


Moreover, sv still has the same eigenvalue. Thus, we
constrain the eigenvector to be of unit length:

||v|| = 1

Linear Algebra, Part II 11/20


Characteristic Polynomial
◮ Eigenvalue equation of matrix A:

Av = λv
Av − λv = 0
(A − λI)v = 0
◮ If nonzero solution for v exists, then it must be the case
that:
det(A − λI) = 0

◮ Unpacking the determinant as a function of λ, we get:

|A − λI| = (λ1 − λ)(λ2 − λ) . . . (λn − λ) = 0


◮ The λ1 , λ2 , . . . , λn are roots of the characteristic
polynomial, and are eigenvalues of A.
Linear Algebra, Part II 12/20
Example
◮ Consider the matrix:
󰀗 󰀘
2 1
A =
1 2
◮ The characteristic polynomial is:
󰀗 󰀘
2−λ 1
det(A − λI) = det = 3 − 4λ + λ2 = 0
1 2−λ
◮ It has roots λ = 1 and λ = 3 which are the two
eigenvalues of A.

◮ We can then solve for eigenvectors using Av = λv:

vλ=1 = [1, −1]T and vλ=3 = [1, 1]T

Linear Algebra, Part II 13/20


Eigendecomposition

◮ Suppose that n × n matrix A has n linearly independent


eigenvectors {v(1) , . . . , v(n) } with eigenvalues
{λ1 , . . . , λn }.

◮ Concatenate eigenvectors to form matrix V.

◮ Concatenate eigenvalues to form vector


λ = [λ1 , . . . , λn ]T .

◮ The eigendecomposition of A is given by:

A = Vdiag(λ)V−1

Linear Algebra, Part II 14/20


Symmetric Matrices
◮ Every real symmetric matrix A can be decomposed into
real-valued eigenvectors and eigenvalues:
A = QΛQT
◮ Q is an orthogonal matrix of the eigenvectors of A, and
Λ is a diagonal matrix of eigenvalues.
◮ We can think of A as scaling space by λi in direction v(i) .

Linear Algebra, Part II 15/20


Eigendecomposition is not Unique

◮ Decomposition is not unique when two eigenvalues are


the same.

◮ By convention, order entries of Λ in descending order.


Then, eigendecomposition is unique if all eigenvalues are
unique.

◮ If any eigenvalue is zero, then the matrix is singular.

Linear Algebra, Part II 16/20


Positive Definite Matrix

◮ A matrix whose eigenvalues are all positive is called


positive definite.

◮ If eigenvalues are positive or zero, then matrix is called


positive semidefinite.

◮ Positive definite matrices guarantee that:

xT Ax > 0 for any nonzero vector x

◮ Similarly, positive semidefinite guarantees: xT Ax ≥ 0

Linear Algebra, Part II 17/20


Singular Value Decomposition (SVD)

◮ If A is not square, eigendecomposition is undefined.

◮ SVD is a decomposition of the form:

A = UDVT

◮ SVD is more general than eigendecomposition.

◮ Every real matrix has a SVD.

Linear Algebra, Part II 18/20


SVD Definition (1)

◮ Write A as a product of three matrices: A = UDVT .

◮ If A is m × n, then U is m × m, D is m × n, and V is
n × n.

◮ U and V are orthogonal matrices, and D is a diagonal


matrix (not necessarily square).

◮ Diagonal entries of D are called singular values of A.

◮ Columns of U are the left singular vectors, and


columns of V are the right singular vectors.

Linear Algebra, Part II 19/20


SVD Definition (2)

◮ SVD can be interpreted in terms of eigendecompostion.

◮ Left singular vectors of A are the eigenvectors of AAT .

◮ Right singular vectors of A are the eigenvectors of AT A.

◮ Nonzero singular values of A are square roots of


eigenvalues of AT A and AAT .

Linear Algebra, Part II 20/20

You might also like