Review of

Matrices and Vectors

1/45
 Vectors & Vector Spaces
Definition of Vector: A collection of complex or real numbers,
generally put in a column  v 
1 Transpose
 
 
v =  "  = [v1 ! v N ]T
 
 
v N 
 

Definition of Vector Addition: Add element-by-element

 a1   b1   a1 + b1 
a =  "  b =  "  a + b =  " 
a N  bN  a N + bN 

2/45
Definition of Scalar: A real or complex number.

If the vectors of interest are complex valued then the set of

scalars is taken to be complex numbers; if the vectors of
interest are real valued then the set of scalars is taken to be
real numbers.

Multiplying a Vector by a Scalar :

 a1   αa1 
a =  "  αa =  " 
a N  αa N 

…changes the vector’s length if |α| ≠ 1

… “reverses” its direction if α < 0
3/45
Arithmetic Properties of Vectors: vector addition and
scalar multiplication exhibit the following properties pretty
much like the real numbers do
Let x, y, and z be vectors of the same dimension and let α
and β be scalars; then the following properties hold:
x+y=y+x
1. Commutativity
αx = xα

(x + y ) + z = y + (x + z )
2. Associativity α(βx) = (αβ)x

α ( x + y ) = αx + α y
3. Distributivity
(α + β)x = αx + βx

1x = x
4. Scalar Unity &
Scalar Zero 0x = 0, where 0 is the zero vector of all zeros
4/45
Definition of a Vector Space: A set V of N-dimensional vectors
(with a corresponding set of scalars) such that the set of vectors
is:
(i) “closed” under vector addition
(ii) “closed” under scalar multiplication

In other words:
• addition of vectors – gives another vector in the set
• multiplying a vector by a scalar – gives another vector in the set

Note: this means that ANY “linear combination” of vectors in the

space results in a vector in the space…
If v1, v2, and v3 are all vectors in a given vector space V, then
3
v = α1v1 + α 2 v 2 + α 3 v 3 = ∑α i v i
i =1
is also in the vector space V.
5/45
Axioms of Vector Space: If V is a set of vectors satisfying the above
definition of a vector space then it satisfies the following axioms:
1. Commutativity (see above)
2. Associativity (see above)
3. Distributivity (see above)
4. Unity and Zero Scalar (see above)
5. Existence of an Additive Identity – any vector space V must
have a zero vector
6. Existence of Negative Vector: For every vector v in V its
negative must also be in V

So… a vector space is nothing more than a set of vectors with

an “arithmetic structure”
6/45
Def. of Subspace: Given a vector space V, a subset of vectors in V
that itself is closed under vector addition and scalar
multiplication (using the same set of scalars) is called a
subspace of V.

Examples:
1. The space R2 is a subspace of R3.
2. Any plane in R3 that passes through the origin is a subspace
3. Any line passing through the origin in R2 is a subspace of R2
4. The set R2 is NOT a subspace of C2 because R2 isn’t closed
under complex scalars (a subspace must retain the original
space’s set of scalars)

7/45
Geometric Structure of Vector Space
Length of a Vector (Vector Norm): For any vector v in CN
we define its length (or “norm”) to be
N N
∑v 2
= ∑ vi
2
2
v 2
= i v
i =1 2
i =1

Properties of Vector Norm:

αv 2
=α v 2

α v1 + β v 2 2
≤ α v1 2
+ β v2 2

v 2
< ∞ ∀v ∈ C N

v 2
= 0 iff v = 0
8/45
Distance Between Vectors: the distance between two
vectors in a vector space with the two norm is defined by:
d ( v1 , v 2 ) = v1 − v 2 2

Note that: d ( v 1 , v 2 ) = 0 iff v 1 = v 2

v1 v1 – v2

v2

9/45
Angle Between Vectors & Inner Product:
v
Motivate the idea in R :
2
 A cos θ 1
A v =  A sin θ  u= 
θ   0 

u
2
Note that: ∑u v
i =1
i i = 1 ⋅ A cos θ + 0 ⋅ A sin θ = A cos θ

Clearly we see that… This gives a measure of the angle

between the vectors.
Now we generalize this idea!

10/45
Inner Product Between Vectors :
Define the inner product between two complex vectors in CN by:
N
< u, v >= ∑ u i vi*
i =1

Properties of Inner Products: < αu, v >= α < u, v >

1. Impact of Scalar Multiplication:
< u, βv >= β * < u, v >

< u, v + z >=< u, v > + < u, z >

2. Impact of Vector Addition:
< u + w, v >=< u, v > + < w, v >

2
3. Linking Inner Product to Norm: v 2
=< v, v >

4. Schwarz Inequality: < u, v > ≤ u 2

v 2

5. Inner Product and Angle: < u, v >

= cos(θ )
(Look back on previous page!) u2 v2
11/45
Inner Product, Angle, and Orthogonality :
< u, v >
= cos(θ )
u2 v2
(i) This lies between –1 and 1;
(ii) It measures directional alikeness of u and v
= +1 when u and v point in the same direction
= 0 when u and v are a “right angle”
= –1 when u and v point in opposite directions

Two vectors u and v are said to be orthogonal when <u,v> = 0

If in addition, they each have unit length they are orthonormal

12/45
Building Vectors From Other Vectors
Can we find a set of “prototype” vectors {v1, v2, …, vM} from
which we can build all other vectors in some given vector space V
by using linear combinations of the vi?
M M
v = ∑α k v k u = ∑ βk vk
k =1 k =1
Same “Ingredients”… just different amounts of them!!!

We want to be able to do is get any vector just by changing the

amounts… To do this requires that the set of “prototype”
vectors {v1, v2, …, vM} satisfy certain conditions.
We’d also like to have the smallest number of members in the
set of “prototype” vectors.
13/45
Span of a Set of Vectors: A set of vectors {v1, v2, …, vM} is said to
span the vector space V if it is possible to write each vector v in V as
a linear combination of vectors from the set:
M
v = ∑αk vk
k =1

This property establishes if there are enough vectors in the

proposed prototype set to build all possible vectors in V.
It is clear that:
1. We need at least N vectors to
Examples in R2 span CN or RN but not just any N
vectors.
2. Any set of N mutually orthogonal
vectors spans CN or RN (a set of
vectors is mutually orthogonal if all
Does not Spans R2
pairs are orthogonal).
Span R2
14/45
Linear Independence: A set of vectors {v1, v2, …, vM} is said to
be linearly independent if none of the vectors in it can be written as
a linear combination of the others.
If a set of vectors is linearly dependent then there is “redundancy”
in the set…it has more vectors than needed to be a “prototype” set!
For example, say that we have a set of four vectors {v1, v2, v3, v4}
and lets say that we know that we can build v2 from v1 and v3…
then every vector we can build from {v1, v2, v3, v4} can also be built
from only {v1, v3, v4}.
It is clear that:
1. In CN or RN we can have no
Examples in R2 more than N linear independent
vectors.
2. Any set of mutually
orthogonal vectors is linear
independent (a set of vectors is
Linearly Not Linearly
mutually orthogonal if all pairs
Independent Independent
are orthogonal).
15/45
Basis of a Vector Space: A basis of a vector space is a set of
linear independent vectors that span the space.
• “Span” says there are enough vectors to build everything
• “Linear Indep” says that there are not more than needed
Orthonormal (ON) Basis: If a basis of a vector space contains
vectors that are orthonormal to each other (all pairs of basis
vectors are orthogonal and each basis vector has unit norm).
Fact: Any set of N linearly independent vectors in CN (RN) is a
basis of CN (RN).
Dimension of a Vector Space: The number of vectors in any
basis for a vector space is said to be the dimension of the space.
Thus, CN and RN each have dimension of N.

16/45
Expansion and Transformation
Fact: For a given basis {v1, v2, …, vN}, the expansion of a vector v
in V is unique. That is, for each v there is only one, unique set of
N
coefficients {α1, α2, … , αN} such that v = α v
∑k =1
k k

In other words, this “expansion” or “decomposition” is unique.

Thus, for a given basis we can make a 1-to-1 correspondence
between vector v and the coefficients {α1, α2, … , αN}.
We can write the coefficients as a vector, too: α = [α1 ! α N ]T

 v1   α1  Expansion can be viewed as a mapping (or

    transformation) from vector v to vector α.
   
v= "  ←
→ α= "  We can view this transform as taking us
  1− to−1   from the original vector space into a new
    vector space made from the coefficient
v N  α N  vectors of all the original vectors.
   
17/45
Fact: For any given vector space there are an infinite number of
possible basis sets.
The coefficients with respect to any of them provides complete
information about a vector…
some of them provide more insight into the vector and are therefore
more useful for certain signal processing tasks than others.

Often the key to solving a signal processing problem lies in finding

the correct basis to use for expanding… this is equivalent to finding
the right transform. See discussion coming next linking DFT to
these ideas!!!!

18/45
 DFT from Basis Viewpoint:
If we have a discrete-time signal x[n] for n = 0, 1, … N-1
x = [x[0] x[1] ! x[ N − 1]]
T
Define vector:
Define a orthogonal basis from the exponentials used in the IDFT:

1  1   1   1 
1  e j 2 π1⋅1 / N   e j 2 π 2⋅1 / N   e j 2 π ( N −1)⋅1 / N 
d0 =  
"

d1 = 
 "


 j 2 π1( N −1) / N 
d2 = 
 "


 j 2 π 2( N −1) / N 
… d N −1 =
 "


 j 2 π( N −1)( N −1) / N 
1 e  e  e 

Then the IDFT equation can be viewed as an expansion of the

signal vector x in terms of this complex sinusoid basis:
N −1 T
1  X [ 0] X [1] X [ N − 1] 
x = ∑ X [k ] d k α= ! 
N
k =0 &
#%#
$  N N N 
αk

kth coefficient coefficient vector 19/45

 Usefulness of an ON Basis
What’s So Good About an ON Basis?: Given any basis
{v1, v2, …, vN} we can write any v in V as
N
v = ∑αk vk
k =1

Given the vector v how do we find the α’s?

• In general – hard! But for ON basis – easy!!
N 
If {v1, v2, …, vN} is an ON basis then v, v i = ∑ α j v j  , v i
 j =1 
N
= ∑α
j =1
j
&
v j , vi
#%# $
δ[ j − i ]

α i = v, v i = αi

ith coefficient = inner product with ith ON basis vector

20/45
Another Good Thing About an ON Basis: They preserve inner
products and norms… (called “isometric”):
If {v1, v2, …, vN} is an ON basis and u and v are vectors
expanded as

N N
Then…. v = ∑αk vk u = ∑ βk vk
k =1 k =1

1. < v ,u > = < α , β > (Preserves Inner Prod.)

2. ||v||2 = ||α||2 and ||u||2 = ||β||2 (Preserves Norms)

So… using an ON basis provides:

• Easy computation via inner products
• Preservation of geometry (closeness, size, orientation, etc.

21/45
Example: DFT Coefficients as Inner Products:
Recall: N-pt. IDFT is an expansion of the signal vector in terms of
N Orthogonal vectors. Thus

X [k ] = x, d k

N −1
= ∑ x[ n ]d *
k [n ]
n =0

N −1
= ∑ x[n]e − j 2πkn / N
n =0

See “reading notes” for some details about normalization issues in this case

22/45
 Matrices
Matrix: Is an array of (real or complex) numbers organized in
rows and columns.  a11 a12 a13 a14 
 
Here is a 3x4 example:  
A = a 21 a 22 a 23 a 24 
 
 
a31 a32 a33 a34 


We’ll sometimes view a matrix as being built from its columns;

The 3x4 example above could be written as:

A = [a1 | a 2 | a 3 | a 4 ] a k = [a1k a2k a3k ]T

We’ll take two views of a matrix:

1. “Storage” for a bunch of related numbers (e.g., Cov. Matrix)
2. A transform (or mapping, or operator) acting on a vector
(e.g., DFT, observation matrix, etc…. as we’ll see)
23/45
Matrix as Transform: Our main view of matrices will be as
“operators” that transform one vector into another vector.
Consider the 3x4 example matrix above. We could use that matrix
to transform the 4-dimensional vector v into a 3-dimensional
vector u:
 v1 
 
 
v 2 
 
u = Av = [a1 | a 2 | a 3 | a 4 ]   = v1a1 + v2 a 2 + v3a 3 + v4 a 4
 v3 
 
 
v  Clearly u is built from the columns
 4
of matrix A; therefore, it must lie
in the span of the set of vectors that
make up the columns of A.
Note that the columns of A are
3-dimensional vectors… so is u. 24/45
Transforming a Vector Space: If we apply A to all the vectors in
a vector space V we get a collection of vectors that are in a
new space called U.
In the 3x4 example matrix above we transformed a 4-dimensional
vector space V into a 3-dimensional vector space U
A 2x3 real matrix A would transform R3 into R2 :

Facts: If the mapping matrix A is square and its columns are

linearly independent then
(i) the space that vectors in V get mapped to (i.e., U) has the
same dimension as V (due to “square” part)
(ii) this mapping is reversible (i.e., invertible); there is an inverse
matrix A-1 such that v = A-1u (due to “square” & “LI” part)
25/45
Transform = Matrix × Vector: a VERY useful viewpoint for all
sorts of signal processing scenarios. In general we can view many
linear transforms (e.g., DFT, etc.) in terms of some invertible
matrix A operating on a signal vector x to give another vector y:
y i = Axi x i = A −1y i

A, A −1

y2
x1 y1
x2

We can think of A and A-1 as

mapping back and forth
A, A −1 between two vector spaces
26/45
 Matrix View & Basis View
Basis Matrix & Coefficient Vector:
Suppose we have a basis {v1, v2, …, vN} for a vector space V.
Then a vector v in space V can be written as: N
v = ∑αk vk
k =1

Another view of this:

 α1 
 
 
α2 
 
v = [v1 | v 2 | ! | v N ]  
&##%##$
NxN matrix  " 
v = Vα
 
 
α 
 N
The “Basis Matrix” V transforms
the coefficient vector into the
original vector v
27/45
Three Views of Basis Matrix & Coefficient Vector:
View #1
Vector v is a linear combination of the columns of basis matrix V.
N
v = ∑αk vk
k =1
View #2
Matrix V maps vector α into vector v.
Now have
a way to go
v = Vα back-and-
View #3 forth
between
There is a matrix, V-1, that maps vector v into vector α. vector v
α = V −1v and its
coefficient
vector α

Aside: If a matrix A is square and has linearly independent columns, then A

is “invertible” and A-1 exists such that A A-1 = A-1A = I where I is the identity
matrix having 1’s on the diagonal and zeros elsewhere.
28/45
Basis Matrix for ON Basis: We get a special structure!!!
Result: For an ON basis matrix V… V-1 = VH
(the superscript H denotes “hermitian transpose”, which consists
of transposing the matrix and conjugating the elements)

To see this:
 < v1 , v1 > < v1 , v 2 > ! < v1 , v N > 
< v , v > < v , v > ! < v 2 , v N > 
VV H = 2 1 2 2

 " ' " 

 
< v N , v 1 > < v N , v 2 > ! < v N , v N >

1 0 ! 0
0 1 ! 0  Inner products are 0 or 1
= =I because this is an ON basis
" ' "
 
 0 0 ! 1 
29/45
 Unitary and Orthogonal Matrices
A unitary matrix is a complex matrix A whose inverse is A-1 = AH
For the real-valued matrix case… we get a special case of “unitary”
the idea of “unitary matrix” becomes “orthogonal matrix”
for which A-1 = AT
Two Properties of Unitary Matrices: Let U be a unitary matrix
and let y1 = Ux1 and y2 = Ux2
1. They preserve norms: ||yi|| = ||xi||.
2. They preserve inner products: < y1, y2 > = < x1, x2 >
That is the “geometry” of the old space is preserved by the unitary
matrix as it transforms into the new space.
(These are the same as the preservation properties of ON basis.)

30/45
DFT from Unitary Matrix Viewpoint:
Consider a discrete-time signal x[n] for n = 0, 1, … N-1.
N −1
We’ve already seen the DFT in a basis viewpoint: 1
x= ∑ N X [k ] d k
k =0 &
#%#
$
αk

Now we can view the DFT as a transform from the Unitary matrix
viewpoint:
1 1 1 ! 1 
 1 e j 2 π1⋅1 / N e j 2 π 2⋅1 / N ! e j 2 π ( N −1)⋅1 / N 
D = [d 0 | d1 | … | d N −1 ] = 
" " " " 
 j 2 π1( N −1) / N j 2 π ( N −1)( N −1) / N 
 1 e e j 2 π 2 ( N −1) / N ! e 
DFT IDFT
~
x = DH x 1 ~
x = Dx
N
(Acutally D is not unitary but N-1/2D is unitary… see reading notes) 31/45
Geometry Preservation of Unitary Matrix Mappings
Recall… unitary matrices map in such a way that the sizes of
vectors and the orientation between vectors is not changed.

A, A −1

y2

x1
x2
y1

A, A −1
Unitary mappings just
“rigidly rotate” the space.
32/45
Effect of Non-Unitary Matrix Mappings

A, A −1

y2

x1
x2
y1

A, A −1

33/45
 More on Matrices as Transforms
We’ll limit ourselves here to real-valued vectors and matrices

y = Ax A maps any vector x in Rn

into some vector y in Rm
m×1 m×n n×1
Mostly interested in two cases:
1. “Tall Matrix” m>n
2. “Square Matrix” m = n

Rn Rm
A y
x

Range(A): “Range Space of A” =

vector y = weighted sum of columns of A set of all vectors in Rm that can be
⇒ may only be able to reach certain y’s reached by mapping
34/45
Range of a “Tall Matrix” (m > n) The range(A) ⊂ Rm
Rn Rm

x A y

“Proof”: Since y is “built” from the n columns of A there are

not enough to form a basis for Rm (they don’t span Rm)

Range of a “Square Matrix” (m = n)

If the columns of A are linearly indep….The range(A) = Rm

…because the columns form a basis for Rm

Otherwise….The range(A) ⊂ Rm
…because the columns don’t span Rm
35/45
Rank of a Matrix: rank(A) = largest # of linearly independent
columns (or rows) of matrix A
For an m×n matrix we have that rank(A) ≤ min(m,n)
An m×n matrix A has “full rank” when rank(A) = min(m,n)
Example: This matrix has rank of 3 because the 4th column cam be
written as a combination of the first 3 columns
1 0 0 1
 
 
0 1 0 2
 
 
A = 0 0 1 1
 
 
0 0 0 0
 
 
0 0 0 0

36/45
Characterizing “Tall Matrix” Mappings
We are interested in answering: Given a vector y, what vector x
mapped into it via matrix A?
“Tall Matrix” (m > n) Case
If y does not lie in range(A), then there is No Solution
If y lies in range(A), then there is a solution (but not
necessarily just one unique solution)

y = Ax
y∉range(A) y∈range(A)

No Solution A full A not

rank full rank

One Solution Many Solutions 37/45

Full-Rank “Tall Matrix” (m > n) Case y = Ax
A
Rn Rm
y
x
Range(A)

For a given y∈range(A)…

there is only one x that maps to it.

This is because the columns of A are linearly independent

and we know from our studies of vector spaces that the
coefficient vector of y is unique… x is that coefficient
vector

By looking at y we can determine which x gave rise to it

38/45
NonFull-Rank “Tall Matrix” (m > n) Case y = Ax
A
Range(A)
Rn Rm
y
x1
x2 A

For a given y∈range(A) there is more than one x that maps

to it

This is because the columns of A are linearly dependent

and that redundancy provides several ways to combine
them to create y

By looking at y we can not determine which x gave rise to it

39/45
Characterizing “Square Matrix” Mappings

Q: Given any y∈Rn can we find an x∈Rn that maps to it?

A: Not always!!!
y = Ax Careful!!! This is quite a
different flow diagram here!!!
A full A not
rank full rank

One Solution
y∉range(A) y∈range(A)

No Solution Many Solutions

When a square A is full rank then its range covers the

complete new space… then, y must be in range(A) and
because the columns of A are a basis there is a way to
build y 40/45
A Full-Rank Square Matrix is Invertible
A square matrix that has full rank is said to be….
“nonsingular”, “invertible”
Then we can find the x that mapped to y using x = A-1y

Several ways to check if n×n A is invertible:

1. A is invertible if and only if (iff) its columns (or rows) are
linearly independent (i.e., if it is full rank)
2. A is invertible iff det(A) ≠ 0
3. A is invertible if (but not only if) it is “positive definite” (see
later)
4. A is invertible if (but not only if) all its eigenvalues are
nonzero

Pos. Def. Invertible

Matrices Matrices

41/45
Eigenvalues and Eigenvectors of Square Matrices
If matrix A is n×n, then A maps Rn → Rn
Q: For a given n×n matrix A, which vectors get mapped into
being almost themselves???
More precisely… Which vectors get mapped to a scalar multiple
of themselves???
Even more precisely… which vectors v satisfy the following:
Av = λv
Input Output
These vectors are “special” and are called the eigenvectors of A.
The scalar λ is that e-vector’s corresponding eigenvalue.
v Av

42/45
“Eigen-Facts for Symmetric Matrices”
• If n×n real matrix A is symmetric, then
– e-vectors corresponding to distinct e-values are orthonormal
– e-values are real valued
– can decompose A as A = VΛ V T
V = [v1 v2 ! vn ] VV T = I

Λ = diag{λ1 , λ2 ,…, λn }
• If, further, A is pos. def. (semi-def.), then
– e-values are positive (non-negative)
– rank(A) = # of non-zero e-values
• Pos. Def. ⇒ Full Rank (and therefore invertible)
• Pos. Semi-Def. ⇒ Not Full Rank (and therefore not invertible)
– When A is P. D., then we can write
A −1 = VΛ −1 V T
{ }
For P.D. A, A-1 has
the same e-vectors and Λ −1 = diag 1 λ , 1 λ ,…, 1 λ
1 2 n
has reciprocal e-values 43/45
 Other Matrix Issues
We’ll limit our discussion to real-valued matrices and vectors
Quadratic Forms and Positive-(Semi)Definite Matrices
Quadratic Form = Matrix form for a 2nd-order multivariate
polynomial
 x1   a11 a12 
Example: x =   A =  

 x2  a 21 a 22 

variable fixed
The quadratic form of matrix A is:

QA ( x1 , x2 ) = x T Ax (1 × 2) ⋅ ( 2 × 2) ⋅ ( 2 × 1) = (1 × 1) scalar

2 2
scalar = ∑∑ ij i j 11 1 22 2 + (a12 + a21 ) x1 x2
a x x = a x 2
+ a x 2

i =1 j =1 44/45
• Values of the elements of matrix A determine the characteristics
of the quadratic form QA(x)
– If QA(x) ≥ 0 ∀x ≠ 0… then say that QA(x) is “positive semi-definite”
– If QA(x) > 0 ∀x ≠ 0… then say that QA(x) is “positive definite”
– Otherwise say that QA(x) is “non-definite”
• These terms carry over to the matrix that defines the Quad Form
– If QA(x) ≥ 0 ∀x ≠ 0… then say that A is “positive semi-definite”
– If QA(x) > 0 ∀x ≠ 0… then say that A is “positive definite”

45/45