Chapter 5

Vector Spaces: Theory and Practice

So far, we have worked with vectors of length n and performed basic operations on them
like scaling and addition. Next, we looked at solving linear systems via Gaussian elimination
and LU factorization. Already, we ran into the problem of what to do if a zero “pivot” is
encountered. What if this cannot be fixed by swapping rows? Under what circumstances
will a linear system not have a solution? When will it have more than one solution? How
can we describe the set of all solutions? To answer these questions, we need to dive deeper
into the theory of linear algebra.

5.1 Vector Spaces

The reader should be quite comfortable with the simplest of vector spaces: R ,R2 , and R3 ,
which represent the points in one-dimentional, two-dimensional, and three-dimensional (real
 space,respectively. A vector x ∈ R is represented by the column of n real numbers
n
valued)
χ0
 .. 
x =  .  which one can think of as the direction (vector) from the origin (the point
χ
 n−1  
0 χ0
   
0 =  ... ) to the point x =  ... . However, notice that a direction is position
0 χn−1
independent: You can think of it as a direction anchored anywhere in Rn .
What makes the set of all vectors in Rn a space is the fact that there is a scaling operation
(multiplication by a scalar) defined on it, as well as the addition of two vectors: The definition
of a space is a set of elements (in this case vectors in Rn ) together with addition and
multiplication (scaling) operations such that (1) for any two elements in the set, the element
that results from adding these elements is also in the set; (2) scaling an element in the set
results in an element in the set; and (3) there is an element in the set, denoted by 0, such
that additing it to another element results in that element and scaling by 0 results in the 0

135
136 Chapter 5. Vector Spaces: Theory and Practice

element.

Example 5.1 Let x, y ∈ R2 and α ∈ R. Then

• z = x + y ∈ R2 ;

• α · x = αx ∈ R2 ; and
' (
0
• 0 ∈ R and 0 · x =
2
.
0

In this document we will talk about vector spaces because the spaces have vectors as their
elements.

Example 5.2 Consider the set of all real valued m × n matrices, Rm×n . Together with
matrix addition and multiplication by a scalar, this set is a vector space.
Note that an easy way to visualize this is to take the matrix and view it as a vector of
length m · n.

Example 5.3 Not all spaces are vector spaces. For example, the spaces of all functions
defined from R to R has addition and multiplication by a scalar defined on it, but it is not
a vectors space. (It is a space of functions instead.)

Recall the concept of a subset, B, of a given set, A. All elements in B are elements in
A. If A is a vector space we can ask ourselves the question of when B is also a vector space.
The answer is that B is a vector space if (1) x, y ∈ B implies that x + y ∈ B; (2) x ∈ B and
α ∈ B implies αx ∈ B; and (3) 0 ∈ B (the zero vector). We call a subset of a vector space
that is also a vector space a subspace.

Example 5.4 Reason that one does not need to explicitly say that the zero vector is in a
(sub)space.

Definition 5.5 Let A be a vector space and let B be a subset of A. Then B is a subspace
of A if (1) x, y ∈ B implies that x + y ∈ B; and (2) x ∈ B and α ∈ R implies that αx ∈ B.
5.2. Why Should We Care? 137

One way to describe a subspace is that it is a subset that is closed under addition and
scalar multiplication.

Example 5.6 The empty set is a subset of Rn . Is it a subspace? Why?

Exercise 5.7 What is the smallest subspace of Rn ? (Smallest in terms of the number of
elements in the subspace.)

5.2 Why Should We Care?

Example 5.8 Consider

     
3 −1 2 8 5
A =  1 2 0 , b0 =  −1  , and b1 =  −1 
4 1 2 7 7

Does Ax = b0 have a solution? The answer is yes: x = (1, −1, 2)T . Does Ax = b1 have a
solution? The answer is no. Does Ax = b0 have any other solutions? The answer is yes.

The above example motivates the question of when a linear system has a solution, when
it doesn’t, and how many solutions it has. We will try to answer that question in this section.
Let A ∈ Rm×n , x ∈ Rn , b ∈ Rm , and Ax = b. Partition
 
χ0
) *  χ1 
 
A → a0 a1 · · · an−1 and x →  ..  .
 . 
χn−1

Then
χ0 a0 + χ1 a1 + · · · + χn−1 an−1 = b.

Definition 5.9 Let {a0 , . . . , an−1 } ⊂ Rm and {χ0 , . . . , χn−1 } ⊂ R. Then χ0 a0 + χ1 a1 +

· · · + χn−1 an−1 is said to be a linear combination of the vectors {a0 , . . . , an−1 }.

We note that Ax = b can be solved if and only if b equals a linear combination of the
vectors that are the columns of A, by the definition of matrix-vector multiplication. This
138 Chapter 5. Vector Spaces: Theory and Practice

observation answers the question “Given a matrix A, for what right-hand side vector, b, does
Ax = b have a solution?” The answer is that there is a solution if and only if b is a linear
combination of the columns (column vectors) of A.
Definition 5.10 The column space of A ∈ Rm×n is the set of all vectors b ∈ Rm for
which there exists a vector x ∈ Rn such that Ax = b.

Theorem 5.11 The column space of A ∈ Rm×n is a subspace (of Rm ).

Proof: We need to show that the column space of A is closed under addition and scalar
multiplication:

• Let b0 , b1 ∈ Rm be in the column space of A. Then there exist x0 , x1 ∈ Rn such that

Ax0 = b0 and Ax1 = b1 . But then A(x0 + x1 ) = Ax0 + Ax1 = b0 + b1 and thus b0 + b1
is in the column space of A.

• Let b be in the column space of A and α ∈ R. Then there exists a vector x such that
Ax = b and hence αAx = αb. Since A(αx) = αAx = αb we conclude that αb is in the
column space of A.

Hence the column space of A is a subspace (of Rm ).

Example 5.12 Consider again

   
3 −1 2 8
A= 1 2 0 , b0 =  −1  .
4 1 2 7

Set this up as two appended systems, one for solving Ax = b0 and the other for solving Ax = 0
(this will allow us to compare and contrast, which will lead to an interesting observation later
on):    
3 −1 2 8 3 −1 2 0
 1 2 0 −1   1 2 0 0 . (5.1)
4 1 2 7 4 1 2 0
Now, apply Gauss-Jordan elimination.

• It becomes convenient to swap the first and second equation:

   
1 2 0 −1 1 2 0 0
 3 −1 2 8   3 −1 2 0  .
4 1 2 7 4 1 2 0
5.2. Why Should We Care? 139

• Use the first row to eliminate the coefficients in the first column below the diagonal:
   
1 2 0 −1 1 2 0 0
 0 −7 2 11   0 −7 2 0  .
0 −7 2 11 0 −7 2 0

• Use the second row to eliminate the coefficients in the second column below the diag-
onal:    
1 2 0 −1 1 2 0 0
 0 −7 2 11   0 −7 2 0 .
0 0 0 0 0 0 0 0
• Divide the first and second row by the diagonal element:
   
1 2 0 −1 1 2 0 0
 0 1 −2/7 −11/7   0 1 −2/7 0  .
0 0 0 0 0 0 0 0

• Use the second row to eliminate the coefficients in the second column above the diag-
onal:    
1 0 4/7 15/7 1 0 4/7 0
 0 1 −2/7 −11/7   0 1 −2/7 0  . (5.2)
0 0 0 0 0 0 0 0
Now, what
) does*this mean? For now, we will focus only on the results for the appended
system A b0 on the left.
• We notice that 0 × χ2 = 0. So, there is no constraint on variable χ2 . As a result, we
will call χ2 a free variable.
• We see from the second row that χ1 − 2/7χ2 = −11/7 or χ1 = −11/7 + 2/7χ2 . Thus,
the value of χ1 is constrained by the value given to χ2 .
• Finally, χ0 + 4/7χ2 = 15/7 or χ0 = 15/7 − 4/7χ2 . Thus, the value of χ0 is also
constrained by the value given to χ2 .
We conclude that any vector of the form
 
15/7 − 4/7χ2
 −11/7 + 2/7χ2 
χ2
solves the linear system. We can rewrite this as
   
15/7 −4/7
 −11/7  + χ2  2/7  . (5.3)
0 1
So, for each choice of χ2 , we get a solution to the linear system by plugging it into Equa-
tion (5.3).
140 Chapter 5. Vector Spaces: Theory and Practice

Example 5.13 We now give a slightly “slicker” way to view Example 5.12. Consider again
Equation (5.2):  
1 0 4/7 15/7
 0 1 −2/7 −11/7  .
0 0 0 0
This represents     
1 0 4/7 χ0 15/7
 0 1 −2/7   χ1  =  −11/7  .
0 0 0 χ2 0
Using blocked matrix-vector multiplication, we find that
' ( ' ( ' (
χ0 4/7 15/7
+ χ2 =
χ1 −2/7 −11/7

and hence ' ( ' ( ' (

χ0 15/7 4/7
= − χ2
χ1 −11/7 −2/7
which we can then turn into
     
χ0 15/7 −4/7
 χ1  =  −11/7  + χ2  2/7 
χ2 0 1

In the above example, we notice the following:

      
15/7 3 −1 2 15/7 8
• Let xp =  −11/7 . Then Axp =  1 2 0   −11/7  =  −1 . In other
0 4 1 2 0 7
words, xp is a particular solution to Ax = b0 . (Hence the p in the xp .)
      
−4/7 3 −1 2 −4/7 0
• Let xn =  
2/7 . Then Axn =  1 2 0   2/7  =  0 . We will see
1 4 1 2 1 0
that xn is in the null space of A, to be defined later. (Hence the n in xn .)

• Now, notice that for any α, xp + αxn is a solution to Ax = b0 :

A(xp + αxn ) = Axp + A(αxn ) = Axp + αAxn = b0 + α × 0 = b0 .

5.2. Why Should We Care? 141

So, the system Ax = b0 has many solutions (indeed, an infinite number of solutions). To
characterize all solutions, it suffices to find one (nonunique) particular solution xp that
satisfies Axp = b0 . Now, for any vector xn that has the property that Axn = 0, we know
that xp + xn is also a solution.
Definition 5.14 Let A ∈ Rm×n . Then the set of all vectors x ∈ Rn that have the
property that Ax = 0 is called the null space of A and is denoted by N (A).

Theorem 5.15 The null space of A ∈ Rm×n is a subspace of Rn .

Proof: Clearly N (A) is a subset of Rn . Now, assume that x, y ∈ N (A) and α ∈ R. Then
A(x + y) = Ax + Ay = 0 and therefore (x + y) ∈ N (A). Also A(αx) = αAx = α × 0 = 0
and therefore αx ∈ N (A). Hence, N (A) is a subspace.

Notice that the zero vector (of appropriate length) is always in the null space of a matrix
A.

Example 5.16 Let us use the last example, but with Ax = b1 : Let us set this up as an
appended system  
3 −1 2 5
 1 2 0 −1  .
4 1 2 7
Now, apply Gauss-Jordan elimination.
• It becomes convenient to swap the first and second equation:
 
1 2 0 −1
 3 −1 2 5 .
4 1 2 7

• Use the first row to eliminate the coefficients in the first column below the diagonal:
 
1 2 0 −1
 0 −7 2 8 .
0 −7 2 11

• Use the second row to eliminate the coefficients in the second column below the diag-
onal:  
1 2 0 −1
 0 −7 2 8 .
0 0 0 3
142 Chapter 5. Vector Spaces: Theory and Practice

Now, what does this mean?

• We notice that 0 × χ2 = 3. This is a contradiction, and therefore this linear system

has no solution!

Consider where we started: Ax = b1 represents

3χ0 + (−1)χ1 + 2χ2 = = 5

1χ0 + 2χ1 + (0)χ2 = = −1
4χ0 + χ1 + 2χ2 = = 7.

Now, the last equation is a linear combination of the first two. Indeed, add the first equation
to the second, you get the third. Well, not quite: The last equation is actually inconsistent,
because if you subtract the first two rows from the last, you don’t get 0 = 0. As a result,
there is no way of simultaneously satisfying these equations.

5.2.1 A systematic procedure (first try)

Let us analyze what it is that we did in Examples 5.12 and 5.13.

• We start with A ∈ Rm×n , x ∈ Rn , and b ∈ Rm where Ax = b and x is to be computed.

) *
• Append the system A b .

• Use the Gauss-Jordan method to transform this appended system into the form
' (
Ik×k ÃT R b̃T
, (5.4)
0(m−k)×k 0(m−k)×(n−k) b̃B

where Ik×k is the k × k identity matrix, ÃT R ∈ Rk×(n−k) , b̃T ∈ Rk , and b̃B ∈ Rm−k .

• Now, if b̃B &= 0, then there is no solution to the system and we are done.

• Notice that (5.4) means that

' (' ( ' (
Ik×k ÃT R xT b̃T
= .
0 0 xB 0

Thus, xT + ÃT R xB = b̃T . This translates to xT = b̃T − ÃT R xB or

' ( ' ( ' (
xT b̃T −ÃT R
= + xB .
xB 0 I(m−k)×(m−k)
5.2. Why Should We Care? 143

' (
b̃T
• By taking xB = 0, we find a particular solution xp = .
0
• By taking, successively, xB = ei , i = 0, . . . , (m − k) − 1, we find vectors in the null
space: ' (
−ÃT R
xni = ei .
I(m−k)×(m−k)
• The general solution is then given by
xp + α0 xn0 + · · · + α(m−k)−1 xn(m−k)−1 .

Example 5.17 We now show how to use these insights to systematically solve the problem
in Example 5.12. As in that example, create the appended systems for solving Ax = b0 and
Ax = 0 (Equation (5.1)).
   
3 −1 2 8 3 −1 2 0
 1 2 0 −1   1 2 0 0 . (5.5)
4 1 2 7 4 1 2 0
We notice that for Ax = 0 (the appended system on the right), the right-hand side never
changes. It is always equal to zero. So, we don’t really need to carry out all the steps for
it, because everything to the left of the | remains the same as it does for solving Ax = b.
Carrying through with the Gauss-Jordan method, we end up with Equation (5.2):
 
1 0 4/7 15/7
 0 1 −2/7 −11/7 
0 0 0 0
 
15/7
Now, our procedure tells us that xp =  −11/7  is a particular solution: it solves Ax =
0 
' ( 4/7 ' (
ÃT R   4/7
b. Next, we notice that = −2/7 so that AT R = , and
I(m−k)×(m−k) −2/7
1
I(m−k)×(m−k) = 1 (since there is only one free variable). So,
   
' ( −4/7 −4/7
−ÃT R
xn = e0 =  2/7  1 =  2/7 
I(m−k)×(m−k)
1 1
The general solution is then given by
   
15/7 −4/7
x = xp + αxn =  −11/7  + α  2/7  ,
0 1
for any scalar α.
144 Chapter 5. Vector Spaces: Theory and Practice

5.2.2 A systematic procedure (second try)

Example 5.18 We now give an example where the procedure breaks down. Note: this
example is borrowed from the book.
Consider Ax = b where
   
1 3 3 2 2
A= 2 6 9 7  and b =  10  .
−1 −3 3 4 10

Let us apply Gauss-Jordan to this:

• Append the system:

 
1 3 3 2 2
 2 6 9 7 10  .
−1 −3 3 4 10

• The boldfaced “1” is the pivot, in the first column. Subtract 2/1 times the first row
and (−1)/1 times the first row from the second and third row, respectively:
 
1 3 3 2 2
 0 0 3 3 6 .
0 0 6 6 12

• The problem is that there is now a “0” in the second column. So, we skip it, and move
on to the next column. The boldfaced “3” now becomes the pivot. Subtract 6/3 times
the second row from the third row:
 
1 3 3 2 2
 0 0 3 3 6 .
0 0 0 0 0

• Divide each (nonzero) row by the pivot in that row to obtain

 
1 3 3 2 2
 0 0 1 1 2 .
0 0 0 0 0
5.2. Why Should We Care? 145

• We can only eliminate elements in the matrix above pivots. So, take 3/1 times the
second row and subtract from the first row:
 
1 3 0 −1 −4
 0 0 1 1 2 . (5.6)
0 0 0 0 0

• This does not have the form advocated in Equation 5.4. So, we remind ourselves of
the fact that Equation 5.6 stands for
 
  χ0  
1 3 0 −1   −4
χ1  
 0 0 1 1 
 χ2  = 2  (5.7)
0 0 0 0 0
χ3
Notice that we can swap the second and third column of the matrix as long as we also
swap the second and third element of the solution vector:
 
  χ0  
1 0 3 −1   −4
χ 2 
 0 1 0 1  χ1  =
 2 . (5.8)
0 0 0 0 0
χ3

• Now, we notice that χ1 and χ3 are the free variables, and with those we can find
equations for χ0 and χ2 as before.
• Also, we can now find vectors in the null space just as before. We just have to pay
attention to the order of the unknowns (the order of the elements in the vector x).
In other words, a specific solution is now given by
   
χ0 −4
 χ2   2 
xp   
 χ1  =  0 


χ3 0
and two linearly independent vectors in the null space are given by the columns of
 
−3 1
 0 −1 
 
 1 0 
0 1
giving us a general solution of
       
χ0 −4 3 1
 χ2   2   0   
      + β  −1  .
 χ1  =  0  + α  1   0 
χ3 0 0 1
146 Chapter 5. Vector Spaces: Theory and Practice

But notice that the order of the elements in the vector must be fixed (permuted):
       
χ0 −4 3 1
 χ1   0   1   0 
       
 χ2  =  2  + α  0  + β  −1  .
χ3 0 0 1

Exercise 5.19 Let A ∈ Rm×n , x ∈ Rm , and b ∈ Rn . Let P ∈ Rm×m be a permutation

matrix. Show that AP T P x = b. Argue how this relates to the transition from Equation 5.7
to Equation 5.8.

Exercise 5.20 Complete Example 5.18 by computing a particular solution and two vectors
in the null space (one corresponding to χ1 = 1, χ3 = 0 and the other to χ1 = 0, χ3 = 1).

5.3 Linear Independence

Definition 5.21 Let {a0 , . . . , an−1 } ⊂ Rm . Then this set of vectors is said to be linearly
independent if χ0 a0 + χ1 a1 + · · · + χn−1 an−1 = 0 implies that χ0 = · · · = χn−1 = 0. A set
of vectors that is not linearly independent is said to be linearly dependent.

Notice that if
χ0 a0 + χ1 a1 + · · · + χn−1 an−1 = 0 and χj &= 0,
then
χj aj = −χ0 a0 + −χ1 a1 − · · · − χj−1 aj−1 − χj+1 aj+1 − · · · − χn−1 an−1
and therefore
χ0 χ1 χj−1 χj+1 χn−1
aj = − a0 + − a1 − · · · − aj−1 − aj+1 − · · · − an−1 .
χj χj χj χj χj
In other words, aj can be written as a linear combination of the other n − 1 vectors. This
motivates the term linearly independent in the definition: none of the vectors can be written
as a linear combination of the other vectors.
) *
Theorem 5.22 Let {a0 , . . . , an−1 } ⊂ Rm and let A = a0 · · · an−1 . Then the vec-
tors {a0 , . . . , an−1 } are linearly independent if and only if N (A) = {0}.
5.3. Linear Independence 147

Proof:

(⇒) Assume {a0 , . . . , an−1 } are linearly independent. We need to show that N(A) = {0}.

χ0
) * 
Assume x ∈ N (A). Then Ax = 0 implies that 0 = Ax = a0 · · · an−1  ...  =
χn−1
χ0 a0 + χ1 a1 + · · · + χn−1 an−1 and hence χ0 = · · · = χn−1 = 0. Hence x = 0.

(⇐) Notice that we are trying to prove P ← Q, where P represents “the vectors {a0 , . . . , an−1 }
are linearly independent” and Q represents “N (A) = {0}”. It suffices to prove the con-
verse: ¬P → ¬Q. Assume that {a0 , . . . , an−1 } are not linearly independent. Then there
exist {χ0 , · · · , χn−1 } with at least one χj &= 0 such that χ0 a0 +χ1 a1 +· · ·+χn−1 an−1 = 0.
Let x = (χ0 , . . . , χn−1 )T . Then Ax = 0 which means x ∈ N (A) and hence N (A) &= {0}.

Example 5.23 The columns of an identity matrix I ∈ Rn×n form a linearly independent
set of vectors.

Proof: Since I has an inverse (I itself) we know that N (I) = {0}. Thus, by Theorem 5.22,
the columns of I are linearly independent.


1 0 0
Example 5.24 The columns of L =  2 −1 0  are linearly independent. If we consider
1 2 3
    
1 0 0 χ0 0
 2 −1 0   χ1  =  0 
1 2 3 χ2 0

and simply solve this, we find that χ0 = 0/1 = 0, χ1 = (0 − 2χ0 )/(−1) = 0, and χ2 = (0 −
χ0 − 2χ1 )/(3) = 0. Hence, N (L) = {0} (the zero vector) and we conclude, by Theorem 5.22,
that the columns of L are linearly independent.

The last example motivates the following theorem:

148 Chapter 5. Vector Spaces: Theory and Practice

Theorem 5.25 Let L ∈ Rn×n be a lower triangular matrix with nonzeroes on its diago-
nal. Then its columns are linearly independent.

Proof: Let L be as indicated and consider Lx = 0. If one solves this via whatever method
one pleases, the solution x = 0 will emerge as the only solution. Thus N (L) = {0} and by
Theorem 5.22, the columns of L are linearly independent.

Exercise 5.26 Let U ∈ Rn×n be an upper triangular matrix with nonzeroes on its diagonal.
Then its columns are linearly independent.

Exercise 5.27 Let L ∈ Rn×n be a lower triangular matrix with nonzeroes on its diagonal.
Then its rows are linearly independent. (Hint: How do the rows of L relate to the columns
of LT ?)

 
1 0 0
 2 −1 0 
Example 5.28 The columns of L =   are linearly independent. If we
 1 2 3 
−1 0 −2
consider    
1 0 0   0
 χ0
 2 −1 0   0 
  χ1  =  
 1 2 3   0 
χ2
−1 0 −2 0
and simply solve this, we find that χ0 = 0/1 = 0, χ1 = (0 − 2χ0 )/(−1) = 0, χ2 = (0 − χ0 −
2χ1 )/(3) = 0. Hence, N (L) = {0} (the zero vector) and we conclude, by Theorem 5.22, that
the columns of L are linearly independent.

This example motivates the following general observation:

' A ∈( R 'have(linearly independent columns and let B ∈ R .

m×n k×n
Theorem 5.29 Let
A B
Then the matrices and have linearly independent columns.
B A
5.4. Bases 149

' (
A
Proof: Proof by contradiction. Assume that is not linearly independent. Then, by
B
' (
A
Theorem 5.22, there exists x ∈ R such that x &= 0 and
n
x = 0. But that means that
B
' ( ' (
Ax 0
= , which in turn implies that Ax = 0. This contradicts the fact that the
Bx 0
columns of A are linearly independent.

' (
A
Corollary 5.30 Let A ∈ R m×n
. Then the matrix has linearly independent
In×n
columns.

Next, we observe that if one has a set of more than m vectors in Rm , then they must be
linearly dependent:
.
Theorem 5.31 Let {a0 , a1 , .., an−1 } ∈ Rm and n > m. Then these vectors are linearly
dependent.

Proof:
) This proof is*a bit more informal than I would like it to be: Consider the matrix
A = a0 · · · an−1 . If one apply the Gauss-Jordan method to this, at most m columns
with pivots will be encountered. The other n − m columns correspond to free variables,
which allow us to construct nonzero vectors x so that Ax = 0.

5.4 Bases

Definition 5.32 Let {v0 , v1 , · · · , vn−1 } ⊂ Rm . Then the span of these vectors,
Span({v0 , v1 , · · · , vn−1 }), is said to be the space of all vectors that are a linear combi-
nation of this set of vectors.
) *
Notice that Span({v0 , v1 , · · · , vn−1 }) equals the column space of the matrix v0 · · · vn .

Definition 5.33 Let V be a subspace of Rm . Then the set {v0 , v1 , · · · , vn−1 } ⊂ Rm is

said to be a spanning set for V if Span({v0 , v1 , · · · , vn−1 }) = V.
150 Chapter 5. Vector Spaces: Theory and Practice

Definition 5.34 Let V be a subspace of Rm . Then the set {v0 , v1 , · · · , vn−1 } ⊂ Rm

is said to be a basis for V if (1) {v0 , v1 , · · · , vn−1 } are linearly independent and (2)
Span({v0 , v1 , · · · , vn−1 }) = V.

The first condition says that there aren’t more vectors than necessary in the set. The
second says there are enough to be able to generate V.

Example 5.35 The vectors {e0 , . . . , em−1 } ⊂ Rm , where ej equals the jth column of the
identity, are a basis for Rm .  
χ0
 
Note: these vectors are linearly independent and any vector x ∈ Rm with x =  ... 
χm−1
can be written as the linear combination χ0 e0 + · · · + χm−1 em−1 .

) *
Example 5.36 Let {a0 , . . . , am−1 } ⊂ Rm and let A = a0 · · · am−1 be invertible. Then
{a0 , . . . , am−1 } ⊂ Rm form a basis for Rm .
Note: The fact that A is invertible means there exists A−1 such that A−1 A = I. Since
Ax = 0 means x = A−1 Ax = A−1 0 = 0, the columns of A are linearly independent. Also,
 y ∈ R , there exists a vector x ∈ R such that Ax = y (namely x = A y).
m m −1
given any vector
χ0
 .. 
Letting x =  .  we find that y = χ0 a0 + · · · + χm−1 am−1 and hence every vector in
χm−1
R is a linear combination of the set {a0 , . . . , am−1 } ⊂ Rm .
m

Now here comes a very important insight:

Theorem 5.37 Let V be a subspace of Rm and let {v0 , v1 , · · · , vn−1 } ⊂ Rm and
{w0 , w1 , · · · , wk−1 } ⊂ Rm both be bases for V. Then k = n. In other words, the number
of vectors in a basis is unique.

Proof: Proof by contradiction. Without loss of generality, let ) us assume that * k > n.
(Otherwise,
) we can
* switch the roles of the two sets.) Let V = v 0 · · · v n−1 and W =
w0 · · · wk−1 . Let xj have the property that wj = V xj . (We know) such a vector x*j
exists because V spans V and wj ∈ V.) Then W = V X, where X = x0 · · · xk−1 .
Now, X ∈ Rn×k and recall that k > n. This means that N (X) contains nonzero vectors
(why?). Let y ∈ N (X). Then W y = V Xy = V (Xy) = V (0) = 0, which contradicts the fact
5.5. Exercises 151

that {w0 , w1 , · · · , wk−1 } are linearly independent, and hence this set cannot be a basis for V.

Note: generally speaking, there are an infinite number of bases for a given subspace.
(The exception is the subspace {0}.) However, the number of vectors in each of these bases
is always the same. This allows us to make the following definition:
Definition 5.38 The dimension of a subspace V equals the number of vectors in a basis
for that subspace.

A basis for a subspace V can be derived from a spanning set of a subspace V by, one-
to-one, removing vectors from the set that are dependent on other remaining vectors until
the remaining set of vectors is linearly independent , as a consequence of the following
observation:
Definition 5.39 Let A ∈ Rm×n . The rank of A equals the number of vectors in a basis
for the column space of A. We will let rank(A) denote that rank.

Theorem 5.40 Let {v0 , v1 , · · · , vn−1 } ⊂ Rm be a spanning set for subspace V

and assume that vi equals a linear combination of the other vectors. Then
{v0 , v1 , · · · , vi−1 , vi , vi−1 , · · · , vn−1 } is a spanning set of V.

Similarly, a set of linearly independent vectors that are in a subspace V can be “built
up” to be a basis by successively adding vectors that are in V to the set while maintaining
that the vectors in the set remain linearly independent until the resulting is a basis for V.
Theorem 5.41 Let {v0 , v1 , · · · , vn−1 } ⊂ Rm be linearly independent and assume that
{v0 , v1 , · · · , vn−1 } ⊂ V . Then this set of vectors is either a spanning set for V or there
exists w ∈ V such that {v0 , v1 , · · · , vn−1 , w} are linearly independent.

5.5 Exercises
(Most of these exercises are borrowed from “Linear Algebra and Its Application” by Gilbert
Strang.)
1. Which of the following subsets of R3 are actually subspaces?
(a) The plane of vectors x = (χ0 , χ1 , χ2 )T ∈ R3 such that the first component χ0 = 0.
In other words, the set of all vectors
 
0
 χ1 
χ2
152 Chapter 5. Vector Spaces: Theory and Practice

where χ1 , χ2 ∈ R.
(b) The plane of vectors x with χ0 = 1.
(c) The vectors x with χ1 χ2 = 0 (this is a union of two subspaces: those vectors with
χ1 = 0 and those vectors with χ2 = 0).
(d) All combinations of two given vectors (1, 1, 0)T and (2, 0, 1)T .
(e) The plane of vectors x = (χ0 , χ1 , χ2 )T that satisfy χ2 − χ1 + 3χ0 = 0.

2. Describe the column space and nullspace of the matrices

' ( ' ( ' (
1 −1 0 0 3 0 0 0
A= , B= , and C = .
0 0 1 2 3 0 0 0

3. Let P ⊂ R3 be the plane with equation x + 2y + z = 6. What is the equation of the

plane P0 through the origin parallel to P? Are P and/or P0 subspaces of R3 ?

4. Let P ⊂ R3 be the plane with equation x + y − 2z = 4. Why is this not a subspace?

Find two vectors, x and y, that are in P but with the property that x + y is not.

5. Find the echolon form U , the free variables, and the special (particular) solution of
Ax = b for
' ( ' (
0 1 0 3 β0
(a) A = ,b= . When does Ax = b have a solution? (When
0 2 0 6 β1
β1 = ?.) Give the complete solution.
   
0 0 β0
 1 2   
(b) A =  , b =  β1 . When does Ax = b have a solution? (When ...)
 0 0   β2 
3 6 β3
Give the complete solution.

6. Write the complete soluton x = xp + xn to these systems:

 
' ( u ' (
1 2 2  1
v =
2 4 5 4
w

and  
' ( u ' (
1 2 2  v = 1
2 4 4 4
w

7. Which of these statements is a correct definition of the rank of a given matrix A ∈

Rm×n ? (Indicate all correct ones.)
5.5. Exercises 153

(a) The number of nonzero rows in the row reduced form of A.

(b) The number of columns minus the number of rows, n − m.
(c) The number of columns minus the number of free columns in the row reduced
form of A. (Note: a free column is a column that does not contain a pivot.)
(d) The number of 1s in the row reduced form of A.

8. Let A, B ∈ Rm×n , u ∈ Rm , and v ∈ Rn . The operation B := A + αuv T is often called

a rank-1 update. Why?

9. Find the complete solution of

x +3y +z = 1
2x +6y +9z = 5 .
−x −3y +3z = 5

10. Find the complete solution of

 
 x  
1 3 1 2   1
 2 6 4 8  y  =  3 .
 z 
0 0 2 4 1
t
154 Chapter 5. Vector Spaces: Theory and Practice

5.6 The Answer to Life, The Universe, and Everything

We complete this chapter by showing how many answers about subspaces can be answered
from the upper-echolon form of the linear system.
To do so, consider again
 
  x  
1 3 1 2   1
 2 6 4 8  y  =  3 
 z 
0 0 2 4 1
t

from Question 10 in the previous section. Reducing this to upper echolon format yields
   
1 3 1 2 1 1 3 1 2 1
 2 6 4 8 3  →  0 0 2 4 1 .
0 0 2 4 1 0 0 0 0 0

Here the boxed entries are the pivots (the first nonzero entry in each row) and they identify
that the corresponding variables (x and z) are dependent variables while the other variables
(y and t) are free variables.

Give the general solution to the problem To find the general solution to problem, you
recognize that there are two free variables (y and t) and a general solution can thus be given
by
     
 0     
  + α 1  + β  0 .
     
0 0 1
 
 0 
Here xp = 
 is a particular (special) solution that solves the system. To obtain it,

0
you set the free variables to zero and solve for the values in the boxes:
 
 
' ( x 1
1 3 1 2  0 
 = 1 
0 0 2 4  z 
0
0

or
x +z = 1
2z = 1
5.6. The Answer to Life, The Universe, and Everything 155

 
1/2
 0 
so that z = 1/2 and x = 1/2 yielding a particular solution xp =  
 1/2 .
0
Next, we have to find the two vectors in the null space of the matrix. To obtain the first,
we set the first free variable to one and the other(s) to zero:
 
 
' ( x 0

1 3 1 2  1    
= 0
0 0 2 4  z 
0
0
or
x +3 × 1 +z = 0
2z = 0
 
−3
 1 
so that z = 0 and x = −3, yielding the first vector in the null space xn0 = 
 0 .


0
To obtain the second, we set the second free variable to one and the other(s) to zero:
 
 
' ( x 0

1 3 1 2  0    
= 0
0 0 2 4  z 
0
1
or
x +z +2 × 1 = 0
2z +4 × 1 = 0
so thatz = −4/2
 = −2 and x = −z − 2 = 0, yielding the second vector in the null space
0
 0 
xn1 =  
 −2 .
1
And thus the general solution is given by
     
1/2 −3 0
 0     
  + α 1  + β  0 ,
 1/2   0   −2 
0 0 1
where α and β are scalars.

Find a specific (particular) solution to the problem The above procedure yields the
particular solution as part of the first step.
156 Chapter 5. Vector Spaces: Theory and Practice

Find vectors in the null space The first step is to figure out how many (linear independent)
vectors there are in the null space. This equals the number of free variables. The above
procedure then gives you a step-by-step procedure for finding that many linearly independent
vectors in the null space.

Find linearly independent columns in the original matrix Note: this is equivalent to
finding a basis for the column space of the matrix.
To find the linearly independent columns, you look at the upper echolon form of the
matrix:  
1 3 1 2
 0 0 2 4 
0 0 0 0
with the pivots highlighted. The columns that have pivots in them are linearly independent.
The corresponding columns in the original matrix are linearly independent:
 
1 3 1 2
 2 6 4 8 .
0 0 2 4
   
1 1
Thus, in our example, the answer is  2  and  4  (the first and third column).
0 2

Dimension of the Column Space (Rank of the Matrix) The following are all equal:

• The dimension of the column space.

• The rank of the matrix.

• The number of dependent variables.

• The number of nonzero rows in the upper echelon form.

• The number of columns in the matrix minus the number of free variables.

• The number of columns in the matrix minus the dimension of the null space.

• The number of linearly independent columns in the matrix.

• The number of linearly independent rows in the matrix.

5.6. The Answer to Life, The Universe, and Everything 157

Find a basis for the row space of the matrix. The row space (we will see in the next
chapter) is the space spanned by the rows of the matrix (viewed as column vectors). Reducing
a matrix to upper echelon form merely takes linear combinations of the rows of the matrix.
What this means is that the space spanned by the rows of the original matrix is the same
space as is spanned by the rows of the matrix in upper echolon form. Thus, all you need to
do is list the rows in the matrix in upper echolon form, as column vectors.  
1
 3 
For our example this means a basis for the row space of the matrix is given by   1 


2
 
0
 0 
and  2 .
