https://www.halvorsen.

blog

Linear Algebra
in Python
Hans-Petter Halvorsen
 Free Textbook with lots of Practical Examples

https://www.halvorsen.blog/documents/programming/python/
 Additional Python Resources

https://www.halvorsen.blog/documents/programming/python/
 Contents
• Linear Algebra
• Matrices and Vectors
• The NumPy Library
• Solving Linear Equations
• Python Examples
 Linear Algebra
• Linear Algebra is central and important in almost all areas
of mathematics
• Linear Algebra is the “Mathematics of Data“
• Foundation: Vectors and Matrices
• Linear Equations
• Modern statistics and data analysis depends on Linear
Algebra
• Linear Algebra plays an important role in advanced
Control Engineering
• Linear Algebra plays an important role in Machine
Learning
 Matrices
A matrix is a two-dimensional data structure where numbers are arranged into rows
and columns.
A general matrix is given by:

𝑎!! ⋯ 𝑎!"
𝐴= ⋮ ⋱ ⋮
𝑎#! ⋯ 𝑎#"
Where 𝑛 is number of rows and 𝑚 is number of columns.

Matrices are very important data structures for many mathematical and scientific
calculations.
 Examples
Example of a 3 x 3 matrix:
1 5 3
𝐴= 4 6 6 Example of a 4 x 2 matrix:
3 8 9
1 5
Example of a 3 x 4 matrix: 4 5
𝐴=
3 2
1 5 3 4 7 8
𝐴= 4 5 7 8
7 8 9 3
 Vectors
A vector is a one-dimensional data structure where numbers are arranged into rows
or columns.
A general vector is given by: 𝑥!
𝑥$
𝑥= ⋮
𝑥#
Where n is number of rows and m is number of columns.

Matrices and vectors are very important data structures for many mathematical and scientific
calculations.
 Vectors
Assume the vector 𝑥: The Length of vector 𝑥 is given by:
𝑥!
𝑥" 𝑥 = 𝑥$𝑥 = 𝑥!" + 𝑥"" + ⋯ + 𝑥#"
𝑥= ⋮
𝑥# The length of a vector most makes sense
for 2 or 3 dimensional vectors.
The Transpose of vector 𝑥 is It can be visualized like this:
Example:
𝑥 = 𝑥!
$ 𝑥" ⋯ 𝑥#
𝑣 = 3, 4

In order to find the length of 𝑣 we use Pythagoras like this:

𝑣 = 3" + 4" = 9 + 16 = 25 = 5
 Standard Python
Python doesn't have a built-in type for matrices. However, we can treat list of a list as a matrix.
a = [1, 3, 7, 2]
Example of a 3 x 4 matrix:
1 3 7 2
𝐴= 5 8 −9 0 print("a =", a)
6 −7 11 12
Example of vector: A = [[1, 3, 7, 2],
1
3 [5, 8, -9, 0],
𝑎= 7 [6, -7, 11, 12]]
2
print("A =", A)
So we can define vectors and matrices with standard Python, but standard Python has no
support for manipulation and calculation of them.
But fortunately we can use the NumPy package for creating matrices and for matrix
manipulation.
Linear algebra (numpy.linalg)
• The NumPy library has a submodule for Linear
Algebra, namely numpy.linalg
https://numpy.org/doc/stable/reference/routines.linalg.html

• The SciPy library also contains a linalg submodule,

and there is overlap in the functionality provided
by the SciPy and NumPy submodules.
https://docs.scipy.org/doc/scipy/reference/linalg.html#module-
scipy.linalg
 numpy.array
Let's use the Array feature in the NumPy Library: import numpy as np

a = np.array([[1],
[3],
Example of a 3 x 4 matrix: [7],
[2]])
1 3 7 2
𝐴= 5 8 −9 0 print("a =", a)
6 −7 11 12
A = np.array([[1, 3, 7, 2],
Example of vector:
[5, 8, -9, 0],
1 [6, -7, 11, 12]])
3
𝑎= 7
print("A =", A)
2
 numpy.matrix
import numpy as np
Let's use the Matrix feature in the NumPy Library:
a = np.array([[1],
[3],
Example of a 3 x 4 matrix: [7],
1 3 7 2 [2]])
𝐴= 5 8 −9 0
6 −7 11 12 print("a =", a)

Example of vector: 1 A = np.matrix([[1, 3, 7, 2],

3 [5, 8, -9, 0],
𝑎= 7
[6, -7, 11, 12]])
2
print("A =", A)
 numpy.matrix
Note!
It is no longer recommended to use this class, even for linear
algebra. Instead use regular arrays. The class may be
removed in the future. See NumPy documentation for details.

More Information:

https://numpy.org/doc/stable/reference/generated/numpy.matrix.html
 Matrix Addition, Subtraction, Multiplication
0 1 1 0
Given the following: 𝐴= 𝐵=
−2 −3 3 −2

Matrix Addition:
0 1 1 0 0+1 1+0 1 1
𝐴+B= + = =
−2 −3 3 −2 −2 + 3 −3 + (−2) 1 −5

Matrix Subtraction:
0 1 1 0 0−1 1−0 −1 1
𝐴−B= − = =
−2 −3 3 −2 −2 − 3 −3 − (−2) −5 −1
Matrix Multiplication:

0 1 1 0 0;1+1;3 0 ; 0 + 1 ; (−2) 3 −2
𝐴B = == =
−2 −3 3 −2 −2 ; 1 − 3 ; 3 −2 ; 0 − 3 ; (−2) −11 6
 Matrix Addition, Subtraction, Multiplication
Matrix Addition, Subtraction, Multiplication import numpy as np

In this example we use numpy.matrix A = np.matrix([[0, 1],

[-2, -3]])
Given the following matrices:
B = np.matrix([[1, 0],
0 1 A+B = [3, -2]])
𝐴= [[ 1 1]
−2 −3
[ 1 -5]] C = A + B
1 0 A-B = print("A+B =", C)
𝐵= [[-1 1]
3 −2
[-5 -1]] C = A - B
Note! A*B = print("A-B =", C)
It is no longer recommended to use this
class, even for linear algebra. Instead use [[3 -2]
C = A * B
regular arrays. The class may be removed [-11 6]] print("A*B =", C)
in the future. See NumPy documentation.
 import numpy as np

Cont. A = np.array([[0, 1],

[-2, -3]])

B = np.array([[1, 0],
[3, -2]])
Matrix Addition, Subtraction, Multiplication
C = A + B
print("A+B =", C)
In this example we use numpy.array C = A - B
print("A-B =", C)

Given the following matrices: C = A * B

print("A*B =", C) # Not Working!, only elementwise
A+B = [[ 1 1]
0 1 [ 1 -5]]
Multiplication!
𝐴= A-B = [[-1 1]
−2 −3 [-5 -1]]
#Working Alternative 1
C = A.dot(B)
print("A*B =", C)
A*B = [[ 0 0]
1 0
𝐵= [-6 6]] #Working Alternative 2
3 −2 C = np.dot(A,B)
print("A*B =", C)
A*B = [[ 3 -2]
[-11 6]]
#Working Alternative 3
A*B = [[ 3 -2]
C = np.mat(A) * np.mat(B)
[-11 6]]
print("A*B =", C)
A*B = [[ 3 -2]
[-11 6]] #Working Alternative 4
A*B = [[ 3 -2] C = np.matmul(A,B)
[-11 6]] print("A*B =", C)
 Matrix Multiplication
• In matrix multiplication the matrices don't need to be
quadratic, but the inner dimensions need to be the same.
• The size of the resulting matrix will be the outer dimensions.

𝐴 × 𝐵 = 𝐶
𝑛×𝑚 𝑚×𝑝 𝑛×𝑝
The resulting matrix will
Inner dimensions be the outer dimensions
need to be the same
 import numpy as np

Examples A = np.array([[1, 5, 3],

[4, 6, 6],
[3, 8, 9]])
1 5 3 B = np.array([[1, 5, 3, 4],
𝐴= 4 6 6 1 5 3 4
[4, 5, 7, 8],
[7, 8, 9, 3]])
3 8 9 𝐵= 4 5 7 8 C = np.array([[1],

1 (3×3) 7 8 9 3 [4],
[3]])

𝐶= 4 (3×4) D = np.array([[1, 5, 3]])

3 𝐷= 1 5 3 M = np.matmul(A,B)
print("M=", M)
(3×1) (1×3)
#M = np.matmul(B,A) # Not Working!

𝐴×𝐵 = 3×3 × 3×4 = (3×4) M = np.matmul(A,C)

print("M=", M)
𝐵×𝐴 = 3×4 × 3×3 = Not Legal
#M = np.matmul(C,A) # Not Working!
𝐴×𝐶 = 3×3 × 3×1 = (3×1)
M = np.matmul(C,D)
C×𝐴 = 3×1 × 3×3 = Not Legal print("M=", M)
C×𝐷 = 3×1 × 1×3 = (3×3) M = np.matmul(D,C)
𝐷×𝐶 = 1×3 × 3×1 = (1×1) etc. print("M=", M)
 Transpose of a Matrix
A general matrix is given by:
𝑎@@ 𝑎@A ⋯ 𝑎@B
𝑎A@ 𝑎AA ⋯ 𝑎AB Where 𝑛 is number of rows
𝐴= ⋮ ⋮ ⋱ ⋮ and 𝑚 is number of columns
𝑎C@ 𝑎CA ⋯ 𝑎CB (𝑛×𝑚)

The transpose of matrix 𝐴 is then:

𝑎// 𝑎0/ ⋯ 𝑎1/
.
𝑎/0 𝑎00 ⋯ 𝑎10
𝐴 = ⋮ ⋮ ⋱ ⋮
𝑎/2 𝑎02 ⋯ 𝑎12 (𝑚×𝑛)
 Transpose of a Matrix - Examples

1 5 6
1 3 7 2
3 8 −7
𝐴 = 5 8 −9 0 𝐴D =
7 −9 11
6 −7 11 12
2 0 12

1 5
4 5 1 4 3 7
𝐵= 𝐵D =
3 2 5 5 2 8
7 8
Transpose of a Matrix import numpy as np

A = np.array([[1, 3, 7, 2],
[5, 8, -9, 0],
1 3 7 2 [6, -7, 11, 12]])
𝐴= 5 8 −9 0 print("A="); print(A)
6 −7 11 12
Atr = np.transpose(A)
Transpose of A=
𝐴$ = ?
[[ 1 5 6] print("Transpose of A="); print(Atr)
[ 3 8 -7]
1 5 [ 7 -9 11] B = np.array([[1, 5],
[ 2 0 12]] [4, 5],
4 5
𝐵= [3, 2],
3 2 [7, 8]])
7 8 print("B="); print(B)
Transpose of B=
[[1 4 3 7] Btr = np.transpose(B)
𝐵$ =? [5 5 2 8]]
print("Transpose of B="); print(Btr)
 Determinant of a Matrix

• The Determinant of a matrix is a special number that can be

calculated from square matrices

What is the Determinant used for?

• The determinant helps us find the inverse matrix (which we
will cover later)
• The Determinant will give us useful information when dealing
with Systems of Linear Equations (which we will cover later)
• Used in advanced Control Engineering theory
• Etc.
 Determinant of a Matrix
Given a matrix 𝐴 the Determinant is given by:

𝑑𝑒𝑡 𝐴 = 𝐴

For a 2𝑥2 matrix we have:

𝑎@@ 𝑎@A
𝐴= 𝑎 𝑑𝑒𝑡 𝐴 = 𝐴 = 𝑎@@ 𝑎AA − 𝑎A@ 𝑎@A
A@ 𝑎 AA
Example:
1 2
𝐴= 𝑑𝑒𝑡 𝐴 = 𝐴 = 1 6 4 − 3 6 2 = 4 − 6 = −2
3 4
 Determinant of a Matrix
For a 3𝑥3 matrix we have: We develop the determinant along a row or a column.
𝑎!! 𝑎!" 𝑎!%
𝐴 = 𝑎"! 𝑎"" 𝑎"%
𝑎%! 𝑎%" 𝑎%%
𝑎"" 𝑎"# 𝑎!" 𝑎!# 𝑎!" 𝑎!#
𝑑𝑒𝑡 𝐴 = 𝐴 = 𝑎!! 𝑎#" 𝑎## − 𝑎"! 𝑎#" 𝑎## + 𝑎#! 𝑎"" 𝑎"#

Here we have developed the determinant along the first column

We see that the determinant of a higher order system can be expressed as a sum of lower
order determinants

It will be a little more complicated and time-consuming work for systems with larger order,
so we need a programming language like python to handle this.
 Determinant of a Matrix
Example:
−1 3 0
𝐴= 2 1 −5 1 −5 3 0 3 0
𝑑𝑒𝑡 𝐴 = (−1) −2 +1
1 4 −2 4 −2 4 −2 1 −5

This gives:
1 −5
= −2 − −20 = 18
4 −2
3 0
= −6 − 0 = −6
4 −2 3 0
= −15 − 0 = −15
1 −5
Finally:

𝑑𝑒𝑡 𝐴 = −18 + 12 − 15 = −21

 Determinant of a Matrix
Given the following Matrices: import numpy as np
import numpy.linalg as la
1 2
𝐴= −1 3 0
3 4 A = np.array([[1, 2],
𝐵= 2 1 −5 [3, 4]])
𝑑𝑒𝑡 𝐴 = −2 1 4 −2
Adet = la.det(A)
𝑑𝑒𝑡 𝐵 = −21
print(Adet)

B = np.array([[-1, 3, 0],
Python Solution:
[2, 1, -5],
-2.0000000000000004 [1, 4, -2]])
-21.00000000000001
Bdet = la.det(B)

print(Bdet)
 Inverse Matrices
The inverse of a quadratic matrix 𝐴 is defined by: 𝐴&! Note: 𝐴𝐴&! = 𝐴&! 𝐴 = 𝐼

For a 2𝑥2 matrix we have:

The inverse 𝐴&! is then given by
𝑎!! 𝑎!"
𝐴= 𝑎 𝑎"" 1 𝑎"" −𝑎!"
"!
𝐴&! =
𝑑𝑒𝑡(𝐴) −𝑎"! 𝑎!!
Example:

1 2 ! 4 −2 Where:
𝐴= 𝐴&! = '()(+)
3 4 −3 1 𝑑𝑒𝑡 𝐴 = 𝐴 = 1 * 4 − 3 * 2 = 4 − 6 = −2
! 4 −2 −2 1
This gives: 𝐴&! = =
&" −3 1 1.5 −0.5
It will be more complicated for systems with larger order, so we need a programming language
like python to handle this.
 Inverse Matrices
Given the following Matrices: import numpy as np
import numpy.linalg as la
1 2
𝐴= −1 3 0
3 4 A = np.array([[1, 2],
𝐵= 2 1 −5 [3, 4]])
1 4 −2
−2 1
𝐴&! = Ainv = la.inv(A)
1.5 −0.5
𝐵 &! =?
Python Solution: print(Adet)

B = np.array([[-1, 3, 0],
[[-2. 1. ]
[2, 1, -5],
[ 1.5 -0.5]]
[1, 4, -2]])
[[-0.85714286 -0.28571429 0.71428571] Binv = la.inv(B)
[ 0.04761905 -0.0952381 0.23809524]
[-0.33333333 -0.33333333 0.33333333]]
print(Binv)
 Eigenvalues
The Eigenvalues for a given matrix 𝐴 is: det 𝜆𝐼 − 𝐴 = 0
0 1
Example: 𝐴= 𝑑𝑒𝑡 𝐴 = 𝐴 = 𝑎!! 𝑎"" − 𝑎"! 𝑎!"
−2 −3

1 0 0 1 𝜆 0 0 1 𝜆 −1
𝜆 − = − =
0 1 −2 −3 0 𝜆 −2 −3 2 𝜆+3
−𝑏 ± 𝑏 " − 4𝑎𝑐
𝑥=
det 𝜆𝐼 − 𝐴 = 𝜆(𝜆 + 3) − (−1)(2) = 0 2𝑎
−3 ± 3" − 4 ; 1 ; (2) −3 ± 1
"
𝜆 + 3𝜆 + 2 = 0 𝜆= =
2;1 2 𝜆! = −1, 𝜆" = −2
In Python we use the function eig(). For more details:

https://numpy.org/doc/stable/reference/generated/numpy.linalg.eig.html
 Eigenvalues
import numpy as np
0 1 import numpy.linalg as la
𝐴=
−2 −3 A = np.array([[0, 1],
[-2, -3]])
−1
𝑒𝑖𝑔 𝐴 = Aeig, v = la.eig(A)
−2
print(Aeig)

𝜆! = −1, 𝜆$ = −2 [-1. -2.]

 Matrices Rules
Some important Matrices Rules:

𝐴𝐵 ≠ 𝐵𝐴
𝐴 𝐵𝐶 = 𝐴𝐵 𝐶
𝐴 + 𝐵 𝐶 = 𝐴𝐶 + 𝐵𝐶
𝐶 𝐴 + 𝐵 = 𝐶𝐴 + 𝐶𝐵
det 𝐴𝐵 = det 𝐴 det 𝐵
det 𝐴% = det(𝐴)
𝐴𝐴&! = 𝐴&! 𝐴 = 𝐼
 import numpy as np

Matrices Rules A = np.array([[0, 1],

[-2, -3]])
0 1
𝐴= 1 0 B = np.array([[1, 0],
−2 −3 𝐵= [3, -2]])
3 −2 1 1
𝐶=
1 −5 C = np.array([[1, 1],
[1, -5]])
Some important Matrices Rules:
print("AB Not Equal BA")
𝐴𝐵 ≠ 𝐵𝐴 L = np.matmul(A,B)
𝐴 𝐵𝐶 = 𝐴𝐵 𝐶 print("L=");print(L)

𝐴 + 𝐵 𝐶 = 𝐴𝐶 + 𝐵𝐶 R = np.matmul(B,A)
print("R=");print(R)
𝐶 𝐴 + 𝐵 = 𝐶𝐴 + 𝐶𝐵
print("A(BC) = (AB)C")
det 𝐴𝐵 = det 𝐴 det 𝐵 L = np.matmul(A,np.matmul(B,C))
det 𝐴D = det(𝐴) print("L=");print(L)

𝐴𝐴R@ = 𝐴R@𝐴 = 𝐼 R = np.matmul(np.matmul(A,B),C)

print("R=");print(R)
 import numpy as np

Matrices Rules A = np.array([[0, 1],

[-2, -3]])
0 1
𝐴= 1 0 B = np.array([[1, 0],
−2 −3 𝐵= [3, -2]])
3 −2 1 1
𝐶=
1 −5 C = np.array([[1, 1],
[1, -5]])
Some important Matrices Rules:
print("(A+B)C = AC + BC")
𝐴𝐵 ≠ 𝐵𝐴 L = np.matmul(A+B,C)
𝐴 𝐵𝐶 = 𝐴𝐵 𝐶 print("L=");print(L)

𝐴 + 𝐵 𝐶 = 𝐴𝐶 + 𝐵𝐶 R = np.matmul(A,C) + np.matmul(B,C)
print("R=");print(R)
𝐶 𝐴 + 𝐵 = 𝐶𝐴 + 𝐶𝐵
print("AA^(-1) = A^(-1)A = I")
det 𝐴𝐵 = det 𝐴 det 𝐵 L = np.matmul(A, la.inv(A))
det 𝐴D = det(𝐴) print("L=");print(L)

𝐴𝐴R@ = 𝐴R@𝐴 = 𝐼 R = np.matmul(la.inv(A),A)

print("R=");print(R)
https://www.halvorsen.blog

Solving Linear
Equations
Hans-Petter Halvorsen
 Linear Equations
Given the following linear equations:
𝑎!! 𝑥! + 𝑎!" 𝑥" + 𝑎!# 𝑥# + ⋯ = 𝑏!
𝑎"! 𝑥! + 𝑎"! 𝑥" + 𝑎"# 𝑥# + ⋯ = 𝑏"
⋯
These equations can be set on the following general form:

𝐴𝑥 = 𝑏
Where A is a matrix, x is a vector with the unknowns and b Solution:
is a vector of constants
𝑥! 𝑏!
𝑎!! ⋯ 𝑎!- 𝑥" 𝑏" 𝑥 = 𝐴;/ 𝑏
𝐴= ⋮ ⋱ ⋮ 𝑥 = ⋮ 𝑏 =
⋮
𝑎#! ⋯ 𝑎#- 𝑥# 𝑏# (assuming 𝐴&! is possible )
 Example
Given the following linear equations:

𝑥/ + 2𝑥0 = 5
3𝑥/ + 4𝑥0 = 6
We want to put the equations on the following general form:

𝐴𝑥 = 𝑏
This gives: The solution is given by:

!"
𝐴=
1 2
𝑏=
5 𝑥!
𝑥= 𝑥
𝑥=𝐴 𝑏
3 4 6 "
 Example - Python
Python code:
import numpy as np
import numpy.linalg as la This gives the following solution:
[[-4. ]
A = np.array([[1, 2], [ 4.5]]
[3, 4]])
This means:
b = np.array([[5], 𝑥! = −4
[6]]) 𝑥" = 4.5

Ainv = la.inv(A) Which is the same as the solution we

got from our manual calculations
x = Ainv.dot(b)
Note! The A matrix must be square and of full-
print(x) rank, i.e. the inverse matrix needs to exists.
 Example – Python (Alt2)
We can also use the linalg.solve()function
x = np.linalg.solve(A, b)
Python code:
import numpy as np This gives the following solution:
[[-4. ]
This means:
A = np.array([[1, 2], [ 4.5]]
𝑥! = −4
[3, 4]]) 𝑥" = 4.5

b = np.array([[5], Which is the same as the solutions we

[6]]) got from the other methods

x = np.linalg.solve(A, b) Note! The A matrix must be square and of full-

rank, i.e. the inverse matrix needs to exists.
print(x)
Python Example – Least Square
Python code: Given the following linear equations:
import numpy as np 𝑥@ + 2𝑥A = 5
A = np.array([[1, 2], 3𝑥@ + 4𝑥A = 6
Note! In this example the
[3, 4], 𝐴 matrix is not quadratic, 7𝑥@ + 8𝑥A = 9
[7, 8]]) and the inverse of 𝐴 does
not exist! We need to use 𝐴𝑥 = 𝑏
b = np.array([[5], another method, e.g., the
[6], Least Square Method
[9]]) The results becomes:
(LSM) [[-3.5 ]
[ 4.17857143]]
x = np.linalg.lstsq(A, b, rcond=None)[0]

print(x) 𝑥! = 3.5, 𝑥" = 4.2

 Additional Python Resources

https://www.halvorsen.blog/documents/programming/python/
Hans-Petter Halvorsen
University of South-Eastern Norway
www.usn.no

E-mail: hans.p.halvorsen@usn.no
Web: https://www.halvorsen.blog