Machine Learning

By:
Dr SOMARAJU SUVVARI
Assistant Professor,
Dept. of CSE, NIT Patna
 Numerical Vectors
(Source: Machine Learning Foundations - Supervised, Unsupervised, and Advanced Learning
by Taeho Jo, springer publication)
 Numerical Vectors
 Numerical vector is defined as a finite ordered set of numerical values.

 The number of elements in each numerical vector is called its dimension.

 Every raw data should be converted into numerical vectors in using machine
learning algorithms for real problems.

 x = [x1 x2 . . . xd ].

 The d values, x1, x2, . . . , xd , become the elements of the vector, x

 The number of elements, d, is called the dimension of the vector - x.

 Note that the elements, x1, x2, . . . , xd , are ordered.

 Numerical Vectors
Operations on vectors

The addition and the deletion of two numerical vectors are simple operations on them.

The scalar multiplication on a numerical vector is to multiply it by a constant value, and

the norm of a numerical vector generates a scalar value which represents it.

The inner product of two numerical vectors generates a scalar value which indicates the
similarity between them, and the cosine similarity between them is a normalized
similarity metric between zero and one.

The outer product of two vectors generates a matrix as the opposite operation to the
inner product.
 Numerical Vectors
 A numerical vector is expanded into a matrix which consists of more than one
vectors as rows or columns.

 The rows in the matrix are called row vectors, and the columns in it are called
column vectors.
 Numerical Vectors
Operations on numerical vectors
 Scalar value is a one dimensional vector.

 A numerical vector is a finite ordered set which contains more than one scalar
value.

 It is possible to plot vectors visually until three dimensional space.

 Numerical Vectors
Operations on numerical vectors
 Definition

 The scalar value is a single numerical value and is notated by x ∈ ℛ

 A scalar value is viewed as a special type of numerical vector: a vector

with only a single element.

 A scalar value is mentioned as a one dimensional vector.

 The norm of a vector is the unary operation on a numerical vector for

expressing it as a scalar value.
 Numerical Vectors
Operations on numerical vectors
 Definition

 The number of elements is finite in the numerical vector, it is called

dimension and notated by d.

 When d = 1, it becomes a scalar value, and when d = 2 or d = 3, each

numerical vector may be visualized in the 2 or 3 dimensional plane.

 When d > 3, it is impossible to visualize the numerical vector; it is viewed

as an ordered finite set of numerical values.

 If the dimension of the vector, x, is d, notated by x ∈ 𝓡d

 Numerical Vectors
Operations on numerical vectors
Set & Vector
 In the vector, its elements are ordered.

 In the set, its elements are unordered.

 In the vector, the number of elements is fixed, whereas in the set, the number of
elements is variable.

 In the numerical vector, its elements are always given as numerical vectors,
whereas in the set, its elements are always given as various types of data.

 Operations on vectors are performed in the one to one style, whereas those on
sets are performed in the style of all possible pairs.
 Numerical Vectors
Operations on numerical vectors
 Addition, deletion, and the scalar multiplication are the binary operations on
numerical vectors - generate a numerical vector as their output.

 Addition

 Let us define x1 = [x11 x12 . . . x1d ] and x2 = [x21 x22 . . . x2d ]

 x1 + x2 = [x11 + x21 x12 + x22 . . . x1d + x2d ]

 The sum of n vectors, x1, x2, . . . , xn, is defined as

 x1 + 0 = x1
 Numerical Vectors
Operations on numerical vectors
 Addition, deletion, and the scalar multiplication are the binary operations on
numerical vectors - generate a numerical vector as their output.

 Deletion

 x1 - x2 = [x11 - x21 x12 - x22 . . . x1d - x2d ]

 x1 + (-x2) = [x11 + (- x21) x12 + (- x22) . . . x1d + (-x2d )]

 Scalar multiplication

 c ・ x1 = [c.x11 c.x12 . . . c.x1d ]

Operations on numerical vectors
Numerical Vectors
 Norm -

 It is used for representing a numerical vector as a scalar value which indicates the vector size.

 The norm of the vector, x1, is expressed as

 The dimension of the vector becomes a norm in r = 0,

 When r = ∞, the maximum element in the vector becomes its norm.

 In studying the machine learning algorithms, r is usually set to one or two.

 Numerical Vectors
Operations on numerical vectors
 Inner Product -

 The inner product is the summation of product of one to one element of two
vectors.

 When two vectors are directed identically, the inner product is maximized, but
when they are directed perpendicular to each other, the inner product becomes
zero.

 If a vector, x1, is a normal vector, its norm is one.

 ||x1|| = 1
 Numerical Vectors
Operations on numerical vectors
 Inner Product -

 A vector may be converted into its normal form by dividing its element by its
norm.

 The multiplication of norm of a vector, x2, by the normal vector, x1 is same to

norm of vector, x2.

 ||x1|| ・ ||x2|| = ||x2||

 Inner product of two vectors x1 = [x11 x12 . . . x1d ] and x2 = [x21 x22 . . . x2d ] is
 Numerical Vectors
Operations on numerical vectors
 Inner Product -

 A scalar value is generated as the output in any dimensional vectors.

 The relation between the inner product and the product of norms of two vectors is
expressed as

|| x1|| ・ ||x2|| ≥ ||x1 . x2||

 Numerical Vectors
Operations on numerical vectors
 Inner Product -

 Orthogonality - when two vectors are in their perpendicular direction, the inner
product of them is given as zero value.

 Two vectors are notated by x1 = [x11 x12 . . . x1d ] and x2 = [x21 x22 . . . x2d ], and if
x1・x2 = 0 two vectors, x1 and x2, are orthogonal.

 When the norms of two vectors x1 and x2 are one as ||x1|| = ||x2|| = 1, and they are
orthogonal each other, this case is called orthonormal.

 When two vectors, x1 and x2, are with same direction, ||x1||・||x2|| = ||x1・x2||
 Numerical Vectors
Operations on numerical vectors
 Outer Product -

 Let us mention the operation which is opposite to the inner product, called outer
product.

 Two vectors are given as x1 = [x11 x12 . . . x1d ] and x2 = [x21 x22 . . . x2d ].

 A matrix is generated from applying the outer product to the two vectors.
 Numerical Vectors
Operations on numerical vectors
 Outer Product -

 In the inner product, a scalar value is generated as the output, where as a d×d
matrix is generated in the outer product.

 In implementing the machine learning algorithms, the inner product is used

more frequently than the outer product.
 Numerical Vectors
Linear Independence

 A vector may be expressed by a linear combination of more than one product of a

vector and a coefficient; the vectors which are involved in the linear combination
are called spans.

 Only when all coefficients are zero, the linear combination of vectors becomes
zero vector; this case is called linearly independent of spans.

 If the spans are linearly dependent on each other, a particular vector in the
spans is expressed by a linear combination of the others.
 Numerical Vectors
Linear Independence

 A particular d dimensional vector, x ∈ Rd ,is expressed as a linear combination of

vectors, v1, v2, . . . , vm, vi ∈ Rd .

 The set of vectors, v1, v2, . . . , vm, that is used for expressing the vector, x, is
called the spanning set of the vector, x.

 The spanning set is notated by S = {v1, v2, . . . , vm}, and the linear combination
of the vectors in the spanning set is expressed for the vector x into

 ci is a constant coefficient.
 Numerical Vectors
Linear Independence

 Example

 x = [x1 x2 x3] , and the set, S = {[1 0 0], [0 1 0], [0 0 1]}, is expressed for the
vector, x, in

 x = x1[1 0 0] + x2[0 1 0] + x3[0 0 1]

 The vectors in the spanning set are independent of each other, then
 Numerical Vectors
Linear Dependence

 Let us mention the linear dependency among the vectors in the spanning set, v1,
v2, . . . , vm.

 In the spanning set, a vector, vi , exists, satisfying in non-zero

coefficients.

 One among v1, v2, . . . , vm, vi is expressed by a linear combination of the others
in the spanning set, as shown


 Numerical Vectors
Linear Dependence

 For example, the spanning set, S = {[1 0 0], [0 1 0], [2 0 0]}, is a typical example
of the linear dependency, since v3 = 2・v1 + 0・v2.
 Numerical Vectors
Matrix Operations

 Addition

 Multiplication
 AB != BA

 AI = A

 AA-1 = I

 Inverse Matrix

 The identity matrix is characterized as diagonal, square, and symmetry

 Not all square matrix have its inverse matrix; if its inverse matrix is available, the matrix is
invertible.
 Numerical Vectors
Matrix Operations
Inverse Matrix (Gaussian-Jordan elimination process)

 Step – 1
 Numerical Vectors
Matrix Operations
Inverse Matrix (Gaussian-Jordan elimination process)

 Step-2

 The matrix, A, is assumed to be a n × n matrix, the n row vectors of the matrix,

A, a1, a2, . . . , an, and the n row vectors of the identity matrix, r1, r2, . . . , rn, and
the above augmented matrix is changed by updating each row of both matrices, A
and I, by
 Numerical Vectors
Matrix Operations

Inverse Matrix (Gaussian-Jordan elimination process)

 Step-2
 Numerical Vectors
Matrix Operations

Inverse Matrix (Gaussian-Jordan elimination process)

 where aij-1 is the element of the inverse matrix, A−1.

 The augmented matrix which is presented above is changed into one which is
presented

by updating each row of both, B and C by using the following eq.,

 Numerical Vectors
Matrix Operations

 The inverse matrix is used for solving a linear equation system

 It is represented as

 Ax = c

 x = A-1 c
 Numerical Vectors
Vectors & Matrices

 Determinant

 The determinant of a particular matrix is defined as a scalar value which

represents itself.

 The invertibility of a matrix is determined by its determinant value; if its

determinant is non-zero, it is invertible.

 Consider the below 2 X 2 matrix

 The determinant of the 2 X 2matrix is

 Numerical Vectors
Vectors & Matrices

 Determinant

 The inverse matrix of A is given as

 If the determinant of the matrix is zero, it is not invertible

 Numerical Vectors
Vectors & Matrices

Determinant

 Let us consider the determinant of n × n matrix as a general square matrix


We define the submatrix, Aij , where ith row and jth column are deleted

 The determinant of the matrix, A, is defined as The determinant of the n × n matrix is

computed by downsizing the matrix until 2 × 2, iteratively.

 It takes the quadratic complexity in computing the determinant of the matrix in

implementing it.
 Numerical Vectors
Vectors & Matrices

Determinant - Properties
 If, at least, a row or a column is a zero vector, the determinant of the matrix is zero.

 If the matrix, B, is obtained by interchanging any two columns or any two rows of the matrix, A,
the determinant of the matrix, B, becomes the negative determinant of the matrix, A;

 |B| = −|A|

 If, at least, two columns or two rows are identical to each other, the determinant of the matrix
becomes zero.

 If the matrix, B, is obtained by multiplying any row or any column of the matrix, A, by a scalar
value, k, the determinant of the matrix, B, is |B| = k|A|.
 Numerical Vectors
Vectors & Matrices

Eigen Value and Vector

 The Eigen vector is used for characterizing a matrix by multiplying itself with a scalar and is
called an Eigenvalue.

 The multiplication of a n × n matrix and a n × 1 Eigen vector is expressed as the multiplication of

an Eigen value and an Eigen vector.

 The Eigen value, λ, and the Eigen vector, v, are expressed as the matrix, A, mathematically

Av = λv

 The matrix, A, in the left side of v is assumed as a square matrix, and the above equation is used
for diagonalizing a matrix into a simplified one.

 Let us consider the Eigen vectors and value

 Numerical Vectors
Vectors & Matrices

Eigen Value and Vector

 The Eigen vector [v1 v2] and the Eigen value 𝜆 of the 2×2 matrix are expressed
as:

 It can be modified in the following manner:

 Numerical Vectors
Vectors & Matrices

Eigen Value and Vector

 Numerical Vectors
Vectors & Matrices

Eigen Value and Vector

 The determinant of the matrix, A − λI, is zero for finding the Eigen vectors and values expressed
as:
 The determinant of the matrix, A2×2 − λI2×2, is zero in the case of 2 ×2 matrix and is shown as:
 Numerical Vectors
Vectors & Matrices

Eigen Value and Vector Properties:

 A triangular matrix, A, is given as:

 A’s diagonal elements are given as its Eigen values.

 If the Eigen values are non zeros, the matrix is invertible.

 If an Eigen value of the matrix, A, is given as λ, the Eigen value of its inverse matrix is given as 1/λ.

 If the Eigen value of the matrix, A is a diagonal matrix, are given as its Eigen values.

 If the Eigen values are non-zeros, the matrix is invertible.

 If an Eigen value of the matrix, A, is given as, the Eigen value of the matrix, An, is λn.
THANK YOU