Unit 5 - SVD

Prepared by: Varun Rao (Dean, Data Science & AI)

For: Data Science - 3rd years

Singular Value Decomposition is one of the important concepts in linear

algebra. To understand the meaning of singular value decomposition
(SVD), one must be aware of the related concepts such as matrix, types of
matrices, transformations of a matrix, etc. As this concept is connected to
various concepts of linear algebra, it’s become challenging to learn the
singular value decomposition of a matrix.

The Singular Value Decomposition (SVD) of a matrix is a factorization of

that matrix into three matrices. It has some interesting algebraic properties
and conveys important geometrical and theoretical insights about linear
transformations. It also has some important applications in data science.

Applications of SVD

The original paper Eigenfaces for Recognition came out in 1991. Before this,

most of the approaches for facial recognition dealt with identifying individual

features such as the eyes or the nose and developing a face model by the

position, size, and relationships among these features.

The Eigenface approach sought to extract the relevant information in a face

image, encode it as efficiently as possible, and compare one face encoding

with a database of models encoded similarly.

The encoding is obtained by expressing each face as a linear combination of

the selected eigenfaces in the new face space.

Let us break the approach down into five steps:

1. Collect a training set of faces as the training set

2. Find the most important features by finding the directions of maximum
variance – the eigenvectors or the eigenfaces
3. Choose top M eigenfaces corresponding to the highest eigenvalues.
These eigenfaces now define a new face space
4. Project all the data in this face space
5. For a new face, project it into the new face space, find the closest
face(s) in the space, and classify the face as a known or an unknown
face

Some more applications are as follows:

 ● SVD has some fascinating algebraic characteristics and conveys
relevant geometrical and theoretical insights regarding linear
transformations.
● SVD has some critical applications in data science too.
● Mathematical applications of the SVD involve calculating the matrix
approximation, rank of a matrix and so on.
● The SVD is also greatly useful in science and engineering.
● It has some applications of statistics, for example, least-squares fitting
of data and process control.

Matrix Factorization
Matrix decomposition (or) matrix factorization is an approximation of a
matrix into a product of matrices. They are used to implement efficient
matrix algorithms.

“The general idea is that you are able to multiply all of the resulting ‘factor’

matrices together to obtain the original matrix.”

Ex: 6 = 2 * 3 , where 2,3 are treated as factors of 6 and we can generate 6

again by product of those factors(i.e 2,3).

In similar fashion, A = B.C , A can be expressed as product of two lower

dimensional matrices B, C. Here, k=no. of dimensions (hyper parameter).

 a) Purpose and Ways to decompose a matrix

There are many different matrix decomposition techniques, each finds use

among a particular class of problems.

To be brief, these are two broad classes of decomposition:

a) Some only apply to ‘n x n’ matrices

b) Some apply to more general ‘m x n’ matrices

1)To solve linear equations, of which mostly the matrix decomposes into

2 parts.
 ● LU decomposition factorizes a matrix into a Lower triangle and an

Upper triangular matrix.

● QR decomposition decomposes a matrix A into a product A = QR

of an orthogonal matrix Q and an upper triangular matrix R.

● Cholskey decomposition etc.

● Non-negative matrix factorization (NMF) decomposes a matrix A

into two matrices that have non-negative elements. (A must have

non-negative elements too)

2) Eigenvalue-based decomposition, mostly applicable to square

matrices. where the matrix decomposes into 3 parts(final rotation, scaling,

initial rotation).

● PCA is a transform that uses eigen decomposition to obtain the

transform matrix.

● Singular Value Decomposition(SVD) factorizes any matrix with

any dimension as 3 parts USV’ .

b) What are the benefits of decomposing a matrix ?

 ● Matrix factorization which separates a matrix into two other

matrices that are typically much easier to solve than the original

matrix. This not only makes the problem easy to solve, but it

reduces the amount of time required by a computer to calculate

the answer.

● Matrix decomposition is mostly used to solve linear systems in an

easy way or quickly.

● Matrix factorization reduces a computer's storage space for

matrices, instead of storing the large non factorized matrix(A). We

can use less storage for its factors(i.e B, C), sometimes even

smaller when the rank of the matrix is small.

Collaborative filtering–based recommendation engines

To address some of the limitations of content-based filtering, collaborative
filtering uses similarities between users and items simultaneously to provide
recommendations. This allows for serendipitous recommendations; that is,
collaborative filtering models can recommend an item to user A based on
the interests of a similar user B. Furthermore, the embeddings can be
learned automatically, without relying on hand-engineering of features.

A Movie Recommendation Example

Consider a movie recommendation system in which the training data
consists of a feedback matrix in which:

● Each row represents a user.

● Each column represents an item (a movie).
The feedback about movies falls into one of two categories:

● Explicit— users specify how much they liked a particular movie by

providing a numerical rating.
● Implicit— if a user watches a movie, the system infers that the user is
interested.

To simplify, we will assume that the feedback matrix is binary; that is, a
value of 1 indicates interest in the movie.

When a user visits the homepage, the system should recommend movies
based on both:

● similarity to movies the user has liked in the past

● movies that similar users liked

To put it simply, collaborative filtering is a recommendation system that

creates a prediction based on a user’s previous behaviors.

Recommendation systems have made their way into our day-to-day online
surfing and have become unavoidable in any online user’s journey.

Why do we need recommender systems?

The main objective of the system is to provide the best user experience.
Therefore, companies strive to connect the users with the most relevant
things according to their past behavior and get them hooked to their
content.

The recommender system suggests which text should be read next, which
movie should be watched, and which product should be bought, creating a
stickiness factor to any product or service. Its unique algorithms are
designed to predict a users’ interest and suggest different products to the
users in many different ways and retain that interest till the end.

Needless to say that we see the implementation of this system in our daily
lives. Many online sellers implement recommender systems to generate
sales through machine learning (ML). Many retail companies generate a
high volume of sales by adopting and implementing this system on their
websites. The pioneering organizations using recommenders like Netflix
and Amazon have introduced their algorithms of recommendation systems
to hook their customers.

Before diving into the in-depth mechanics, it is necessary to know that this
system removes useless and redundant information. It intelligently filters
out all information before showing it to the front users.

To understand the recommender system better, it is a must to know that

there are three approaches to it being:

1. Content-based filtering
2. Collaborative filtering
3. Hybrid filtering model

Let’s take a closer look at all three of them to see which one could better fit
your product or service.

1. Content-based filtering
Many of the product’s features are required to implement content-based
filtering instead of user feedback or interaction. It is a machine learning
technique that is used to decide the outcomes based on product
similarities.

Content-based filtering algorithms are designed to recommend products

based on the accumulated knowledge of users. This technique is all about
comparing user interest with product features, so it is essential to provide a
significant feature of products in the system. It should be the first priority
before designing a system to select the favorite features of each buyer.

These two strategies can be applied in a possible combination. Firstly, a list

of features is provided to the user to select the most interesting features.

Secondly, the algorithms keep the record of all the products chosen by the
user in the past and make up the customer’s behavioral data. The buyer’s
profile rotates around the buyer’s choices, tastes, and preferences and
shapes the buyer’s rating. It includes how many times a single buyer clicks
on interested products or how many times liked those products in wishlists.

Content-based filtering consists of a resemblance between the items. The

proximity and similarity of the product are measured based on the similar
content of the item. When we talk about the content, it includes genre, the
item category, and so on.

Let’s take the example of recommender systems in movies. Suppose you

have four movies in which the user starts off liking only two movies at first.
Still, the 3rd movie is similar to the 1st movie in terms of the genre, so the
system will automatically suggest the 3rd movie. It is something that is
automatically generated by a content-based recommender system based
on the similarity of content.

Just imagine the power of content-based recommender systems, and the

possibilities are endless. For example, when we have a drama film that the
user has not seen or liked before, this genre will be excluded from their
profile altogether. Therefore, a user only gets their recommendation of the
genre that is already existing in their profile. The system would never
suggest any movie out of their genres to present the best user experience.
2. Collaborative Filtering
Collaborative filtering needs a set of items that are based on the user’s
historical choices. This system does not require a good amount of product
features to work. An embedding or feature vector describes each item and
User, and it sinks both the items and the users in a similar embedding
location. It creates enclosures for items and users on its own.

Other purchaser’s reactions are taken into consideration while suggesting a

specific product to the primary user. It keeps track of the behavior of all
users before recommending which item is mostly liked by users. It also
relates similar users by similarity in preference and behavior towards a
similar product when proposing a product to the primary customer.

Two sources are used to record the interaction of a product user. First,
through implicit feedback, User likes and dislikes are recorded and noticed
by their actions like clicks, listening to music tracks, searches, purchase
records, page views, etc.

Collaborative Filtering is the most famous application suggestion engine

and is based on calculated guesses; the people who liked the product will
enjoy the same product in the future. This type of algorithm is also known
as a product-based collaborative shift. In this Filtering, users are filtered
and associated with each User in place of items. In this system, only users’
behavior is considered. Only their content and profile information is not
enough. The User giving a positive rating to products will be associated
with other User’s behavior giving a similar rating.

The main idea behind this approach is suggesting new items based on the
closeness in the behavior of similar customers.

3. Hybrid Filtering
A hybrid approach is a mixture of collaborative and content-based filtering
methods while making suggestions; the film’s context also considers. The
user-to-item relation and the user-to-user relation also play a vital role at
the time of the recommendation. This framework gives film
recommendations as per the user’s knowledge, provides unique
recommendations, and solves a problem if the specific buyer ignores
relevant data. The user’s profile data is collected from the website, film’s
context also considers the user’s watching film and the data of the scores
of the movie.

The data consist of aggregating similar calculations. This method is called

the hybrid approach, in which both methods are used to produce the
results. When this system is compared with other approaches, this system
has higher suggestions accuracy. The main reason is the absence of
information about the filtering’s domain dependencies and the people’s
interest in a content-based system.

When these two approaches work together, you will get more knowledge,
leading to better results; it explores the new paths to significant underlying
content and collaborative filtering methods with buyer behavior data.

Restaurant Dish Recommendation Engine

Content-based recommendation and collaborative filtering

recommendation are the two most common systems. The aim of this

post is to build a simple content-based recommendation system. We

will see how we can scrape data, and build a recommender engine.

Python has great tools for web scraping such as “BeautifulSoup”,

“Selenium”, etc. We need “requests” library for fetching content from a
website and then you can parse and return the data if you use
“BeautifulSoup” library. In this project, we’d extract the data into a
data frame using the “Selenium” library. “Selenium” has to open a
browser for each request to get a website. For this reason, you need a
webdriver sending the requests.

The dataset contains different types of recipes such as desserts,

thanksgiving, Christmas, etc. In total, 2000+ recipe details are

available in our dataset. Each recipe consists of the below features

● Description

● Ingredients

● Preparation Steps

● Recipe Rate

● “Make It Again” Ratio

We also scraped more information about the recipes. The data on the

above contains the most reviewed recipes. So, each item has at least

one review from a user. The reviews have also been collected to

understand the general perspectives of the users about a recipe. In

order to understand the perspectives of the users, we applied

sentiment analysis to the reviews. We measured the polarities of the

people using ‘TextBlob”. The polarity numbers range between -1 and 1.

If the number is less than 0 we can say that the review is negative or it

is greater than 0 the review is positive.

Data Cleaning

Before building the engine, you need to clean the data. For that, we

should clear the texts from the characters and words that we do not

need. While we were scraping the data, some special characters came

with that such as ‘/n’ among the words. We started to clean those

characters and removed the stop words and then, applied classic
clearing techniques for natural language processing the following

steps:

● Removing punctuation

● Obtaining the words’ stems

● Removing numeric characters

● Converting letters to lowercase

Recommendation Engine

We are going to build a recommender engine based on the ingredients,

description, and preparation steps. The engine will make a

recommendation according to positive reviews to the users’.

In order to create a recommendation engine, we need a vector of the

matrix (in this case we use “TF-IDF Vectorizer”) to find the similarities

between the recipes. Afterward, we use the “linear_kernel” function

from “sklearn” to calculate cosine similarities. It measures the cosine

of the angles between the two vectors. The vectors that we created with
TF IDF based on the recipe information. As I said, the engine will

make a recommendation based on user reviews. The following steps

explain the working system.

1. Determining the positive sentiments of the user with

‘TextBlob’ and filtering the reviews

2. Choosing randomly out of the positive reviews and finding a

recipe that user’s like

3. Recommending top 10 recipes with the highest cosine score

based on the recipe that we choose

from sklearn.feature_extraction.text import

TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
from sklearn.metrics.pairwise import
cosine_similarity

tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf.fit_transform(df['Infos'])
cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)
def get_recommendations(name,
cosine_sim=cosine_sim):
# Get the index of the food that matches the title
idx = indices[name]

# Get the pairwsie similarity scores of all dishes

with that food
sim_scores = list(enumerate(cosine_sim[idx]))

# Sort the dishes based on the similarity scores

sim_scores = sorted(sim_scores, key=lambda x:
x[1], reverse=True)

# Get the scores of the 10 most similar dishes

sim_scores = sim_scores[1:11]

# Get the food indices

food_indices = [i[0] for i in sim_scores]

return df['Name'].iloc[food_indices]
This will be the following output for The Food Recommendation Engine.