Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Implementing a Randomized SVD Algorithm and its

Performance Analysis
Injamamul Haque Ahmed

Abstract:- Dimension reducing techniques are becoming patterns, it can be really expensive and we might have to
more and more dominant in data science and model wait for a significant while.
predictions because it is much more efficient and
comfortable working on a small set of data than very In Modern world, we have now with the bigger and
large data. More often than not the reduced lower bigger sets of data is the higher and higher dimensional
dimensional representation seems to contain the same measurements or we can say the measuring dimensions is
properties as that of the higher dimensional space. increasing rapidly at a fast pace. But the key concept of
Additionally, big sets of data prove to be a problem in Singular Value Decomposition is that even in very large
terms of computational environment on both memory data sets there are actually a few properties or features that
and processing power and hence the need for we care about, say for our understanding and building
dimensionality reduction is key. models. So despite the increase in measuring dimensions we
can still have a low intrinsic rank available to us.
This paper discusses a dimension reducing
technique for low rank matrices called Randomized Randomized Singular Value Decomposition (RSVD)
Singular Value Decomposition (RSVD). It is a is an emerging technique in Randomized Linear Algebra
probabilistic method combined with the theory of that essentially randomly samples the column space of our
structured matrices to aid matrix factorization of any matrix, say 𝑋. So the basic idea is we randomly sample the
order. Probabilistic methods were considered to be column space of our parent matrix 𝑋 and with high
chaotic in nature at the start but the numerical results probability we are going to find a subspace that will be
are found to be quite reliable and stable. First we try and spanned by the dominating columns of 𝑈 matrix from the
show the mathematics involved in randomized singular Singular Value Decomposition.
value decompositions using concepts of Linear Algebra
and then try and formulate an algorithm based on our Note: The key motivation or the assumption while
mathematics. A random (𝟓𝟎𝟎𝟎, 𝟏𝟎𝟎𝟎) matrix is chosen working with Randomized Singular Value Decomposition is
to be factorized into a low rank 10, which is our target that there exists a low rank available that we want to
rank. uncover. We mostly call that rank as the target rank.

Generally, RSVD can be implanted using any II. ALGEBRA INVOLVED

programming language but here we use python. In
python a SVD can be implemented using a single line of The RSVD can be implemented in two main steps
code whereas the RSVD is another few lines of code. To using various Numerical Linear Algebra Techniques but all
match the theoretical performance and numerical of the algorithms generally have the same basic algebraic
performance we use RStudio to compute error construction as we discuss below.
percentage and runtime checks for both SVD and RSVD
to give us an idea of why the later proves to be a better, We want to factorize Matrix 𝑋(𝑚, 𝑛) such that 𝑋 =
fast and robust technique. The codes we developed below 𝑈𝑟 Σ𝑟 𝑉𝑟𝑇 where r is the target rank we want and 𝑈 represents
not only work for matrices but also for any grey scaled matrix of left singular vectors, Σ represents rectangular
images. diagonal matrix of singular values and 𝑉 represents a matrix
of right singular vectors.
There are various versions of the RSVD algorithm,
but we stick to the method of randomized range finder STEP 1:
method and in later parts also put in test the power 1. We take a random projection matrix 𝑃𝜖ℝ𝑛×𝑟 , where 𝑟
iteration method. is the target rank.
2. Then 𝑍 = 𝑋𝑃 𝑖. 𝑒 we shrink the column space down to 𝑟
I. INTRODUCTION and
𝑍 = 𝑄𝑅 𝑖. 𝑒 we do 𝑄𝑅 factorization in 𝑍 in place of 𝑋
We know and have already computed Singular Value itself.
Decompositions (SVD) for large data sets and although it is 3. 𝑄 gives an orthonormal basis for 𝑍 but also for 𝑋.
a fantastic tool for crunching that data down into a lower 4. Next we now project our Original data matrix 𝑋 onto
dimensional representation for extracting key features and subspace 𝑄.

IJISRT21OCT507 www.ijisrt.com 804

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
STEP 2: The Algorithm
5. Now that we have orthogonal subspace for column space
of 𝑍 and 𝑋, so we project our matrix 𝑋 into the lower STEP1:
dimensional subspace 𝑄.
6. 𝑌 = 𝑄𝑇 𝑋, this is much more lower dimensional. Given an m × n matrix A and an integer 𝑙, this scheme
7. Next we compute the SVD of 𝑌 𝑖. 𝑒 𝑌 = 𝑈𝑌 Σ𝑉 𝑇 . Based computes an m× 𝑙 orthonormal matrix Q whose range
on properties of random matrix theory and structured approximates the range of A.
matrices, the Σ, 𝑉 of 𝑌 should be same as that of Σ, 𝑉 of
𝑋 discussed before, since U is the orthonormal basis that
spans the columns of 𝑋. 1. Draw an n × 𝑙 Gaussian random matrix Ω.
8. 𝑈𝑋 = 𝑄𝑈𝑌 . 2. Form the m × 𝑙 matrix Y = AΩ.
3. Construct an m× 𝑙 matrix Q whose columns form an
Generally this method of randomized singular value orthonormal basis for the range of Y, e.g., using the
decomposition outperforms the classical singular value QR factorization Y = QR.
decomposition and these claims can be supported by STEP2:
numerical experiments and followed by their detailed error
analysis. The specific benefits of randomized techniques 4. Form B = Q*A.
depend on the computational environment. Consider the
model problem of finding the k dominant components of the 5. Compute an SVD of the small matrix: B = 𝑈𝑌 ΣV*.
singular value decomposition of an m × n matrix. (i) For a 6. Set U = Q𝑼𝒀 .
dense input matrix, randomized algorithms require
O(mnlog(k)) floating-point operations (flops) in contrast to
O(mnk) for classical algorithms. (ii) For a sparse input The computation of Y in step 2 is vulnerable to round-
matrix, the flop count matches classical Krylov subspace off errors. When high accuracy is required, we sometimes
methods, but the randomized approach is more robust and incorporate an orthonormalization step between each
can easily be reorganized to exploit multiprocessor application of A and A* as we will see when we apply the
architectures. (iii) For a matrix that is too large to fit in fast power iterations.
memory, the randomized techniques require only a constant
number of passes over the data, as opposed to O(k) passes The next section is all about implementing this
for classical algorithms. algorithm into working. We use python and jupyter
notebook to compile our results. These ideas can be easily
Randomized Algorithm implemented on any programming platform such as matlab,
In recent times there has been numerous developments R, python, C++, java etc. These codes are all original work
in randomized algorithms for matrix factorization. There are and done from the scratch.
various ways in which STEP 1 discussed above can be
computed but in this paper we stick to the method of III. IMPLEMENTATION
randomized range finder and later use power scheme to see
how the it improves the runtime of the algorithms discussed The piece of code below is same as the algorithm
in Finding Structures with Randomness - Probabilistic written in the box above. It follows the same algebraic
Algorithm for Constructing Approximate Matrix pattern, though there are two factors in the code that needs a
Decomposition: N.Halko, P.G.Martinsson, J.A.Tropp. little explanation.

Problem Description 1. p(Oversampling): Often we see it is very useful to

In the fast part Suppose we have matrix 𝐴 = sample a few more columns than the prescribed target
(5000,1000) and there exist a low intrinsic rank of 10, then rank as it has been found to provide better approximate
we compute rank 𝑟( 𝑖. 𝑒 10 ) truncated single value results when used.
decomposition of 𝐴 using a fast randomized algorithm and 2. q(Power iterations):Often in dealing with matrices the
implement using some programming language. singular value doesn’t seem to decay fast enough and in
that case we multiply the matrix with some kind of a
Secondly we compare how the algorithm we have used factor that makes the singular value drop off faster. The
compare to the standard singular value decomposition in code in the third cell shows the implementation the
terms of runtime and accuracy. We also try and give an idea power scheme and the graphs below show how with
of how fast our algorithm is to normal singular value increase in the value of 𝑞 from 1 to 5, the singular values
decomposition. drop of faster. For high accuracy the factor 𝐴𝐴∗ is
multiplied on the right hand side of the equation of 𝑌 in
the second substep of step 1 in the algorithm above.
Though the use of power iterations are quite useful and
provide fast results, it is very costly and hence its
implementation since we have to move through the data
sets multiple times.

IJISRT21OCT507 www.ijisrt.com 805

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

The deterministic singular value decomposition in python is a single line of code and the randomized singular value
decomposition is a few lines of codes. The above written code is very robust and work easily for matrices of arbitrary order and
can even be implemented to shapes and pictures with easy matrix reconstruction codes. In fact randomized algorithms are finding
high usage in graphics designing and image compression.

The next piece of code below is to illustrate the working and significance of power iterations as 𝑞 is run in the range 1 to 5.
The graphs and the plots below show the comparison between deterministic singular value decomposition and then randomized
ones with power iterations.

IJISRT21OCT507 www.ijisrt.com 806

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

IV. GRAPHICAL RESULTS AND INTERPRETATION

The above run codes generate graphs of power iterations:

The results show that as and when go on increasing the Often we observe our singular values doesn’t decay of
value of 𝑞 = 1,2,3,4,5, the singular values decays of faster that fast but for example using the factor (𝐴𝐴∗ )𝑞 in substep
and faster and the slope of the graphs goes down and down. 2 of step 1 in our algorithm makes the singular values decay
The randomized singular value decomposition relies on the much faster. This factor 𝑞 in theory is what we call a power
fact that singular values decay vey very rapidly so that we iteration.
can approximate the dominant ones and not have them be
contaminated by lower energy ones.

IJISRT21OCT507 www.ijisrt.com 807

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Computational Time of Randomized Singular Value
Decomposition 2. package(“microbenchmark”): This package helps us
The specific benefits of the randomized techniques produce accurate timings of the said algorithm.
depend upon the computational environment we are working
with. Analysis of computational time gives us an idea of A few other packages are also used which can be easily
how the algorithm performs under various conditions and is installed using install.package().
an important topic of study in performance analysis. This
section compares the computational time or the runtime of The accuracy of the approximation can be controlled
the deterministic SVD and the randomized algorithm that we via the two tuning parameters 𝑝 and 𝑞 . The former
have discussed above for the same parent matrix. parameter is used to oversample the basis, and is set by
default to 𝑝 = 10. This setting guarantees a good basis with
We use specific tools from RStudio to make our work high probability in general. The parameter 𝑞 can be used to
easier. The reason we used R is because it provides better compute additional power iterations (subspace iterations).
and comfortable statistical computing and graphics as By default this parameter is set to 𝑞 = 2, which yields a good
compared to other programming languages. The same performance in our numerical experiments, i.e., the default
analysis though could be done via other platforms as well. values show an optimal trade-off between speed and
accuracy in standard situations.
Before we move on to the numerical calculations, we
discuss a few important R packages we used to compute the Below here is R code for compiling the runtime for our
results. algorithm for the same parent matrix describes above in the
problem description. The graph below also provides us an
1. The rsvd() function: This package helps us compute low idea of how fast in runtime is the randomized algorithm in
rank singular value decomposition as in our algorithm. contrast to the deterministic singular value decomposition.
The interface of the function is

𝑟𝑠𝑣𝑑(𝐴, 𝑘, 𝑛𝑢 = 𝑁𝑈𝐿𝐿, 𝑛𝑣 = 𝑁𝑈𝐿𝐿, 𝑝 = 10, 𝑞

= 2, 𝑠𝑑𝑖𝑠𝑡 = "𝑛𝑜𝑟𝑚𝑎𝑙)

IJISRT21OCT507 www.ijisrt.com 808

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Note: To plot this figure one must install the package experiment was carried out in a machine with Intel(R)
ggplot2 which can be easily found and implemented in Core(TM) i5-7300HQ CPU @ 2.50GHz and RAM of
RStudio. 8.00GB.

Computational Accuracy
In the following we evaluate the performance of the
randomized SVD routine of ours with the classical SVD
routine based on the error percentage. The relative
theoretical reconstruction error is computed as

∥ 𝐴 − 𝐴𝐾 ∥𝐹
∥ 𝐴 ∥𝐹

Where 𝐴𝐾 = 𝑈𝐾 Σ𝐾 𝑉𝐾𝑇 , is the rank K- matrix. To fully

exploit the power of randomized algorithm we use the
enhanced R distribution. This allows the user to control the
trade-off between computational time and accuracy,
depending on the application. Note that we have set the
precision parameter of the svd() routines to tol = 1e-5.

Below is the RStudio code to compile the error for

both the randomized technique and the deterministic SVD.
The graph shows in truth how efficient and fast the
The matrix can easily be generated as with symbols having
randomized technique is to the normal classical method.
their usual meanings,
Theoretically, a randomized technique is 4-7 times quicker
than the normal deterministic method and our numerical
R> A <- matrix(rnorm(m * r), m, r) %*% matrix(rnorm(r
experiments are in accordance with it. One must also note
* n), r, n)
that the computational times and results might vary
accordingly based on the system and environment used. This

IJISRT21OCT507 www.ijisrt.com 809

Volume 6, Issue 10, October – 2021 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

The error in the runtime of our randomized algorithm

is found to be 4.567596𝑒 − 13, and for the deterministic
standard approach the error is 8.591737𝑒 − 13. We notice
that the error percentage is higher in classical SVD than
RSVD. We must also note that in the rsvd() package the
oversampling factor and power iteration is by default set at
𝑝 = 10, 𝑞 = 2 as discussed above. The results may vary
once we change these parameters.

V. CONCLUSIONS

This paper discusses the idea behind formulation of a

randomized algorithm for random matrices having a low
rank in it. The randomized algorithm constructed in this
paper is found to be robust and is faster in computational or
runtime in contrast to the deterministic classical SVD. The
proposed algorithm is also found to be highly accurate and
the error approximations are found to be very small and in
accordance with the theoretical studies. Overall we can say
that though the deterministic SVD makes a very good tool
for low rank matrix factorization, the randomized approach
beats this classical approach because it is much faster,
efficient and almost produces the same accurate results with
a low error term when deployed.

REFERENCES

[1]. N.Halko, P.G. Martinsson, J.A. Tropp, “Finding

structures with randomness: Probabilistic Algorithms
for Constructing Approximate Matrix Decomposition”,
SIAM Vol-53,No.2,pp-217-288.
[2]. N.B. Erichson, S. Voronin, S.L. Brunton, J.N. Kutz,
“Randomized Matrix Decomposition Using R”,
Journal of Statistical Software, Vol-89, Issue-11.

IJISRT21OCT507 www.ijisrt.com 810