OPTIMIZATION METHOD

MINOR PROJECT
LEVENBERG-MARQUARDT METHOD
SEMESTER 6
3 SSCM
2019/2020
Topic : Efficiency of Levenberg-Marquardt in solving non-linear least
squares

NO. GROUP MEMBER MATRIC NO

1. NOR LAILA BINTI MUSTAKIM A17SC0123
2. ANIS SYAKIRAH BINTI MOHAMAD HANAFIAH A17SC0019

LECTURER: SIR ISMAIL BIN KAMIS

DATE SUBMITTED: 18 MAY 2020

CONTENT
Abstract .......................................................................................................................

Chapter 1
1.1 Introduction of Optimization and Nonlinear least square problems……………….
1.2 Objective ………………………………………………………….........................
1.3 Scope of study…………………………………………………………..................
1.4 Significances of study…………………………………………………………......

Chapter 2
2.1 Introduction of Levenberg-Marquardt....…………………………………………..
2.2 Method of optimization…………………………………………………………....
2.3 Theory of the method……………………………………………………………...
2.4 Algorithm of the method…………………………………………………………..
2.4 Example of the method…………………………………………………………….

Chapter 3
3.1 Application of the method…………………………………………………………
3.2 Solving the problem ……………………………………………………………...
3.3 Conclusion ………………………………………………………………………..

References .......................................................................................................................

Minute Meeting Report ................................................................................................

Abstract
 There are many method introduced by mathematician in solving case of non-linear
least problem. In this study, we will discuss about Levenberg-Marquardt method which
involve the minimization problems arise especially in least squares curve fitting. The main
purposes for this study is to analyze the efficiency of Levenberg-Marquardt method in
solving the non-linear least square problem. The objective of this study are to identify the
problem of non-linear least square in real life and apply the Levenberg-Marquardt method to
solve non-linear least square problem and to develop a coding of Levenberg-Marquardt
formula. In order to achieve the result, we will use a software which is (matlab or excel). At
the end of this study, we can conclude that Levenberg-Marquardt can be used to solve the
non-linear least square formula.
 Chapter 1

1.1 Introduction of Optimization and the Non-linear least square problem.

Nowadays, optimization had been widely used in more areas such as optimal control,
engineering, economics, finance and etc. Optimization is about finding the largest or smallest
values which can be attained by functions of several real variables subject to constraint [1].

In this writing, we will focus on the method of Levenberg-Marquardt in solving the

non-linear least square problem. Before we begin to the theory and algorithm of the method,
we will briefly explain the non-linear least square problem in real life. The optimization of
nonlinear Least Squares is a special case of unconstrained optimization. We have the same
objective as the linear case

m
1 ' 1
g ( x )= r ( x ) r ( x )= ∑ r i ¿
2 2 i=1

But the residuals

r i ( x )= y i−f ( t i , x ) , i=1 , … , m,

are nonlinear as “the model” f( ,x) is a nonlinear function with being the independent
variables and the vector of parameters to be estimated[2].

Non-linear least squares is the form of least analysis used to fit a set of m observations
with a model that is nonlinear in some forms of nonlinear regression. The basis of the method
is to approximate the model by a linear one and to refine the parameters by successive
iterations. In economic theory, the non-linear least squares method is applied in probit
regression, threshold regression, smooth regression and Box-Cox transformed regressors.
Here are some example of non-linear least square problem in real life:

i. Bell shaped fit of Dorsal concentration

 ii. UV- vis-spectrocopic titration on H20

1.2 Objectives of the study

The objectives of this study are;

1. To identify the problem of non-linear least squares in real life.

2. To apply the Levenberg-Marquardt on non-linear least square.

3. To develop a coding involving the problem solving by using Levenberg-Marquardt

algorithm.

1.3 Scope of the study

We will limit our study to unconstrained Non-linear least square problem. To solve the
problem we use Levenberg-Marquardt and the problem is unconstrained type of problem

1.4 Significance of the study

This study contributes knowledge to better understand and solve non-linear least square
problem by using the Levenberg-Marquardt. Besides, mathematician also can further
investigate the modification of the present method to solve the more difficult problem in the
optimization areas.
 Chapter 2
2.1 Introduction
The Levenberg-Marquardt algorithm was developed by Kenneth Levenberg and
Donald Marquardt [5]. Also, the Levenberg-Marquardt algorithm was developed in the early
1960s to solve non-linear squared problems. The least-squares problem arises in the context
of fitting a parameter function to a set of measured data points by minimizing the sum of
squared errors between the data point and function. If the load function is not linear in the
parameter, the least-squares is nonlinear. The non-linear squares method periodically reduces
the sum of the square error between the function and the data points as measured by the
update sequence to the parameter value [6].

2.2 Method of optimization

The Levenberg-Marquardt algorithm combines two minimization methods the
gradient descent method and the Gauss-Newton method. In the gradient descent method, the
sum of squared errors is reduced by updating the parameter to the steepest descent. In the
Gauss-Newton method, the sum of squared errors is reduced by assuming the local least-
squares function quadratic and finding the quadratic of the minimum. The Levenberg-
Marquardt method acts more like a gradient-descent method when its parameter is far from
its optimum value and acts more like the Gauss-Newton method when the parameter
approaches its optimal value [6].

2.3 Theory of the method

To solve non-linear least-squares problems, LMA is commonly used as a standard

algorithm. A combination of the gradient descent and the Gauss-Newton method appears in
this algorithm. In many cases, LMA can guarantee the solution through its adaptive
behaviour. In the case of back-propagation, BP is expressed as gradient descent, the algorithm
slows down and does not provide an optimal solution. On the contrary, if BP is expressed as
Gauss-Newton, the algorithm has the highest probability of providing an optimal solution. In
this algorithm, the calculation of Hessian calculations, H is shown in the Equation and its
gradient calculation, g is expressed in

H = JT J

g = JT e
where Jacobian matrix represents by J and e indicate a vector of a network error. And then the
LMA behaves as Newton is expressed by the following equation:

xk+1 = xk – ¿

where xk+1 is a new weight that calculated as gradient function and current weight x k using
Newton algorithm [7].
The Levenberg-Marquardt algorithm combines the steepest descent which is far from
the optimal values and the Gauss-Newton methods that close to the optimal value, by taking
advantage of the high speed of the Gauss-Newton algorithm and the high stability of the
steepest descent method. The Levenberg-Marquardt algorithm converges slower than the
Gauss-Newton algorithm and is faster than the steepest descent method. Thus, Lavernberg
Marquardt solves the problem of minimizing nonlinear functions numerically, very quickly
and with a stable convergence. Later, due to the Lavernberg Marquardt weakness,
computational complexity grew exponentially as it dealt with large Jacobians and the need
to reverse the large matrix [5].

2.4 Algorithm of Levenberg-Marquardt

The Levenberg-Marquardt (LM) algorithm is an iterative technique that locates the

minimum of a multivariate function that is expressed as the sum of squares of non-linear real-
valued functions [3]. It have become a standard technique for non-linear least-square
problems [4]. Below is the complete pseudocode of Levenberg-Marquardt algorithm:-
Input: A vector function f : R m → Rn with ≥ m , a measurement vector x ∈ Rn and an initial
parameters estimate p0 ∈ Rm
m
Output: A vector p+¿∈ R ¿ minimizing ¿|x−f ( p )|∨¿2 ¿.

Algorithm:

k :=0 ; v :=2; p := p0

A ;=J T J ; ϵ p ≔ x−f ( p ) ; g ≔ J T ϵ p ;

stop≔ (||g||∞ ≤ ε 1) ; μ ≔ τ∗max i=1 , …, m ( A ii ) ;

while ( not stop )∧(k < k max )

k ≔k + 1;

repeat

S olve( A+ μ I ) δ p=g

if (||δ p||≤ ε 2|| p||)

stop≔true ;

else

pnew ;= p+ δ p

2 2

ρ≔
(||ϵ || −||x−f ( p )|| ) ;
p new
T
δ p ( μ δ p+ g )

pnew ;=δ p ;

A ;=J T J ; ϵ p ≔ x−f ( p ) ; g ≔ J T ϵ p ;
2
(
stop≔ (||g||∞ ≤ ε 1) ∨ ||ϵ p|| ≤ ε 3 ; )
μ :=μ∗max ⁡¿ ; v ≔2 ;

else

μ ;=μ∗v ; v ≔2∗v ;

endif

endif

until ( ρ> 0 )∨(stop )

endwhile
p+¿≔ p ;¿
2.5 Example of the method

Chapter 3

3.1 Application of the method

In order to minimize the deviation of the seafloor topography, which is a multi-beam

random error, caused by the error of sound velocity, the Levenberg-Marquardt (LM)
algorithm is proposed to correct the velocity.
Nowadays, multi-beam systems that integrate navigation technology and satellite
positioning, computer science and digital sensors are widely used in areas such as marine
engineering, underwater archaeology and marine national defence construction. The accuracy
of multi-beam oceanographic topography measurement is highly dependent on the accuracy
of the sound velocity profile (SVP) measurements. As the velocity of sound changes in space
and time as the seawater flows, the different sound velocity profile structures can also cause
changes in the acoustic wave propagation path in the sea, leading to oceanic distortion, and
the original sea level will appear "smiling face" "and" frowning face ".
Three methods of correcting voice velocity are used to solve the aforementioned
deviations. Firstly, based on the replacement of the voice velocity profile, which is the
principle of minimum distance or minimum time. However, the effects of correcting this
method are limited and difficult to meet the actual needs. The second is the customization of
some parameters during the data processing system. The downside of this method is that
manual operation is slower and lacks detailed theory. The third is to optimize the structure of
simulating velocity profiles based on the sound velocity formula, which unfortunately
requires more samples and parameters, and more complex calculations.

So, to meet the requirements of a submarine topography with multi-beam voice,

automatic data processing without manual operation, and high accuracy with low complexity,
the Levenberg-Marquardt (LM) algorithm is introduced for voice velocity correction in
handling multi-beam echo-sounding data [8].

3.2 Solving the problem

Correction Model Based on Levenberg-Marquardt Algorithm

LM is one of a kind non-linear square optimization algorithm. In this case, the LM

algorithm aims to find a set of the sound of velocity parameters that are optimized to
straighten the curved seabed field and to ensure optimum displacement and actual
displacement. The formation of a sound beam model can obtain horizontal, longitudinal, and
theoretical transitions after a sound wave is transmitted through all layers of water, which is
then used as a reference correction. By using the error in the velocity profile, the distortion of
the "smiling face" and "frown face" of the model of observation can be established. Consider
the one-way transmission beam obtained in the entire process of sound as if there were no
errors, in other words, it moves with the same T as in the theoretical model.

Using the LM algorithm to correct the sound velocity and seafloor distortion errors,
assume that the vertical displacement of a particular beam is Z j, and the horizontal
displacement is X j. By stratification, we know the height of the n-1 layer, the sound velocity
of each layer, the Snell ray parameters and the true value of the one-way sound wave
propagation. Without considering the surface velocity error, we obtain the observation
equations with the following indirect adjustments

X j=❑ni=1 Δ t ij . C2ij p j +(T j −❑n−1

−1 2
i=1 Δt i )C nj p j

Z j=(τ j−❑n−1 n−1

i=1 Δ t ij ). C nj cos θ nj +❑i =1 Δ Z ij

Where i is the number of layers, 1=1,2,...,n. j is the number of sound beams, j=1,2,.....,m. Δ t ij
is

ΔZ ij ΔZ ij
Δt ij = =
Cij cos θij C ij ¿ ¿

The number of emitted beams is often greater than the number of sound velocity parameters.
Therefore, we can apply the principles of the LM algorithm to perform linear inversions.
Taking X ijo and Zijo as the approximation of the true value of vertical displacement and the
actual value of the horizontal displacement while the sound beam crosses layer i , d X i and dZ i
as correction of the approximation and the true value, therefore, the indirect linear model of
the adjustment function is

❑❑ X j= X oj +d X j=F j (Ĉ1 , Ĉ2 , ...., Ĉ i)

n
o o o
¿ F j (C ,C ,.... , C )+ ∑ ❑ Aij ĉ i
1 2 i
i=1

❑❑ Z j=Z oj +d Z j=H j (Ĉ 1 ,Ĉ 2 , .... ,Ĉ i )

n
¿ H j (C o1 , Co2 , .... ,C oi )+ ∑ ❑ Bij ĉ i
i=1
where Ĉ is the true value of sound velocity, C o is the observed value of sound velocity, Aij
and Bijare the coefficients of sound velocity correction for the LM linearization model,
namely, the derivative of the vertical displacement and horizontal displacement on the sound
velocity. Solving simultaneous equations X jand Z J would get m×n Jacobi matrix of
A and B in which Aij and Bij is given by

∂ Fj Δ Z I (C oi 2−C 2n) sin3 θ1 j

Aij = ❑c =
o
∂C i i ¿¿

C on si n2 θ 1 j
❑

Bij =
∂H j
❑ =
∂C i c
o
i
o o 2

√
Δ Z i C C si n θ1 j 1−
1 n

¿¿¿
Co1 2
❑

where the subscripts of X and Z represent the number of the sound beam.
Solving simultaneous equations of the horizontal displacement and vertical
displacement of the beam, the sound velocity correction of partial derivative matrix A and B
could get the total matrix D=¿, then solve the equations to get the sound velocity correction
as in Aij

ĉ =¿

where Matrix P is the weight matrix of the observed value. The setting of weight should be
considered that the reliability of the edge beam sounding data in the multi-beam system is
lower than the central beam Bij. According to the precision of the actual measurement for the
various sound beams, the lower the accuracy is, the smaller the weight of the beam is. The
matrix L is written by L=

The actual research indicates that in matrix L because some characteristic values of the
coefficient of the square D T PD are small, and the condition number of equations is large, the
inverse square matrix is singular, therefore, the solution tends to diverge. To solve this
problem, we use the least-squares damping method ĉ, which adds the term damping after the
αI phalanx. The basic idea of this method is to suppress the growth of correction to make
linear approximation an approximate efficiency. At the same time, we improve the state of
the equation, so that L turn to

ĉ=¿

where I is the unit matrix, α is the damping factor, the size of which depends on the actual
situation, that is, if the coefficient of the square DT PD has a relatively large singularity, then
a larger damping coefficient is required. Contrariwise, choosing a smaller coefficient, and the
correction value of the underwater observation value matrix V is

V =Bĉ−L

Add distortion correction values to the seabed topographic data, based on central beam
measurements to assess the accuracy of their corrections. If the accuracy does not meet the
requirements for multi-beam sound accuracy, add a correction of the sound velocity to the
original sound profile data as the initial parameter to calculate, and repeated calculations will
be performed to meet the requirements of IHO (International Hydrographic Organization) [8].

3.3 Conclusion
It analyzes the major causes of underwater area distortion due to velocity errors in the
multi-beam system for flat area measurement, combined with indirect adjustments and LM
algorithms, developing sound velocity profiles and underwater sea level distortion correction
technology. Research shows that in shallow seawater, the new sound velocity profile
calculated by the LM algorithm is closer to the actual profile than the error sound profile.
This technique effectively removes the divergence of the seafloor topography, minimizes
depth error, and satisfies the demand for high accuracy in shallow sea field measurements.
Moreover, the calculation step is very concise and convenient [8].
The Levenberg - Marquardt algorithm solves the problems of both the gradient
method and the Gauss-Newton method with the combination of both algorithms. It is
considered one of the most efficient algorithms. Nevertheless, the Levenberg - Marquardt
algorithm has its drawbacks. One problem is that the inversion of the Hessian matrix needs to
be calculated each time to update the weight and there may be some updates in each iteration.
For small sizes, the computation is efficient, but for large networks, such as image
recognition problems, this inverse calculation can be a disaster and the speed achieved by the
second-order approach may be lost. In that case, the Levenberg - Marquardt algorithm may
be slower than the steepest descent algorithm. Another problem is that the Jacobian matrix
must be stored for calculation, and its size is P × M × N, where P is the number of patterns,
M is the number of outputs, and N is the number of weights. For large-size training patterns,
the cost of memory for Jacobian matrix storage may be too large to be practical. Also, the
Levenberg - Marquardt algorithm only applies to multilayer perceptron networks. Although
there are still some unresolved problems for Levenberg - Marquardt, for small and medium-
sized networks, the Levenberg - Marquardt algorithm is very efficient [9].
 REFERENCE

1. Kiwiel, K. C. Methods of Descent for Nondifferentiable Optimization. Volume 1133 of

lecture notes in mathematics. Pennsylvania State University. 1985.

2. Manfred G. , Dietmar M. & Enrico S. Numerical method and Optimization in Finance.

16 Aug 2019 (Business and Economic). Retrieved date: 16/7/2020.

3. D.W. Marquardt. An algorithm for the Least Squares Estimation of Non-linear

Parameters. SIAM Journal of Applied Mathematics, 11920:431-441, Jun 1963.

4. Manolis I. A. Lourakis. A Brief Description of the Levenberg-Marquardt Algorithm

Implemened by Levmar. Institute of Computer Science, Greece. February 11, 2005.

5. Levenberg-Marquardt Algorithm. (n.d.). Retrieved from

https://www.sciencedirect.com/topics/engineering/levenberg-marquardt-algorithm

6. Gavin, H.P. (2013). The Levenberg-Marquardt method for nonlinear least squares curve-
fitting problems c ©.

7. Du, Y.-C., & Stephanus, A. (2018). Levenberg-Marquardt Neural Network Algorithm for
Degree of Arteriovenous Fistula Stenosis Classification Using a Dual Optical
Photoplethysmography Sensor. Sensors, 18(7), 2322. doi:10.3390/s18072322

8. Ma, K., Xu, W., & Xu, J. (2017). The application of levenberg-marquardt algorithm in
multi-beam sounding data. 2017 IEEE 2nd Information Technology, Networking, Electronic
and Automation Control Conference (ITNEC). doi: 10.1109/itnec.2017.8285088

9. Yu, H., & Wilamowski, B. M. (2010). Levenberg–Marquardt Training, Auburn University.

 MINUTES OF MEETING

Date: 27 April 2020

Time: 8.00 am - 9.00 am
Venue: Whatsapp
Group Member :

No. Group Members Matric card no. Signature

1. Nor Laila Binti Mustakim A17SC0123 Laila

2. Anis Syakirah binti Mohamad Hanafiah A17SC0019 Syakirah

Description :
- During the meeting, we made a few research about the method of optimization that
was not in our syllabus based on lecturer requirement.
- Then we collected our opinion and discussed the method that we are going to do for
our minor project.
- After that, we decided and chose the method that both of us agree for the minor
project.
- Lastly, we decided to continue our discussion for the project on the next meeting
which will be 4 May 2020.
MINUTES OF MEETING

Date: 4 May 2020

Time: 8.00 am - 9.00 am
Venue: Whatsapp
Group Member :

No. Group Members Matric card no. Signature

1. Nor Laila Binti Mustakim A17SC0123 Laila

2. Anis Syakirah binti Mohamad Hanafiah A17SC0019 Syakirah

Description :
- After doing some research and searching for the information about the method for our
minor project we decided to change our method for our project due to some
misinformation during our last meeting.
- Then after a little bit more details and cautious in searching the method that we can do
for our project, we decided another method that we have not learn or see yet which is
Levenberg-Marquardt method.
- We also discussed and divided our work by part so that it easier to get our project
done smoothly.

MINUTES OF MEETING
Date: 11 May 2020
Time: 8.00 am - 9.00 am
Venue: Whatsapp
Group Member :

No. Group Members Matric card no. Signature

1. Nor Laila Binti Mustakim A17SC0123 Laila

2. Anis Syakirah binti Mohamad Hanafiah A17SC0019 Syakirah

Description :
- During the meeting, we discussed the content of our project with more details.
- Also, we exchange opinion on our parts that we discussed in the last meeting.
- Then, decided to compile our works on google document so that easier to check and
discuss each other parts.

MINUTES OF MEETING

Date: 17 May 2020

Time: 3.00 pm
Venue: Whatsapp
Group Member :

No. Group Members Matric card no. Signature

1. Nor Laila Binti Mustakim A17SC0123 Laila

2. Anis Syakirah binti Mohamad Hanafiah A17SC0019 Syakirah

Description :
- We compiled our works and rechecked our content in the project.
- Lastly, we sent our minor project on 18 May 2020.