Curve Fitting

Curve Fitting Using Least-Square Principle
P. Sam Johnson
February 6, 2020
P. Sam Johnson (NIT Karnataka) Curve Fitting Using Least-Square Principle February 6, 2020 1 / 32
Overview
The study of approximation theory involves two general types of problems.

One problem arises when a function is given explicitly, but we wish to
find a simpler type of function, such as a polynomial, that can be used to
determine approximate values of the given function.
The Taylor polynomial of degree n about the number x0 is an

excellent approximation to an (n + 1)-times differentiable function f
in a small neighbourhood of x0 .
The second problem in approximation theory is concerned with fitting

functions to given data and find the “best” function in a certain class to
represent the data.
The Lagrange interpolating polynomials are approximating

polynomials and they fit certain data. Limitations of these techniques
are considerd, and other avenues of approach are discussed in the lecture.
Introduction
In many branches of applied mathematics and engineering sciences, we

come across experiments and problems which involve two variables.
For example, it is known that the speed v of a ship varies with the horse
power p of an engine according to the formula p = a + bv 3 . Here a and b
are the constants to be determined. For this purpose we take several sets
of readings of speeds and the corresponding horse powers.
The problem is to find the best values of v and p. Thus the general
problem is to find a suitable relation or law that may exist between the
variables x and y from a given set of observed values
(xi , yi ), i = 1, 2, . . . , n. Such relation connecting x and y is known as
empirical law.
Introduction
The process of finding the equation of the “curve of best fit” which may
be most suitable for predicting the unknown values is known as curve
fitting.
The following are standard methods for curve fitting.
1. Graphical method
2. Method of group averages
3. Method of moments
4. Method of least squares.
We discuss the method of least squares in the lecture.
Introduction
Usually a mathematical equation is fitted to experimental data by plotting

the data on a “graph sheet” and then passing a straight line through the
data points.
The method has the obvious drawback in that the straight line drawn may
not be unique. The method of least squares is probably the most
systematic procedure to fit a “unique curve” using given data points and
is widely used in practical computations. It can also be easily implemented
on a digital computer.
Fitting a Straight Line (Linear Form)
Let y = a0 + a1 x be the straight line to be fitted to the given data.
The problem of finding the equation of the best linear approximation

requires that values of a0 and a1 be found to minimize
m
X
S(a0 , a1 ) = |yi − (a0 + a1 xi )|.
i=1
This quantity is called the absolute deviation.

To minimize a function of two variables, we need to set its partial

derivatives to zero and simultaneously solve the resulting equations.
In the case of the absolute deviation, we need to find a0 and a1 with

∂S ∂S
=0 and = 0.
∂a0 ∂a1
The difficulty is that the absolute-value funciton is not differentiable at
zero, and we may not be able to find solutions to this pair of equations.
The least squares approach to this problem involves determining the best
approximating line when the error involved is the sum of the squares of the
differences between the y -values on the approximating line and the given
y -values.
Hence, the sum of the squares of the errors,

m
X
S= [yi − (a0 + a1 xi )]2 .
i=1
For S to be minimum, we have

m
∂S X
= −2 [yi − (a0 + a1 xi )]
∂a0
i=1
and
m
∂S X
= −2 xi [yi − (a0 + a1 xi )].
∂a1
i=1
The above equations are simplified to

m
X m
X
ma0 + a1 xi = yi
i=1 i=1
and
m
X m
X m
X
ma0 xi + a1 xi2 = xi yi .
i=1 i=1 i=1
Since the xi and yi are known quantities, the above two equations (called
the normal equations), can be solved for the two unknown a0 and a1 .
∂S ∂S
Differentiating ∂a 0
and ∂a 1
with respect to a0 to a1 respectively, we find
∂2S ∂2S
and
∂a02 ∂a12
and both will be positive at the points. Hence these values provide a
minimum of S.
∂2S ∂2S ∂2S ∂2S
Let Sa0 a0 = ∂a02
, S a1 a1 = ∂a12
, S a0 a1 = ∂a0 ∂a1
and Sa1 a0 = ∂a1 ∂a0
.
m
X
Sa0 a0 = −2 (−1) = 2m > 0
i=1
Xm m
X
Sa1 a1 = −2 xi (−xi ) = 2 xi2 > 0
i=1 i=1
Xm m
X
Sa1 a0 = −2 xi (−1) = 2 xi
i=1 i=1
Xm m
X m
X
Sa0 a0 Sa1 a1 − (Sa1 a0 )2 = 4m xi2 − 4 xi xi
i=1 i=1 i=1
for minimum we should get Sa0 a0 Sa1 a1 − (Sa1 a0 )2 > 0. This is guaranteed from the
Cauchy-Schwarz inequality
X m √ X m 1/2
xi < m xi2 .
i=1 i=1
Another approach of finding the equation of the best linear approximation

requires that values of a0 and a1 be found to minimize
n o
S(a0 , a1 ) = max |yi − (a0 + a1 xi )| .
1≤i≤m
This is commonly called a minimax problem and cannot be handled by

elementary techniques.
The minimax approach generally assigns too much weight to a bit of

data that is badly in error, whereas the absolute deviation method does
not give sufficient weight to a point that is considerably out of line with
the approximation.
The least squares approach puts substantially more weight on a point

that is out of line with the rest of the data but will not allow that point to
completely dominate the approximation.
Laws Reduicible to the Linear Law
We give below some of the laws in common use, indicating the way these
can be reduced to the linear form by suitable subsitutions.
Law Transformation Converted Law

y = mx n + c x n = X and y = Y Y = mX + c
y = ax n log10 x = X and log10 y = Y Y = nX + c, c = log10 a
xn y
y = ax n + b log x log x = X and log x = Y Y = aX + b
y = ae x x = X and log10 y = Y Y = mX + c,
m = log10 e, log10 a
y
xy = ax + by x = X and y = Y Y = bX + a
Exercise
1. Convert the following equations into linear form
x
(a) y = a+bx (c) xay = b (e) xy = ax + b
ax+b (d) y = b (f) y = ax + bx 2
(b) y = x x(x−a)
(g) y = ax + bxy .
Polynomial of the nth degree
Let the polynomial of the nth degree, y = a0 + a1 x + a2 x 2 + · · · + an x n be

fitted to the data points (xi , yi ), i = 1, 2, . . . , m. We then have
m
X
S= [yi − (a0 + a1 xi + a2 xi2 + · · · + an xin )]2 .
i=1
We ge the following normal equations
Xm m
X m
X m
X
2 n
ma0 + a1 xi + a2 xi + · · · + an xi = yi
i=1 i=1 i=1 i=1
m
X m
X m
X m
X
a0 xi + a1 xi2 + · · · + an xin+1 = xi yi
i=1 i=1 i=1 i=1
.. .. .. .. .. ..
. . . . . .
m
X m
X Xm m
X
a0 xin + a1 xin+1 + · · · + an xi2n = xin yi .
i=1 i=1 i=1 i=1
x y xy ··· x ny
x0 y0 x0 y0 ··· x0n y0
x1 y1 x1 y1 ··· x1n y1
.. .. .. .. ..
. . . . .
x y x y ··· xn y
Pmm Pmm Pmm m Pmm mn
i=1 xi i=1 yi i=1 xi yi ··· i=1 xi yi
After substituting the values of

m
X m
X m
X m
X
xi , yi , xi yi and xin yi
i=1 i=1 i=1 i=1
in the above normal equations, we get (n + 1) equations in (n + 1)

unknowns. Solving them will give the values of a0 , a1 , . . . , an . Thus the
polynomial of the nth degree, y = a0 + a1 x + a2 x 2 + · · · + an x n be fitted
to the data points (xi , yi ), i = 1, 2, . . . , m.
Plausible (Feasible) Values
Consider the m linear equations in n unknowns :
a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
.. .. .. .
. . . = ..
am1 x1 + am2 x2 + · · · + amn xn = bm .
We find the values of unknowns x1 , x2 , . . . , xn as nearly as possible.
Applying the principle

Pm of least squares, these values can be obtained by
2
minimizing S = i=1 (ai1 x1 + · · · + ain xn − bi ) using the conditions of
minima,
∂S ∂S . ∂S
= 0, = 0, .. , = 0.
∂x1 ∂x2 ∂xn
We get n equations. Solving these equations, we get most plausible
(feasible) values of x1 , x2 , . . . , xn . One can verify that at the values of
x1 , x2 , . . . , xn , determinant of every principal minor of the following matrix
is positive :  
Sa0 a0 Sa0 a1 . . . Sa0 an
Sa a Sa a . . . Sa a 
 1 0 1 1 1 n 
 .. .. .. ..  .
 . . . . 
San a0 San a1 . . . San an
Exercise
2. Find the most plausible values of x, y and z from the equations
x + 3y − 3z = −14
4x + y + 4z = 21
3x + 2y − 5z = 5
x − y + 2z = 3
by forming the normal equations.
Nonlinear Curve Fitting : Power Function
We consider a power function, y = ax c to fit the given data points
(xi , yi ), i = 1, 2, . . . , m.
Taking logarithms of both sides, we obtain the relation
log y = log a + c log x,
which is of the form Y = a0 + a1 X , where Y = log y , a0 = log a, a1 = c

and X = log x.
Hence the procedure outlined earlier can be followed to evaluate a0 and a1 .

Then a and c can be calculated from the formulae a0 = log a and c = a1 .
Nonlinear Curve Fitting : Exponential function
Let the curve

y = a 0 e a1 x
be fitted to the given data.
Then, as before, taking logarithms of both sides, we get
log y = log a0 + a1 x,
which can be written in the form
Z = A + Bx,
where Z = log y , A = log a0 and B = a1 .
The problem therefore reduces to finding a least-squares straight line

through the given data.
Fitting the data with given curve
Let the set of data points be
(xi , yi ), i = 1, 2, . . . , m,
and let the curve given by

y = f (x)
be fitted to this data. At x = xi , the experimental (or observed) value of
the ordinate is yi and the corresponding value on the fitting curve is f (xi ).
If ei is the error of approximation at x = xi , then we have
ei = yi − f (xi ).
Fitting the data with given curve
If we write
S = [y1 − f (x1 )]2 + [y2 − f (x2 )]2 + · · · + [ym − f (xm )]2

= e12 + e22 + · · · + em
2
then the method of least squares consists in minimizing S, i.e., the sum of
the squares of the errors.
Nodes (xi , yi ) are in red coloured points.
The curve y = f (x) fitted with the data

is shown in blue.
Weighted Least Squares Approximation
We have minimized the sum of squares of the errors. A more general

approach is to minimize the weighted sum of the squares of the errors
taken over all data points. If this sum is denoted by S, then we have
S = W1 [y1 − f (x1 )]2 + W2 [y2 − f (x2 )]2 + · · · + Wm [ym − f (xm )]2

= W1 e12 + W2 e 2 + · · · + Wm em
2
.
In the above equation, the Wi are prescribed positive numbers and are
called weights.
A weight is prescribed according to the relative accuracy of a data

point. If all the data points are accurate, we set Wi = 1 for all i. We
consider again the linear and nonlinear cases below.
Linear Weighted Least Squares Approximation
Let y = a0 + a1 x be the Straight line to be fitted to the given data points,

(x1 , y1 ), . . . , (xm , ym ). Then
m
X
S(a1 , a2 ) = Wi [yi − (a0 + a1 xi )]2 .
i=1
For maxima or minima, we have

∂S ∂S
= =0
∂a0 ∂a1
which gives
m
∂S X
= −2 Wi [yi − (a0 + a1 xi )] = 0
∂a0
i=1
and
m
∂S X
= −2 Wi [yi − (a0 + a1 xi )]xi = 0.
∂a1
i=1
Simplification yields the system of equations for a0 and a1 :
m
X m
X m
X
a0 Wi + a1 Wi x i = Wi y i
i=1 i=1 i=1
and
m
X m
X m
X
a0 Wi xi + a1 Wi xi2 = Wi xi yi
i=1 i=1 i=1
which are the normal equations in this case and are solved to obtain a0
and a1 .
Suppose that in data, a point (x0 , y0 ) is known to be more reliable

than the others. Then we prescribe a weight (say, 10) corresponding
to this point only and all other weights are taken as unity.
We consider with an increased weight, say 100, corresponding to
(x0 , y0 ), then the approximation becomes better when the
weight is increased.
Nonlinear Weighted Least Squares Approximation
We now consider the least squares approximation of a set of m data points
(xi , yi ), i = 1, 2, . . . , m,
by a polynomial of degree n < m. Let
y = a0 + a1 x + a2 x 2 + · · · + an x n
be fitted to the given data points. We then have

m
X
S(a0 , a1 , . . . , an ) = Wi [yi − (a0 + a1 xi + · · · + an xin )]2 .
i=1
If a minimum occurs at (a0 , a1 , . . . , an ), then we have

∂S ∂S ∂S ∂S
= = = ··· = = 0.
∂a0 ∂a1 ∂a2 ∂an
These conditions yield the normal equations
m
X m
X m
X m
X
a0 Wi + a1 Wi xi + · · · + an Wi xin = W i yi
i=1 i=1 i=1 i=1
m
X m
X m
X m
X
a0 Wi xi + a1 Wi xi2 + · · · + an Wi xin+1 = Wi xi yi
i=1 i=1 i=1 i=1
.. .. .. .. ..
. . . . .
m
X m
X X m m
X
a0 Wi xin + a1 Wi xin+1 + · · · + an Wi xi2n = Wi xin yi .
i=1 i=1 i=1 i=1
There are (n + 1) equations in (n + 1) unknowns a0 , a1 , . . . , an .
If the xi ’s are distinct with n < m, then the equations possses a ’unique’
solution.
Method of Least Squares for Continuous Functions
We considered the least squares approximations of discrete data. We shall

discuss the least squares approximation of a continuous function on [a, b].
The summations in the normal equations are now replaced by definite

integrals.
Let y (x) = a0 + a1 x + a2 x 2 + · · · + an x n be chosen to minimize

Z b
S(a0 , a1 , . . . , an ) = W (x)[y (x) − (a0 + a1 x + a2 x 2 + · · · + an x n )]2 dx.
a
The necessary conditions for minimum are given by

∂S ∂S ∂S
= = ··· = = 0.
∂a0 ∂a1 ∂an
Hence
Z b
−2 W (x)[y (x) − (a0 + a1 x + a2 x 2 + · · · + an x n )]dx = 0
a
Z b
−2 W (x)[y (x) − (a0 + a1 x + a2 x 2 + · · · + an x n )]xdx = 0
a
Z b
−2 W (x)[y (x) − (a0 + a1 x + a2 x 2 + · · · + an x n )]x 2 dx = 0
a
.. .. .. .. .. ..
. . . . . .
Z b
−2 W (x)[y (x) − (a0 + a1 x + a2 x 2 + · · · + an x n )]x n dx = 0
a
Rearrangement of terms gives the system
Z b Z b Z b Z b
a0 W (x)dx + a1 xW (x)dx + · · · + an x n W (x)dx = W (x)y (x)dx
a a a a
Z b Z b Z b Z b
a0 xW (x)dx + a1 x 2 W (x)dx + · · · + an x n+1 W (x)dx = xW (x)y (x)dx
a a a a
.. .. .. .. .. .. ..
. . . . . . .
Z b Z b Z b Z b
a0 x n W (x)dx + a1 x n+1 W (x)dx + · · · + an x 2n W (x)dx = x n W (x)y (x)dx.
a a a a
The system comprises (n + 1) normal equations in (n + 1) unknowns,

a0 , a1 , a2 , . . . , an and they always possess a ’unique’ solution.
Exercise
3. Construct a least squares quadratic approximation to the function
y (x) = sin x on [0, π/2] with respect to the weight function
W (x) = 1.
Exercises
4. If P is the pull required to lift a load W by means of a pulley block,
find a linear law of the form P = mW + C , connecting P and W ,
using the following data.
P (in kg.) 12 15 21 25
W (in kg.) 50 70 100 120
Compute P when W = 150 kg.
5. By the method of least squares, find the straight line that best fits
the following data.
x 1 2 3 4 5
y 14 27 40 55 68
6. Fit a straight line to the following data and estimate the value of y
corresponding to x = 6.
x 0 5 10 15 20 25
y 12 15 17 22 24 30
Exercises
7. Fit a second degree parabola by taking x as the independent variable.
x 1 1 2 3 4
y 1 5 10 22 38
8. Using the method of least squares, fit a curve of the form y = ab x to
the following data.
x 1 2 3 4
y 4 11 35 100
9. Fit a curve of the form y = ab x to the following data.
Year (x) 1951 1952 1953 1954 1955 1956 1957
Production
in tone (y ) 201 263 314 395 427 504 612
10. Fit a curve of the form y = ax b for the following data, where a and b
are constants.
x 61 26 7 10
y 350 400 500 600
Exercises
11. Using the principle of least squares, fit an equation of the form
y = ae bx (a > 0) to the data.
x 1 2 3 4
y 1.65 2.7 4.5 7.35
12. The pressure and volume of a gas are related by the equation
pc λ = k (λ and k are constants). Fit this equation for the following
data, using the principle of least squares.
p 0.5 1.0 1.5 2.0 2.5 3.0
v 1.62 1.00 0.75 0.62 0.52 0.46
13. Two quantities of x and y are measured and corresponding values are
given in the following table.
x 20 40 60 80 100 120
y 5.5 9.1 14.9 22.8 33.3 46
Find a second degree parabola to the data.
References
1. Richard L. Burden and J. Douglas Faires, Numerical Analysis -

Theory and Applications, Cengage Learning, Singapore.
2. Kendall E. Atkinson, An Introduction to Numerical Analysis, Wiley
India.
3. David Kincaid and Ward Cheney, Numerical Analysis -
Mathematics of Scientific Computing, American Mathematical
Society, Providence, Rhode Island.
4. S.S. Sastry, Introductory Methods of Numerical Analysis, Fourth
Edition, Prentice-Hall, India.

Curve Fitting

Uploaded by

Copyright:

Available Formats

Curve Fitting

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Curve Fitting

Uploaded by

Copyright:

Available Formats

Curve Fitting Using Least-Square Principle

The study of approximation theory involves two general types of problems.

The Taylor polynomial of degree n about the number x0 is an

The second problem in approximation theory is concerned with fitting

The Lagrange interpolating polynomials are approximating

In many branches of applied mathematics and engineering sciences, we

The following are standard methods for curve fitting.

We discuss the method of least squares in the lecture.

Usually a mathematical equation is fitted to experimental data by plotting

Let y = a0 + a1 x be the straight line to be fitted to the given data.

The problem of finding the equation of the best linear approximation

This quantity is called the absolute deviation.

To minimize a function of two variables, we need to set its partial

In the case of the absolute deviation, we need to find a0 and a1 with

Hence, the sum of the squares of the errors,

For S to be minimum, we have

The above equations are simplified to

Another approach of finding the equation of the best linear approximation

This is commonly called a minimax problem and cannot be handled by

The minimax approach generally assigns too much weight to a bit of

The least squares approach puts substantially more weight on a point

Law Transformation Converted Law

Let the polynomial of the nth degree, y = a0 + a1 x + a2 x 2 + · · · + an x n be

After substituting the values of

in the above normal equations, we get (n + 1) equations in (n + 1)

Consider the m linear equations in n unknowns :

a11 x1 + a12 x2 + · · · + a1n xn = b1

We find the values of unknowns x1 , x2 , . . . , xn as nearly as possible.

Applying the principle

by forming the normal equations.

We consider a power function, y = ax c to fit the given data points

Taking logarithms of both sides, we obtain the relation

log y = log a + c log x,

which is of the form Y = a0 + a1 X , where Y = log y , a0 = log a, a1 = c

Hence the procedure outlined earlier can be followed to evaluate a0 and a1 .

Let the curve

Then, as before, taking logarithms of both sides, we get

which can be written in the form

where Z = log y , A = log a0 and B = a1 .

The problem therefore reduces to finding a least-squares straight line

Let the set of data points be

and let the curve given by

If ei is the error of approximation at x = xi , then we have

S = [y1 − f (x1 )]2 + [y2 − f (x2 )]2 + · · · + [ym − f (xm )]2

Nodes (xi , yi ) are in red coloured points.

The curve y = f (x) fitted with the data

We have minimized the sum of squares of the errors. A more general

S = W1 [y1 − f (x1 )]2 + W2 [y2 − f (x2 )]2 + · · · + Wm [ym − f (xm )]2

A weight is prescribed according to the relative accuracy of a data

Let y = a0 + a1 x be the Straight line to be fitted to the given data points,

For maxima or minima, we have

Suppose that in data, a point (x0 , y0 ) is known to be more reliable

We now consider the least squares approximation of a set of m data points

by a polynomial of degree n < m. Let

be fitted to the given data points. We then have

If a minimum occurs at (a0 , a1 , . . . , an ), then we have

There are (n + 1) equations in (n + 1) unknowns a0 , a1 , . . . , an .

We considered the least squares approximations of discrete data. We shall

The summations in the normal equations are now replaced by definite

Let y (x) = a0 + a1 x + a2 x 2 + · · · + an x n be chosen to minimize