ECM6 Computational Methods :

Slide 1 of 5

Lecture 11c
Nonlinear Regression
Brian G. Higgins
Department of Chemical Engineering & Materials Science
University of California, Davis
April 2014, Hanoi, Vietnam

ECM6Lecture11cVietnam_2014.nb

Overview of Nonlinear Regression

There are times when we need to determine parameters in a model that appear nonlinearly.
Recall in our previous discussions on linear regression we were always able to reduce the problem to

Y = a0 + a1 X1 + a2 X2 + + aN XN
Here the unknown parameters ai appear linearly in the regression model. But this is not always the case
For example, suppose we have vapor pressure versus temperature data that we want to use to determine the coefficients in an Antoine equation:

P HTL = Exp a -

b
T+c

Note that we cannot transform this equation into a linear model. Nevertheless, we can still define a least
squares solution to determine the parameters. That is
M

E2 HPL = HP HTk L - Pk L2
k=1

and then look for the parameters such that E2 HPL is minimized. Again the local minimum with respect to
the parameters a, b, c is given by

P
a

= 0,

P
b

= 0,

P
c

=0

These equations are not linear, however. Nonetheless, we can still solve them using a Newton iteration
method. Let us consider the following example:

ECM6Lecture11cVietnam_2014.nb

A Nonlinear Regression Model

Consider the following example
We are given temperature (K) versus vapor pressure (kPa) data for n-Hexane:
88280, 8.65589<, 8290, 13.9557<, 8300, 21.6206<, 8310, 32.5923<,
8320, 47.9046<, 8330, 68.2326<, 8340, 94.8424<, 8350, 129.398<,
8360, 172.859<, 8370, 226.84<, 8380, 292.981<, 8390, 372.821<, 8400, 468.262<<

We would like to fit this data to the Antoine model given by (14). As before we define the data and the
necessary functions to extract out the temperature and vapor pressure values from the data list. We
also define our Antoine model, and the expression of the square of the errors E2

Mathematica Code
In[1]:=

data = 88280, 8.615<, 8290, 13.866<, 8300, 21.696<, 8310, 32.588<, 8320, 47.845<,
8330, 68.14697<, 8340, 94.879<, 8350, 129.325<, 8360, 172.806<,
8370, 226.774<, 8380, 292.951<, 8390, 372.788<, 8400, 468.248<<;

Functions to extract the data

In[6]:=

T@i_D := data@@i, 1DD; P@i_D := data@@i, 2DD; M = Length@dataD;

Mathematical Model: Antoine Equation

In[7]:=

Antoine@T_D := ExpBa -

b
T+c

Sum of Squares function

M
In[8]:=

E2 @g_, M_D := Hg@T@iDD - P@iDL2

i=1

Here is what the square of the errors E2 looks like using the Antoine Model:
In[9]:=

E2 @Antoine, MD
2

Out[9]=

J- 8.615 + a- 280+c N + J- 13.866 + a- 290+c N + J- 21.696 + a- 300+c N +

2

J- 32.588 + a- 310+c N + J- 47.845 + a- 320+c N + J- 68.147 + a- 330+c N + J- 94.879 + a- 340+c N +

b

J- 129.325 + a- 350+c N + J- 172.806 + a- 360+c N + J- 226.774 + a- 370+c N +

J- 292.951 + a- 380+c N + J- 372.788 + a- 390+c N + J- 468.248 + a- 400+c N

Then we take the derivative of E2 with respect to the parameters a,b,c:

ECM6Lecture11cVietnam_2014.nb

In[11]:=

eqns = 8D@E2 @Antoine, MD, aD 0, D@E2 @Antoine, MD, bD 0, D@E2 @Antoine, MD, cD 0<
b

Out[11]=

:2 a- 280+c J- 8.615 + a- 280+c N + 2 a- 290+c J- 13.866 + a- 290+c N + 2 a- 300+c J- 21.696 + a- 300+c N +

b

2 a- 310+c J- 32.588 + a- 310+c N + 2 a- 320+c J- 47.845 + a- 320+c N + 2 a- 330+c J- 68.147 + a- 330+c N +

b

2 a- 340+c J- 94.879 + a- 340+c N + 2 a- 350+c J- 129.325 + a- 350+c N + 2 a- 360+c J- 172.806 + a- 360+c N +

b

2 a- 370+c J- 226.774 + a- 370+c N + 2 a- 380+c J- 292.951 + a- 380+c N +

2 a- 390+c J- 372.788 + a- 390+c N + 2 a- 400+c J- 468.248 + a- 400+c N 0,
b

2 a- 280+c J- 8.615 + a- 280+c N

-

280 + c
b

H290 + cL2

H300 + cL2

H320 + cL2
b

2 b a- 340+c J- 94.879 + a- 340+c N

+

H330 + cL2

2 b a- 330+c J- 68.147 + a- 330+c N

H360 + cL2
b

2 b a- 370+c J- 226.774 + a- 370+c N

2 b a- 380+c J- 292.951 + a- 380+c N

+

H370 + cL2

2 b a- 360+c J- 172.806 + a- 360+c N

+

H350 + cL2

H340 + cL2

2 b a- 350+c J- 129.325 + a- 350+c N

H380 + cL2
b

2 b a- 390+c J- 372.788 + a- 390+c N

H390 + cL2

2 b a- 300+c J- 21.696 + a- 300+c N

2 b a- 320+c J- 47.845 + a- 320+c N

+

H310 + cL2

2 b a- 290+c J- 13.866 + a- 290+c N

2 b a- 310+c J- 32.588 + a- 310+c N

0,

400 + c
b

2 b a- 280+c J- 8.615 + a- 280+c N

2 a- 400+c J- 468.248 + a- 400+c N

-

390 + c

380 + c
b

2 a- 390+c J- 372.788 + a- 390+c N

2 a- 380+c J- 292.951 + a- 380+c N

-

370 + c

360 + c
b

2 a- 370+c J- 226.774 + a- 370+c N

2 a- 360+c J- 172.806 + a- 360+c N

-

350 + c

340 + c

H280 + cL2

2 a- 340+c J- 94.879 + a- 340+c N

2 a- 350+c J- 129.325 + a- 350+c N

320 + c

300 + c

2 a- 330+c J- 68.147 + a- 330+c N

330 + c

2 a- 320+c J- 47.845 + a- 320+c N

-

310 + c
b

2 a- 300+c J- 21.696 + a- 300+c N

-

290 + c

2 a- 310+c J- 32.588 + a- 310+c N

2 a- 290+c J- 13.866 + a- 290+c N

2 b a- 400+c J- 468.248 + a- 400+c N

+

H400 + cL2

0>

This is a set of nonlinear transcendental equations. We can solve these equations using a Newton's
method (i.e. use FindRoot). But we need to supply initial guesses for the parameters

ECM6Lecture11cVietnam_2014.nb

This is a set of nonlinear transcendental equations. We can solve these equations using a Newton's
method (i.e. use FindRoot). But we need to supply initial guesses for the parameters
In[15]:=
Out[15]=

sol = FindRoot@eqns, 88a, 16<, 8b, 3200<, 8c, - 37<<, MaxIterations InfinityD
8a 14.0599, b 2827.25, c - 42.6153<

In this case we have increased the number of iterations allowed from the default value of 100 to Infinity.
Note that the residual is quite small (< 0.01) but not vanishingly small.
In[19]:=
Out[19]=

E2 @Antoine, MD . sol
0.0192802

Here is a plot of the data and the model with the fitted parameters:
In[16]:=

plt1 = Plot@Antoine@TD .sol, 8T, 280, 400<, PlotStyle 8Thick, Blue<,

Frame True, FrameLabel 8Style@"T", 16D, Style@"PHTL", 16D<D;
plt2 = ListPlot@data, PlotStyle 8PointSize@LargeD, Red<D;
Show@plt1, plt2, PlotRange AllD

400

Out[18]=

PHTL

300

200

100

0
280

300

320

340

360

380

400

Using Mathematicas FindFit Function

There is also a nonLinear fitting function called FindFit. The syntax for the function is
FindFit[data, model,parameters, variables,options]

Here is how we use the NonLinearFit with the previous data

? FindFit
FindFit@data, expr, pars, varsD finds numerical values of the parameters pars that make expr give a best fit to
data as a function of vars. The data can have the form 88x1 , y1 , , f1 <, 8x2 , y2 , , f2 <, <, where
the number of coordinates x, y, is equal to the number of variables in the list vars. The data
can also be of the form 8 f1 , f2 , <, with a single coordinate assumed to take values 1, 2, .
FindFit@data, 8expr, cons<, pars, varsD finds a best fit subject to the parameter constraints cons.

ECM6Lecture11cVietnam_2014.nb

In[20]:=

Out[20]=

result = FindFitBdata, ExpBa -

b
HT + cL

F, 8a, b, c<, T, MaxIterations 1000F

8a 14.0599, b 2827.25, c - 42.6153<

Note that these parameters are the same that our code produced, but the number of iterations was less
In[21]:=

plt1 = Plot@Antoine@TD .result, 8T, 280, 400<, PlotStyle 8Thick, Blue<,

Frame True, FrameLabel 8Style@"T", 16D, Style@"PHTL", 16D<D;
plt2 = ListPlot@data, PlotStyle 8PointSize@LargeD, Red<D;
Show@plt1, plt2, PlotRange AllD

400

300

Out[23]=

PHTL

200

100

0
280

300

320

340

360

380

400

ECM6Lecture11cVietnam_2014.nb

Visualizing the Minimization Process (Optional)

Here is a way to see how the iteration converges by extrating out the paramter values during the
minimization process. We use the functions Reap and Sow in conjunction with StepMonitor. Check the
Help section to learn how to use these functions. These calculations make use of more advanced
Mathematica functions.
In[24]:=

8fit, steps< = ReapBFindFitBdata, ExpBa -

b
HT + cL

F, 8a, b, c<, 8T<,

MaxIterations 10 000, WorkingPrecision 16, StepMonitor Sow@8a, b, c<DFF;

FindFit::precw :
The precision of the data and model function HMachinePrecisionL is less than the specified WorkingPrecision H16L.

The values of the parameters {a,b,c} during the iteration are stored in the variable steps; the final value
of the fit parameters is given by the variable fit. The data in steps is of the form
88a1 , b1 , c1 <, 8a2 , b2 , c2 <, , 8an , bn , cn <<
We can plot this data in 3D by using ListPlot3D
In[25]:=

plt1 = ListPlot3D@stepsD

Out[25]=

The following shows the location of the optimum value for the parameters in the space of all values
obtained during the iteration.

ECM6Lecture11cVietnam_2014.nb

In[26]:=

Show@8plt1, Graphics3D@8Red, PointSize@LargeD, Point@8a, b, c< . fitD<D<D

Out[26]=

We can also use the previous definition for the least squares function and evaluate it at the iteration
points during the computation
In[27]:=

res = E2 @Antoine, MD
2

Out[27]=

J- 8.615 + a- 280+c N + J- 13.866 + a- 290+c N + J- 21.696 + a- 300+c N +

2

J- 32.588 + a- 310+c N + J- 47.845 + a- 320+c N + J- 68.147 + a- 330+c N + J- 94.879 + a- 340+c N +

b

J- 129.325 + a- 350+c N + J- 172.806 + a- 360+c N + J- 226.774 + a- 370+c N +

J- 292.951 + a- 380+c N + J- 372.788 + a- 390+c N + J- 468.248 + a- 400+c N

To do so we creat a set of rules from the data stored in the variable steps:
In[28]:=

paramRules =
steps . 8a1_Real, b1_Real, c1_Real< 8Rule@a, a1D, Rule@b, b1D, Rule@c, c1D<;

Then we plot the least squares error as a functioin of the iteration number
In[39]:=

ListPlot@res . paramRules, Joined True, Frame True, PlotStyle Thick,

FrameLabel 8Style@"number of iterations n", 16D, Style@"E2 ", 16D<D
500 000

400 000

E2

Out[39]=

300 000

200 000

100 000

0
0

20

40

60

80

100

number of iterations n
After about 90 iterations the value of the least squares error settles down to a small number. The total
number of iterations is

ECM6Lecture11cVietnam_2014.nb

In[35]:=
Out[35]=

num = Length@Flatten@res . paramRulesDD

116

Here is a blow-up of the region near the minimum value for the least squared error:
In[38]:=

ListPlot@Take@Flatten@res . paramRulesD, - 20D,

Joined True, Frame True, PlotStyle -> Thick,
FrameLabel 8Style@"number of iterations n", 16D, Style@"E2 ", 16D<,
FrameTicks 88True, True<,
8880, num - 20<, 85, num - 5<, 810, num - 10<, 815, num - 15<, 820, num<<, True<<D
40

E2

30

20

Out[38]=

10

0
96

111

106

101

number of iterations n

116

10

ECM6Lecture11cVietnam_2014.nb

References
These notes and the examples were adapted from the follow texts:
M. J. Maron, Numerical Analysis. A Practical Approach, 2nd Edition, Macmillan Publishing Company,
1987
A. J. Pettoprezzo, Introductory Numerical Analysis, Dover Publications, 1984