Welcome to

Numerical Methods for Civil Engineers

ภาคการศึกษาที่ 3 ปการศึกษา 2548

Lecture 1 -
MATLAB
Introduction
Instructor: Mongkol JIRAVACHARADET

School of Civil Engineering

Institute of Engineering

Suranaree University of Technology

Textbook
Applied Numerical Methods
With MATLAB for Engineers
and Scientists

STEVEN C. CHAPRA

McGraw-Hill International Edition

 References
Numerical Methods For Engineers with Personal Computer Applications,
(Third Edition) by Chapra, S.C. and R.P. Canale, McGraw-Hill, 1998

Numerical Methods with MATLAB : Implementations and Applications,

Gerald W. Recktenwald, Prentice-Hall, 2000

An Introduction to Numerical Methods : A MATLAB Approach,

Abdelwahab Kharab and Ronald B. Guenther, Chapman & Hall/CRC, 2002

Numerical Methods using MATLAB,

John H. Mathews and Kurtis D. Fink, Prentice-Hall, 2004

The Matlab 7 Handbook , Mathwork Inc.

KEEP THESE BOOKS! They are excellent career references

(at least for a while)

Topics Covered
• Introduction to Matlab • Interpolation

• Approximations and Errors • Numerical Integration

• Roots of Equations • Ordinary Differential Equations

• Linear Systems • Optimization

• Curve Fitting
 Conduct of Course
Homework/Projects/Quizzes 30 %

Midterm Exam 30 %

Final Exam 40 %

Grading Policy
Final Score Grade
100 - 90 A
89 - 85 B+
84 - 80 B
79 - 75 C+
74 - 70 C
69 - 65 D+
64 - 60 D
59 - 0 F
 WARNINGS !!!
1) Participation expected, check by quizzes

2) Study in groups but submit work on your own

3) No Copying of Matlab code

4) Submit Homework at the beginning of class

5) Late homework with penalty 30%

6) No make up quizzes or exams

MATLAB
The Language of Technical Computing

M ATLAB เปนโปรแกรมที่รวมการคํานวณทาง
คณิตศาสตร การแสดงผล และภาษาที่มีประสิทธิภาพ
เพื่อสรางสภาวะแวดลอมที่ยืดหยุนสําหรับการคํานวณ
ทางเทคนิค ดวยสถาปตยกรรมแบบเปด ทําใหเปนการงาย
ที่จะใช MATLAB ในการสํารวจขอมูล สรางอัลกอริธึม หรือ
สรางเครื่องมือคํานวณตามที่ผูใชตองการ

The latest version Matlab 7.3 R2006b

www.mathworks.com
 MATLAB = MATrix LABoratory
- Math and computation

- Algorithm development

- Modeling, Simulation, and Prototyping

- Data analysis, exploration, and visualization

- Scientific and Engineering Graphics

- Many toolboxes for solving problems:

Control System Toolbox Statistics Toolbox

Signal Processing Toolbox Optimization Toolbox

System Identification Toolbox Partial Diff. Equation Toolbox

Neural Network Toolbox Symbolic Math Toolbox

MATLAB Educational Sites

http://www.mathworks.com/
http://www.eece.maine.edu/mm/matweb.html

http://www.math.utah.edu/lab/ms/matlab/matlab.html
http://www.math.mtu.edu/~msgocken/intro/intro.html
 MATLAB 1

- Getting started

- Basic Arithmetic

- Built-in Functions

- Built-in Variables

- Vector & Matrix

The MATLAB System

The MATLAB system consists of five main parts:

Development Environment: set of tools and facilities that help you use MATLAB
functions and files. Many of these tools are graphical user interfaces. It includes the
MATLAB desktop and Command Window, a command history, an editor and
dedugger, and browsers for viewing help, the workspace, files, and the search path.

The MALAB Mathematical Function Library: a vast collection of computational

algorithms.

The MATLAB Language: This is a high-level matrix/array language with control

flow statements, functions, data structures, input/output, and objected-oriented
programming features.

Graphics: MATLAB has extensive facilities 2-D and 3-D data visualization,
animation, and presentation graphics.

The MATLAB Application Program Interface (API): allows you to write C and
Fortran programs that interact with MATLAB.
Getting Started

MATLAB Desktop

Getting Started

Command Window

Command prompt >> DEMO

Getting Started

Editor/Debugger

Basic Arithmetic
Calculator functions work as you'd expect:

>>(1+4)*3

ans =

15
+ and - are addition, / is division, * is multiplication, ^ is an exponent.

>> 5*5*5
>> 5^(-2.5)
>> 3*(23+14.7-4/6)/3.5

Last-line editing Up Arrow

 >> 2 + 6 - 4 Upper & Lower Case
ans = >> a=2;
4 >> A=3;
The value in ans can be recalled: >> 2*a

>> ans/2 >> 2*A

ans = 2 Several commands in the same line:

>>x=2;y=6+x,x=y+7
Assign value to a variable:

>> a = 5 … For too long command:

>> Num_Apples = 10;
>> b = 6
>> Num_Orange = 25;
>> c = b/a
>> Num_Pears = 12;
>> Num_Fruit=Num_Apples+Num_Orange...
+ Num_Pears

Built-in Functions

>> sin(pi/4) >> pi sin(x), cos(x), tan(x),

ans = ans = sqrt(x), log(x), log10(x),
0.7071 3.1416 asin(x), acos(x), atan(x)

The output of each command can be suppress by using semicolon ;

>> x = 5;
>> y = sqrt(59);
>> z = log(y) + x^0.25
z =
3.5341
 The commas allow more than one command
on a line:

>> a = 5; b = sin(a), c = sinh(a)

b =
-0.9589
c =
74.2099

MATLAB Variables is created whenever it appears

on the left-hand of “ = “

>> t = 5;
>> t = t + 2
t =
7

Any variable appearing on the right-hand side of

“ = “ must already be defined. Format
>> x = 2*z
>> pi
??? Undefined function or variable ‘z’
>> format long
>> who >> pi
>> whos >> format short
>> clear >> format bank

Use long variable names is better to remember >> format short e

and understandable for others >> format long e
>> format compact
>> radius = 5.2;
>> format loose
>> area = pi*radius^2;
>> format
Built-in Variables
Use by MATLAB, Should not be assigned to other values

Variable Meaning

ans value of an expression when not assigned to variable

eps floating-point precision
i, j unit imaginary numbers, i = j = −1
pi π = 3.14159265 . . .
realmax largest positive floating-point number
realmin smallest positive floating-point number
Inf ∞, a number larger than realmax, result of 1/0
NaN not a number (0/0)

>> x = 0; >> help log

>> 5/x On-line Help:
>> x/x >> lookfor cosine

Matrices and Magic Squares

In MATLAB, a matrix is a rectangular array of numbers.

scalars = 1-by-1 matrices

vectors = one row or column matrices

it is usually best to think of everything

as a matrix.

The matrices operations in MATLAB are

designed to be as natural as possible.

MATLAB allows you to work with entire

matrices quickly and easily.

Renaissance engraving Melencolia I by the German artist and amateur

mathematician Albrecht Dürer.
Entering Matrices

You can enter matrices into MATLAB in several different ways:

• Enter an explicit list of elements.
• Load matrices from external data files.
• Generate matrices using built-in functions.
• Create matrices with your own functions in M-files.

Start by entering matrix as a list of its elements.

You only have to follow a few basic conventions:
• Separate the elements of a row with blanks or commas.
• Use a semicolon, ; , to indicate the end of each row.
• Surround the entire list of elements with square brackets, [ ] .

To enter matrix, simply type in the Command Window

>> A = [16 3 2 13; 5 10 11 8; 9 6 7 12; 4 15 14 1]

Row and Column Vectors

Row vector >> A = [ 2 3 5 7 11]

Column vector >> A = [ 2; 3; 5; 7; 11]
Transposition >> At = A’

Arrays Operations

>> A = [ 3 5 7 9 11 ] >> A / B
>> A(3) >> A ./ B
>> length(A) >> A .^ 2
>> clear(A) >> odd = 1:2:11
>> B = [ 2 4 6 8 10 ] >> even = 2:2:12
>> A + B >> natural = 1:6
>> A - B >> angle = 0:pi/10:pi;
>> A * B >> sin(angle)
>> A .* B
Generate matrices using built-in functions

>> A = zeros(4) >> A = zeros(3,4)

>> A = ones(4)
>> A = eye(4)
>> A = magic(4)

Elementary Matrix Operations

>> S = A + B >> [m , n] = size(A)

>> D = A - B >> det(A)
>> A*B >> inv(A)
>> C = [ 10 11; 12 13; 14 15]; >> v = [1 2 3];
>> A*C >> A = diag(v)
>> A^2 >> B = diag([1 2 1 2])
>> L = log10(A) >> w = diag(B)

Subscripts

The element in row i and column j of A is denoted by A(i , j)

A = [16 3 2 13; 5 10 11 8; 9 6 7 12; 4 15 14 1]

MATLAB displays the matrix you just entered.
A =
16 3 2 13
5 10 11 8
9 6 7 12
4 15 14 1

For example, A(4,2) is the number in the fourth row and second column.
>> A(4,2)
ans =
15
Submatrix

>> A(1,:)

>> A(:,2)

>> A(3:4,1:2)

Juxtaposition

>> B = [9 8 2 5 ; 4 5 6 7 ; 2 1 3 4]

>> [A B]

>> size(ans)

>> [A ; B]

The Colon Operator

The colon, :, is one of the most important MATLAB operators.

>> 1:10
To obtain nonunit spacing, specify an increment.
For example,
and
>> 100:-7:50 >> 0:pi/4:pi
Linspace

linspace function creates row vectors with equally spaced elements.

>> u = linspace(0.0,0.25,5)
>> v = linspace(0,9,4)’
>> x = linspace(0,pi/6,6*pi);

>> s = sin(x); >> t = tan(x);

>> c = cos(x); >> [x’ s’ c’ t’]

Workspace Browser

The MATLAB workspace consists of the set of variables (named arrays)

built up during a MATLAB session and stored in memory.
You add variables to the workspace by using functions, running M-files, and
loading saved workspaces.
To view the workspace and information about each variable,
use the Workspace browser, or use the functions who and whos.

Example 2.1 Transportation route analysis

The following table gives data for the distance travel along five truck routes and the
corresponding time required to traveled each route. Use the data to compute the
average speed required to drive each route. Find the route that has the highest
average speed.

1 2 3 4 5
Distance (miles) 560 440 490 530 370
Time (hrs) 10.3 8.2 9.1 10.1 7.5

Solution:
>>d = [560, 440, 490, 530, 370]
>>t = [10.3, 8.2, 9.1, 10.1, 7.5]
>>speed = d./t
speed =
54.3689 53.6585 53.8462 52.4752 49.3333
>>[highest_speed, route] = max(speed)
highest_speed = route =
54.3689 1
 Numerical Methods for Civil Engineers

Lecture 2 - MATLAB 2

- Save & Load

- Input & Output
- Plotting Graph

Mongkol JIRAVACHARADET

SURANAREE INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Save/Load Data to/from External Files

The save and load Commands
>> clear
>> x = 0:5; y=5*x;
>> save xyfile
>> save(‘xyfile’,’x’,’y’)
>> XY = [x’ y’];
>> save xyvals.txt XY -ascii

Loading Matrices from mat Files

>> clear
>> x = linspace(0,2*pi); y = cos(x); z = sin(x);
>> save trigvar
>> clear
 >> whos
>> load trigvar
>> whos

Loading Data from Plain Text Files

>> clear
>> whos
>> XY = load(‘xyvals.txt’)
>> x = XY(:,1)
>> y = XY(:,2)

Input/Output Commands

Input Output - Screen

- File
User MATLAB

Command Description

disp (A) Displays the contents, but not the name, of the
Array A.
disp (‘text’) Displays the text string enclosed within single quotes.

fprintf Control the screen’s output display format.

x = input(‘text’) Displays the text in quotes, waits for user input from
the keyboard, and stores the value in x.

x = input(‘text’,’s’) Displays the text in quotes, waits for user input from
the keyboard, and stores the input as a string in x.
INPUT AND OUTPUT

Prompting for User Input

>> x = input(‘Enter a value for x’);

By default, the input function returns a numerical value.

To obtain a string input, a second parameter, ‘s’ , must be provided.

>> yourname = input(‘Enter your name ‘,‘s’);

function s = inputAbuse
% inputAbuse Use input messages to compute sum of 3 variables

x = input(‘Enter the first variable to be added ‘);

y = input(‘Enter the second variable to be added ‘);
z = input(‘Enter the third variable to be added ‘);
s = x+y+z;

Text Output

The disp Function

>> disp(‘My favorite color is red’)

>> Speed = 63;

>> disp(‘The vehicle’s predicted speed is:’)
>> disp(Speed)

Since disp requires only one argument, the message and variable
must be combined into a single string.

>> yourName = input(‘enter your name ‘,’s’);

>> disp([‘Your name is ‘,yourName])
Display Formats with the fprintf Command

Syntax Description
fprintf(‘format’,A,...) Displays the elements of the array A, and any
addition array arguments, according to the format
specified in the string ‘format’
‘format’ structure %[-][number1.number2]C, where
number1 specifies the minimum field width,
number2 specifies the number of digits to the
right of the decimal point, and C contains control
codes and format codes. Items in brackets are
optional. [-] specifies left justified.

fprintf(format)
fprintf(format,variables)
fprintf(fid,format,variables)

>> fprintf(‘Warning: x is negative\n’)

fprintf (Continue…)

Format Codes Description

%e Scientific format with lowercase e.

%E Scientific format with uppercase E.
%f Decimal format.
%g %e or %f, whichever is shorter.

Control Codes Description

\n Start new line.

\r Beginning of new line.
\b Backspace.
\t Tab.
“ Apostrophe.

\\ Backslash.
fprintf(format,variables)

>> name = ‘Elvis’; age = str2num(datestr(now,10)-1935;

>> fprintf(‘%s is %d years old\n’,name,age);

Elvis is 65 years old

>> fprintf(‘The speed is: %3.1f\n’,Speed)

The speed is: 63.2

>>r = [2.25:20:42.25];
2.25 14.137
>>circum = 2*pi*r;
22.25 139.8
>>y = [r;circum];
42.25 265.46
>>fprintf(‘%5.2f %11.5g\n’,y)

Low-Level Input/Output Functions

fopen open file Permission:
fid = fopen(filename, permission); ‘r’ read
‘rt’ read plain text file
...
‘w’ write
fclose(fid)
‘wt’ write plain text file
‘a’ append
fscanf read data from file ‘r+’ read & write
‘w+’ truncate or create
x = fscanf(fid, format); for read & write
x = fscanf(fid, format, size); ‘a+’ read & append
‘W’ write without
automatic flushing
fscanf print data to file ‘A’ append without
automatic flushing
fprintf(fid, format, variables);
fprintf(fid,format,variables)

>> x = ...
>> fout = fopen(‘myfile.dat’,’wt’);
>> fprintf(fout,’ k x(k)\n’);
>> for k = 1:length(x)
>> fprintf(fout,’%4d %5.2f\n’,k,x(k));
>> end
>> fclose(fout)

Creating a Plot

The plot function has different forms, depending on the input arguments.
If y is a vector, plot(y) produces a piecewise linear graph of the elements of y
versus the index of the elements of y.
If you specify two vectors as arguments, plot(x,y) produces a graph of y versus
x.
For example, these statements use the colon operator to create a vector of x values
ranging from zero to 2π, compute the sine of these values, and plot the result.

>> x = 0:pi/100:2*pi;
>> y = sin(x);
>> plot(x,y)
Now label the axes and add a title. The characters \pi create the symbol π.

>> xlabel('x = 0:2\pi')

>> ylabel('Sine of x')
>> title('Plot of the Sine
>> Function','FontSize',12)
Multiple Data Sets in One Graph

Multiple x-y pair arguments create multiple graphs with a single call to plot.
MATLAB automatically cycles through a predefined (but user settable) list of colors to
allow discrimination among sets of data.
For example, these statements plot three related functions of x, each curve in a
separate distinguishing color.

>> y2 = sin(x-.25);
>> y3 = sin(x-.5);
>> plot(x,y,x,y2,x,y3)

The legend command provides an easy

way to identify the individual plots.

>> legend('sin(x)','sin(x-.25)','sin(x-.5)')

Specifying Line Styles and Colors

It is possible to specify color, line styles, and markers (such as plus signs or
circles) when you plot your data using the plot command.

>> plot(x,y,'color_style_marker')
color_style_marker is a string containing from one to four characters
(enclosed in single quotation marks) constructed from a color, a line style, and
a marker type:
• Color strings are 'c', 'm', 'y', 'r', 'g', 'b', 'w', and 'k'.
These correspond to cyan, magenta, yellow, red, green, blue, white, and black.
• Linestyle strings are '-' for solid, '--' for dashed, ':' for dotted, '-.' for dash-dot.
Omit the linestyle for no line.
• Marker types are '+', 'o', '*', and 'x' and the filled marker types are :
's' for square, 'd' for diamond, '^' for up triangle, 'v' for down triangle,
'>' for right triangle, '<' for left triangle, 'p' for pentagram, 'h' for hexagram,
and none for no marker.
>> x1 = 0:pi/100:2*pi;
>> x2 = 0:pi/10:2*pi;
>> plot(x1,sin(x1),'r:',x2,sin(x2),'r+')
Adding Plots to an Existing Graph

The hold command enables you to add plots to an existing graph. When you type

hold on
MATLAB does not replace the existing graph when you issue another plotting
command; it adds the new data to the current graph, rescaling the axes if
necessary.
For example, these statements first create a contour plot of the peaks function, then
superimpose a pseudocolor plot of the same function.
[x,y,z] = peaks;
contour(x,y,z,20,'k')
hold on
pcolor(x,y,z)
shading interp
hold off
The hold on command causes the pcolor
plot to be combined with the contour
plot in one figure.
 Numerical Methods for Civil Engineers

Lecture 3 - MATLAB 3
Programming with MATLAB :

- Script m-files

- Function m-files

- Flow Control

Mongkol JIRAVACHARADET

SURANAREE INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Script Files
You can perform operations in MATLAB in two ways:
1. Enter commands directly in the Command Window.
2. Run a script M-file which contains all the commands – one at a time.
M-file is just a plain text file ended with extension “.m”
To open the built-in editor, select menu: File > New > M-file
A Script to Plot Functions
trigplot.m

t = linspace(0,2*pi);
y1 = sin(t);
y2 = cos(t);
y3 = y1.*y2;
plot(t,y1,’-’,t,y2,’.’,t,y3,’—’);

>> trigplot
Replace the plot statement with

plot(t,y1,’-’,t,y2,’:’,t,y3,’—’);
axis([0 2*pi -1.5 1.5])
legend(‘sin(t)’,’cos(t)’,’sin(t)*cos(t)’);

>> close all; trigplot

Use TEX notation to display “θ”

legend(‘sin(\theta)’,’cos(\theta)’,
’sin(\theta)*cos(\theta)’);
xlabel(‘\theta (radius)’,’FontName’,’Times’,
’FontSize’,14)

>> close all; trigplot

>> x = linspace(0,2*pi);
>> y1 = sin(x);
>> y2 = cos(x);
>> y3 = y1.*y2;
>> plot(x,y1,'-',x,y2,':',x,y3,'--');
>> axis([0 2*pi -1.5 1.5])
>>
legend('sin(\theta)','cos(\theta)','sin(\theta)*cos(\theta)')
>> xlabel('\theta (radius)','FontName','Times','FontSize',14)
>> title('Plot of simple trigonometric functions',...
'FontName','Times','FontSize',12)
Function m-Files
input parameters
Command Window
>>
function

output parameters

function [outputParameterList] = functionName(inputPramaterList)

output argument
function name
Input argument
pyt.m
function h = pyt(a, b)
% PYT hypotenuse of a right-angled triangle
h = sqrt(a.^2 + b.^2);

>> pyt(3,4)

average.m is a simple function that calculates the average of the

elements in a vector.
function y = average(x)
% AVERAGE Mean of vector elements.
% AVERAGE(X), where X is a vector, is the mean of vector elements.
y = sum(x)/length(x); % Actual computation
>> z = 1:99;
>> average(z)
ans =
50
The H1 Line is the first help text line.
>> lookfor average
AVERAGE Mean of vector elements.

Help Text : >> help average

AVERAGE Mean of vector elements.
AVERAGE(X), where X is a vector, is the mean of vector elements.
Creating P-Code Files

You can convert average.m into a pseudocode called P-code file.

>> pcode average

Text file: Pseudocode:

can be viewed can’t be viewed
by any editor by any editor

- Faster for large program

- Use to hide algorithm

FLOW CONTROL
MATLAB has several flow control constructs:

• if
• switch & case
• for
• while
• continue
• break
if Conditional Control
The if statement evaluates a logical expression and executes a group of statements
when the expression is true.
The optional elseif and else keywords provide for the execution of alternate groups
of statements.
An end keyword, which matches the if, terminates the last group of statements.

Condition:
if condition
Equal A == B
expression
Not equal A ~= B
elseif condition
Greater A>B
expression
Smaller A<B
else Greater or equal A >= B
expression Smaller or equal A <= B
end AND &
OR |

Example 1:
if a < 0
disp(‘a is negative’);
end

Example 2:
if a < 0, disp(‘a is negative’); end

Example 3:
if x >= y
c = x^2 - y;
elseif y/x > 2.0
c = log(y/x);
else
c = x + y;
end
switch & case
The switch statement executes groups of statements based on the value of a variable or
expression.
The keywords case and otherwise delineate the groups.
Only the first matching case is executed. There must always be an end to match the switch.

switch expression
case value1 switch sign(x)
block of statements case -1
disp(‘x is negative’);
case value2
case 0
block of statements disp(‘x is exactly zero’);
. case 1
. disp(‘x is positive’);
. otherwise
disp(‘sign test fail’);
otherwise
end
block of statements
end

for
The for loop repeats a group of statements a fixed,
predetermined number of times.
A matching end delineates the statements.

>> P = zeros(5, 5);

>> for k = 1:5

for l = 1:5
P(k, l) = pyt(k, l);
end
end

>> P
while
The while loop repeats a group of statements
an indefinite number of times under control of a logical condition.

A matching end delineates the statements.

>> x=1;
while condition >> while 1+x > 1
expressions x = x/2;
end
end >> x

LOOP
Break Loops break
return
break
Return Loops return

Here is a complete program, illustrating while, if, else, and end,

that uses interval bisection to find a zero of a polynomial.

a = 0; fa = -Inf;
b = 3; fb = Inf; fb
while b-a > eps*b
x = (a+b)/2;
fx = x^3-2*x-5;
if sign(fx) == sign(fa)
a = x; fa = fx; x
else a b
b = x; fb = fx;
end
end fa
x

The result is a root of the polynomial x3 - 2x - 5, namely

x =
2.09455148154233
Figure Windows

Graphing functions automatically open a new figure window if there are no

figure windows already on the screen. If a figure window exists, MATLAB uses
that window for graphics output. If there are multiple figure windows open,

MATLAB targets the one that is designated the “current figure” (the last figure used
or clicked in).

To make an existing figure window the current figure, you can click the mouse while
the pointer is in that window or you can type

figure(n)
where n is the number in the figure title bar. The results of subsequent graphics
commands are displayed in this window.

To open a new figure window and make it the current figure, type

figure

Multiple Plots in One Figure

The subplot command enables you to display multiple plots in the same window or
print them on the same piece of paper. Typing
subplot(m,n,p)
partitions the figure window into an m-by-n matrix of small subplots and selects
the pth subplot for the current plot.
The plots are numbered along first the top row of the figure window, then the second
row, and so on.
For example, these statements plot data in four different subregions of the figure
window.

>> t = 0:pi/10:2*pi;
>> [X,Y,Z] = cylinder(4*cos(t));
>> subplot(2,2,1); mesh(X)
>> subplot(2,2,2); mesh(Y)
>> subplot(2,2,3); mesh(Z)
>> subplot(2,2,4); mesh(X,Y,Z)
3-D Plot
First, create 1D vectors describing the grids in the x- and y-directions:
>> x = (0:2*pi/20:2*pi)';
>> y = (0:4*pi/40:4*pi)';
Next, ``spread'' these grids into two dimensions using meshgrid:
>> [X,Y] = meshgrid(x,y);
>> whos
Evaluate a function z = f(x,y) of two variables on the rectangular grid:
>> z = cos(X).*cos(2*Y);

Plotting commands:
>> mesh(x,y,z)
>> surf(x,y,z)
>> contour(x,y,z)
 Numerical Methods for Civil Engineers

Lecture 4
Approximations and Errors
- Significant Figures - Truncation Errors
- Accuracy & Precision - Taylor Series
- Error Definitions - Order Notation
- Round-off Errors

Mongkol JIRAVACHARADET

SURANAREE INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Composite surfaces and

terrain modelling
Mixed approximation methods
Significant Figures 49 km/h 2 digits
Car speed = ? 48.8 km/h 3 digits
48.9 km/h 3 digits
48.86 km/h 4 digits

ZERO is NO significant: 184500, 0.1845, 0.01845, 0.00001845 are 4 digits

Use of Significant Figures in Numerical Methods

1) Numerical method yield approximation results. So we have

to specify how confident in our approximated results.

2) Specific quantities such as π , e , or 7 cannot be expressed

exactly by a limited number of digits.

π = 3.141592653589793238462643 . . .

But computers can contain only . . .

>> format short ; pi = 3.1416 (5 digits)

>> format long ; pi = 3.14159265358979 (15 digits)
Try also exp(1), sqrt(7)

* Omission of the remaining significant figures is called round-off error.

 Accuracy and Precision
Increasing accuracy
Increasing precision

UNCERTAINTY
ACCURACY
BIAS

Computes value
close to true value

P R E C I S I O N Computed value close to each other

Error Definitions

True value = Approximation + Error

Numerical Error Et = true value - approximation

true value - approximation

Relative Error εt = 100%
true value

Approximation Error
Iterative approach for better and better approximations

current approximation - previous approximation

εa = 100%
current approximation
EXAMPLE 4.1 Error Estimates for Iterative Methods

x2 x3 xn
Approximate e0.5 by using the series e = 1+ x +
x
+ +L +
2! 3! n!

true value: >> format long ; exp(0.5)

>> ans =
1.648721 . . .

1st estimate: ex = 1
2nd estimate: ex = 1 + x e0.5 = 1 + 0.5 = 1.5

1.648721 − 1.5
true error: εt = × 100% = 9.02%
1.648721

1.5 − 1
approximate error: εa = × 100% = 33.3%
1.5

Terms Results εt (%) εa (%)

1 1 39.3
2 1.5 9.02 33.3
3 1.625 1.44 7.69
4 1.645833333 0.175 1.27
5 1.648437500 0.0172 0.158
6 1.648697917 0.00142 0.0158

After six terms are included, the approximate error falls below εa = 0.05%,
and the computation is terminated.

>> Term = 1:6

>> et = [39.3 9.02 1.44 0.175

0.0172 0.00142]

>> ea = [33.3 7.69 1.27 0.158

0.0158]

>> plot(Term,et,'o-‘,
Term(2:6),ea,'+-')
ROUND-OFF ERRORS
Precision : Computers retain only a fixed number of significant figures
All computations in MATLAB are done in double precision

FORMAT LONG E Floating point format with 15 digits. ROUND-OFF

True value π = 3.141592653589793238462643 . . .

MATLAB pi = 3.141592653589793e+000

Creation of erroneous digit

>> format long e % display all of the significant digits
>> 2.6 + 0.2
>> ans + 0.2
>> ans + 0.2

Computer Representation of Numbers

Computers manipulate binary numbers

Base 10 Conversion Base 2

1 1 = 20 0000 0001
2 2 = 21 0000 0010
4 4 = 22 0000 0100
8 8 = 23 0000 1000
9 8+1 = 23+20 0000 1001
10 8+2 = 23+21 0000 1010
27 16 + 8 + 2 + 1 = 24+23+23+21 0001 1011

>> dec2bin(27)
>> type dec2bin
Bits & Bytes

BIT = Binary digit (0 or 1)

=0 =1
BYTE = Group of 8 bits

= 0001 1011
= 27
MAX BYTE NUMBER = 1111 1111 = 27+26+25+24+23+22+21+20 = 255

Integer Representation

Computer uses 2 bytes = 16 bits & 1st bit for sign

1 0 0 0 0 0 1 0 1 1 0 1 0 0 1 1

Sign Number

Upper limit = 214 + 213 + . . . + 22 + 21 + 20

= 215 - 1 = 32,767
Zero = 0000 0000 0000 0000
- Zero = 1000 0000 0000 0000
Redundant
Integer range: [ -32768 to 32767 ]
Floating-point Numbers

Stored in binary equivalent of scientific notation

123.456 = 123.456 × 100 = 1.23456 × 102 = 123456 × 10-3
37.5 = 0.3750 × 102
-812.5 = - 0.8125 × 103
Signed exponent
0.005781 = 0.5781 × 10-2

Sign Mantissa

How computer stored floating-point number

Signed exponent Mantissa

Sign

MATLAB uses the IEEE floating-point standard

Single Precision 32-bit

Sign Signed exponent Mantissa
1 bit 8 bit 23 bit
Range: +1.18 × 10-38 to +3.40 × 1038
−3.40 × 1038 to −1.18 × 10-38

Double Precision 64-bit

Sign Signed exponent Mantissa
1 bit 11 bit 53 bit
Range: +2.23 × 10-308 to +1.80 × 10308
−1.80 × 10308 to −2.23 × 10-308
MATLAB Discrete Approximation

denormal

under under
overflow usable range usable range overflow
flow flow

-10+308 -10-308 0 10-308 10308

- realmax - realmin realmin realmax

>> format long e

>> 10*realmax
>> realmin/10 denormal (fewer significant digits)
>> realmin/1e16

Effect of Order of Operations

0.99 + 0.0044 + 0.0042 = 0.9986

3-Digits: (0.99 + 0.0044) + 0.0042 = 0.998

0.99 + (0.0044 + 0.0042) = 0.999

x approximate x* to t significant digits when

x − x*
< 5 ×10− t
x
0.998 − 0.9986
Example: = 6.012 × 10−4 < 5 × 10 −3
0.998
= 3 significant digits
TRUNCATION ERRORS
result from using approximation in place of an exact mathematical
procedure
Example: Derivative of velocity

dv ∆v v(ti +1 ) − v(ti )
≅ =
dt ∆t ti +1 − ti

dv
Velocity

Continuous
dt Discrete
∆v
∆t

Time
ti ti+1

Bungee Jumper Problem

rate of change of velocity with respect to time,

dv c
= g − d v2
dt m
where v = vertical velocity (m/s), t = time (s),
g = gravity acceleration ( ≅ 9.81 m/s2)
cd = drag coefficient (kg/m)
m = jumper’s mass (kg)
Analytical solution by solving differential equation,

gm  gcd  ex − e− x
v(t ) = tanh t  tanh( x) = x − x
cd  m  e +e
Example: Compute velocity of a free fall bungee jumper with a mass of 70 kg.
Use a drag coefficient of 0.25 kg/m.

9.81(70)  9.81(0.25) 
v (t ) = tanh  t  = 52.41 tanh(0.1872t )
0.25  70 
>> t=0:2:12;
>> v=52.41*tanh(0.1872*t);
>> [t’ v’] Analytical Solution for the bungee jumper problem
60
Terminal velocity
ans =
0 0 50

2.0000 18.7541
4.0000 33.2506 Velocity, m/s 40

6.0000 42.3829 30
8.0000 47.4160
10.0000 49.9874 20
12.0000 51.2501
10

>> plot(t,v) 0
0 2 4 6 8 10 12
Time, s

Numerical Solution to the differential equation

Rate of change of velocity can be

approximated by v(ti+1)

True slope
dv ∆v v (t i +1 ) − v (t i )
≅ = ∆v dv/dt
dt ∆t t i +1 − t i Approx. slope
∆v v (t i +1 ) − v (ti )
=
Substitute into dv/dt = g – (cd/m)v2 to give v(ti) ∆t t i +1 − t i

v (t i +1 ) − v (t i ) c
= g − d v (t i )2
t i +1 − t i m
ti ti+1 t
Rearrange equation to yield
∆t
 c 
v (t i +1 ) = v ( t i ) + g − d v (t i )2 (t i +1 − t i )
 m 
Euler’
Euler’s method:
dv i
v ( t i +1 ) = v (t i ) + ∆t New value = old value x step size
dt
Example 2: Numerical Solution to the Bungee Jumper Problem
Perform the same computation as previous example but use
the Euler’s method. Employ a step size ∆t = 2 sec.

 c 
v (t i +1 ) = v ( t i ) + g − d v (t i )2 (t i +1 − t i )
 m 

@ start ti = 0 s, ti+1 = 2 s, v(0) = 0 m/s:

 0.25 2 
v ( 2) = 0 + 9.81 − (0)  × 2 = 19.62 m/s
 70 

Next step ti = 2 s, ti+1 = 4 s, v(2) = 19.62 m/s:

 0.25  t(s) v(m/s)

v ( 4) = 19.62 + 9.81 − (19.62) 2  × 2 = 36.49 m/s 0 0
 70 
2 19.62
4 36.49
MATLAB: 6 46.60
>> 0 [Enter] 8 50.71
>> ans+(9.81-(0.25/70)*ans^2)*2 [Enter] 10 51.96
[Up Arrow] [Enter] many time . . . 12 52.30

>> hold on
>> v2 = [0 19.62 36.49 46.6 50.71 51.96 52.3]
>> plot(t,v2,'+-')

Analytical Solution for the bungee jumper problem

60
Terminal velocity
50
Approx. solution
Velocity, m/s

40
Analytical solution
30

20

10

0
0 2 4 6 8 10 12
Time, s
TAYLOR SERIES
Predict function value of one point in term of function value
and its derivatives at other points

f(x)
f(xi) Zero-order approx.
f(xi+1) ≅ f(xi)
First o
rder
Known Se f(xi+1) ≅ f(xi) + f’(xi)h
co
nd
Tr or Slope @ xi
ue de
r

f(xi+1) ≅ f(xi) + f’(xi)h + f’’(xi)h2/2!

f(xi+1): want to know

xi xi+1 x
h

N th-order Approximations

f ′′( x i ) 2 f ′′′( x i ) 3 f ( n ) ( xi ) n
f ( x i +1 ) ≅ f ( xi ) + f ′( x i )h + h + h +L+ h
2! 3! n!
Taylor series expansion:
f ′′( x i ) 2 f ′′′( x i ) 3 f ( n ) ( xi ) n
f ( x i +1 ) = f ( xi ) + f ′( x i )h + h + h +L+ h + Rn
2! 3! n!

Note that, an equal sign replaces the approximate sign.

A reminder term is included to account for all terms from n+1 to infinity:

f ( n +1) (ξ ) n +1
Remainder: Rn = h where xi < ξ < xi+1
(n + 1)!
Truncation error of order n + 1

f ( n +1) (ξ ) n +1
Rn = h = O(h n +1 ) = Truncation error of order hn+1
(n + 1)!

Where ξ is not known but lies somewhere between xi and xi+1

xi ξ xi+1
f(x) =Pn(x) + O(hn+1)

x2 xn
Example: e = 1+ x + +L + +L
x

2! n!
2
x
e x = 1 + x + + O( x3 ) → error of order x3
2!
x2 xn
e = 1 + x + + L + + O( x n +1 ) → error of order x n +1
x

2! n!

Taylor series of a function

Exponential: f ( x) = e x
f ′′( xi ) 2 f ( n ) ( xi ) n
from f ( xi +1 ) = f ( xi ) + f ′( xi ) h + h + ... + h
2! n!
Set: xi+1 = x and xi = 0 so h = xi+1 - xi = x

f ( x) = e x , f ′( x) = e x , ..., f ( n) ( x) = e x
So
f (0) = e0 = 1, f ′(0) = 1, ..., f ( n) ( x) = 1

x 2 x3 xn
f ( x) = e = 1 + x + + + ... +
x

2! 3! n!
 Taylor Series Expansions for Some Common Functions

x3 x5 x 2 n −1
sin( x) = x − + − L − (−1) n
for all x
3! 5! (2n − 1)!
x2 x4 n x
2n
cos( x) = 1 − + − L + (−1) for all x
2! 4! (2n)!
x 2 x3 xn
e = 1+ x + + + L +
x
for all x
2! 3! n!
x 2 x3 x 4 n x
n
ln(1 + x) = x − + − − L − (−1) -1 ≤ x ≤ 1
2! 3! 4! n!
x3 x5 x2n
arctan( x) = x − + − L − (−1) n
-1 ≤ x ≤ 1
3! 5! (2n − 1)!
p ( p − 1) 2 p ( p − 1)( p − 2) 3
(1 + x) p = 1 + px + x + x + L for x < 1
2! 3!

Example: Taylor series approximation of function sin(x)

x3 x5
Plot the first n = 1 to 6 terms of the Taylor series of sin( x ) = x − + −L
6 120
1.5
sin(x)
n=1
1 n=2
n=3
0.5

-0.5

-1

-1.5
-6 -4 -2 0 2 4 6

>> x = -2*pi:pi/30:2*pi;
>> y = sin(x);plot(x,y)
>> axis([-2*pi 2*pi -1.5 1.5])
>> hold on
>> y1 = x; plot(x,y1)
>> y2 = x-x.^3/6; plot(x,y2)
>> y3 = x-x.^3/6+x.^5/120; plot(x,y3)
>> legend('sin(x)','n = 1','n = 2','n = 3')
Example: Approximation of function cos(x) by Taylor Series Expansion

Use Taylor series expansions with n = 0 to 6 to approximate f(x) = cos x at xi+1 = π /3

on the basis of the value of f(x) and its derivatives at xi = π /4.

Solution:

Step size: h = π /3 - π /4 = π /12 True value: cos(π /3) = 0.5

Zero-order approximation: f(π /3) ≅ cos(π /4) = 0.707106781

which represents a percent relative error of

0.5 − 0.707106781
εt = × 100% = 41.4%
0.5

First-order approximation: f’(x) = -sin(x)

f(π /3) ≅ cos(π /4) – sin(π /4)(π /12) = 0.521986659

which has εt = 4.40%

Second-order approximation: f’’(x) = -cos(x)

2
π  π   π  π  cos(π / 4)  π 
f   ≅ cos  − sin   −   = 0.497754491
3 4  4  12  2  12 

which has εt = 0.449%

The process can be continued and the results listed as in

Order n f(π/3) εt(%)

0 0.707106781 41.4
1 0.521986659 4.40
2 0.497754491 0.449
3 0.499869147 2.62 x 10-2
4 0.500007551 1.51 x 10-3
5 0.500000304 6.08 x 10-5
6 0.499999988 2.44 x 10-6
Example: Evaluating the Series for sin(x)

x3 x5 x 2 n −1
sin( x) = x − + − L − (−1) n

3! 5! (2n − 1)!

function ssum = sinser(x)

% sinser Evaluate the series of sin function
tol = 5e-9; % Tolerance on accumulated sum
nmax = 15; % Maximum number of terms
term = x; ssum = term; % Initialize series
fprintf(‘Series approximation to sin(%f)\n’,x);
fprintf(‘ k term ssum\n’);
fprintf(‘%3d %11.3e %12.8f\n’,1,term,ssum);

− x2
Recursion: Tk = Tk − 2 , k = {1, 3, 5,...}
k (k − 1)

for k=3:2:(2*nmax-1)
term = -term*x*x/(k*(k-1));
ssum = ssum + term;
fprintf(‘%3d %11.3e %12.8f\n’,k,term,ssum);
if abs(term/ssum)<tol, break; end
end
fprintf(‘\nTruncation error after %d terms is %g\n\n’,
(k+1)/2,abs(ssum-sin(x)));

>> sinser(pi/6);
 Numerical Methods for Civil Engineers

Lecture 5
Roots of Equations
- Graphical Methods
- Fixed-
Fixed-Point Iteration
- Bisection Method
- Newton’s Method
- Secant Method

fx = - Roots of Polynomials

Mongkol JIRAVACHARADET

SURANAREE INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

What Are Roots?

Quadratic formula: f(x) = ax2 + bx + c = 0

f(x)
− b ± b 2 − 4ac
x=
2a

x
0 x1 x2

Roots = Values of x that make f(x) = 0

GRAPHICAL METHODS
Plot the function and observe where it crosses the x axis.
Example: Throw a Turtle out of a Plane !
Drag = cv
Consider the forces acting on a turtle thrown out from a plane.
According to Newton’s law :

Net force acting on dv

= m 1
the turtle dt
where m = is the mass of the turtle (kg),
v = velocity of the turtle (m/s), Gravity = mg
t = time (s)

For a turtle in free fall, Net force = mg – cv

dv
m = mg − cv 2
where c is some coefficient of friction. dt

Let’s now find the solution. Divide Eq. (2) by m gives:

gm  − 
ct
dv c
=g− v v (t ) = 1 − e m 
dt m c  


if v = 0 at t = 0

gm/c
v(t)

t
Problem: From the velocity equation of the turtle, find the value of the drag
coefficient c such that a turtle of mass m = 5 can attain a prescribed velocity,
v = 10, at a set period of time, t = 9. Use g = 10.

In the context of the problem mentioned above, m, v, and t are constant

(parameters) of the problem.

The only variable that we can adjust is c.

To solve this problem, rewrite the velocity equation as

gm  −  50  
ct 9c
−
f (c ) =  1 − e m  − v (t ) f (c ) = 1 − e 5  − 10
c  
 c  

So our task of solving the problem reduces to merely finding the value of c,
such that f( c ) =0, for the given values of v, m and t.

Graphical method

วิธีที่งายที่สุดคือการพล็อตกราฟ f(x) ซึ่งรากของสมการ f(x) = 0 ก็คือจุดที่กราฟ

ตัดแกน x นั่นเอง สังเกตดวยตาเปลา
f(x)

x
Graphical method : NOT precise

ZOOM

ZOOM

Double root?

Bracketing Method

Coarse level search for roots over large interval

1) Subdividing large interval into smaller subinterval
2) Examine sign at the end of each subinterval

f (x) + + + + - - - + +

+++++++++ ++++
----------
x1 x2 x
Possible ways in an interval
Part (a) and (c) : f(xl) and f(xu) same sign

No roots or Even number of roots

Part (b) and (d) : f(xl) and f(xu) different signs

One roots or Odd number of roots

Exceptions:

(a) Multiple roots

(b) Discontinuous function

Bracketing Algorithm
Given: f (x), xmin, xmax, n
dx = (xmax-xmin)/n % Size of bracket interval
xa = xmin % Initialize left side of test bracket
i = 0 % Initialize counter
while i < n
i = i + 1
xb = xa + dx
if f (xa) and f (xb) have different sign
save [ xa , xb ] for further root finding
end
xa = xb
end
 Detecting Sign Change : f (xa) & f (xb)
1st Solution: f (xa) × f (xb) < 0 ?
>> format long e
>> fa = 1e-120; fb = -2e-300;
>> fa * fb < realmin
ans = 0

2nd Solution: Use sign function

>> fa = 1e-120; fb = -2e-300;

>> sign(fa) ~= sign(fb) return true value = 1
ans = 1

Example: Graphical Methods

Plot the function and observe where it crosses x axis

f(x)
f ( x) =
667.38
x
(1 − e −0.147 x ) − 40 40

x f(x)
4 34.115 20
8 17.653
12 6.067
0.00 16
16 -2.269 0
4 8 12 20 x
20 -8.401
-10

Observe: x ≈ 15 → f (15) = -0.4133

 Interpolation (False-position method)
Assume as linear function for short interval

6.607

12 16 2.269

6.607
Interpolation: x = 12 + (16 − 12) = 14.977 f (14.977) = -0.3691
6.607 − (−2.269)

f (15) = -0. 4133, f (14) = 1.582

1.582
2nd Interpolation: x = 14 + (15 − 14) = 14.793 f (14.793) = -0.013
1.582 − (−0.4133)

Root Finding Methods

Two major classes of methods:
Bracketing methods: start with guesses that bracket, or contain, the root
and then systematically reduce the width of the bracket

§ Graphical method
f(x)
f(x) = 0
§ False-position method
x=?
§ Bisection method

Open Methods: require only a single starting value of x without bracketing

the root.

§ Fixed-point iteration
§ Newton-Raphson method
§ Secant method
Fixed-Point Iteration
1) Rewrite original equation f(x) = 0 into another form x = g(x).
x2 + 3
Example: x − 2x + 3 = 0
2
⇒ x =
2
sin x = 0 ⇒ x = sin x + x
2) Select initial value x0

3) Predict new xi+1 as a function of old xi xi+1 = g (xi)

4) Use iteration xi+1 = g(xi) to find a value that reaches convergence.

Iterative until satisfy

xi +1 − xi
εa = ×100%
xi +1

Example: Finding root of f (x) = x - e-x = 0 → xi +1 = e − xi

1
i xi εa (%)
True value = 0.5671
0 0 0.8
1 1 100
Approximated Root

2 0.3679 171.8
0.6
3 0.6922 46.9
4 0.5005 38.3
5 0.6062 17.4 0.4

6 0.5454 11.2
7 0.5796 5.90 0.2
8 0.5601 3.48
9 0.5711 1.93 0
10 0.5649 1.11 0 1 2 3 4 5 6 7 8 9 10 11 12
Number of Iteration

>> 0
>> exp(-ans)
 Two-Curve Graphical Method
1 1

0.8 f ( x) = e − x
0.6 f ( x) = e− x − x 0.8
0.4
f(x) = exp(-x) - x

0.2 Root 0.6

0
-0.2 0.4 f ( x) = x
-0.4
-0.6 0.2
-0.8
-1 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
x x

>> x = 0 : 0.1 : 1; >> x = 0 : 0.1 : 1;

>> plot(x, exp(-x)-x) >> plot(x, x, x, exp(-x))

Convergence
1

y2 = g(x) y1 = x
0.8

0.6922
0.6062
0.6

y 0.5005

0.4
0.3679

0.2

0
0 0.2 0.4 0.6 0.8 1
x
 Divergence
1

0.8
y1 = x
y2 = g(x)
0.6

y
0.4

0.2

x0
0
0 0.2 0.4 0.6 0.8 1
x

Bisection Method
Repeatedly halve interval while bracketing root
f (x) +

+
xb
xa xm x
-
-
1) Choose interval [ xa , xb ] which has sign-change
2) Compute midpoint of interval xm = ( xa + xb ) / 2
3) Select subinterval which has sign-change and repeat 2)
Example: Use bisection method find the root of equation

(1 − e −0.147 x ) − 40
667.38 True value of the root:
f ( x) = 14.7802
x
Estimate root at midpoint: f(x)
40
12 + 16
xr = = 14
2
14.7802 − 14 Initial interval
εt = × 100% = 5.279% 20
14.7802 [ 12 , 16 ]

Select interval with sign-change: 16

+ - 0
4 8 12 20 x
f (14) = 1.582 → [ 14 , 16 ]
-10

15 − 14
Next estimation: xr = (14+16)/2 = 15 εa = ×100% = 6.667%
14

Termination Criteria and Error Estimates

Stop computation when εa < εs , Ex. εs = 0.5%

Iteration xa xb xr εa(%) εt(%)

1 12 16 14 5.279
2 14 16 15 6.667 1.487
3 14 15 14.5 3.448 1.896
4 14.5 15 14.75 1.695 0.204
5 14.75 15 14.875 0.840 0.641
6 14.75 14.875 14.8125 0.422 0.219

< εs = 0.5%
Bisection Algorithm
initialize: a = . . . , b = . . .
for k = 1, 2, . . . less susceptible to
xm = a + (b-a)/2 roundoff error than
if sign(f (xm)) = sign(f (a)) (a+b)/2
a = xm
else
b = xm
end
if converged, stop
end

Example: Apply Bisection to x - x1/3 - 2 = 0

bisect.m
function xm = bisect(a, b, n)
if nargin<3, n=15; end % Default number of iterations
fa = a - a^(1/3) - 2; % Initial value of f(a) and f(b)
fb = b - b^(1/3) - 2;
fprintf(‘ k a xmid b f(xmid)\n’);
for k = 1:n
xm = a + 0.5*(b-a); % Computing midpoint
fm = xm - xm^(1/3) - 2; % Function value at midpoint
fprintf(‘%3d %12.8f %12.8f %12.8f %12.3e\n’, k, a, xm, b, fm);
if sign(fm) == sign(fa) % Root lies in [xm, b] replace a
a = xm;
fa = fm;
else % Root lies in [xm, a] replace b
b = xm;
fb = fm;
end
end
Newton’s Method
First-order Taylor series: f ( xi +1 ) ≅ f ( xi ) + f ′( xi )( xi+1 − xi )
f ( xi )
Set f ( xi +1 ) = 0 → xi +1 = xi −
f ′( xi )
f(x)

slope = f’(x)
f(xi)

Root

xi+1 xi x
f ( xi )
xi − xi +1 =
f ′( xi )

Approximation Sequence of Newton’s Method

f(x)

f(x1)

x2
x3 x1 x
f(x2)
Failure of Newton’s Method
Case 1: Inflection point in vicinity of root

f(x)

x2 x1 x3
x

Failure of Newton’s Method

Case 2: Oscillate around local maximum or minimum

f(x)

x
Failure of Newton’s Method
Case 3: Jump away for several roots

f(x)

Failure of Newton’s Method

Case 4: Disaster from zero slope

f(x)

x
 Example: Apply Newton’s Method to x - x1/3 - 2 = 0

newton.m

function x = newton(x0, n)
if nargin<2, n=5; end % Default number of iterations
x = x0; % Initial guess
fprintf(‘ k f(x) dfdx x(k+1)\n’);
for k = 1:n
f = x - x.^(1/3) - 2; % Function value
dfdx = 1 - (1/3)*x.^(-2/3); % Derivative value
x = x - f/dfdx;
fprintf(‘%3d %12.3e %12.3e %18.15f\n’, k-1, f, dfdx, x);
end

Secant Method
for function whose derivatives are difficult to evaluate
Derivative approximated by backward finite difference
f ( xi −1 ) − f ( xi ) xi +1 = xi −
f ( xi )
f ′ ( xi ) ≅
xi −1 − xi f ′( xi )

f ( xi ) ( xi −1 − xi )
f(xi)
xi +1 = xi −
f ( xi −1 ) − f ( xi )
f(xi-1)

x
xi-1 xi
Example: Use secant method to estimate root of e-x - x = 0
Initial estimate x-1 = 0 and x0 = 1.0
True root = 0.56714329. . .

First iteration:
x-1 = 0 f (x-1) = 1.00000
x0 = 1 f (x0) = -0.63212
−0.63212(0 − 1)
x1 = 1 − = 0.61270 ε t = 8.0%
1 − ( −0.63212)

Second iteration:
x0 = 1 f (x0) = -0.63212
x1 = 0.61270 f (x1) = -0.07081
−0.07081(1 − 0.61270)
x2 = 0.61270 − = 0.56384 ε t = 0.58%
−0.63212 − (−0.07081)

Third iteration:
x1 = 0.61270 f (x1) = -0.07081
x2 = 0.56384 f (x2) = 0.00518
0.00518(0.61270 − 0.56384)
x3 = 0.56384 − = 0.56717
−0.07081 − ( −0.00518)
ε t = 0.0048%
Roots of Polynomials

f n ( x ) = a0 + a1 x + a2 x 2 + L + an x n
where n = order of polynomial
a’s = constant coefficients

Roots of polynomials:
(1) nth-order equation has n real or complex roots
(2) If n is odd, at least one root is real.

(3) Complex roots exist with conjugate pairs (λ + µi and λ − µi)

Ordinary Differential Equation (ODE)

d2y dy
a2 2 + a1 + a0 y = F (t )
dt dt

d2y dy
General Solution: a2 2 + a1 + a0 y = 0
dt dt

Set y = ert → a2 r 2 ert + a1rert + a0 ert = 0

Canceling exponents → a2 r 2 + a1r + a0 = 0

Characteristic equation
 Characteristic Equation a2 r 2 + a1r + a0 = 0
Roots of equation by quadratic formula

r1 − a1 ± a12 − 4a2 a0 EIGENVALUES

=
r2 2a2

(1) If a12 - 4a2a0 > 0, roots are real

y
Overdamped case
y = c1er1t + c2er2t

(2) If a12 - 4a2a0 = 0, single root is real

y
Critically damped case
y = ( c1 + c2t ) e rt

(3) If a12 - 4a2a0 < 0, two roots are complex conjugate

Underdamped case y
r1
= λ ± µi t
r2
y = c1eλt cos µ t + c2 eλt sin µ t
 f1 ( x) = x 2 − 3 x + 2
r1 3 ± 32 − 4(2) 2 Distinct real roots
= =
r2 2 1

f 2 ( x) = x 2 − 10 x + 25
r1 10 ± 102 − 4(25) 5 Repeated real roots
= =
r2 2 5

f3 ( x) = x 2 − 17 x + 72.5
Complex roots
r1 17 ± 17 2 − 4(72.5) 8.5 + 0.5i
= =
r2 2 8.5 − 0.5i

Roots of Polynomials
3

2.5

1.5
y = f(x)

0.5

-0.5
distinct repeated complex
real roots real roots roots
-1
0 2 4 6 8 10

x
MATLAB’s roots Function
Compute eigenvalues of companion matrix
4th-order polynomial: c5 λ 4 + c4 λ 3 + c3λ 2 + c2λ + c1 = 0
 −c2 / c1 −c3 / c1 −c4 / c1 −c5 / c1 
 1 0 0 0 
Companion
A= 
Matrix  0 1 0 0 
 
 0 0 1 0 
Eigenvalue problem Aυ = λυ
f1 ( x) = x 2 − 3 x + 2 →  root([1 -3 2])
f 2 ( x) = x 2 − 10 x + 25 →  root([1 -10 25])
f3 ( x) = x 2 − 17 x + 72.5 →  root([1 -17 72.5])
 Numerical Methods for Civil Engineers

Lecture 6
System of Linear Algebraic Equations

- Basic Concepts
- Gaussian Elimination
- Backward Substitution
- LU Decomposition

Mongkol JIRAVACHARADET

SURANAREE INSTITUTE OF ENGINEERING

UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Linear Algebraic Equations

A system of n linear equations in n unknowns,

a11x1 + a12x2 + . . . a1nxn = b1

a21x1 + a22x2 + . . . a2nxn = b2
n equations
. .
. .
. .
an1x1 + an2x2 + . . . annxn = bn

n unknowns
Example: c 1 + 3c 2 − 5c 3 = 7
2c 1 − 8c 2 + 3c 3 = −2
5c 1 + 7c 2 − 4c 3 = 9

Matrix Form: Augment Form:

1 3 − 5  c 1   7  1 3 − 5 7 
2 − 8 3  c  = − 2   2 − 8 3 − 2
   2    
5 7 − 4  c 3   9  5 7 − 4 9 

Symbolic Notation A x = b

System of linear algebraic equations Ax=b

 a11 a12 L a1n   x1   b1  m = number of rows
a L a2n   x2  b 
 21 a22 =  2
 M M O M  M  M n = number of columns
    
am1 am2 L amn   xm  bm 
where aij and bi are constants, i = 1,2,…,m, j = 1,2,…,n

Solution : Ax = b MATLAB :
A-1A x = A-1b - Use matrix inversion,

x = A-1b >> x = inv(A)*b

- Use backslash operator,
>> x = A\b
 Requirements for a Solution, Am×n
m rows and n columns

m=n
Solving Ax = b for n unknowns from n equations

m>n
Overdetermined systems
(e.g., least-squares problems)

m<n
Underdetermined systems
(e.g., optimization)

Matrix Rank
rank(A) = number of linearly independent columns in A
>> A = [3 -4 1;6 10 2;9 -7 3];
>> rank(A)

ans =

For matrix A with m rows and n columns, and Ax = b has a solution

- if rank(A) < n , → infinite number of solutions

- if rank(A) = n , → unique solution

For a system of n equations in n unknowns written in the form Ax = b,

the solution x exists and is unique for any b if and only if rank(A) = n.
Consistency Test
Element of vector x are coefficients in the linear equation
 a11   a12   a1n   b1 
a  a  a  b 
Ax = b ⇔ x1   + x2   + L + xn  2 n  =  2 
21 22

 M   M   M  M
       
 am1   am 2   amn  bm 

 a11 a12 a13 L a1n b1 

a a11 a11 L a11 b2 
 11
Augmented matrix: A% =  a11 a11 a11 L a11 b3  = [Ab]
 
M M M O M M
 a11 a11 a11 L a11 bm 

Consistent when A and A% have the same rank

Example: An Inconsistent System from Data Fitting

x 1 2 3
Experiment data: Practical Model: y = α x + β
y 2 1 0.5
Data → Model:
3
α + β = 2
2 2α + β = 1
1 3α + β = 0.5

0
Matrix Form:
 1 1  2
-1  2 1 α  1 
 
 β  =  
-2  3 1    0.5
0 1 2 3 4

Consistency test : >> A=[1 1; 2 1; 3 1]; b=[2; 1; 0.5];

>> rank(A)
>> rank([A b])
2nd Experiment: 1 1  2  1  1  2 
 α
         
 2 1   β  = 1  ⇒ ( −1) 2  + ( 3) 1  = 1 
3 1    0  3  1  0 

Consistency test: [α β ] = [− 1 3 ]
T T

>> A = [1 1; 2 1; 3 1]; b = [2; 1; 0];

>> rank(A)
>> rank([A b]) 3

2 Exact solution
1

-1

-2
0 1 2 3 4
Nonsingular Case

§ If the coefficient matrix A is nonsingular, then it is invertible and we can

solve Ax = b as follows:

Ax =b x = A-1 b
§ This solution is therefore unique. Also, if b = 0, it follows that the unique
-1
solution to Ax = 0 is x = A 0 = 0.

§ Thus if A is nonsingular, then the only solution to Ax = 0 is the trivial

solution x = 0.

Naive Gauss Elimination

The most basic systematic scheme for solving system of linear equations.
The procedure consisted of two steps:

1) Forward elimination of the linear equation matrix using row operation

to obtain an upper triangular matrix.

2) Backward substitution to solve for the unknowns

elimination backward

row operation substitution

Matrix Upper triangular Solution

 Forward elimination

a11 a12 a13 b1  a11 a12 a13 b1 

a b2  ⇒  ′ ′ b2′ 
 21 a22 a23  a22 a23
a31 a32 a33 b3   ′′
a33 b3′′ 

Backward substitution

x3 = b3′′ / a33
′′

x2 = (b2′ − a23
′ x 3 ) / a22
′

x1 = (b1 − a13 x3 − a12 x 2 ) / a11

Substitutions

Backward substitution: Upper triangular matrix

bn
xn =
ann
n
1
xi = (bi − ∑ aij x j ), i = n − 1, n − 2, L, 2, 1
aii j = i +1

Forward substitution: Lower triangular matrix

b1
x1 =
a11
i −1
1
x i = (bi − ∑ aij x j ), i = 1, 2, 3, L, n
aii j =1
EXAMPLE : Naive Gauss Elimination

− 3 x1 + 2 x2 − x3 = −1 − 3 2 − 1 − 1 R1

6 x1 − 6 x2 + 7 x3 = −7 ⇒  6 − 6 7 − 7 R2
 
3 x1 − 4 x2 + 4 x3 = −6  3 − 4 4 − 6  R3

Forward elimination (Row operation):

Pivot element

− 3 2 − 1 − 1 Pivot row
 0 − 2 5 − 9
R2 + 2×R1, R3+R1: ⇒  
 0 − 2 3 − 7

− 3 2 − 1 − 1
R3 - R2: ⇒
 0 −2 5 − 9 
 
 0 0 −2 2

− 3 2 − 1 − 1 − 3 x1 + 2 x 2 − x 3 = −1
 0 −2 5 − 9 ⇒ − 2 x 2 + 5 x3 = −9

 0 0 −2 2 − 2 x3 = 2

Backward substitution:
1
− 3 x1 + 2 x2 − x3 = −1 ⇒ x1 = ( −1 − 2 x2 + x3 ) = 2
−3
1
− 2x 2 + 5 x3 = −9 ⇒ x2 = ( −9 − 5 x3 ) = 2
−2
2
− 2x3 = 2 ⇒ x3 = = −1
−2

MATLAB: >> A = [-3 2 -1;6 -6 7;3 -4 4]

>> b = [-1;-7;-6]
>> x=A\b
Gaussian Elimination with Pivoting
To prevent failure from zero diagonal elements
 2 4 −2 −2 − 4  R1
1 2 4 −3 5  R2
Augmented matrix: A% = [ A b ] = 
 −3 −3 8 −2 7  R3
 
 −1 1 6 −3 7  R4

First row operation: 2 4 −2 −2 − 4 

0 0 5 −2 7 
R2 – R1/2, 
R3 + 3R1/2, 0 3 5 −5 1 
R4 + R1/2  
0 3 5 −4 5 

next elimination fail

PIVOTING
Exchange row to avoid zero pivot element

2 4 −2 −2 − 4  2 4 −2 −2 − 4 
0 0 5 −2 7  0 3 5 −4 5 
 
0 3 5 −5 1  0 3 5 −5 1 
   
0 3 5 −4 5  0 0 5 −2 7 

R3=R3-R2
2 4 −2 −2 − 4  2 4 −2 −2 − 4 
0 3 5 −4 5  0 3 5 −4 5 
 
0 0 0 −1 −4  0 0 5 −2 7 
   
0 0 5 −2 7  0 0 0 −1 4 
Solving Systems with the Backslash Operator

MATLAB
Ax =b x = A-1 b x =A\b

A \ . . . means multiply on the left by the inverse of A

>> A = [2 4 -2 -2;1 2 4 -3;-3 -3 8 -2;-1 1 6 -3]

>> b = [-4;5;7;7]
>> x = A\b

x =

1.0000
2.0000
3.0000
4.0000

LU Decomposition

To solve several Ax = b systems with the same A but different b

LU decomposition avoids repeating steps of Gaussian elimination on A
Main idea is to record the steps used in Gaussian elimination.

Consider the matrix: 1 − 2 3 

A = 2 − 5 12 
0 2 − 10 

Gaussian elimination:
1 − 2 3 
(2)
0 − 1 6 
R2 – 2×R1  
0 2 − 10 
For recording
Gaussian elimination (con’t): Lower triangular matrix:
 1 − 2 3 1 0 0 
R3 – 0×R1 (2) − 1 6
  ⇒ L = 2 1 0 
R3 + 2×R1 (0 ) (−2) 2 0 − 2 1

Mysterious coincidence !!!

 1 0 0   1 − 2 3  1 − 2 3 
LU = 2 1 0  0 − 1 6  = 2 − 5 12  = A
    
0 − 2 1 0 0 2 0 2 − 10 

MATLAB: >> L = [1 0 0;2 1 0;0 -2 1]

>> U = [1 -2 3;0 -1 6;0 0 2]
U >> A = L*U
A =
A 1 -2 3
L 2 -5 12
0 2 -10

Using LU to solve equations

Ly = b : Forward substitution
Ax = b → LUx = b
Ux = y : Backward substitution
For Gauss elimination with pivoting:

P = identity matrix with same rows

PA = LU switched as A in pivoting.

For example PA : (switch R2 ↔ R3)

1 0 0 1 − 2 3  1 − 2 3 
0 0 1 2 − 5 12  = 0 2 − 10 
    
0 1 0  0 2 − 10  2 − 5 12 

Ly = b’
PAx = Pb ≡ b’ → LUx = b’
Ux = y
Gauss elimination as LU decomposition

Gauss elimination can be used to decompose A into L and U as illustrated

for a three-equation system,

a11 a12 a13   x1   b1  Forward a11 a12 a13 

a a23   x 2  = b2   0 a′ ′  = U
 21 a22 elimination
 22 a23 
a31 a32 a33   x 3  b3   0 0 a33 
′′

Store row operation @ zeros position:

 a11 a12 a13 

(f ) a′ a′
′23 
a21 a
a where f21 = , f31 = 31 , and f32 = 32
 21 22  a11 a11 ′
a22
(f31 ) (f32 ) a33′′ 

1 0 0 a11 a12 a13 

[A] → [L][U] L = f21 1 0 U =  0 a22
′ ′ 
a23 
f31 f32 1  0 0 a33 
′
′

EXAMPLE : LU Decomposition with Gauss Elimination

From previous example,

− 3 2 − 1 a11 a12 a13 
 
A =  6 − 6 7  = a21 a22 a23 
a21 6
First step: R2+2R1 f21 = = = −2,
a11 − 3
 3 − 4 4  a31 a32 a33 
a31 3
Second step: R3+R1 f31 = = = −1,
a11 − 3
− 3 2 − 1 a11 a12 a13 
 ′ a′23 
After the second step of forward elimination ⇒  0 − 2 5  =  0 a22 
′
a32 −2  0 − 2 3   0 a32 a33 
 ′ ′
Third step: R3-R2 f32 = = =1
′
a22 −2
− 3 2 − 1 a11 a12 a13 
 ′ 
After forward elimination, U =  0 − 2 5  =  0 a22
′ a23

 
 ′′ 
 0 0 − 2  0 0 a33 

1 0 0  1 0 0
 0 = − 2 1 0
Lower triangular matrix, L = f21 1
f31 f32 1  − 1 1 1

Check LU = A or not ?
Using LU to solve equations (con’t)
Ly = b
Ax = b → LUx = b
− 3 2 − 1  x1   − 1 Ux = y
 6 − 6 7   x  = − 7
   2  
 3 − 4 4   x2  − 6 

Ly = b
1 0 0  y1   b1   1 0 0  y 1   − 1
f 0  y  = b  − 2 1 0   y  =  − 7 
 21 1  2  2    2  
f31 f32 1  y 3  b3   − 1 1 1 y 3  − 6

y1 = b1 y1 = −1

y 2 = b2 − y 1 f21 y 2 = −7 − ( −1)(−2) = −9

y 3 = b3 − y 1 f31 − y 2 f32 y 3 = −6 − ( −1)( −1) − ( −9) × 1 = 2

Ux = y
a11 a12 a13   x1   y1  − 3 2 − 1  x1   − 1
 0 a′ ′  x2  = y 2 
a23  0 − 2 5   x  =  − 9
 22        2  
 0 0 ′′   x3   y 3 
a33  0 0 − 2  x3   2 
y3 2
x3 = x3 = = −1
′′
a33 −2
′
y 2 − x3 a23 − 9 − ( −1)(5)
x2 = x2 = =2
′
a22 −2
y 1 − x 2 a12 − x3 a13 − 1 − 2 × 2 − ( −1)( −1)
x1 = x1 = =2
a11 −3
MATLAB Function: lu

>> [L, U] = lu(A)

− 3 2 − 1  x1   − 1
 6 − 6 7   x  =  − 7
   2  
 3 − 4 4   x2  − 6

LU Decomposition:

>> A = [-3 2 -1;6 -6 7;3 -4 4];

>> b = [-1; -7; -6];
>> [L,U] = lu(A)

Solve equations:

>> y = L\b
>> x = U\y

Various LU decompositions

Crout’
Crout’s reduction (U has ones on the diagonal)
l11 0 0  1 u12 u13  a11 a12 a13 
l 0  × 0 1 u23  = a21 a22 a23 
 21 l 22
l 31 l 32 l 33  0 0 1  a31 a32 a33 

Doolittle’
Doolittle’s method (L has ones on the diagonal)
1 0 0  u11 u12 u13  a11 a12 a13 
l 0  ×  0 u 22 u23  = a21 a22 a23 
 21 1
l 31 l 32 1  0 0 u33  a31 a32 a33 

Cholesky’
Cholesky’s method (The diagonal terms are the same value for
the L and U matrices)
l11 0 0  u11 u12 u13  a11 a12 a13 
l 0  ×  0 u 22 u 23  = a21 a22 a23 
 21 l 22
l 31 l 32 l 33   0 0 u33  a31 a32 a33 

where, lii = uii

Cholesky Decompositions
One of the most popular approaches based on the fact that a symmetric
matrix can be decomposed as
- [A] is a symmetric matrix
[A] = [U]T [U] - Computational advantages:
- Half storage
- Half computation time

Decomposition by recurrence relations. For the i th row:

i −1
uii = aii − ∑ uki2
k =1

i −1
aij − ∑ uki ukj
uij = k =1
for j = i + 1, K, n
uii

EXAMPLE : Cholesky Decomposition

4 8 54  For the first row (i = 1): u11 = a11 = 4 = 2

A =  8 32 216  a 8
u12 = 12 = = 4
54 216 1,494  u11 2
a 54
For the second row (i = 2): u13 = 13 = = 27
u11 2
u22 = a22 − u122
= 32 − 42 = 4
u11 u12 u13 
a23 − u12u13 216 − 4 × 27
u23 = = = 27 U =  0 u 22 u 23 
u22 4
 0 0 u33 
For the third row (i = 3):

u33 = a33 − u13

2
− u23
2
= 1,494 − 27 2 − 27 2 = 6

2 4 27  4 8 27 
Thus, Cholesky yields U = 0 4 27  ⇒ U T U =  8 32 216  = A
  
0 0 6  27 216 1,494 
MATLAB Function: chol

>> U = chol(A)
4 8 54 
A =  8 32 216 
54 216 1,494 

Cholesky Decomposition:

>> A = [4 8 54;8 32 216;54 216 1494];

>> U = chol(A)

Test:

>> U’*U

Example : Member forces in truss

1000 N

[ΣM2 = 0] V3(2) – 1000(1.5) = 0 → V3 = 750 N 1

90o
0.866 m
[ΣFx = 0] H2 = 0 F1
F3

[ΣFy = 0] V2 = 250 N H2 2 30o 60o 3

@ node 1 :
F2
V2 V3
[ΣFx = 0] F1cos30o – F3cos60o = 0
1.5 m 0.5 m
[ΣFy = 0] F1sin30o + F3sin60o = 1000
@ node 3 :
@ node 2 :
[ΣFx = 0] – F2 + F3cos60o = 0
[ΣFx = 0] – F1cos30o + F2 = 0
[ΣFy = 0] F3 = 750/sin60o = 866 N
[ΣFy = 0] F1sin30o = 250
F2 = 866cos60o = 433 N
F1 = 250/sin30o = 500 N
F1sin30o + F3sin60o = 1000  sin 30° 0 sin 60°   F1  1,000 
− cos 30° 1 0  F2  =  0 
– F1cos30o + F2 = 0 
 0 − 1 cos 60° F3   0 
– F2 + F3cos60o = 0

>> A=[sin(pi/6) 0 sin(pi/3);-cos(pi/6) 1 0;0 -1 cos(pi/3)]

>> b=[1000;0;0]

>> F=A\b

F =

500.0000
433.0127
866.0254
 Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression

< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit

y = α x +β

No exact solution but many approximated solutions

 Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error

defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the

error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors

n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:

∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
 Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286

α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
 OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7

sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)

A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571

1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is

1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.

The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313

2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208

7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024

12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686

17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527

22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435

27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375

32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333

37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301

42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277

47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258

52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242

57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229

62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218

67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209

72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201

77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194

82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188

87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182

92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177

97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090

Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:

S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:

∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.

MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in

y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979

n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225

Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and

a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2

Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857

>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
 Error Bounds

By passing an optional second output parameter from polyfit as an input

to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:

∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 

∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';

>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y
 Numerical Methods for Civil Engineers

Lecture 7 Curve Fitting

< Linear Regression

< Polynomial Regression
< Multiple Linear Regression

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

LINEAR REGRESSION
We want to find the curve that will fit the data.

Candidate lines for curve fit

y = α x +β

No exact solution but many approximated solutions

 Error Between Model and Observation

y Observation: [ xi yi ]

Model: y = α x + β

Error: ei = yi – α xi – β

x
Criteria for a “Best” Fit
Find the BEST line which minimize the sum of error for all data

BEST line with error minimized ?

The problem is how to minimize the error.

We can use the error

defined as:

ei = y i − yˆ
Where
ŷ = α x i + β However, the errors can
cancel one another and
still be wrong.
ERROR Definition

To avoid ± signs cancellation, the

error may be defined as:

ei = y i − yˆ
But, the error minimization is going to have problems.
The solution is the minimization of the sum of squares.

S = ∑ (ei )
2

This will give a least square solution.

Least-Square Fit of a Straight Line

Minimize sum of the square of the errors

n n
S r = ∑ e = ∑ (y i − β − αx i )
2 2
i
i =1 i =1

Differentiate with respect to each coefficient:

∂S r
= −2 ∑( y i − β − αx i )
∂β
∂S r
= −2 ∑[( y i − β − αx i ) x i ]
∂α
Setting derivatives = 0 :

0 = ∑ y i − ∑ β − ∑ αx i

0 = ∑ y i x i − ∑ βx i − ∑ αx i2

From Σβ = n β , express equations as set of 2 unknowns ( β , α )

nβ + α ∑ x i = ∑ y i

β ∑ x i + α ∑ x i2 = ∑ y i x i

Solve equations simultaneously:

1
∑ xi yi − ∑ xi ∑ yi S xy
α= n α=
1 S xx
∑ xi2 − ( ∑ xi )
2

β = y −α x
where y and x are the mean of y and x
1
Define: S xy = Σxi yi − Σxi Σyi
n Approximated y for any x is
1
S xx = Σxi2 − ( Σxi ) ŷ = α x + β
2

n
1
S yy = Σyi2 − ( i)
Σ
2
y
n
Example: Fit a straight line to x and y values

xi yi xi2 xi yi yi2
n =7
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 28
3 2.0 9 6.0 4 x = =4
4 4.0 16 16.0 16 7
5 3.5 25 17.5 12.25
6 6.0 36 36.0 36 24
7 5.5 49 38.5 30.25 y = = 3 . 4286
7
Σ 28 24 140 119.5 105

(119.5) − (28)(24) / 7
α= = 0.8393 Least-square fit:
(140) − (28) 2 / 7
y = 0 . 8393 x + 0 . 0714
β = 3.4286 − 0.8393(4) = 0.0714

How good is our fit?

Sum of the square of the errors:

S r = ∑ e = ∑ ( yi − β − α xi ) = ( S xx S yy − S xy2 ) / S xx
n n
2 2
i
i =1 i =1

n
St = ∑ ( yi − y ) = S yy
2
Sum of the square around the mean:
i =1

Sr
Standard errors of the estimation: sy / x =
n−2

St
Standard deviation: sy =
n−2
 Linear regression
sy > sy/x
sy sy/x
y

St − S r S xy2
Coefficient of determination r2 = =
St S xx S yy

r2 คืออัตราสวนการแปรเปลี่ยนคา y ที่เกิดจากการเปลี่ยนคา x

For perfect fit Sr = 0 and r = r2 = 1

Example: error analysis of the linear fit y = 3.4286

α = 0.8393
( yi − y ) ( yi - β - α xi)2
2
xi yi β = 0.0714
1 0.5 8.5765 0.1687
22.7143
2 2.5 0.8622 0.5626 sy =
3 2.0 2.0408 0.3473 7−2
4 4.0 0.3265 0.3265 = 2.131
5 3.5 0.0051 0.5896
2.9911
6 6.0 6.6122 0.7972 sy / x =
7 5.5 4.2908 0.1993 7−2
= 0.773
Σ 28 24 22.7143 2.9911
St Sr
Since sy/x < sy , linear regression has merit.
22.7143 − 2.9911
r= = 0.868 = 0.932
22.7143
Linear model explains 86.8% of original uncertainty.
 OR Example: error analysis of the linear fit

xi yi x i2 xi yi y i2
S xx = 140 − 282 / 7 = 28
1 0.5 1 0.5 0.25
2 2.5 4 5.0 6.25 S yy = 105 − 242 / 7 = 22.7
3 2.0 9 6.0 4
4 4.0 16 16.0 16
5 3.5 25 17.5 12.25 S xy = 119.5 − 28 × 24 / 7 = 23.5
6 6.0 36 36.0 36
7 5.5 49 38.5 30.25 Sr = (28 × 22.7 − 23.52 ) / 28
Σ 28 24 140 119.5 105 = 2.977

Since sy/x < sy , linear regression has merit. 22.7

sy = = 2.131
7−2
23.52 2.977
r =
2
= 0.869 sy / x = = 0.772
28 × 22.7 7−2

Linear model explains 86.9% of original uncertainty.

Confidence Interval (CI)

A confidence interval is an interval in which a measurement or trial
falls corresponding to a given probability.

yˆ i ± ∆
yˆ i
y y

x xi x
For CI 95%, you can be 95% confident that the two curved
confidence bands enclose the true best-fit linear regression line,
leaving a 5% chance that the true line is outside those boundaries.
A 100 (1 - α) % confidence interval for yi is given by
Confidence interval 95% → α = 0.05

1 ( xi − x ) 2
yˆ i ± tα / 2 s y / x +
n S xx

Example: to estimate y when x is 3.4 using 95% confidence interval:

xi yi yˆ = α x + β = 0.8363(3.4) + 0.0714 = 2.9148

95% Confidence → α = 0.05 → tα/2 = t0.025(df = n-2 = 5) = 2.571

1 0.5
2 2.5
1 (3.4 − 4)2
3 2.0 Interval: 2.9148 ± (2.571) (0.772) +
4 4.0 7 28
5 3.5
6 6.0 2.9148 ± 0.7832
7 5.5

T-Distribution
Probability density function of
t 0.025 t 0.025 the t distribution:
t 0.005 t 0.005
(1 + x 2 /ν ) − (ν +1) / 2
f ( x) =
t
B(0.5, 0.5ν ) ν
95% where B is the beta function and
99% ν is a positive integer
shape parameter.

The formula for the beta function is

1
B(α , β ) = ∫ t α −1 (1 − t ) β −1 dt
0
The following is the plot of the t probability density function for 4 different
values of the shape parameter.

ν = df
Degree of freedom

In fact, the t distribution with ν equal to 1 is a Cauchy distribution.

The t distribution approaches a normal distribution as ν becomes large.
The approximation is quite good for values of ν > 30.

Critical Values of t
Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

1 3.078 6.314 12.706 31.821 63.657 318.313

2 1.886 2.920 4.303 6.965 9.925 22.327
3 1.638 2.353 3.182 4.541 5.841 10.215
4 1.533 2.132 2.776 3.747 4.604 7.173
5 1.476 2.015 2.571 3.365 4.032 5.893

6 1.440 1.943 2.447 3.143 3.707 5.208

7 1.415 1.895 2.365 2.998 3.499 4.782
8 1.397 1.860 2.306 2.896 3.355 4.499
9 1.383 1.833 2.262 2.821 3.250 4.296
10 1.372 1.812 2.228 2.764 3.169 4.143

11 1.363 1.796 2.201 2.718 3.106 4.024

12 1.356 1.782 2.179 2.681 3.055 3.929
13 1.350 1.771 2.160 2.650 3.012 3.852
14 1.345 1.761 2.145 2.624 2.977 3.787
15 1.341 1.753 2.131 2.602 2.947 3.733

16 1.337 1.746 2.120 2.583 2.921 3.686

17 1.333 1.740 2.110 2.567 2.898 3.646
18 1.330 1.734 2.101 2.552 2.878 3.610
19 1.328 1.729 2.093 2.539 2.861 3.579
20 1.325 1.725 2.086 2.528 2.845 3.552
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

21 1.323 1.721 2.080 2.518 2.831 3.527

22 1.321 1.717 2.074 2.508 2.819 3.505
23 1.319 1.714 2.069 2.500 2.807 3.485
24 1.318 1.711 2.064 2.492 2.797 3.467
25 1.316 1.708 2.060 2.485 2.787 3.450

26 1.315 1.706 2.056 2.479 2.779 3.435

27 1.314 1.703 2.052 2.473 2.771 3.421
28 1.313 1.701 2.048 2.467 2.763 3.408
29 1.311 1.699 2.045 2.462 2.756 3.396
30 1.310 1.697 2.042 2.457 2.750 3.385

31 1.309 1.696 2.040 2.453 2.744 3.375

32 1.309 1.694 2.037 2.449 2.738 3.365
33 1.308 1.692 2.035 2.445 2.733 3.356
34 1.307 1.691 2.032 2.441 2.728 3.348
35 1.306 1.690 2.030 2.438 2.724 3.340

36 1.306 1.688 2.028 2.434 2.719 3.333

37 1.305 1.687 2.026 2.431 2.715 3.326
38 1.304 1.686 2.024 2.429 2.712 3.319
39 1.304 1.685 2.023 2.426 2.708 3.313
40 1.303 1.684 2.021 2.423 2.704 3.307

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

41 1.303 1.683 2.020 2.421 2.701 3.301

42 1.302 1.682 2.018 2.418 2.698 3.296
43 1.302 1.681 2.017 2.416 2.695 3.291
44 1.301 1.680 2.015 2.414 2.692 3.286
45 1.301 1.679 2.014 2.412 2.690 3.281

46 1.300 1.679 2.013 2.410 2.687 3.277

47 1.300 1.678 2.012 2.408 2.685 3.273
48 1.299 1.677 2.011 2.407 2.682 3.269
49 1.299 1.677 2.010 2.405 2.680 3.265
50 1.299 1.676 2.009 2.403 2.678 3.261

51 1.298 1.675 2.008 2.402 2.676 3.258

52 1.298 1.675 2.007 2.400 2.674 3.255
53 1.298 1.674 2.006 2.399 2.672 3.251
54 1.297 1.674 2.005 2.397 2.670 3.248
55 1.297 1.673 2.004 2.396 2.668 3.245

56 1.297 1.673 2.003 2.395 2.667 3.242

57 1.297 1.672 2.002 2.394 2.665 3.239
58 1.296 1.672 2.002 2.392 2.663 3.237
59 1.296 1.671 2.001 2.391 2.662 3.234
60 1.296 1.671 2.000 2.390 2.660 3.232
 Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

61 1.296 1.670 2.000 2.389 2.659 3.229

62 1.295 1.670 1.999 2.388 2.657 3.227
63 1.295 1.669 1.998 2.387 2.656 3.225
64 1.295 1.669 1.998 2.386 2.655 3.223
65 1.295 1.669 1.997 2.385 2.654 3.220

66 1.295 1.668 1.997 2.384 2.652 3.218

67 1.294 1.668 1.996 2.383 2.651 3.216
68 1.294 1.668 1.995 2.382 2.650 3.214
69 1.294 1.667 1.995 2.382 2.649 3.213
70 1.294 1.667 1.994 2.381 2.648 3.211

71 1.294 1.667 1.994 2.380 2.647 3.209

72 1.293 1.666 1.993 2.379 2.646 3.207
73 1.293 1.666 1.993 2.379 2.645 3.206
74 1.293 1.666 1.993 2.378 2.644 3.204
75 1.293 1.665 1.992 2.377 2.643 3.202

76 1.293 1.665 1.992 2.376 2.642 3.201

77 1.293 1.665 1.991 2.376 2.641 3.199
78 1.292 1.665 1.991 2.375 2.640 3.198
79 1.292 1.664 1.990 2.374 2.640 3.197
80 1.292 1.664 1.990 2.374 2.639 3.195

Confidence Interval
80% 90% 95% 98% 99% 99.8%
df 0.10 0.05 0.025 0.01 0.005 0.001

81 1.292 1.664 1.990 2.373 2.638 3.194

82 1.292 1.664 1.989 2.373 2.637 3.193
83 1.292 1.663 1.989 2.372 2.636 3.191
84 1.292 1.663 1.989 2.372 2.636 3.190
85 1.292 1.663 1.988 2.371 2.635 3.189

86 1.291 1.663 1.988 2.370 2.634 3.188

87 1.291 1.663 1.988 2.370 2.634 3.187
88 1.291 1.662 1.987 2.369 2.633 3.185
89 1.291 1.662 1.987 2.369 2.632 3.184
90 1.291 1.662 1.987 2.368 2.632 3.183

91 1.291 1.662 1.986 2.368 2.631 3.182

92 1.291 1.662 1.986 2.368 2.630 3.181
93 1.291 1.661 1.986 2.367 2.630 3.180
94 1.291 1.661 1.986 2.367 2.629 3.179
95 1.291 1.661 1.985 2.366 2.629 3.178

96 1.290 1.661 1.985 2.366 2.628 3.177

97 1.290 1.661 1.985 2.365 2.627 3.176
98 1.290 1.661 1.984 2.365 2.627 3.175
99 1.290 1.660 1.984 2.365 2.626 3.175
100 1.290 1.660 1.984 2.364 2.626 3.174

∞ 1.282 1.645 1.960 2.326 2.576 3.090

Polynomial Regression
Second-order polynomial:

y = a0 + a1x + a2 x2

Sum of the squares of the residuals:

S r = ∑( y i − a 0 − a 1 x i − a 2 x i2 ) 2

Take derivative with respect to each coefficients:

∂S r
= −2 ∑( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 0

∂S r
= −2 ∑ x i ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 1

∂S r
= −2 ∑ x i2 ( y i − a 0 − a 1 x i − a 2 x i2 )
∂a 2

Normal equations:

( )
n a 0 + (∑ x i )a 1 + ∑ x i2 a 2 = ∑ y i

(∑ x i )a 0 + (∑ x i2 )a + (∑ x )a
1 i
3
2 = ∑ x iy i

(∑ x )a + (∑ x
i
2
0 i
3
)a + (∑ x )a
1 i
4
2 = ∑ x i2 y i

For mth-order polynomial: y = a0 + a1x + a2 x2 + . . . + amxm

We have to solve m+1 simultaneous linear equations.

MATLAB polyfit Function
For second-order polynomial, we can define
 x12 x1 1  y1 
 2  y   c1 
, Y =  2  , C =  c2 
x x2 1
A= 2
M M M M   
 2     c3 
 xm xm 1  ym 
and show that C = ( A ' A) −1 A ' Y or C=A -1Y
Fit norm Fit QR
>> C = polyfit(x, y, n)
>> [C, S] = polyfit(x, y, n)

x = independent variable C = coeff. of polynomial in

y = dependent variable descending power
n = degree of polynomial S = data structure for polyval
function

Example: Fit a second-order polynomial to the data.

( yi − y )
2
xi yi ( yi - a0 - a1xi - a2xi2)2
0 2.1 544.44 0.14332
1 7.7 314.47 1.00286
2 13.6 140.03 1.08158
3 27.2 3.12 0.80491
4 40.9 239.22 0.61951
5 61.1 1272.11 0.09439

Σ 15 152.6 2513.39 3.74657

From the given data: m=2 ∑ xi = 15 ∑ xi4 = 979

n=6 ∑ yi = 152.6 ∑ xi yi = 585.6

x = 2.5 ∑ xi2 = 55 ∑ xi2 yi = 585.6

y = 25.433 ∑ xi3 = 225

Simultaneous linear equations
6 15 25  a0   152.6 
15    
 55 225   a1  =  585.6 
55 225 979  a2  2488.8

Solving these equation gives a0 = 2.47857, a1 = 2.35929, and

a2 = 1.86071.
Least-squares quadratic equation:

y = 2.47857 + 2.35929x + 1.86071x2

Coefficient of determination:

2513.39 − 3.74657
r= = 0.99851 = 0.99925
2513.39

Solving by MATLAB polyfit Function

>> x = [0 1 2 3 4 5];
>> y = [2.1 7.7 13.6 27.2 40.9 61.1];
>> c = polyfit(x, y, 2)
>> [c, s] = polyfit(x, y, 2)
>> st = sum((y - mean(y)).^2)
>> sr = sum((y - polyval(c, x)).^2)
>> r = sqrt((st - sr) / st)
MATLAB polyval Function
Evaluate polynomial at the points defined by the input vector

>> y = polyval(c, x)
where x = Input vector
y = Value of polynomial evaluated at x
c = vector of coefficient in descending order

Y = c(1)*xn + c(2)*x(n-1) + ... + c(n)*x + c(n+1)

Example: y = 1.86071x2 + 2.35929x + 2.47857

>> c = [1.86071 2.35929 2.47857]

Polynomial Interpolation
70
60
50
40
y

30
20
10
0
0 1 2 3 4 5
x

>> y2 = polyval(c,x)
>> plot(x, y, ’o’, x, y2)
 Error Bounds

By passing an optional second output parameter from polyfit as an input

to polyval.

>> [c,s] = polyfit(x,y,2)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')

Interval of ±2∆ = 95% confidence interval

Linear Regression Example:

xi yi
1 0.5
2 2.5
3 2.0
4 4.0
5 3.5
6 6.0
7 5.5

>> [c,s] = polyfit(x,y,1)

>> [y2,delta] = polyval(c,x,s)

>> plot(x,y,'o',x,y2,'g-',x,y2+2*delta,'r:',x,y2-2*delta,'r:')
Multiple Linear Regression

y = c0 + c1x1 + c2x2 + . . . + cpxp

Example case: two independent variables y = c0 + c1x1 + c2x2

Sum of squares of the residual: S r = ∑ ( yi − c0 − c1 x1i − c2 x2 i ) 2

Differentiate with respect to unknowns:

∂S r
= −2 ∑( yi − c0 − c1 x1i − c2 x2 i )
∂c0
∂S r
= −2 ∑ x1i ( yi − c0 − c1 x1i − c2 x2i )
∂c1
∂S r
= −2 ∑ x2i ( yi − c0 − c1 x1i − c2 x2i )
∂c2

Setting partial derivatives = 0 and expressing result in matrix form:

 n ∑ x1i ∑ x2i  c0   ∑ yi 

∑ x    
 1i ∑ x12i ∑ x1i x2i   c1  =  ∑ x1i yi 
 ∑ x2i ∑ x1i x2i ∑ x22i  c2  ∑ x2i yi 

Example:
x1 x2 y  6 16.5 14  c0   54 
16.5    
 76.25 48  c1  = 243.5
0 0 5
2 1 10  14 48 54  c2   100 
2.5 2 9
1 3 0
c0 = 5
4 6 3 c1 = 4
7 2 27
c2 = −3
Multivariate Fit in MATLAB
c0 + c1x11 + c2x12 + . . . + cpx1p = y1
c0 + c1x21 + c2x22 + . . . + cpx2p = y2
.
.
.
c0 + c1xm1 + c2xm2 + . . . + cpxmp = ym
Overdetermined system of equations: A c = y
 x11 x12 L x1 p 1  c0   y1 
x x22 L x2 p 1 c  y 
A= , c =   , and y =  2 
21 1

 M M O M M M M 
     
 xm1 xm 2 L xmp 1  p 
c  ym 
Fit norm >> c = (A’*A)\(A’*y)
Fit QR >> c = A\y

Example:
x1 x2 y
0 0 5
2 1 10
2.5 2 9
1 3 0
4 6 3
7 2 27

>> x1=[0 2 2.5 1 4 7]';

>> x2=[0 1 2 3 6 2]';
>> y=[5 10 9 0 3 27]';
>> A=[x1 x2 ones(size(x1))];
>> c=A\y
 Numerical Methods for Civil Engineers

Lecture 8 Interpolation

Linear Interpolation
Quadratic Interpolation
Polynomial Interpolation
Piecewise Polynomial Interpolation

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Visual Interpolation

60
40 80

20 100
km/h

0 120

Vehicle speed is approximately 49 km/h

Interpolation between data points

Consider a set of xy data collected during an experiment.

We use interpolation technique to estimate y at x where there’s no data.

y
known data

What is the
corresponding value
of y for this x ?

BASIC IDEAS
From the known data ( xi , yi ), interpolate yˆ = F ( xˆ ) for xˆ ≠ x i

x1 y1 y F(x)
M M
xi y i interpolate ŷ
xˆ → yˆ
x i +1 y i +1
M M
x
xn yn x̂
Determining coefficient a1, a2, . . . , an of basis function F(x)

F(x) = a1Φ1(x) + a2Φ2(x) + . . . + anΦn(x)

Polynomials are often used as the basis functions.

F(x) = a1 + a2 x + a3 x2 + . . . + an xn-1
Interpolation v.s. Curve Fitting

y known data
curve fit
interpolation

x
Curve fitting: fit function & data not exactly agree
Interpolation: function passes exactly through known data

Interpolation & Extrapolation

Interpolation approximate within the range of independent variable
of the given data set.

Extrapolation approximate outside the range of independent variable

of the given data set.
y

x1 x2 x
Linear Interpolation

The most common way to estimate a data point between 2 points.

The function is estimated by a straight line drawn between them.

Interpolated Point

Linear, Quadratic, and Cubic Interpolations

2 points 3 points 4 points

Linear = 1st order Quadratic = 2nd order Cubic = 3rd order

f(x) = a0 + a1x f(x) = a0 + a1x + a2x2 f(x) = a0 + a1x+ a2x2 + a3x3

General formula for an (n – 1)th-order polynomial:

f(x) = a0 + a1x+ a2x2 +…+ anxn-1

Linear Interpolation

Connect two data points with a straight line

y1
Using similar triangles:

F ( xˆ ) = yˆ yˆ − y 0 y − y0
= 1
xˆ − x0 x1 − x 0
y0

x0 x̂ x1

y1 − y 0
F ( xˆ ) = yˆ = y 0 + ( xˆ − x0 )
x1 − x0

Quadratic Interpolation
Second-order polynomial interpolation using 3 data points

(x1, y1) Convenient form:

f2 ( x ) = b0 + b1( x − x0 ) + b2 ( x − x0 )( x − x1 )

(x2, y2) f2 ( x ) = a0 + a1x + a2 x 2

where a0 = b0 + b1x0 + b2 x 0 x1
(x0, y0) a1 = b1 + b2 x0 + b2 x1
a2 = b2
Substitute f2 ( x0 ) = y 0 → b0 = y 0
y1 − y 0
Substitute f2 ( x1 ) = y 1 → b1 =
x1 − x 0
y 3 − y 2 y 2 − y1
−
x3 − x 2 x 2 − x1
Substitute f2 ( x 2 ) = y 2 → b2 =
x3 − x1
EXAMPLE : Linear & Quadratic Interpolation ln(1) = 0
ln(2) = 0.6932
Estimate ln(2) by using linear & quadratic interpolation ln(4) = 1.3863
ln(6) = 1.7918
Linear interpolation from x1 = 1 to x2 = 6

1.7918 − 0
f1(2) = 0 + ( 2 − 1) = 0.3584 → ε t = 48 .3%
6 −1
Linear interpolation from x1 = 1 to x2 = 4

1.3863 − 0
f1(2) = 0 + ( 2 − 1) = 0.4621 → ε t = 33 .3%
4 −1
Quadratic interpolation from x1 = 1 to x2 = 4 and x3 = 6

1.3863 − 0
b0 = 0 , b1 = = 0.4621
4 −1
1.7918 − 1.3863
− 0.4621
b3 = 6 − 4 = −0.05187
6 −1

Substitute b0, b1 and b2 into equation

f2(x) = 0 + 0.4621(x - 1) - 0.05187(x - 1)(x - 4)

which can be evaluated at x = 2 for

f2(2) = 0 + 0.4621(2 - 1) - 0.05187(2 - 1)(2 - 4)

f2(2) = 0.5658 → εt = 18.4%

f2(x) = – 0.05187x2 + 0.7251x – 0.6696

2
MATLAB :
>> x = 1:0.01:7;
>> y = log(x);
>> y2= -0.05187*x.^2+...
0.7251*x-0.6696; 1
>> plot(x,y,x,y2)

1 2 3 4 5 6 7
Polynomial Interpolation

Finding Pn-1(x) of degree n-1 that passes through n known data pairs
Linear = 1 : 2 pt.
Pn-1(x) = c1xn-1 + c2xn-2 + . . . + cn-1x + cn Quadratic = 2 : 3 pt.
Cubic = 3 : 4 pt.

Vandermonde Systems n pairs of (x, y)

n equations
n unknowns

Polynomial pass through each of data points

Example: Construct a quadratic interpolating function

y = c1x2 + c2x + c3
that pass through (x, y) support points (-2, -2), (-1, 1), and (2, -1)

Substitute known points into equation:

-2 = c1 (-2)2 + c2 (-2) + c3
1 = c1 (-1)2 + c2 (-1) + c3
-1 = c1 (2)2 + c2 (2) + c3
Rewritten in matrix form:
 4 −2 1  c1   −2 
1 −1 1  c2  =  1 

 2 2 1  c3   −1
  x12 x1 1  c1   y1 
 2 
Vandermonde Matrix
 x2 x2 1 c2  =  y2 
 x32 x3 1  c3   y3 


MATLAB:
>> x = [-2 -1 2]’;
>> A = [x.^2 x ones(size(x))];
or use the built-in vander function
>> A = vander([-2 -1 2]);
>> y = [-2 1 -1]’;
>> c = A\y
c =
-0.9167
0.2500
2.1667

MATLAB : polyfit and polyval Functions

Find y = c1x2 + c2x + c3 that pass through (x, y) support points

(-2, -2), (-1, 1), and (2, -1).

>> x = [-2 -1 2];

>> y = [-2 1 -1];
>> c = polyfit(x,y,2)
c =
-0.9167 0.2500 2.1667

We can then use the polyval function to perform an interpolation as in

Ex. To interpolate y at x = 1

>> polyval(c,1)
ans =
1.5000
Also, we can use the polyval function to plot the result as in

>> xhat = -3:0.1:3;

>> yhat = polyval(c,xhat);
>> plot(xhat,yhat,x,y,'o')

-2

-4

-6

-8
-3 -2 -1 0 1 2 3

Polynomials Wiggle
2nd-order
3rd-order
y
4th-order
5th-order

xi 1 2 3 4 5 6 7 8 9 10
yi 3.5 3.0 2.5 2.0 1.5 -2.4 -2.8 -3.2 -3.6 -4.0
MATLAB’s Command Lines to Demonstrate Polynomials Wiggle
>> x = [1 2 3 4 5 6 7 8 9 10];
>> y = [3.5 3.0 2.5 2.0 1.5 -2.4 -2.8 -3.2 -3.6 -4.0];
>> x0 = 1:0.1:10;
>> y2 = polyval(polyfit(x(4:6), y(4:6), 2), x0);
>> y3 = polyval(polyfit(x(4:7), y(4:7), 3), x0);
>> y4 = polyval(polyfit(x(3:7), y(3:7), 4), x0);
>> y5 = polyval(polyfit(x(3:8), y(3:8), 5), x0);
>> axis([0 10 -5 5])
>> plot(x, y, ‘o’)
>> hold on
>> plot(x0, y2)
>> plot(x0, y3)
>> plot(x0, y4)
>> plot(x0, y5)

Piecewise Polynomial Interpolation

Using a set of lower degree interpolants on subinterval of
the whole domain
= breakpoint or knot
y

x
Piecewise-linear interpolation
Piecewise-quadratic interpolation
Piecewise-cubic interpolation
f’(x) and f’’(x) continuous at breakpoint = cubic spline
MATLAB’s Built-in Interpolation Functions

Funnction Description

interp1 1-D interpolation with piecewise polynomials.

interp2 2-D interpolation with nearest neighbor, bilinear,

or bicubic interpolants.

interp3 3-D interpolation with nearest neighbor, bilinear,

or bicubic interpolants.
interpft 1-D interpolation of uniformly spaced data using
Fourier Series (FFT).

interpn n-D extension of methods used by interp3.

spline 1-D interpolation with cubic-splines using

not-a-knot or fixed-slope end conditions.

interp1 Built-in Function

1-D interpolation with one of the following 4 methods:

1. Nearest-neighbor uses piecewise-constant function.

Interpolant discontinue at midpoint between knots

2. Linear interpolation uses piecewise-linear polynomials.

3. Cubic interpolation uses piecewise-cubic polynomials.

Interpolant and f’ (x) are continuous.

4. Spline interpolation uses cubic splines. This option performs

the same interpolation as built-in function spline.
How to use interp1 ?

>> yhat = interp1(y, xhat)

>> yhat = interp1(x, y, xhat)
>> yhat = interp1(x, y, xhat, method)

where y = tabulated values to be interpolated.

x = independent values. If not given x=1:length(y).
xhat = values at which interpolant be evaluated.
method = ‘nearest’, ‘linear’, ‘cubic’ or ‘spline’

Example: Interpolation with piecewise-polynomials

>> x = [1 2 3 4 5 6 7 8 9 10];
>> y = [3.5 3.0 2.5 2.0 1.5 -2.4 -2.8 -3.2 -3.6 -4.0];
>> xhat = 1:0.1:10; % eval interpolant at xhat
>> yn = interp1(x, y, xhat, ‘nearest’);
>> plot(x, y, ‘o’, xhat, yn); pause;
>> yl = interp1(x, y, xhat, ‘linear’);
>> plot(x, y, ‘o’, xhat, yl); pause;
>> yc = interp1(x, y, xhat, ‘cubic’);
>> plot(x, y, ‘o’, xhat, yc); pause;
>> ys = interp1(x, y, xhat, ‘spline’);
or >> ys = spline(x, y, xhat);
>> plot(x, y, ‘o’, xhat, ys);
EXAMPLE : Linear interpolation by interp1 function

Estimate the value of y when x is equal to 3.5

x 0 1 2 3 4 5
y 15 10 9 6 2 0

>> x = 0:5; 15

>> y = [15 10 9 6 2 0];

>> interp1(x,y,3.5) 10

ans =
4 5

Plot Graph:

>> xhat = 0:0.2:5; 0

0 1 2 3 4 5

>> yhat = interp1(x,y,xhat);

>> plot(x,y,'bo-',xhat,yhat,'rx')
 Numerical Methods for Civil Engineers

Lecture 9 Numerical Integration

- Basic Ideas
- Symbolic vs.
vs. Numerical Integration
- Trapezoid Rule
- Simpson’
Simpson’s Rule
- MATLAB quad and quad8
quad8 Functions

Mongkol JIRAVACHARADET
SURAN AR EE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

BASIC IDEAS
b
I = ∫ f ( x ) dx = Area under curve f ( x )
a
Node
f (x)

Trapezoidal
region

x
a b
Approximated by piecewise-linear:
Area = Sum of trapezoidal regions
Symbolic Math Toolbox
Symbolic Math Toolboxes incorporate symbolic computation into the numeric
environment of MATLAB.
Symbolic Integration with MATLAB

>> I = int(f) % Indefinite integral

>> I = int(f, v) % Designating integration variable
>> I = int(f, a, b) % Definite integral on close interval
>> I = int(f, v, a, b)
where f = symbolic expression

Variables must be define as symbolic by sym or syms

>> x = sym(‘x’), y = sym(‘y’), z = sym(‘z’)
or >> syms x y z

Example : I = ∫ ( x 3 − c ) dx
a

>> syms x a b c
>> I = int(x^3-c,x,a,b)

I =

1/4*b^4-c*b-1/4*a^4+c*a
What is Integration?
The integral is equivalent to the area under the curve.
f(x)

I = ∫ f ( x ) dx
a I
x
a b
Examples of how integration is used to evaluate areas

Net force due to

wind blowing
against the building

Cross-sectional area of a river

Newton-Cotes Formulas
Replace a complicated function with a polynomial that is easy to integrate
b b
I = ∫ f ( x ) dx ≅ ∫ fn ( x ) dx
a a

where fn ( x ) = a0 + a1x + L + an x n −1 + an +1x n

f(x) f(x)

x x
a b a b

Approximate area by a straight line Approximate area by a parabola

Trapezoidal Rule

f(x) 1st degree polynomial = Linear line

f(b)
f (b ) − f (a )
f1( x ) = f (a) + ( x − a)
f1(x) b−a

f (a ) + f ( b )
b
f(a) I = ∫ f1( x ) dx = (b − a )
a
2
x
a b

Area of trapezoid = width x average height

width = b – a = h
f (a ) + f ( b )
average height =
2

Composite Trapezoidal Rule

Improve accuracy by dividing interval into subintervals

f(x) f5
f4
f3
f2
f1

x1 x2 x3 x4 x5

x5 x2 x3 x4 x5

∫ f ( x ) dx = ∫ f ( x ) dx + ∫ f ( x ) dx + ∫ f ( x ) dx
x1 x1 x2 x3
+ ∫ f ( x ) dx
x4
f(x)
There are n segment with equal width:

b–a
h =
n
x2 x3 xn

I = ∫ f ( x ) dx + ∫ f ( x ) dx + L + ∫ f ( x ) dx
x1 x2 x n −1

f ( x1 ) + f ( x 2 ) f ( x2 ) + f ( x 3 )
I =h +h +L
2 2
f ( x n−1 ) + f ( x n )
x1 x2 x3 x4 x5 x6 +h
2
x1 = a b–a xn = b
h=
n

f +f f +f f +f h  n −1

I = h 1 2 + h 2 3 + L + h n −1 n I = f1 + 2 ∑ fi + fn 
2 2 2 2  i =2 

f(x)
f4 f5

MEAN
f2 f3

f6
f1

x1 x2 x3 x4 x5 x6

 n −1

b–a
f
1

+ 2 ∑
i =2
f i + f n

Substitute h = I = (b − a )
n 2n
width average height
Example 9-1 Use Trapezoidal rule to numerically integrate
f ( x ) = 0.2 + 25 x − 200 x 2 + 675 x 3 − 900 x 4 + 400 x 5
from a = 0 to b = 0.8. Note that the exact value of the integral can be determined
analytically to be 1.640533.

or using MATLAB’s Symbolic Math Toolbox :

>> x = sym(‘x’)
>> I=int(0.2+25*x-200*x^2+675*x^3-900*x^4+400*x^5,x,0,0.8)

I = 3076/1875 = 1.6405

Single Trapeziod: f(0) = 0.2 and f(0.8) = 0.232 f(x)

f (a ) + f (b )
I = (b − a)
2
0.2 + 0.232
= (0.8 − 0) = 0.1728 Error
2

1.6405 − 0.1728
εt = × 100% = 89 .5% Integral estimate
1.6405 0 0.8

2 Trapeziods: n = 2 (h = 0.4) :

f(0) = 0.2 f(0.4) = 2.456 f(0.8) = 0.232

 n −1

f1 + 2 ∑ fi + fn 
I = (b − a)  i =2 
2n

0.2 + 2(2.456 ) + 0.232

I = (0.8 ) = 1.0688
4

1.6405 − 1.0688
εt = × 100 % = 34.9%
1.6405
MATLAB’s trapz function
Trapezoidal numerical integration

>> z = trapz(y) % Integral of y with unit spacing

>> z = trapz(x, y) % Integral of y with respect to x
90o
Example: ∫ sin x dx
0o
>> angle = 0:15:90;
>> x = (pi*angle/180);
>> y = sin(x);
>> z = trapz(x,y)
>> z =
0.9943

Example 9-2 Use MATLAB’s trapz function to numerically integrate

f ( x ) = 0.2 + 25 x − 200 x 2 + 675 x 3 − 900 x 4 + 400 x 5
from a = 0 to b = 0.8. Note that the exact value of the integral can be determined
analytically to be 1.640533.

For n = 2 : >> x = linspace(0,0.8,3);

>> fx = 0.2+25*x-200*x.^2+675*x.^3-900*x.^4+400*x.^5;
>> I = trapz(x,fx)
I =
1.0688

n I εt

2 1.0688 34.9
3 1.3639 16.5
4 1.4848 9.49
5 1.5399 6.13
6 1.5703 4.28
Simpson’s Rules
Using higher-order polynomials to connect the points

f(x) f(x)

x x
Simpson’s 1/3 rule Simpson’s 3/8 rule
Quadratic connecting 3 points Cubic connecting 4 points

Simpson’s 1/3 Rules

Quadratic connecting 3 points x2 x2
f(x)
f(x1) f2(x) I= ∫ f ( x)dx ≅ ∫ f 2 ( x)dx
f(x2) x0 x0
f(x0)
( x − x1 )( x − x2 )
f 2 ( x) = f ( x0 )
( x0 − x1 )( x0 − x2 )
( x − x0 )( x − x2 )
+ f ( x1 )
( x1 − x0 )( x1 − x2 )
( x − x0 )( x − x1 )
x0 x1 x2 + f ( x2 )
( x2 − x0 )( x2 − x1 )

h x −x
I≅ [ f ( x0 ) + 4 f ( x1 ) + f ( x2 )] , h = 2 0
3 2
Example 9-3 Use Simpson’s 1/3 rule to numerically integrate
f ( x ) = 0.2 + 25 x − 200 x 2 + 675 x 3 − 900 x 4 + 400 x 5
from a = 0 to b = 0.8. Note that the exact value of the integral can be determined
analytically to be 1.640533.

Solution: n = 2 (h = 0.4) :

f(0) = 0.2 f(0.4) = 2.456 f(0.8) = 0.232

0.4
I = (0.2 + 4( 2.456) + 0.232) = 1.3675
3

1.6405 − 1.3675
εt = × 100 % = 16.6%
1.6405

Composite Simpson’s 1/3 Rules

b−a
x2 x4 xn

I = ∫ f ( x ) dx + ∫ f ( x ) dx + L + ∫ f ( x ) dx where h = , n = even number

x0 x2 xn − 2
n

I=
h
[f0 + 4f1 + f2 ] + h [f2 + 4f3 + f4 ] + L + h [fn−2 + 4fn −1 + fn ]
3 3 3
1 4 1 1 4 1
1 4 1 1 4 1 1 4 1

4 2
4
1 2
4 1
2 4
4 2

a b

(b − a)  n −1 n −2 
I=
3n 
 0
f ( x ) + 4 ∑ i j =∑
f ( x ) + 2 f ( x j ) + f ( x n 
)
i =1,3,5 2, 4, 6 
Example 9-4 Use composite Simpson’s 1/3 rule with n = 4 to numerically integrate
f ( x ) = 0.2 + 25 x − 200 x 2 + 675 x 3 − 900 x 4 + 400 x 5
from a = 0 to b = 0.8. Note that the exact value of the integral can be determined
analytically to be 1.640533.

Solution: n = 4 (h = 0.2) :

f(0) = 0.2 f(0.2) = 1.288 f(0.4) = 2.456 f(0.6) = 3.464 f(0.8) = 0.232

(b − a )  n −1 n−2 
I=
3n 
 0
f ( x ) + 4 ∑ i j =∑
f ( x ) + 2 f ( x j ) + f ( x n 
)
i =1,3,5 2, 4,6 

0.8
I = ( 0.2 + 4(1.288 + 3.464) + 2( 2.456) + 0.232) = 1.624
3( 4)

1.6405 − 1.624
εt = × 100 % = 1.04%
1.6405

Simpson’s 3/8 Rules

Cubic connecting 4 points b b
I = ∫ f ( x)dx ≅ ∫ f 2 ( x)dx
a a
f(x)
3h
I≅ [ f ( x0 ) + 3 f ( x1 ) + 3 f ( x2 ) + f ( x3 )]
8
h = ( x3 − x0 ) / 3

x
x0 x1 x2 x3
a b
(b − a)
or… I = [f ( x0 ) + 3 f ( x1) + 3 f ( x2 ) + f ( x3 )]
8
Truncation Errors

Interval
Method Et width Et
1 3 (b − a )3
Trapezoid h f ′′(ξ ) h =b−a f ′′(ξ )
12 12
1 5 (4) b−a (b − a )5 (4)
Simpson’s 1/3 h f (ξ ) h= f (ξ )
90 2 2880

3 5 (4) b−a (b − a )5 (4)

Simpson’s 3/8 h f (ξ ) h= f (ξ )
80 3 6480
Simpson’s 3/8 rule is used when number of segments is odd.

Apply Simpson’s 1/3 and 3/8 Rules

To handle multiple application with odd number of intervals

f(x)

x
1/3 rule 3/8 rule
Example 9-5 Use Simpson’s 1/3 + 3/8 rule with n = 5 to numerically integrate
f ( x ) = 0.2 + 25 x − 200 x 2 + 675 x 3 − 900 x 4 + 400 x 5
from a = 0 to b = 0.8. Note that the exact value of the integral can be determined
analytically to be 1.640533.
f(x)
Solution: n = 5 (h = 0.16) :
f(0) = 0.2 f(0.16) = 1.297
f(0.32) = 1.743 f(0.48) = 3.186
f(0.64) = 3.182 f(0.8) = 0.232
For the first two segments use Simpson’s 1/3 :
0.16
I = (0.2 + 4(1.297) + 1.743) = 0.380
3
For the last three segments use Simpson’s 3/8 : x
0 0.16 0.32 0.48 0.64 0.8
0.48
I = (1.743 + 3(3.186 + 3.182) + 0.232)
8
1/3 rule 3/8 rule
= 1.265
1.6405 − 1.645
Total integral : I = 0.380 + 1.265 = 1.645 εt = × 100 % = 0.274 %
1.6405

MATLAB’s quad and quad8 Functions

quad : low order method, quad8: high order method

>> q = quad(’f',a,b)
Approximates the integral of f(x) from a to b within a relative
error of 1e-3 using an adaptive recursive Simpson's rule.

’f' is a string containing the name of the function.

>> quad(‘sin’, 0, pi/2)

Function f must return a vector of output values if given

a vector of input values.
 0.8
Example: ∫
0
0.2 + 25x − 200 x 2 + 675x3 − 900 x 4 + 400 x5 dx

poly5.m
function p = poly5(x)
p=0.2+25*x-200*x.^2+675*x.^3-900*x.^4+400*x.^5;

>> quad(‘poly5’, 0, 0.8)

ans =

1.6405

Example: Computing the Length of a Curve

x(t ) = sin(2t ), y (t ) = cos(t ), z (t ) = t

where t ∈ [ 0, 3π ]
>> t = 0:0.1:3*pi;
>> plot3(sin(2*t), cos(t), t)
10

0
1
0.5 1
0 0.5
0
-0.5
- 0.5
-1 -1
 3π

∫
Length of the curve:
4 cos(2t )2 + sin(t )2 + 1 dt
Norm of derivative
0

hcurve.m
function f = hcurve(t)
f = sqrt(4*cos(2*t).^2 + sin(t).^2 + 1);

>> len = quad(‘hcurve’,0,3*pi)

len =
17.2220

Example: Double Integration

y max x max

∫ ∫
y min x min
f ( x, y ) dx dy

For example: f (x, y) = y sin(x) + x cos(y)

integrnd.m

function out = integrnd(x,y)

out = y*sin(x) + x*cos(y);

>> xmin = pi; xmax = 2*pi;

>> ymin = 0; ymax = pi;
>> result = dblquad(‘integrnd’,xmin,xmax,ymin,ymax);
result =
-9.8698
 Numerical Methods for Civil Engineers

Lecture 10 Ordinary Differential Equations

- Basic Ideas
- Euler’s Method
- Higher Order One-step Methods
- Predictor-Corrector Approach
- Runge-Kutta Methods
- Adaptive Stepsize Algorithms

Mongkol JIRAVACHARADET
SURANAREE INSTITUTE OF ENGINEERING
UNIVERSITY OF TECHNOLOGY SCHOOL OF CIVIL ENGINEERING

Ordinary Differential Equations

Rate of change of one variable with respect to another.

dy
= f (t , y )
dt
where t = independent variable
y = dependent variable

Initial Condition y(t0) = y0 x

k
Example: Mass-spring system
m
c
d 2x dx
ma + cv + kx = m + c + kx = 0
dt dt
EULER’S METHOD
(t − t0 ) 2
From Taylor series: y (t ) = y (t0 ) + (t − t0 ) y ′(t0 ) + y ′′(t0 ) +L
2!
Retaining only first derivative: y (t ) = y0 + h f (t0 , y0 )
where h = (t - t0), y0 = y(t0), and f(t0, y0) = y’(t0)
y Predict
error
True
y1 = y0 + h f (t0 , y0 )
y2 = y1 + h f (t1 , y1 )
h
yi = yi −1 + h f (ti −1 , yi −1 )
t0 t1 t

New value = Old value + slope x step size

Example: Manual Calculation with Euler’s Method

dy
= t − 2 y, y (0) = 1
dt

Exact solution: y = ( 2t − 1 + 5e −2t )

1
4
Set h = 0.2
Euler
ti f (ti-1, yi-1) yi=yi-1+h f (ti-1, yi-1) Exact Error

0.0 NA initial cond. = 1.0 1.0000 0

0.2 0-2(1) = -2.0 1.0+0.2(-2.0) = 0.6 0.6879 -0.0879
0.4 0.2-2(0.6) = -1.0 0.6+0.2(-1.0) = 0.4 0.5117 -0.1117
0.6 0.4-2(0.4) = -0.4 0.4+0.2(-0.4) = 0.32 0.4265 -0.1065
Implementing Euler’s Method

odeEuler.m
function [t,y] = odeEuler(diffeq,tn,h,y0)
% Input: diffeq = (string) name of m-file that
% evaluate right hand side of ODE
% tn = stopping value of independent variable
% h = stepsize
% y0 = initial condition at t=0

t = (0:h:tn)’;
n = length(t);
y = y0*ones(n,1);

for i=2:n
y(i) = y(i-1)+h*feval(diffeq,t(i-1),y(i-1));
end

dy
Example: = t − 2 y, y (0) = 1, h = 0.2
dt
rhs1.m
function dydt = rhs1(t,y)
dydt = t-2*y;

>> [t,y] = odeEuler(‘rhs1’,0.6,0.2,1.0)

t =
0
0.2000
0.4000
0.6000
y =
1.0000
0.6000
0.4000
0.3200
 Exact Solution :
>> y0 = (1/4)*(2*t-ones(size(t))+5*exp(-2*t))
y0 =
1.0000
0.6879
0.5117
0.4265

>> t0=0:0.01:0.6;
>> y0 = (1/4)*(2*t0-ones(size(t0))+5*exp(-2*t0));
>> plot(t0,y0,t,y)

Improvements of Euler’s Method

Heun’s Method: estimate slope from derivatives at the beginning
and at the end of the interval.

y Euler:
y0i = yi-1 + hk1 slope:
k2 = f(ti, y0i)

slope:
k1 = f(ti-1, yi-1)

h
t
ti-1 ti
 k1 + k 2 f (t i −1, y i −1 ) + f (t i , y i0 )
Average slope = =
2 2

y
 k + k2 
y i = y i −1 + h  1 
 2 

Use slope:
(k1 + k2)/2

h
t
ti-1 ti

Heun’s Method
dy k1 = f (ti −1 , yi −1 )
= f (t , y ) y i0
dt
k2 = f (ti −1 + h, yi −1 + hk1 )
Euler
k +k
yi = yi −1 + h 1 2
2
Heun
True
yi-1

ti-1 ti
Predictor-Corrector Approach

Heun’s Method:
k1 = f (ti −1 , yi −1 )
Euler’s Method:
k2 = f (ti −1 + h, yi −1 + hk1 ) yi = yi −1 + h f (ti −1 , yi −1 )
k1 + k2
yi = yi −1 + h
2

Predictor: yi0 = yi −1 + h f (ti −1 , yi −1 )

f (ti −1 , yi −1 ) + f (ti , yi0 )

Corrector: yi = yi −1 + h
2

Iterating the Corrector of Heun’s Method

To improve the estimation

f (ti −1 , yi −1 ) + f (ti , y ) 0

yi yi −1 + h i

2
Example: Use Heun’s method to integrate

y ′ = 4e 0.8 x − 0.5 y , y (0) = 2

From x = 0 to 4, step size = 1

Analytical solution: y=
1.3
(
4 0.8 x −0.5 x
e −e ) + 2e −0.5 x

Predictor, yi0 = yi −1 + h f (ti −1 , yi −1 )

y10 = 2 + 1 (4e 0 − 0.5(2)) = 5

f (ti −1 , yi −1 ) + f (ti , yi0 )

Corrector, yi = yi −1 + h
2
(4e 0 − 0.5(2)) + (4e 0.8(1) − 0.5(5))
y1 = 2 + 1 = 6.7011
2

True value: y =
1.3
(
4 0.8(1) −0.5(1)
e −e ) + 2e −0.5(1) = 6.1946

6.1946 − 6.7011
Relative error, Et = ×100% = 8.18%
6.1946

(4e 0 − 0.5(2)) + (4e 0.8(1) − 0.5(6.7011))

2 nd
Corrector, y1 = 2 + 1
2
= 6.2758, Et = 1.31%

(4e 0 − 0.5(2)) + (4e 0.8(1) − 0.5(6.2758))

rd
3 Corrector, y1 = 2 + 1
2
= 6.3821, Et = 3.03%
Error may increases for large step sizes.
Iteration using up-arrow in MATLAB

>> y = 2+1*(4*exp(0)-0.5*2) % Predictor of y1

y = 5

>> y = 2+(1/2)*((4*exp(0)-0.5*2)+(4*exp(0.8*1)-0.5*y))
y = 6.7011 % 1st Corrector of y1

Press and Enter

y = 6.7011 % 2nd Corrector of y1

y = 6.2758 % 3rd Corrector of y1

y = 6.3821 % 4th Corrector of y1

y = 6.3555 % 5th Corrector of y1

y = 6.3609 % until no change

MATLAB’s Implementation
func1.m
function dydx = func1(x,y)
dydx = 4*exp(0.8*x)-0.5*y;

odeHeun.m
function [x,y] = odeHeun(diffeq,xn,h,y0,iter)
% Input: iter = number of corrector iterations

x = (0:h:xn)’;
n = length(x);
y = y0*ones(n,1);
for i=2:n
y(i) = y(i-1)+h*feval(diffeq,x(i-1),y(i-1));
for j=1:iter
y(i) = y(i-1)+h*(feval(diffeq,x(i-1),y(i-1)) ...
+ feval(diffeq,x(i),y(i)))/2;
end
end
 For x = 0 to 4, step size = 1, y(0) = 2, Iteration = 15

>> [x,y] = odeHeun(‘func1’,4,1,2,15)

x =
0
1
2
3
4
y = y_true = Et =
2.0000 2.0000 0.00
6.3609 6.1946 2.68
15.3022 14.8439 3.09
34.7433 33.6772 3.17
77.7351 75.3390 3.18

Compare with Euler’s Method:

>> [x,y] = odeEuler(‘func1’,4,1,2)

x =
0
1
2
3
4
y = Et =
2.0000 0.00
5.0000 19.28
11.4022 23.19
25.5132 24.24
56.8493 24.54
 Comparison of True Solution with Euler’s and Heun’s Method

>> xtrue = (0:0.1:4)’;

>> ytrue = (4/1.3)*(exp(0.8*xtrue)-exp(-0.5*xtrue))...
+ 2*exp(-0.5*xtrue);
>> [xeuler,yeuler] = odeEuler(‘func1’,4,1,2);
>> [xheun,yheun] = odeHeun(‘func1’,4,1,2,15);
>> plot(xtrue,ytrue,xeuler,yeuler,’o’,xheun,yheun,’+’)
80
True
Euler
60
Heun 15

40
y

20

0
0 1 2 3 4
x

RUNGE-KUTTA (RK) METHODS

General Formula using Weighted Average of slopes
m
yi = yi −1 + ∑ γ l kl
l =1
For Heun’s method γ1 = γ2 = 0.5
m
The weights must satisfy ∑γ
l =1
l =1

Second-Order RK (Ralston’s) Method:

1 2 
yi = yi −1 + h  k1 + k 2 
3 3 
k1 = f ( xi −1 , yi −1 )
 3 3 
k2 = f  xi −1 + h, yi −1 + k1h 
 4 4 
 Third-Order Runge-Kutta Methods

h
yi = yi −1 + ( k1 + 4k2 + k3 )
6
where
k1 = f ( xi −1 , yi −1 )
 1 1 
k2 = f  xi −1 + h, yi −1 + k1h 
 2 2 
k3 = f ( xi −1 + h, yi −1 − k1h + 2k 2 h )

Fouth-Order Runge-Kutta Methods

The most popular RK methods
h
Classical 4th-order RK: yi = yi −1 + ( k1 + 2k2 + 2k3 + k4 )
6
where
k1 = f ( xi −1 , yi −1 )
 1 1 
k2 = f  xi −1 + h, yi −1 + k1h 
 2 2 
 1 1 
k3 = f  xi −1 + h, yi −1 + k2 h 
 2 2 
k4 = f ( xi −1 + h, yi −1 + k3 h )
Example: Use 4th-order RK method to integrate

y ′ = 4e 0.8 x − 0.5 y , y (0) = 2

From x = 0 to 4, step size = 1
Step 1: x0 = 0, y0 = 2

k1 = 4e0 − 0.5(2) = 3
1
k2 = 4e0.8(0.5) − 0.5(2 + × 3 × 1) = 4.2173
2
1
k3 = 4e0.8(0.5) − 0.5(2 + × 4.2173 ×1) = 3.9130
2
k4 = 4e0.8(1) − 0.5(2 + 3.9130 ×1) = 5.9457

1
y1 = 2 + (3 + 2 × 4.2173 + 2 × 3.9130 + 5.9457 )
6
= 6.2011

True value: y =
1.3
(
4 0.8(1) −0.5(1)
e −e ) + 2e −0.5(1) = 6.1946

6.1946 − 6.2010
Relative error, Et = ×100% = 0.1038%
6.1946
MATLAB’s Implementation
odeRK4.m
function [x,y] = odeRK4(diffeq,xn,h,y0)

x = (0:h:xn)’;
n = length(x);
y = y0*ones(n,1);
for i=2:n
k1 = feval(diffeq,x(i-1),y(i-1));
k2 = feval(diffeq,x(i-1)+h/2,y(i-1)+k1*h/2);
k3 = feval(diffeq,x(i-1)+h/2,y(i-1)+k2*h/2);
k4 = feval(diffeq,x(i-1)+h,y(i-1)+k3*h);
y(i) = y(i-1)+h/6*(k1+2*k2+2*k3+k4);
end

For x = 0 to 4, step size = 1, y(0) = 2, Iteration = 15

>> [x,y] = odeRK4(‘func1’,4,1,2)

x =
0
1
2
3
4
y = y_true = Et(%) =
2.0000 2.0000 0.00
6.2010 6.1946 0.1038
14.8625 14.8439 0.1250
33.7213 33.6772 0.1309
75.4392 75.3390 0.1328
Adaptive Stepsize Algorithms
MATLAB’s ode23 and ode45 Routines
ode23 use second- and third-order RK simultaneously
ode45 use fourth- and fifth-order RK simultaneously

Example: y ′ = 4e 0.8 x − 0.5 y , y (0) = 2 func1.m

For x = 0 to 4, y(0) = 2
>> x0 = 0; xn = 4; y0 = 2;
>> [x45,y45] = ode45(‘func1’,[x0,xn],y0);
>> xtrue = (0:0.1:4)’;
>> ytrue = (4/1.3)*(exp(0.8*xtrue)...
- exp(-0.5*xtrue))+ 2*exp(-0.5*xtrue);
>> plot(xtrue,ytrue,x45,y45)
 Numerical Methods for Civil Engineers

Lecture 11 OPTIMIZATION
- Basic Ideas
- One-dimensional Unconstrained Optimization
- Golden-Section Search
- Quadratic Interpolation
- Newton’s Method
- Multidimensional Unconstrained Optimization
- Direct Methods
- Gradient Methods

Suranaree
UNIVERSITY OF TECHNOLOGY Mongkol JIRAVACHARADET
NAKORNRATCHSIMA School of Civil Engineering
BASIC IDEAS

f (x) f ′( x) = 0
Maximum
f ′′( x) < 0

f ( x) = 0
Root Root
0 x
Root

f ′( x) = 0
Minimum
f ′′( x) > 0
Examples of Optimization Problems in Engineering

- Design aircraft for minimum weight and maximum strength.

- Optimal trajectories of space vehicles.
- Design civil engineering structures for minimum cost.
- Predict structural behavior by minimizing potential energy.
- Material-cutting strategy for minimum cost.
- Design pump and heat transfer equipment for maximum efficiency.
- Shortest route of salesperson visiting various cities during one sales trip.
- Optimal pipeline network.
- Optimal planning and scheduling.
- Statistical analysis and models with minimum error.
- Minimize waiting and idling times.
One-dimensional Unconstrained Optimization
Global & Local Optima

f (x) Global
maximum
Local
maximum

0 x

Local
Global minimum
minimum
GOLDEN-SECTION SEARCH

- Similar to Bisection root finding method

- Define interval that contains single answer

- Need 3 points to detect maximum or minimum

- Pick 4th points then choose the first or last three points.
Intermediate
Point
Choice of Intermediate Points

Maximum
f (x)

0 x
xL xU

First L0
iteration L0 = L1 + L2
L1 L2
Second
iteration L1 L2
L2 =
L0 L1
 L1 L2
From two condition, =
L1 + L2 L1
Set R = L2 / L1,
1
1+ R = or R2 + R −1 = 0
R
Solving for positive root,

−1 + 1 − 4(−1) 5 −1
R= = = 0.61803…
2 2
R = 0.61803 = Golden ratio since it allows optima to be
found efficiently
The Parthenon in Athens

0.61803 x

x
This golden ratio were considered aesthetically pleasing
by the Greeks.
Initial Step of the Golden-section Search
Eliminate
f (x) f (x1) Maximum
f (x2)

d
d
xL xU x
x2 x1
1) Guess initial bracket xL and xU
2) Choose two interior points x1 and x2 according to golden ratio,
5 −1
d= ( xU − xL )
2
x1 = xL + d , x2 = xU − d
3) If f (x1) > f (x2), eliminate [ xL , x2 ] and set x2 = xL for next round
Next Step to Complete the Algorithm
f (x) Maximum

xU x
xL x2 x1

Old x2 Old x1
4) Only new x1 need to be determined,

5 −1
x1 = xL + ( xU − xL )
2
Example: Golden-Section Search to find maximum
x2
f ( x) = 2sin x − , xL = 0 and xU = 4
10
Solution: (1) Create two interior points
5 −1
d= (4 − 0) = 2.472
2
x1 = 0 + 2.472 = 2.472
x2 = 4 − 2.472 = 1.528
(2) Evaluate function at interior points,
2.4722
f ( x1 ) = f (2.472) = 2sin(2.472) − = 0.63
10
f ( x2 ) = f (1.528) = 1.765
(3) Because f(x2) > f(x1), eliminate upper part,
New xU = x1 = 2.472 OLD
New x1 = x2 = 1.528
xL x2 x1 xU
(4) Compute new x2
5 −1 NEW
d= (2.472 − 0) = 1.528
2
x2 = 2.472 − 1.528 = 0.944 xL x2 x1 xU

(5) Evaluate function at x2

New f ( x1 ) = Old f ( x2 ) = 1.765
New f ( x2 ) = f (0.944) = 1.531
Because f(x1) > f(x2), eliminate lower part, . . .
Tabulated results:

i xL x2 f(x2) x1 f(x1) xU d

1 0 1.5279 1.7647 2.4721 0.6300 4.0000 2.4721

2 0 0.9443 1.5310 1.5279 1.7647 2.4721 1.5279
3 0.9443 1.5279 1.7647 1.8885 1.5432 2.4721 0.9443
4 0.9443 1.3050 1.7595 1.5279 1.7647 1.8885 0.5836
5 1.3050 1.5279 1.7647 1.6656 1.7136 1.8885 0.3607
6 1.3050 1.4427 1.7755 1.5279 1.7647 1.6656 0.2229
7 1.3050 1.3901 1.7742 1.4427 1.7755 1.5279 0.1378
8 1.3901 1.4427 1.7755 1.4752 1.7732 1.5279 0.0851
Quadratic Interpolation
2nd polynomial provides good approximation near optimum
True maximum Quadratic
f (x) approximation
of maximum

x0 x1 x3 x2 x

f ( x0 )( x12 − x22 ) + f ( x1 )( x22 − x02 ) + f ( x2 )( x02 − x12 )

x3 =
2 f ( x0 )( x1 − x2 ) + 2 f ( x1 )( x2 − x0 ) + 2 f ( x2 )( x0 − x1 )
Example: Quadratic Interpolation to find maximum
x2
f ( x) = 2sin x − , x0 = 0, x1 =1 and x2 = 4
10
Solution: (1) Evaluate function values,
x0 = 0 f ( x0 ) = 0
x1 = 1 f ( x1 ) = 1.5829
x2 = 4 f ( x2 ) = −3.1136

0(12 − 42 ) + 1.5829(42 − 02 ) + ( −3.1136)(02 − 12 )

x3 = = 1.5055
2(0)(1 − 4) + 2(1.5829)(4 − 0) + 2( −3.1136)(0 − 1)
 (2) Because f(1.5055) = 1.7691, employ golden-section search,
eliminate lower guess x0 = 0,

x0 = 1 f ( x0 ) = 1.5829
x1 = 1.5055 f ( x1 ) = 1.7691
x2 = 4 f ( x2 ) = −3.1136

1.5829(1.50552 − 42 ) + 1.7691(42 − 12 ) + ( −3.1136)(12 − 1.50552 )

x3 =
2(1.5829)(1.5055 − 4) + 2(1.7691)(4 − 1) + 2( −3.1136)(1 − 1.5055)
= 1.4903

After 5 iterations, x = 1.4276 gives maximum value of 1.7757

Newton’s Method

f ( xi )
f ( x) = 0 → xi +1 = xi −
f ′( xi )

Define new function, g ( x ) = f ′( x )

At optimal point x*, f ′( x*) = g ( x*) = 0

g ( xi ) f ′( xi )
g ( x) = 0 → xi +1 = xi − → xi +1 = xi −
g ′( xi ) f ′′( xi )
Example: Use Newton’s method to find maximum of
x2
f ( x) = 2sin x − , Initial guess: x0 = 2.5
10
Solution: (1) Evaluate 1st and 2nd derivatives of the function,
x
f ′( x) = 2 cos x −
5
1
f ′′( x) = −2sin x −
5
(2) Substitute into Newton’s formula,

2 cos xi − xi / 5
xi +1 = xi −
−2sin xi − 1/ 5
(3) Substituting initial guess,
2 cos 2.5 − 2.5 / 5
x1 = 2.5 − = 0.99508, f (0.99508) = 1.57859
−2sin 2.5 − 1/ 5

(4) Second iteration,

2 cos 0.995 − 0.995 / 5
x2 = 0.995 − = 1.46901, f (1.46901) = 1.77385
−2sin 0.995 − 1/ 5

i x f(x)
0 2.5 0.57194
1 0.99508 1.57859
2 1.46901 1.77385
3 1.42764 1.77573
4 1.42755 1.77573

After 4 iterations, result converges rapidly to the true value.

Multidimensional Unconstrained Optimization
Example: 2-D Topographic Map of 3-D Mountain

y Line of constant f

x
2D searches: Ascending a mountain (maximum) or
Descending into a valley (minimization)
DIRECT METHODS
Random Search
Univariate and Pattern Searches

GRADIENT METHODS

Gradients and Hessians

Steepest Ascent Method
Random Search
Repeatedly evaluates function at randomly selected points

Example: f ( x , y ) = y − x − 2 x 2 − 2 xy − y 2

Domain boundaries: x = -2 to 2 and y = 1 to 3

Write an M-file objfun1.m
function f = objfun1(x,y)
f = y-x-2*x^2-2*x*y-y^2;

>> rand % Uniformly distributed random numbers

% on the interval (0.0,1.0).
ans =
0.2311
Random search algorithm: rndsrch.m

function [maxx,maxy,maxf] = rndsrch(objfun,xmin,…

xmax,ymin,ymax)

iter = 100;
maxf = -inf;

for i = 1:iter
x = xmin + (xmax - xmin)*rand;
y = ymin + (ymax - ymin)*rand;
fn = feval(objfun,x,y);
if fn > maxf
maxf = fn;
maxx = x;
maxy = y;
end
end
>>[maxx,maxy,maxf]=rndsrch('objfun1',-2,2,1,3)

iter maxx maxy maxf

50 -0.9024 1.5913 1.2048
100 -0.8050 1.3092 1.2120
200 -1.1409 1.7708 1.2133
500 -1.0006 1.4983 1.2500
1000 -1.0253 1.5030 1.2489

TRUE -1.0000 1.5000 1.2500

Univariate and Pattern Searches
- More efficient and still no require derivative evaluation
- Change one variable at time while others constant
y

Pattern directions
4 3

1
2 x
From (1) move along x axis with y constant to max at (2)
Powell’s Method
Use pattern directions to find optimum efficiently
- Pt. 1 and 2 are obtained by 1-D
y
search from different starting pts.

- Line formed by 1 and 2 will be

directed toward maximum
2 Conjugate direction
1

x
GRADIENT METHODS
Use derivative to locate optima

Gradient : ∆y

∆f
∆h
∆x
∆f
= Slope along axis h
∆h
∂f ∂f
∇f = i + j = Steepest direction
∂x ∂y

The vector is called “del f “ = directional derivative of f (x , y)

Hessian
To determine whether maximum, minimum or saddle

2
∂ f ∂ f  ∂ f 
2 2 2
H = 2 − 
∂x ∂y  ∂x ∂y 
2

∂2 f
• If H > 0 and > 0 → local minimum
∂x 2

∂2 f
• If H > 0 and < 0 → local maximum
∂x 2

• If H < 0 → saddle point

Steepest Ascent Method
Climbing a hill with maximum slope

Repeat the process until

the maximum is reached

1 Follow steepest ascent path

until f (x, y) stop increasing

0 : Starting point (x0, y0)

x

Problems: 1) Determine the best direction of steepest ascent

2) Determine the best value along that search direction
 Example: Steepest ascent method

f ( x, y ) = xy 2 , Starting point (2, 2)

>> contour(x,y,fxy)
4
x=0:0.1:4;
y=0:0.1:4; 3
fxy=zeros(length(y),length(x));
for i=1:length(y)
2

y
for j=1:length(x)
fxy(i,j)=x(j)*y(i)^2;
1
end
end
0
0 1 2 3 4
x
>> surf(x,y,fxy)
>> mesh(x,y,fxy)
Determine elevation: f (2, 2) = 2 × 22 = 8

Evaluate partial derivatives,

∂f
= y 2 = 22 = 4
∂x
∂f
= 2 xy = 2(2)(2) = 8
∂y

Determine the gradient, ∇ f = 4i + 8 j 63.40

8
−1
Direction, θ = tan   = 63.40 relative to x axis
4
Magnitude of ∇f is 42 + 82 = 8.944

We will gain 8.944 units of elevation rise for a unit distance

advanced along this steepest path.

63.4o
y 2

0
0 1 2 3 4
x
 Relationship between steepest direction and x-y coordinate

Starting at x0, y0 any point in the

y ∇ f = 4i + 8 j
gradient direction can be computed by
10 ∂f
x = x0 + h
1
h= ∂x
∂f
y = y0 + h
∂y
2
0

Then, go along h axis to find maximum

h=

x
2 6 ∂f ∂f
Define g (h) = f ( x0 + h, y0 + h)
∂x ∂y
Set g ′(h*) = 0 to find h *
Example: Developing 1-D function along a gradient direction

f ( x, y ) = 2 xy + 2 x − x 2 − 2 y 2 , Starting point (−1,1)

Partial derivative at starting point:
∂f
= 2 y + 2 − 2 x = 2(1) + 2 − 2(−1) = 6
∂x
∂f
= 2 x − 4 y = 2(−1) − 4(1) = −6
∂y
Function along h axis:
∂f ∂f
f ( x0 + h, y0 + h) = f (−1 + 6h,1 − 6h)
∂x ∂y
= 2(−1 + 6h)(1 − 6h) + 2(−1 + 6h) − (−1 + 6h) 2 − 2(1 − 6h) 2
g (h) = −180h 2 + 72h − 7
Locate the maximum,

g ′(h*) = −360h * +72 = 0

h* = 0.2
Locate (x, y) coordinates according to h = 0.2,

x = −1 + 6(0.2) = 0.2

y = 1 − 6(0.2) = −0.2

Set new starting point to (0.2, -0.2) and repeat the process.
 MATLAB Optimization Toolbox
fminunc : the unconstrained optimization function

f ( x, y ) = e x (4 x 2 + 2 y 2 + 4 xy + 2 y + 1), Starting point (−1, 1)

Write an M-file objfun2.m

function f = objfun2(x,y)
f = exp(x)*(4x^2+2*y^2+4*x*y+2*y+1);

>>x0 = [-1,1]; % Starting point

>>options = optimset(‘LargeScale’,’off’);
>>[x,fval,exitflag,output]=fminunc(‘objfun2’,x0,options);

x =
0.5000 -1.0000
 Numerical Methods for Civil Engineers

Lecture 12 Constrained Optimization

- Linear Programming

- Nonlinear Constrained Optimization

- MATLAB’s fmincon Function

Suranaree
UNIVERSITY OF TECHNOLOGY Mongkol JIRAVACHARADET
NAKORNRATCHSIMA School of Civil Engineering
LINEAR PROGRAMMING
Standard Form

Objective Function: Maximize Z = c1x1+ c2x2+ . . . + cnxn

where cj = payoff of jth activity
xj = magnitude of jth activity
Z = total payoff due to all activities

Constraints: ai1x1 + ai2x2 + . . . + ainxn
bi

where aij = amount of ith resource used by jth activity

bi = amount of ith resource available
Example: Setting Up the Linear Programming Problem
Producing product with limited resources: Gas processing plant

Product Resource
Resource Regular Premium Availability

Raw gas 7 m3/ton 11 m3/ton 77 m3/week

Production time 10 hr/ton 8 hr/ton 80 hr/week
Storage 9 ton 6 ton
Profit 150 / ton 175 / ton

Set x1 = amount of regular gas produced

x2 = amount of premium gas produced
Total profit = 150x1 + 175x2

Maximize Z = 150x1 + 175x2 Objective function

Total gas used = 7x1 + 11x2

cannot exceed 77 m3/week
7x1 + 11x2
77 Constraints

10x1 + 8x2
80

x1
9 x2
6

x1 , x2  0
Graphical Solution Maximize Z = 150x1 + 175x2

x2 7 x1 + 11x2 ≤ 77 (1)
10x1+8x2 = 80 10 x1 + 8 x2 ≤ 80 (2)
10x1+8x2
80 0 ≤ x1 ≤ 9 (3)
7
0 ≤ x2 ≤ 6 (4)
7x1+11x2 = 77
6

7x1+11x2
77

x1
0
9 11
Increasing Objective Function From Z = 150x1 + 175x2
For Z = 0, 150x1 + 175x2 = 0
x2

Unique solution
Z = 1400

x1
0
Z=0 Z = 600
Other Possible Outcomes
x2
Objective function parallel
to constraint.
Alternate Solution

0 x1
No Feasible Solution

x2

0 x1
Unbounded Problems

x2

0 x1
Simplex Method
Assumption: optimal solution will be an extreme point

x2
Extreme points: corner points
where two constraints meet

Z = 1400
Objective function
is LINEAR

x1
0 Z=0 Z = 600
Slack Variables
Reformulate constraint inequalities as equalities
slack variable
7 x1 + 11x2 ≤ 77 → 7 x1 + 11x2 + S1 = 77

Resource constraint How much slack of the resource

is available ?

IF . . .

S1 = + Surplus resource is not fully used.

S1 = - Exceeded the constraint.

S1 = 0 Exactly meet the constraint. All resource is used.

Fully Augmented Version

Maximize Z = 150x1 + 175x2

subject to

7 x1 + 11x2 ≤ 77 7 x1 + 11x2 + S1 = 77
10 x1 + 8 x2 ≤ 80 10 x1 + 8 x2 + S2 = 80
0 ≤ x1 ≤ 9 x1 + S3 =9
0 ≤ x2 ≤ 6 x2 + S4 =6
x1 , x2 , S1 , S 2 , S3 , S4 ≥ 0

Linear algebraic equations

solving 4 equations with 6 unknowns
 Algebraic Solution
Underspecified : more unknowns (6) than equations (4)
x2 x1 = 9, S3 = 0
Extreme Point Zero Variables
10x1 + 8x2 = 80
S2 = 0 A x1, x2
x2 = 6, S4 = 0 B x2, S2
6 C S1, S2
E D
D S1, S4
C E x1, S4
7x1 + 11x2 = 77 @ Extreme point :
S1 = 0
A 4 equations with 4 unknowns
B
0 The problem can be solved.
x1
9
For point E : Setting x1 = S4 = 0 nonbasic variables
The standard form reduced to

0
7 x1 + 11x2 + S1 = 77 11x2 + S1 = 77
0
10 x1 + 8 x2 + S2 = 80 8 x2 + S2 = 80
0
x1 + S3 =9 S3 =9
0
x2 + S4 =6 x2 =6

which can be solved for

x1 x2 S1 S2 S3 S4

0 6 11 32 9 0

basic variables
 Simplex Method Implementation S1 = 77
Start at point A : Setting x1 = x2 = 0 S2 = 80
S3 = 9
S4 = 6

Tableau: Z - 150x1 - 175x2 - 0S1 - 0S2 - 0S3 - 0S4 = 0

Basic Z x1 x2 S1 S2 S3 S4 Solution Intercept

Z 1 -150 -175 0 0 0 0 0
S1 0 7 11 1 0 0 0 77 11
S2 0 10 8 0 1 0 0 80 8
S3 0 1 0 0 0 1 0 9 9
S4 0 0 1 0 0 0 1 6 ∞
Graphical Depiction of Simplex Method

x2 Change x1 to basic variable (entering variable)

Move from A horizontally
S2
Check intercept:
6 S4
E D Intercept = Solution
S3 (for each S) x1 coeff.
C

S1 S2 give min. intercept = 8

A B F
0 x1
8 11

Therefore, change S2 to nonbasic(leaving) variable.

Move to point B : Setting x2 = S2 = 0 7 x1 + S1 = 77
10 x1 = 80
x1 + S3 = 9
S4 = 6
Pivot row S2: Dividing row by 10 and replacing S2 by x1
Basic Z x1 x2 S1 S2 S3 S4 Solution
Z 1 -150 -175 0 0 0 0 0
S1 0 7 11 1 0 0 0 77 x -150
x1 0 1 0.8 0 0.1 0 0 8
S3 0 1 0 0 0 1 0 9
S4 0 0 1 0 0 0 1 6

Z 1 0 -55 0 15 0 0 1200
Perform same operation on the remaining rows,
Basic Z x1 x2 S1 S2 S3 S4 Solution Intercept
Z 1 0 -55 0 15 0 0 1200
S1 0 0 5.4 1 -0.7 0 0 21 3.889
x1 0 1 0.8 0 0.1 0 0 8 10
S3 0 0 -0.8 0 -0.1 1 0 1 -1.25
S4 0 0 1 0 0 0 1 6 6

- From A to B, objective function has increased to 1200

- S1 give min. positive intercept = 3.889 is entering variable
- Move from B to C
Final Tableau:
Basic Z x1 x2 S1 S2 S3 S4 Solution
Z 1 0 0 10.19 7.87 0 0 1414
S1 0 0 1 0.19 -0.13 0 0 3.89
x1 0 1 0 -0.15 0.20 0 0 4.89
S3 0 0 0 0.15 -0.20 1 0 4.11
S4 0 0 0 -0.19 0.13 0 1 2.11

- No negative coeff. in objective func. row, No more increasing.

Final solution: x1 = 3.89, x2 = 4.89

Z = 1414
MATLAB’s fmincon Function

f ( x) = e x1 ( 4 x12 + 2 x22 + 4 x1 x2 + 2 x2 + 1)

subject to the constraints: x1 x2 − x1 − x2 ≤ −1.5

x1 x2 ≥ −10

rewrite the constraints in the form c(x)
0 :

x1 x2 − x1 − x2 + 1.5 ≤ 0

− x1 x2 − 10 ≤ 0
 Write M-file confun.m for the constraints

function [c, ceq] = confun(x)

% nonlinear inequality constraints
c = [1.5 + x(1)*x(2) - x(1) - x(2);
-x(1)*x(2) - 10];
% nonlinear equality constraints
ceq = [];

> x0 = [-1,1];
> options = optimset(‘LargeScale’,’off’);
> [x, fval] = . . .
fmincon(‘objfun’,x0,[],[],[],[],[],[],’confun’,options)

x =
-9.5474 1.0474
fval =
0.0236