Interpolation and approximation

Given a set of points {(xj , yj ), j = 1, · · · , n}, and given a functional

⃗ nd “best” β
form f (x; β), ⃗ so that f (x; β) ⃗ “models” the data.

Interpolation Approximation

⃗ = yj
f (xj ; β) ⃗ + εj = yj
f (xj ; β)
εj is “noise”, E(εj ) = 0

1/19
Least squares approximation

E(εj ) = 0
E(ε2j ) = σj2
σj ≈ const

n 
∑ 2
⃗ =  ⃗  ⇒ min
2
χ (β) yj − f (xj ; β)
j=1

2/19
Weighted least squares

E(εj ) = 0
E(ε2j ) = σj2

If σj are signi cantly different (heteroscedasticity)

 2
 ⃗ 
∑
n yj − f (xj ; β)
⃗ =
χ2 (β) ⇒ min
j=1
σj2

3/19
Least absolute deviations

Also known as L1 regression:

n 
∑ 
⃗ =  ⃗  ⇒ min
S(β) yj − f (xj ; β)
j=1

4/19
Total least squares

Also known as Orthogonal distance regression: minimize the sum of

squares of orthogonal distances from observations to the curve.

Can be more appropriate e.g. if both variables, x and y have

measurement errors.
5/19
Linear least squares

6/19
Least squares approximation

E(εj ) = 0
E(ε2j ) = σj2
σj ≈ const

n 
∑ 2
⃗ =  ⃗  ⇒ min
2
χ (β) yj − f (xj ; β)
j=1

7/19
Linear least squares

Consider an ordinary least squares problem,

n 
∑ 2
⃗ =  ⃗ 
ξ(β)  j
y − f (x j ; β)  ⇒ min
j=1

⃗ a linear combination
Let the model, f (x; β), is a linear function of β,
of m basis functions, φk (x)

∑
m
⃗ =
f (x; β) βk φk (x)
j=1

Typically, want m < n.

8/19
Linear least squares

⃗ a linear combination
Let the model, f (x; β), is a linear function of β,
of m basis functions, φk (x)

∑
m
⃗ =
f (x; β) βk φk (x)
j=1

The basis functions need not be linear:

▶ polynomials: φk (x) = xk
▶ Fourier series: φk (x) = eisk x
▶ φk (x) = xk log x
▶ ...

9/19
Linear least squares

We minimize with respect to β⃗

∑
n
⃗ =
ξ(β) |zj |2
j=1

where (j = 1, . . . , n)
( )
zj = yj − β1 φ1 (xj ) + β2 φ2 (xj ) + · · · + βm φm (xj )

Which is eqivalent to
2
ξ(β) = y − Aβ⃗ 2

with y = (y1 , · · · , yn )T and Akj = φk (xj ).

10/19
Design matrix

The design matrix A is an n × m matrix

 
φ1 ( ) φ2 ( ) · · · φm ( )
 φ1 ( ) φ2 ( ) · · · φm ( )
A= 


···
φ1 ( ) φ2 ( ) · · · φm ( )

The dimensions of the design matrix is # of observations × # of

parameters

11/19
Design matrix

The design matrix A is an n × m matrix

 
φ1 (x1 ) φ2 (x1 ) · · · φm (x1 )
 φ1 (x2 ) φ2 (x2 ) · · · φm (x2 ) 
A= 


···
φ1 (xn ) φ2 (xn ) · · · φm (xn )

The dimensions of the design matrix is # of observations × # of

parameters

11/19
Example: straight line t

The model is
⃗ = β1 + β2 x
f (x; β)

m=2: φ1 (x) = 1 , φ2 (x) = x

and the design matrix is
 
1 x1
1 x2 
 
A = . . 
 .. .. 
1 xn

12/19
Linear least squares

Normal equations

13/19
Linear least squares: normal equations

To minimize the quadratic form

⃗ = y − Aβ⃗ 2
ξ(β) 2

set the derivatives to zero,

∂ ⃗ = 0,
ξ(β) j = 1, · · · , m
∂βk

And obtain the normal equations:

⃗ = AT y
AT A β

14/19
Linear least squares: normal equations

Normal equations
⃗ = AT y
AT A β
give a formal solution of a linear least squares problem.

However, ( )
cond AT A = [cond A]2
so that typically the system of normal equations is very poorly
conditioned.

15/19
Linear least squares

QR factorization of the design matrix

16/19
Linear least squares: QR factorization

Recall that a matrix A can be factorized into

A = QR

where Q is orthogonal (QT Q = 1) and R is upper triangular.

Since a design matrix is thin and tall (m < n), last n − m rows of R
are zero:
[ ]
R1
A=Q
0
where dim R1 = m

17/19
Linear least squares: QR factorization

Since the 2-norm of a vector is invariant under a rotation by an

orthogonal matrix Q, we rotate the residual y − Aβ⃗
2 ( ) 2

ξ(β) = y − Aβ⃗ = QT y − Aβ⃗

[ ] 2
T R1 ⃗

= Q y − β
0

Next, write [ ]
T f
Q y=
r
with dim f = m and dim r = n − m.

18/19
Linear least squares: QR factorization

This way, 2
⃗ =
ξ(β)

f − R1 β⃗ + ∥r∥
2

⃗ satis es
And the minimum of ξ(β)

R1 β⃗ = f

19/19
Linear least squares: QR factorization

This way, 2
⃗ =
ξ(β)

f − R1 β⃗ + ∥r∥
2

⃗ satis es
And the minimum of ξ(β)

R1 β⃗ = f

Algorithm
▶ Factorize the design matrix A = QR
▶ Rotate y → QT y (only need m rows ⇒ thin QR)
▶ Solve R1 β⃗ = f by back substitution.

19/19