Smoothing Techniques in Image

Processing

Prof. PhD. Vasile Gui

Polytechnic University
of Timisoara
Content

 Introduction
 Briefreview of linear operators
 Linear image smoothing techniques
 Nonlinear image smoothing techniques
Introduction

Why do we need image

smoothing?

What is “image” and what

is “noise”?

– Frequency spectrum
– Statistical properties
Brief Review of Linear Operators
[Pratt 1991]
 Generalized 2D linear operator
M −1 N −1
g (m, n) = ∑∑ O ( j , k ; m, n) f ( j , k )
j =0 k =0
 Separable linear operator:
O( j , k ; m, n) = OR ( j; m)OC (k ; n)
M −1 N −1
g ( m, n) = ∑O R ( j ; m) ∑OC ( k ; n) f ( j , k )
j =0 k =0
 Space invariant operator:
O( j , k ; m, n) = O(m − j , n − k ) = H (m − j , n − k )
M −1 N −1
g ( m, n) = ∑∑h( m − j , n − k ) f ( j , k )
j =0 k =0

Convolution sum
Brief Review of Linear Operators
h22 h21 h20
L-1
h12 h11 h10
 Geometrical
2 L-1
H h02 h01 h0 2 h22 h21 h20
0
h12 h11 h10

G
L-1
2
L-1 h22 h21 h20
h02 h01 h0
0
interpretation
of 2D
2 h12 h11 h10
h02 h01 h0

F
0
N-
L+1
convolution
N

N+L-1

h22 h21 h20

h12 h11 h10 h22 h21 h20
h02 h01 h0 h12 h11 h10
0
h02 h01 h0
Brief Review of Linear Operators

 Matrix formulation
t = Af
 Unitary transform
−1
A =A *T
 Separable transform


T = A FAC is
Transform
T
R
invertible

f = A *Tt = A *T Af = If
Brief Review of Linear Operators

 Basis vectors are orthogonal

1, k = l
a a l = δ( k , l ) = 
*T
k
0, k ≠ l
 Inner product and energy are conserved

|| t || 2 = t *T t = (Af)*T (Af) =
 Unitary *T
Af = f *aTrotation
f *T Atransform: If = f *Tinf N-dimensional
=|| f || 2 vector space
Brief Review of Linear Operators

x2

ξ 2 ξ 1

x1

 2D vector space interpretation

of a unitary transform
Brief Review of Linear Operators

 DFT unitary matrix

W 0 W 0 W0 ⋅ ⋅ ⋅ W0
 0 
W W 1
W 2
⋅ ⋅ ⋅ W N −1 
W 0 W 2 W4 ⋅ ⋅ ⋅ W 2 ( N −1) 
1  
A=  ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
N 
⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
 
 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
W 0 W N W 2 ( N −1) ⋅ ⋅ ⋅ W ( N −1) 
2

i 2π 2π 2π
W N = exp{− } = cos( ) − i sin( )
N N N
Brief Review of Linear Operators

 1D DFT
1 N −1
i 2π
t (u ) =
N
∑n =0
f (n) exp{−
N
un}, u = 0,1,..., N − 1

 2D DFT M −1 N −1
1 u v
t (u , v) = ∑∑
MN m =0 n =0
f (m, n) exp{−i 2π( m + n)}
M N
u = 0,1,..., M − 1, v = 0,1,..., N − 1
Brief Review of Linear Operators

 Circular convolution theorem

N −1 N −1
g (m, n) = ∑∑ h(m − j , n − k ) f ( j , k )
j =0 k =0
1
t g ( u, v ) = t f ( u, v ) t h ( u, v )
N
 Periodicity of f,h and g with N are assumed
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

1 1  1 1
1 1  1 1 1
1  1
h= 2 = [1 1  1] *
L    1 L L 
  
1 1  1 1
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 Separable
 Canbe computed recursively, resulting in roughly 4
operations per pixel

m,n

m,n+1

− +

L pixels
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

Optimality properties
 Signal and additive white noise
N N N
1 1 1
g=
N
∑
k =1
( f k + nk ) =
N
∑
k =1
fk +
N
∑n .
k =1
k

N
1
z=
N
∑n .
k =1
k

 Noise varianceN is reducedN NN times

N
1 1
σ = E{z z} = 2 E{∑∑ nl n k } = 2
2
z
T
∑∑ E{n n } = l k
N k =1 l =1 N k =1 l =1

1 N N 1 2
2 ∑∑
δ (l , k ) σ 2
= σ
N k =1 l =1 N
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 Unknown constant signal plus noise

 Minimize MSE of the estimation g:
N
ε (g) = ∑ ( fk − g)2
2

k =1

∂ε 2 ( g )
=0
∂g
N
1
gˆ =
N
∑f
k =1
k
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 i.i.d. Gaussian signal with unknown mean.

( f − µ) 2
p ( f | µ) = ct. exp{− }
2σ 2

 Given the observed samples, maximize

N N
1
∏
k =1
p ( f k | µ ) = ct . exp{−
2σ 2
∑( f
k =1
k − µ) 2 }

 Optimal solution: arithmetic mean

Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 Frequency response:
Non-monotonically decreasing with frequency

1 1 1 1
1  1 1 
1 1 1 = [111]∗ 1 = hx ∗ hy
9  3 3 
1 1 1 1
( N −1)/ 2
i 2π
t (u) = hx (0) + ∑
n =− ( N −1)/ 2
hx (n) exp{−
N
un}

1 2 2π
t (u ) = + cos( u )
3 3 N
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 An example
           
 8 5 8 8 5 8  1 1 1  7 7 7 7 7 7 
1 
 8 5 8 8 5 8  ∗  1 1 1 =  7 7 7 7 7 7 
9
 8 5 8 8 5 8  1 1 1  7 7 7 7 7 7 
           

           
 8 5 8 5 8 5  1 1 1  6 7 6 7 6 7 
1 
 8 5 8 5 8 5  ∗  1 1 1 =  6 7 6 7 6 7 
9
 8 5 8 5 8 5  1 1 1  6 7 6 7 6 7 
           
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

 Image smoothed with

3× 3, 5× 5, 9× 9
and 11 × 11 box filters
Linear Image smoothing techniques
Box filters. Arithmetic mean L× L operator

Lena image filtered with

Original Lena image
5x5 box filter
Linear Image smoothing techniques
Binomial filters [Jahne 1995]

 Computes a weighted average of pixels in

the window
 Less blurring, less noise cleaning for the
same size
 The family of binomial filters can be defined
recursively
 The coefficients can be found from (1+x)n
Linear Image smoothing techniques
Binomial filters. 1D versions

1
b1 = [1 1]
2
1
b2 = [1 2 1] = b 1 * b 1
4
1
b = [1 4 6 4 1] = b 1 * b 1 * b 1 * b 1
4

16
1
b = [1 6 15 20 15 6 1] = b 1 * b 1 * b 1 * b 1 * b 1 * b 1
6

64
As size increases, the shape of the filter is closer to a Gaussian one
Linear Image smoothing techniques
Binomial filters. 2D versions

1  1 2 1 
1 1 1
b 2 = [1 2 1] * 2 = 2 4 2
4 4 16
1  1 2 1 

1  1 4 6 4 1 
 4 4 16 24 16 4
1 1   1  
b = [1 4 6 4 1] * 6 =
4
6 24 36 24 6
16 16   256  
4
   4 16 24 16 4 
1  1 4 6 4 1
Linear Image smoothing techniques
Binomial filters. Frequency response

1
[
b = 1 2 1
2

4 −
] t (u ) =
1 1 2π
+ cos( u )
2 2 N

b4 = [
1
16
1 4 6 4 1
−
] 1 1
2 2
2π
t (u ) = [ + cos( u )] 2
N

Monotonically decreasing
with frequency
Linear Image smoothing techniques
Binomial filters. Example

Original Lena image Lena image filtered Lena image filtered

with binomial 5x5 kernel with box filter 5x5
Linear Image smoothing techniques
Binomial and box filters. Edge blurring comparison

 Linear filters have to compromise smoothing

with edge blurring
Step edge

Result of size L
box filter

Size L binomial

L
Nonlinear image smoothing
The median filter [Pratt 1991]

 Block diagram
f(1)
f1
f(2)
f2

. . f(m)
. Order . select
.
. samples
f(N) median
fN

f (1) ≤ f ( 2 ) ≤ f (3) ≤  f ( N )

N +1
m= N is odd
2
Nonlinear image smoothing
The median filter

 Numerical example

3 7 8
2 3 7 6
4 6 7

2, 3, 3, 4, 6, 7, 7, 7, 8
Nonlinear image smoothing
The median filter

 Nonlinearity

median{ f1 + f2} ≠ median{ f1} + median{ f2}.

However:
median{ c f } = c median{ f },
median{ c + f } = c + median{ f }.

• The filter selects a sample from the window, does

not average
• Edges are better preserved than with liner filters
• Best suited for “salt and pepper” noise
Nonlinear image smoothing
The median filter

Noisy image 5x5 median filtered 5x5 box filter

Nonlinear image smoothing
The median filter

 Optimality
 Grey level plateau plus noise. Minimize sum of
absolute differences:
N
ε( g ) = ∑ | f k − g |
k =1
 Result:
g = median{ f 1 , f 2 , f 3 , , f N } = f ( m )
 If 51% of samples are correct and 49% outliers,
the median still finds the right level!
Nonlinear image smoothing
The median filter

 Caution: points, thin lines and corners are erased by the median filter

Test images

Results of 3× 3 pixel median filter

Nonlinear image smoothing
The median filter

 Cross shaped window can correct some of the problems

Results of the 9 pixel cross shaped window median filter

Nonlinear image smoothing
The median filter

 Implementing the median filter

 Sorting needs O(N2) comparisons
– Bubble sort
– Quick sort
– Huang algorithm (based on histogram)
– VLSI median
Nonlinear image smoothing
The median filter

 VLSI median block diagram

x1 x(1)

x2 x(2)

x3 x(3)
a Min(a,b)
x4 x(4)
b Max(a,b)
x5 x(5)

x6 x(6)

x7 x(7)
Nonlinear image smoothing
The median filter

 Color median filter

– There is no natural ordering in 3D (RGB) color
space
– Separate filtering on R,G and B components does
not guarantee that the median selects a true
sample from the input window
– Vector median filter, defined as the sample
minimizing the sum of absolute deviations from all
the samples
– Computing the vector median is very time
consuming, although several fast algorithms exist
Nonlinear image smoothing
The median filter

 Example of color
median filtering
 5x5 pixels window
 Up: original image
 Down: filtered image
Nonlinear image smoothing
The weighted median filter

 The basic idea is to give higher weight to some samples, according to

their position with respect to the center of the window
 Each sample is given a weight according to its spatial position in the
window.
 Weights are defined by a weighting mask
 Weighted samples are ordered as usually
 The weighted median is the sample in the ordered array such that
neither all smaller samples nor all higher samples can cumulate more
than 50% of weights.
 If weights are integers, they specify how many times a sample is
replicated in the ordered array
Nonlinear image smoothing
The weighted median filter

 Numerical example for the weighted median filter

1 2 1

2 3 2

1 2 1

3 7 8
2 3 7 6
4 6 7

2, 2, 3, 3, 3, 3, 4, 6, 6, 7, 7, 7, 7, 7, 8
Nonlinear image smoothing
The multi-stage median filter

 Better detail preservation

R3

R2 • • • R4

• • •
m5
R1 • • • • •

• • •

• • •

mi = median( Ri ), i =1,2,3,4.

Result = median( m1,m2,m3,m4,m5 )

Nonlinear image smoothing
Rank-order filters (L filters)

 Block diagram
weights

f(1)
f1 ∗a1
f(2)
f2
∗a2
y
. .
.
ordering . Σ sum
. .

f(N)
fN
∗aN

∑a
k =1
k =1 y = a1f(1) + a2 f(2) +...+ aN f(N)
Nonlinear image smoothing
Rank-order filters (L filters)

 Some examples
a1=1

Percentile . . .
Min
Particular cases
filters (rank of grey level
selection): aN=1 morphological
filters
. . . Max
0%,
50% am=1

. . . . . .
100% Median

a1=.5 aN=.5
. . .
Mid range
Nonlinear image smoothing
Rank-order filters (L filters)

 Optimality
N
ε( f ) = ∑ | f ( k ) − f |r Minkovski distance
k =1

Case r = 1: best estimator is median.

Case r = 2, best estimator is arithmetic mean.
Case r → ∞, best estimator is mid-range.
Nonlinear image smoothing
Rank-order filters (L filters)

 Alpha-trimmed mean filter

1 /(2Q + 1), fo r m − Q ≤ k ≤ m + Q
ak = 
 0, o th erw ise

α = (2Q + 1) / N, defines de degree of averaging

α = 1 corresponds to arithmetic mean
α = 1 / N corresponds to the median filter

Properties: in between mean and median

Nonlinear image smoothing
Rank-order filters (L filters)

Median of absolute differences trimmed mean

 Better smoothing than the median filter and good edge preservation

M 1 = median{ f1 , f 2 , f 3 ,  , f N }
M 2 = median{| f i − M 1 |}, i = 1,2,..., N
output = average{ f i : | f i − M 1 | ≤ M 2 }

M2 is a robust estimator of the variance

Nonlinear image smoothing
Rank-order filters (L filters)

k nearest neighbour (kNN) median filter

 Median of the k grey values nearest by rank
to the central pixel
 Aim: same as above
Nonlinear image smoothing
Selected area filtering [Nagao 1980]

 Kuwahara type filtering

 Form regions Ri
 Compute mean, mi and variance, reg 1 reg 2

Si for each region. reg 4

 Result = mi : mi < mj for all j
different from i, i,e. the mean of reg 3

the most homogeneous region

 Matsujama & Nagao reg 5 reg 6

reg 8 reg 7
Nonlinear image smoothing
Conditional mean

 Pixels in a neighbourhood are averaged only if they

differ from the central pixel by less than a given
threshold:
L L
g (m, n) = ∑ ∑ h(k , l ) f (m − k , n − l ),
k =− L l =− L

1, if | f (m − k , n − l ) − f (m, n) | < th

h( k , l ) = 
0, otherwise

L is a space scale parameter and th is a range scale parameter

Nonlinear image smoothing
Conditional mean

 Example with L=3,

th=32
Nonlinear image smoothing
Bilateral filter [Tomasi 1998]

 Space and range are treated in a similar way

 Space and range similarity is required for the averaged pixels
 Tomasi and Manduchi [1998] introduced soft weights to penalize the
space and range dissimilarity.

h(k , l ) = s (k , l )r ( f (m − k , n − l ) − f (m, n))

1
g ( m, n) =
K k l
∑∑ h( k , l ) f ( m − k , n − l ),

K = ∑∑h( k , l )
k l

s() and r() are space and range similarity functions (Gaussian
functions of the Euclidian distance between their arguments).
Nonlinear image smoothing
Bilateral filter

 The filter can be seen as weighted averaging in the

joint space-range space (3D for monochromatic
images and 5D – x,y,R,G,B - for colour images)
 The vector components are supposed to be properly
normalized (divide by variance for example)
 The weights are given by:

|| x c − x || 2
h(x) = exp{− }
s
h(x) = K(d(x c − x); s )
Nonlinear image smoothing
Bilateral filter

 Example of Bilateral filtering

 Low contrast texture has been removed
 Yet edges are well preserved
Nonlinear image smoothing
Mean shift filtering [Comaniciu 1999, 2002]

 Mean shift filtering replaces each pixel’s value with the most probable local value, found
by a nonparametric probability density estimation method.

{x i }i =1...n xi ∈ R d
 The multivariate kernel density estimate obtained in the point x with the kernel K(x) and
window radius r is:

1 n
 x − xi 
fˆ (x) = d
nr
∑
i =1
K 
 r


 For the Epanechnikov kernel, the estimated normalized density gradient is proportional
to the mean shift:

r2 ∇ ˆ f ( x) 1
d + 2 fˆ (x)
= M r ( x) =
nx
∑x
x i ∈S r ( x )
i −x

S is a sphere of radius r, centered on x and nx is the number of samples inside the sphere
Nonlinear image smoothing
Mean shift filtering

 The mean shift procedure is

a gradient ascent method to
find local modes (maxima)
of the probability density and
is guaranteed to converge.
 Step1: computation of the
mean shift vector Mr(x).
 Step2: translation of the
window Sr(x) by Mr(x).
 Iterations start for each pixel
(5D point) and tipically
converge in 2-3 steps.
Nonlinear image smoothing
Mean shift filtering

 Example1.
Nonlinear image smoothing
Mean shift filtering

Detail of a 24x40
window from the
cameraman image
a) Original data
b) Mean shift paths for some
points
c) Filtered data
d) Segmented data
Nonlinear image smoothing
Mean shift filtering

 Example 2
Nonlinear image smoothing
Mean shift filtering

Comparison to bilateral filtering

 Both methods based on simultaneous processing of both the
spatial and range domains
 While the bilateral filtering uses a static window, the mean shift
window is dynamic, moving in the direction of the maximum
increase of the density gradient.
REFERENCES
 D. Comaniciu, P. Meer: Mean shift analysis and applications. 7th
International Conference on Computer Vision, Kerkyra, Greece, Sept.
1999, 1197-1203.
 D. Comaniciu, P. Meer: Mean shift: a robust approach toward feature
space analysis. IEEE Trans. on PAMI Vol. 24, No. 5, May 2002, 1-18.
 M. Elad: On the origin of bilateral filter and ways to improve it. IEEE
trans. on Image Processing Vol. 11, No. 10, October 2002, 1141-1151.
 B. Jahne: Digital image processing. Springer Verlag, Berlin 1995.
 M. Nagao, T. Matsuiama: A structural analysis of complex aerial
photographs. Plenum Press, New York, 1980.
 W.K. Pratt: Digital image processing. John Wiley and sons, New York
1991
 C. Tomasi, R. Manduchi: Bilateral filtering for gray and color images.
Proc. Sixth Int’l. Conf. Computer Vision , Bombay, 839-846.