ENGN4528 and ENGN6528

Computer Vision
Image Processing: Basics

Dr Shaodi You
1. Data61 - CSIRO
2. Research School of Engineering, The Australian National University
 Review:
The computer vision problem
• Make a computer see and understand images

• We know it is physically possible – we do it every

day and effortlessly!

Real world Sensing device Interpreting device Interpretation

scene

A person/
A person with
folded arms/
Prof. Pietro
Perona/ etc.
Camera Obscura

3
 Slide Credit: Saverese

Projection matrix
R,T
jw

kw
Ow
iw

x  KR t X
x: Image Coordinates: (u,v,1)
K: Intrinsic Matrix (3x3)
R: Rotation (3x3)
t: Translation (3x1)
Extrinsic Matrix X: World Coordinates: (X,Y,Z,1)

4
Human visual processing

6
 Image processing operations

• Point-wise operations
o histogram operation
• Neighborhood operations
• Spatial domain (image convolution)
o Frequency domain (Fourier transform)
o Binary image analysis
• Geometric operations
o Rotation, translation, affine…

7
Recall: digital image representation
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93

8
I = f(x,y): R2 R (grey image);
or R2  R3 (colour image)

Origin

y
Image “After snow storm” f(x,y)

9
I = f(x,y)

10
 Point Operations

 Operation on Pixel's
value
 Context free
 Operation can be
performed on the image
histogram
 Example:
g x, y   M  f x, y 

g x, y     f x, y   

11
 Neighbourhood Operations

 Operation depends on
Pixel's value and its
spatial neighbours.

 Context dependant.
g x, y   M  f i, j  | (i, j )  N x, y 

 Example:

g  x, y    f i, j  / n
i , jN ( x , y )

12
 Geometric Operations

 Operation depend on
pixel's coordinates.

 Independent of pixels
value.

 Context free. g x, y   f Gx, y 

 Example: translation

g x, y   f x  a, y  b
13
 Point operations:

(Histogram operation)

14
 Point Operations

• A point operation can be defined as a mapping

function: vnew  M vold 
where v stands for gray values.
• M(v) takes any value v in the source image into
vnew in the destination image.
• Simplest case - Linear Mapping:

M v    v  

• Often implemented on the image histogram.

15
 Histogram

The histogram of a digital image with gray values

is the discrete function

nk
p ( rk ) 
n
nk: Number of pixels with gray value rk
n: total Number of pixels in the image

The function p(rk) represents the fraction of the total

number of pixels with gray value rk.
16
 Histogram
• Histogram provides a global description of the
appearance of the image.

• If we consider the gray values in the image as

realizations of a random variable R, with some
probability density, histogram provides an
approximation to this probability density.

• In other words,
Pr(R  rk )  p(rk )

17
 Some Typical Histograms
The shape of a histogram provides useful information for
image global appearance.

Dark image

18
Bright image

Low-contrast image

19
High-contrast image

20
Application: image enhancement

21
 Histogram-based image
enhancement/image editing

• Histogram modification

• Histogram Equalization

• Histogram Matching

22
 Histogram modification

• Given a point operation:

vb  M v a 
• M(va) takes any value va in image A and maps it into vb in
image B.
• Requirement: the mapping M is a non descending function
(M-1 exists).
• In this case, the area under Ha between 0 and va is equal to the
area under Hb between 0 and vb.

23
 Monotonicity Requirement
• If it is required to map the full gray-level range (256
values) to its full range while keeping the gray-level
order, a non-decreasing monotonic mapping
function is needed:

M(v)
255

stretching

v
255
contraction 24
• The area under Ha between 0 and va is equal to the
area under Hb between 0 and vb=M(va)

v
M(v)

v
Hb

Ha

v
25
 Contrast Enhancement
• If most of the gray-levels in the image are in
[u1 u2], the following mapping increases the
image contrast.
• The values u1 and u2 can be found by using
the image’s accumulated histogram.
255

M(v)

u1 u2 v
255 26
Power-law transformations

s  cr

27
28
 Histogram Equalization
• Visual contrast between objects depends on the their
gray-level separation.
• How can we improve the contrast after an image has
been acquired?

Hard to discriminate Doesn’t help This is better

What about this?

29
 Histogram Equalization

• For a better visual discrimination of an image we

would like to re-assign gray-levels so that gray-
level resource will be optimally assigned.

• Our goal: finding a gray-level transformation M(v)

such that:
– The histogram Hb is as flat as possible.
– The order of gray-levels is maintained.
– The histogram bars are not fragmented.
30
Example Original image and its histogram

31
Histogram equalized image and its histogram

32
Hist. Eq.: Example 1

3500 3500

3000 3000

2500 2500

2000 2000

1500 1500

1000 1000

500 500

0 0
0 50 100 150 200 250 0 50 100 150 200 250
33
Original Equalized
Hist. Eq.: Example 2

3000
3000
2500
2500
2000
2000
1500
1500
1000
1000
500
500
0
0 0 50 100 150 200 250
0 50 100 150 200 250

Original Equalized 34
 Comments:
Histogram equalization may not always produce desirable
results, particularly if the given histogram is very narrow. It
can produce false edges and regions. It can also increase
image “graininess” and “patchiness.”

35
36
 Histogram Matching
• Histogram matching is a process where an image is
modified such that its histogram matches that of another
(reference) image.

• A common application of this is to match the images

from two sensors with slightly different responses, or
from a sensor whose response changes over time.

• Homework: think about how to implement “histogram

matching” ? (hint: use “histogram equalization” as an
intermediate step.).
37
Histogram matching example

38
2D Geometric Transformations

(Spatial transformation, Image

warping)

39
 Image Warping

http://www.jeffrey-martin.com

40
Some slides from Steve Seitz
 Image Warping
Point operation and neighborhood operation:
change range of image
g(x) = h(f(x))

f f
h
x x

Image warping: change domain of image

g(x) = f(h(x))
f f
h
x x
41
 Image Warping
Image filtering: change range of image
g(x) = h(f(x))

f g
h

Image warping: change domain of image

g(x) = f(h(x))
f g
h

42
 Image Warping
Why?
- texture mapping

- image processing (rotation, zoom

in/out, etc)

- image morphing/blending

- image mosaic (stitching)

- image based-modeling &

rendering

43
 Geometric transformation
Examples of transformation

translation rotation aspect

affine perspective cylindrical

44
Transformation Function

Transform the domain of an image to a

desired geometry
45
 Definition: Image Warping
Source Image: Image to be used as the
reference. The geometry of this image is no
changed

Target Image: this image is obtained by

transforming the reference image.

(x,y): coordinates of points in the reference image

(x’,y’): (or [u,v]) coordinates of points in the target

image
46
 Definition: Image Warping
Control points: Unique points in the reference and target
images. The coordinates of corresponding control points in
images are used to determine a transformation function.

Source Image Target Image

47
 Types of Image Warping
Simple geometric transformations:
- Rotation;
- Similarity
- Affine mapping
- Projective mapping

Other general types of transformations

free-form deformation

48
 Types of Transformation

MATLAB functions: griddata, interp2, maketform, imtransform

49
49
 Scaling
• Scaling a coordinate means multiplying each of
its components by a scalar
• Uniform scaling means this scalar is the same
for all components:

2

50
 Scaling
• Non-uniform scaling: different scalars per
component:

x’=x  2;
y’=y0.5

51
 Scaling

• Scaling operation: x '  ax

y '  by

• Or, in matrix form:

 x '   a 0  x 
 y '   0 b   y 
    
scaling matrix S
52
What’s inverse of S?
 Scale

a=1/2

 x'  a 0   x 
 y '  0 1 / a   y 
    

53
53
 Rotation

y
y’

x’
θ
x

 x'  cos  sin    x 

 y '   sin  cos    y 
  

54
54
Rotation Example

θ=3o

55
55
 Shear

square parallelogram

 x' 1 0  x   d x 
 y '   s 1  y   d 
      y 

56
56
Shear Example

 x '   1 0   x  0 
 y '  .5 1  y   1
      

57
57
 Affine Transform:
Rotation+Scaling+Sheer+Translation

square parallelogram

 x'  a11 a12   x   d x 

 y '  a     
   21 a22   y  d y 
58
58
Affine Transform Example

 x' .5 1   x  0

 y '  .5  2  y   1
      

59
59
 Projective Transform
B’
A B
A’
C’

D C

D’
square quadrilateral
a1 x  a2 y  a3
x '
a7 x  a8 y  1
a 4 x  a5 y  a 6
y' 
a7 x  a8 y  1

60
60
Projective Transform Example

[ 0 0; 1 0; 1 1; 0 1] [-4 2; -8 -3; -3 -5; 6 3]

61
61
Projective Image stitching

62
62
How to compute image warping ?

63
Image Warping Computation
x u

y v

S(x,y) T(u,v)

Given a coordinate transform function [f,g[ (or the inverse i.e.,

[F,G] ) and source image S(x,y), how do we compute a
transformed target image T(u,v)?

64
 Forward Warping
x u

y v

S(x,y) T(u,v)

Forward warping algorithm:

for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
65
copy pixel at source S(x,y) to T(u,v)
 Forward Warping
x u

y v

S(x,y) T(u,v)

Forward warping algorithm:

for y = ymin to ymax
for x = xmin to xmax - Any problems for forward
warping?
u = f(x,y); v = g(x,y)
66
copy pixel at source S(x,y) to T(u,v)
 Forward Warping
x u

y v

S(x,y) T(u,v)

Q: What if the transformed pixel located between pixels?

67
 Forward Warping
x u

y v

S(x,y) T(u,v)

Q: What if the transformed pixel located between pixels?

A: Distribute color among neighboring pixels

- known as “splatting”
68
 Forward Warping
• Iterate over source, sending pixels to destination
• Some source pixels map to the same dest. pixels
• Some dest. pixels may have no corresponding
source
• Holes in reconstruction
• Must splat etc.
x u

y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)
69
 Forward Warping
• Iterate over source, sending pixels to destination
• Some source pixels map to the same dest. pixels
• Some dest. pixels may have no corresponding
source
• Holes in reconstruction - How to remove the holes?

• Must splat etc.

x u

y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)
70
 Forward Warping
• Iterate over source, sending pixels to destination
• Some source pixels map to the same dest. pixels
• Some dest. pixels may have no corresponding
source
• Holes in reconstruction - How to remove the holes?

• Must “splat” etc.

x u

y v
for y = ymin to ymax
for x = xmin to xmax
u = f(x,y); v = g(x,y)
copy pixel at source S(x,y) to T(u,v)
71
 Inverse Warping
x u

y v

S(x,y) T(u,v)

Inverse warping algorithm:

for v = vmin to vmax
for u = umin to umax
x = F(u,v); y = G(u,v)
72
copy pixel at source S(x,y) to T(u,v)
 Inverse Warping
x u

y v

S(x,y) T(u,v)

Q: What if pixel comes from “between” two pixels?

A: Interpolate color values from neighboring pixels

73
 Inverse Warping
• Iterate over dest., finding pixels from source
• Non-integer evaluation source image, resample
• May oversample source
• But no holes
• Simpler, better than forward mapping

x u

for v = vmin to vmax y v

for u = umin to umax
x = F(u,v); y = G(u,v)
copy pixel at source S(x,y) to T(u,v)

74
 Forward vs. inverse warping

• Q: which is better?

• A: usually inverse—eliminates holes

– however, it requires an invertible warp function—not
always possible...

75
 2D Geometric Transformations in
Homogeneous Coordinates Representation

76
 Homogeneous Coordinates

• Homogeneous
coordinates
– represent coordinates
in 2 dimensions with a
3-vector
 x
 x  homogeneous coords  
 y      y 
   1 

77
 Homogeneous Coordinates

• Q: How can we represent translation as a

3x3 matrix?

x'  x  t x
y'  y  t y

78
 Homogeneous Coordinates

• A: Using the rightmost column:

 x '   x  t x  1 0 tx  x
 y '   y  t   0 1
 
t y  y
   y 
 1   1  0 0 1   1 

79
 Translation
Homogeneous Coordinates
• Example of
translation
 x' 1 tx  x   x  tx 

0
 y '   
0 1 t y  y  y  t 
     y

 1  0 0 
1   1   1 


tx = 2
ty = 1
80
Basic 2D Transformations in homogeneous
representation

 x '  1 0 t x   x   x '  s x 0 0  x 
 y '  0 1 t   y   y '   0 sy 0  y 
   y     
 1  0 0 1   1   1   0 0 1  1 

Translate Scale

 x' cos   sin  0  x   x '  1 shx 0  x 

 y '   sin  0  y   y '   sh
   cos 
   y 1 0  y 
 1   0 0 1  1   1   0 0 1  1 

Rotate Shear

81
 Affine Transformations

• Affine transformations are

 x'  a11 a12   x   d x 
combinations of …
 y '  a     
– Linear transformations, and    21 a22   y  d y 
– Translations
• Properties of affine
transformations:
– Origin does not necessarily
map to origin
 x'  a b c  x 
– Lines map to lines  y '  d e f  y 
– Parallel lines remain  w  0 0 1   w
parallel   
– Ratios are preserved
– Closed under composition
82
– Models change of basis
 Transformation Composition

• Transformations can be combined by

matrix multiplication
 x'   1 0 tx  cos   sin  0  sx 0 0   x 
 y '    0 1 ty   sin  cos  0  0 sy 0   y 
 w'  0 0 1   0 0 1      
  0 0 1    w
   
p’ = T(tx,ty) R() S(sx,sy) p

83
 Projective Transformations
a1 x  a2 y  a3
x '
• Projective transformations … a7 x  a8 y  1
a 4 x  a5 y  a 6
– Affine transformations, and y' 
a7 x  a8 y  1
– Projective warps
• Properties of projective transformations:
– Origin does not necessarily map to origin
– Lines map to lines
– Parallel lines do not necessarily remain parallel
– Ratios are not preserved
– Closed under composition  x'   a b c   x 
 y '   d e f   y 
– Models change of basis  w'  g h i   w
    
84
 Hierarchy of 2D image transformations
(stratification)

These transformations are a nested set of groups 85

• Closed under composition and inverse is a member
 Other types of warping

http://astronomy.swin.edu.au/~pbourke/projection/imagewarp/
86
86
Free Form Warping

87
87
Image Morphing

88
 Image morphing
image #1 cross-fading image #2

warp morphing warp

89
 Summary
• Point operation and histogram equalization
can enhance the image contrast
• Geometrical transformation is used in
image wrapping
• Read textbook Section 3.1 & Section 3.6.

• Next lecture on Binary image analysis and

Morphology (Chapter 3.3)
90