MAN-522: COMPUTER VISION

SET-2
Projections and
Camera Calibration
 Image formation
• How are objects in the world captured in
an image?
 Physical parameters of
image formation
• Geometric
– Type of projection
– Camera pose
• Optical
– Sensor’s lens type
– focal length, field of view, aperture
• Photometric
– Type, direction, intensity of light reaching sensor
– Surfaces’ reflectance properties
 Image formation

• Let’s design a camera

– Idea 1: put a piece of film in front of an object
– Do we get a reasonable image?

Slide by Steve Seitz

 Pinhole camera

• Add a barrier to block off most of the rays

– This reduces blurring
– The opening is known as the aperture
– How does this transform the image?

Slide by Steve Seitz

 Pinhole camera
• Pinhole camera is a simple model to approximate
imaging process, perspective projection.

Image
plane

Virtual pinhole
image

If we treat pinhole as a point, only one ray

from any given point can enter the camera.

Fig from Forsyth and Ponce

 Camera obscura

In Latin, means
‘dark room’

"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January

24, 1544, and later he used this illustration of the event in his book De Radio
Astronomica et Geometrica, 1545. It is thought to be the first published illustration of
a camera obscura..."
Hammond, John H., The Camera Obscura, A Chronicle

http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html
 Camera obscura

Jetty at Margate England, 1898.

An attraction in the late

Around 1870s
19th century

http://brightbytes.com/cosite/collection2.html
Adapted from R. Duraiswami
 Camera obscura at home

http://blog.makezine.com/archive/2006/02/how_to_room_
Sketch from http://www.funsci.com/fun3_en/sky/sky.htm sized_camera_obscu.html
Perspective effects
 Perspective effects
• Far away objects appear smaller

Forsyth and Ponce

Perspective effects
 Perspective effects
• Parallel lines in the scene intersect in the image
• Converge in image on horizon line
Image plane
(virtual)
pinhole

Scene
 Projection properties
• Many-to-one: any points along same ray map to
same point in image
• Points  points
• Lines  lines (collinearity preserved)
• Distances and angles are not preserved
• Degenerate cases:
– Line through focal point projects to a point.
– Plane through focal point projects to line
– Plane perpendicular to image plane projects to part of
the image.
 Perspective and art
• Use of correct perspective projection indicated in
1st century B.C. frescoes
• Skill resurfaces in Renaissance: artists develop
systematic methods to determine perspective
projection (around 1480-1515)

Raphael Durer, 1525

 Perspective projection equations
• 3d world mapped to 2d projection in image plane
Image
plane
Focal
length

Optical
Camera axis
frame

‘ Scene / world
‘’ ’ points

Scene point Image coordinates

Forsyth and Ponce
 Homogeneous coordinates
Is this a linear transformation?
• no—division by z is nonlinear
Trick: add one more coordinate:

homogeneous image homogeneous scene

coordinates coordinates

Converting from homogeneous coordinates

Slide by Steve Seitz

 Perspective Projection Matrix
• Projection is a matrix multiplication using
homogeneous coordinates:

 x
1 0 0 0    x 
0 1 0 0  y 
  y  x
(f' , f' )
y
  z    z z
0 0 1 / f ' 0    z / f '
 
1 divide by the third
coordinate to convert back
to non-homogeneous
coordinates

Complete mapping from world points to image pixel

positions?
Slide by Steve Seitz
Perspective projection & calibration
• Perspective equations so far in terms of
camera’s reference frame….
• Camera’s intrinsic and extrinsic parameters
needed to calibrate geometry.

Camera
frame
Perspective projection & calibration
World Extrinsic:
frame
Camera frame World frame

Intrinsic:
Image coordinates relative to
camera  Pixel coordinates
Camera
frame

Camera to World to 3D
2D Perspective
pixel coord.
point = trans. matrix
projection matrix camera coord.
trans. matrix
point
(3x1) (3x4) (4x1)
(3x3) (4x4)
 Weak perspective
• Approximation: treat magnification as constant
• Assumes scene depth << average distance to
camera
Image World
plane points:
 Orthographic projection
• Given camera at constant distance from scene
• World points projected along rays parallel to
optical access
 Pinhole size / aperture
How does the size of the aperture affect the
image we’d get?
Larger

Smaller
 Adding a lens

focal point

• A lens focuses light onto the film

– Rays passing through the center are not deviated
– All parallel rays converge to one point on a plane
located at the focal length f
Slide by Steve Seitz
Pinhole vs. lens
 Cameras with lenses

focal point
optical center
(Center Of Projection)

• A lens focuses parallel rays onto a single focal

point
• Gather more light, while keeping focus; make
pinhole perspective projection practical
Camera Parameters
 Imaging Geometry

W
Object of Interest
in World Coordinate
System (U,V,W)
V
U
 Imaging Geometry

Camera Coordinate
Y System (X,Y,Z).

X Z
• Z is optic axis
f • Image plane located f units
out along optic axis
• f is called focal length
 Imaging Geometry

W
Y y
X Z x

V
U

Forward Projection onto image plane.

3D (X,Y,Z) projected to 2D (x,y)
 Imaging Geometry

W
Y y
X Z x

V
u U

v Our image gets digitized

into pixel coordinates (u,v)
 Imaging Geometry
Camera Image (film) World
Coordinates Coordinates W
Coordinates
Y y
X Z x

V
u U

v Pixel
Coordinates
 Forward Projection

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

We want a mathematical model to describe

how 3D World points get projected into 2D
Pixel coordinates.

Our goal: describe this sequence of

transformations by a big matrix equation!
 Backward Projection

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

Note, much of vision concerns trying to

derive backward projection equations to
recover 3D scene structure from images
(via stereo or motion)

But first, we have to understand forward projection…

 Forward Projection

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

3D-to-2D Projection
• perspective projection

We will start here in the middle, since we’ve already

talked about this when discussing stereo.
 Basic Perspective Projection
Scene Point Perspective Projection Eqns
P = (X,Y,Z)
X
Image Point y x f
Y
p = (x,y,f) X Y
Z
x Y
Z
y
y f
x
X
f Z
Z
O

O.Camps, PSU
 Basic Perspective Projection
Scene Point Perspective Projection Eqns
P = (X,Y,Z)
X
Image Point y x f
Y
p = (x,y,f) X Y
Z
x Y
Z
y
y f
x
X
f Z
Z derived via similar
O
triangles rule

x X

f
Z
O.Camps, PSU
 Basic Perspective Projection
Scene Point Perspective Projection Eqns
P = (X,Y,Z)
X
Image Point y x f
Y
p = (x,y,f) X Y
Z
x Y
Z
y
y f
x
X
f Z
Z derived via similar
O
triangles rule
X Y
y
x
f f
Z Z
O.Camps, PSU
 Basic Perspective Projection
Scene Point Perspective Projection Eqns
P = (X,Y,Z)
X
Image Point y x f
Y
p = (x,y,f) X Y
Z
x Y
Z
y
y f
x
X
f Z
Z
O

So how do we represent this as a matrix equation?

We need to introduce homogeneous coordinates.

O.Camps, PSU
 Homogeneous Coordinates
Represent a 2D point (x,y) by a 3D point (x’,y’,z’) by
adding a “fictitious” third coordinate.

By convention, we specify that given (x’,y’,z’) we can

recover the 2D point (x,y) as
x' y'
x y
z' z'
Note: (x,y) = (x,y,1) = (2x, 2y, 2) = (k x, ky, k)
for any nonzero k (can be negative as well as positive)
 Perspective Matrix Equation
(in Camera Coordinates)

X X 
x f  x'  f 0 0 0  
 y '   0  Y
Z
   f 0 0  
Y Z 
y f  z '   0 0 1 0  
Z 1
 Forward Projection

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

Rigid Transformation (rotation+translation)

between world and camera coordinate systems
 World to Camera Transformation
PC PW
W

X
U
Z V

Avoid confusion: Pw and Pc are not two different

points. They are the same physical point, described
in two different coordinate systems.
 World to Camera Transformation
PC PW
W

X
U
R Z V
C
Rotate to Translate by - C
align axes (align origins)

P C = R ( PW - C )
Matrix Form, Homogeneous Coords
P C = R ( PW - C )

X r11 r12 r13  1 0 0  cx U

Y r21 r22 r23  0 1 0  cy V
Z r31 r32 r33  0 0 1  cz W
1 0 0 0 1 0 0 0 1 1
 Example: Simple Stereo System
Y Z
(X,Y,Z)
z
left y
camera
located at ( , )
z
(0,0,0) y ( , )
x right
camera
Tx located at
x (Tx,0,0)

X
Left camera located at world origin (0,0,0)
and camera axes aligned with world coord axes.
 Simple Stereo, Left Camera

X 1r11 r012 r013  1 0 0 0cx U

Y 0r21 r122 r023  0 1 0 0cy V
Z 0r31
r032 r133  0 0 1 0cz W
1 0 0 0 1 0 0 0 1 1

camera axes aligned located at world

with world axes position (0,0,0)

=
 Simple Stereo Projection Equations
Left camera
 Example: Simple Stereo System
Y Z
(X,Y,Z)
z
left y
camera
located at ( , )
z
(0,0,0) y ( , )
x right
camera
Tx located at
x (Tx,0,0)

X
Right camera located at world location (Tx,0,0)
and camera axes aligned with world coord axes.
 Simple Stereo, Right Camera

X 1r11 r012 r013  1 0 0 -T

 cxx U
Y 0r21 r122 r023  0 1 0 0cy V
Z 0r31
r032 r133  0 0 1 0cz W
1 0 0 0 1 0 0 0 1 1

camera axes aligned located at world

with world axes position (Tx,0,0)

=
 Simple Stereo Projection Equations
Left camera

Right camera
 Bob’s sure-fire way(s) to
figure out the rotation
X r11 r12 r13  1 0 0  cx U
Y r21 r22 r23  1 about
0forget 0 thiscy V
while thinking
Z r31 r32 r33  0 rotations
0about 1  cz W
1 0 0 0 1 0 0 0 1 1

PC = R P W
This equation says how vectors in the world coordinate
system (including the coordinate axes) get transformed
into the camera coordinate system.
 Figuring out Rotations
X r11 r12 r13  U
Y r21 r22 r23  V PC = R P W
Z r31 r32 r33  W
1 0 0 0 1 1
what if world x axis (1,0,0) corresponds to camera axis (a,b,c)?

X
a r11 r12 r13  U
1 X
a ra11 r12 r13  U
1
Y
b r21 r22 r23  V
0 Y
b rb21 r22 r23  V
0
Z
c r31 r32 r33  0
W Z
c rc31 r32 r33  0
W
1 0 0 0 1 1 1 0 0 0 1 1
we can immediately write down the first column of R!
 Figuring out Rotations
and likewise with world Y axis and world Z axis...

same axis in camera coords axis is world coords

X r11 r12 r13  U

Y r21 r22 r23  V
Z r31 r32 r33  W
1 0 0 0 1 1

world X axis (1,0,0) world Z axis (0,0,1)

in camera coords in camera coords
world Y axis (0,1,0)
in camera coords
 Figuring out Rotations
Alternative approach: sometimes it is easier to specify
what camera X,Y,or Z axis is in world coordinates. Then
do rearrange the equation as follows.

PC = R P W R-1PC = PW RTPC = PW

r11 r21 r31  X U

r12 r22 r32  Y V
r13 r23 r33  Z W
0 0 0 1 1 1
 Figuring out Rotations
r11 r21 r31  X U
r12 r22 r32  Y V RTPC = PW
r13 r23 r33  Z W
0 0 0 1 1 1
what if camera X axis (1,0,0) corresponds to world axis (a,b,c)?

r11 r21 r31  X

1 U
a ra11 r21 r31  X
1 U
a
r12 r22 r32  Y
0 V
b rb12 r22 r32  Y
0 V
b
r13 r23 r33  Z
0 W
c 0
rc13 r23 r33  Z W
c
0 0 0 1 1 1 0 0 0 1 1 1
we can immediately write down the first column of RT,
(which is the first row of R).
 Figuring out Rotations
and likewise with camera Y axis and camera Z axis...

same axis in camera coords axis is world coords

X r11 r12 r13  U

Y r21 r22 r23  V
Z r31 r32 r33  W
1 0 0 0 1 1
camera X axis (1,0,0)
in world coords

camera Z axis (0,0,1)

camera Y axis (0,1,0) in world coords
in world coords
 Example

y
z
x

0 0 1 0 -1 0
Rtrain 0 -1 0 Rfly 0 0 1
1 0 0 -1 0 0
 Note: External Parameters
also often written as R,T
X r11 r12 r13  1 0 0  cx U
Y r21 r22 r23  0 1 0  cy V
Z r31 r32 r33  0 0 1  cz W
1 0 0 0 1 0 0 0 1 1

R ( PW - C ) r11 r12 r13 tx

r21 r22 r23 ty
= R PW - R C
r31 r32 r33 tz
= R PW + T 0 0 0 1
 Summary

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z
We now know how to transform 3D world
coordinate points into camera coords, and
then do perspective project to get 2D points
in the film plane.

Next time: pixel coordinates

 Recall: Imaging Geometry

W
Object of Interest
in World Coordinate
System (U,V,W)
V
U
 Imaging Geometry

Camera Coordinate
Y System (X,Y,Z).

X Z
• Z is optic axis
f • Image plane located f units
out along optic axis
• f is called focal length
 Imaging Geometry

W
Y y
X Z x

V
U

Forward Projection onto image plane.

3D (X,Y,Z) projected to 2D (x,y)
 Imaging Geometry

W
Y y
X Z x

V
u U

v Our image gets digitized

into pixel coordinates (u,v)
 Imaging Geometry
Camera Image (film) World
Coordinates Coordinates W
Coordinates
Y y
X Z x

V
u U

v Pixel
Coordinates
 Forward Projection

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

We want a mathematical model to describe

how 3D World points get projected into 2D
Pixel coordinates.

Our goal: describe this sequence of

transformations by a big matrix equation!
 Intrinsic Camera Parameters

World Camera Film Pixel

Coords Coords Coords Coords
U X x u
V Y y v
W Z

Affine Transformation
 Intrinsic parameters

• Describes coordinate transformation

between film coordinates (projected image)
and pixel array
• Film cameras: scanning/digitization
• CCD cameras: grid of photosensors

still in T&V section 2.4

 Intrinsic parameters (offsets)
film plane
pixel array
(projected image)
ox (0,0) u (col)

oy x v (row)
(0,0)

X Y
u  f  ox v  f  oy
Z Z
ox and oy called image center or principle point
 Intrinsic parameters
sometimes one or more coordinate axes are flipped (e.g. T&V section 2.4)

film plane pixel array

ox (0,0) u (col)

oy y
v (row)
x
(0,0)

X Y
u  f  ox v  f  oy
Z Z
 Intrinsic parameters (scales)
sampling determines how many rows/cols in the image

film scanning
resolution
pixel array

C cols x R rows
CCD
analog

resample
 Effective Scales: sx and sy

1 X 1Y
u s f  ox v  f  oy
x Z sy Z

Note, since we have different scale factors in x and y,

we don’t necessarily have square pixels!

Aspect ratio is sy / sx

O.Camps, PSU
 Perspective projection matrix
Adding the intrinsic parameters into the
perspective projection matrix:
X 
 x'  f / s x 0 ox 0  
 y '   0  Y
   f / sy oy 0  
Z 
 z '   0 0 1 0  
1
To verify:
x’
u 1 X Y
1
z’ u s f  ox v  f  oy
y’ x Z sy Z
v
z’
O.Camps, PSU
 Note:
Sometimes, the image and the camera coordinate systems
have opposite orientations: [the book does it this way]

X X 
f  ( u  o x ) s x  x'  f / s x 0  ox 0  
Z  y '   0   Y
    f / sy  oy 0
Y Z 
f  ( v  o y )s y  z '   0 0 1 0  
Z 1
 Note 2
In general, I like to think of the conversion as
a separate 2D affine transformation from film
coords (x,y) to pixel coordinates (u,v):
X 
u’ a11 a12 xa' 13f 0 0 0  
    Y
v’ a21 a22 ya' 23  0
   f 0 0  
w’ Z 
0 0  z 1'   0 0 1 0  
1
Maff Mproj

u = Mint PC = Maff Mproj PC

 Huh?

Did he just say it was “a fine” transformation?

No, it was “affine” transformation, a type of

2D to 2D mapping defined by 6 parameters.

More on this in a moment...

 Summary : Forward Projection
World Camera Film Pixel
Coords Coords Coords Coords
U X x u
V Mext Y Mproj Maff
y v
W Z

U Mext X Mint u
V Y v
W Z

U u
M
V m11 m12 m13 m14 v
W m21 m22 m23 m24
m31 m31 m33 m34
Summary: Projection Equation
Film plane Perspective
World to camera
to pixels projection

Maff Mproj Mext

Mint

M
Intro to Image Mappings
 Image Mappings
Overview

from R.Szeliski
 Geometric Image Mappings
Geometric
image transformation transformed image

(x,y)
(x’,y’)

x’ = f(x, y, {parameters})
y’ = g(x, y, {parameters})
 Linear Transformations
(Can be written as matrices)
Geometric
image transformation transformed image

(x,y)
(x’,y’)

x’ x
y’ = M(params) y
1 1
 Translation
y y’
transform
1

ty

0
0 1 x tx x’

equations matrix form

 Scale
y y’
transform S

0 0
0 1 x 0 S x’

equations matrix form

 Rotation
y y’
transform
1


)
0
0 1 x x’

equations matrix form

 Euclidean (Rigid)
y y’
transform 

)
1
ty

0
0 1 x tx x’

equations matrix form

 Partitioned Matrices

http://planetmath.org/encyclopedia/PartitionedMatrix.html
 Partitioned Matrices

2x1 2x2 2x1 2x1

1x1 1x2 1x1 1x1 matrix form

equation form
Another Example (from last time)
X r11 r12 r13 tx U
Y r21 r22 r23 ty V
Z r31 r32 r33 tz W
1 0 0 0 1 1
3x1 3x3 3x1 3x1
PC R T PW
1x1 = 1x3 1x1 1x1
1 0 1 1

PC = R P W + T
 Similarity (scaled Euclidean)
y y’ S
transform 

)
1
ty

0
0 1 x tx x’

equations matrix form

 Affine
y y’
transform
1

0
0 1 x x’

equations matrix form

 Projective
y y’
transform
1

0
0 1 x x’
Note!

equations matrix form

Summary of 2D Transformations
 Summary of 2D Transformations

Euclidean
 Summary of 2D Transformations

Similarity
 Summary of 2D Transformations

Affine
 Summary of 2D Transformations

Projective
 Summary of 2D Transformations

from R.Szeliski