Cameras and Stereo

CSE 455
Linda Shapiro

1
Müller-Lyer Illusion

http://www.michaelbach.de/ot/sze_muelue/index.html

• What do you know about perspective projection?

• Vertical lines?
• Other lines?
2
Image formation

Object Film

Let’s design a camera

• Idea 1: put a piece of film in front of an object
• Do we get a reasonable image?
3
Pinhole camera

Object Barrier Film

Add a barrier to block off most of the rays

• This reduces blurring
• The opening known as the aperture
• How does this transform the image?

4
Adding a lens

“circle of
confusion”

A lens focuses light onto the film

• There is a specific distance at which objects are “in focus”
– other points project to a “circle of confusion” in the image
• Changing the shape of the lens changes this distance
5
Lenses

focal point
optical center
(Center Of Projection)

A lens focuses parallel rays onto a single focal point

• focal point at a distance f beyond the plane of the lens
– f is a function of the shape and index of refraction of the lens
• Aperture of diameter D restricts the range of rays
– aperture may be on either side of the lens
• Lenses are typically spherical (easier to produce)
6
• Real cameras use many lenses together (to correct for aberrations)
Thin lenses

Thin lens equation:

• Any object point satisfying this equation is in focus

7
Digital camera

A digital camera replaces film with a sensor array

• Each cell in the array is a Charge Coupled Device (CCD)
– light-sensitive diode that converts photons to electrons
• CMOS is becoming more popular (esp. in cell phones)
– http://electronics.howstuffworks.com/digital-camera.htm
8
Issues with digital cameras
Noise
– big difference between consumer vs. SLR-style cameras
– low light is where you most notice noise
Compression
– creates artifacts except in uncompressed formats (tiff, raw)
Color
– color fringing artifacts from Bayer patterns
Blooming
– charge overflowing into neighboring pixels
In-camera processing
– oversharpening can produce halos
Interlaced vs. progressive scan video
– even/odd rows from different exposures
Are more megapixels better?
– requires higher quality lens
– noise issues
Stabilization
– compensate for camera shake (mechanical vs. electronic)
More info online, e.g.,
• http://electronics.howstuffworks.com/digital-camera.htm 9
• http://www.dpreview.com/
Projection
Mapping from the world (3d) to an image (2d)
• Can we have a 1-to-1 mapping?
• How many possible mappings are there?

An optical system defines a particular

projection. We’ll talk about 2:

1. Perspective projection (how we see “normally”)

2. Orthographic projection (e.g., telephoto lenses)

10
Modeling projection

3D point
C
negative z axis

The coordinate system

• We will use the pin-hole model as an approximation
• Put the optical center (Center Of Projection) at the origin
• Put the image plane (Projection Plane) in front of the COP
• The camera looks down the negative z axis
– we need this if we want right-handed-coordinates

11
Modeling projection

x/z = x´/-d
x´ = -d(x/z)

y/z = y´/-d
y´ = -d(y/z)
Projection equations
• Compute intersection with PP of ray from (x,y,z) to COP
• Derived using similar triangles

• We get the projection by throwing out the last coordinate:

(x´,y´) 12
Homogeneous coordinates
Is this a linear transformation?
• no—division by z is nonlinear
Trick: add one more coordinate:

homogeneous image homogeneous scene

coordinates coordinates

Converting from homogeneous coordinates

13
Perspective Projection
Projection is a matrix multiply using homogeneous coordinates:

divide by third coordinate

projection matrix 3D point 2D point

This is known as perspective projection

• The matrix is the projection matrix

14
Perspective Projection Example
1. Object point at (10, 6, 4), d=2

é x ù é ù
é 1 0 ù
0 ê ú é 1 0 ù
0 ê
10
ê
0
úê y ú ê
0 ú
úê 6 ú = é 10 6 -2 ù
ê 0 1 0 ú = 0 1
ê z ú ê
0 0 0 ú
ê 0 0 -1 / d 0 úê ê úê 4 ú ë û
ë û 1 ú ë 0 0 -1 / 2 0 ûê ú
ë û ë 1 û
Þ x' = -5, y' = -3
2. Object point at (25, 15, 10)

é x ù é 25 ù
é 1 0 ù
0 ê ú é 1 0 ù
0 ê
ê
0
úê y ú ê
0 ú
úê 15 ú é
ê 0 1 0 0 ú =ê 0 1 0 0 ú = ë 25 15 -5 ùû
ê ú
ê 0 0 -1 / d 0 úê z ú ê 0 0 -1 / 2 0 úê 10 ú
ë û 1 ë ûê 1 ú
ë û ë û
Þ x' = -5, y' = -3
Perspective projection is not 1-to-1!
15
Perspective Projection
How does scaling the projection matrix change the transformation?

SAME
16
Perspective Projection

• What happens to parallel lines?

• What happens to angles?
• What happens to distances?

17
Perspective Projection
What happens when d?

18
Orthographic projection
Special case of perspective projection
• Distance from the COP to the PP is infinite

Image World

• Good approximation for telephoto optics

• Also called “parallel projection”: (x, y, z) → (x, y)
• What’s the projection matrix?

19
Orthographic Projection

• What happens to parallel lines?

• What happens to angles?
• What happens to distances?

20
Camera parameters
How many numbers do we need to describe a camera?

• We need to describe its pose in the world

• We need to describe its internal parameters

21
 A Tale of Two Coordinate Systems

v y

COP
u
Camera
w
Two important coordinate systems: o x
1. World coordinate system z “The World”
2. Camera coordinate system

22
Camera parameters
•To project a point (x,y,z) in world coordinates into a
camera
•First transform (x,y,z) into camera coordinates
•Need to know
– Camera position (in world coordinates)
– Camera orientation (in world coordinates)
•Then project into the image plane
– Need to know camera intrinsics
•These can all be described with matrices

23
 3D Translation
• 3D translation is just like 2D with one more
coordinate

x′ 1 0 0 tx x
y′ = 0 1 0 ty y
z′ 0 0 1 tz z
1 0 0 0 1 1

= [x+tx, y+ty, z+tz, 1]T

24
3D Rotation (just the 3 x 3 part shown)
About X axis: 1 0 0 About Y: cosθ 0 sinθ
0 cosθ –sinθ 0 1 0
0 sinθ cosθ -sinθ 0 cosθ

About Z axis: cosθ –sinθ 0

sinθ cosθ 0
0 0 1

General (orthonormal) rotation matrix used in practice:

r11 r12 r13
r21 r22 r23
r31 r32 r33
25
Camera parameters
A camera is described by several parameters
• Translation T of the optical center from the origin of world coords
• Rotation R of the image plane
• focal length f, principal point (x’c, y’c), pixel size (sx, sy)
• blue parameters are called “extrinsics,” red are “intrinsics”
Projection equation

• The projection matrix models the cumulative effect of all parameters

• Useful to decompose into a series of operations
identity matrix

[tx, ty, tz]T

intrinsics projection rotation translation
• The definitions of these parameters are not completely standardized
26
– especially intrinsics—varies from one book to another
 Extrinsics
• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points
right, y-axis points up, z-axis points backwards)

Step 1: Translate by -c

image
plane
0

camera
27
 Extrinsics
• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points
right, y-axis points up, z-axis points backwards)

Step 1: Translate by -c
How do we represent
translation as a matrix
multiplication?

28
 Extrinsics
• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points
right, y-axis points up, z-axis points backwards)

Step 1: Translate by -c
Step 2: Rotate by R

3x3 rotation matrix

29
 Extrinsics
• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points
right, y-axis points up, z-axis points backwards)

Step 1: Translate by -c
Step 2: Rotate by R

30
 Perspective projection

(converts from 3D rays in camera

(intrinsics) coordinate system to pixel coordinates)

f is the focal
in general,
length of the
camera

: aspect ratio (1 unless pixels are not square)

: skew (0 unless pixels are shaped like rhombi/parallelograms)

: principal point ((0,0) unless optical axis doesn’t intersect projection plane at origin)
31
 Focal length
• Can think of as “zoom”

24mm 50mm

200mm 800mm

• Related to field of view 32

 Projection matrix

intrinsics
projection rotation translation

33
Projection matrix
arbitrary 3D point
=

image plane

0 (in homogeneous image coordinates)

34
Distortion

No distortion Pin cushion Barrel

Radial distortion of the image

• Caused by imperfect lenses
• Deviations are most noticeable for rays that pass through
the edge of the lens

35
Correcting radial distortion

from Helmut Dersch 36

 Where does all this lead?
• We need it to understand stereo
• And 3D reconstruction
• It also leads into camera calibration, which is
usually done in factory settings to solve for
the camera parameters before performing an
industrial task.
• The extrinsic parameters must be determined.
• Some of the intrinsic are given, some are
solved for, some are improved.
37
Camera Calibration
The idea is to snap
images at different
depths and get a
lot of 2D-3D point
correspondences.

x1, y1, z1, u1, v1

x2, y2, z1, u2, v2
.
.
xn, yn, zn, un, vn

Then solve a system

of equations to get
camera parameters.
38
Stereo

CSE 455
Linda Shapiro

39
40
Amount of horizontal movement is
…
…inversely proportional to the distance from the camera

41
 Depth from Stereo
• Goal: recover depth by finding image coordinate x’
that corresponds to x
X
X

x z
x x’

f f
x'
C Baseline C’
B

42
 Depth from
X
disparity

x  x f
 z
O  O z x x’

f f
O Baseline O’
B

B f
disparity  x  x 
z
Disparity is inversely proportional to depth.
43
 Depth from Stereo
• Goal: recover depth by finding image coordinate x’ that
corresponds to x
• Sub-Problems
1. Calibration: How do we recover the relation of the cameras (if
not already known)?
2. Correspondence: How do we search for the matching point x’?

x'
44
 Correspondence Problem

x ?

• We have two images taken from cameras with different

intrinsic and extrinsic parameters

• How do we match a point in the first image to a point in the

second? How can we constrain our search?

45
Key idea: Epipolar constraint
X

x x’

x’

x’

Potential matches for x have to lie on the corresponding line l’.

Potential matches for x’ have to lie on the corresponding line l.

46
 Epipolar geometry: notation X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)

47
 Epipolar geometry: notation X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)
• Epipolar Lines - intersections of epipolar plane with image
planes (always come in corresponding pairs)
48
Example: Converging cameras

49
Example: Motion parallel to
image plane

50
 Epipolar constraint
X

x x’

• If we observe a point x in one image, where

can the corresponding point x’ be in the
other image?

51
 Epipolar constraint
X

x x’

x’

x’

• Potential matches for x have to lie on the corresponding

epipolar line l’.

• Potential matches for x’ have to lie on the corresponding

epipolar line l.
52
Epipolar constraint example

53
Epipolar constraint: Calibrated case
X

x x’

• Assume that the intrinsic and extrinsic parameters of the cameras are
known
• We can multiply the projection matrix of each camera (and the image
points) by the inverse of the calibration matrix to get normalized
image coordinates
• We can also set the global coordinate system to the coordinate
system of the first camera. Then the projection matrices of the two
cameras can be written as [I | 0] and [R | t] 54
Simplified Matrices for the 2
Cameras

= (R | T)

55
Epipolar constraint: Calibrated case
X = (x,1)T

x x’ = Rx+t

t
R

The vectors Rx, t, and x’ are coplanar

56
Epipolar constraint: Calibrated case
X

x x’

x   [ t  ( R x )]  0 xT E x  0 with E  [ t ] R

Essential Matrix E
(Longuet-Higgins, 1981)

57
The vectors Rx, t, and x’ are coplanar
Epipolar constraint: Calibrated case
X

x x’

x   [ t  ( R x )]  0 xT E x  0 with E  [ t ] R
• E x is the epipolar line associated with x (l' = E x)
• ETx' is the epipolar line associated with x' (l = ETx')
• E e = 0 and ETe' = 0
• E is singular (rank two)
• E has five degrees of freedom 58
Epipolar constraint: Uncalibrated
case
X

x x’

• The calibration matrices K and K’ of the two

cameras are unknown
• We can write the epipolar constraint in
terms of unknown normalized coordinates:
xˆ  E xˆ  0
T
xˆ  K x, xˆ   K  xˆ 
1 1
59
 Epipolar constraint: Uncalibrated
case
X

x x’

ˆxT E xˆ  0   T
x F x  0 with F  K E K
T 1

ˆx  K 1 x
Fundamental Matrix
xˆ   K  x 
1 (Faugeras and Luong, 1992)

60
Epipolar constraint: Uncalibrated
case
X

x x’

ˆxT E xˆ  0   T
x F x  0 with F  K E K
T 1

• F x is the epipolar line associated with x (l' = F x)

• FTx' is the epipolar line associated with x' (l' = FTx')
• F e = 0 and FTe' = 0

61
 The eight-point algorithm
x  (u, v,1)T , x  (u, v,1)  f11 
f 
 12 
 f13 
 f11 f12 f13  u   
 f 21 
u v 1  f 21 f 22 f 23   v   0 uu uv u vu vv v u v 1  f 22   0
 
 f 31 f 32 f 33  1  A  f 23 
f 
 31 
 f 32 
Smallest f 
 33 
eigenvalue of
Minimize: ATA
N

 i i
( x
i 1
T
F x ) 2

under the constraint

||F||2=1
62
 Comparison of estimation
algorithms

8-point Normalized 8-point Nonlinear least squares

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel 63
 Moving on to stereo…
Fuse a calibrated binocular stereo pair to
produce a depth image
image 1 image 2

Dense depth map

Many of these slides adapted from

64
Steve Seitz and Lana Lazebnik
 Depth from
X
disparity

x  x f
 z
O  O z x x’

f f
O Baseline O’
B

B f
disparity  x  x 
z
Disparity is inversely proportional to depth.
65
 Basic stereo matching algorithm

• If necessary, rectify the two stereo images to transform

epipolar lines into scanlines
• For each pixel x in the first image
– Find corresponding epipolar scanline in the right image
– Search the scanline and pick the best match x’
– Compute disparity x-x’ and set depth(x) = fB/(x-x’)
66
 Simplest Case: Parallel images
Epipolar constraint:
x E x   0, E  t  R
T

R=I t = (T, 0, 0)
x

0 0 0 
x’
E  t  R  0 0 T
 
t 0 T 0 

0 0 0  u   0 
     
u v 10 0  T  v   0 u v 1  T   0 Tv  Tv
0 T 0  1   Tv 
 
68
The y-coordinates of corresponding points are the same
Stereo image rectification

69
 Stereo image rectification
• Reproject image planes
onto a common plane
parallel to the line
between camera centers

• Pixel motion is horizontal

after this transformation

• Two homographies (3x3

transform), one for each
input image reprojection
 C. Loop and Z. Zhang. Computing
Rectifying Homographies for Stereo
Vision. IEEE Conf. Computer Vision
and Pattern Recognition, 1999.

70
 Example
Unrectified

Rectified

71
 Left Right

scanline

Matching cost
disparity

• Slide a window along the right scanline and

compare contents of that window with the
reference window in the left image
• Matching cost: SSD, SAD, or normalized correlation
72
 Correspondence search
Left Right

scanline

SSD 73
 Correspondence search
Left Right

scanline

Norm. corr 74
 Effect of window size

W=3 W = 20
• Smaller window
+ More detail
– More noise

• Larger window
+ Smoother disparity maps
– Less detail
– Fails near boundaries 75
Failures of correspondence search

Textureless surfaces Occlusions, repetition

Non-Lambertian surfaces, specularities 76

 Results with window search
Data

Window-based matching Ground truth

77
 How can we improve window-based matching?

• So far, matches are independent for each

point

• What constraints or priors can we add?

78
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at
most one matching point in the other image

79
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at most
one matching point in the other image
• Ordering
– Corresponding points should be in the same order in
both views

80
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at most
one matching point in the other image
• Ordering
– Corresponding points should be in the same order in
both views

Ordering constraint doesn’t 81hold

 Priors and constraints
• Uniqueness
– For any point in one image, there should be at most one
matching point in the other image
• Ordering
– Corresponding points should be in the same order in both
views
• Smoothness
– We expect disparity values to change slowly (for the most
part)

82
Stereo as energy minimization

• What defines a good stereo

correspondence?
1. Match quality
• Want each pixel to find a good match in the
other image
83
2. Smoothness
 Matching windows:
Similarity Measure Formula

Sum of Absolute Differences (SAD)

Sum of Squared Differences (SSD)

Zero-mean SAD

Locally scaled SAD

Normalized Cross Correlation (NCC)

SAD SSD NCC Ground truth

84
http://siddhantahuja.wordpress.com/category/stereo-vision/
 Real-time stereo

Nomad robot searches for meteorites in Antartica

http://www.frc.ri.cmu.edu/projects/meteorobot/index.html

• Used for robot navigation (and other

tasks)
– Several software-based real-time stereo 85
Stereo reconstruction pipeline
• Steps
– Calibrate cameras
– Rectify images
– Compute disparity
– Estimate depth
What will cause errors?

• Camera calibration errors

• Poor image resolution
• Occlusions
• Violations of brightness constancy (specular reflections)
• Large motions
• Low-contrast image regions
86
Multi-view stereo ?

87
 Using more than two images

Multi-View Stereo for Community Photo Collections

M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz
Proceedings of ICCV 2007,
88