MRGCV Computer Vision 1

Chapter 1
2D-3D geometry in homogenous
coordinates and projective camera

MRGCV Computer Vision

JM Martínez Montiel
MRGCV Computer Vision 2

Index

1. Geometry in homogeneous coordinates

– Points and lines in 2D
– Points and planes in 3D
– Lines in 3D
2. SVD
3. Perspective projection P matrix
4. Non-projective distortion.
5. Tsai camera model
6. Camera calibration with planar pattern
MRGCV Computer Vision 3

1. 2D points
Pixel inhomogeneous coordinates
in an image
𝑥𝑥
𝒙𝒙 = 𝑦𝑦 600
200 𝑥𝑥

in homogeneous coordinates
𝒙𝒙� = 𝑥𝑥,
� 𝑦𝑦,
� 𝑤𝑤
� 200
Connection inhomogeneous ⟺ 400
homogeneous
𝒙𝒙� = 𝑥𝑥,
� 𝑦𝑦, � =𝑤𝑤
� 𝑤𝑤 � 𝑥𝑥, 𝑦𝑦, 1 = 𝑤𝑤�
� 𝒙𝒙

𝒙𝒙, augmented vector

�
�
𝒙𝒙 = 𝑥𝑥, 𝑦𝑦, 1

𝑤𝑤=0,
� points at infinity, no 𝑦𝑦
inhomogeneous representation 200 −600 600 3
400 ≈ −1200 200 ≈ 1
1 −3 0 0
MRGCV Computer Vision 4

1. 2D lines 𝑙𝑙 = 𝑎𝑎 𝑏𝑏 𝑐𝑐
Also in homogenous coordinates 𝑐𝑐
𝒍𝒍̃ = 𝑎𝑎, 𝑏𝑏, 𝑐𝑐
− 𝑥𝑥
𝑎𝑎
Corresponding to line equation
𝒙𝒙𝒍𝒍̃ = 𝑎𝑎𝑎𝑎 + 𝑏𝑏𝑏𝑏 + 𝑐𝑐 = 0
� −𝑏𝑏, 𝑎𝑎
𝑐𝑐
− 𝑛𝑛𝑥𝑥 , 𝑛𝑛𝑦𝑦
Normalizing 𝑏𝑏
� , 𝑑𝑑
𝒍𝒍 = 𝑛𝑛� 𝑥𝑥 , 𝑛𝑛� 𝑦𝑦 , 𝑑𝑑 = 𝒏𝒏 � =1
𝒏𝒏
𝑎𝑎, 𝑏𝑏

𝑐𝑐
𝑦𝑦 𝑑𝑑 =
𝑎𝑎2 + 𝑏𝑏2
MRGCV Computer Vision 5

1. 2D points and lines

2 points define a line

(+ )(− )
𝒍𝒍̃ = 𝒙𝒙
�1 × 𝒙𝒙
�2
× =
2 lines define a point
(− )(+ )

� = 𝒍𝒍̃1 × 𝒍𝒍̃ 2
𝒙𝒙
× =
Example 0   4  1 
 4  (+ )(− )
line defined by points 1 and 0 l ≈  
1 1  − 4
parallel lines meet at infinity
vector normal (b,-a)
× =
directional vector (a, b) a   a   b 
point as the intersection of 2 lines b  y  b  x = − a 
   
 c   d   0 
MRGCV Computer Vision 6

1. 3D points
Inhomogeneous point
coordinates
𝒙𝒙 = 𝑥𝑥, 𝑦𝑦, 𝑧𝑧
𝑍𝑍 𝑤𝑤
� 𝑥𝑥, 𝑦𝑦, 𝑧𝑧, 1 , 𝑤𝑤
�≠0
Homogeneous coordinates
𝒙𝒙� = 𝑥𝑥,
� 𝑦𝑦,
� 𝑧𝑧,̃ 𝑤𝑤
�

Connection inhomogeneous ⟺ 𝑧𝑧 𝑋𝑋
homogeneous 𝑥𝑥,
� 𝑦𝑦,
� 𝑧𝑧,̃ 0
𝑥𝑥
𝒙𝒙� = 𝑥𝑥,
� 𝑦𝑦, � =𝑤𝑤
� 𝑧𝑧,̃ 𝑤𝑤 � 𝑥𝑥, 𝑦𝑦, 𝑧𝑧, 1 = 𝑤𝑤�
� 𝒙𝒙 𝑦𝑦
𝑌𝑌
𝒙𝒙, augmented vector
�

𝑤𝑤=0,
� points at infinity
MRGCV Computer Vision 7

1. 3D planes
𝑎𝑎, 𝑏𝑏, 𝑐𝑐
In homogeneous coordinates
� = 𝑎𝑎, 𝑏𝑏, 𝑐𝑐, 𝐷𝐷
𝒎𝒎 𝑛𝑛�𝑥𝑥 , 𝑛𝑛�𝑦𝑦 , 𝑛𝑛�𝑧𝑧

Corresponding to the plane

equation
�
𝒙𝒙𝑚𝑚
� = 𝑎𝑎𝑎𝑎 + 𝑏𝑏𝑏𝑏 + 𝑐𝑐𝑐𝑐 + 𝐷𝐷 = 0

normalizing
� , 𝑑𝑑
𝒎𝒎 = 𝑛𝑛� 𝑥𝑥 , 𝑛𝑛� 𝑦𝑦 , 𝑛𝑛� 𝑧𝑧 , 𝑑𝑑 = 𝒏𝒏 �
𝒏𝒏 𝐷𝐷
𝑑𝑑 =
𝑎𝑎2 + 𝑏𝑏2 + 𝑐𝑐 2
MRGCV Computer Vision 8

1. 3D lines
𝒓𝒓� = 𝜇𝜇�
𝒑𝒑 + 𝜆𝜆�
𝒒𝒒
𝒓𝒓 = 1 − 𝜆𝜆 𝒑𝒑 + 𝜆𝜆𝒒𝒒
Given two points in inhomogeneous coordintaes, 𝒑𝒑, 𝒒𝒒. The ℛ
coordiantes of other point 𝒓𝒓 in the line ℛ is computed 𝒑𝒑
as: 𝒓𝒓 = 1 − 𝜆𝜆 𝒑𝒑 + 𝜆𝜆𝒒𝒒
𝒒𝒒
If 0 ≤ 𝜆𝜆 ≤ 1, it codes the segment with ending points 𝒑𝒑, 𝒒𝒒

In homogeneous coordinates
𝒓𝒓� = 𝜇𝜇 𝒑𝒑
� + 𝜆𝜆𝒒𝒒
�
� 𝟏𝟏
𝒎𝒎
It can be represented also a the intersection of two planes
𝑎𝑎1 𝑥𝑥 + 𝑏𝑏1 𝑦𝑦 + 𝑐𝑐1𝑧𝑧 + 𝐷𝐷1 = 0 ℛ
𝑎𝑎2𝑥𝑥 + 𝑏𝑏2 𝑦𝑦 + 𝑐𝑐2𝑧𝑧 + 𝐷𝐷2 = 0 � 𝟐𝟐
𝒎𝒎
Transformations betweew points and planes coding by means of
SVD
� 𝟏𝟏 �
𝒎𝒎 𝒙𝒙 = 𝟎𝟎 Modificación de: https://commons.wikimedia.org/

� 𝟐𝟐 �
𝒎𝒎 𝒙𝒙 = 𝟎𝟎 wiki/File:Is-plane-plane.png Autor: Ag2gaeh
MRGCV Computer Vision 9

2. SVD. Linear mapping

𝑓𝑓: ℛ 𝑛𝑛 → ℛ 𝑚𝑚
𝒙𝒙 ↝ 𝒚𝒚 = 𝑓𝑓 𝒙𝒙
ℛ 𝑛𝑛 ℛ 𝑚𝑚
ker ( f )
0

im( f )

𝑟𝑟 = rank 𝑨𝑨
ker 𝑓𝑓 = 𝒙𝒙 ∈ ℛ 𝑛𝑛 ⁄𝑨𝑨𝒙𝒙 = 𝟎𝟎 vector space dimension 𝑛𝑛 − 𝑟𝑟
im 𝑓𝑓 = 𝒚𝒚 ∈ ℛ 𝑚𝑚 ⁄∃𝒙𝒙 ∈ ℛ 𝑛𝑛 𝑨𝑨𝒙𝒙 = 𝒚𝒚 vector space dimension 𝑟𝑟
MRGCV Computer Vision 10

2. SVD

𝑼𝑼, 𝑺𝑺, 𝑽𝑽 = svd 𝑨𝑨 𝑼𝑼𝑚𝑚×𝑚𝑚 = 𝒖𝒖1 , ⋯ , 𝒖𝒖𝑚𝑚

𝑨𝑨𝒎𝒎×𝒏𝒏 = 𝑼𝑼𝒎𝒎×𝒎𝒎 𝑺𝑺𝒎𝒎×𝒏𝒏 𝑽𝑽𝑻𝑻𝒏𝒏×𝒏𝒏 𝑽𝑽𝑛𝑛×𝑛𝑛 = 𝒗𝒗1 , ⋯ , 𝒗𝒗𝑛𝑛

𝑤𝑤1 > 𝑤𝑤2 > 𝑤𝑤3 ⋯> 𝑤𝑤𝑟𝑟 >0 𝑤𝑤𝑟𝑟+1 = 0, ⋯ 𝑤𝑤min(𝑚𝑚,𝑛𝑛) = 0
𝑤𝑤1 ⋯ 0
⋮ ⋱ ⋮
𝑚𝑚 > 𝑛𝑛 0 ⋯ 𝑤𝑤𝑛𝑛
𝑺𝑺𝑚𝑚×𝑛𝑛 = 𝟎𝟎 𝑚𝑚−𝑛𝑛 ×𝑛𝑛
𝑤𝑤1 ⋯ 0
𝑚𝑚 ≤ 𝑛𝑛 ⋮ ⋱ ⋮ � 𝟎𝟎𝑚𝑚× 𝑛𝑛−𝑚𝑚
0 ⋯ 𝑤𝑤𝑚𝑚
MRGCV Computer Vision 11

2. SVD
𝑟𝑟 𝑛𝑛 − 𝑟𝑟
𝑚𝑚 > 𝑛𝑛
𝑤𝑤1
⋱ 𝒗𝒗1𝑇𝑇
𝑤𝑤𝑟𝑟 ⋮
𝒗𝒗𝑇𝑇𝑟𝑟
𝑨𝑨 = 𝒖𝒖1 ⋯ 𝒖𝒖𝑟𝑟 𝒖𝒖𝑟𝑟+1 ⋯ 𝒖𝒖𝑚𝑚 0
⋱ 𝒗𝒗𝑇𝑇𝑟𝑟+1 ker 𝑓𝑓 basis
0 ⋮ all elements
𝒏𝒏×𝒏𝒏
Im 𝑓𝑓 basis 𝒗𝒗𝑇𝑇𝑛𝑛 map to 0
all elements 𝟎𝟎 𝑚𝑚−𝑛𝑛 ×𝑛𝑛
have inverse image
MRGCV Computer Vision 12

2. SVD

𝑟𝑟 𝑚𝑚 − 𝑟𝑟
𝑚𝑚 ≤ 𝑛𝑛
𝑤𝑤1 𝒗𝒗1𝑇𝑇
⋱ ⋮
𝑤𝑤𝑟𝑟 𝒗𝒗𝑇𝑇𝑟𝑟
𝑨𝑨 = 𝒖𝒖1 ⋯ 𝒖𝒖𝑟𝑟 𝒖𝒖𝑟𝑟+1 ⋯ 𝒖𝒖𝑚𝑚 𝟎𝟎𝑚𝑚× 𝑛𝑛−𝑚𝑚
0 𝒗𝒗𝑇𝑇𝑟𝑟+1 ker 𝑓𝑓 basis
⋱ ⋮ all elements
0 𝑚𝑚×𝑚𝑚 𝒗𝒗𝑇𝑇𝑛𝑛 map to 0
Im 𝑓𝑓 basis
all elements
have inverse image
MRGCV Computer Vision 13

2. SVD applications
a) Numerical matrix rank estimation
– Number of wj≠0; wj<10e-6, float; wj<10-e12, double
b) Solving homogenous linear systems
𝑨𝑨𝑨𝑨 = 𝟎𝟎
1. The solution is vector space. A basis for computing the whole
null space, 𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉
2. 𝑟𝑟 = rank 𝑨𝑨
3. Null space dimensión is 𝑛𝑛 − 𝑟𝑟 𝒗𝒗1𝑇𝑇
4. Null space basis 𝑛𝑛 − 𝑟𝑟 vectors, from 𝑽𝑽 ⋮
ℛ 𝑛𝑛 ℛ 𝑚𝑚 𝒗𝒗𝑇𝑇𝑟𝑟
ker ( f ) 𝒗𝒗𝑇𝑇𝑟𝑟+1
0 ker 𝑓𝑓 basis
⋮ all elements
im( f ) 𝒗𝒗𝑇𝑇𝑛𝑛 map to 0

𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 , any linear combination of 𝒗𝒗𝑟𝑟+1 , ⋯ , 𝒗𝒗𝑛𝑛

MRGCV Computer Vision 14

2. SVD applications
c) Solving non-homogeneous linear systems
𝑨𝑨𝑨𝑨 = 𝒃𝒃
1. The solution is an affine space. We compute the whole space
𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 + 𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒕𝒕
2. 𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 𝑨𝑨𝑨𝑨 = 0
3. 𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑 𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒕𝒕 = 𝑨𝑨+ 𝒃𝒃
inv(𝑤𝑤1 ) 1
𝑖𝑖𝑖𝑖 𝑤𝑤𝑗𝑗 ≠ 0
𝑨𝑨+ = 𝒗𝒗1 ⋯ 𝒗𝒗𝑚𝑚 ⋱ 𝑼𝑼𝑇𝑇 inv(𝑤𝑤𝑗𝑗 ) �𝑤𝑤𝑗𝑗
inv(𝑤𝑤𝑚𝑚 ) 0 𝑖𝑖𝑖𝑖 𝑤𝑤𝑗𝑗 = 0
𝑨𝑨+ Moore-Penrose pseudoinverse
𝑟𝑟 < 𝑛𝑛 underconstrained
𝑟𝑟 = 𝑛𝑛 unique solution
n< 𝑟𝑟 overconstrained least mean squared error solution
𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑 = min 𝑨𝑨𝑨𝑨 − 𝒃𝒃
𝒙𝒙
MRGCV Computer Vision 15

Geometric intuition
non-singular A

𝑨𝑨2×2 Ax = b
x
𝑟𝑟 = 2, full rank
ℛ 2 ℛ2

𝑨𝑨𝑨𝑨 = 𝒃𝒃 𝑏𝑏
𝒙𝒙
MRGCV Computer Vision 16

Geometric intuition
singular A
𝑨𝑨2×2
𝑟𝑟 = 1, rank deficient
𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 ℛ 2 ℛ2
𝑨𝑨𝑨𝑨 = 𝟎𝟎 𝑨𝑨𝑨𝑨 = 𝒄𝒄
𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑 = 𝑨𝑨+ 𝒄𝒄 𝑨𝑨𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑
𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 + 𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒕𝒕

𝑨𝑨𝑨𝑨 = 𝒅𝒅 𝒅𝒅 𝒄𝒄
𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑 = 𝑨𝑨+ 𝒅𝒅
𝒙𝒙𝒉𝒉𝒉𝒉𝒉𝒉 + 𝒙𝒙𝒑𝒑𝒑𝒑𝒑𝒑𝒕𝒕
Fig 2.6.1 Press et at. 2002
MRGCV Computer Vision 17

2. SVD applications

d) Numerical approximation of rank deficient matrices

i. Numerically computed matrix never are rank deficient
ii. 𝑨𝑨𝑛𝑛×𝑛𝑛 Can be approximated by the rank 𝑟𝑟 matrix
𝑟𝑟

𝑨𝑨𝑟𝑟 = � 𝑤𝑤𝑘𝑘 𝒖𝒖𝑘𝑘 𝒗𝒗𝑇𝑇𝑘𝑘

𝑘𝑘=1
𝑨𝑨𝑟𝑟 = min 𝑨𝑨 − 𝑨𝑨𝑙𝑙 ∀𝑨𝑨𝑙𝑙 rank(𝑟𝑟)
𝑨𝑨𝑙𝑙
MRGCV Computer Vision 18

3D point intersection of 3D planes 3D planes containing 3D points

• N Planes, N rows in the linear system • N Planes, N rows in the linear system

𝝅𝝅1 , 𝝅𝝅2, ⋯ , 𝝅𝝅𝑛𝑛 𝑿𝑿1 , 𝑿𝑿2, ⋯ , 𝑿𝑿𝑛𝑛

𝝅𝝅1𝑇𝑇 𝑿𝑿1𝑇𝑇
𝑨𝑨𝑛𝑛×4 𝑿𝑿4×1 = 0 𝑨𝑨 = ⋮ 𝑨𝑨𝑛𝑛×4 𝝅𝝅4×1 = 0 𝑨𝑨 = ⋮
𝝅𝝅𝑇𝑇𝑛𝑛 𝑿𝑿𝑇𝑇𝑛𝑛
𝑨𝑨 = 𝑼𝑼𝑼𝑼𝑽𝑽𝑻𝑻 𝑽𝑽𝑛𝑛×4 = 𝒗𝒗1, ⋯ , 𝒗𝒗𝑛𝑛 𝑨𝑨 = 𝑼𝑼𝑼𝑼𝑽𝑽𝑻𝑻 𝑽𝑽𝑛𝑛×4 = 𝒗𝒗1, ⋯ , 𝒗𝒗4
rank(𝐴𝐴) = 1, alll the points are the same.
rank(𝐴𝐴) = 1, alll the planes are the same. Solution Solution vector space dimensión 3
vector space dimensión 3
𝜋𝜋 ≈ 𝜆𝜆2 𝒗𝒗2 + 𝜆𝜆3 𝒗𝒗3 + 𝜆𝜆4 𝒗𝒗4
𝑋𝑋 ≈ 𝜆𝜆2 𝒗𝒗2 + 𝜆𝜆3 𝒗𝒗3 + 𝜆𝜆4 𝒗𝒗4 rank(𝐴𝐴) = 2, two points in a line, plane
rank(𝐴𝐴) = 2, planes intersect in a line. bundle
Solution vector space dimensión 2 Solution vector space dimensión 2
𝑋𝑋 ≈ 𝜆𝜆3 𝒗𝒗3 + 𝜆𝜆4 𝒗𝒗4 𝜋𝜋 ≈ 𝜆𝜆3 𝒗𝒗3 + 𝜆𝜆4 𝒗𝒗4
rank(𝐴𝐴) = 3, planes intersect in a point. rank(𝐴𝐴) = 3, planes contained in a plane.
Solution vector space dimensión 1 Solution vector space dimensión 1

𝑋𝑋 ≈ 𝜆𝜆4 𝒗𝒗4 𝜋𝜋 ≈ 𝜆𝜆4 𝒗𝒗4

rank(𝐴𝐴) = 4, No solution rank(𝐴𝐴) = 4, No solution
Only trivial solution. Only trivial solution.
MRGCV Computer Vision 19

Pinhole camera
virtual image
m(x,y) image plane optical axis
plane
f
optical centre

optical axisz z
x f optical axis
x M(x,y)
y y M (X,Y,Z)
M (X,Y,Z)
pinhole camera virtual image plane pinhole camera
X Y X Y
x = − f y = − f x = f y = f
Z Z Z Z

– accurately models camera geometry

– image plane in front of the optical centre
– infinite depth of field
2.2 Photometric image formation. Szleliski (2011).
1.2.2 Intrinsic Parameters. Forsyth (2012)
MRGCV Computer Vision 20

3.- Perspective projection

Optical centre
𝑶𝑶 Z

X x 𝒙𝒙[𝑥𝑥, 𝑦𝑦]
𝑿𝑿[𝑋𝑋, 𝑌𝑌, 𝑍𝑍]
Y f y

• The camera detect “projection rays”

• Projection ray determined by the optical centre 𝑂𝑂, and the
scene point 𝑿𝑿
• The image, 𝒙𝒙 𝑥𝑥, 𝑦𝑦 , is the way the camera detects the ray
MRGCV Computer Vision 21

3. Pinhole geometry
Triangle similarity OAX and Oax,
Optical Centre Rule of three
O
f →Z
Z x→X
Then
X f x[x,y] X
x= f
Z
Y X[X,Y,Z] Similarly in Y axis

Focusing in the 𝑋𝑋 axis Y

y= f
Z Z
X X[X,Z]
x X Z
O
a x A

f
MRGCV Computer Vision 22

3. Perspective projection

O Z
𝐶𝐶
X x x[x,y]
X[X,Y,Z]
Y f y
X Y
Image of point [𝑋𝑋, 𝑌𝑌, 𝑍𝑍] is [𝑥𝑥, 𝑦𝑦] x= f y= f
Z Z
In homogeneous coordinates
𝑠𝑠𝑠𝑠 𝜆𝜆𝜆𝜆𝜆𝜆𝜆𝜆 𝑋𝑋 𝑤𝑤𝑋𝑋
𝑥𝑥 = = = 𝑓𝑓 𝑠𝑠𝑥𝑥 𝑓𝑓 0 0 0
𝑠𝑠 𝜆𝜆𝑤𝑤𝑤𝑤 𝑍𝑍 𝑠𝑠𝑠𝑠 = 𝜆𝜆 0 𝑤𝑤𝑤𝑤
𝑓𝑓 0 0
𝑠𝑠𝑦𝑦 𝜆𝜆𝜆𝜆𝜆𝜆𝜆𝜆 𝑌𝑌 𝑤𝑤𝑤𝑤
𝑦𝑦 = = = 𝑓𝑓 𝑠𝑠 0 0 1 0
𝑠𝑠 𝜆𝜆𝑤𝑤𝑤𝑤 𝑍𝑍 𝑤𝑤 𝐶𝐶
MRGCV Computer Vision 23

3 From image to pixels

81 98 118 145 153 146 213 214 126 91 91 108 115 114 111

47 113 118 139 152 199 ## 160 ## 115 90 84 109 104 95

37 111 113 135 146 148 163 201 214 181 96 84 111 119 109

74 111 104 138 170 129 125 195 142 164 106 88 126 118 108

51 111 107 140 161 122 112 176 121 127 96 84 123 122 125

42 117 101 136 121 92 105 128 136 115 137 90 122 118 120

42 82 91 86 99 75 100 111 145 120 107 84 123 119 130

7 56 66 66 103 67 90 115 139 123 86 95 119 119 114

11 52 48 51 104 60 94 98 141 108 81 91 103 115 118

11 34 37 39 113 77 90 93 118 106 77 100 106 121 119

2 22 22 40 116 113 83 135 116 141 97 113 97 132 124

3 26 27 41 104 103 80 127 107 136 105 153 174 145 134

3 27 29 51 104 104 102 114 130 131 120 144 171 175 158

3 30 32 60 118 96 105 106 105 119 112 141 141 131 144

7 31 41 103 48 111 110 97 113 120 105 147 135 119 129

Optical 2D image image gathered

2 34 55 217 78 121 94 93 99 109 106 130 199 144 118

projected on the sensor by the sensor array in the computer memory

coding the 2D image

https://drive.google.com/file/d/1uolUCdk-
TwqIPfySovsNVrMsX-HKAk9I/view?usp=sharing
MRGCV Computer Vision 24
3. From image to pixels
Digital sensor is a pixel matrix: N_cols
N_row, N_cols number of rows/columns

𝑑𝑑𝑥𝑥 , 𝑑𝑑𝑦𝑦 pixel size (meter) 𝑑𝑑𝑥𝑥

𝑐𝑐𝑥𝑥 , 𝑐𝑐𝑦𝑦 principal point (pixel) 𝑢𝑢
𝑠𝑠𝑘𝑘 , skew, usually 0
𝑥𝑥
𝑢𝑢 = + 𝑐𝑐𝑥𝑥 𝑥𝑥 = 𝑢𝑢 − 𝑐𝑐𝑥𝑥 𝑑𝑑𝑥𝑥

𝑑𝑑𝑦𝑦
𝑑𝑑𝑥𝑥 𝑐𝑐𝑥𝑥 , 𝑐𝑐𝑦𝑦 𝑥𝑥
𝑣𝑣

N_row
𝑦𝑦 𝑦𝑦 = 𝑣𝑣 − 𝑐𝑐𝑦𝑦 𝑑𝑑𝑦𝑦
𝑣𝑣 = + 𝑐𝑐𝑦𝑦
𝑑𝑑𝑦𝑦
𝑦𝑦
1� 𝑠𝑠𝑘𝑘 𝑐𝑐𝑥𝑥
𝑢𝑢 𝑑𝑑𝑥𝑥 𝑥𝑥 𝑢𝑢, 𝑣𝑣 (pixel)
𝑣𝑣 = 0 1� 𝑐𝑐𝑦𝑦 𝑦𝑦 𝑥𝑥, 𝑦𝑦 (meter)
1 𝑑𝑑𝑦𝑦 1 Intrinsics, canonical
0 0 1 calibration 𝜃𝜃int perspective
projection
1� 𝑠𝑠𝑘𝑘 𝑐𝑐𝑥𝑥 𝑓𝑓 𝑤𝑤𝑋𝑋 𝑤𝑤𝑋𝑋
𝑠𝑠𝑢𝑢 𝑑𝑑𝑥𝑥 0 0 0 𝑠𝑠𝑢𝑢 𝑓𝑓𝑥𝑥 𝑠𝑠𝑘𝑘 𝑐𝑐𝑥𝑥 1 0 0 0
𝑤𝑤𝑤𝑤 𝑤𝑤𝑤𝑤
𝑠𝑠𝑠𝑠 = 𝜆𝜆 1� 𝑠𝑠𝑠𝑠 = 𝜆𝜆 0 𝑓𝑓𝑦𝑦 𝑐𝑐𝑦𝑦 0 1 0 0
0 𝑑𝑑𝑦𝑦 𝑐𝑐𝑦𝑦 0 𝑓𝑓 0 0
𝑤𝑤𝑤𝑤 𝑤𝑤𝑤𝑤
𝑠𝑠 0 0 1 0 𝑠𝑠 0 0 1 0 0 1 0
𝑤𝑤 𝑤𝑤 𝐶𝐶
0 0 1 𝐶𝐶
𝑲𝑲,
𝑓𝑓 𝑓𝑓
𝑓𝑓𝑥𝑥 = , 𝑓𝑓 = hortiz. Vert. focal length (pixel), 𝑓𝑓 focal length (meter)
𝑑𝑑𝑥𝑥 𝑦𝑦 𝑑𝑑𝑦𝑦
MRGCV Computer Vision 25

3.Perspective projection. P Matrix

� 𝑐𝑐 = 𝑻𝑻𝐶𝐶𝐶𝐶 𝑿𝑿
𝑿𝑿 � 𝑊𝑊 =
Intrinsics, canonical
extrinsic 𝑹𝑹𝐶𝐶𝐶𝐶 𝒕𝒕𝐶𝐶𝐶𝐶 �
calibration 𝜃𝜃int perspective 𝜃𝜃
parameters ext 𝑿𝑿𝑊𝑊
projection 0 1
𝑲𝑲,
𝑠𝑠𝑥𝑥 𝑓𝑓𝑥𝑥 𝑠𝑠𝑘𝑘 𝑐𝑐𝑥𝑥 𝑋𝑋
1 0 0 0
𝑹𝑹𝐶𝐶𝐶𝐶 𝒕𝒕𝐶𝐶𝐶𝐶 𝑌𝑌
𝑠𝑠𝑠𝑠 = 𝜆𝜆 0 𝑓𝑓𝑦𝑦 𝑐𝑐𝑦𝑦 0 1 0 0 𝑥𝑥 𝑦𝑦
𝟎𝟎 1 𝑍𝑍
𝑠𝑠 0 0 1 0 0 1 0
1 𝑊𝑊 𝑍𝑍

𝑋𝑋 𝐶𝐶
𝑷𝑷 = 𝑲𝑲 𝑹𝑹𝐶𝐶𝐶𝐶 �𝒕𝒕𝐶𝐶𝐶𝐶
𝑹𝑹𝑊𝑊𝑊𝑊 , 𝒕𝒕𝑊𝑊𝑊𝑊
� 𝑊𝑊 𝜆𝜆 ≠ 0
𝑥𝑥� = 𝜆𝜆𝑷𝑷𝑿𝑿 𝑿𝑿[𝑋𝑋, 𝑌𝑌, 𝑍𝑍]

𝑠𝑠𝑢𝑢 𝑋𝑋
𝑠𝑠𝑣𝑣 = 𝜆𝜆𝑷𝑷 𝑌𝑌 𝒙𝒙 𝑥𝑥, 𝑦𝑦 , [𝑢𝑢, 𝑣𝑣]
𝑠𝑠 𝑍𝑍
1 𝑊𝑊
MRGCV Computer Vision 26
Cushion distortion

4. Lens radial distortion xd

xu
xd

C
Barrel
Distortion

𝜅𝜅1 , 𝜅𝜅2 , 𝜅𝜅3 distortion coefficients

𝑥𝑥𝑑𝑑 = 𝑥𝑥𝑢𝑢 1 + 𝜅𝜅1 𝑟𝑟𝑢𝑢2 + 𝜅𝜅2 𝑟𝑟𝑢𝑢4 + 𝜅𝜅3 𝑟𝑟𝑢𝑢6
Theoretical undistorted
coordinates 𝑦𝑦𝑑𝑑 = 𝑦𝑦𝑢𝑢 1 + 𝜅𝜅1 𝑟𝑟𝑢𝑢2 + 𝜅𝜅2 𝑟𝑟𝑢𝑢4 + 𝜅𝜅3 𝑟𝑟𝑢𝑢6
Distorted
coordinates
𝑟𝑟𝑢𝑢 = 𝑥𝑥𝑢𝑢2 + 𝑦𝑦𝑢𝑢2
From distorted to
undistorted, non-linear
equation, iterative
Szeliski, Section 2.1.6, lens distortion.
OpenCV “Camera calibration and 3D reconstruction.
https://docs.opencv.org/3.4.11/d9/d0c/group__calib3d.html
MRGCV Computer Vision 27

5. Camera calibration (summary)

• Projection model: 𝒖𝒖 = 𝑷𝑷 𝑿𝑿𝑤𝑤 , 𝜃𝜃𝑖𝑖𝑖𝑖𝑖𝑖 , 𝜃𝜃𝑒𝑒𝑒𝑒𝑒𝑒
1. Transform to camera: 𝑿𝑿𝒄𝒄 = 𝑹𝑹𝑐𝑐𝑐𝑐 𝑿𝑿𝑤𝑤 + 𝒕𝒕𝑐𝑐𝑐𝑐
2. Theoretical projection normalize 𝑋𝑋𝑐𝑐 𝑌𝑌𝑐𝑐
camera 𝑓𝑓 = 1: 𝑥𝑥𝑢𝑢 = 𝑦𝑦𝑢𝑢 =
𝑍𝑍𝑐𝑐 𝑍𝑍𝑐𝑐
𝑥𝑥𝑑𝑑 = 𝑥𝑥𝑢𝑢 1 + 𝜅𝜅1 𝑟𝑟𝑢𝑢2 + 𝜅𝜅2 𝑟𝑟𝑢𝑢4 + 𝜅𝜅3 𝑟𝑟𝑢𝑢6
3. Radial distortion:
𝑦𝑦𝑑𝑑 = 𝑦𝑦𝑢𝑢 1 + 𝜅𝜅1 𝑟𝑟𝑢𝑢2 + 𝜅𝜅2 𝑟𝑟𝑢𝑢4 + 𝜅𝜅3 𝑟𝑟𝑢𝑢6

4. Transform to pixels:
𝑢𝑢 = 𝑓𝑓𝑥𝑥 𝑥𝑥𝑑𝑑 + 𝑐𝑐𝑥𝑥
𝑣𝑣 = 𝑓𝑓𝑦𝑦 𝑦𝑦𝑑𝑑 + 𝑐𝑐𝑦𝑦
• Parameter to Calibrate:
• Extrinsic, 𝜃𝜃𝑒𝑒𝑒𝑒𝑒𝑒 𝑹𝑹𝑐𝑐𝑐𝑐 , 𝒕𝒕𝑐𝑐𝑐𝑐
• Intrinsic, 𝜃𝜃𝑖𝑖𝑖𝑖𝑖𝑖 𝑓𝑓𝑥𝑥 , 𝑓𝑓𝑦𝑦 , 𝑐𝑐𝑥𝑥 , 𝑐𝑐𝑦𝑦 , 𝜅𝜅1 , 𝜅𝜅2 , 𝜅𝜅3
MRGCV Computer Vision 28

5. Camera model
Relation between pixels and the projection rays

By Amfeli - Own work, CC BY 4.0, By Romski - Own work, CC BY-SA 3.0,

https://commons.wikimedia.org/w/i https://commons.wikimedia.org/w/index.
ndex.php?curid=65481123 php?curid=15995059

[1] Ángel Miguel Sánchez

(https://commons.wikimedia.org/wiki/
File:Sta_Maria_Naranco.jpg), „
Sta Maria Naranco“, modified,
https://creativecommons.org/licenses/
by-sa/3.0/es/deed.en
MRGCV Computer Vision 29

5. calibration
• Standard cameras (e.g mobile phone):
– Optical zoom modifies intrinsic parameters
– Focus (autofocus) also modifies the intrinsic
parameters
• Metric cameras (rather rare today because Price):
– Known calibration
MRGCV Computer Vision 30

5. Planar pattern calibration

• Software
– OpenCV
http://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/ca
mera_calibration.html
– Vicalib
https://github.com/arpg/vicalib
– Kalibr
https://github.com/ethz-asl/kalibr
• Better result if images with point of view producing well
conditioned equations Enough 6 -8 of the calibration
pattern.
• Planar pattern of known proportion.
MRGCV Computer Vision 31

5 Calibration patterns

Vicalib

Aruco OpenCV

OpenCV
MRGCV Computer Vision 32

OpenCV calibration example

MRGCV Computer Vision 33

Planar pattern calibration

1. Nice example of self-calibration

𝑗𝑗 𝑗𝑗 𝑗𝑗 2
𝑗𝑗
argmin � 𝐮𝐮𝑖𝑖 − 𝐏𝐏 𝐗𝐗 𝑖𝑖 , 𝜃𝜃𝑖𝑖𝑖𝑖𝑖𝑖 , 𝜃𝜃𝑒𝑒𝑒𝑒𝑒𝑒
𝑗𝑗 𝑗𝑗
𝜃𝜃𝑒𝑒𝑒𝑒𝑒𝑒 ,𝜃𝜃𝑖𝑖𝑖𝑖𝑖𝑖 ,𝐗𝐗 𝑖𝑖 𝑖𝑖,𝑗𝑗
1. The same camera in all the views.
2. Pattern approximately planar.
3. Matches exploiting the prior knowledge of the pattern.
4. Initial guess exploiting approximate planarity [Zhang 2000].
5. Self-calibrated BA can estimate :
1. All camera poses (up to scale)
2. The accurate pattern shape including the lack of planarity (up to
scale)
3. The camera intrinsic parameters

Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on
pattern analysis and machine intelligence, 22(11), 1330-1334.
MRGCV Computer Vision 34

Bibliography
Geomtery+camera models:

Richard Szeliski, “Computer Vision: Algorithms and Applications” 2nd Edition

2022. http://research.microsoft.com/en-us/um/people/szeliski/Book/
Sección 2.1 Geometric primitives and transformations.
Planar pattern calibration
Tutorioal OpenCV
http://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camer
a_calibration.html
Vicalib, herramienta calibración incluyendo cámara fish-eye
https://github.com/arpg/vicalib
SVD
William H. Press, Brian P. Flannery, Saul A. Teukolsky, William T. Vetterling
:”Numerical Recipes in C : The Art of Scientific Computing”. Second
Edition 2002. Capítulo 2.6
Golub G.H y Van Loan C.F. “Matrix Computations”. Second Edition
The John Hopkins University Press. 1989