CS 598 3D Vision:

Multi-View Geometry

Shenlong Wang
UIUC

Some materials borrowed from Matthew O’Toole, Kris Kitani, Jianxiong Xiao, Derek Hoeim, Sanja Fidler 1
Logistics

• Quiz 1 – If we didn’t reach out, it's satisfactory!

• Quiz 2 – Will be out tonight (due next Tuesday).
• Group assignment is out!
• Survey due date has been extended (Sept 26 → Oct 3). Do meet earlier to
conduct a literature review and select 25+ papers, then organize them into
groups and assign jobs within the group.
• Role-playing group: 1) discuss your tackling plans with us during Thursday
office hours, the week before your presentation, or arrange a quick ad-hoc
meeting. 2) share your presentation for feedback three days before your
group presentation.
2
Today’s Agenda

● Camera Calibration
● Structure from Motion
● Other Cameras

3
Big picture: 3 key components in 3D
3D Points
(Structure)

Camera
Correspondences
(Motion)

Angjoo Kanazawa
Camera Calibration
How do I know K?

3D object
Image Plane

Camera Center

intrinsic extrinsic
parameters parameters

5
Camera Calibration
● Inputs : A collection of images with points whose 2D image coordinates and 3D world
coordinates are known.
● Outputs: The 3×3 camera intrinsic matrix, the rotation and translation of each image.

Capture multiple images of the Finding camera parameters by

checkerboard from different viewpoints Find checkerboard corners
minimizing 3D-2D reprojection err

6
Image credit: OpenCV
 Camera Calibration
● Minimizing the reprojection error

Extrinsic Intrinsic Detection Corner Points Perspective Projection Known 3D Location

7
 Camera Calibration

https://www.mathworks.com/help/vision/camera-calibration.html
https://github.com/ethz-asl/kalibr
8
https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
Structure-from-Motion

Each pair of 2D-2D correspondence establish

triagulaler relationships
9
Structure-from-Motion

10
Image credit: Jianxiong Xiao
Structure-from-Motion

● Establish 2D-2D correspondences across images

● Jointly refine camera pose and 3D points in an optimization framework

11
Two View Reconstruction

Bundle
Match
Adjustment

12
Keypoints Detection
● Step 1: Detect distinctive keypoints that are suitable for matching

13
Descriptor for each point
● Step 2: Compute visual descriptors (SIFT features)

SIFT

SIFT

14
Descriptor for each point
● Step 3: Measure pairwise distance / similarity between features

SIFT SIFT

SIFT SIFT

15
Match Points
● Step 3: Measure pairwise distance / similarity between features

SIFT SIFT

SIFT SIFT

16
Match Points
SIFT (scale-invariant feature transform)
● Step 1: Detect distinctive keypoints that are suitable for matching
● Step 2: Compute oriented histogram gradient features
● Step 3: Measure distance between each pair

17
Match Points
● How many pair-wise matching I need to conduct?

SIFT SIFT

SIFT SIFT

18
Match Points
● What if there are bad matches?

SIFT SIFT

SIFT SIFT

19
Match Points in Practice

How can we make SIFT matching faster than exhaustive search?

- Approximate nearest neighbor search
- Hashing, KD-tree, etc.

How can we ensure a pair of match is good?

- Ratio test: my nearest neighbor should be much better than other candidates
- Consistency-check: (1) keypoint A’s nearest neighbor in image 2 is keypoint
B; (2) keypoint B’s nearest neighbor in image 1 is also keypoint A.

20
Camera Calibration
● Inputs : A collection of images with points whose 2D image coordinates and 3D world
coordinates are known.
● Outputs: The 3×3 camera intrinsic matrix, the rotation and translation of each image.

Capture multiple images of the Finding camera parameters by

checkerboard from different viewpoints Find checkerboard corners
minimizing 3D-2D reprojection err

21
Image credit: OpenCV
 Camera Calibration
● Minimizing the reprojection error

Extrinsic Intrinsic Detection Corner Points Perspective Projection Known 3D Location

22
Two View Reconstruction

Bundle
Match
Adjustment

23
Fundamental Matrix

24
Eight-Point Algorithm
• Given a correspondence
• Assume

• We can get

25
Eight-Point Algorithm
• Given 8 correspondences

• Nontrivial solution
• f is in null space of A
SVD!
26
Eight-Point Algorithm

● Rank constraint

● Minimize Frobenius norm

subject to

27
Rank Constraint

Before After 28
RANSAC Estimation
● For many times

○ Pick 8 points

○ Compute a solution for using these 8 points

○ Count number of inliers that with geometric error close to 0

● Pick the one with the largest number of inliers

● Only the inliers are kept as correspondences

29
30
Essential Matrix

31
 Essential Matrix Decomposition
● Essential matrix E to R and t

Try to verify by yourself 32

Extending to Multiple Views

33
Image credit: Noah Snavely
Multi-view Triangulation

Are we guaranteed to
converge?

34
Multi-view Triangulation

What could be changed?

35
 Bundle Adjustment

Extrinsic 3D Points Detection Keypoints Perspective Projection Unknow 3D Location

36
 Bundle Adjustment

What is the difference between calibration vs structure from motion?

Extrinsic 3D Points Detection Keypoints Perspective Projection Unknow 3D Location

37
Continuous Optimization
MAP inference: find the best configuration that minimize the energy

There is no universal solution. Inference algorithm choice is depending on:

● Continuous vs Discrete Variables: numerical approach or search-based
● Energy Functions: convex, submodular, piecewise linear, quadratic, etc.
● Graphical Model Structures: containing loops or not; having high-order
terms or not? 38
MAP Inference: Gradient Descent
● Minimize continuous-valued energy based models by numerical optimization:

● Pros: simple and straightforward, works for all differentiable energies

● Cons: (sub-)differentiability requirements and slow to convergence

39
MAP Inference: Newton Method
● For twice-differentiable energy function, one could use Newton’s method:

40
MAP Inference: Newton Method
● For twice-differentiable energy function, one could use Newton’s method:

● Pros: capturing curvature, better convergence, less likely stuck, less tuning
● Cons: expensive to compute inverse Hessian, hard to scale

41
MAP Inference: Gauss-Newton
● If the energy has a sum of square form:

● For each iteration t:

○ Taylor approximation for the residual function:

○ Solving least square:

42
How to get the solution? Today’s Quiz
Multi-View Stereo
● Input: images from several viewpoints
with known camera poses and calibration

● Output: 3D object model

Why are SFM 3D points insufficient?

44
Measuring the matching cost

45
Measuring the matching cost

46
Colmap: Photometric + Geometric Cost + View Select
● Photometric consistency: normalized cross correlation

● Geometry consistency: forward-backward reprojection error

47
Pixelwise View Selection for Unstructured Multi-View Stereo, 2016
MVSNet

48
MVSNet: Depth Inference for Unstructured Multi-view Stereo, 2018
3D Reconstruction: SFM + MVS

49
Image credit: Google, Michael Keass
Visual SLAM: Online SFM

50
Image credit: Yang et al. ICRA 2021
Camera Distortion

51
Image credit: OpenCV
Camera Distortion
● Remember to cv2.undistort the image if you want to reason in 3D.

52
Image credit: OpenCV
Event Cameras

53
Image credit: Davide Scaramuzza
Fisheye Camera / Omnidirectional Camera

54
Image credit: OZ robotics
What I Didn’t Cover
● Stereo Rectification
Making two stereo camera frontal parallel.

● Five-Point Algorithms
Recover Essential/Fundamental Matrix from 2D-2D Correspondences

● Projection Matrix Decomposition

Recover R and t from camera projection matrix

● Essential Decomposition
Recover R and t from essential matrix estimation

● Perspective-n-Projection (PnP)
Recover R and t from 2D-3D correspondences
55
Check Szeliski or MVG Book if you want to know these concepts