Novel Methods for Estimating Surface Normals from Affine Transformations

Abstract

The aim of this paper is to describe different estimation techniques in order to deal with point-wise surface normal estimation from calibrated stereo configuration. We show here that the knowledge of the affine transformation between two projections is sufficient for computing the normal vector unequivocally. The formula which describes the relationship among the cameras, normal vectors and affine transformations is general, since it works for every kind of cameras, not only for the pin-hole one. However, as it is proved in this study, the normal estimation can optimally be solved for the perspective camera. Other non-optimal solutions are also proposed for the problem. The methods are tested both on synthesized data and real-world images. The source codes of the discussed algorithms are available on the web.

A Appendix

Algorithm for Optimal Normal Estimation. The task is to minimize the cost function defined in Eq. 12 with respect to normal vector n. The scale of the vector is arbitrary, only the direction of the normal is required. Such kind of problems are typically solved using Lagrange-multipliers, however, it cannot be applied here since the derivatives become very difficult. For this reason, we utilize another constraint for the normal: let the sum of the coordinates be 1. Thus, n is written as $n=[n_x,n_y,1-n_x-n_y]^T$. Equation 12 can be reformulated as,

$$\begin{aligned} \arg \min _{m}\sum _{k=1}^{4}\left( \frac{m^{T}q_{k}+w_{k,z}}{\alpha m^{T}q_{5}+\alpha w_{5,z}}-a_{k}\right) ^{2}, \end{aligned}$$

where $m=[n_x,n_y]$, $q_i=[w_{i,x}-w_{i,z},w_{i,y}-w_{i,z}]^T$. (Indices x, y, and z denote the first, second, and third coordinates of vectors, respectively.)

The minima/maxima can be obtained by taking the derivative with respect to vector m:

$$\begin{aligned} 2\sum _{k=1}^{4} \beta _k \gamma _k =0 , \end{aligned}$$

where

$$\begin{aligned} \beta _k=\left( \frac{m^{T}q_{k}+w_{k,z}}{\alpha m^{T}q_{5}+\alpha w_{5,z}}-a_{k}\right) , \\ \gamma _k =\left( \alpha \frac{(m^{T}q_{5}+w_{5,z})q_{k}-(m^{T}q_{k}+w_{k,z})q_{5}}{(\alpha m^{T}q_{5}+\alpha w_{5,z})^{2}}\right) . \end{aligned}$$

After taking the lowest common multiple of the fractions, the left side should be equal to zero as $\sum _{k=1}^{4} \delta _k \kappa _k =0$, where

$$\begin{aligned} \delta _k=\left( m^{T}q_{k}+w_{k,z}-a_{k}\alpha m^{T}q_{5}-a_{k}\alpha w_{5,z}\right) ,\\ \kappa _k =\left( (m^{T}q_{5}+w_{5,z})q_{k}-(m^{T}q_{k}+w_{k,z})q_{5}\right) . \end{aligned}$$

It can be simplified as $\sum _{k=1}^{4} e^1_k e^2_k =0$, where

$$\begin{aligned} e^1_k=\left( m^{T}(q_{k}-a_{k}\alpha q_{5})+(w_{k,z}-a_{k}\alpha w_{5,z})\right) ,\\ e^2_k=\left( (m^{T}q_{5})q_{k}-(m^{T}q_{k})q_{5}+w_{5,z}q_{k}-w_{k,z}q_{5}\right) . \end{aligned}$$

This is an equation with a 2D-vector:

$$\begin{aligned} \sum _{k=1}^{4} r \left( \begin{array}{c} m^{T}\left( q_{5}q_{k,x}-q_{i}q_{5,x}\right) +w_{5,z}q_{k,x}-w_{k,z}q_{5,x}\\ m^{T}\left( q_{5}q_{k,y}-q_{i}q_{5,y}\right) +w_{5,z}q_{k,y}-w_{k,z}q_{5,y}\end{array}\right) =0 , \end{aligned}$$

where $r=\left( m^{T}(q_{k}-a_{k}\alpha q_{5})+(w_{k,z}-a_{k}\alpha w_{5,z})\right)$.

By introducing the $m=[x,y]^{T}$ notation, the vector equation is modified as follows

$$\begin{aligned} \sum _{k=1}^{4}(\varOmega _{k}x+\varPsi _{k}y+\varGamma _{k})\left( \begin{array}{c} \varOmega _{k}^{1}x+\varPsi _{k}^{1}y+\varGamma _{k}^{1}\\ \varOmega _{k}^{2}x+\varPsi _{k}^{2}y+\varGamma _{k}^{2}\end{array}\right) =0 , \end{aligned}$$

where

$$\begin{aligned} \begin{array}{cc} \varOmega _{k}=q_{k,x}-\alpha q_{5,x}a_{k} , &{} \varPsi _{k}=q_{k,y}-\alpha q_{5,y}a_{k} ,\\ \varGamma _{k}=w_{k,z}-a_{k}\alpha w_{5,z} ,&{} \varOmega _{k}^{1}=0 ,\\ \varPsi _{k}^{1}=q_{5,y}q_{k,x}-q_{k,y}q_{5,x} ,&{} \varGamma _{k}^{1}=w_{5,z}q_{k,x}-w_{k,z}q_{5,x},\\ \varOmega _{k}^{2}=q_{5,x}q_{k,y}-q_{k,x}q_{5,y},&{} \varPsi _{k}^{2}=0,\\ \varGamma _{k}^{2}=w_{5,z}q_{i,y}-w_{i,z}q_{5,y}.&{} ~ \end{array} \end{aligned}$$

The rows of the vector equation give two special quadratic curves. They are written by their implicit equations as $\sum _{k=1}^{4}A_{k}^{l}x^{2}+\sum _{k=1}^{4}B_{k}^{l}y^{2}+\sum _{k=1}^{4}C_{k}^{l}xy+\sum _{k=1}^{4}D_{k}^{l}x+\sum _{k=1}^{4}E_{k}^{l}y+\sum _{k=1}^{4}F_{k}^{l}=0$, where $A_{k}^{l}=\varOmega _{k}\varOmega _{k}^{l}$, $B_{k}^{l}=\varPsi _{k}\varPsi _{k}^{l}$, $C_{k}^{l}=\varOmega _{k}^{l}\varPsi _{k}+\varPsi _{k}^{l}\varOmega _{k}$, $D_{k}^{l}=\varOmega _{k}^{l}\varGamma _{k}+\varGamma _{k}^{l}\varOmega _{k}$, $E_{k}^{l}=\varPsi _{k}^{l}\varGamma _{k}+\varGamma _{k}^{l}\varPsi _{k}$ and $F_{k}^{l}=\varGamma _{k}\varGamma _{k}^{l}$, $l\in {1,2}$. They are special because $A_{k}^{1}=0$ and $B_{k}^{2}=0$.

The solution of the optimal method described in the study (within appendix) is given by the intersection of two quadratic equations.

$$\begin{aligned} B_{1}y^{2}+C_{1}xy+D_{1}x+E_{1}y+F_{1}=0 , \\ A_{2}x^{2}+C_{2}xy+D_{2}x+E_{2}y+F_{2}=0. \end{aligned}$$

Parameter y can be obtained from the latter equation as

$$\begin{aligned} y=-\frac{A_{2}x^{2}+D_{2}x+F_{2}}{C_{2}x+E_{2}}. \end{aligned}$$

Substituting y into the first equation the following expression is obtained

$$\begin{aligned} B_{1}\left( \frac{A_{2}x^{2}+D_{2}x+F_{2}}{C_{2}x+E_{2}}\right) ^{2}- C_{1}x\frac{A_{2}x^{2}+D_{2}x+F_{2}}{C_{2}x+E_{2}}+D_{1}x \\ -E_{1}\frac{A_{2}x^{2}+D_{2}x+F_{2}}{C_{2}x+E_{2}}+F_{1}=0. \end{aligned}$$

If both sides are multiplied with $(C_{2}x+E_{2})^{2}$, then the equation modifies as follows

$$\begin{aligned}&B_{1}(A_{2}x^{2}+D_{2}x+F_{2})^{2}- C_{1}x\left( A_{2}x^{2}+D_{2}x+F_{2}\right) \left( C_{2}x+E_{2}\right) \\&\quad +D_{1}x\left( C_{2}x+E_{2}\right) ^{2}- E_{1}\left( A_{2}x^{2}+D_{2}x+F_{2}\right) \left( C_{2}x+E_{2}\right) + F_{1}\left( C_{2}x+E_{2}\right) ^{2}=0 \end{aligned}$$

This is a fourth-order polynomial where the coefficients are as follows

$$\begin{aligned} \begin{array}{cc} x^{4}: &{} B_{1}A_{2}^{2}-C_{1}A_{2}C_{2} \\ x^{3}: &{} 2B_{1}A_{2}D_{2}-C_{1}A_{2}E_{2}- C_{1}D_{2}C_{2}+D_{1}C_{2}^{2}-E_{1}A_{2}C_{2} \\ \begin{array}{c} x^{2}: \\ ~ \end{array} &{} \begin{array}{c} B_{1}D_{2}^{2}+2B_{1}A_{2}F_{2}- C_{1}D_{2}E_{2}-C_{1}F_{2}C_{2}+2D_{1}C_{2}E_{2} \\ -E_{1}A_{2}E_{2}-E_{1}D_{2}C_{2}+F_{1}C_{2}^{2} \end{array} \\ \begin{array}{c} x^{1}: \\ ~ \end{array} &{} \begin{array}{c} 2B_{1}D_{2}F_{2}-C_{1}F_{2}E_{2}+ D_{1}E_{2}^{2}-E_{1}D_{2}E_{2}-E_{1}F_{2}C_{2} \\ +2F_{1}C_{2}E_{2} \end{array} \\ x^{0}: &{} B_{1}F_{2}^{2}-E_{1}F_{2}E_{2}+F_{1}E_{2}^{2} \end{array} \end{aligned}$$

Remark that the equation $C_{2}x+E_{2}=0$ can also be considered. (In this case the first equation is independent from y.)

Fig. 12.

Quadratic curves.

An example for two quadratic curves with three real intersections is visualized in Fig. 12. (The parameters of curves are $B1 = -1.9055$, $C1 = 2.2632$, $D1 = 2.8577$, $E1 = -9.4392$, $F1 = 7.7081$, and $A2 = -2.2632$, $C2 = 1.9055$, $D2 = -4.2074$, $E2 = 2.3903$, $F2 = -1.1190$).

Affine Parameters from Homography. The affine parameters can be obtained from the homography between the stereo image pairs. Let us assume that the homography H is given. Then the correspondence between the coordinates on the first (u and v) and second ($u'$ and $v'$) images is written as

$$\begin{aligned} u'=\frac{h_{1}^{T}\left[ u,v,1\right] ^{T}}{h_{3}^{T}\left[ u,v,1\right] ^{T}}, \\ v'=\frac{h_{2}^{T}\left[ u,v,1\right] ^{T}}{h_{3}^{T}\left[ u,v,1\right] ^{T}}, \end{aligned}$$

where the $3 \times 3$ homography matrix H is written as

$$\begin{aligned} H=\left[ \begin{array}{c} h_1^T \\ h_2^T \\ h_3^T \end{array} \right] = \left[ \begin{array}{ccc} h_{11} &{} h_{12} &{} h_{13} \\ h_{21} &{} h_{22} &{} h_{23} \\ h_{31} &{} h_{32} &{} h_{33} \end{array} \right] . \end{aligned}$$

The affine parameters come from the partial derivatives of the perspective plane-plane transformation. The top left element $a_{11}$ of affine transformation matrix is as follows

$$\begin{aligned} a_{11}=\frac{\partial u'}{\partial u}=\frac{h_{11}h_{3}^{T}\left[ u,v,1\right] ^{T}-h_{31}h_{1}^{T}\left[ u,v,1\right] ^{T}}{\left( h_{3}^{T}\left[ u,v,1\right] ^{T}\right) ^{2}}= \frac{h_{11}-h_{31}u'}{s} , \end{aligned}$$

where $s=h_3^T [u,v,1]^T$. The other components of affine matrix are obtained similarly

$$\begin{aligned} a_{12}=\frac{\partial u'}{\partial v}=\frac{h_{12}-h_{32}u'}{s} , \\ a_{21}=\frac{\partial v'}{\partial u}=\frac{h_{21}-h_{31}v'}{s} , \\ a_{22}=\frac{\partial v'}{\partial v}=\frac{h_{22}-h_{32}v'}{s}. \end{aligned}$$