An N-Point Linear Solver for Line and Motion Estimation with Event Cameras

Abstract

Event cameras respond primarily to edges—formed by strong gradients—and are thus particularly well-suited for line-based motion estimation. Recent work has shown that events generated by a single line each satisfy a polynomial constraint which describes a manifold in the space-time volume. Multiple such constraints can be solved simultaneously to recover the partial linear velocity and line parameters. In this work, we show that, with a suitable line parametrization, this system of constraints is actually linear in the unknowns, which allows us to design a novel linear solver. Unlike existing solvers, our linear solver (i) is fast and numerically stable since it does not rely on expensive root finding, (ii) can solve both minimal and overdetermined systems with more than 5 events (i.e. $N\geq 5$), and (iii) admits the characterization of all degenerate cases and multiple solutions. The found line parameters are singularity-free and have a fixed scale, which eliminates the need for auxiliary constraints typically encountered in previous work. To recover the full linear camera velocity we fuse observations from multiple lines with a novel velocity averaging scheme that relies on a geometrically-motivated residual, and thus solves the problem more efficiently than previous schemes which minimize an algebraic residual. Extensive experiments in synthetic and real-world settings demonstrate that our method surpasses the previous work in numerical stability, and operates over 600 times faster.

Project page:  https://mgaoling.github.io/eventail/

1 Introduction

Man-made scenes contain a multitude of straight lines, and exploiting these lines for motion estimation is an important feature of modern mobile vision systems like AR/VR devices and robotic systems [17, 25, 42, 23]. However, computer vision algorithms aimed at leveraging these line features still suffer from fundamental limitations when using standard frame-based sensing: During high-speed motion and challenging illumination conditions, these sensors suffer from motion blur and saturation effects, which have deleterious effects on line feature extraction. Event cameras [10] are biologically inspired sensors that address the limitation of frame-based sensors by instead only measuring the changes in intensity at a per-pixel level, and they do this with high dynamic range, low motion blur, high temporal resolution, and high spatial data-sparsity.

Due to their working principle, event cameras respond primarily to edges—formed by strong gradients—and are thus particularly well-suited for line-based motion estimation. A recent breakthrough [12] in event-based motion estimation introduced an incidence relation that enforces the intersection of bearing vectors emitted by events and a corresponding line that generates those events. Using this relation, a 5-point minimal solver was designed that recovers the parameters of a minimal two-point-two-plane parametrization of the line and two velocity ratios in the plane perpendicular to the line, using the Gröbner basis method and polynomial elimination theory. However, this solver suffers from several limitations: First, it relies on a non-minimal line representation using four degrees of freedom (DoF) that (i) fails to realize that, in the absence of scale, only three DoF are needed, and (ii) encounters singularities when describing lines parallel to the two planes. Secondly, to solve for the motion and line parameters, previous work employs a polynomial solver, which (i) is by definition minimal and thus incapable of incorporating more than five events, and (ii) relies on root finding algorithms that are expensive to run and suffer from instabilities that are not easily detected.

This work addresses these limitations with two important innovations: First, it introduces a new line representation based on the angle-axis representation of a rotation matrix, which is singularity-free and only depends on three DoF and thus implicitly enforces scale ambiguity. Second, in formulating the incidence relation with this new parametrization, we derive a simple algorithm for determining motion and line parameters that only relies on solving a linear system and simple vector operations and is thus orders of magnitude faster than the polynomial solver in [12]. This linear system is easily extended to $N>5$ events, with a minimal increase in complexity, which enables solution refinement with inliers when employing random sample consensus (RANSAC) schemes.

The proposed solver sheds light on all possible cases of degenerate solutions, how they arise, and all additional solutions that arise from symmetries in the incidence relation. Finally, expressing the incidence relation in terms of our new line parametrization allows us to fully characterize and visualize the types of manifolds circumscribed by events generated by a single line. Our contributions are:

• A minimal, three DoF representation of lines based on the angle-axis representation of a rotation matrix. This representation encodes a reference frame centered at the 3D line, and enforces a unit distance to the closest point.

• A linear algorithm for determining line and motion parameters from a set of events triggered by a line. This algorithm is fast, and extensible to multiple events, and sheds light on degenerate and multiplicitous solutions.

• A simpler and faster scheme for fusing partial linear velocity measurements from multiple lines, based on geometrically-motivated constraints, instead of solving algebraic equations.

We validate our method extensively in simulated and real-world settings. In particular, our solver is on average 600 times faster, taking only $3.25\text{\,}\mathrm{\SIUnitSymbolMicro s}$ for five events, compared to $2046\text{\,}\mathrm{\SIUnitSymbolMicro s}$ for [12], while achieving a similar performance.

2 Related Work

Ego-motion estimation is crucial for intelligent mobile devices and thus has been the subject of extensive research over the past few decades. The sub-class of vision-based solutions comprises single-camera, stereo-camera, and multi-camera solutions that are potentially supported by an inertial measurement unit. A review would go beyond the scope of the present paper, and the reader is kindly referred to recent reviews such as the one by Cadena et al. [4]. This paper focuses on motion estimation with event cameras.

The latter is a challenging problem that is initially often addressed for constrained scenarios such as 2D motion [39], known depth or 3D structure [40, 5, 27, 7, 3, 6, 46], pure rotation [8], and homographic warping [9, 20, 34, 24, 31]. The community has furthermore explored the combination with other sensors such as standard cameras [20, 38, 16], inertial units [38, 45], or a second event camera [44]. In turn, the present paper considers motion in 3D with a single event camera and in arbitrary environments. Different optimization-based [32, 21, 28], filter-based [19, 45] and learning-based [26, 13] solutions have already been presented. Of particular interest to the present work are methods that rely on line features [43, 21].

In the spirit of original works on monocular visual odometry [29], the present work addresses the relatively unexplored topic of geometric incidence relationships for local relative motion calculation with an event camera. Geometric solutions remain important to date owing to their ability to find solutions with optimality guarantees, and potentially certificates, under known assumptions, unlike optimization-, filter- or learning-based solvers, which often lack these guarantees. Given that events are primarily triggered by high-gradient appearance boundaries, the dominant feature for event-based motion estimation is given by lines. Weng et al. [41] and Hartley et al. [15] have proposed closed-form solutions for frame-based cameras. Tri-focal tensor geometry inspired the first closed-form solution for event cameras [30]. An important characteristic of this solver is that it relies on a local constant velocity motion model and makes use of the first-order dynamics of the camera. This is important as it permits the inclusion of the time-stamped, asynchronous measurements produced by an event camera. Nonetheless, the method by Peng et al. [30] is not general as it still depends on approximate event-based line-feature extractors [2, 37, 36] rather than events, only.

The most related works to ours are the works by Ieng et al. [18], Seok and Lim [33], and Gao et al. [12], who propose model-based fitting of the manifold locations of the events generated by the observation of a line under motion. In particular, Gao et al. [12] introduces an exact incidence relation that all such events must obey under constant linear velocity. It depends on observable camera motion and 3D line parameters, and thus serves as a basis for joint manifold fitting and motion estimation. Based on this foundation, the present work develops the first linear $N$-point solution to this problem, which not only unlocks unprecedented efficiency, but also a simplified understanding of degenerate geometric conditions.

3 Methodology

Assume a calibrated event camera undergoing an arbitrary six DoF motion, while observing a set of $M$ lines $\{\mathbf{L}_{i}\}_{i=1}^{M}$. Each line generates a set of $N_{i}$ events $\mathcal{E}_{i}=\{e_{ij}\}_{j=1}^{N_{i}}$ where each event $e_{ij}=(\mathbf{x}_{ij},t_{ij},p_{ij})$ is characterized by its pixel coordinate $\mathbf{x}_{ij}$ in the image plane, timestamp $t_{ij}$ (with $\text{\,}\mathrm{\SIUnitSymbolMicro s}$ resolution), and polarity $p_{ij}$.

For a small time window $[t_{s}-\Delta t,t_{s}+\Delta t]$, centered at reference time $t_{s}$, such that the camera motion can be approximated by linear dynamics, the events generated by a single line circumscribe a manifold termed eventail [12]. In Sec. 3.1, we describe the geometric incidence relation [12] which needs to be satisfied by events on this manifold and depends on the observable components of the velocity $\mathbf{v}$ and the line $\mathbf{L}_{i}$ for which we introduce a minimal parametrization. Then, in Sec. 3.2 we will introduce a solver that recovers the line parameters, and observable linear velocity parameters from a set of $N_{i}\geq 5$ events that lie on the manifold. Finally, in Sec. 3.6, we will explore how to recover the full linear velocity from a set of $M\geq 2$ partial velocity observations.

3.1 Incidence Relationship

We reiterate here briefly the incidence relation introduced in [12] and use Fig. 1 for illustration purposes. For simplicity, we will first consider the case of one line, and thus drop the index $i$ from the variables. We furthermore express all quantities in the camera frame centered at time $t_{s}$. The incidence relation enforces that events are triggered by points on the line, such that the line $\mathbf{L}_{j}=[\mathbf{d}_{j}^{\intercal}\,\mathbf{m}_{j}^{\intercal}]^{\intercal}$ (in Plücker coordinates) emanating from an individual event $e_{j}$ triggered at time $t_{j}$ (orange line in Fig. 1) intersects the line $\mathbf{L}=[\mathbf{d}^{\intercal}\,\mathbf{m}^{\intercal}]^{\intercal}$ (blue line in Fig. 1). The condition for intersection of two non-parallel lines is

$$\mathbf{d}^{\intercal}\mathbf{m}_{j}+\mathbf{m}^{\intercal}\mathbf{d}_{j}=0\,.$$ (1)

Plücker line coordinates comprise the line direction $\mathbf{d}\in\mathbb{R}^{3}$ and moment $\mathbf{m}=\mathbf{P}\times\mathbf{d}$, with $\mathbf{P}\in\mathbb{R}^{3}$ being an arbitrary point on the line. As [12] for the event ray $j$ we use the camera position $\mathbf{C}[t_{j}]=\mathbf{P}_{j}$ at time $t_{j}$, and the event bearing vector $\mathbf{f}_{j}^{\prime}=\mathbf{d}_{j}$ rotated into the reference frame at time $t_{s}$. Under first order dynamics, these are $\mathbf{C}[t_{j}]=t_{j}^{\prime}\mathbf{v}$ and $\mathbf{f}_{j}^{\prime}=\mathbf{R}[t_{j}]\mathbf{f}_{j}$ with $t_{j}^{\prime}=t_{j}-t_{s}$. As [12], we assume $\mathbf{R}[t_{j}]$ given and computed by integrating angular rate measurements $\bm{\omega}$ from an available inertial measurement unit (IMU) with $\mathbf{R}[t_{j}]=\text{exp}([\bm{\omega}]_{\times}t_{j}^{\prime})$. Here $\text{exp}(\cdot)$ denotes the matrix exponential and $[\bm{\omega}]_{\times}$ denotes the skew-symmetric matrix associated with the angular rate $\bm{\omega}\in\mathbb{R}^{3}$. The Plücker coordinates are thus $\mathbf{L}_{j}=[\mathbf{f}_{j}^{\prime\intercal}\quad(\mathbf{C}[t_{j}]\times\mathbf{f}_{j}^{\prime})^{\intercal}]^{\intercal}$ and the incidence relation becomes

$$\mathbf{d}^{\intercal}\left(\mathbf{C}\left[t_{j}\right]\times\mathbf{f}^{\prime}_{j}\right)+\mathbf{m}^{\intercal}\mathbf{f}^{\prime}_{j}=0.$$ (2)

The above equation relates measurements $\mathbf{f}_{j}$ and $t_{j}$ to our unknowns $\mathbf{d},\mathbf{m}$ and $\mathbf{v}$, but is still not in a minimal form. This is because, firstly Plücker coordinates are not minimal, and indeed true minimal line representations only have four DoF. Second, in a monocular setup, there is scale ambiguity, and this dictates that the absolute scale of $\mathbf{v}$ and $\mathbf{L}$ is unobservable. Finally, the velocity component along the line direction is unobservable due to the aperture problem. To encapsulate these constraints, in the next part, we will introduce a line representation based on the angle-axis representation of rotation matrices. This rotation matrix simultaneously spans a coordinate frame in which we will express our camera velocity, and the aperture problem can be enforced succinctly.

Figure 1: Incidence relationship between the line $\mathbf{L}$, and the bearing vector $\mathbf{f}_{j}^{\prime}$ of event. We parameterize the line with the rotation matrix $\mathbf{R}_{\ell}=[\mathbf{e}^{\ell}_{1}\,\mathbf{e}^{\ell}_{2}\,\mathbf{e}^{\ell}_{3}]$. Since scale is unobservable, we select the point $-\mathbf{e}^{\ell}_{3}$ at unit depth to lie on the line, and $\mathbf{e}^{\ell}_{1}$ to indicate its direction. Due to the aperture problem, we can only observe the projected camera velocity $\hat{\mathbf{v}}$ with components $u^{\ell}_{y}$ and $u^{\ell}_{z}$ in the $\mathbf{e}^{\ell}_{2}$ and $\mathbf{e}^{\ell}_{3}$ direction respectively.

3.2 Transition into a Minimal Form

The transition into minimal form consists of two steps. We first derive the angle-axis-based line representation by successively eliminating internal constraints in the Plücker line coordinate representation, and then proceed with finding the minimal camera velocity representation. We start by observing that scaling both $\mathbf{d}$ and $\mathbf{m}$ yields the same line, and thus we may choose to fix the scale of $\mathbf{d}$ to be unity. Next, we observe that $\mathbf{m}=\mathbf{P}\times\mathbf{d}$ is by definition perpendicular to $\mathbf{d}$. Moreover, we see that any $\mathbf{P}$ on the same line results in the same moment, and thus we may choose to select it closest to the origin, such that it is perpendicular to $\mathbf{d}$. Since the scale is unobservable, we may furthermore fix the distance from $\mathbf{P}$ to the origin to be unity. Summarizing these observations, we conclude that we may select $\mathbf{P}$ and $\mathbf{d}$ to be perpendicular unit vectors, and in particular we will select $\mathbf{P}=-\mathbf{e}^{\ell}_{3}$ and $\mathbf{d}=\mathbf{e}^{\ell}_{1}$, such that the resulting moment is $\mathbf{m}=-\mathbf{e}^{\ell}_{2}$. We visualize the three unit vectors in Fig. 1, and observe that they span a line-dependent coordinate frame via the rotation matrix $\mathbf{R}_{\ell}=[\mathbf{e}^{\ell}_{1}\,\mathbf{e}^{\ell}_{2}\,\mathbf{e}^{\ell}_{3}]$. Rotation matrices belong to $\mathit{SO}(3)$, and thus it becomes apparent that this line representation can be further compressed via the matrix logarithm $\bm{\theta}_{\ell}=(\text{log}\left(\mathbf{R}_{\ell}\right))^{\vee}=[\theta^{\ell}_{x}\,\theta^{\ell}_{y}\,\theta^{\ell}_{z}]$, to yield only three DoF, which is a minimal line representation in the absence of scale. Here $(\cdot)^{\vee}$ maps the skew-symmetric matrix in the argument to the associated vector.

Note that compared to the two-point-two-plane parametrization [14] in [12], this representation is (i) minimal, relying on three instead of four DoF, and (ii) more flexible, since it does need a reparametrization step for lines that are almost parallel to the $yz$-plane.

We now address the camera velocity parametrization, which we express in the line-dependent coordinate frame.

$$\mathbf{v}=\mathbf{R}_{\ell}\mathbf{u}_{\ell}\,,$$ (3)

introduces the camera velocity $\mathbf{u}_{\ell}=[u_{x}^{\ell}\,u_{y}^{\ell}\,u_{z}^{\ell}]$ expressed in the line coordinate frame. Using these new parametrizations, the incidence relation Eq. 2 becomes

$$t_{j}^{\prime}{\mathbf{e}^{\ell}_{1}}^{\intercal}((\mathbf{R}_{\ell}\mathbf{u}_{\ell})\times\mathbf{f}_{j}^{\prime})-{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2}=0\,.$$ (4)

Cycling through the triple product in the first summand, i.e. $\mathbf{a}^{\intercal}(\mathbf{b}\times\mathbf{c})=\mathbf{c}^{\intercal}(\mathbf{a}\times\mathbf{b})$, we arrive at

$$t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(\mathbf{e}^{\ell}_{1}\times(\mathbf{R}_{\ell}\mathbf{u}_{\ell}))-{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2}=0\,,$$ (5)

which can be expanded and further simplified to

$$t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u_{y}^{\ell}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2}=0\,.$$ (6)

Note that due to the cross product with $\mathbf{e}^{\ell}_{1}$, the camera velocity $u^{\ell}_{x}$ becomes unobservable within this incidence relation, i.e. changing it does not affect the residual. This confirms our intuition that the aperture problem should make velocities along the line direction unobservable. As a result, we focus on only solving for $u^{\ell}_{y}$ and $u^{\ell}_{z}$. Furthermore, observe that these equations are in terms of $\mathbf{e}^{\ell}_{2}$ and $\mathbf{e}^{\ell}_{3}$ instead of the minimal parameters $\bm{\theta}_{\ell}$. In what follows we design the solver around recovering $\mathbf{e}^{\ell}_{2}$ and $\mathbf{e}^{\ell}_{3}$ since it yields a simpler algorithm, yet it should be remembered that the minimal representation can always be recovered using the matrix logarithm of $\mathbf{R}_{\ell}$. We now discuss how to recover the unknowns from a set of incidence relations from multiple events, which is summarized in Alg. 1.

3.3 Five-point Minimal Solver

The incidence relationship in Eq. 6 has five unknowns, three from $\bm{\theta}_{\ell}=[\theta^{\ell}_{x}\,\theta^{\ell}_{y}\,\theta^{\ell}_{z}]$ and two from $\mathbf{u}_{\ell}=[0\,u^{\ell}_{y}\,u^{\ell}_{z}]$, and thus can be solved by stacking a minimum of five such constraints. Since each such constraint originates from a single event, this means that five events are the minimum number to solve this system. This stack of equations is

$$t_{1}^{\prime}{\mathbf{f}_{1}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{1}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2} =0$$

$$t_{2}^{\prime}{\mathbf{f}_{2}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{2}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2} =0$$

$$\vdots$$

$$t_{5}^{\prime}{\mathbf{f}_{5}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{5}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2} =0\,.$$

This system of equations is linear in the unknowns and can be rewritten as a single matrix equation

$$\underbrace{\begin{bmatrix}t_{1}^{\prime}{\mathbf{f}_{1}^{\prime}}^{\intercal}&{\mathbf{f}_{1}^{\prime}}^{\intercal}\\ \vdots&\vdots\\ t_{5}^{\prime}{\mathbf{f}_{5}^{\prime}}^{\intercal}&{\mathbf{f}_{5}^{\prime}}^{\intercal}\\ \end{bmatrix}}_{\doteq\mathbf{A}\in\mathbb{R}^{5\times 6}}\underbrace{\begin{bmatrix}u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3}\\ \mathbf{e}^{\ell}_{2}\end{bmatrix}}_{\doteq\mathbf{x}\in\mathbb{R}^{6\times 1}}=\mathbf{0}\,.$$ (7)

Note that this formulation successfully groups the terms from the events in $\mathbf{A}$ and unknowns in $\mathbf{x}$. We can thus solve for $\mathbf{x}$, and then reconstruct the unknowns from the found solution. Solving Eq. 7 can be done with a singular value decomposition of A and then selecting the last column of $\mathbf{V}$ corresponding to the smallest singular value of $\mathbf{A}$. Let us denote this solution with $\hat{\mathbf{x}}$. Note that $\hat{\mathbf{x}}$ is normalized, and, due to the homogeneous nature of Eq. 7, only known up to parity, i.e. both $\pm\hat{\mathbf{x}}$ are solutions. Note also that this procedure is not limited to using only five events, but can be applied to $N\geq 5$, however in this case the solution will no longer be exact, and instead approximate but with a globally minimal squared residual equal to the smallest singular value of $\mathbf{A}$. The ability to process more than five events sets this method apart from the solver in [12] which uses a fixed elimination template tailored to only five events. Next, we discuss how to recover the unknowns from a solution $\mathbf{\hat{x}}$.

3.4 Recovering the Unknowns from $\hat{\mathbf{x}}$

For simplicity, we will only treat the case with $+\hat{\mathbf{x}}$, but will state that the ambiguity in the parity of the solution to Eq. 7 gives rise to the solution pairs $S_{0},S_{1}$, and $S_{2},S_{3}$. Remembering the definition of $\mathbf{x}$ we write that

$$\hat{\mathbf{x}}=\begin{bmatrix}\lambda\hat{\mathbf{x}}_{1:3}\\ \lambda\hat{\mathbf{x}}_{4:6}\end{bmatrix}=\begin{bmatrix}u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3}\\ \mathbf{e}^{\ell}_{2}\end{bmatrix}\,.$$ (8)

Here $\lambda$ is an unknown scaling factor. However, since the last three entries of $\hat{\mathbf{x}}$ correspond to the unit vector $\mathbf{e}^{\ell}_{2}$, we can simply normalize $\hat{\mathbf{x}}$ by the length of $\hat{\mathbf{x}}_{4:6}$. In what follows we assume that this normalization is done beforehand, and thus ignore this scaling factor by setting $\lambda=1$. Straightforward manipulation yields

$$\mathbf{e}^{\ell}_{2} =\hat{\mathbf{x}}_{4:6}$$ (9a)

$$u^{\ell}_{z} =\hat{\mathbf{x}}_{1:3}^{\intercal}\hat{\mathbf{x}}_{4:6}$$ (9b)

$$u^{\ell}_{y}\mathbf{e}^{\ell}_{1} =\hat{\mathbf{x}}_{1:3}\times\hat{\mathbf{x}}_{4:6}\,.$$ (9c)

From the last equation we can recover $u^{\ell}_{y}$ and $\mathbf{e}^{\ell}_{1}$ by taking the norm, and normalized vector

$$u^{\ell}_{y}=\|\hat{\mathbf{x}}_{1:3}\times\hat{\mathbf{x}}_{4:6}\|\text{ and }\mathbf{e}^{\ell}_{1}=\frac{\hat{\mathbf{x}}_{1:3}\times\hat{\mathbf{x}}_{4:6}}{\|\hat{\mathbf{x}}_{1:3}\times\hat{\mathbf{x}}_{4:6}\|}\,.$$ (10)

Note that this decomposition is not unique, as we may simultaneously flip the signs of $\mathbf{e}^{\ell}_{1}$ and $u^{\ell}_{y}$, resulting in the same product. Finally, we can recover $\mathbf{e}^{\ell}_{3}=\mathbf{e}^{\ell}_{1}\times\mathbf{e}^{\ell}_{2}$. While $\hat{\mathbf{x}}$ is the globally optimal solution to the incidence relation constraints in Eq. 7, it is not immediately clear that the recovered $\mathbf{R}_{l},\mathbf{u}_{l}$ are also globally optimal with respect to this constraint. As proved in the supplementary material it turns out that $\mathbf{R}_{l},\mathbf{u}_{l}$ are globally optimal. Next, we will comment on verifying the correctness of the solution.

3.5 Degenerate Solutions and Solution Multiplicity

First, we state a theorem that the only degenerate cases arise when the matrix $\mathbf{A}$ in Eq. 7 is rank-deficient. This means that, if $\text{rank}(\mathbf{A})\geq 5$, the previously discussed decomposition always succeeds and yields four solutions.

Theorem 1: If $\text{rank}(\mathbf{A})\geq 5$, with $\mathbf{A}$ defined in Eq. 7, the decomposition of $\hat{\mathbf{x}}$ into $\mathbf{u}_{\ell}$ and $\mathbf{R}_{\ell}$ always succeeds and yields four distinct solutions. If $\text{rank}(\mathbf{A})<5$ the solver returns infinitely many solutions.

Note that this theorem also handles cases in which the line passes through the origin. Both the proof of the above theorem and the handling of this case are described in the supplementary material. Rank deficiency of $\mathbf{A}$ occurs when events share the same timestamp $t_{j}$ or motion-corrected bearing vector $\mathbf{f}_{j}^{\prime}$. To identify this scenario, checking the matrix rank before solving for $\hat{\mathbf{x}}$ can be done. After SVD, the second smallest singular value should be checked for being large, since small values indicate near-rank deficiency.

Next, we discuss the multiplicity of solutions. As previously stated, the designed solver returns four distinct solutions if the rank of $\mathbf{A}$ is at least 5. Here we enumerate these solutions (visualized in Fig. 2), stated as a theorem:

Theorem 2: Given a solution $S_{0}=\{\mathbf{e}^{\ell}_{1},\mathbf{e}^{\ell}_{2},\mathbf{e}^{\ell}_{3},u^{\ell}_{y},u^{\ell}_{z}\}$ to the incidence relation in Eq. 6, then

$$S_{1} =\{\mathbf{e}^{\ell}_{1},-\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},u^{\ell}_{y},u^{\ell}_{z}\}\,,$$

$$S_{2} =\{-\mathbf{e}^{\ell}_{1},\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},-u^{\ell}_{y},u^{\ell}_{z}\}\,,$$

$$S_{3} =\{-\mathbf{e}^{\ell}_{1},-\mathbf{e}^{\ell}_{2},\mathbf{e}^{\ell}_{3},-u^{\ell}_{y},u^{\ell}_{z}\}$$

are also solutions. For solutions $S_{1}$ and $S_{2}$ the closest point $-\mathbf{e}_{3}^{\ell}$ on the line is behind the camera, while for solutions $S_{2}$ and $S_{3}$ the line direction $\mathbf{e}^{\ell}_{1}$ is flipped, which represents an ambiguity in the definition of direction of $\mathbf{L}$.

We state the proof in the supplementary material. Note that two configurations correspond to flipping across the xy-plane. We eliminate these solutions by enforcing that the intersection point $\mathbf{P}_{j}$ (see Fig. 1) between the line and event ray is in front of the camera. In the supplementary material, we use this geometric interpretation to characterize manifolds spanned by events in more detail.

(a) Solution $S_{0}$	(b) Solution $S_{1}$	(c) Solution $S_{2}$	(d) Solution $S_{3}$

Figure 2: Multiplicity of solutions to the incidence relation in Eq. 6. While $S_{0}$ and $S_{3}$ have the line in front of the camera, $S_{1}$ and $S_{2}$ have the line behind the camera. The solution pairs $S_{0},S_{3}$ and $S_{1},S_{2}$ differ in the orientation of $\mathbf{e}^{\ell}_{1}$, which comes from the ambiguity of defining the line direction. In solutions with the line behind the camera, the measured projected camera velocity, $\mathbf{R}_{\ell}\mathbf{u}_{\ell}$ returned by the solver is the negative of the true projected velocity $\hat{\mathbf{v}}$. However, these solutions can be discarded by enforcing the condition in Eq. 14.

Algorithm 1 Linear Solver for Line and Partial Motion Parameters

Input: A set of events $\mathcal{E}$ with rotated bearing vectors.
Output: Line parameters $\bm{\theta}_{\ell}$ and projected velocities $\mathbf{u}_{\ell}$.

• Form matrix $\mathbf{A}$ from the set of events $\mathcal{E}$ by Eq. 7 and make sure that $\text{rank}(\mathbf{A})\geq 5$.

• Apply SVD on $\mathbf{A}$ and select the last column of $\mathbf{V}$, denoted with $\hat{\mathbf{x}}$. Both $\pm\hat{\mathbf{x}}$ can be selected.

• Normalize $\hat{\mathbf{x}}$ by $\hat{\mathbf{x}}_{4:6}$, the last three elements.

• Recover $\mathbf{e}^{\ell}_{2},u^{\ell}_{z}$ from Eq. 9.

• Recover $\mathbf{e}^{\ell}_{1},u^{\ell}_{y}$ from Eq. 10. Both $\{\mathbf{e}^{\ell}_{1},u^{\ell}_{y}\}$ and $\{-\mathbf{e}^{\ell}_{1},-u^{\ell}_{y}\}$ can be selected.

• Compile $\mathbf{u}_{\ell}=[0\,u^{\ell}_{y}\,u^{\ell}_{z}]$.

• Compute $\mathbf{e}^{\ell}_{3}=\mathbf{e}^{\ell}_{1}\times\mathbf{e}^{\ell}_{2}$.

• Construct the rotation $\mathbf{R}_{\ell}=[\mathbf{e}^{\ell}_{1}\,\mathbf{e}^{\ell}_{2}\,\mathbf{e}^{\ell}_{3}]$.

• Recover minimal line parameters $\bm{\theta}_{\ell}=(\text{log}\left(\mathbf{R}_{\ell}\right))^{\vee}$.

3.6 Velocity Averaging from Multiple Manifolds

As previously discussed, the linear solver can only recover partial velocities perpendicular to the line generating the events in a single cluster. We now describe how to recover the full velocity from a set of partial observations from multiple lines $\mathbf{L}_{i}$. Each partial observation of the velocity $\mathbf{v}$ is the projection $\hat{\mathbf{v}}_{i}=\mathbf{H}_{i}\mathbf{v}$ into the $\mathbf{e}^{\ell}_{2i}\mathbf{e}^{\ell}_{3i}$-plane where $\mathbf{H}_{i}=\mathbf{I}-\mathbf{e}^{\ell}_{1i}{\mathbf{e}^{\ell}_{1i}}^{\intercal}$, which is visualized in Fig. 3(i). The observation is given by the two projections $u^{\ell}_{yi}$ and $u^{\ell}_{zi}$ scaling the second and third basis vectors of $\mathbf{R}_{{\ell}i}$, respectively. Thus, the correct velocity estimate must satisfy

$$\hat{\mathbf{v}}_{i}=\mathbf{H}_{i}\mathbf{v}=\mathbf{e}^{\ell}_{2i}u^{\ell}_{yi}+\mathbf{e}^{\ell}_{3i}u^{\ell}_{zi}=\mathbf{R}_{{\ell}i}\mathbf{u}_{{\ell}i}\,.$$ (11)

Unlike the velocity averaging scheme in [12] we adopt the following geometrically motivated, but equivalent formalism to solve multiple such equations: Each such constraint (one for each line) can be converted into a homogeneous linear constraint following the steps in Fig. 3(i). We see that the 90^∘ rotated velocity $\mathbf{R}_{{\ell}i}\mathbf{R}_{\frac{\pi}{2}}\mathbf{u}_{{\ell}i}$ must be perpendicular to the projected camera velocity $\hat{\mathbf{v}}_{i}$. Forming the dot product with this vector yields:

$$\hat{\mathbf{v}}_{i}^{\intercal}\mathbf{R}_{{\ell}i}\mathbf{R}_{\frac{\pi}{2}}\mathbf{u}_{{\ell}i} =\mathbf{v}^{\intercal}\left(u^{\ell}_{yi}\mathbf{e}^{\ell}_{3i}-u^{\ell}_{zi}\mathbf{e}^{\ell}_{2i}\right)=0$$ (12)

$$\text{ with }\mathbf{R}_{\frac{\pi}{2}} =\begin{bmatrix}1&0&0\\ 0&0&-1\\ 0&1&0\end{bmatrix}\,.$$

Note that the above constraint remains, even when inserting the alternative solutions $S_{0},S_{3}$ or $S_{1},S_{2}$, since these only lead to a change in the sign. Stacking $M$ such equations, one for each line, can be summarized as:

$$\underbrace{\begin{bmatrix}u^{\ell}_{y1}{\mathbf{e}_{31}^{\ell}}^{\intercal}-u^{\ell}_{z1}{\mathbf{e}_{21}^{\ell}}^{\intercal}\\ \vdots\\ u^{\ell}_{yM}{\mathbf{e}_{3M}^{\ell}}^{\intercal}-u^{\ell}_{zM}{\mathbf{e}_{2M}^{\ell}}^{\intercal}\\ \end{bmatrix}}_{\doteq\mathbf{D}\in\mathbb{R}^{M\times 3}}\mathbf{v}=\mathbf{0}\,.$$ (13)

We solve this equation again with SVD, by selecting the column of $\mathbf{V}$ corresponding with the minimal singular value. From this system, we also conclude that we need a minimum of two lines to recover $\mathbf{v}$. Again, SVD only recovers $\mathbf{v}$ up to an unknown parity, i.e. both $\pm\mathbf{v}$ satisfy the equation. To disambiguate the solution, we enforce that the projected velocity must point in the same direction as $\mathbf{R}_{{\ell}i}\mathbf{u}_{{\ell}i}$ for each line. This can be written concisely as

$$\hat{\mathbf{v}}_{i}^{\intercal}\mathbf{R}_{{\ell}i}\mathbf{u}_{{\ell}i}=\mathbf{v}^{\intercal}\left(u^{\ell}_{yi}\mathbf{e}^{\ell}_{2i}+u^{\ell}_{zi}\mathbf{e}^{\ell}_{3i}\right)>0\,,$$ (14)

If at least one line does not satisfy this condition, the sign of $\mathbf{v}$ should be flipped. Compared to [12], which uses a more expensive Shur-Complement step to eliminate unknown scale factors, our algorithm only requires a single step of SVD, but yields the same results. This is because our algorithm is actually equivalent to that of [12], as will be shown in the supplementary material.

Note that the above line averaging scheme may run into issues, if two lines are parallel. Making, the common $\mathbf{e}_{1}^{\ell}$ component of $\mathbf{v}$ unobservable. However, this case actually induces a rank deficiency on $\mathbf{D}$, as proved in the supplementary material, and can thus easily be discarded.

(i) single line constraint	(ii) multiple line constraints

Figure 3: (i) The line constraint in Eq. 11 dictates that the 90-degree rotated measured velocity $\mathbf{R}_{{\ell}i}\mathbf{R}_{\frac{\pi}{2}}\mathbf{u}_{{\ell}i}$ should be perpendicular to $\hat{\mathbf{v}}_{i}$, or $\mathbf{v}^{\intercal}\mathbf{H}_{i}^{\intercal}\mathbf{R}_{{\ell}i}\mathbf{R}_{\frac{\pi}{2}}\mathbf{u}_{{\ell}i}=0$. (ii) Each such constraint spans a two-dimensional subspace, in which $\mathbf{v}$ must reside. With a minimum of two such subspaces, the velocity can be found.

4 Implementation

We integrate the aforementioned linear solver into a RANSAC framework for parameter determination of each manifold, followed by fitting over all inliers. Diverging from the approach of [12], our implementation adopts the GC-RANSAC framework [1] for robust geometric model estimation. GC-RANSAC enhances the original RANSAC by introducing a few versatile functionalities tailored for early termination, thereby expediting the overall process. Essentially, GC-RANSAC first iteratively selects a minimal, spatially coherent subset of events ($N=5$) from the incoming event stream, applies the proposed linear solver described in Alg. 1, and evaluates the quality of the resulting hypothesis. Each occurrence of a so-far-the-best hypothesis will trigger a local refinement within a subset of its inliers. This procedure is repeated $M$ times to separately identify $M$ manifolds. We will now delve into the critical aspects influencing this process.

Spatially Coherent Sampler: The manifold’s continuous structure allows for the examination of the data’s spatial coherency. We utilize NAPSAC [35] to sample from the incoming event stream. This approach starts by randomly selecting one event in the space-time volume followed by identifying four additional events within a hyper-sphere centered on the initial event, based on a predetermined radius $r$. In practice, these four points, located within the hyper-sphere, are likely to be inliers.

Angular Reprojection Residual: We employ the angular reprojection residual [22] for inliers selection. The objective is to minimally correct the two lines, the bearing vector emanating from an event and the 3D line, so they could converge at a single point (i.e. $\mathbf{P}_{j}$ in Fig. 1). Unlike the typical image reprojection residual, this measurement is invariant to both rotation and scale.

Local Refinement: RANSAC often results in numerous ineffective iterations. Therefore, when a promising hypothesis emerges, it is recommended to perform local refinement using the inlier sets, which can enhance the inlier ratio and decrease the total number of required iterations. In our work, we introduce two distinct methods for local refinement. The first method leverages the over-determined nature of our proposed linear solver, while the second employs non-linear optimization with Levenberg–Marquardt over an algebraic error (i.e. Eq. 6). As suggested in [1], we randomly select a subset from the inlier set ($N=10$) and repeat this procedure $Q$ times.

5 Experiments

We perform evaluations both on synthetic and real data. We first confirm the runtime improvement and numerical stability of our linear solver. Next, we discuss the impact of the number of used events or lines, over different noise setups. We conclude with experiments on a few public real-world sequences, demonstrating the advantage over existing methods. We quantify the accuracy of our results with the same criterion as [12], the direction error $\phi$ between the estimated and the ground truth velocities, given that the scale is not observable. All experiments are conducted on a 32GB RAM desktop with an Intel Core i9-10900F Processor.

5.1 Simulation

We first evaluate the performance of the proposed linear solver under different setups over synthetic data. Readings from individual manifolds are generated as follows. We first sample randomly directed linear and angular velocities of $0.5\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$ and $15\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{/}\mathrm{s}$ magnitude, respectively. The time window length is set to $0.5\text{\,}\mathrm{s}$, and the virtual event camera has a resolution of 640$\times$480 and a focal length of 320 pixels. Next, we sample a random line in 3D with a finite length and sample random events according to the spatiotemporal strategy in [12]. We study three types of noise with different magnitudes: pixel noise ($0.5\text{\,}\mathrm{px}$), timestamp jitter ($0.5\text{\,}\mathrm{ms}$), and gyroscope noise ($5.0\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{/}\mathrm{s}$), which is assumed to be given by an IMU. The magnitude of the pixel noise and the noise on camera angular velocities are consistent within the same noise level but vary in direction. Timestamp noise follows a zero-mean Gaussian. For a more detailed sensitivity study for different noise sources and levels see the supplementary material. We generate one million random line, velocity, and event configurations, and report the mean and median angle error, as well as the minimum and mean runtime in milliseconds of our method and the one in [12].

Table 1: General comparison between the two solvers.

Method	Runtime ($\text{\,}\mathrm{\SIUnitSymbolMicro s}$)		Error Rate (%)
	min.	avg.	$\geq$$$	$\geq$$$
Gröbner  [12]	1893	2046	1.00	0.28
Linear (ours)	3.00	3.25	0.00	0.00

Runtime Analysis: In each run we record the runtime of both solvers. As reflected in Tab. 1, our solver runs over 600 times faster than the Gröbner basis solver in [12].

Numerical Stability: We further analyze the numerical stability of both methods under the noise-free setup. We report the likelihood of the solver to converge to within a low ($1.0\mathrm{°}$, or $0.1\mathrm{°}$) error. Our solver consistently reaches a zero error, unlike the polynomial solver, which, due to numerical instabilities of the elimination template, fails to converge within a $0.1\mathrm{°}$ error range $1\text{\,}\mathrm{\char 37\relax}$ of times.

Table 2: Noise resilience of our linear solver and the solver in [12]. Left corresponds to the mean, right to the median error in degrees.

Method	num.	Pixel Noise	Time. Jitter	Gyro. Noise
	events	($0.5\text{\,}\mathrm{px}$)	($0.5\text{\,}\mathrm{ms}$)	($5.0\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{/}\mathrm{s}$)
Gröbner  [12]	5	7.80/1.67	3.61/0.83	7.48/3.09
Linear (ours)	5	5.53/1.24	2.87/0.73	6.53/2.47
Linear (ours)	10	0.46/0.15	0.17/0.12	1.50/1.17

Table 3: Real-world experiment results. Left corresponds to the mean, right to the median error in degrees.

Seq. Name	CELC+opt [30]	Gröbner  [12]	Linear only	Linear w/ non-min. solver	Linear w/ non-linear opt.
mountain-normal	27.7/29.3	33.5/33.6	25.2/21.4	17.0/17.2	16.5/14.6
desk-normal	26.6/26.6	26.4/26.7	22.7/23.4	19.8/19.2	22.1/20.7
sofa-normal	24.0/26.1	31.5/29.5	21.9/17.6	20.6/16.1	19.9/15.0

Figure 4: Condensed analysis of the number of used events (left) and used lines (right) over three types of representative noise.

Analysis of the Number of Used Events: In each simulation, we sample 1,000 signal events and introduce a type of representative noise to the measurements. From these, we use the first $5\leq K\leq 1,000$ events as input for our linear solver for a fair comparison and document the resulting error. We repeat this simulation a million times, varying $K$, and report the results in Fig. 4. We observe a clear trend that as the number of used events increases, the error decreases markedly, except when noise is introduced to the camera’s angular velocity. This exception occurs because the solver cannot average out the noise on the angular velocity, regardless of the number of events processed. The other two errors approach near zero when 1,000 events are used. A full analysis of the solver’s performance under each noise type can be found in the supplementary material.

Analysis of the Number of Used Lines: Finally, we validate our velocity averaging scheme. We extend our simulation to multiple manifolds. For each run, we sampled ten lines and, within each line, we selected ten signal events with known line associations as input to our solver. The first $2\leq K\leq 10$ solutions (line and partial motion parameters) from each manifold were taken into the linear averaging scheme, and the error was recorded. This simulation was executed 10,000 times. Fig. 4 shows that as the number of used lines increases, the error drops significantly. A comprehensive analysis of the solver’s performance against various noise types is available in the supplementary material. Additionally, we show quantitative results in Tab. 2. Both the Gröbner Solver and our linear solver use five lines with either five or ten events each. Our approach demonstrates a lower error with noisy measurements, and this margin grows further in overdetermined systems (i.e. $N=10$).

5.2 Real-world Experiment

Similarly to [12], we validate our method on the same data sequence from VECtor Benchmark [11]. Unlike the previous work, we first segmented the event data into non-overlapping intervals of $0.1\text{\,}\mathrm{s}$ each and reduced the overall size of the data to approximately 5,000 events per interval for efficiency. Motion-corrected bearing vectors are then calculated by fusing gyroscope readings from the attached IMU. Next, to construct the chosen sampler used in GC-RANSAC, we multiply the timestamp by a scale factor of 1,000 and established a radius of 50 to compose the spatially coherent graph in the space-time volume. We apply an angular reprojection threshold of $0.2\mathrm{°}$ for inlier selection, consistent across both the primary iterations and the local refinement stages. The number of iterations for each manifold fitting is capped at 100 and evaluated manifolds is capped at 10. In Tab. 3, we summarized the performance, including both mean and median errors, across two baseline approaches and three variants of our proposed method. Importantly, as [12] reports, CELC+opt [30] is limited to certain sub-sequences, where spatial-temporal plane clustering is feasible, while, [12] does not suffer from this limitation. We test three configurations of our method: linear only, linear with non-minimal refinement, and linear with non-linear optimization. Each configuration uses GC-RANSAC with the spatially coherent sampler. The latter two configurations perform different operations when a new best hypothesis is found by the sampler: “Linear w/ non-min. solver” samples 10 events from the found inliers and feeds them to the linear solver, resulting in a refined solution. “Linear w/ non-linear opt.” runs on-manifold Levenberg-Marquardt optimization steps (with line parameters $\mathbf{R}_{\ell}\in SO(3)$) over the 10 selected events and minimizes the algebraic error in Eq. 6. We use a subset of 10 inliers to enhance efficiency and reduce errors, as seen in simulated experiments.

Results: In each configuration, our method achieves a lower error than the two baseline approaches. Additionally, local refinement enhances accuracy significantly. Without refinement it already has a $10\text{\,}\mathrm{\char 37\relax}$ lower mean error than [30], and $24\text{\,}\mathrm{\char 37\relax}$ lower mean error than [12]. Introducing non-minimal solver refinement reduces the mean error by another $18\text{\,}\mathrm{\char 37\relax}$ over the “Linear only” baseline and non-linear optimization reduces it by $16\text{\,}\mathrm{\char 37\relax}$.

6 Conclusion and Future Work

This work introduced a novel, efficient, and linear N-point solver for line-based relative motion estimation of an event camera. Compared to existing works that rely on polynomial system solvers, our method is more numerically stable, over 600 times faster, and allows the identification of degenerate cases explicitly. Moreover, we introduce a novel velocity averaging scheme that is simpler and faster than previous approaches. When combined with GC-RANSAC we show improved normalized velocity estimation compared to existing approaches, at a fraction of the runtime. Finally, the solutions found by our solver deliver new insights into event manifolds generated by lines and thus pave the way for line-based motion estimation with events. Moreover, despite focusing on event cameras in this work, our formulation is fully compatible with line detections from standard cameras. Thus the tools developed in this work can benefit both frame- and event-based computer vision. Our next steps consist of adding uncertainties to the partial velocity readings and applying the fusion strategy asynchronously over time as well as in conjunction with IMU measurements.

7 Appendix

Here we report additional results of our algorithm for varying noise sources in Sec. 7.1, before discussing the proofs of Theorem 1 and Theorem 2 in Sec. 7.2 and Sec. 7.3, as well as the connection of our proposed line averaging scheme with that of [12] in Sec. 7.6. Finally, we provide additional visual insights into the manifolds spanned by events generated by a line. We show that these manifolds can be canonicalized, i.e. reduced to a small family of manifolds which are highly interpretable (see Sec. 7.7).

7.1 Noise Sensitivity Analysis

In Fig. 5, we provide additional results of our method in simulation, as we vary the number of events used by our solver, and the magnitude of the various noise sources, e.g. pixel noise, timestamp jitter, gyroscope noise. As expected, we see that all errors decrease as more events are used, and errors increase as more noise is injected. Again, the only noise source that cannot be completely eliminated through addition of events is the gyroscope noise, which introduces systematic errors. Experimentally, we found that $N=10$ events gives a good tradeoff between the speed of the algorithm, and observed errors for all noise levels and sources.

We also present additional results for differing noise sources and magnitudes of our line averaging scheme in Fig. 6, and analyse the resulting errors as the number of used lines increases. Again we see that all errors tend to zero as more lines are used, except for the gyroscope noise.

7.2 Proof of Theorem 1 on Degeneracies

For clarity, we restate Theorem 1 here:

Theorem 1: If $\text{rank}(\mathbf{A})\geq 5$, with $\mathbf{A}$ defined in Eq. 7, the decomposition in Eqs. (8, 9, 10) always succeeds and yields four distinct solutions. If $\text{rank}(\mathbf{A})<5$ the solver returns infinitely many solutions.

Proof: First assume $\text{rank}(\mathbf{A})\geq 5$. Then SVD returns two distinct principle directions $\pm\hat{\mathbf{x}}$. After decomposition, Eq. 10 yields two more solutions, resulting in a total of four distinct solutions. Now assume that the decomposition fails, and this can happen for three reasons:

Failure to normalize in Eq. 8: Normalization may fail if $\hat{\mathbf{x}}_{4:6}$ has zero norm. However, this case is impossible for a matrix $\mathbf{A}$ with rank $\geq 5$ for the following reason: Let $\mathbf{B},\mathbf{C}$ be the three left and right columns of $\mathbf{A}$ (see Eq. 7). Moreover, note that $\mathbf{C}=\mathbf{T}\mathbf{B}$, where $\mathbf{T}=\text{diag}(t^{\prime}_{1},t^{\prime}_{2},...,t^{\prime}_{N})$ is a diagonal matrix, i.e. each row of $\mathbf{B}$ is a multiple of the corresponding row in $\mathbf{C}$.

If $\hat{\mathbf{x}}_{4:6}$ has zero norm, $\hat{\mathbf{x}}_{4:6}=0$. Next, let $\sigma$ be the smallest singular value of $\mathbf{A}$ corresponding to the solution $\hat{\mathbf{x}}$. Then

$$\mathbf{A}^{\intercal}\mathbf{A}\hat{\mathbf{x}} =\sigma\hat{\mathbf{x}}$$ (15)

$$\begin{bmatrix}\mathbf{B}^{\intercal}\\ \mathbf{B}^{\intercal}\mathbf{T}\end{bmatrix}\begin{bmatrix}\mathbf{B}&\mathbf{T}\mathbf{B}\end{bmatrix}\begin{bmatrix}\hat{\mathbf{x}}_{1:3}\\ 0\end{bmatrix} =\begin{bmatrix}\sigma\hat{\mathbf{x}}_{1:3}\\ 0\end{bmatrix}$$ (16)

$$\begin{bmatrix}\mathbf{B}^{\intercal}\mathbf{B}\hat{\mathbf{x}}_{1:3}\\ \mathbf{B}^{\intercal}\mathbf{T}\mathbf{B}\hat{\mathbf{x}}_{1:3}\end{bmatrix} =\begin{bmatrix}\sigma\hat{\mathbf{x}}_{1:3}\\ 0\end{bmatrix}$$ (17)

The last three rows of the last equation are

$$\mathbf{B}^{\intercal}\mathbf{T}\mathbf{B}\hat{\mathbf{x}}_{1:3}=0\,,$$ (18)

and imply either that $\mathbf{B}^{\intercal}(\mathbf{T}\mathbf{B}\hat{\mathbf{x}}_{1:3})=0$, i.e. $\mathbf{T}\mathbf{B}\hat{\mathbf{x}}_{1:3}$ is in the left null-space of $\mathbf{B}$, or $\mathbf{B}\hat{\mathbf{x}}_{1:3}=0$, i.e. $\hat{\mathbf{x}}_{1:3}$ is in the right null-space of $\mathbf{B}$. Both imply that $\text{rank}(\mathbf{B})<3$. This can only be the case if $\text{rank}(\mathbf{A})=5$, following the assumption. This implies that the smallest singular value is $\sigma=0$. From the first three equations above, this implies that $\mathbf{B}\hat{\mathbf{x}}_{1:3}=0$. But then

$$\mathbf{A}\begin{bmatrix}0\\ \mathbf{x}_{1:3}\end{bmatrix}=\mathbf{T}\mathbf{B}\mathbf{x}_{1:3}=0\,,$$ (19)

which implies that $\hat{\mathbf{x}}=[0^{\intercal}\,\mathbf{x}_{1:3}^{\intercal}]$ is also in the null space of $\mathbf{A}$. We now find that both $\hat{\mathbf{x}}_{1}=[0^{\intercal}\,\mathbf{x}_{1:3}^{\intercal}]$ and $\hat{\mathbf{x}}_{2}=[\mathbf{x}_{1:3}^{\intercal}\,0^{\intercal}]$ are in the null-space of $\mathbf{A}$. These vectors are independent, and render the rank of $\mathbf{A}<5$. This is a contradiction.

Failure to recover $\mathbf{e}^{\ell}_{1}$: Recovering $\mathbf{e}^{\ell}_{1}$ fails if the norm of the cross product in Eq. 10 is $0$. This implies that $\hat{\mathbf{x}}_{4:6}=\lambda\hat{\mathbf{x}}_{1:3}$. For similar reasons as above, this implies that $\hat{\mathbf{x}}_{1:3}$ solves both $\mathbf{B}\mathbf{x}=0$ and $\mathbf{C}\mathbf{x}=0$. This implies that $\lambda$ can be freely varied, which would imply a two-dimensional null space of $\mathbf{A}$ and a rank $\leq 4$ which is again a contradiction.

Line passing through the origin at $t^{\prime}=0$: Note that in such a case, $\mathbf{e}_{3}^{\ell}$ would not be defined, and could cause issues in solving. However, we can then use a different definition of the line, with the direction $\mathbf{d}=\mathbf{e}_{1}^{\ell}$, and point on the line $\mathbf{P}=\mathbf{e}_{1}^{\ell}$. The line moment then becomes $\mathbf{m}=\mathbf{P}\times\mathbf{d}=0$. Inserting this into Eq. 2, transforms Eq. 6 into

$${\mathbf{f}_{j}^{\prime}}^{\intercal}(\mathbf{e}_{3}^{\ell}u_{y}^{\ell}-\mathbf{e}_{2}^{\ell}u_{z}^{\ell})=0\,.$$ (20)

However, this would imply that the system in Eq. 7 has a solution of the form $\hat{\mathbf{x}}=[\hat{\mathbf{x}}_{1:3}^{\intercal}\,\mathbf{0}^{\intercal}]^{\intercal}$, with $\hat{\mathbf{x}}_{1:3}=\mathbf{e}_{3}^{\ell}u_{y}^{\ell}-\mathbf{e}_{2}^{\ell}u_{z}^{\ell}$. However, we proved in the last two cases that such a solution form implies that the rank of $\mathbf{A}$ is smaller than 5. Thus ensuring $\text{rank}(\mathbf{A})\geq 5$ is sufficient for discarding the case where the line passes through the origin.

Figure 5: Analysis of the number of used events over three types of representative noise, i.e. pixel noise, timestamp jitter, and gyroscope noise.

Figure 6: Analysis of the number of used lines over three types of representative noise, i.e. pixel noise, timestamp jitter, and gyroscope noise.

We conclude that if $\text{rank}(\mathbf{A})\geq 5$, the decomposition cannot fail, and always returns four distinct solutions. Moreover, we conclude that a $\text{rank}(\mathbf{A})<5$ yields solutions $\hat{\mathbf{x}}$ from a two dimensional nullspace, which yields infinitely many decompositions. $\blacksquare$

7.3 Proof of Theorem 2 on Solution Multiplicity

For clarity, we restate Theorem 2 here:

Theorem 2: Given a solution $S_{0}=\{\mathbf{e}^{\ell}_{1},\mathbf{e}^{\ell}_{2},\mathbf{e}^{\ell}_{3},u^{\ell}_{y},u^{\ell}_{z}\}$ to the incidence relation in Eq. 6, then

$$S_{1} =\{\mathbf{e}^{\ell}_{1},-\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},u^{\ell}_{y},u^{\ell}_{z}\}\,,$$

$$S_{2} =\{-\mathbf{e}^{\ell}_{1},\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},-u^{\ell}_{y},u^{\ell}_{z}\}\,,$$

$$S_{3} =\{-\mathbf{e}^{\ell}_{1},-\mathbf{e}^{\ell}_{2},\mathbf{e}^{\ell}_{3},-u^{\ell}_{y},u^{\ell}_{z}\}$$

are also solutions. These four solutions are visualized in  Fig. 2. For solutions $S_{1}$ and $S_{2}$ the closest point $-\mathbf{e}_{3}^{\ell}$ on the line is behind the camera, while for solutions $S_{2}$ and $S_{3}$ the line direction $\mathbf{e}^{\ell}_{1}$ is flipped, which represents an ambiguity in the definition of direction of $\mathbf{L}$.

Proof: We will only prove solutions $S_{1}$ and $S_{2}$ since $S_{3}$ can be derived from a composition of $S_{1}$ and $S_{2}$. Inserting $S_{1}=\{{\mathbf{e}^{\ell}_{1}}^{\prime},{\mathbf{e}^{\ell}_{2}}^{\prime},{\mathbf{e}^{\ell}_{3}}^{\prime},{u^{\ell}_{y}}^{\prime},{u^{\ell}_{z}}^{\prime}\}=\{\mathbf{e}^{\ell}_{1},-\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},u^{\ell}_{y},u^{\ell}_{z}\}$ into Eq. 6, we have

$$\ \ \ t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}({u^{\ell}_{z}}^{\prime}{\mathbf{e}^{\ell}_{2}}^{\prime}-{u^{\ell}_{y}}^{\prime}{\mathbf{e}^{\ell}_{3}}^{\prime})+{\mathbf{f}_{j}^{\prime}}^{\intercal}{\mathbf{e}^{\ell}_{2}}^{\prime}$$

$$= \ \ \ t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}(-\mathbf{e}^{\ell}_{2})-u^{\ell}_{y}(-\mathbf{e}^{\ell}_{3}))+{\mathbf{f}_{j}^{\prime}}^{\intercal}(-\mathbf{e}^{\ell}_{2})$$

$$= \ \ \ -(t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2})$$

$$= \ \ \ 0\hskip 210.55022pt$$

and $S_{2}=\{{\mathbf{e}^{\ell}_{1}}^{\prime\prime},{\mathbf{e}^{\ell}_{2}}^{\prime\prime},{\mathbf{e}^{\ell}_{3}}^{\prime\prime},{u^{\ell}_{y}}^{\prime\prime},{u^{\ell}_{z}}^{\prime\prime}\}=\{-\mathbf{e}^{\ell}_{1},\mathbf{e}^{\ell}_{2},-\mathbf{e}^{\ell}_{3},-u^{\ell}_{y},u^{\ell}_{z}\}$ yields

$$\ \ \ t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}({u^{\ell}_{z}}^{\prime\prime}{\mathbf{e}^{\ell}_{2}}^{\prime\prime}-{u^{\ell}_{y}}^{\prime\prime}{\mathbf{e}^{\ell}_{3}}^{\prime\prime})+{\mathbf{f}_{j}^{\prime}}^{\intercal}{\mathbf{e}^{\ell}_{2}}^{\prime\prime}$$

$$= \ \ \ t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-(-u^{\ell}_{y})(-\mathbf{e}^{\ell}_{3}))+{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2}$$

$$= \ \ \ t_{j}^{\prime}{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}^{\ell}_{2}-u^{\ell}_{y}\mathbf{e}^{\ell}_{3})+{\mathbf{f}_{j}^{\prime}}^{\intercal}\mathbf{e}^{\ell}_{2}$$

$$= \ \ \ 0\hskip 210.55022pt$$

$\blacksquare$

7.4 Handling of Parallel Lines

As mentioned in the main text, parallel lines may cause difficulties in identifying the $\mathbf{e}_{1}^{\ell}$ direction of the camera velocity. However, we can identify this case easily by checking the rank of $\mathbf{D}$. If it is lower than 2, we can discard the sample, and select a new one, or even use another RANSAC loop to select pairs of lines until the rank of $\mathbf{D}$ is at least 2. Let us now prove that parallel lines cause a rank deficiency in $\mathbf{D}$.

Proof: We will proceed in showing that if two lines are parallel, the two corresponding rows $\mathbf{r}_{1}=u_{y1}^{\ell}\mathbf{e}_{31}^{\ell}-u_{z1}^{\ell}\mathbf{e}_{21}^{\ell}$ and $\mathbf{r}_{2}=u_{y2}^{\ell}\mathbf{e}_{32}^{\ell}-u_{z2}^{\ell}\mathbf{e}_{22}^{\ell}$ in $\mathbf{D}$ are parallel and will thus result in rank deficiency (see Eq. 13). Expanding $\mathbf{v}$ in the two line coordinate frames yields

$$\lambda_{1}\mathbf{v}=u_{x1}^{\ell}\mathbf{e}_{11}^{\ell}+u_{y1}^{\ell}\mathbf{e}_{21}^{\ell}+u_{z1}^{\ell}\mathbf{e}_{31}^{\ell}$$ (21)

$$\lambda_{2}\mathbf{v}=u_{x2}^{\ell}\mathbf{e}_{12}^{\ell}+u_{y2}^{\ell}\mathbf{e}_{22}^{\ell}+u_{z2}^{\ell}\mathbf{e}_{32}^{\ell}$$ (22)

with unknown scale factors $\lambda_{1},\lambda_{2}$. For parallel lines $\mathbf{e}^{\ell}_{11}=\mathbf{e}^{\ell}_{12}\doteq\mathbf{e}^{\ell}_{1}$. Computing $\mathbf{e}^{\ell}_{1}\times\mathbf{v}$ in two ways (with two expansions of $\mathbf{v})$, we recover exactly the rows of $\mathbf{D}$ by

$$\lambda_{1}(\mathbf{e}^{\ell}_{1}\times\mathbf{v})=u_{y1}^{\ell}\mathbf{e}_{31}^{\ell}-u_{z1}^{\ell}\mathbf{e}_{21}^{\ell}=\mathbf{r}_{1}$$ (23)

$$\lambda_{2}(\mathbf{e}^{\ell}_{1}\times\mathbf{v})=u_{y2}^{\ell}\mathbf{e}_{32}^{\ell}-u_{z2}^{\ell}\mathbf{e}_{22}^{\ell}=\mathbf{r}_{2}$$ (24)

It follows that $\mathbf{r}_{1}=\frac{\lambda_{1}}{\lambda_{2}}\mathbf{r}_{2}$, i.e. they are parallel. $\blacksquare$

7.5 Global Optimality of $\mathbf{R}_{l}$ and $\mathbf{u}_{l}$

As noted in the main text, while the SVD-based solver which recovers $\hat{\mathbf{x}}$ from a set of incidence relations (Eq. 7) finds a globally optimal solution $\hat{\mathbf{x}}$, it is not clear if the decomposed solution $\mathbf{R}_{l},\mathbf{u}_{l}$ is also optimal with respect to the same objective. We prove this here.

Proof: We will prove this by way of contradiction. Assume given the SVD-based solution

$$\hat{\mathbf{x}}=\arg\min_{\mathbf{x}}\|\mathbf{A}\mathbf{x}\|^{2}\quad\text{such that}\quad\|\mathbf{x}\|^{2}=1.$$

which is globally optimal, and decomposition $\mathbf{R}_{l},\mathbf{u}_{l}$ with $\hat{\mathbf{x}}=\mathbf{x}(\mathbf{R}_{l},\mathbf{u}_{l})$. Assume that there exists a different, more optimal $\mathbf{R}^{\prime}_{l},\mathbf{u}^{\prime}_{l}$ with $\hat{\mathbf{x}}^{\prime}=\mathbf{x}(\mathbf{R}^{\prime}_{l},\mathbf{u}^{\prime}_{l})$. Then $\|\mathbf{A}\hat{\mathbf{x}}^{\prime}\|^{2}<\|\mathbf{A}\hat{\mathbf{x}}\|^{2}$ but this is impossible since it would imply that $\hat{\mathbf{x}}^{\prime}$ is more optimal than $\hat{\mathbf{x}}$, but $\hat{\mathbf{x}}$ is already optimal. This implies that the objective is already optimal in $\mathbf{R}_{l},\mathbf{u}_{l}$ which concludes the proof. $\blacksquare$

7.6 Connection between the Proposed Line Averaging Scheme and [12]

The presented velocity averaging scheme is conceptually simpler, and lends itself to geometric interpretation, unlike the scheme in [12]. However, surprisingly these schemes are actually equivalent, as will be demonstrated next. In [12], Eq. 11 is used to set up a number of constraints

$${\mathbf{e}_{2i}^{\ell}}^{\intercal}\mathbf{v}-\lambda_{i}u_{yi}^{\ell}=0$$ (25)

$${\mathbf{e}_{3i}^{\ell}}^{\intercal}\mathbf{v}-\lambda_{i}u_{zi}^{\ell}=0$$ (26)

Introducing unknowns $\mathbf{v}$ and $\{\lambda_{i}\}$, one for each line. Stacking multiple of these equations results in a system

$$\underbrace{\begin{bmatrix}{\mathbf{e}_{21}^{\ell}}^{\intercal}&-u_{y1}^{\ell}&\cdots&0\\ {\mathbf{e}_{31}^{\ell}}^{\intercal}&-u_{z1}^{\ell}&\cdots&0\\ \vdots&\ddots&\ddots&\vdots\\ {\mathbf{e}_{2M}^{\ell}}^{\intercal}&0&\cdots&-u_{yM}^{\ell}\\ {\mathbf{e}_{3M}^{\ell}}^{\intercal}&0&\cdots&-u_{zM}^{\ell}\\ \end{bmatrix}}_{\mathbf{E}}\begin{bmatrix}\mathbf{v}\\ \lambda_{1}\\ \vdots\\ \lambda_{M}\end{bmatrix}=\mathbf{0}\,.$$ (27)

This system is then multiplied from the left with $\mathbf{E}^{\intercal}$, and the Shur complement trick is employed to eliminate the extraneous variables $\lambda_{i}$, resulting in the equation $\mathbf{F}\mathbf{v}=\mathbf{0}$, where we use the following definitions:

$$\mathbf{F} =\mathbf{U}-\mathbf{W}\mathbf{V}^{-1}\mathbf{W}^{\intercal}$$ (28)

$$\mathbf{U} =\sum_{i=1}^{M}(\mathbf{e}^{\ell}_{2i}{\mathbf{e}^{\ell}_{2i}}^{\intercal}+\mathbf{e}^{\ell}_{3i}{\mathbf{e}^{\ell}_{3i}}^{\intercal})$$ (29)

$$\mathbf{V} =\text{diag}\left({u_{y1}^{\ell}}^{2}+{u_{z1}^{\ell}}^{2},\cdots,{u_{yM}^{\ell}}^{2}+{u_{zM}^{\ell}}^{2}\right)$$ (30)

$$\mathbf{W}^{\intercal} =\begin{bmatrix}-u_{y1}^{\ell}{\mathbf{e}_{21}^{\ell}}^{\intercal}-u_{z1}^{\ell}{\mathbf{e}_{31}^{\ell}}^{\intercal}\\ \vdots\\ -u_{yM}^{\ell}{\mathbf{e}_{2M}^{\ell}}^{\intercal}-u_{zM}^{\ell}{\mathbf{e}_{3M}^{\ell}}^{\intercal}\\ \end{bmatrix}$$ (31)

Inserting the equations, and simplifying we get

$$\mathbf{F}= \sum_{i=1}^{M}(\mathbf{e}^{\ell}_{2i}{\mathbf{e}^{\ell}_{2i}}^{\intercal}+\mathbf{e}^{\ell}_{3i}{\mathbf{e}^{\ell}_{3i}}^{\intercal})$$

$$-\frac{\left(-u_{yi}^{\ell}\mathbf{e}_{2i}^{\ell}-u_{zi}^{\ell}\mathbf{e}_{3i}^{\ell}\right)\left(-u_{yi}^{\ell}{\mathbf{e}_{2i}^{\ell}}^{\intercal}-u_{zi}^{\ell}{\mathbf{e}_{3i}^{\ell}}^{\intercal}\right)}{{u_{yi}^{\ell}}^{2}+{u_{zi}^{\ell}}^{2}}$$ (32)

$$= \sum_{i=1}^{M}\frac{1}{{u_{yi}^{\ell}}^{2}+{u_{zi}^{\ell}}^{2}}\left({u_{zi}^{\ell}}^{2}\mathbf{e}_{2i}^{\ell}{\mathbf{e}_{2i}^{\ell}}^{\intercal}+{u_{yi}^{\ell}}^{2}\mathbf{e}_{3i}^{\ell}{\mathbf{e}_{3i}^{\ell}}^{\intercal}\right.$$

$$-\left.u_{yi}^{\ell}u_{zi}^{\ell}\mathbf{e}^{\ell}_{3i}{\mathbf{e}^{\ell}_{2i}}^{\intercal}-u_{yi}^{\ell}u_{zi}^{\ell}\mathbf{e}^{\ell}_{2i}{\mathbf{e}^{\ell}_{3i}}^{\intercal}\right)$$ (33)

$$= \sum_{i=1}^{M}\frac{\left(u_{yi}^{\ell}\mathbf{e}_{3i}^{\ell}-u_{zi}^{\ell}\mathbf{e}_{2i}^{\ell}\right)\left(u_{yi}^{\ell}\mathbf{e}_{3i}^{\ell}-u_{zi}^{\ell}\mathbf{e}_{2i}^{\ell}\right)^{\intercal}}{{u_{zi}^{\ell}}^{2}+{u_{zi}^{\ell}}^{2}}$$ (34)

$$= \hat{\mathbf{D}}^{\intercal}\hat{\mathbf{D}}$$ (35)

with

$$\hat{\mathbf{D}}=\begin{bmatrix}\frac{1}{\sqrt{{u_{y1}^{\ell}}^{2}+{u_{z1}^{\ell}}^{2}}}\left(u_{y1}^{\ell}\mathbf{e}_{31}^{\ell}-u_{z1}^{\ell}\mathbf{e}_{21}^{\ell}\right)\\ \vdots\\ \frac{1}{\sqrt{{u_{yM}^{\ell}}^{2}+{u_{zM}^{\ell}}^{2}}}\left(u_{yM}^{\ell}\mathbf{e}_{3M}^{\ell}-u_{zM}^{\ell}\mathbf{e}_{2M}^{\ell}\right)\\ \end{bmatrix}$$ (36)

The linear system thus becomes

$$\mathbf{F}\mathbf{v} =\mathbf{0}$$ (37)

$$\hat{\mathbf{D}}^{\intercal}\hat{\mathbf{D}}\mathbf{v} =\mathbf{0}$$ (38)

$$\hat{\mathbf{D}}^{\intercal}(\hat{\mathbf{D}}\mathbf{v}) =\mathbf{0}$$ (39)

if $\hat{\mathbf{D}}^{\intercal}$ has full rank this implies that

$$\hat{\mathbf{D}}\mathbf{v} =\mathbf{0}$$ (40)

Note that $\hat{\mathbf{D}}$ is identical to $\mathbf{D}$ in Eq. 13 up to normalization of each velocity $\mathbf{u}_{i}^{\ell}$ separately. This normalization strategy can be seamlessly integrated into the computation of $\mathbf{D}$. Moreover, computing $\mathbf{D}$ is much simpler than computing $\mathbf{F}$.

7.7 Canonicalization of the Manifold

(i) event transformations

(ii) eventail in 2D

Figure 7: Events triggered by a line observed by an event camera span a non-linear manifold called éventail (A). We show that this manifold imposes a linear constraint on the partial camera velocity and line parameters. With this insight, we can design a linear solver for these quantities that is both fast and highly interpretable, and characterize all manifolds (A) by transforming them into canonical form via rotation compensation (B), and transformation into the line coordinate frame (C). In the $\hat{y}^{\ell}t$-plane, these manifolds trace a family of curves (ii), depending on the configuration of $u^{\ell}_{z}$ and $u^{\ell}_{y}$.

The incidence relation in Eq. 6 yields a simple way to visualize the manifold in its canonical form, and also shows the dependence on the line velocity parameters $u^{\ell}_{y}$ and $u^{\ell}_{z}$. To reach this canonical form, we simply rotate the bearing vectors of all events into the line-dependent coordinate frame by replacing $\mathbf{f}^{\prime}=\mathbf{R}_{\ell}\hat{\mathbf{f}}^{\prime}$. We visualize this transformation in Fig. 7, where we transition from raw events in normalized coordinates (A), derotated events (B), and then events in the line reference frame (C). This coordinate frame corresponds with that of a line that is parallel to the camera’s $x$-axis. Doing this replacement yields

$$t^{\prime}\hat{\mathbf{f}_{j}^{\prime}}^{\intercal}(u^{\ell}_{z}\mathbf{e}_{y}-u^{\ell}_{y}\mathbf{e}_{z})+{\hat{\mathbf{f}_{j}^{\prime}}}^{\intercal}\mathbf{e}_{y}=0\,,$$ (41)

where $\mathbf{e}_{x/y/z}$ corresponds to the unit vectors in the camera coordinate frame. Distributing and diving out the third component of $\hat{\mathbf{f}_{j}^{\prime}}$, i.e. transitioning to normalized coordinates in the new line reference frame, we reach

$$0 =t^{\prime}(u^{\ell}_{z}\hat{y}^{\ell}-u^{\ell}_{y})+\hat{y}^{\ell}\implies \hat{y}^{\ell}=\frac{u^{\ell}_{y}t^{\prime}}{1+u^{\ell}_{z}t^{\prime}}$$ (42)

where $\hat{y}^{\ell}$ is the event $y$-coordinate in normalized, line coordinates. This form describes the shape of the manifold in two dimensions and is visualized in Fig. 7(ii) for varying $u^{\ell}_{y}$ and $u^{\ell}_{z}$.

From these visualization we make a number of observations: First, configurations with $u^{\ell}_{z}=0$ trace straight lines, corresponding to planar manifolds in the line coordinate frame. Note, however, that in the derotated frame (B) these may still be non-planar. Second, we see that $u^{\ell}_{z}<0$ induces a curvature in the manifold which increases as time progresses. This configuration corresponds to a camera approaching the line, and thus the reduced distance increases the apparent motion, which results in a larger slope. Finally, $u^{\ell}_{z}>0$ results in flattened curves. This corresponds to cameras retracting from the line, which reduces the apparent motion, and thus reduces the slope in the manifold.