Open Access
Add to BibSonomy
Abstract
We have built and characterized a compact, simple and flexible 3D camera based on interferometric fringe projection and stereo reconstruction. The camera uses multi-frame active stereo as basis for 3D reconstruction, providing full-field 3D images with 3D measurement standard deviation of 0.09 mm, 12.5 Hz 3D image capture rate and 3D image resolution of 500 × 500 pixels. Interferometric projection enables a compact, low-power projector that consumes < 1 W of electrical power. The key component in the projector, a movable micromirror, has undergone initial vibration, thermal vacuum cycling (TVAC) and radiation testing, with no observed component degradation. The system's low power, small size and component longevity makes it well suitable for space applications.
© 2021 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Structured Light and Active Stereo (multi-frame) systems can deliver high accuracy 3D data at short range, through projection of multiple patterns. However, these systems tend to be bulkier and more complex than their Assisted Stereo (single frame) counterparts, as the projectors are normally DMD based. These projectors can project any image, a feature not required for multi-frame systems, where sinusoidal or binary patterns are the patterns of choice.
This work demonstrates a 3D measurement system using an extremely simple projector based on interferometric fringe projection and is a continuation of the results reported in [1]. The interferometric projector allows us to project sinusoidal patterns with a defined frequency and phase onto the scene. Such a projector gives a very small, robust, and low-power system with excellent 3D measurement abilities, providing dense 3D point clouds and 0.1 mm 3D measurement accuracy. The narrow spectrum of the light source simplifies sunlight filtering. The 3D measurement system was developed for use cases in space, where simplicity, compactness and low power are attractive features.
The core component of the system is a highly planar and robust tip-tilt-piston micromirror. The mirror allows us to realize a Michelson interferometer for flexible and fast pattern projection, and to control phase and frequency of the projected patterns. Unlike traditional DMD based systems [2] the control electronics can be extremely simple, only requiring four control voltages for positioning the mirror for sufficient control over pattern frequency and phase. The mirror itself is highly robust, and has undergone initial TVAC, vibration and gamma radiation testing without showing signs of degradation.
Our system compares favorably to other existing 3D imaging principles. It provides much of the same 3D imaging performance of 3D structured light systems, having sub-mm accuracy and full-field 3D data. At the same time, it retains much of the hardware simplicity of active/passive stereo cameras like Intel Realsense [3], not requiring advanced projection optics and parts like DMDs [2,4]. It is also able to provide full-field 3D images, where the depth is individually measured in each pixel, unlike active stereo systems employing a single static pattern [3].
At the same time, the 3D imaging system poses unique challenges in terms of pattern design, pattern decoding and laser speckle. The limitation of only being able to project sinusoidal patterns means that we cannot resort to traditional Gray Code – Phase Step (GCPS) methods [5], line sweep methods or more complex patterns [6]. Instead we use multi-frequency phase shifting [7], where we similar to Mirdeghan et al. [8] use an optimization approach for maximum accuracy and robustness. Existing research has done theoretical considerations on frequency selection [9] or used linear frequency sweeps [10]. We explore the use of other mathematical primitives (Golomb rulers) and data-driven experimental frequency optimization as our solution for the problem. We furthermore outline the design principles we have used for choosing the patterns, to enable later robust pattern decoding and management of laser speckle.
The rest of the paper is organized as follows. In section 2, we present the principle of interferometric fringe projection and estimate the requirements for such a principle. Then we present a piezoelectric micromirror developed at SINTEF MiNalab, and show that the characteristics of this mirror is suitable for the application. We present images of the projected fringes. In section 3, we discuss the 3D reconstruction algorithm, in light of the limitations of our projected patterns. We discuss how to select a set of patterns that enables robust and precise 3D reconstruction, and we demonstrate successful 3D reconstruction. In section 4, we present the assembled 3D camera, 3D images and we quantify the 3D measurement performance of our system.
We have demonstrated that we can build a fringe projector based on a Michelson interferometer, using a MEMS mirror to adjust phase and frequency of the mirror. Phase and frequency control is sufficient for a robust 3D reconstruction algorithm. We choose an active stereo setup in order to overcome problems related to pattern irregularities and drift, and we show that proper pattern selection provides robust and accurate 3D measurements. Our 3D measurement system shows a measurement uncertainty of 0.1 mm, 500 × 500 3D pixels and up to 12.5 Hz 3D image capture rate.
2. Hardware requirements and design
Many 3D applications require a compact, robust, and low power system, with the possibility of filtering out background illumination, particularly sunlight. Small size facilitates camera integration and enables mounting on robotic arm or drone. Low power reduces thermal issues and enables battery driven / off-grid operation, while sunlight filtering enables outdoor applications. In collaboration with ESA project officers, we define a set of target specifications for a competitive and relevant 3D imaging system, listed in Table 1.
Table 1. 3D camera target specifications.
A projection system based on a diode laser can be compact and efficient due to the use of a high brightness source with high electrical to optical (wall plug) efficiency, and it is well suited for sunlight filtering due to the narrow emission spectrum of the laser. For these reasons, we choose a laser based projector. As noted in the introduction, there are 3D reconstruction algorithms available that require only the projection of sinusoids with varying spatial frequency and phase. A very simple method for generating such patterns using a laser diode would be to use an interferometer, and we choose to base our projector on a Michelson interferometer. The 3D camera is schematically shown in Fig. 1, with the Michelson interferometer placed between the two cameras. We choose to use two cameras + projector, such that our system is an Active stereo system. One advantage of choosing this configuration is that 3D reconstruction relies on camera – camera calibration, instead of projector-camera calibration.
2.1 Interferometric fringe projection
Interferometric fringe projection is based on the interference between two mutually coherent, angularly displaced beams. We choose to generate these beams by propagating the light from a single mode laser diode through a Michelson interferometer, but other similar approaches have been demonstrated, such as the use of a Fabry-Pérot interferometer [11] or the interference between single mode fibers [12]. Our Michelson interferometer is shown schematically in Fig. 1 and Fig. 2. The movable mirror is positioned at an angle relative to the reference mirror. After recombining the two beams in the beam splitter, we focus the two beams, thereby generating two focus points. We can describe the interference pattern by considering the interference between these two point sources, which gives a situation very similar to the Young double slit interferometer. In the case of a collimated beam through the interferometer, the reference arm generates a point source on the optical axis, and the movable arm generates a point located at a distance $d = f\tan (\theta )$ from the optical axis. Here, f is the focal length of the lens, and $\theta $ is the angular displacement of the beam from the movable arm (, the mirror being displaced by $\theta /2$). If also allowing a piston movement in the movable arm, we can generate a relative phase shift $\delta \varphi = \frac{{4\pi }}{\lambda }\mathrm{\Delta }z$, where $\mathrm{\Delta }z$ is the piston movement of the movable mirror. The interference pattern at a distance R from the interferometer, and a distance x from the optical axis can, in the limit of $x \ll R$, be expressed as
We observe that the spatial frequency of the pattern is given by the displacement angle of the movable mirror, while the phase of the pattern is given by the piston movement of the movable mirror. From Eq. (1), we can estimate the required angular displacement and positioning accuracy of the movable mirror. These parameters are estimated below, and summarized in Table 2. A fringe density on the order of 1 - 10 fringes pr degree in the image is suitable for 3D measurements. 1 finge pr degree would require:
where we have used $\frac{x}{R} = \tan \alpha $, and $\alpha $ being the angle in image space. For small angles, we get where the final result assumes $\lambda = 800\; nm$, $f = 10\; mm$ and $\alpha = \frac{\pi }{{180}}.$ 10 fringes pr degree would require 23 mrad tilt of the mirror. Moreover, for an un-distorted sinusoidal pattern, we require the mirror to be highly planar, typically on the order of $\lambda /10$. Similarly, a phase shift of 2$\pi /3$ will require a piston movement of $133\; nm$, and the precision in piston movement should be comparable to the flatness of the mirror, i.e. $\lambda /10$. While the exact numbers depend on system design, we require a mirror with modest deflection angle, but excellent positioning accuracy.Table 2. Mirror requirements and performance.
One further requirement of the mirror is repositioning speed. From Table 1, we will require our 3D measurement system to acquire on the order of 150 images pr second, i.e. every 6.7 ms. If we would like the pattern to be stationary, and available for camera exposure most of this time, repositioning needs to take on the order of 1–2 ms. The mirror requirements are listed in Table 2.
2.2 Piezoelectric micromirror
In previous publications [13,14], we have presented an ultra-planar piezoelectric micromirror with tip/tilt and piston abilities. Peak to valley planarity has been reported to 15 nm both in actuated and non-actuated state, well below our requirement of $\lambda /10$. While this planarity was measured in static mode, the effects of dynamic mirror deformation will be negligible [15] due to the combination of high stiffness, small actuation distance and the fact that the mirror is kept stationary during exposure. The mirror is a gold coated silicon mirror with an aperture of 3 mm. The mirror is suspended on a closed silicon membrane, actuated by a PZT thin film with 8 segments, as shown in Fig. 3.
Fig. 2. (Left) Optical layout of the interferometric projector. Components: Laser: ThorLabs L785H1, SM laser diode, Laser collimating lens: ThorLabs A397-B, f = 11 mm, Beam splitter: ThorLabs BS005, Reference mirror: ThorLabs PF05-03-P01, ½”. Movable mirror: SINTEF MiNa-lab Hex Pimp, 3 mm aperture, Projection lenses: Edmund Optics #38-333-INK, f = 20 mm and Edmund Optics #49-531-INK, f = -30 mm. (Right) Miniaturized projector design with footprint 32 × 16 mm.
Fig. 3. Schematic layout of the piezoelectric micromirror. Left: cross section schematics. Right: Microscope image of the mirror chip, showing the mirror aperture, the 8 PZT segments as well as electrical connections. The chip is 8 × 8 mm, and the mirror has a diameter of 3 mm.
We apply a suitable control voltage loop and record the mirror deflection using a ThorLabs PDP90A position sensing device (PSD). A distance of 580 mm between the mirror and the PSD results in a measurement accuracy of 0.7 µrad of the deflected laser beam or 2 nm elevation of one edge of the mirror. An example mirror positioning sequence with 10 equally large voltage steps and 100 Hz re-positioning rate is shown in Fig. 4 (left), but the sequence is freely programmable to provide any desired set of frequency and phase, limited only by the angular deflection of the mirror. We observe re-positioning within 2 ms, adequate for our purposes, with stationary mirror positions in between movements. In this sequence we use only half of the available electrodes and maximum mirror deflection is around 3 mrad. Maximum mirror deflection with all electrodes active is around 6 mrad, which is well suited for our application. We also observe that the mirror deflection does not vary linearly with voltage, an effect which will be discussed further in section3.3.
Fig. 4. Mirror repositioning. Left: Mirror voltage loop with 10 discrete positions, with a new position every 10 ms. The small peak in front of each new position is an intended position overshoot to reduce piezoelectric creep. Right: Close-up of relative motion of the mirror during camera exposure for one of the steps (step 4 from the bottom in the figure to the left).
To improve stability during exposure we move the mirror beyond the intended position and back. This overshoot can be seen as a small spike before each plateau in Fig. 4 (left). The overshoot reduces time dependent deformation (creep) of the thin-film PZT material [16], and ensures a stationary projected pattern during camera exposure. Figure 4 (right) shows a close-up of the mirror position during exposure for one of the steps over 10 consecutive sequences. The fact that each sequence is almost on top of the other is an indication of the excellent repeatability of the mirror position. We observe that the stability of the mirror during exposure is limited by piezoelectric creep, and not by repositioning stability or mirror oscilations (ringing). The mirror drift varies for the different steps, but are below 5 µrad for all but the longest step. By tuning the overshoot more carefully, it is possible to further improve mirror stability during exposure. However, a relative deflection of 5 µrad equates to a lift of one mirror edge by 15 nm and is more than adequate for our purposes.
However, we observe that the bending force from the actuator on the mirror does change the peak to valley planarity. We measure that at maximum actuation this change can be in the range of 30 nm or in the order of the piezoelectric creep that we observe. This is still well below the target of λ/10 and is not a limiting factor for 3D accuracy in the 3D camera presented here.
Comparing with the requirements from section 2, we find that the mirror is well suited for our application, in terms of planarity, positioning accuracy and repositioning speed. The performance of the mirror is compared with our requirements, listed in Table 2.
2.3 Sample patterns
With the results presented in sections 2.1 and 2.2, we proceeded to build the interferometer shown in Fig. 2 (left). The length of the interferometer arms (including phase shifts at mirror surfaces) was adjusted to ensure maximum modulation contrast, implying that the arms are of equal optical length to within the coherence length of the laser and that the intensity profiles of the arms align. We record the pattern as projected on a planar wall, shown in Fig. 5. (left). We observe a sinusoidally varying pattern, but with varying local amplitude and local frequency. The slowly varying intensity envelope is that of the Gaussian beam emitted from the diode laser. Circular shapes are diffraction rings / interference from optical elements and contaminations on optical surfaces. We observe speckle, which we measure to have a speckle contrast close to proportional to the f-number of the camera lens. We also observe that the speckle pattern is non-stationary, and that the speckle patterns are un-correlated between the two cameras. These findings combined turn speckle into a source of measurement noise, as the projected patterns will start to look different when observed from the two cameras. This effect will be further discussed in section 4.
Fig. 5. (Left) Pattern projected on a planar surface. (Right) Cross section of the intensity distribution. The profile is sinusoidal, but with variations in local amplitude and frequency.
To investigate stability of the projected patterns, we let the system run for 8 hours while capturing images. As shown in Fig. 6, we find that the spatial frequency of the fringes is highly stable, and that the phase of the patterns varies by up to 60 degrees. The frequency of the patterns is determined predominantly by the actuation angle of the movable mirror, while the phase of the pattern will, in addition to piston movement in the mirror, also depend on differences in thermal fluctuations between the two interferometer arms and drift in laser wavelength. The variability of projected phase poses constraints on our reconstruction algorithm, not allowing the algorithm to make assumptions of exact phase, phase shift and freqencies.
Fig. 6. (Left) Cross section of the projected fringe pattern, with 10 samples captured over 8 hours. The stability in intensity and frequency is excellent. (Right) A zoom-in shows that the phase of the projected pattern varies by up to 60 degrees.
3. Pattern selection and 3D reconstruction algorithms
Our 3D reconstruction algorithm is designed with a basis in stereo algorithms. This means that we are interested in first establishing a subpixel disparity estimate between camera 1 and 2, and subsequently performing a homographic transform to convert the disparity estimates into a 3D pointcloud.
The system works by illuminating the scene with patterns of selected spatial frequencies and phase – horizontal sine waves as exemplified in Fig. 5. (left). The goal is that every pixel receives its unique temporal “signature” that makes it possible to determine for a pixel A in camera 1 which is the corresponding pixel B in camera 2. As we have a stereo system, we can use the epipolar constraint [17] to constrain the search in the vertical direction, meaning that the projected codes only need to be unique in the horizontal direction. Unlike traditional stereo, the use of temporally changing patterns that cover the whole field-of-view allows us to capture detailed, full 3D data of surfaces that do not have any structure or surface texture.
The use of sinusoidal patterns would normally hint towards the use of $\textrm{atan}$ based reconstruction algorithms for matching [7], possibly combined with number theoretic or other approaches [18] for resolving the phase ambiguities . However, as the mirror cannot guarantee an exact phase shift between patterns (confer section 2.2) and the phase will drift over time, we have selected to use a conservative approach relying on zero-normalized cross-correlation (ZNCC) instead, noting that it outperforms many other decoders [8]. In addition, ZNCC can easily be made robust towards differences in signal amplitude between the two cameras, unlike other disparity metrics like sum of absolute differences.
3.1 Disparity estimation algorithm
We receive a stack of images ${F_1}({x,y,i} )$ and ${F_2}({x,y,i} )$ from each camera where $x,y\; \in ({0,{R_1}} ),\; ({0,{R_2}} )$ are pixel indices and $i \in ({1,N} )$ is corresponding image index. Camera calibration [19] and image rectification ensures that corresponding rows in the resulting rectified image stacks ${R_1}({x,y,i} )$ and ${R_2}({x,y,i} )$ are on the same epipolar plane.
Horizontally we match the per-pixel signatures against each other on a row-by-row basis. We normalize each pixel by
where $\mu $ and $\sigma $ are per-pixel mean and standard deviation. In our experience, this normalization reduces signal variations due to specular reflections, laser speckle, ambient light, and similar imaging artifacts, which would otherwise negatively affect results. The metric is also highlighted as the best performing by Mirdeghan et al. [8].We find the stereo disparity by calculating the cross-correlation matrix $C({{x_1},{x_2}} )= \mathop \sum \nolimits_{i = 1}^N {N_1}({x_1}, y,i ){N_2}({{x_2},y,i} )$ and subsequently for each pixel ${x_1} $ in camera 1 determining the pixel index ${x_2}$ that maximizes the cross-correlation between the per-pixel signatures, e.g. $\mathop {\textrm{argmax}}\limits_{{x_2} \in ({0,{R_1}} )} C({x1,x2} )$. An example cross-correlation matrix $C $ (CCM) for a single row is shown in Fig. 7 (right). We observe that there is a clear peak trajectory in the CCM except in the middle, where scene shadowing results in no signature being present.
Fig. 7. Left: Example 3D result imaging of a brick toy build of a Mars Rover. Red line indicates camera row picked out for illustrating example cross-correlation matrix (CCM) at the right. The peak (in pink) indicates corresponding pixels from camera 1 and 2. For the same epipolar line, each column in the CCM reflect a single pixel in camera 1, and each row a single pixel in camera 2. In the left part of the selected camera row, there is no signal, as the object in the foreground casts a shadow here.
We determine the maximum position of the peak, and refine the estimate by fitting a parabola to the peak location and its neighboring CCM elements. The location of the parabola peak is used as the subpixel peak location and disparity estimate. To speed up calculations, we establish a disparity window related to the working range of the camera. This means that we can calculate a sparse CCM matrix and limit the search window for the peak detection.
We calculate depth data directly from stereo disparity estimates by using a homographic transform after median filtering the disparity estimates. The homographic transform is established on basis on the Zhang's camera calibration method [19]. An example 3D image is shown in Fig. 7 (left).
3.2 Pattern selection – design considerations
A correct selection of patterns will enable the system to perform well even though the projection system induces numerous artifacts, e.g. laser speckle, slightly unreliable frequency/phase information, and partially specular objects. By selecting the right set of patterns, we can enable the disparity estimation algorithm to reduce the impact of these artifacts.
The objective is thus to select an optimal set S of $N $ patterns $S = \{{{f_i},\; {\theta_i}} \},\; i \in [{1, \ldots ,N} ]$ where pattern i has frequency ${f_i}$ and ${\theta _i}$, and where we assume the projected signal output to be ${P_i}(x )= a(x )\cos ({x{f_i} + {\theta_i}} )+ b(x )$ where $x \in X$ is the horizontal position in the outputted “image” for the horizontal field-of-view X, i is pattern index, and $a(x )$ and $b(x )$ are functions indicating the amplitude and DC level of the outputted signal. The patterns will be emitted sequentially in time, enabling each point in the scene to receive a unique temporal signature.
In terms of optimality, the selected patterns should:
- 1. Provide the highest possible accuracy of measurements.
- 2. Provide the lowest possible probability of confounding two temporal signatures
To perform the frequency and phase selection, we have used the following design principles:
- 1. The highest projection frequency should always be chosen as part of the set S. In previous work [20] we have established that the accuracy of the system is proportional to the the highest frequency ${f_{max}} = \mathop {\max }\limits_\textrm{i} {f_i}$ of the projected patterns.
- 2. For each frequency, we should project all three phases $\left[ {0,\frac{{2\pi }}{3},\frac{{4\pi }}{3}} \right]$. This allows for later robust signal normalization that cancels out differences in projected/observed DC level and signal amplitude (Eq. (4)), as the mean and standard deviation of the three phases combined will be constant. This allows us to reduce the impact of e.g. specular surfaces in the reconstruction algorithm.
- 3. There should be a minimal amount of artifacts interfering with the main lobe in the cross-correlation matrix.
- 4. Beyond these principles, we should select the patterns that provide the maximum robustness (e.g. unambiguity) within the relevant depth working range for the system. Our consideration is that we should select the patterns that provide the minimum maximum peak height outside the main lobe within the relevant disparity range for the system. (Maximum side lobe suppression.).
Design principles 1 and 2 can be met by pre-selecting phases and one of the frequencies. Optimizing for design principles 3 and 4 revolves around the cross-correlation matrix for the patterns themselves. As the patterns projected are not pure sinusoidal patterns, we have chosen to use experimental data to determine the cross-correlation matrix of the patterns. This allows to also include other effects like laser speckle etc directly into the analysis. In practice we do this by capturing an image set of a planar, matte surface for a linear set of frequencies, and studying the cross-correlation matrix and its properties for selected rows in the image.
Figure 8 shows an example of a pattern cross-correlation matrix. We observe a main lobe along the diagonal which reflects the correct correlation, and multiple sidelobes indicating false matches. As the disparity estimate is found by detecting the maximum location in the cross-correlation matrix, the height of the sidelobes relative to the mainlobe provide an indication of the chance of confusing the true answer (mainlobe) with a false detection (sidelobe). Furthermore, the width of the mainlobe provides an indication of the expected precision of the later 3D measurements, as a narrower mainlobe will allow for a more precise subpixel positioning.
Fig. 8. Example cross-correlation matrix (CCM) used for determining optimal set of patterns. Cameras are imaging a planar, white surface. Rows are camera 1, columns are camera 2. We see the main lobe as the center diagonal in the CCM, this is the true match. We observe side lobes outside the main lobe. By reducing the amplitude of these relative to the main lobe we can reduce the probability of false detections.
3.3 Frequency selection strategies
We have considered three different pattern selection strategies, linear frequency sweep, Golomb rulers and an exhaustive search. The issue we are facing is that if we have two different frequencies ${f_1}$ and ${f_2}$, we would expect a beat note (manifesting as a sidelobe) with a frequency ${f_1} - {f_2}$. To reduce the sidelobe amplitude, our overall strategy is to distribute the energy of the sidelobes across the entire disparity range. As basis for the frequency selection problem, we captured images of a planar, matte surface with 30 different frequencies between 35 and 80 fringes in the image, each with 3 phase values separated by approx. $\frac{{2\pi }}{3}$. The frequencies are generated by applying control voltages to the mirror that are linearly increasing. There is a slightly non-linear response between applied voltage and projected frequency, as observed in in Fig. 4 (left). We chose to use all 3 phases, and 5 selected frequencies for 3D reconstruction, i.e. a total of 15 images, in line with our 3D acquisition rate goal in Table 1.
3.3.1 Linear sweep
Selecting pattern frequencies S for the linear frequency sweep means selecting equidistant frequencies ${f_i} = i \cdot \frac{{({{f_{\textrm{max}}} - {f_{\textrm{min}}}} )}}{{N - 1}} + {f_{\textrm{min}}}$ as our pattern set. With our frequency span, the frequency separation is approx. 11 fringes. This results in a pattern cross-correlation matrix as shown in Fig. 9. (top, left), where we see some aggregation of sidelobes which raises their combined amplitude. From a mathematical point of view, a linear frequency sweep with frequency separation $\mathrm{\Delta }f$ should result in a CCM with $\mathrm{\Delta }f$ side lobes to each side of the main lobe (in our case 11 to each side), each equally high as the main lobe. The absence of such features in the CCM confirms that the recorded frequencies are not linearly spaced, i.e. that the angular displacement of the mirror is not linear with applied voltage, as noted in section 2.2.
Fig. 9. Cross-correlation matrices for different frequency selection strategies. Red lines indicate relevant disparity range excluding main lobe. Max sidelobe indicates maximum amplitude (99.9% percentile) of sidelobes within the disparity range. Linear (section 3.3.1), Golomb (section 3.3.2) and Exhaustive (section 3.3.3) use five different frequencies, and fulfill capture time constraints. Full uses 30 different frequencies and is included as a reference showing excellent side lobe suppression, although too slow for many practical use cases.
3.3.2 Golomb rulers
A Golomb ruler is a set of marks at integer positions along a ruler such that no two pairs of marks are the same distance apart. More precisely, an $n$-mark Golomb ruler is a set of n distinct non-negative integers $({{a_1},{a_2}, \ldots ,\; {a_n}} )$ such that the positive difference $|{{a_i} - {a_j}} |$ computed over all possible pairs of different integers $i,j \in [{1,\; \ldots ,\; n} ]$ with $i \ne j$ are distinct [21]. By selecting frequencies according to a Golomb ruler, we should be able to distribute the beat frequencies evenly into the disparity range window. We have used the Golomb ruler [0,1,4,9,11] in our experiments. The result is shown in Fig. 9. (top, right). Visually the CCM looks cleaner in the sidelobe region, but numerical results indicate otherwise – we get results similar to the linear sweep. Again, we believe this is due to that the frequency response is slightly non-linear relative to the voltage applied.
3.3.3 Optimization approach / exhaustive search
In this strategy we used an optimization approach to select the best frequencies. Considering design principle 1 and 2, the highest frequency of the interferometer had to be part of the pattern set, and for each frequency we should use all three phases. This reduces the selection problem to choosing the four best frequencies out of the remaining 29 frequencies. This is a set of $M = \left( {\begin{array}{c} {29}\\ 4 \end{array}} \right) = 23751$ possibilities which makes the problem computationally tractable and enabling us to calculate all pattern cross-correlation matrices ${C_m},\; m \in [{0,M} ]$ selecting the optimal set. As optimization criteria $G({{C_m}} )$ we measured the maximum sidelobe response outside the mainlobe and within the defined disparity range for the pattern cross-correlation matrix. An exhaustive search enabled us to find the optimal $m^{\prime}$ maximizing $G({{C_m}} )$.
The cross-correlation matrix is shown in Fig. 9. We see that the sidelobes in the disparity window are lower, and numerical results confirm the visual indication.
4. Results
4.1 3D camera
The 3D camera was assembled as shown in Fig. 10, with the interferometer shown in Fig. 2 (Left). This interferometer design has a footprint of approx. 100 × 125 mm, and is chosen for simplicity of assembly. A far smaller interferometer with a footprint of 32 × 16 mm is shown in Fig. 2 (Right), demonstrating the potential for miniaturization. Physical characteristics of the system are listed in Table 3. The interferometric projector is mounted in a dust-proof compartment above the two cameras forming the stereo pair. The movable mirror is mounted with an angular offset with respect to the reference mirror, such that we can access between approx. 35 and 80 fringes in our images, by tilting the movable mirror. The two cameras are mounted with a baseline distance of 135 mm, and at an inward angle of 4°. We use two IDS uEye UI-3060CP-M-GL Rev.2 cameras with a 25 mm Tamron M1123FM25 lens nominally set to f/2.8 (also tested at f/5.6 for increased depth of focus). We use a MIDOPT BN785/27 NIR bandpass filter to suppress ambient illumination. The control electronics board is not visible in Fig. 10, but is mounted between the two cameras. The control electronics board provides an analog control voltage sequence with 20 kHz sample rate for the movable mirror, as well as triggers for the two cameras such that the two cameras and the mirror movement are in sync. It also provides a constant drive current for the diode laser. Data from the cameras is returned to a PC using two USB3 cables.
Fig. 10. (Left) Assembled 3D camera. The interferometric projector is mouted on top, while two cameras forming a stereo pair, as well as control electronics are mounted below. (Right) Close-up showing the interferometer. The laser is shown at the bottom, beam splitter in the center, movable mirror on top, reference mirror to the left and expansion lens to the right.
4.2 3D imaging results
We have used toy building bricks as example objects for 3D imaging. The bricks expose well many problematic surfaces for 3d imaging, as they have e.g. specular surfaces, minute details and large variations in color and albedo. At the same time, the recovered shapes are easily recognizable. Figure 11 shows example 3D pointclouds for imaging the toy bricks and a toy brick build of a Mars Rover. Results were obtained using the frequencies determined by exhaustive search.
Fig. 11. Example 3D results. Axis units in mm. Left: Capture of the toy bricks. We can see that the small bumps on the toy bricks are clearly visible. Right: Capture of the toy brick build of a Mars Rover in front of a planar surface. We see that the shape is well recovered.
For quantitative evaluation of 3D measurement performance, we image a planar surface with a known checkerboard pattern, each square being ${\mathrm{\Delta }_C} = 20$ mm. We acquire sequences of $N = 16$ 3D images $XYZ({r,c,d,i} )$ where we for each pixel coordinate $({r,c} )\in [{1,500} ]$ in image $i \in [{0,16} ] $ report the value in mm for each of the dimensions $d \in [{1,3} ]$.
We perform subpixel corner detection in the accompanying grayscale image to determine the corner pixel locations ${p_j}$ that provide the pixel position $({{r_{{p_j}}},{c_{{p_j}}}} )$ for corner index k, and establish the corner locations ${C_j} = XYZ({{r_{{p_j}}},{c_{{p_j}}},d,i} )$ in 3D space by linearly interpolating the 3D images at the corner locations. The underlying metric used for further analysis corner-to-corner distance relative to the reference distance ${\mathrm{\Delta }_C}$, which we calculate as ${\mathrm{\Delta }_{C{C_k}}} = |{|{{C_j}_1 - {C_j}_2} |} |- {\mathrm{\Delta }_C}$ where $({j_1},{j_2}) \in K$ are adjacent corners on the chessboard in the set $k \in K$ of all adjacent ${N_K}$ corners.
We report the following metrics:
- • Average standard deviation in Z in mm per pixel (SZ), e.g. $\overline {{\sigma _z}} = \frac{1}{{{R^2}}}\mathop \sum \limits_{r,c \in [{0,R} )} {\sigma _z}({r,c} )$ where ${\sigma _z}({r,c} )$ is the per-pixel standard deviation for the Z measurements. This provides an indication of the noise in the underlying disparity estimates, capturing effects due to mainlobe width, laser speckle, pattern artifacts and projector repeatability.
- • Mean corner-to-corner distance vs reference distance (MC), e.g. the average of all ${\mathrm{\Delta }_{CC}}_k$. This is provides primarily information on the quality of the stereo calibration of the system, in combination with any systematic deviations in the underlying disparity estimates.
- • Standard deviation in corner-to-corner distance (RC). The standard deviation of all ${\mathrm{\Delta }_{CC}}_k$. This measures partly the calibration, but also the noise in the underlying disparity estimates and the corner detection noise.
It should be noted that we used the same checkerboard for calibrating and verifying the system. This allows us to avoid any issues related to difference in scale when printing the checkerboard patterns.
In Table 4, we present the 3D measurement results at 60 cm distance, using apertures of f/2.8 and f/5.6, the latter being used for increased depth of focus. We observe that when using 15 images for 3D reconstruction, SZ is <0.1 mm with f/2.8, and increases close to linearly to 0.15 mm at f/5.6. This is a result of the increased speckle contrast when using a smaller aperture. MC is close to 0 and indicates that the stereo calibration is successful. The stereo calibration is based on images without speckle, and is thus unaffected by f-number, meaning that the MC metric is unaffected. The RC metric measures a mixture of Z standard deviation, calibration artifacts and corner detection noise, and is somewhat affected by the increase of speckle due to smaller aperture. For completeness, we include the results using 90 images for 3D reconstruction showing SZ < 0.05 mm. In addition, the increased number of frequencies significantly suppresses the sidelobes, as seen in the confusion matrix in Fig. 9 (bottom, right).
Table 3. Summary of 3D camera performance characteristics.
Table 4. 3D measurement performance on planar surface.
4.3 Piezoelectric micromirror – environmental tests
The piezoelectric micromirror is a critical component of the interferometer [13]. Thin film piezoelectric devices are also a family of devices that have not been conclusively tested for environmental stressors, particularly for use cases in space. Therefore, extensive studies of the effect of space-related stressors on the micromirror were conducted to ensure reliable operation. The test parameters are listed in.
The effects of the separate and cumulative tests, 16 electrodes in each set, were compared with a reference sample set, using polarization, capacitance, deflection, impedance, and time-dependent dielectric breakdown measurements to analyze the state of the PZT film. While detailed results and discussions on this work can be found in [22], we summarize our main findings here.
Compared with the pristine devices, we observe no reduction in electromechanical properties or the electrical lifetime due to the stress tests. On the contrary, an improvement in specific properties, particularly energy efficiency, leakage, and lifetime, was generally measured. The improvements after environmental stress originate from various effects of the stressors on the defect chemistry of the Pb(Zr,Ti)O3 (PZT) piezoelectric layer, particularly the redistribution of point defects [23,24]. Leakage vs. time curves, Weibull analysis, and visual inspection revealed soft dielectric breakdown as the primary failure mode, particularly lead and oxygen vacancies, migrating to and piling up at the electrode interfaces [25,26].
As shown in Table 5, our test samples underwent either single environmental stressor or cumulative tests. One sample set was kept for reference.
Table 5. Environmental test parameters
The effects of the separate and cumulative tests, 16 electrodes in each set, were compared with a reference sample set, using polarization, capacitance, deflection, impedance, and time-dependent dielectric breakdown measurements to analyze the state of the PZT film. While detailed results and discussions on this work can be found in [22], we summarize our main findings here.
Compared with the pristine devices, we observe no reduction in electromechanical properties or the electrical lifetime due to the stress tests. On the contrary, an improvement in specific properties, particularly energy efficiency, leakage, and lifetime, was generally measured. The improvements after environmental stress originate from various effects of the stressors on the defect chemistry of the Pb(Zr,Ti)O3 (PZT) piezoelectric layer, particularly the redistribution of point defects [23,24]. Leakage vs. time curves, Weibull analysis, and visual inspection revealed soft dielectric breakdown as the primary failure mode, particularly lead and oxygen vacancies, migrating to and piling up at the electrode interfaces [25,26].
Overall, the qualification tests show that PZT-based thin film micromirrors are well suited for space operation, however, operation at elevated temperature will significantly reduce the mean time to failure (MTF). For each electrode, the estimated MTF at constant operation was found to be 12 years at 70°C and 10 V electrode voltage, and 1200 years at 30°C and 10 V electrode voltage. Typical stressors encountered in space are significantly less aggressive than stressors commonly encountered on earth, e.g., humidity [27,28], demonstrating the space-readiness of PZT-based piezoelectric MEMS devices.
5. Conclusion and discussion
We argue that a single mode laser diode is a suitable light source for a compact, power efficient and simple projector with potential for sunlight filtering, due to a combination of high brightness, high wall plug efficiency and narrow emission spectrum. We have introduced the angularly displaced Michelson interferometer as a configuration for projecting sine patterns with controllable phase and frequency, and shown that our piezoelectric micromirror has suitable characteristics in terms of positioning accuracy, planarity and repositioning speed.
Further, we have adapted our 3D reconstruction algorithms to handle a pattern set which will exhibit slight deviations from target frequency and phase. We demonstrate that our algorithms successfully reconstructs 3D based on images acquired by two cameras in a stereo configuration, and provide robust and high quality 3D data with a measurement precision of 0.09 mm at a 3D image capture rate of 12.5 Hz.
Natural further steps would be to utilize the potential for miniaturization to build a smaller version of the camera, to implement narrow-band sunlight filtering and to reduce the computational load related to the 3D reconstruction and move towards FPGA – based data processing.
Funding
European Space Agency (4000129212/19/NL/TFD).
Acknowledgments
Portions of this work were presented at the OSA Imaging and Applied Optics Congress (3D, COSI, DH, ISA, pcAOP) in 2021, 3Th2D.1.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are available upon reasonable request to the corresponding author.
References
1. Ø. Skotheim, H. Schumann-Olsen, J. Thorstensen, A. N. Kim, M. Lacolle, K. Haugholt, and T. Bakke, “A Real-Time 3D Range Image Sensor based on a novel Tip-Tilt-Piston Micro-mirror and Dual Frequency Phase Shifting,” Proc. SPIE 9393, Three-Dimensional Image Process. Meas. (3DIPM), Appl. 93930A (2015).
2. F. Couweleers, O. Skotheim, and B. W. Tveiten, “Optical 3D shape measurement with structured light,” in 8th International Symposium on Measurement and Quality Control in Production1860, 53–58 (2004).
3. M. Carfagni, R. Furferi, L. Governi, C. Santarelli, M. Servi, F. Uccheddu, and Y. Volpe, “Metrological and critical characterization of the intel D415 stereo depth camera,” Sensors 19(3), 489 (2019). [CrossRef]
4. Y. Wang and S. Zhang, “Superfast multifrequency phase-shifting technique with optimal pulse width modulation,” Opt. Express 19(6), 5149 (2011). [CrossRef]
5. G. Sansoni and A. Patrioli, “Noncontact 3D sensing of free-form complex surfaces,” in Proceedings SPIE, Videometrics and Optical Methods for 3D Shape Measurement (2000).
6. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, “A state of the art in structured light patterns for surface profilometry,” Pattern Recognit. 43(8), 2666–2680 (2010). [CrossRef]
7. J. Thorstensen and J. Thielemann, “Adaptive multi-frequency phase stepping for optimal 3D depth reconstruction,” in Imaging and Applied Optics 2018 (3D, AO, AIO, COSI, DH, IS, LACSEA, LS&C, MATH, PcAOP), OSA Technical Digest (2018), p. JTu4A.15.
8. P. Mirdehghan, W. Chen, and K. N. Kutulakos, “Optimal Structured Light a la Carte,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018).
9. C. E. Towers, D. P. Towers, and J. D. C. Jones, “Generalized frequency selection in multifrequency interferometry,” Opt. Lett. 29(12), 1348 (2004). [CrossRef]
10. F. B. Djupkep Dizeu, J. Boisvert, M. A. Drouin, G. Godin, M. Rivard, and G. Lamouche, “Frequency Shift Triangulation: A Robust Fringe Projection Technique for 3D Shape Acquisition in the Presence of Strong Interreflections,” in Proceedings - 2019 International Conference on 3D Vision, 3DV 2019 (2019).
11. C. S. Park, K. W. Park, U. Jung, J. Kim, S. W. Kang, and H. R. Kim, “Dynamic fringe pattern generation using an electrically tunable liquid crystal fabry-perot cell for a miniaturized optical 3-D surface scanning profilometer,” Mol. Cryst. Liq. Cryst. 526(1), 28–37 (2010). [CrossRef]
12. F. Wu, H. Zhang, M. J. Lalor, and D. R. Burton, “Novel design for fiber optic interferometric fringe projection phase-shifting 3-D profilometry,” Opt. Commun. 187(4-6), 347–357 (2001). [CrossRef]
13. T. Bakke, A. Vogl, O. Żero, F. Tyholdt, I.-R. Johansen, and D. Wang, “A novel ultra-planar, long-stroke and low-voltage piezoelectric micromirror,” J. Micromech. Microeng. 20(6), 064010 (2010). [CrossRef]
14. T. Bakke and I.-R. Johansen, “A robust, non-resonant piezoelectric micromirror,” in 16th International Conference on Optical MEMS and Nanophotonics (IEEE, 2011), pp. 171–172.
15. R. A. Conant, J. T. Nee, K. Y. Lau, and R. S. Muller, “Dynamic deformation of scanning mirrors,” in 2000 IEEE/LEOS International Conference on Optical MEMS (Cat. No.00EX399) (IEEE, n.d.), pp. 49–50.
16. E. M. Bourim, H.-Y. Kim, J.-S. Yang, J.-W. Yang, K.-S. Woo, J.-H. Song, and S.-K. Yun, “Creep behavior of undoped and La–Nb codoped PZT based micro-piezoactuators for micro-optical modulator applications,” Sens. Actuators, A 155(2), 290–298 (2009). [CrossRef]
17. Y. Liu, N. Pears, P. L. Rosin, and P. Huber, 3D Imaging, Analysis and Applications (Springer, 2020).
18. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]
19. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). [CrossRef]
20. G. Bouquet, J. Thorstensen, K. A. H. Bakke, and P. Risholm, “Design tool for TOF and SL based 3D cameras,” Opt. Express 25(22), 27758–27769 (2017). [CrossRef]
21. R. C. Alperin and V. Drobot, “Golomb Rulers,” Math. Mag. 84(1), 48–55 (2011). [CrossRef]
22. R. Dahl-Hansen and J. Gjessing, is preparing a manuscript to be called “Thin film PZT-based micormirros for space-applications.”
23. B. Akkopru-Akgun, W. Zhu, M. T. Lanagan, and S. Trolier-McKinstry, “The effect of imprint on remanent piezoelectric properties and ferroelectric aging of PbZr0.52Ti0.48O3 thin films,” J. Am. Ceram. Soc. 102(9), 5328–5341 (2019). [CrossRef]
24. D. Damjanovic, “Ferroelectric, dielectric and piezoelectric properties of ferroelectric thin films and ceramics,” Rep. Prog. Phys. 61(9), 1267–1324 (1998). [CrossRef]
25. S. W. Ko, W. Zhu, C. Fragkiadakis, T. Borman, K. Wang, P. Mardilovich, and S. Trolier-McKinstry, “Improvement of reliability and dielectric breakdown strength of Nb-doped lead zirconate titanate films via microstructure control of seed,” J. Am. Ceram. Soc. 102(3), 1211–1217 (2019). [CrossRef]
26. B. Akkopru-Akgun, T. J. M. Bayer, K. Tsuji, C. A. Randall, M. T. Lanagan, and S. Trolier-McKinstry, “The influence of Mn doping on the leakage current mechanisms and resistance degradation behavior in lead zirconate titanate films,” Acta Mater. 208, 116680 (2021). [CrossRef]
27. R. P. Dahl-Hansen, J. M. Polfus, E. Vøllestad, B. Akkopru-Akgun, L. Denis, K. Coleman, F. Tyholdt, S. Trolier-Mckinstry, and T. Tybell, “Electrochemically driven degradation of chemical solution deposited ferroelectric thin-films in humid ambient,” J. Appl. Phys. 127(24), 244101 (2020). [CrossRef]
28. R. P. Dahl-Hansen, F. Tyholdt, J. Gjessing, A. Vogl, P. Wittendorp, J. Vedum, and T. Tybell, “On the Effect of Water-Induced Degradation of Thin-Film Piezoelectric Microelectromechanical Systems,” J. Microelectromech. Syst. 30(1), 105–115 (2021). [CrossRef]
