On Plenoptic Multiplexing and Reconstruction

Abstract

Photography has been striving to capture an ever increasing amount of visual information in a single image. Digital sensors, however, are limited to recording a small subset of the desired information at each pixel. A common approach to overcoming the limitations of sensing hardware is the optical multiplexing of high-dimensional data into a photograph. While this is a well-studied topic for imaging with color filter arrays, we develop a mathematical framework that generalizes multiplexed imaging to all dimensions of the plenoptic function. This framework unifies a wide variety of existing approaches to analyze and reconstruct multiplexed data in either the spatial or the frequency domain. We demonstrate many practical applications of our framework including high-quality light field reconstruction, the first comparative noise analysis of light field attenuation masks, and an analysis of aliasing in multiplexing applications.

Appendices

Appendix A: Proof of PSM Theorem

Throughout this paper, we assume that the spatial basis functions $\varvec{\sigma }_j ( \vec {x})$ are super-pixel-periodic², i.e., $\varvec{\sigma }_j ( \vec {x}) = \varvec{\sigma }_j ( \vec {x}+ t \Delta \vec {x}_s), \, \forall t \in \mathbb{Z }.$ The offset between successive super-pixels is denoted as $\Delta \vec {x}_s.$ This notation allows us to define a sampling operator as

$$\begin{aligned} \mathrm{I\!I\!I}_k(\vec {x}) = \sum \limits _{t \in \mathbb{Z }} \delta \left( \vec {x}+ t \Delta \vec {x}_s+ \Delta \vec {x}_k \right), \end{aligned}$$

(23)

where $\Delta \vec {x}_k$ is the offset of individual samples within each super-pixel. This sampling operator basically extracts a sub-image or channel $k=1 \ldots M$ from an interleaved sensor image, where only corresponding sub-pixels within the super-pixels are included in each channel. Sampling such a channel $\widetilde{\varvec{i}}_k (\vec {x})$ from a sensor image $\varvec{i}(\vec {x}),$ in combination with Eq. 4, results in the following expression

$$\begin{aligned} \widetilde{\varvec{i}}_k (\vec {x})&= \mathrm{I\!I\!I}_k(\vec {x}) \varvec{i}(\vec {x}) = \mathrm{I\!I\!I}_k(\vec {x}) \left( \sum \limits _{j=1}^{N} \varvec{\sigma }_j (\vec {x}) \varvec{\rho }_j \left( \vec {x}\right)\right) \nonumber \\&= \sum \limits _{j=1}^{N} \varvec{\sigma }_j^k \mathrm{I\!I\!I}_k(\vec {x}) \varvec{\rho }_j \left( \vec {x}\right). \end{aligned}$$

(24)

As discussed in Sect. 3.3, the spatial basis functions $\varvec{\sigma }_j (\vec {x})$ become spatially-invariant constants $\varvec{\sigma }_j^k$ in the sampled channels because of spatial periodicity of the basis. Each channel $\tilde{\varvec{i}}_k (\vec {x})$ is thus associated with $N$ constants $\varvec{\sigma }_j^k$ defined by the spatial basis. Reconstructing a channel $\varvec{i}_k (\vec {x})$ from its sampled representation $\tilde{\varvec{i}}_k (\vec {x})$ is performed by convolving with a reconstruction filter kernel $\varvec{f}(\vec {x}):$

$$\begin{aligned} \varvec{i}_k (\vec {x}) = \widetilde{\varvec{i}}_k (\vec {x}) \otimes \varvec{f}(\vec {x}). \end{aligned}$$

(25)

According to the sampling theorem, the original signal must be spatially band-limited for this reconstruction to be a faithful representation.

Due to the spatial basis being a set of constants at every position in the interpolated channels $\varvec{i}_k (\vec {x}),$ and the plenoptic basis being spatially-invariant, a convolution with the filter kernel can be formulated as

$$\begin{aligned} \varvec{i}_k (\vec {x})&= \left( \sum \limits _{j=1}^{N} \varvec{\sigma }_j^k \mathrm{I\!I\!I}_k(\vec {x}) \varvec{\rho }_j \left( \vec {x}\right)\right) \otimes \varvec{f}(\vec {x})\\&= \sum \limits _{j=1}^{N} \varvec{\sigma }_j^k \int _\fancyscript{P}\varvec{\pi }_j (\vec {p}) \bigg ( \mathrm{I\!I\!I}_k(\vec {x}) \varvec{l}_\lambda (\vec {x},\vec {p})\otimes \varvec{f}(\vec {x})\bigg ) d\vec {p}\nonumber \end{aligned}$$

(26)

Equation 26 shows that all channels $\varvec{i}_k (\vec {x})$ are locally related to the sampled and reconstructed plenoptic function via a linear combination. It also shows that applying a linear filter to the measured channels, i.e., before decorrelation, is equivalent to applying the same filter to the plenoptic function itself, i.e., after decorrelation. $\square$

Appendix B: Proof of PFM Theorem

The proof of Theorem 2 follows Equations 6-8. To provide additional detail, we start with Eq. 8

$$\begin{aligned} \mathcal F _{x} \left\{ \varvec{i}(\vec {x}) \right\} \! = \! \sum \limits _{k=1}^M \left( \! \delta \left( {\vec {\omega }_x- \Delta \vec {\omega }_x^k} \right) \! \otimes \! \sum \limits _{j=1}^N \hat{\varvec{\sigma }}_j^{\,k}\hat{\varvec{\rho }}_j\left(\vec {\omega }_x\right) \! \right), \end{aligned}$$

(27)

where $k \Delta \vec {\omega }_x$ has been replaced by $\Delta \vec {\omega }_x^k,$ a vector to the center frequency of a Fourier tile (Fig. 13, right), to allow for generalized sampling patterns. If the signal is properly band-limited the image’s Fourier transform separates into disjoint sets, each encoding one Fourier tile $\hat{\varvec{i}}_k (\vec {\omega }^{\prime }_x)$ (Fig. 13, right). To separate the notation for spatial frequencies in the sensor image and for Fourier tiles we use the substitution $\vec {\omega }^{\prime }_x= \vec {\omega }_x- \Delta \vec {\omega }_x^k$.

Fig. 13

Information overlaps in the Fourier domain if the signal is not suitably band-limited (left). With the appropriate band-limitation, the Fourier representation decomposes into distinct correlated Fourier tiles $\hat{\varvec{i}}_k.$ The arrowin the right figure indicates the position $\Delta \vec {\omega }_x^k$ of the Dirac peaks, i.e., the center frequencies, of a Fourier tile

The Fourier tiles can be cropped from the Fourier transformed sensor image by applying a rect filter:³

$$\begin{aligned} \hat{\varvec{i}}_k (\vec {\omega }^{\prime }_x)&= \mathrm rect ^k(\vec {\omega }^{\prime }_x) \delta \left( {\vec {\omega }^{\prime }_x} \right) \otimes \sum \limits _{j=1}^N \hat{\varvec{\sigma }}_j^{\,k}\hat{\varvec{\rho }}_j\left(\vec {\omega }^{\prime }_x\right) \\&= \mathrm rect ^k(\vec {\omega }^{\prime }_x) \sum \limits _{j=1}^N \hat{\varvec{\sigma }}_j^{\,k}\int _\fancyscript{P}\varvec{\pi }(\vec {p}) \hat{\varvec{l}}_\lambda (\vec {\omega }^{\prime }_x, \vec {p}) d\vec {p}.\nonumber \end{aligned}$$

(28)

The convolution with a Dirac train in Eq. 8 reduces to a convolution with a single Dirac peak $\delta \left( {\vec {\omega }^{\prime }_x} \right)$ because of band-limitation. This is a unit operation and thus removed from the equations. $\square$

In addition, by inverse Fourier transforming and sampling the Fourier tile $\hat{\varvec{i}}_k (\vec {\omega }^{\prime }_x)$ we see that the reconstruction filter is indeed a sinc:

$$\begin{aligned} \varvec{i}_k(\vec {x}) \! = \! \sum \limits _{j=1}^{N} \varvec{\sigma }_j^k \int _\fancyscript{P}\varvec{\pi }_j (\vec {p}) \bigg ( \mathrm{I\!I\!I}_k(\vec {x}) \varvec{l}_\lambda (\vec {x},\vec {p})\otimes \mathrm sinc (\vec {x}) \!\bigg ) d\vec {p}\nonumber \\ \end{aligned}$$

(29)