Neural Subspaces for Light Fields
Pages 1685 - 1695
Abstract
We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.
References
[1]
B. Y. Feng and A. Varshney, “SIGNET: Efficient neural representation for light fields,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2021, pp. 14204–14213.
[2]
V. Sitzmann, J. N. P. Martel, A. Bergman, D. B. Lindell, and G. Wetzstein, “Implicit neural representations with periodic activation functions,” in Proc. Adv. Neural Inf. Process. Syst., 2020, pp. 7462–7473.
[3]
M. Tancik et al., “Fourier features let networks learn high frequency functions in low dimensional domains,” in Proc. Adv. Neural Inf. Process. Syst., 2020, pp. 7537–7547.
[4]
B. Mildenhall, P.P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing scenes as neural radiance fields for view synthesis,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 405–421.
[5]
M. Magnor and B. Girod, “Data compression for light-field rendering,” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 3, pp. 338–343, Apr. 2000.
[6]
C. Chang, X. Zhu, P. Ramanathan, and B. Girod, “Light field compression using disparity-compensated lifting and shape adaptation,” IEEE Trans. Image Process., vol. 15, no. 4, pp. 793–806, Apr. 2006.
[7]
A. Jagmohan, A. Sehgal, and N. Ahuja, “Compression of lightfield rendered images using coset codes,” in Proc. 37th Asilomar Conf. Signals, Syst. Comput.2003, pp. 830–834.
[8]
X. Zhu, A. Aaron, and B. Girod, “Distributed compression for large camera arrays,” in Proc. IEEE Workshop Statist. Signal Process., 2003, pp. 30–33.
[9]
M. Broxton et al., “Immersive light field video with a layered mesh representation,” ACM Trans. Graph., vol. 39, pp. 86:1–86:15, 2020.
[10]
X. Jiang, M. L. Pendu, R. A. Farrugia, and C. Guillemot, “Light field compression with homography-based low-rank approximation,” IEEE J. Sel. Topics Signal Process., vol. 11, no. 7, pp. 1132–1145, Oct. 2017.
[11]
S. Pratapa and D. Manocha, “HMLFC: Hierarchical motion-compensated light field compression for interactive rendering,” Comput. Graph. Forum, vol. 38, no. 8, pp. 1–12, Nov. 2019.
[12]
G. Sullivan, J. Ohm, W. Han, and T. Wiegand, “Overview of the high efficiency video coding (HEVC) standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 12, pp. 1649–1668, Dec. 2012.
[13]
D. Mukherjee et al., “The latest open-source video codec VP9 - an overview and preliminary results,” in Proc. Picture Coding Symp., 2013, pp. 390–393.
[14]
M. L. Pendu, C. Guillemot, and A. Smolic, “A fourier disparity layer representation for light fields,” IEEE Trans. Image Process., vol. 28, no. 11, pp. 5740–5753, Nov. 2019.
[15]
M. L. Pendu and A. Smolic, “High resolution light field recovery with fourier disparity layer completion, demosaicing, and super-resolution,” in Proc. IEEE Int. Conf. Comput. Photogr., 2020, pp. 1–12.
[16]
E. Dib, M. L. Pendu, and C. Guillemot, “Light field compression using fourier disparity layers,” in Proc. IEEE Int. Conf. Image Process., 2019, pp. 3751–3755.
[17]
M. L. Pendu, C. Ozcinar, and A. Smolic, “Hierarchical fourier disparity layer transmission for light field streaming,” in Proc. IEEE Int. Conf. Image Process., 2020, pp. 2606–2610.
[18]
J. Mairal, F. R. Bach, J. Ponce, and G. Sapiro, “Online dictionary learning for sparse coding,” in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 689–696.
[19]
E. Miandji, J. Kronander, and J. Unger, “Learning-based compression of surface light fields for real-time rendering of global illumination scenes,” in Proc. SIGGRAPH Asia Tech. Briefs, 2013, pp. 1–4.
[20]
J. Sulam, V. Papyan, Y. Romano, and M. Elad, “Multilayer convolutional sparse modeling: Pursuit and dictionary learning,” IEEE Trans. Signal Process., vol. 66, no. 15, pp. 4090–4104, Aug. 2018.
[21]
A. Abdi, A. Payani, and F. Fekri, “Learning dictionary for efficient signal compression,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2017, pp. 3689–3693.
[22]
M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311–4322, Nov. 2006.
[23]
E. Miandji, S. Hajisharif, and J. Unger, “A unified framework for compression and compressed sensing of light fields and light field videos,” ACM Trans. Graph., vol. 38, pp. 1–18, 2019.
[24]
S. Hajisharif, E. Miandji, P. Larsson, K. Tran, and J. Unger, “Light field video compression and real time rendering,” Comput. Graph. Forum, vol. 38, pp. 265–276, 2019.
[25]
S. Pratapa and D. Manocha, “RLFC: Random access light field compression using key views and bounded integer sequence encoding,” in Proc. ACM SIGGRAPH Symp. Interactive 3D Graph. Games, 2019, pp. 1–10.
[26]
J. Nystad, A. Lassen, A. Pomianowski, S. Ellis, and T. Olson, “Adaptive scalable texture compression,” in Proc. 4th ACM SIGGRAPH / Eurographics Conf. High-Perform. Graph., 2012, pp. 105–114.
[27]
N. K. Kalantari, T. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Trans. Graph., vol. 35, pp. 1–10, 2016.
[28]
M. Bemana, K. Myszkowski, H.-P. Seidel, and T. Ritschel, “X-fields: Implicit neural view-, light-and time-image interpolation,” ACM Trans. Graph., vol. 39, no. 6, pp. 1–15, 2020.
[29]
V. Sitzmann, S. Rezchikov, W. Freeman, J. Tenenbaum, and F. Durand, “Light field networks: Neural scene representations with single-evaluation rendering,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2021, pp. 19313–19325.
[30]
B. Y. Feng, Y. Zhang, D. Tang, R. Du, and A. Varshney, “PRIF: Primary ray-based implicit function,” in Proc. Eur. Conf. Comput. Vis., 2022, pp. 138–155.
[31]
B. Y. Feng, S. Jabbireddy, and A. Varshney, “VIINTER: View interpolation with implicit neural representations of images,” in Proc. ACM SIGGRAPH Asia Conf. Proc., 2022, pp. 1–9.
[32]
Y. Zhang, T. van Rozendaal, J. Brehmer, M. Nagel, and T. Cohen, “Implicit neural video compression,” 2021,.
[33]
G. W. Stewart, “An updating algorithm for subspace tracking,” IEEE Trans. Signal Process., vol. 40, no. 6, pp. 1535–1541, Jun. 1992.
[34]
F. D. L. Torre and M. J. Black, “A framework for robust subspace learning,” Int. J. Comput. Vis., vol. 54, pp. 117–142, 2004.
[35]
B. Wang and Z. Tu, “Sparse subspace denoising for image manifolds,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 468–475.
[36]
N. Vaswani, T. Bouwmans, S. Javed, and P. Narayanamurthy, “Robust subspace learning: Robust PCA, robust subspace tracking, and robust subspace recovery,” IEEE Signal Process. Mag., vol. 35, no. 4, pp. 32–55, Jul. 2018.
[37]
A. Adams, “The stanford light field archive,” 2008. [Online]. Available: http://lightfield.stanford.edu/lfs.html
[38]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.
[39]
Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 2472–2481.
[40]
J. Flynn et al., “DeepView: View synthesis with learned gradient descent,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 2362–2371.
[41]
G. Wu, M. Zhao, L. Wang, Q. Dai, T. Chai, and Y. Liu, “Light field reconstruction using deep convolutional network on EPI,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1638–1646.
[42]
S. Winkler, E. D. Gelasca, and T. Ebrahimi, “Toward perceptual metrics for video watermark evaluation,” in Proc. Appl. Digit. Image Process. XXVI, 2003, pp. 371–378.
[43]
D. Li, R. Du, A. Babu, C. D. Brumar, and A. Varshney, “A log-rectilinear transformation for foveated 360-degree video streaming,” IEEE Trans. Vis. Comput. Graph., vol. 27, no. 5, pp. 2638–2647, May 2021.
[44]
M. Tancik et al., “Learned initializations for optimizing coordinate-based neural representations,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 2846–2855.
[45]
A. Bergman, P. Kellnhofer, and G. Wetzstein, “Fast training of neural lumigraph representations using meta learning,” in Proc. Adv. Neural Inf. Process. Syst., 2021, pp. 172–186.
[46]
X. Meng, R. Du, J. F. JaJa, and A. Varshney, “3D-Kernel foveated rendering for light fields,” IEEE Trans. Vis. Comput. Graph., vol. 27, no. 8, pp. 3350–3360, Aug. 2021.
[47]
C. Koniaris, M. Kosek, D. Sinclair, and K. Mitchell, “Compressed animated light fields with real-time view-dependent reconstruction,” IEEE Trans. Vis. Comput. Graph., vol. 25, no. 4, pp. 1666–1680, Apr. 2018.
Index Terms
- Neural Subspaces for Light Fields
Index terms have been assigned to the content through auto-classification.
Recommendations
Opacity light fields: interactive rendering of surface light fields with view-dependent opacity
I3D '03: Proceedings of the 2003 symposium on Interactive 3D graphicsWe present new hardware-accelerated techniques for rendering surface light fields with opacity hulls that allow for interactive visualization of objects that have complex reflectance properties and elaborate geometrical details. The opacity hull is a ...
Sentiment Analysis in the Light of LSTM Recurrent Neural Networks
Long short-term memory LSTM is a special type of recurrent neural network RNN architecture that was designed over simple RNNs for modeling temporal sequences and their long-range dependencies more accurately. In this article, the authors work with ...
Learning at the Speed of Light: A New Type of Optical Neural Network
OSC '08: Proceedings of the 1st international workshop on Optical SuperComputingMost, if not all, optical hardware-based neural networks are slow during the neural learning phase. This limitation has been not only a speed bottleneck, but it has contributed to the lack of wide-spread use of optical neural systems. We present a novel ...
Comments
Information & Contributors
Information
Published In
© 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 December 2022
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 06 Oct 2024
Other Metrics
Citations
Cited By
View allView Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in