SpectralSplatsViewer: An Interactive Web-Based Tool for Visualizing Cross-Spectral Gaussian Splats
Article No.: 22, Pages 1 - 10
Abstract
Spectral rendering accurately simulates light-material interactions by considering the entire light spectrum, unlike traditional rendering methods that use limited color channels like RGB. This technique is particularly valuable in industries to assess visual quality before production. Moreover, Spectral imaging finds extensive applications in fields like agriculture for plant disease detection, cultural heritage for preservation, forensic science, environment monitoring and medical science among others. Advances in generating novel views from images have been achieved through methods like NERF and Gaussian splatting, which outperforms others in terms of quality. This paper introduces a web-based viewer built on the Viser framework for visualizing and comparing cross-spectral Gaussian splats from different views and during various training stages. This viewer supports real-time collaboration and comprehensive visual comparison, enhancing user experience in spectral data analysis. We conduct a user study and performance analysis to confirm its effectiveness and usability for different application scenarios, while also proposing potential enhancements for increased functionality.
References
[1]
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild. arXiv preprint arXiv:1906.02569 (2019).
[2]
Matthias Alfeld, Maud Mulliez, Jonathan Devogelaere, Laurence de Viguerie, Philippe Jockey, and Philippe Walter. 2018. MA-XRF and hyperspectral reflectance imaging for visualizing traces of antique polychromy on the Frieze of the Siphnian Treasury. Microchemical Journal 141 (2018), 395–403. https://doi.org/10.1016/j.microc.2018.05.050
[3]
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. 2021. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. arxiv:2103.13415 [cs.CV]
[4]
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5460–5469.
[5]
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2023. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 19697–19705.
[6]
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022a. TensoRF: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision. 333–350.
[7]
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. 2022b. Hallucinated neural radiance fields in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12943–12952.
[8]
Ather Abbas Sanghyun Park Gibeom Nam Jae-Hyun Yoo Kyunghyun Kim Hong Tae Kim JongCheol Pyo Do Hyuck Kwon, Seok Min Hong and Kyung Hwa Cho. 2023. Deep learning-based super-resolution for harmful algal bloom monitoring of inland water. GIScience & Remote Sensing 60, 1 (2023), 2249753. https://doi.org/10.1080/15481603.2023.2249753
[9]
Antonin Dufay, David Murray, Romain Pacanowski, 2019. The Malia rendering framework. https://pacanows.gitlabpages.inria.fr/MRF.
[10]
Luca Fascione, Johannes Hanika, Marco Leone, Marc Droske, Jan Schwarzhaupt, Tomislav Davidovic, Alexander Weidlich, and Johannes Meng. 2018. Manuka: A batch-shading architecture for spectral path tracing in movie production. ACM Transactions on Graphics 37, 3 (2018), 31:1–31:18. https://doi.org/10.1145/3182161
[11]
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5491–5500.
[12]
Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. 2023. Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection. arXiv preprint arXiv:2303.10840 (2023).
[13]
Federico Grillini, Lavinia de Ferri, George Alexis Pantos, Sony George, and Margunn Veseth. 2024. Reflectance imaging spectroscopy for the study of archaeological pre-Columbian textiles. Microchemical Journal 200 (2024), 110168. https://doi.org/10.1016/j.microc.2024.110168
[14]
Wenzel Jakob. 2010. Mitsuba 2: Physically based renderer. http://www.mitsuba-renderer.org.
[15]
Alexander T Jebb, Vivian Ng, and Louis Tay. 2021. A Review of Key Likert Scale Development Advances: 1995-2019. Frontiers in Psychology 12 (2021), 637547. https://doi.org/10.3389/fpsyg.2021.637547
[16]
JENOPTIK. Accessed: 31-05-2024. EVIDIR alpha thermal imaging camera and infrared modules – one size for all variants. https://www.jenoptik.com/products/cameras-and-imaging-modules/thermographic-camera/thermal-imaging-camera.
[17]
Yoonwoo Jeong, Seungjoo Shin, and Kibaek Park. 2022. NeRF-Factory: An awesome PyTorch NeRF collection. https://github.com/kakaobrain/NeRF-Factory/.
[18]
Morris A. Jette and Tim Wickberg. 2023. Architecture of Slurm Workload Manager. In Job Scheduling Strategies for Parallel Processing: 26th Workshop, JSSPP 2023, St. Petersburg, FL, USA, May 19, 2023, Revised Selected Papers (St. Petersburg, FL, USA). Springer-Verlag, Berlin, Heidelberg, 3–23. https://doi.org/10.1007/978-3-031-43943-8_1
[19]
Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. 2023. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. arXiv preprint arXiv:2311.17977 (2023).
[20]
Cerys Jones, Christina Duffy, Adam Gibson, and Melissa Terras. 2020. Understanding multispectral imaging of cultural heritage: Determining best practice in MSI analysis of historical artefacts. Journal of Cultural Heritage 45 (2020), 339–350. https://doi.org/10.1016/j.culher.2020.03.004
[21]
Kim Jun-Seong, Kim Yu-Ji, Moon Ye-Bin, and Tae-Hyun Oh. 2022. HDR-Plenoxels: Self-calibrating high dynamic range radiance fields. In Proceedings of the European Conference on Computer Vision. 384–401.
[22]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (July 2023). https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
[23]
Marco Landi and Giuseppe Maino. 2011. Multispectral Imaging and Digital Restoration for Paintings Documentation. In Image Analysis and Processing – ICIAP 2011, Giuseppe Maino and Gian Luca Foresti (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 464–474.
[24]
L. Lanteri and C. Pelosi. 2021. 2D and 3D ultraviolet fluorescence applications on cultural heritage paintings and objects through a low-cost approach for diagnostics and documentation. In Optics for Arts, Architecture, and Archaeology VIII, Haida Liang and Roger Groves (Eds.). Vol. 11784. International Society for Optics and Photonics, SPIE, 1178417. https://doi.org/10.1117/12.2593691
[25]
Boyi Li, Junting Xie, Yifu Zhang, Lingjie Zhang, Weiming Wang, and Hongdong Li. 2022. SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Fields. ACM Transactions on Graphics (TOG) 41, 4, Article 109 (2022).
[26]
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. 2023. NerfAcc: Efficient Sampling Accelerates NeRFs.arXiv preprint arXiv:2305.04966 (2023).
[27]
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7206–7215.
[28]
Naoki Matsumoto and Akihiro Suda. 2024. bypass4netns: Accelerating TCP/IP Communications in Rootless Containers. arxiv:2402.00365 [cs.NI]
[29]
Micasense. Accessed: 2024-04-24. Micasense RedEdge-MX DUAL. https://drones.measurusa.com/products/micasense-rededge-mx-dual.
[30]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
[31]
Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ricardo Martin-Brualla, and Jonathan T. Barron. 2022. MultiNeRF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF. https://github.com/google-research/multinerf
[32]
Peyman Moghadam, Daniel Ward, Ethan Goan, Srimal Jayawardena, Pavan Sikka, and Emili Hernandez. 2017. Plant Disease Detection Using Hyperspectral Imaging. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA). 1–8. https://doi.org/10.1109/DICTA.2017.8227476
[33]
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41, 4 (2022), 102:1–102:15.
[34]
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. 2022. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8270–8280.
[35]
David Murray, Alban Fichet, and Romain Pacanowski. 2021. Efficient Spectral Rendering on the GPU for Predictive Rendering. Springer, 673–698. https://doi.org/10.1007/978-1-4842-7185-8_42
[36]
Nerfstudio Project. 2023. Viser: A library for interactive 3D visualization in Python. https://github.com/nerfstudio-project/viser.
[37]
Marion Nimier-David, Daniele Vicini, Thomas Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia) 38, 6 (December 2019), 203:1–203:17. https://doi.org/10.1145/3355089.3356498
[38]
Matt Pharr. 2020. PBRT version 4. https://github.com/mmp/pbrt-v4.
[39]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically Based Rendering: From Theory to Implementation (3rd ed.). Morgan Kaufmann.
[40]
Matteo Poggi, Pierluigi Zama Ramirez, Fabio Tosi, Samuele Salti, Luigi Di Stefano, and Stefano Mattoccia. 2022. Cross-Spectral Neural Radiance Fields. In Proceedings of the International Conference on 3D Vision. 3DV.
[41]
Rizwan Qureshi, Muhammad Uzair, Khurram Khurshid, and Hong Yan. 2019. Hyperspectral document image processing: Applications, challenges and future prospects. Pattern Recognition 90 (2019), 12–22. https://doi.org/10.1016/j.patcog.2019.01.026
[42]
Silios. Accessed: 2024-04-24. Off-the-shelf Snapshot Multispectral Cameras. https://www.silios.com/multispectral-imaging.
[43]
Towaki Takikawa, Or Perel, Clement Fuji Tsang, Charles Loop, Joey Litalien, Jonathan Tremblay, Sanja Fidler, and Maria Shugrina. 2022. Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research. https://github.com/NVIDIAGameWorks/kaolin-wisp.
[44]
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. 2023. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings(SIGGRAPH ’23).
[45]
The ART development team. 2018. The Advanced Rendering Toolkit. https://cgg.mff.cuni.cz/ART.
[46]
N.T. Vetrekar, R. Raghavendra, and R.S. Gad. 2016. Low-cost multi-spectral face imaging for robust face recognition. In 2016 IEEE International Conference on Imaging Systems and Techniques (IST). 324–329. https://doi.org/10.1109/IST.2016.7738245
[47]
Chen Wang, Xian Wu, Yuan-Chen Guo, Song-Hai Zhang, Yu-Wing Tai, and Shi-Min Hu. 2022. NeRF-SR: High quality neural radiance fields using supersampling. In Proceedings of the 30th ACM International Conference on Multimedia. 6445–6454.
[48]
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Adv. Neural Inf. Proc. Syst. 354 (2021), 27171–27183.
[49]
Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. 2023. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3272–3283.
[50]
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. 2022. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5438–5448.
[51]
I.-Chang Yang, Kuang-Wen Hsieh, Chao-Yin Tsai, Yu-I. Huang, Yu-Liang Chen, and Suming Chen. 2014. Development of an automation system for greenhouse seedling production management using radio-frequency-identification and local remote sensing techniques. Engineering in Agriculture, Environment and Food 7, 1 (2014), 52–58. https://doi.org/10.1016/j.eaef.2013.12.009
[52]
Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. 2023. BakedSDF: Meshing neural SDFs for real-time view synthesis. In Proceedings of the ACM SIGGRAPH 2023 Conference, Erik Brunvand, Alla Sheffer, and Michael Wimmer (Eds.). 46:1–46:9.
[53]
Anam Zahra, Rizwan Qureshi, Muhammad Sajjad, Ferhat Sadak, Mehmood Nawaz, Haris Ahmad Khan, and Muhammad Uzair. 2024. Current advances in imaging spectroscopy and its state-of-the-art applications. Expert Systems with Applications 238 (2024), 122172. https://doi.org/10.1016/j.eswa.2023.122172
[54]
Yi Zhang, Xiaoyang Huang, Bingbing Ni, Teng Li, and Wenjun Zhang. 2023. Frequency-Modulated Point Cloud Rendering with Easy Editing. arXiv preprint arXiv:2303.07596 (2023).
Index Terms
- SpectralSplatsViewer: An Interactive Web-Based Tool for Visualizing Cross-Spectral Gaussian Splats
Recommendations
A novel progressive refinement algorithm for full spectral rendering
This paper presents a new progressive refinement algorithm for full spectral rendering. This algorithm adopts wavelet transformation to efficient represent full spectral data. To our knowledge, this is the first approach to employing such a ...
3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting
Computer Vision – ECCV 2024AbstractThe use of 3D Gaussians as representation of radiance fields has enabled high quality novel view synthesis at real-time rendering speed. However, the choice of optimising the outgoing radiance of each Gaussian independently as spherical harmonics ...
Performance Improvement of Physically Based Spectral Rendering Using Stochastic Sampling
CCIW 2013: Proceedings of the 4th International Workshop on Computational Color Imaging - Volume 7786In recent years, many researchers in computer graphics pay attention to spectral rendering where light transport is modeled with multiple wavelengths instead of just using the red, green and blue components. This paper discusses some of the advantages ...
Comments
Information & Contributors
Information
Published In
September 2024
220 pages
ISBN:9798400706899
DOI:10.1145/3665318
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- EU Commission
Conference
WEB3D '24
Sponsor:
WEB3D '24: The 29th International ACM Conference on 3D Web Technology
September 25 - 27, 2024
Guimarães, Portugal
Acceptance Rates
Overall Acceptance Rate 27 of 71 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 185Total Downloads
- Downloads (Last 12 months)185
- Downloads (Last 6 weeks)59
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in