Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Pro-Cam SSfM: projector–camera system for structure and spectral reflectance from motion

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image-based reconstruction of an object’s 3D shape having the wavelength-by-wavelength spectral reflectance property enables higher-fidelity object 3D modeling compared with typical RGB-based modeling. In this paper, we propose a novel projector–camera system for practical and low-cost acquisition of a dense object 3D model with the spectral reflectance property. Different from existing spectral 3D data acquisition systems that use a dedicated multispectral camera or light, we use a standard RGB camera and an off-the-shelf projector as active illumination for both the 3D reconstruction and the spectral reflectance estimation. We first propose a calibration-free multi-view structured-light method to reconstruct the 3D points while estimating the intrinsic parameters and the poses of both the camera and the projector, which are alternately moved around the object during our image acquisition procedure. We then exploit the projector for multispectral imaging and estimate the spectral reflectance of each 3D point based on a novel spectral reflectance estimation method considering the geometric relationship between the reconstructed 3D points and the estimated projector positions. Experimental results on both synthetic and real data demonstrate that our system can precisely acquire a dense spectral 3D model using off-the-shelf devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Aanæs, H., Jensen, R.R., Vogiatzis, G., Tola, E., Dahl, A.B.: Large-scale data for multiple-view stereopsis. Int. J. Comput. Vis. 120(2), 153–168 (2016)

    Article  MathSciNet  Google Scholar 

  2. Aeschbacher, J., Wu, J., Timofte, R.: In defense of shallow learned spectral reconstruction from RGB images. In: Proc. of IEEE Int, Conf. on Computer Vision (ICCV) Workshops pp. 471–479 (2017)

  3. Agarwal, S., Mierle, K.: Ceres solver. http://ceres-solver.org

  4. Aliaga, D.G., Xu, Y.: Photogeometric structured light: a self-calibrating and multi-viewpoint framework for accurate 3D modeling. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp. 1–8 (2008)

  5. Arad, B., Ben-Shahar, O.: Sparse recovery of hyperspectral signal from natural RGB images. In: Proc. of European Conf. on Computer Vision (ECCV) pp. 19–34 (2016)

  6. Baek, S.H., Kim, I., Gutierrez, D., Kim, M.H.: Compact single-shot hyperspectral imaging using a prism. ACM Trans. Graph. 36(6), 1–12 (2017)

    Article  Google Scholar 

  7. Behmann, J., Mahlein, A.K., Paulus, S., Dupuis, J., Kuhlmann, H., Oerke, E.C., Plümer, L.: Generation and application of hyperspectral 3D plant models: methods and challenges. Mach. Vis. Appl. 27(5), 611–624 (2016)

    Article  Google Scholar 

  8. Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)

    Article  Google Scholar 

  9. Cao, X., Du, H., Tong, X., Dai, Q., Lin, S.: A prism-mask system for multispectral video acquisition. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2423–2435 (2011)

    Article  Google Scholar 

  10. Caspi, D., Kiryati, N., Shamir, J.: Range imaging with adaptive color structured light. IEEE Trans. Pattern Anal. Mach. Intell. 20(5), 470–480 (1998)

    Article  Google Scholar 

  11. Chane, C.S., Mansouri, A., Marzani, F., Boochs, F.: Integration of 3D and multispectral data for cultural heritage applications: survey and perspectives. Image Vis. Comput. 31(1), 91–102 (2013)

    Article  Google Scholar 

  12. Chi, C., Yoo, H., Ben-Ezra, M.: Multi-spectral imaging by optimized wide band illumination. Int. J. Comput. Vis. 86, 140–151 (2010)

    Article  Google Scholar 

  13. Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: Meshlab: an open-source mesh processing tool. In: Proc. of Eurographics Italian Chapter Conference pp. 129–136 (2008)

  14. Devlin, K., Chalmers, A., Wilkie, A., Purgathofer, W.: Tone reproduction and physically based spectral rendering. Eurographics State of the Art Report pp. 1–23 (2002)

  15. Fu, Y., Zheng, Y., Zhang, L., Huang, H.: Spectral reflectance recovery from a single RGB image. IEEE Trans. Comput. Imaging 4(3), 382–394 (2018)

    Article  Google Scholar 

  16. Furukawa, R., Inose, K., Kawasaki, H.: Multi-view reconstruction for projector camera systems based on bundle adjustment. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 69–76 (2009)

  17. Furukawa, R., Nagamatsu, G., Kawasaki, H.: Simultaneous shape registration and active stereo shape reconstruction using modified bundle adjustment. In: Int. Conf. on 3D Vision (3DV), pp. 453–462 (2019)

  18. Furukawa, R., Sagawa, R., Kawasaki, H., Sakashita, K., Yagi, Y., Asada, N.: One-shot entire shape acquisition method using multiple projectors and cameras. Proc. of Pacific-Rim Symposium on Image and Video Technology (PSIVT) pp. 107–114 (2010)

  19. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010)

    Article  Google Scholar 

  20. Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F.J., Marín-Jiménez, M.J.: Simultaneous reconstruction and calibration for multi-view structured light scanning. J. Vis. Commun. Image Represent. 39, 120–131 (2016)

    Article  Google Scholar 

  21. Geng, J.: Structured-light 3D surface imaging: a tutorial. Adv. Opt. Photonics 3(2), 128–160 (2011)

    Article  Google Scholar 

  22. Greg, T., Marc, L.: The stanford 3d scanning repository. http://graphics.stanford.edu/data/3Dscanrep/

  23. Han, S., Sato, I., Okabe, T., Sato, Y.: Fast spectral reflectance recovery using DLP projector. Int. J. Comput. Vis. 110(2), 172–184 (2014)

    Article  Google Scholar 

  24. Herakleous, K., Poullis, C.: 3DUNDERWORLD-SLS: an open-source structured-light scanning system for rapid geometry acquisition. CoRR abs/1406.6595 (2014)

  25. Hirai, K., Nakahata, R., Horiuchi, T.: Measuring spectral reflectance and 3D shape using multi-primary image projector. In: Proc. of Int. Conf. on Image and Signal Processing (ICISP) pp. 137–147 (2016)

  26. Inokuchi, S.: Range imaging system for 3-D object recognition. ICPR 1984, 806–808 (1984)

    Google Scholar 

  27. Ito, S., Ito, K., Aoki, T., Tsuchida, M.: A 3D reconstruction method with color reproduction from multi-band and multi-view images. In: Proc. of Asian Conf. on Computer Vision (ACCV) pp. 236–247 (2016)

  28. Jia, Y., Zheng, Y., Gu, L., Subpa-Asa, A., Lam, A., Sato, Y., Sato, I.: From RGB to spectrum for natural scenes via manifold-based mapping. In: Proc. of IEEE Int. Conf. on Computer Vision (ICCV) pp. 4705–4713 (2017)

  29. Jiang, J., Liu, D., Gu, J., Süsstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras?. In: Proc. of Workshop on Applications of Computer Vision (WACV) pp. 168–179 (2013)

  30. Kazhdan, M., Hoppe, H.: Screened Poisson surface reconstruction. ACM Trans. Graph. 32(3), 1–13 (2013)

    Article  MATH  Google Scholar 

  31. Kim, K., Torii, A., Okutomi, M.: Multi-view inverse rendering under arbitrary illumination and albedo. In: Proc. of European Conf. on Computer Vision (ECCV) pp. 750–767 (2016)

  32. Kim, M.H., Harvey, T.A., Kittle, D.S., Rushmeier, H., Dorsey, J., Prum, R.O., Brady, D.J.: 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Trans. Graph. 31(4), 1–11 (2012)

    Google Scholar 

  33. Kim, M.H., Rushmeier, H., Ffrench, J., Passeri, I., Tidmarsh, D.: Hyper 3D: 3D graphics software for examining cultural artifacts. ACM J. Comput. Cult. Herit. 7(3), 1–19 (2014)

    Article  Google Scholar 

  34. Kitahara, M., Okabe, T., Fuchs, C., Lensch, H.P.A.: Simultaneous estimation of spectral reflectance and normal from a small number of images. In: Proc. of Int. Conf. on Computer Vision Theory and Applications (VISAPP) pp. 303–313 (2015)

  35. Ley, A., Hänsch, R., Hellwich, O.: Syb3r: a realistic synthetic benchmark for 3D reconstruction from images. In: Proc. of European Conf. on Computer Vision (ECCV), pp. 236–251 (2016)

  36. Li, C., Monno, Y., Hidaka, H., Okutomi, M.: Pro-cam ssfm: projector-camera system for structure and spectral reflectance from motion. In: Proc. of the IEEE/CVF Int. Conf. on Computer Vision (ICCV), pp. 2414–2423 (2019)

  37. Li, C., Monno, Y., Okutomi, M.: Spectral MVIR: joint reconstruction of 3d shape and spectral reflectance. In: IEEE Int. Conf. on Computational Photography (ICCP), pp. 1–12 (2021)

  38. Li, C., Torii, A., Okutomi, M.: Robust, precise, and calibration-free shape acquisition with an off-the-shelf camera and projector. In: Proc. of IEEE Conf. on Consumer Electronics (ICCE) pp. 1–6 (2018)

  39. Liang, J., Zia, A., Zhou, J., Sirault, X.: 3D plant modelling via hyperspectral imaging. In: Proc. of IEEE Int, Conf. on Computer Vision Workshops (ICCVW) pp. 172–177 (2013)

  40. Lourakis, M., Argyros, A.A.: SBA: a software package for generic sparse bundle adjustment. ACM Trans. Math. Softw. 36(1), 1–30 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  41. Mansouri, A., Lathuiliere, A., Marzani, F., Voisin, Y., Gouton, P.: Toward a 3D multispectral scanner: an application to multimedia. IEEE Multimed. 14(1), 40–47 (2007)

    Article  Google Scholar 

  42. Maurer, D., Ju, Y.C., Breuß, M., Bruhn, A.: Combining shape from shading and stereo: a variational approach for the joint estimation of depth, illumination and albedo. In: Proc. of British Machine Vision Conference (BMVC) pp. 76–1–14 (2016)

  43. Mélou, J., Quéau, Y., Durou, J., Castan, F., Cremers, D.: Variational reflectance estimation from multi-view images. J. Math. Imaging Vis. 60(9), 1527–1546 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  44. Monno, Y., Kikuchi, S., Tanaka, M., Okutomi, M.: A practical one-shot multispectral imaging system using a single image sensor. IEEE Trans. Image Process. 24(10), 3048–3059 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  45. Nguyen, H., Nguyen, D., Wang, Z., Kieu, H., Le, M.: Real-time, high-accuracy 3D imaging and shape measurement. Appl. Opt. 54(1), A9–A17 (2015)

    Article  Google Scholar 

  46. Nguyen, R.H.M., Prasad, D.K., Brown, M.S.: Training-based spectral reconstruction from a single RGB image. In: Proc. of European Conf. on Computer Vision (ECCV) pp. 186–201 (2014)

  47. Ozawa, K., Sato, I., Yamaguchi, M.: Hyperspectral photometric stereo for a single capture. J. Opt. Soc. Am. A 34(3), 384–394 (2018)

    Article  Google Scholar 

  48. Park, J.I., Lee, M.H., Grossberg, M.D., Nayar, S.K.: Multispectral imaging using multiplexed illumination. In: Proc. of IEEE Int. Conf. on Computer Vision (ICCV) pp. 1–8 (2007)

  49. Parkkinen, J.P., Hallikainen, J., Jaaskelainen, T.: Characteristic spectra of munsell colors. JOSA A 6(2), 318–322 (1989)

    Article  Google Scholar 

  50. Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: Proc. of IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 3212–3217 (2009)

  51. Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Pattern Recognit. 43(8), 2666–2680 (2010)

    Article  MATH  Google Scholar 

  52. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp. 4104–4113 (2016)

  53. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp. 519–528 (2006)

  54. Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops pp. 939–947 (2018)

  55. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. 25(3), 835–846 (2006)

    Article  Google Scholar 

  56. Weinmann, M., Schwartz, C., Ruiters, R., Klein, R.: A multi-camera, multi-projector super-resolution framework for structured light. In: Proc. of Int. Conf. on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT) pp. 397–404 (2011)

  57. Wug Oh, S., Brown, M.S., Pollefeys, M., Joo Kim, S.: Do it yourself hyperspectral imaging with everyday digital cameras. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) pp. 2461–2469 (2016)

  58. Zia, A., Liang, J., Zhou, J., Gao, Y.: 3D reconstruction from hyperspectral images. In: Proc. of IEEE Winter Conf. on Applications of Computer Vision (WACV) pp. 318–325 (2015)

Download references

Acknowledgements

This work was partly supported by JSPS KAKENHI Grant Number 17H00744 and 21K17762.

Author information

Authors and Affiliations

  1. Department of Systems and Control Engineering, School of Engineering, Tokyo Institute of Technology, Tokyo, Japan

    Chunyu Li, Yusuke Monno & Masatoshi Okutomi

Authors
  1. Chunyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yusuke Monno
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masatoshi Okutomi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chunyu Li.

Ethics declarations

Conflict of interest

The authors declared that they have no conflicts of interest to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 16653 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., Monno, Y. & Okutomi, M. Pro-Cam SSfM: projector–camera system for structure and spectral reflectance from motion. Vis Comput 39, 1651–1666 (2023). https://doi.org/10.1007/s00371-022-02434-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02434-0

Keywords

Search

Navigation