research-article

Open access

Dynamic Projection Mapping of Deformable Stretchable Materials

Authors:

Muhammad Twaha Ibrahim,

Gopi Meenakshisundaram,

Aditi MajumderAuthors Info & Claims

VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

Article No.: 35, Pages 1 - 5

https://doi.org/10.1145/3385956.3418970

Published: 01 November 2020 Publication History

All formats PDF

Abstract

We present a method for dynamic projection mapping on deformable, stretchable and elastic materials (e.g. cloth) using a time of flight (ToF) depth camera (e.g. Azure Kinect or Pico-Flexx) that come equipped with an IR camera. We use Bezier surfaces to model the projection surface without explicitly modeling the deformation. We devise an efficient tracking method that tracks the boundary of the surface material using the IR-Depth camera. This achieves realistic mapping even in the interior of the surface, with simple markers (e.g. black dots or squares) or without markers entirely, such that the projection appears to be printed on the material. The surface representation is updated in real-time using GPU based computations. Further, we also show that the speed of these updates is limited by the camera frame rate and therefore can be adopted for higher speed cameras as well. This technique can be used to project on several stretchable moving materials to change their appearance.

Supplementary Material

MP4 File (a35-ibrahim-supplement.mp4)

Download
65.72 MB

References

[1]

Yuichiro Fujimoto, Ross T. Smith, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, Hirokazu Kato, and Bruce H. Thomas. 2014. Geometrically-Correct Projection-Based Texture Mapping onto a Deformable Object. In IEEE Transactions on Visualization and Computer Graphics. 20. 540-9.

Digital Library

[2]

Tomohiro Sueishi, Hiromasa Oku, and Masatoshi Ishikawa. 2015. Robust high-speed tracking against illumination changes for dynamic projection mapping. In IEEE Virtual Reality (VR), Arles, 2015, pp. 97-104.

[3]

Christian Siegl, Matteo Colaianni, Lucas Thies, Justus Thies, Michael Zollhöfer, Shahram Izadi, Marc Stamminger, and Frank Bauer. 2015. Real-time pixel luminance optimization for dynamic multi-projection mapping. In ACM Trans. Graph. 34, 6, Article 237 (November 2015), 11 pages.

Digital Library

[4]

Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2015. Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology (VRST ’15). Association for Computing Machinery, New York, NY, USA, 149–152.

Digital Library

[5]

Yi Zhou, Shuangjiu Xiao, Ning Tang, Zhiyong Wei, and Xu Chen. 2016. Pmomo: Projection Mapping on Movable 3D Object. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 781–790.

Digital Library

[6]

Christian Siegl, Vanessa Lange, Marc Stamminger, Frank Bauer, and Justus Thies. 2017. FaceForge: Markerless Non-Rigid Face Multi-Projection Mapping. In IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 11, pp. 2440-2446, Nov. 2017.

Digital Library

[7]

Vanessa Lange, Christian Siegl, Matteo Colaianni, Marc Stamminger, and Frank Bauer. 2017. Robust blending and occlusion compensation in dynamic multi-projection mapping. In Proceedings of the European Association for Computer Graphics: Short Papers (EG ’17). Eurographics Association, Goslar, DEU, 1–4.

Digital Library

[8]

Gaku Narita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2017. Dynamic Projection Mapping onto Deforming Non-Rigid Surface Using Deformable Dot Cluster Marker. In IEEE Transactions on Visualization and Computer Graphics, vol. 23, no. 3, pp. 1235-1248, 1 March 2017.

Digital Library

[9]

Leo Miyashita, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2018. MIDAS projection: markerless and modelless dynamic projection mapping for material representation. In ACM Trans. Graph. 37, 6, Article 196 (November 2018), 12 pages.

Digital Library

[10]

Yuri Mikawa, Tomohiro Sueishi, Yoshihiro Watanabe, and Masatoshi Ishikawa. 2018. VarioLight: hybrid dynamic projection mapping using high-speed projector and optical axis controller. In SIGGRAPH Asia 2018 Emerging Technologies (SA ’18). Association for Computing Machinery, New York, NY, USA, Article 17, 1–2.

Digital Library

[11]

Hirotaka Asayama, Daisuke Iwai, and Kosuke Sato. 2018. Fabricating Diminishable Visual Markers for Geometric Registration in Projection Mapping. In IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 2, pp. 1091-1102, 1 Feb. 2018.

[12]

Philipp Kurth, Vanessa Lange, Christian Siegl, Marc Stamminger, and Frank Bauer. 2018. Auto-Calibration for Dynamic Multi-Projection Mapping on Arbitrary Surfaces. In IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 11, pp. 2886-2894, Nov. 2018.

[13]

Shingo Kagami, and Koichi Hashimoto. 2019. Animated Stickies: Fast Video Projection Mapping onto a Markerless Plane through a Direct Closed-Loop Alignment. In IEEE Transactions on Visualization and Computer Graphics, vol. 25, no. 11, pp. 3094-3104, Nov. 2019.

[14]

Lihui Wang, Hongjin Xu, Yunpu Hu, Satoshi Tabata, and Masatoshi Ishikawa. 2019. Dynamic Depth-of-Field Projection for 3D Projection Mapping. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, Paper INT021, 1–4.

Digital Library

[15]

Christoph Resch, Peter Keitler, and Gudrun Klinker. 2014. Sticky projections — A new approach to interactive shader lamp tracking. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, 2014, pp. 151-156.

[16]

Naoki Hashimoto, Ryo Koizumi, and Daisuke Kobayashi. 2017. Dynamic Projection Mapping with a Single IR Camera. Int. J. Comput. Games Technol. 2017 (February 2017), 2.

Digital Library

[17]

Christian Siegl, Matteo Colaianni, Marc Stamminger, and Frank Bauer. 2016. Stray-light compensation in dynamic projection mapping. In SIGGRAPH ASIA 2016 Technical Briefs (SA ’16). Association for Computing Machinery, New York, NY, USA, Article 21, 1–4.

Digital Library

[18]

Vanessa Lange, Christian Siegl, Matteo Colaianni, Philipp Kurth, Marc Stamminger, and Frank Bauer. 2016. Interactive Painting and Lighting in Dynamic Multi-Projection Mapping. In De Paolis L., Mongelli A. (eds) Augmented Reality, Virtual Reality, and Computer Graphics. AVR 2016. Lecture Notes in Computer Science, vol 9769. Springer, Cham

[19]

Daniel Saakes, Hui-Shyong Yeo, Seung-Tak Noh, Gyeol Han, and Woontack Woo. 2016. Mirror Mirror: An On-Body T-shirt Design System. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 6058–6063.

Digital Library

[20]

Amit H. Bermano, Markus Billeter, Daisuke Iwai, and Anselm Grundhöfer. 2017. Makeup Lamps: Live Augmentation of Human Faces via Projection. Comput. Graph. Forum 36, 2 (May 2017), 311–323.

[21]

Jürgen Steimle, Andreas Jordt, and Pattie Maes. 2013. Flexpad: highly flexible bending interactions for projected handheld displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 237–246.

Digital Library

[22]

Mahdi Abbaspour Tehrani, Gopi M, and Aditi Majumder. 2019. Automated Geometric Registration for Multi-Projector Displays on Arbitrary 3D Shapes Using Uncalibrated Devices. In IEEE Transactions on Visualization and Computer Graphics.

[23]

Chris Harris and Mike Stephens. 1988. A Combined Corner and Edge Detector. In Alvey Vision Conference.

[24]

Bruce D. Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121—130

[25]

Carlo Tomasi and Takeo Kanade. 1991. Detection and Tracking of Point Features. Carnegie Mellon University Technical Report CMU-CS-91-132, April 1991

[26]

Jianbo Shi and Carlo Tomasi. 1994. Good Features to Track. IEEE Conference on Computer Vision and Pattern Recognition, pages 593–600, 1994.

[27]

Ramesh Raskar, Michael S. Brown, Ruigang Yang, Wei-Chao Chen, Greg Welch, Herman Towles, B. Seales, and Henry Fuchs. 1999. Multi projector displays using camera-based registration. Proc. of IEEE Vis, 1999

[28]

Behzad Sajadi and Aditi Majumder. Automatic registration of multiple projectors on swept surfaces. 2010. In ACM Virtual Reality and Software Technology, 2010.

Digital Library

[29]

Behzad Sajadi and Aditi Majumder. 2011. Auto-calibrating projectors for tiled displays on piecewise smooth vertically extruded surfaces. In IEEE Transactions on Visualization and Computer Graphics 2011

[30]

Behzad Sajadi and Aditi Majumder. 2009. Markerless View Independent Geometric Registration of Multiple Distorted Projectors on Vertically Extruded Surfaces Using a Single Uncalibrated Camera. In IEEE Transactions on Visualization and Computer Graphics, 2009.

[31]

Ramesh Raskar. 2000. Immersive planar displays using roughly aligned projectors. In Proc. of IEEE VR, 2000

Digital Library

[32]

Michael S. Brown and W. Brent Seales. 2002. A practical and flexible tiled display system. In 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings., Beijing, China, 2002, pp. 194-203.

[33]

Tyler Johnson and Henry Fuchs. 2007. Real-time projector tracking on complex geometry using ordinary imagery. In IEEE CVPR Workshop on ProjectorCamera Systems (PROCAMS), 2007

[34]

Parinya Punpongsanon, Daisuke Iwai, and Kosuke Sato. 2020. FleXeen: Visually Manipulating Perceived Fabric Bending Stiffness in Spatial Augmented Reality. In IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 2, pp. 1433-1439, 1 Feb. 2020.

[35]

Ramesh Raskar, Greg Welch, Kok-Lim Low, and Deepak Bandyopadhyay. 2001. Shader Lamps: Animating Real Objects with Image-Based Illumination. In Eurographics Workshop on Rendering Techniques EGSR 2001, pp. 89-102.

[36]

Ramesh Raskar, Jeroen van Baar, Paul Beardsley, Thomas Willwacher, Srinivas Rao, and Clifton Forlines. 2006. ILamps: geometrically aware and self-configuring projectors. In ACM SIGGRAPH 2006 Courses (SIGGRAPH ’06). Association for Computing Machinery, New York, NY, USA, 7–es.

Digital Library

[37]

Tyler Johnson, Florian Gyarfas, Rick Skarbez, Herman Towles, and Henry Fuchs. 2007. A Personal Surround Environment: Projective Display with Correction for Display Surface Geometry and Extreme Lens Distortion. In Proceedings of IEEE Virtual Reality Conference, Virtual Reality 2007, pp. 147-154.

[38]

Katherine L. Bouman, Bei Xiao, Peter Battaglia, and William T. Freeman. 2013. Estimating the material properties of fabric from video. In Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1984–1991.

[39]

Wenyan Bi and Bei Xiao. 2016. Perceptual constancy of mechanical properties of cloth under variation of external forces. In Proceedings of the ACM Symposium on Applied Perception (SAP ’16). Association for Computing Machinery, New York, NY, USA, 19–23.

Digital Library

[40]

Takahiro Kawabe, Taiki Fukiage, Masataka Sawayama, and Shin'ya Nishida. 2016. Deformation Lamps: A Projection Technique to Make Static Objects Perceptually Dynamic. ACM Trans. Appl. Percept. 13, 2, Article 10 (March 2016), 17 pages.

Digital Library

[41]

Daniel Moreno and Gabriel Taubin. 2012. Simple, Accurate, and Robust Projector-Camera Calibration. In 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, Zurich, 2012, pp. 464-471.

Digital Library

[42]

http://mesh.brown.edu/calibration/software.html

[43]

Julien Pilet, Vincent Lepetit and Pascal Fua. 2005. Real-time nonrigid surface detection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), San Diego, CA, USA, 2005, pp. 822-828 vol. 1.

Digital Library

[44]

Ibai Leizea, Hugo Álvarez, Iker Aguinaga and Diego Borro. 2014. Real-time deformation, registration and tracking of solids based on physical simulation. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, 2014, pp. 165-170.

[45]

Dat Tien Ngo, Jonas Östlund and Pascal Fua. 2016. Template-Based Monocular 3D Shape Recovery Using Laplacian Meshes. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 1, pp. 172-187, 1 Jan. 2016.

Digital Library

[46]

Antoine Petit, Vincenzo Lippiello and Bruno Siciliano. 2015. Real-time tracking of 3D elastic objects with an RGB-D sensor. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, 2015, pp. 3914-3921.

Digital Library

Cited By

IWAI D(2024)Projection mapping technologies: A review of current trends and future directionsProceedings of the Japan Academy, Series B10.2183/pjab.100.012100:3(234-251)Online publication date: 11-Mar-2024
https://doi.org/10.2183/pjab.100.012
Ibrahim MMajumder A(2024)Multi-Projector Dynamic Spatially Augmented Reality on Deformable Surfaces2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00372(1156-1157)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00372
Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372097
Show More Cited By

Recommendations

Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array
VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

In recent years, projection mapping has attracted much attention in a variety of fields. Generally, however, the objects in projection mapping are limited to rigid and static or quasistatic objects. Dynamic projection mapping onto a deformable object ...
Dynamic projection mapping on deformable stretchable materials using boundary tracking
Abstract
We present the first method for dynamic projection mapping on rectangular, deformable and stretchable elastic materials using consumer-grade time of flight (ToF) depth cameras. We use a B-Spline patch to model the projection surface ...
Graphical abstract

Display Omitted
Highlights
- Exploit properties of rational B-spline patches to model the projection surface.
Tangible projection mapping: dynamic appearance augmenting of objects in hands
SA '18: SIGGRAPH Asia 2018 Emerging Technologies

We propose a technique of dynamic appearance augmentation by projection to object in user's hands, named Tangible Projection Mapping. This technique allows users to hold a target object freely, and augments appearance of that in various postures by user'...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology

November 2020

429 pages

ISBN:9781450376198

DOI:10.1145/3385956

Editors:
Robert J. Teather
Carleton University, Canada
,
Chris Joslin
Carleton University, Canada
,
Wolfgang Stuerzlinger
Simon Fraser University, Canada
,
Pablo Figueroa
Universidad de los Andes, Colombia
,
Yaoping Hu
University of Calgary, Canada
,
Anil Ufuk Batmaz
Simon Fraser University, Canada
,
Wonsook Lee
University of Ottawa, Canada
,
Francisco Ortega
Colorado State University, USA

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2020

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

VRST '20

Sponsor:

VRST '20: 26th ACM Symposium on Virtual Reality Software and Technology

November 1 - 4, 2020

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
994
Total Downloads

Downloads (Last 12 months)209
Downloads (Last 6 weeks)23

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

IWAI D(2024)Projection mapping technologies: A review of current trends and future directionsProceedings of the Japan Academy, Series B10.2183/pjab.100.012100:3(234-251)Online publication date: 11-Mar-2024
https://doi.org/10.2183/pjab.100.012
Ibrahim MMajumder A(2024)Multi-Projector Dynamic Spatially Augmented Reality on Deformable Surfaces2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00372(1156-1157)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00372
Ibrahim MGopi MMajumder A(2024)Real-Time Seamless Multi-Projector Displays on Deformable SurfacesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209730:5(2527-2537)Online publication date: May-2024
https://doi.org/10.1109/TVCG.2024.3372097
Li YYin WLi JXie X(2024)Physics-Based Efficient Full Projector Compensation Using Only Natural ImagesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328168130:8(4968-4982)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3281681
Tehrani MIbrahim MMajumder AGopi M(2024)3D Gamut Morphing for Non-Rectangular Multi-Projector DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.327743630:8(4724-4738)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3277436
Ibrahim MMajumder AGopi MSayadi LVyas R(2024)Illuminating precise stencils on surgical sites using projection-based augmented realitySmart Health10.1016/j.smhl.2024.10047632(100476)Online publication date: Jun-2024
https://doi.org/10.1016/j.smhl.2024.100476
Ibrahim MMajumder A(2023)Spatially Augmented Reality on Non-rigid Dynamic Surfaces2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00350(1009-1010)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00350
Ibrahim MGopi MMajumder A(2023)Projector-Camera Calibration on Dynamic, Deformable Surfaces2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00295(905-906)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00295
Ibrahim MGopi MMajumder A(2023)Self-Calibrating Dynamic Projection Mapping System for Dynamic, Deformable Surfaces with Jitter Correction and Occlusion Handling2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00044(293-302)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00044
Ibrahim MMajumder AGopi M(2022)Dynamic projection mapping on deformable stretchable materials using boundary trackingComputers and Graphics10.1016/j.cag.2022.01.004103:C(61-74)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.cag.2022.01.004
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten