research-article

Enhancing User Experiences of Mobile-Based Augmented Reality via Spatial Augmented Reality: : Designs and Architectures of Projector-Camera Devices

Author:

Thitirat Siriborvornratanakul Academic Editor:

Hoshang KolivandAuthors Info & Claims

Advances in Multimedia, Volume 2018

https://doi.org/10.1155/2018/8194726

Published: 01 January 2018 Publication History

Abstract

As smartphones, tablet computers, and other mobile devices have continued to dominate our digital world ecosystem, there are many industries using mobile or wearable devices to perform Augmented Reality (AR) functions in their workplaces in order to increase productivity and decrease unnecessary workloads. Mobile-based AR can basically be divided into three main types: phone-based AR, wearable AR, and projector-based AR. Among these, projector-based AR or Spatial Augmented Reality (SAR) is the most immature and least recognized type of AR for end users. This is because there are a small number of commercial products providing projector-based AR functionalities in a mobile manner. Also, prices of mobile projectors are still relatively high. Moreover, there are still many technical problems regarding projector-based AR that have been left unsolved. Nevertheless, it is projector-based AR that has potential to solve a fundamental problem shared by most mobile-based AR systems. Also the always-visible nature of projector-based AR is one good answer for solving current user experience issues of phone-based AR and wearable AR systems. Hence, in this paper, we analyze what are the user experience issues and technical issues regarding common mobile-based AR systems, recently widespread phone-based AR systems, and rising wearable AR systems. Then for each issue, we propose and explain a new solution of how using projector-based AR can solve the problems and/or help enhance its user experiences. Our proposed framework includes hardware designs and architectures as well as a software computing paradigm towards mobile projector-based AR systems. The proposed design is evaluated by three experts using qualitative and semiquantitative research approaches.

References

[1]

Deloitte, “What is digital economy? unicorns, transformation and the internet of things,” 2017, https://www.deloitte.com/mt/en/pages/technology/articles/mt-what-is-digital-economy.html.

[2]

O. Bimber and R. Raskar, Spatial Augmented Reality: Merging Real and Virtual Worlds, A. K. Peters, Ltd., Natick, Ma, USA, 2005.

[3]

S. Levy, “Google glass 2.0 is a startling second act,” 2017, https://www.wired.com/story/google-glass-2-is-here/.

[4]

C. Harrison, H. Benko, and A. D. Wilson, “OmniTouch: Wearable multitouch interaction everywhere,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST'11, pp. 441–450, USA, October 2011.

Digital Library

[5]

O. Bimber, F. Coriand, A. Kleppe, E. Bruns, S. Zollmann, and T. Langlotz, “Superimposing pictorial artwork with projected imagery,” IEEE MultiMedia, vol. 12, no. 1, pp. 16–26, 2005.

Digital Library

[6]

M. R. Mine, J. Van Baar, A. Grundhöfer, D. Rose, and B. Yang, “Projection-based augmented reality in Disney theme parks,” The Computer Journal, vol. 45, no. 7, pp. 32–40, 2012.

Digital Library

[7]

O. Bimber and D. Iwai, “Superimposing dynamic range,” ACM Transactions on Graphics, vol. 27, no. 5, article no. 150, 2008.

Digital Library

[8]

H. Benko, A. D. Wilson, and F. Zannier, “Dyadic projected spatial augmented reality,” in Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 645–656, USA, October 2014.

Digital Library

[9]

T. Siriborvornratanakul, “Multimodal interface towards smartphones: The use of pico projector, passive RGB imaging and active infrared imaging,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Preface, vol. 8563, pp. 41–50, 2014.

[10]

P. Mistry, P. Maes, and L. Chang, “WUW—wear Ur world: a wearable gestural interface,” in Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems (CHI '09), pp. 4111–4116, ACM, Boston, Mass, USA, April 2009.

Digital Library

[11]

J. Schöning, M. Löchtefeld, M. Rohs, A. Krüger, and S. Kratz, “Map torchlight: A mobile Augmented Reality camera projector unit,” in Proceedings of the 27th International Conference Extended Abstracts on Human Factors in Computing Systems, CHI 2009, pp. 3841–3845, USA, April 2009.

Digital Library

[12]

M. Lochtefeld, M. Gehring, J. Schoning, and A. Kruger, “Shelftorchlight: augmenting a shelf using a camera projector unit,” in Proceedings of the in Proceedings of the Internation Conference on Pervasive Computing, pp. 1–4, 2010.

[13]

T. Siriborvornratanakul and M. Sugimoto, “Clutter-aware adaptive projection inside a dynamic environment,” in Proceedings of the VRST 2008 ACM Symposium on Virtual Reality Software and Technology, pp. 241–242, France, October 2008.

Digital Library

[14]

T. Yoshida, Y. Hirobe, H. Nii, N. Kawakami, and S. Tachi, “Twinkle: Interacting with physical surfaces using handheld projector,” in Proceedings of the IEEE Virtual Reality 2010, VR 2010, pp. 87–90, USA, March 2010.

Digital Library

[15]

M. Funk, T. Kosch, K. Wolf, P. Knierim, S. Mayer, and A. Schmidt, “Automatic projection positioning based on surface suitability,” in Proceedings of the 5th ACM International Symposium on Pervasive Displays, PerDis 2016, pp. 75–79, Finland, June 2016.

Digital Library

[16]

R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs, “The office of the future: a unified approach to image-based modeling and spatially immersive displays,” in Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '98), pp. 179–188, Orlando, Fla, USA, July 1998.

Digital Library

[17]

D. Cotting, M. Naef, M. Gross, and H. Fuchs, “Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display,” in Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 100–109, Arlington, VA, USA.

Digital Library

[18]

D. Cotting and M. Gross, “Interactive environment-aware display bubbles,” in Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology 2006, UIST'06, pp. 245–254, Switzerland, October 2006.

Digital Library

[19]

T. Siriborvornratanakul, Ubiquitous projection for vision-based augmented reality [Ph.D. thesis], Department of Electrical Engineering and Information Systems, University of Tokyo, 2011.

[20]

A. D. Wilson, “PlayAnywhere: A compact interactive tabletop projection-vision system,” in Proceedings of the UIST 2005: 18th Annual ACM Symposium on User Interface Software and Technology, pp. 83–92, usa, October 2005.

[21]

D. Molyneaux, S. Izadi, D. Kim, O. Hilliges, S. Hodges, X. Cao, A. Butler, and H. Gellersen, “Interactive environment-aware handheld projectors for pervasive computing spaces,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Preface, vol. 7319, pp. 197–215, 2012.

Digital Library

[22]

K. D. D. Willis, I. Poupyrev, S. E. Hudson, and M. Mahler, “SideBySide: Ad-hoc multi-user interaction with handheld projectors,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST'11, pp. 431–440, USA, October 2011.

Digital Library

[23]

A. D. Wilson, H. Benko, S. Izadi, and O. Hilliges, “Steerable augmented reality with the Beamatron,” in Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST 2012, pp. 413–422, usa, October 2012.

[24]

B. R. Jones, H. Benko, E. Ofek, and A. D. Wilson, “Illumiroom: Peripheral projected illusions for interactive experiences,” in Proceedings of the CHI ’13 Extended Abstracts on Human Factors in Computing Systems (CHI EA’13), pp. 2825–2826, Paris, France, April 2013.

Digital Library

[25]

K. D. D. Willis, T. Shiratori, and M. Mahler, “HideOut: Mobile projector interaction with tangible objects and surfaces,” in Proceedings of the 7th ACM International Conference on Tangible, Embedded and Embodied Interaction, TEI 2013, pp. 331–338, Spain, February 2013.

Digital Library

[26]

K. Fujii, M. D. Grossberg, and S. K. Nayar, “A projector-camera system with real-time photometric adaptation for dynamic environments,” in Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, pp. 814–821, USA, June 2005.

Digital Library

[27]

J. Nosta, “Glass is back and google has big plans in health,” 2017, https://www.forbes.com/sites/johnnosta/2017/07/18/glass-is-back-and-google-has-big-plans-in-health/.

[28]

S. Dent, “Google glass is officially back with clearer vision,” 2017, https://www.engadget.com/2017/07/18/google-glass-is-officially-back-with-a-clearer-vision/.

[29]

M. O'Brien, “DHL supply chain growing use of augmented reality glasses,” 2017, http://multichannelmerchant.com/operations/dhl-supply-chain-growing-use-augmented-reality-glasses/.

[30]

G. Suzuki, T. Murase, and Y. Fujii, “Projecting recorded expert hands at real size, at real speed, and onto real objects for manual work,” in Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI 2016, pp. 13–17, USA, March 2016.

Digital Library

[31]

A. Sutcliffe, O. de Bruijn, B. Gault, T. Fernando, and K. Tan, “Comparing interaction in the real world and cave virtual environments,” in People and Computers XVIII - Design for Life, environments., S. Fincher, P. Markopoulos, D. Moore, and R. Ruddle, Eds., pp. 347–361, 2005.

[32]

C. Winkler, M. Löchtefeld, D. Dobbelstein, A. Krüger, and E. Rukzio, “SurfacePhone: A mobile projection device for single- and multiuser everywhere tabletop interaction,” in Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI 2014, pp. 3513–3522, Canada, May 2014.

Digital Library

[33]

M. Leung, M. Tomitsch, and A. V. Moere, “Designing a personal visualization projection of online social identity,” in Proceedings of the 29th Annual CHI Conference on Human Factors in Computing Systems, CHI 2011, pp. 1843–1848, Canada, May 2011.

Digital Library

[34]

G. Avveduto, F. Tecchia, and H. Fuchs, “Real-world occlusion in optical see-through AR displays,” in Proceedings of the the 23rd ACM Symposium, pp. 1–10, Gothenburg, Sweden, November 2017.

Digital Library

[35]

D. Martinez, B. Dennis, J. Knibbe, G. Lewis, A. Haslam, M. Whiteley, E. Latimer, and D. Coyle, “ReflectoSlates: Personal overlays for tabletops combining camera-projector systems and retroreflective materials,” in Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, CHI EA 2014, pp. 2071–2076, Canada, May 2014.

Digital Library

[36]

A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs, “Pinlight displays: Wide field of view augmented reality eyeglasses using defocused point light sources,” ACM Transactions on Graphics, vol. 33, no. 4, article no. 89, 2014.

Digital Library

[37]

L.-W. Chan, H.-T. Wu, H.-S. Kao, J.-C. Ko, H.-R. Lin, M. Y. Chen, J. Hsu, and Y.-P. Hung, “Enabling beyond-surface interactions for interactive surface with an invisible projection,” in Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, UIST 2010, pp. 263–272, USA, October 2010.

Digital Library

[38]

S. Follmer, M. K. Johnson, E. H. Adelson, and H. Ishii, “deForm: An interactive malleable surface for capturing 2.5d arbitrary objects, tools and touch,” in Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST'11, pp. 527–536, USA, October 2011.

Digital Library

[39]

K. Akasaka, R. Sagawa, and Y. Yagi, “A sensor for simultaneously capturing texture and shape by projecting structured infrared light,” in Proceedings of the 6th International Conference on 3-D Digital Imaging and Modeling, 3DIM 2007, pp. 375–381, Canada, August 2007.

Digital Library

[40]

J. C. Lee, S. Hudson, and P. Dietz, “Hybrid infrared and visible light projection for location tracking,” in Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST 2007, pp. 57–60, USA, October 2007.

Digital Library

[41]

T. Siriborvornratanakul, “Mobile projectors: the review of non-laser and laser technologies,” KMITL Journal of Information Technology, vol. 2, no. 1, 2013.

[42]

K. Hosoi, V. N. Dao, A. Mori, and M. Sugimoto, “CoGAME: Manipulation using a handheld projector,” in Proceedings of the ACM SIGGRAPH 2007: Emerging Technologies, SIGGRAPH 2007, 2007.

Digital Library

[43]

R. Raskar, J. Van Baar, P. Beardsley, T. Willwacher, S. Rao, and C. Forlines, “iLamps: Geometrically aware and self-configuring projectors,” in Proceedings of the ACM SIGGRAPH 2003, pp. 809–818, USA, July 2003.

Digital Library

[44]

R. Sukthankar, R. G. Stockton, and M. D. Mullin, “Smarter presentations: Exploiting homography in camera-projector systems,” Proceedings of the IEEE International Conference on Computer Vision, vol. 1, pp. 247–253, 2001.

[45]

O. Bimber, A. Emmerling, and T. Klemmer, “Embedded entertainment with smart projectors,” The Computer Journal, vol. 38, no. 1, pp. 48–55, 2005.

Digital Library

[46]

O. Bimber, D. Iwai, G. Wetzstein, and A. Grundhofer, “The visual computing of projector-camera systems,” in Proceedings of the ACM SIGGRAPH 2008 Classes, p. 84, USA, August 2008.

Digital Library

[47]

A. Grundhöfer and O. Bimber, “Real-time adaptive radiometric compensation,” IEEE Transactions on Visualization and Computer Graphics, vol. 14, no. 1, pp. 97–108, 2008.

Digital Library

[48]

C. Lewis, “Using the thinking-aloud method in cognitive interface design,” IBM Technical Report, RC-9265, 1982.

Cited By

Eswaran MRaju Bahubalendruni M(2024)Augmented reality aided object mapping for worker assistance/training in an industrial assembly contextComputers and Industrial Engineering10.1016/j.cie.2023.109663185:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.cie.2023.109663
Anvo NThuruthel TTaha Hde Silva LAl-Tabbaa ABrilakis IIida F(2023)Automated 3D Mapping, Localization and Pavement Inspection with Low Cost RGB-D Cameras and IMUsTowards Autonomous Robotic Systems10.1007/978-3-031-43360-3_23(279-291)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-43360-3_23
Ilchev SIlcheva Z(2021)Laser Projection System for Continuous Operation in Manufacturing and Educational Use CasesProceedings of the 22nd International Conference on Computer Systems and Technologies10.1145/3472410.3472416(12-17)Online publication date: 18-Jun-2021
https://dl.acm.org/doi/10.1145/3472410.3472416

Index Terms

Enhancing User Experiences of Mobile-Based Augmented Reality via Spatial Augmented Reality: Designs and Architectures of Projector-Camera Devices
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems

As shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality

Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...

Comments

Information & Contributors

Information

Published In

cover image Advances in Multimedia

Advances in Multimedia Volume 2018, Issue

2018

453 pages

ISSN:1687-5680

EISSN:1687-5699

Issue’s Table of Contents

Copyright © 2018 Thitirat Siriborvornratanakul.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Publisher

Hindawi Limited

London, United Kingdom

Publication History

Published: 01 January 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Eswaran MRaju Bahubalendruni M(2024)Augmented reality aided object mapping for worker assistance/training in an industrial assembly contextComputers and Industrial Engineering10.1016/j.cie.2023.109663185:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.cie.2023.109663
Anvo NThuruthel TTaha Hde Silva LAl-Tabbaa ABrilakis IIida F(2023)Automated 3D Mapping, Localization and Pavement Inspection with Low Cost RGB-D Cameras and IMUsTowards Autonomous Robotic Systems10.1007/978-3-031-43360-3_23(279-291)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-43360-3_23
Ilchev SIlcheva Z(2021)Laser Projection System for Continuous Operation in Manufacturing and Educational Use CasesProceedings of the 22nd International Conference on Computer Systems and Technologies10.1145/3472410.3472416(12-17)Online publication date: 18-Jun-2021
https://dl.acm.org/doi/10.1145/3472410.3472416

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents