Abstract
This paper proposes an augmented reality content authoring system that enables ordinary users who do not have programming capabilities to easily apply interactive features to virtual objects on a marker via gestures. The purpose of this system is to simplify augmented reality (AR) technology usage for ordinary users, especially parents and preschool children who are unfamiliar with AR technology. The system provides an immersive AR environment with a head-mounted display and recognizes users’ gestures via an RGB-D camera. Users can freely create the AR content that they will be using without any special programming ability simply by connecting virtual objects stored in a database to the system. Following recognition of the marker via the system’s RGB-D camera worn by the user, he/she can apply various interactive features to the marker-based AR content using simple gestures. Interactive features applied to AR content can enlarge, shrink, rotate, and transfer virtual objects with hand gestures. In addition to this gesture-interactive feature, the proposed system also allows for tangible interaction using markers. The AR content that the user edits is stored in a database, and is retrieved whenever the markers are recognized. The results of comparative experiments conducted indicate that the proposed system is easier to use and has a higher interaction satisfaction level than AR environments such as fixed-monitor and touch-based interaction on mobile screens.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
AndAR (2014) AndAR-Android augmented reality. http://code.google.com/p/andar/. Accessed 30 May 2014
ARToolKit (2014) ARToolKit. http://www.hitl.washington.edu/artoolkit/. Accessed 30 May 2014
Azuma RT (1997) A survey of augmented reality. Presence 6(4):355–385
Azuma R, Bishop G (1994) Improving static and dynamic registration in an optical see-through HMD. In: Proceedings of the 21st ACM annual conference on computer graphics and interactive techniques, pp 197–204
Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–47
Bajura M, Neumann U (1995) Dynamic registration correction in video-based augmented reality systems. IEEE Comput Graph Appl 15(5):52–60
Billinghurst M, Kato H, Poupyrev I (2001) The magicbook-moving seamlessly between reality and virtuality. IEEE Comput Graph Appl 21(3):6–8
Billinghurst M, Hakkarainen M, Woodward C (2008) Augmented assembly using a mobile phone. In: Proceedings of the 7th ACM international conference on mobile and ubiquitous multimedia, pp 84–87
Buchmann V, Violich S, Billinghurst M, Cockburn A (2004) FingARtips: Gesture based direct manipulation in augmented reality. In: Proceedings of the 2nd ACM international conference on computer graphics and interactive techniques in Australasia and South East Asia, pp 212–221
Bunnun P, Subramanian S, Mayol-Cuevas WW (2013) In-situ interactive image-based model building for augmented reality from a handheld device. Virtual Real 17(2):137–146
Coquillart S, Göbel M (2004) Authoring of mixed reality applications including multi-marker calibration for mobile devices. In: Eurographics symposium on virtual environments, pp 1–9
Doliotis P, Athitsos V, Kosmopoulos D, Perantonis S (2012) Hand shape and 3D pose estimation using depth data from a single cluttered frame. In: Bebis G, Boyle R, Parvin B, Koracin D, Fowlkes C (eds) Advances in visual computing. Springer, Berlin, Heidelberg, pp 148–158
Dorfmuller-Ulhaas K, Schmalstieg D (2001) Finger tracking for interaction in augmented environments. In: Proceedings IEEE and ACM international symposium on augmented reality, pp 55–64
Grimm P, Haller M, Paelke V, Reinhold S, Reimann C, Zauner R (2002) AMIRE-authoring mixed reality. In: The first IEEE international workshop augmented reality toolkit, pp 2–pp
Hackenberg G, McCall R, Broll W (2011) Lightweight palm and finger tracking for real-time 3d gesture control. In: 2011 IEEE virtual reality conference (VR), pp 19–26
Harviainen T, Korkalo O, Woodward C (2009) Camera-based interactions for augmented reality. In: Proceedings of the ACM international conference on advances in computer entertainment technology, pp 307–310
Henrysson A, Billinghurst M (2007) Using a mobile phone for 6 DOF mesh editing. In: Proceedings of the 8th ACM SIGCHI New Zealand chapter’s international conference on computer-human interaction: design centered HCI, pp 9–16
Hoff WA, Nguyen K, Lyon T (1996) Computer-vision-based registration techniques for augmented reality. In: Photonics East’96, international society for optics and photonics, pp 538–548
Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings IWAR’99, pp 85–94
Kato H, Billinghurst M, Poupyrev I, Imamoto K, Tachibana K (2000) Virtual object manipulation on a table-top AR environment. Proc ISAR 2000:111–119
Kaufmann H, Schmalstieg D (2003) Mathematics and geometry education with collaborative augmented reality. Comput Graph 27(3):339–345
Langlotz T, Mooslechner S, Zollmann S, Degendorfer C, Reitmayr G, Schmalstieg D (2012) Sketching up the world: in situ authoring for mobile augmented reality. Pers Ubiquit Comput 16(6):623–630
Ledermann F, Schmalstieg D (2005) April: a high-level framework for creating augmented reality presentations. In: Proceedings. VR 2005, pp 187–194
Lee T, Hollerer T (2007) Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In: 11th IEEE international symposium on wearable computers, pp 83–90
Lee U, Tanaka J (2012) Hand controller: image manipulation interface using fingertips and palm tracking with Kinect depth data. In: Proceedings of Asia Pacific conference on computing human interact, pp 705–706
Lee GA, Nelles C, Billinghurst M, Kim GJ (2004) Immersive authoring of tangible augmented reality applications. In: Proceedings of the 3rd IEEE/ACM international symposium on mixed and augmented reality, pp 172–181
Li D, Weng D, Li Y, Xie J (2013) Touchablear: a new experience of augmented reality. Proc ICVRV 2013:37–42
Liang H, Yuan J, Thalmann D (2012) 3D fingertip and palm tracking in depth image sequences. In: Proceedings of the 20th ACM international conference on Multimedia, pp 785–788
Lund A (2001) Measuring usability with the use questionnaire. Usability Interface 8(2):3–6
MacIntyre B, Gandy M, Dow S, Bolter JD (2004) DART: A toolkit for rapid design exploration of augmented reality experiences. In: Proceedings of the 17th annual ACM symposium on user interface software and technology, pp 197–206
Maier P, Tonnis M, Klinker G (2010) Designing and comparing two-handed gestures to confirm links between user controlled objects. Proc ISMAR 2010:251–252
Ng L, Oon S, Ong S, Nee A (2011) GARDE: a gesture-based augmented reality design evaluation system. Int J Interact Design Manuf (IJIDeM) 5(2):85–94
NyARToolKit (2014) NyARToolKit. http://arforglass.org/?project=nyartoolkit/. Accessed 30 May 2014
Oculus VR (2014) Oculus VR developer center. http://developer.oculusvr.com/. Accessed 30 May 2014
Oikonomidis I, Kyriazis N, Argyros AA (2011) Efficient model-based 3D tracking of hand articulations using Kinect. Proc BMVC 2011:1–11
Oikonomidis I, Kyriazis N, Argyros AA (2012) Tracking the articulated motion of two strongly interacting hands. Proc CVPR 2012:1862–1869
Okuma T, Kiyokawa K, Takemura H, Yokoya N (1998) An augmented reality system using a real-time vision based registration. In: Proceedings fourteenth international conference on pattern recognition, vol 2, pp 1226–1229
Ong S, Wang Z (2011) Augmented assembly technologies based on 3D bare-hand interaction. CIRP Ann-Manuf Technol 60(1):1–4
OpenNI (2014) OpenNI. https://github.com/OpenNI/OpenNI/. Accessed 30 May 2014
Poupyrev I, Tan DS, Billinghurst M, Kato H, Regenbrecht H, Tetsutani N (2002) Developing a generic augmented-reality interface. Computer 35(3):44–50
QCAR (2014) QCAR. http://developer.qualcomm.com/dev/augmented-reality/. Accessed 30 May 2014
Radu I, MacIntyre B (2009) Augmented-reality scratch: a tangible programming environment for children. In: Proceedings of conference on interaction design for children, Como, Italy
Ren Z, Meng J, Yuan J, Zhang Z (2011a) Robust hand gesture recognition with Kinect sensor. In: Proceedings of the 19th ACM international conference on Multimedia, pp 759–760
Ren Z, Yuan J, Zhang Z (2011b) Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: Proceedings of the 19th ACM international conference on Multimedia, pp 1093–1096
Schmalstieg D, Fuhrmann A, Hesina G, Szalavári Z, Encarnaçao LM, Gervautz M, Purgathofer W (2002) The studierstube augmented reality project. Presence Teleoper Virtual Environ 11(1):33–54
Seichter H, Looser J, Billinghurst M (2008) Composar: An intuitive tool for authoring AR applications. In: Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality, pp 177–178
Shim J, Kong M, Yang Y, Seo J, Han T-D (2014) Interactive features based augmented reality authoring tool. Proc ICCE 2014:47–50
SoftKinetic (2014) SoftKinetic: the interface is you. http://www.softkinetic.com/. Accessed 30 May 2014
Thomas BH, Piekarski W (2002) Glove based user interaction techniques for augmented reality in an outdoor environment. Virtual Real 6(3):167–180
Wang Y, Langlotz T, Billinghurst M, Bell T (2009) An authoring tool for mobile phone AR environments. Proc N Z Comput Sci Res Stud Conf 9:1–4
Whitton MC, Pisano ED, Fuchs H (1996) Technologies for augmented reality systems: Realizing ultrasound-guided needle biopsies. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 439–446
Wilson AD (2010) Using a depth camera as a touch sensor. In: Proceedings ACM international conference on interactive tabletops and surfaces, pp 69–72
Wozniewski M, Warne P (2011) Towards in situ authoring of augmented reality content. In: Proceedings ISMAR 2011
Yao Y, Fu Y (2012) Real-time hand pose estimation from RGB-D sensor. Proc ICME 2012:705–710
Acknowledgments
This work was supported by a National Research Foundation of Korea (NRF) Grant (No. NRF-2015R1A2A1A10055673) funded by the Korea government (MEST).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shim, J., Yang, Y., Kang, N. et al. Gesture-based interactive augmented reality content authoring system using HMD. Virtual Reality 20, 57–69 (2016). https://doi.org/10.1007/s10055-016-0282-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-016-0282-z