poster

VRySmart: a Framework for Embedding Smart Devices in Virtual Reality

Authors:

Akhmajon Makhsadov,

Donald Degraen,

Felix Kosmalla,

Kamila Mushkina,

Antonio KrügerAuthors Info & Claims

CHI EA '22: CHI Conference on Human Factors in Computing Systems Extended Abstracts

Article No.: 358, Pages 1 - 8

https://doi.org/10.1145/3491101.3519717

Published: 28 April 2022 Publication History

Abstract

As immersive virtual experiences find their way into our living room entertainment, they are becoming part of our daily technological consumption. However, state-of-the-art virtual reality (VR) remains disconnected from other digital devices in our environment, such as smartphones or tablets. As context switches between acting in the virtual environment and resolving external notifications negatively influence immersion, we look towards integrating smart devices into virtual experiences. To this aim, we present the VRySmart framework. Through either optical marker tracking or simultaneous localization and mapping (SLAM), embedded smart devices can be used as VR controllers with different levels of integration while their content is incorporated into the virtual context to support the plausibility of the illusion. To investigate user impressions, we conducted a study (N = 10) where participants used a smartphone in four different virtual scenarios. Participants positively assessed smart device usage in VR. We conclude by framing implications for future work.

Supplementary Material

MP4 File (3491101.3519717-talk-video.mp4)

Talk Video

Download
282.73 MB

MP4 File (3491101.3519717-video-figure.mp4)

Video Figure

Download
282.73 MB

MP4 File (3491101.3519717-video-preview.mp4)

Video Preview

Download
51.75 MB

References

[1]

Jodi Aronson. 1995. A Pragmatic View of Thematic Analysis. The Qualitative Report (April 1995). https://doi.org/10.46743/2160-3715/1995.2069

[2]

Sabah Boustila, Thomas Guégan, Kazuki Takashima, and Yoshifumi Kitamura. 2019. Text Typing in VR Using Smartphones Touchscreen and HMD. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 860–861. https://doi.org/10.1109/VR.2019.8798238

[3]

Florian Daiber, Donald Degraen, André Zenner, Frank Steinicke, Oscar Javier Ariza Núñez, and Adalberto L. Simeone. 2020. Everyday Proxy Objects for Virtual Reality. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375165

Digital Library

[4]

Donald Degraen, Bruno Fruchard, Frederik Smolders, Emmanouil Potetsianakis, Seref Güngör, Antonio Krüger, and Jürgen Steimle. 2021. Weirding Haptics: In-Situ Prototyping of Vibrotactile Feedback in Virtual Reality through Vocalization. Association for Computing Machinery, New York, NY, USA, 936–953. https://doi.org/10.1145/3472749.3474797

[5]

Donald Degraen, Michal Piovarči, Bernd Bickel, and Antonio Krüger. 2021. Capturing Tactile Properties of Real Surfaces for Haptic Reproduction. Association for Computing Machinery, New York, NY, USA, 954–971. https://doi.org/10.1145/3472749.3474798

[6]

Donald Degraen, Anna Reindl, Akhmajon Makhsadov, André Zenner, and Antonio Krüger. 2020. Envisioning Haptic Design for Immersive Virtual Environments. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS’ 20 Companion). Association for Computing Machinery, New York, NY, USA, 287–291. https://doi.org/10.1145/3393914.3395870

Digital Library

[7]

Donald Degraen, André Zenner, and Antonio Krüger. 2019. Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300479

Digital Library

[8]

Paulo Dias, Luis Afonso, Sérgio Eliseu, and Beatriz Sousa Santos. 2018. Mobile Devices for Interaction in Immersive Virtual Environments. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces (Castiglione della Pescaia, Grosseto, Italy) (AVI ’18). Association for Computing Machinery, New York, NY, USA, Article 19, 9 pages. https://doi.org/10.1145/3206505.3206526

Digital Library

[9]

Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. A Reconfigurable Mobile Head-Mounted Display Supporting Real World Interactions. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411763.3451765

[10]

Sarthak Ghosh, Lauren Winston, Nishant Panchal, Philippe Kimura-Thollander, Jeff Hotnog, Douglas Cheong, Gabriel Reyes, and Gregory D. Abowd. 2018. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4(2018), 1447–1456. https://doi.org/10.1109/TVCG.2018.2793698

Digital Library

[11]

Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed. 2019. Mixing Realities for Sketch Retrieval in Virtual Reality. In The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry (Brisbane, QLD, Australia) (VRCAI ’19). Association for Computing Machinery, New York, NY, USA, Article 50, 2 pages. https://doi.org/10.1145/3359997.3365751

Digital Library

[12]

Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4021–4033. https://doi.org/10.1145/3025453.3025683

Digital Library

[13]

Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, and Enrico Rukzio. 2018. FaceDisplay: Towards Asymmetric Multi-User Interaction for Nomadic Virtual Reality. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173628

Digital Library

[14]

Keisuke Hattori and Tatsunori Hirai. 2020. Inside-out Tracking Controller for VR/AR HMD Using Image Recognition with Smartphones. In ACM SIGGRAPH 2020 Posters (Virtual Event, USA) (SIGGRAPH ’20). Association for Computing Machinery, New York, NY, USA, Article 23, 2 pages. https://doi.org/10.1145/3388770.3407430

[15]

Teresa Hirzle, Jan Rixen, Jan Gugenheimer, and Enrico Rukzio. 2018. WatchVR: Exploring the Usage of a Smartwatch for Interaction in Mobile Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188629

Digital Library

[16]

Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph. D. Dissertation. University of North Carolina at Chapel Hill, USA. Advisor(s) Frederick P. Brooks Jr. http://www.cs.unc.edu/techreports/01-017.pdf

[17]

Daniel Kharlamov, Brandon Woodard, Liudmila Tahai, and Krzysztof Pietroszek. 2016. TickTockRay: Smartwatch-Based 3D Pointing for Smartphone-Based Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology(Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA, 365–366. https://doi.org/10.1145/2993369.2996311

Digital Library

[18]

Youngwon R. Kim and Gerard J. Kim. 2017. HoVR-Type: Smartphone as a typing interface in VR using hovering. In 2017 IEEE International Conference on Consumer Electronics (ICCE). 200–203. https://doi.org/10.1109/ICCE.2017.7889285

[19]

Hai-Ning Liang, Yuwei Shi, Feiyu Lu, Jizhou Yang, and Konstantinos Papangelis. 2016. VRMController: An Input Device for Navigation Activities in Virtual Reality Environments. In Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry - Volume 1(Zhuhai, China) (VRCAI ’16). Association for Computing Machinery, New York, NY, USA, 455–460. https://doi.org/10.1145/3013971.3014005

Digital Library

[20]

Fabrice Matulic, Aditya Ganeshan, Hiroshi Fujiwara, and Daniel Vogel. 2021. Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445583

[21]

Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815

[22]

Niels Christian Nilsson, André Zenner, and Adalberto L. Simeone. 2021. Propping Up Virtual Reality With Haptic Proxies. IEEE Computer Graphics and Applications 41, 05 (sep 2021), 104–112. https://doi.org/10.1109/MCG.2021.3097671

[23]

Niels Christian Nilsson, André Zenner, Adalberto L. Simeone, Donald Degraen, and Florian Daiber. 2021. Haptic Proxies for Virtual Reality: Success Criteria and Taxonomy. In Proceedings of the 1st Workshop on Everyday Proxy Objects for Virtual Reality. (EPO4VR ’21).

[24]

Rufat Rzayev, Sven Mayer, Christian Krauter, and Niels Henze. 2019. Notification in VR: The Effect of Notification Placement, Task and Environment. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (Barcelona, Spain) (CHI PLAY ’19). Association for Computing Machinery, New York, NY, USA, 199–211. https://doi.org/10.1145/3311350.3347190

Digital Library

[25]

Gian-Luca Savino. 2020. Virtual Smartphone: High Fidelity Interaction with Proxy Objects in Virtual Reality. arXiv preprint arXiv:2010.00942(2020).

[26]

Oliver Schneider, Bruno Fruchard, Dennis Wittchen, Bibhushan Raj Joshi, Georg Freitag, Donald Degraen, and Paul Strohmeier. 2022. Sustainable Haptic Design: Improving Collaboration, Sharing, and Reuse in Haptic Design Research. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/3491101.3503734

Digital Library

[27]

Oliver Schneider, Karon MacLean, Colin Swindells, and Kellogg Booth. 2017. Haptic experience design: What hapticians do and where they need help. International Journal of Human-Computer Studies 107 (2017), 5–21. https://doi.org/10.1016/j.ijhcs.2017.04.004 Multisensory Human-Computer Interaction.

Digital Library

[28]

Oliver S. Schneider and Karon E. MacLean. 2014. Improvising design with a Haptic Instrument. In 2014 IEEE Haptics Symposium (HAPTICS). 327–332. https://doi.org/10.1109/HAPTICS.2014.6775476

[29]

Marco Speicher, Jan Ehrlich, Vito Gentile, Donald Degraen, Salvatore Sorce, and Antonio Krüger. 2019. Pseudo-Haptic Controls for Mid-Air Finger-Based Menu Interaction. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3312927

Digital Library

[30]

Anthony Steed and Simon Julier. 2013. Design and implementation of an immersive virtual reality system based on a smartphone platform. In 2013 IEEE Symposium on 3D User Interfaces (3DUI). 43–46. https://doi.org/10.1109/3DUI.2013.6550195

[31]

Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Bjoern Hartmann. 2020. TransceiVR: Bridging Asymmetrical Communication Between VR Users and External Collaborators. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 182–195. https://doi.org/10.1145/3379337.3415827

Digital Library

[32]

Kashyap Todi, Donald Degraen, Brent Berghmans, Axel Faes, Matthijs Kaminski, and Kris Luyten. 2016. Purpose-Centric Appropriation of Everyday Objects as Game Controllers. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems(San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 2744–2750. https://doi.org/10.1145/2851581.2892448

Digital Library

[33]

Curtis B. Wilkes, Dan Tilden, and Doug A. Bowman. 2012. 3D User Interfaces Using Tracked Multi-touch Mobile Devices. In Joint Virtual Reality Conference of ICAT - EGVE - EuroVR, Ronan Boulic, Carolina Cruz-Neira, Kiyoshi Kiyokawa, and David Roberts (Eds.). The Eurographics Association. https://doi.org/10.2312/EGVE/JVRC12/065-072

[34]

André Zenner, Donald Degraen, and Antonio Krüger. 2019. Addressing Bystander Exclusion in Shared Spaces During Immersive Virtual Experiences. In Proceedings of the 1st Workshop on Challenges Using Head-Mounted Displays in Shared and Social Spaces (socialHMD ’19).

[35]

André Zenner, Marco Speicher, Sören Klingner, Donald Degraen, Florian Daiber, and Antonio Krüger. 2018. Immersive Notification Framework: Adaptive & Plausible Notifications in Virtual Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3188505

Digital Library

[36]

Li Zhang, Huidong Bai, Mark Billinghurst, and Weiping He. 2020. Is This My Phone? Operating a Physical Smartphone in Virtual Reality. In SIGGRAPH Asia 2020 XR(Virtual Event, Republic of Korea) (SA ’20). Association for Computing Machinery, New York, NY, USA, Article 12, 2 pages. https://doi.org/10.1145/3415256.3421499

Cited By

Degraen DSpeicher MZenner AKrüger A(2024)Exploring User-Defined Interactions for Virtual RealityProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688882(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688882
Greenslade MClark ALukosch S(2024)Becoming Q: Using Design Workshops to Explore Everyday Objects as Interaction Devices for an Augmented Reality Spy GameProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682096(1-11)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682096
Zhu FSousa MSidenmark LGrossman T(2024)PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642582(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642582
Show More Cited By

Recommendations

A Preliminary Study on Full-Body Haptic Stimulation on Modulating Self-motion Perception in Virtual Reality
Augmented Reality, Virtual Reality, and Computer Graphics
Abstract
We introduce a novel experimental system to explore the role of vibrotactile haptic feedback in Virtual Reality (VR) to induce the self-motion illusion. Self-motion (also called vection) has been mostly studied through visual and auditory stimuli ...
Evaluation of haptic virtual reality user interfaces for medical marking on 3D models
Highlights
- Two haptic VR interfaces are proposed to interact with volumetric medical images for improving current computer-aided medical diagnosis and planning.
- Compared with the traditional 2D interface using a screen-based 2D display and a ...
Abstract
Three-dimensional (3D) visualization has been widely used in computer-aided medical diagnosis and planning. To interact with 3D models, current user interfaces in medical systems mainly rely on the traditional 2D interaction techniques by ...
Envisioning Haptic Design for Immersive Virtual Environments
DIS' 20 Companion: Companion Publication of the 2020 ACM Designing Interactive Systems Conference

Current techniques for haptics in immersive virtual environments (IVEs) allow users to perceive materials while exploring virtual surfaces. However, these experiences are usually restricted to the properties defined during the design phase of the IVE. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

3066 pages

ISBN:9781450391566

DOI:10.1145/3491101

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
331
Total Downloads

Downloads (Last 12 months)85
Downloads (Last 6 weeks)9

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Degraen DSpeicher MZenner AKrüger A(2024)Exploring User-Defined Interactions for Virtual RealityProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688882(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688882
Greenslade MClark ALukosch S(2024)Becoming Q: Using Design Workshops to Explore Everyday Objects as Interaction Devices for an Augmented Reality Spy GameProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682096(1-11)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682096
Zhu FSousa MSidenmark LGrossman T(2024)PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642582(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642582
Stellmacher CMathis FWeiss YLoerakker MWagener NSchöning J(2024)Exploring Mobile Devices as Haptic Interfaces for Mixed RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642176(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642176
Gomi RTakashima KOnishi YFujita KKitamura Y(2023)UbiSurface: A Robotic Touch Surface for Supporting Mid-air Planar Interactions in Room-Scale VRProceedings of the ACM on Human-Computer Interaction10.1145/36264797:ISS(376-397)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626479
Lee JJin TLee SLee SBae S(2023)Stereoscopic Viewing and Monoscopic Touching: Selecting Distant Objects in VR Through a Mobile DeviceProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606809(1-7)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606809
Zhu FLyu ZSousa MGrossman T(2022)Touching The Droid: Understanding and Improving Touch Precision With Mobile Devices in Virtual Reality2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00099(807-816)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00099

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents