research-article

Scenariot: Spatially Mapping Smart Things Within Augmented Reality Scenes

Authors:

Karthik RamaniAuthors Info & Claims

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

Paper No.: 219, Pages 1 - 13

https://doi.org/10.1145/3173574.3173793

Published: 21 April 2018 Publication History

Abstract

The emerging simultaneous localizing and mapping (SLAM) based tracking technique allows the mobile AR device spatial awareness of the physical world. Still, smart things are not fully supported with the spatial awareness in AR. Therefore, we present Scenariot, a method that enables instant discovery and localization of the surrounding smart things while also spatially registering them with a SLAM based mobile AR system. By exploiting the spatial relationships between mobile AR systems and smart things, Scenariot fosters in-situ interactions with connected devices. We embed Ultra-Wide Band (UWB) RF units into the AR device and the controllers of the smart things, which allows for measuring the distances between them. With a one-time initial calibration, users localize multiple IoT devices and map them within the AR scenes. Through a series of experiments and evaluations, we validate the localization accuracy as well as the performance of the enabled spatial aware interactions. Further, we demonstrate various use cases through Scenariot.

Supplementary Material

ZIP File (pn2347.zip)

Caption File: scenariot_preview_video.srt

Download
.90 KB

suppl.mov (pn2347-file5.mp4)

Supplemental video

Download
12.93 MB

suppl.mov (pn2347.mp4)

Supplemental video

Download
58.40 MB

References

[1]

Abdulrahman Alarifi, AbdulMalik Al-Salman, Mansour Alsaleh, Ahmad Alnafessah, Suheer Al-Hadhrami, Mai A Al-Ammar, and Hend S Al-Khalifa. 2016. Ultra wideband indoor positioning technologies: Analysis and recent advances. Sensors 16, 5 (2016), 707.

[2]

Isaac Amundson and Xenofon Koutsoukos. 2009. A survey on localization for mobile wireless sensor networks. Mobile entity localization and tracking in GPS-less environnments (2009), 235--254.

Digital Library

[3]

Ferran Argelaguet and Carlos Andujar. 2009. Visual feedback techniques for virtual pointing on stereoscopic displays. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology. ACM, 163--170.

Digital Library

[4]

ASUS. 2017. ZenFone-AR. (2017). Retrieved September 1, 2017 from = https://www.asus.com/us/Phone/ZenFoneAR-ZS571KL/.

[5]

Shuanghua Bai and Houduo Qi. 2016. Tackling the flip ambiguity in wireless sensor network localization and beyond. Digital Signal Processing 55 (2016), 85--97.

Digital Library

[6]

Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic interaction: designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces. ACM, 121--130.

Digital Library

[7]

Ingwer Borg and Patrick JF Groenen. 2005. Modern multidimensional scaling: Theory and applications. Springer Science & Business Media.

[8]

Sebastian Boring, Dominikus Baur, Andreas Butz, Sean Gustafson, and Patrick Baudisch. 2010. Touch projector: mobile interaction through video. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2287--2296.

Digital Library

[9]

Barry Brumitt, John Krumm, Brian Meyers, and Steven Shafer. 2000. Ubiquitous computing and the role of geometry. IEEE Personal Communications 7, 5 (2000), 41--43.

[10]

Yu-Hsiang Chen, Ben Zhang, Claire Tuna, Yang Li, Edward A Lee, and Bjorn Hartmann. 2013. A context menu for the real world: Controlling physical appliances through head-worn infrared targeting. Technical Report. CALIFORNIA UNIV BERKELEY DEPT OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCES.

[11]

Jose A Costa, Neal Patwari, and Alfred O Hero III. 2006. Distributed weighted-multidimensional scaling for node localization in sensor networks. ACM Transactions on Sensor Networks (TOSN) 2, 1 (2006), 39--64.

Digital Library

[12]

Adrian A de Freitas, Michael Nebeling, Xiang'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K Dey. 2016. Snap-to-it: A user-inspired platform for opportunistic device interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5909--5920.

Digital Library

[13]

Jan De Leeuw and Patrick Mair. 2011. Multidimensional scaling using majorization: SMACOF in R. Department of Statistics, UCLA (2011).

[14]

Carmelo Di Franco, Enrico Bini, Mauro Marinoni, and Giorgio C Buttazzo. 2017a. Multidimensional scaling localization with anchors. In Autonomous Robot Systems and Competitions (ICARSC), 2017 IEEE International Conference on. IEEE, 49--54.

[15]

Carmelo Di Franco, Amanda Prorok, Nikolay Atanasov, Benjamin P Kempke, Prabal Dutta, Vijay Kumar, and George J Pappas. 2017b. Calibration-free network localization using non-line-of-sight ultra-wideband measurements. In IPSN. 235--246.

Digital Library

[16]

Ivan Dokmanic, Reza Parhizkar, Juri Ranieri, and Martin Vetterli. 2015. Euclidean distance matrices: Essential theory, algorithms, and applications. IEEE Signal Processing Magazine 32, 6 (2015), 12--30.

[17]

Hans Gellersen, Carl Fischer, Dominique Guinard, Roswitha Gostner, Gerd Kortuem, Christian Kray, Enrico Rukzio, and Sara Streng. 2009. Supporting device discovery and spontaneous interaction with spatial references. Personal and Ubiquitous Computing 13, 4 (2009), 255--264.

Digital Library

[18]

Google. 2017. ARCore. (2017). Retrieved September 1, 2017 from = https://developers.google.com/ar/.

[19]

Uwe Gruenefeld, Abdallah El Ali, Wilko Heuten, and Susanne Boll. 2017. Visualizing out-of-view objects in head-mounted augmented reality. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 81.

Digital Library

[20]

Anhong Guo, Jeeeun Kim, Xiang'Anthony' Chen, Tom Yeh, Scott E Hudson, Jennifer Mankoff, and Jeffrey P Bigham. 2017. Facade: Auto-generating tactile interfaces to appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5826--5838.

Digital Library

[21]

Dirk Hahnel, Wolfram Burgard, Dieter Fox, Ken Fishkin, and Matthai Philipose. 2004. Mapping and localization with RFID technology. In Robotics and Automation, 2004. Proceedings. ICRA'04. 2004 IEEE International Conference on, Vol. 1. IEEE, 1015--1020.

[22]

Valentin Heun, James Hobin, and Pattie Maes. 2013. Reality editor: programming smarter objects. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 307--310.

Digital Library

[23]

Yi Jiang and Victor CM Leung. 2007. An asymmetric double sided two-way ranging for crystal offset. In Signals, Systems and Electronics, 2007. ISSSE'07. International Symposium on. IEEE, 525--528.

[24]

Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. Em-sense: Touch recognition of uninstrumented, electrical and electromechanical objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 157--166.

Digital Library

[25]

David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-aware controls: Designing remote controls for ubiquitous computing ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 187--198.

Digital Library

[26]

Sikun Lin, Hao Fei Cheng, Weikai Li, Zhanpeng Huang, Pan Hui, and Christoph Peylo. 2017a. Ubii: Physical World Interaction Through Augmented Reality. IEEE Transactions on Mobile Computing 16, 3 (2017), 872--885.

Digital Library

[27]

Yung-Ta Lin, Yi-Chi Liao, Shan-Yuan Teng, Yi-Ju Chung, Liwei Chan, and Bing-Yu Chen. 2017b. Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360 Video Using Spatial Picture-in-Picture Previews. In Proceedings of the 30th Annual Symposium on User Interface Software and Technology. ACM, --.

Digital Library

[28]

Simon Mayer, Markus Schalch, Marian George, and Gábor Sörös. 2013. Device recognition for intuitive interaction with the web of things. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, 239--242.

Digital Library

[29]

Microsoft. 2017. Hololens. (2017). Retrieved September 1, 2017 from = https://www.microsoft.com/en-us/hololens.

[30]

Gang Pan, Jiahui Wu, Daqing Zhang, Zhaohui Wu, Yingchun Yang, and Shijian Li. 2010. GeeAir: a universal multimodal remote control device for home appliances. Personal and Ubiquitous Computing 14, 8 (2010), 723--735.

Digital Library

[31]

Charith Perera, Arkady Zaslavsky, Peter Christen, and Dimitrios Georgakopoulos. 2014. Context aware computing for the internet of things: A survey. IEEE Communications Surveys & Tutorials 16, 1 (2014), 414--454.

[32]

Sudeep Pillai and John Leonard. 2015. Monocular slam supported object recognition. arXiv preprint arXiv:1506.01732 (2015).

[33]

Math.Net Project. 2017. Math.Net Numerics. (2017). Retrieved September 1, 2017 from = https://numerics.mathdotnet.com/.

[34]

Kun Qian, Chenshu Wu, Zimu Zhou, Yue Zheng, Zheng Yang, and Yunhao Liu. 2017. Inferring motion direction using commodity wi-fi for interactive exergames. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 1961--1972.

Digital Library

[35]

Jun Rekimoto, Yuji Ayatsuka, Michimune Kohno, and Haruo Oba. 2003. Proximal Interactions: A Direct Manipulation Technique for Wireless Networking. In Interact, Vol. 3. 511--518.

[36]

Enrico Rukzio, Karin Leichtenstern, Vic Callaghan, Paul Holleis, Albrecht Schmidt, and Jeannette Chin. 2006. An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. UbiComp 2006: Ubiquitous Computing (2006), 87--104.

Digital Library

[37]

Spencer Russell, Gershon Dublon, and Joseph A Paradiso. 2016. Hearthere: Networked sensory prosthetics through auditory augmented reality. In Proceedings of the 7th Augmented Human International Conference 2016. ACM, 20.

Digital Library

[38]

Renato F Salas-Moreno, Richard A Newcombe, Hauke Strasdat, Paul HJ Kelly, and Andrew J Davison. 2013. Slam++: Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1352--1359.

Digital Library

[39]

Dominik Schmidt, David Molyneaux, and Xiang Cao. 2012. PICOntrol: using a handheld projector for direct control of physical devices through visible light. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 379--388.

Digital Library

[40]

Eldon Schoop, Michelle Nguyen, Daniel Lim, Valkyrie Savage, Sean Follmer, and Björn Hartmann. 2016. Drill Sergeant: Supporting physical construction projects through an ecosystem of augmented tools. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1607--1614.

Digital Library

[41]

Unity3D. 2017. Unity3D. (2017). Retrieved September 1, 2017 from = https://unity3d.com/.

[42]

Pasi Välkkynen and Timo Tuomisto. 2005. Physical Browsing Research. PERMID 2005 (2005), 35--38.

[43]

S Venkatesh and RM Buehrer. 2007. Non-line-of-sight identification in ultra-wideband systems based on received signal statistics. IET Microwaves, Antennas & Propagation 1, 6 (2007), 1120--1130.

[44]

Edward J Wang, Tien-Jui Lee, Alex Mariakakis, Mayank Goel, Sidhant Gupta, and Shwetak N Patel. 2015. Magnifisense: Inferring device interaction using wrist-worn passive magneto-inductive sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 15--26.

Digital Library

[45]

Roy Want, Kenneth P Fishkin, Anuj Gujar, and Beverly L Harrison. 1999. Bridging physical and virtual worlds with electronic tags. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 370--377.

Digital Library

[46]

wikitude. 2017. wikitude. (2017). Retrieved September 1, 2017 from = https://www.wikitude.com/.

[47]

Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4000--4008.

Digital Library

[48]

Sang Ho Yoon, Yunbo Zhang, Ke Huo, and Karthik Ramani. 2016. TRing: Instant and Customizable Interactions with Objects Using an Embedded Magnet and a Finger-Worn Device. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 169--181.

Digital Library

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Qian XWang TXu XJonker TTodi K(2024)Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642158(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642158
Xi MPerera MAnderson SAdcock M(2024)Towards Situated Imaging2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00019(85-89)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00019
Show More Cited By

Index Terms

Scenariot: Spatially Mapping Smart Things Within Augmented Reality Scenes
1. Human-centered computing
  1. Human computer interaction (HCI)
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

A magic lens for revealing device interactions in smart environments
SA '14: SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications

Keeping track of device interactions in smart environments is a challenging task for everyday users. Given the expected high number of communicating devices in future smart homes, it will become increasingly important to put users more in control of ...
Ubiquitous home simulation using augmented reality
CEA'07: Proceedings of the 2007 annual Conference on International Conference on Computer Engineering and Applications

Computing paradigm is moving toward ubiquitous computing in which devices, software agents, and services are all expected to seamlessly integrate and cooperate in support of human objectives. Augmented reality (AR) can naturally complement ubiquitous ...
A smart environment for augmented learning through physical books
ICME '11: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo

Today, augmented reality is evolving towards sophisticated approaches exploiting the opportunities offered by immersive environments and high quality 3D graphics. Such systems can prove to be very useful in the context of education, especially when ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

April 2018

8489 pages

ISBN:9781450356206

DOI:10.1145/3173574

General Chairs:
Regan Mandryk
University of Saskatchewan, Canada
,
Mark Hancock
University of Waterloo, Canada
,
Program Chairs:
Mark Perry
Brunel University London, UK
,
Anna Cox
University College London, UK

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF CMMI
NSF IIP

Conference

CHI '18

Sponsor:

SIGCHI

CHI '18: CHI Conference on Human Factors in Computing Systems

April 21 - 26, 2018

Montreal QC, Canada

Acceptance Rates

CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
1,229
Total Downloads

Downloads (Last 12 months)101
Downloads (Last 6 weeks)8

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Qian XWang TXu XJonker TTodi K(2024)Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642158(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642158
Xi MPerera MAnderson SAdcock M(2024)Towards Situated Imaging2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)10.1109/AIxVR59861.2024.00019(85-89)Online publication date: 17-Jan-2024
https://doi.org/10.1109/AIxVR59861.2024.00019
Galvão MFogliaroni PGiannopoulos INavratil GKattenbeck MAlinaghi N(2024)GeoAR: a calibration method for Geographic-Aware Augmented RealityInternational Journal of Geographical Information Science10.1080/13658816.2024.235532638:9(1800-1826)Online publication date: 20-May-2024
https://doi.org/10.1080/13658816.2024.2355326
Zhu ZLiu ZZhang YZhu LHuang JVillanueva AQian XPeppler KRamani K(2023)LearnIoTVR: An End-to-End Virtual Reality Environment Providing Authentic Learning Experiences for Internet of ThingsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581396(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581396
Li RSeyed TMarquardt NOfek EHodges SSinclair MRomat HPahud MSharma JBuxton WHinckley KRiche N(2023)AdHocProx: Sensing Mobile, Ad-Hoc Collaborative Device Formations using Dual Ultra-Wideband RadiosProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581300(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581300
Ye HLeng JXiao CWang LFu H(2023)ProObjAR: Prototyping Spatially-aware Interactions of Smart Objects with AR-HMDProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580750(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580750
Xiao FZhang STang SShen SDong HZhong Y(2023)WiSion: Bolstering MAV 3D Indoor State Estimation by Embracing Multipath of WiFiIEEE Transactions on Vehicular Technology10.1109/TVT.2022.320571072:1(253-266)Online publication date: Jan-2023
https://doi.org/10.1109/TVT.2022.3205710
Fleck PCalepso AHubenschmid SSedlmair MSchmalstieg D(2023)RagRug: A Toolkit for Situated AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.315705829:7(3281-3297)Online publication date: 1-Jul-2023
https://doi.org/10.1109/TVCG.2022.3157058
Wang QZhang QSun WBoulay CKim KBarmaki R(2023)A scoping review of the use of lab streaming layer framework in virtual and augmented reality researchVirtual Reality10.1007/s10055-023-00799-827:3(2195-2210)Online publication date: 2-May-2023
https://doi.org/10.1007/s10055-023-00799-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten