research-article

Open access

ARticulate: One-Shot Interactions with Intelligent Assistants in Unfamiliar Smart Spaces Using Augmented Reality

Authors:

Mark W. Newman,

Prabal DuttaAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 6, Issue 1

Article No.: 7, Pages 1 - 24

https://doi.org/10.1145/3517235

Published: 29 March 2022 Publication History

Abstract

Smart space technologies have entered the mainstream home market. Most users currently interact with smart homes that they (or an acquaintance) have set up and know well. However, as these technologies spread to commercial or public environments, users will need to frequently interact with unfamiliar smart spaces where they are unaware of the available capabilities and the system maintainer will not be present to help. Users will need to quickly and independently 1) discover what is and is not possible, and 2) make use of available functionality. Widespread adoption of smart space systems will not be possible until this discoverability issue is solved. We design and evaluate ARticulate, an interface that allows users to have successful smart space interactions with an intelligent assistant while learning transferable information about the overall set of devices in an unfamiliar space. Our method of using Snapchat-like contextual photo messages enhanced by two technologies---augmented reality and autocomplete---allows users to determine available functionality and achieve their goals in one attempt with a smart space they have never seen before, something no existing interface supports. The ability to easily operate unfamiliar smart spaces improves the usability of existing systems and removes a significant obstacle to the vision of ubiquitous computing.

References

[1]

Amazon.com, Inc. 2021. Echo Dot (3rd Gen) - Smart speaker with Alexa - Charcoal. https://web.archive.org/web/20210805031908/https://www.amazon.com/Echo-Dot/dp/B07FZ8S74R Accessed: 2021--08--10.

[2]

Joseph B Bayer, Nicole B Ellison, Sarita Y Schoenebeck, and Emily B Falk. 2016. Sharing the small moments: Ephemeral social interaction on Snapchat. Information, Communication & Society 19, 7 (2016), 956--977.

[3]

James H Bradford. 1995. The human factors of speech-based interfaces: a research agenda. ACM SIGCHI Bulletin 27, 2 (1995), 61--67.

Digital Library

[4]

John Brooke. 1996. SUS: A quick and dirty usability scale. Usability Evaluation in Industry 189 (1996).

[5]

Barry Brumitt and Jonathan J Cadiz. 2001. "Let there be light": Examining interfaces for homes of the future. In Human Computer Interaction. INTERACT'01. IFIP TC. 13 International Conference on Human Computer Interaction. IOS Press, Amsterdam, Netherlands; 2001; xxvii+ 897 pp. 375--82.

[6]

Fei Cai and Maarten De Rijke. 2016. A survey of query auto completion in information retrieval. Foundations and Trends in Information Retrieval 10, 4 (2016), 273--363.

Digital Library

[7]

Pew Research Center. 2021. Social Media Use in 2021. https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021 Accessed: 2021--08--10.

[8]

Fang Chen. 2006. Designing human interface in speech technology. Springer Science & Business Media.

[9]

Meghan Clark, Mark W. Newman, and Prabal Dutta. 2017. Devices and data and agents, oh my: How smart home abstractions prime end-User mental models. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3, Article 44 (Sept. 2017), 44:1--44:26 pages.

Digital Library

[10]

Eric Corbett and Astrid Weber. 2016. What can I say? Addressing user experience challenges of a mobile voice user interface for accessibility. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services. 72--82.

Digital Library

[11]

Adrian A de Freitas, Michael Nebeling, Xiang 'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K Dey. 2016. Snap-to-it: A user-inspired platform for opportunistic device interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 5909--5920.

Digital Library

[12]

Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. 1997. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies 1, 4 (1997), 208--217.

[13]

George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. 1987. The vocabulary problem in human-system communication. Commun. ACM 30, 11 (1987), 964--971.

Digital Library

[14]

Christine Geeng and Franziska Roesner. 2019. Who's in control? Interactions in multi-user smart homes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.

Digital Library

[15]

Google. 2019. Google Assistant. https://assistant.google.com Accessed: 2019--02--14.

[16]

Google. 2019. Google Home. https://store.google.com/us/product/google_home Accessed: 2019--09--19.

[17]

Google. 2019. How Google autocomplete works in Search. https://www.blog.google/products/search/how-google-autocomplete-works-search Accessed: 2019--02--15.

[18]

Korinna Grabski and Tobias Scheffer. 2004. Sentence completion. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 433--439.

Digital Library

[19]

Amy He. 2020. Amazon maintains convincing lead in US smart speaker market. https://www.emarketer.com/content/amazon-maintains-convincing-lead-in-us-smart-speaker-market Accessed: 2021--07--06.

[20]

Danny Yuxing Huang, Noah Apthorpe, Frank Li, Gunes Acar, and Nick Feamster. 2020. Iot inspector: Crowdsourcing labeled network traffic from smart home devices at scale. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1--21.

Digital Library

[21]

Candace A Kamm, Diane J Litman, and Marilyn A Walker. 1998. From novice to expert: The effect of tutorials on user expertise with spoken dialogue systems. In Fifth International Conference on Spoken Language Processing. Citeseer.

[22]

Demetrios Karis and Kathryn M. Dobroth. 1991. Automating services with speech recognition over the public switched telephone network: Human factors considerations. IEEE Journal on Selected Areas in Communications 9, 4 (1991), 574--585.

Digital Library

[23]

Laurent Karsenty. 2002. Shifting the design philosophy of spoken natural language dialogue: From invisible to transparent systems. International Journal of Speech Technology 5, 2 (2002), 147--157.

[24]

Jette Kofoed and Malene Charlotte Larsen. 2016. A snap of intimacy: Photo-sharing practices among young people on social media. First Monday (2016).

[25]

Vinay Koshy, Joon Sung Sung Park, Ti-Chung Cheng, and Karrie Karahalios. 2021. "We just use what they give us": Understanding passenger user perspectives in smart homes. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--14.

Digital Library

[26]

Josephine Lau, Benjamin Zimmerman, and Florian Schaub. 2018. Alexa, are you listening? Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1--31.

Digital Library

[27]

Yubing Li, Kun Zhao, Meichen Duan, Wei Shi, Liangliang Lin, Xinyi Cao, Yang Liu, and Jizhong Zhao. 2020. Control Your Home With a Smartwatch. IEEE Access 8 (2020), 131601--131613.

[28]

Axel Liljencrantz. 2005. Friendly Interactive Shell. https://fishshell.com Accessed: 2019--09--20.

[29]

Sven Mayer, Gierad Laput, and Chris Harrison. 2020. Enhancing mobile voice assistants with WorldGaze. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--10.

Digital Library

[30]

Sarah Mennicken and Elaine M Huang. 2012. Hacking the natural habitat: An in-the-wild study of smart homes, their development, and the people who live in them. In International Conference on Pervasive Computing. Springer, 143--160.

Digital Library

[31]

Robert C Miller, Victoria H Chou, Michael Bernstein, Greg Little, Max Van Kleek, David Karger, and MC Schraefel. 2008. Inky: A sloppy command line for the web with rich visual feedback. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology. 131--140.

Digital Library

[32]

Arnab Nandi and HV Jagadish. 2011. Guided interaction: Rethinking the query-result paradigm. Proceedings of the VLDB Endowment 4, 12 (2011), 1466--1469.

Digital Library

[33]

Robert Neßelrath, Chensheng Lu, Christian H Schulz, Jochen Frey, and Jan Alexandersson. 2011. A gesture based system for context-sensitive interaction with smart homes. In Ambient Assisted Living. Springer, 209--219.

[34]

Jenni Niemelä-Nyrhinen and Janne Seppänen. 2020. Visual communion: The photographic image as phatic communication. New Media & Society 22, 6 (2020), 1043--1057.

[35]

Sophie Nyborg. 2015. Pilot users and their families: Inventing flexible practices in the smart grid. Science & Technology Studies (2015).

[36]

Debajyoti Pal, Tuul Triyason, and Suree Funikul. 2017. Smart homes and quality of life for the elderly: a systematic review. In 2017 IEEE International Symposium on Multimedia (ISM). IEEE, 413--419.

[37]

François Portet, Michel Vacher, Caroline Golanski, Camille Roux, and Brigitte Meillon. 2013. Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects. Personal and Ubiquitous Computing 17, 1 (2013), 127--144.

Digital Library

[38]

Jun Rekimoto and Katashi Nagao. 1995. The world through the computer: Computer-augmented interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on User Interface and Software Technology. 29--36.

Digital Library

[39]

Franziska Roesner, Brian T Gill, and Tadayoshi Kohno. 2014. Sex, lies, or kittens? Investigating the use of Snapchat's self-destructing messages. In International Conference on Financial Cryptography and Data Security. Springer, 64--76.

[40]

Eric Rose, David Breen, Klaus H Ahlers, Chris Crampton, Mihran Tuceryan, Ross Whitaker, and Douglas Greer. 1995. Annotating real-world objects using augmented reality. In Computer Graphics. Elsevier, 357--370.

[41]

Frode Eika Sandnes, Jo Herstad, Andrea Marie Stangeland, and Fausto Orsi Medola. 2017. UbiWheel: a simple context-aware universal control concept for smart home appliances that encourages active living. In 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE, 1--6.

[42]

SmartThings, Inc. 2021. SmartThings. https://web.archive.org/web/20210727135832/https://www.smartthings.com/ Accessed: 2021--08--10.

[43]

Snap, Inc. 2021. SEC Quarterly Report Form 10-Q. https://investor.snap.com/financials/sec-filings/default.aspx Accessed: 2021--08--10.

[44]

Yang Song, Mingyang Guan, Wee Peng Tay, Choi Look Law, and Changyun Wen. 2019. UWB/LiDAR Fusion For Cooperative Range-Only SLAM. In 2019 International Conference on Robotics and Automation (ICRA). 6568--6574.

[45]

Ivan E Sutherland. 1968. A head-mounted three dimensional display. In Proceedings of the December 9--11, 1968, Fall Joint Computer Conference, Part I. 757--764.

Digital Library

[46]

Arto Vihavainen, Juha Helminen, and Petri Ihantola. 2014. How novices tackle their first lines of code in an IDE: Analysis of programming session traces. In Proceedings of the 14th Koli Calling International Conference on Computing Education Research. ACM.

Digital Library

[47]

Ryen W White. 2018. Skill discovery in virtual assistants. Commun. ACM 61, 11 (2018), 106--113.

Digital Library

[48]

Ryen W White and Gary Marchionini. 2007. Examining the effectiveness of real-time query expansion. Information Processing & Management 43, 3 (2007), 685--704.

Digital Library

[49]

Andrew Wilson and Hubert Pham. 2003. Pointing in intelligent environments with the WorldCursor. In Interact. Citeseer.

[50]

Jason Wither, Stephen DiVerdi, and Tobias Höllerer. 2009. Annotation in outdoor augmented reality. Computers & Graphics 33, 6 (2009), 679--689.

Digital Library

[51]

Jong-bum Woo and Youn-kyung Lim. 2015. User experience in do-it-yourself-style smart homes. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 779--790.

[52]

Huiyue Wu, Shaoke Zhang, Jiayi Liu, Jiali Qiu, and Xiaolong Zhang. 2019. The gesture disagreement problem in free-hand gesture interaction. International Journal of Human-Computer Interaction 35, 12(2019), 1102--1114.

[53]

Nicole Yankelovich. 1996. How do users know what to say? Interactions 3, 6 (1996), 32--43.

Digital Library

[54]

Elizabeth Zoltan-Ford. 1991. How to get people to say and type what computers can understand. International Journal of Man-Machine Studies 34, 4 (1991), 527--547.

Digital Library

Cited By

Qian XWang TXu XJonker TTodi K(2024)Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended RealityProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642158(1-17)Online publication date: 11-May-2024
https://doi.org/10.1145/3613904.3642158
Zheng WXue Y(2024)Clinical Nursing Intelligent Interaction System Based on Virtual Augmented Reality Technology2024 IEEE 2nd International Conference on Image Processing and Computer Applications (ICIPCA)10.1109/ICIPCA61593.2024.10709300(1757-1761)Online publication date: 28-Jun-2024
https://doi.org/10.1109/ICIPCA61593.2024.10709300
Shvedenko VShvedenko V(2024)Ensuring a harmonious state of smart space when there is a conflict of interest of its elementsSmart Spaces10.1016/B978-0-443-13462-3.00005-4(235-255)Online publication date: 2024
https://doi.org/10.1016/B978-0-443-13462-3.00005-4
Show More Cited By

Index Terms

ARticulate: One-Shot Interactions with Intelligent Assistants in Unfamiliar Smart Spaces Using Augmented Reality
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies

Recommendations

Using Augmented Reality and Internet of Things to improve accessibility of people with motor disabilities in the context of Smart Cities

Smart Cities need to be designed to allow the inclusion of all kinds of citizens. For instance, motor disabled people like wheelchair users may have problems to interact with the city. Internet of Things (IoT) technologies provide the tools to include ...
A Personal Smart Space approach to realising Ambient Ecologies

One of the main aims of research in the area of ubiquitous or pervasive computing is that of dealing with the rapidly growing number of sensors, devices and artefacts in the user's environment, which have an increasing amount of intelligence built in ...
Cerberus: A Context-Aware Security Scheme for Smart Spaces
PERCOM '03: Proceedings of the First IEEE International Conference on Pervasive Computing and Communications

Ubiquitous computing has fueled the idea of constructing sentient, information-rich "smart spaces" that extend the boundaries of traditional computing to encompass physical spaces, embedded devices, sensors, and other machinery. To achieve this, smart ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 6, Issue 1

March 2022

1009 pages

EISSN:2474-9567

DOI:10.1145/3529514

Issue’s Table of Contents

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 March 2022

Published in IMWUT Volume 6, Issue 1

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Semiconductor Research Corporation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
775
Total Downloads

Downloads (Last 12 months)217
Downloads (Last 6 weeks)18

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Qian XWang TXu XJonker TTodi K(2024)Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended RealityProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642158(1-17)Online publication date: 11-May-2024
https://doi.org/10.1145/3613904.3642158
Zheng WXue Y(2024)Clinical Nursing Intelligent Interaction System Based on Virtual Augmented Reality Technology2024 IEEE 2nd International Conference on Image Processing and Computer Applications (ICIPCA)10.1109/ICIPCA61593.2024.10709300(1757-1761)Online publication date: 28-Jun-2024
https://doi.org/10.1109/ICIPCA61593.2024.10709300
Shvedenko VShvedenko V(2024)Ensuring a harmonious state of smart space when there is a conflict of interest of its elementsSmart Spaces10.1016/B978-0-443-13462-3.00005-4(235-255)Online publication date: 2024
https://doi.org/10.1016/B978-0-443-13462-3.00005-4
Mattioli APaternò F(2023)A Mobile Augmented Reality App for Creating, Controlling, Recommending Automations in Smart HomesProceedings of the ACM on Human-Computer Interaction10.1145/36042427:MHCI(1-22)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604242
Minder BWolf PBaldauf MVerma S(2023)Voice assistants in private households: a conceptual framework for future research in an interdisciplinary fieldHumanities and Social Sciences Communications10.1057/s41599-023-01615-z10:1Online publication date: 19-Apr-2023
https://doi.org/10.1057/s41599-023-01615-z
Andrao MBalducci FBreve BCena FDesolda GDeufemia VGena CMatera MMattioli APaternò FSantoro CTreccani BVernero FZancanaro M(2023)Understanding Concepts, Methods and Tools for End-User Control of Automations in Ecosystems of Smart Objects and ServicesEnd-User Development10.1007/978-3-031-34433-6_7(104-124)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-34433-6_7

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents