Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3332165.3347945acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Context-Aware Online Adaptation of Mixed Reality Interfaces

Published: 17 October 2019 Publication History

Abstract

We present an optimization-based approach for Mixed Reality (MR) systems to automatically control when and where applications are shown, and how much information they display. Currently, content creators design applications, and users then manually adjust which applications are visible and how much information they show. This choice has to be adjusted every time users switch context, i.e., whenever they switch their task or environment. Since context switches happen many times a day, we believe that MR interfaces require automation to alleviate this problem. We propose a real-time approach to automate this process based on users' current cognitive load and knowledge about their task and environment. Our system adapts which applications are displayed, how much information they show, and where they are placed. We formulate this problem as a mix of rule-based decision making and combinatorial optimization which can be solved efficiently in real-time. We present a set of proof-of-concept applications showing that our approach is applicable in a wide range of scenarios. Finally, we show in a dual-task evaluation that our approach decreased secondary tasks interactions by 36%.

Supplementary Material

SRT File (ufp8699vfc.srt)
Video figure captions
MP4 File (ufp8699pv.mp4)
Preview video
MP4 File (ufp8699vf.mp4)
Supplemental video
MP4 File (p147-lindlbauer.mp4)

References

[1]
J.K. Aggarwal and Michaelsahngwon Ryoo. 2011. Human Activity Analysis: A Review. ACM Comput. Surv. 43, 3, Article 16 (April 2011), 43 pages. http://dx.doi.org/10.1145/1922649.1922653
[2]
Ronald Azuma and Chris Furmanski. 2003. Evaluating Label Placement for Augmented Reality View Management. In Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '03). IEEE Computer Society, Washington, DC, USA, 66--. http://dl.acm.org/citation.cfm?id=946248.946790
[3]
Brian P. Bailey and Shamsi T. Iqbal. 2008. Understanding Changes in Mental Workload During Execution of Goal-directed Tasks and Its Application for Interruption Management. ACM Trans. Comput.-Hum. Interact. 14, 4, Article 21 (Jan. 2008), 28 pages. http://dx.doi.org/10.1145/1314683.1314689
[4]
Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive Optimization of Menu Systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 331--342. http://dx.doi.org/10.1145/2501988.2502024
[5]
Michael Bartels and Sandra P. Marshall. 2012. Measuring Cognitive Workload Across Different Eye Tracking Hardware Platforms. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12). ACM, New York, NY, USA, 161--164. http://dx.doi.org/10.1145/2168556.2168582
[6]
Blaine Bell, Steven Feiner, and Tobias Höllerer. 2001. View Management for Virtual and Augmented Reality. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology (UIST '01). ACM, New York, NY, USA, 101--110. http://dx.doi.org/10.1145/502348.502363
[7]
John Brooke and others. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.
[8]
Anind K Dey. 2016. Context-Aware Computing. In Ubiquitous Computing Fundamentals. Chapman and Hall/CRC, 335--366.
[9]
Stephen DiVerdi, Tobias Höllerer, and Richard Schreyer. 2004. Level of Detail Interfaces. In Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '04). IEEE Computer Society, Washington, DC, USA, 300--301. http://dx.doi.org/10.1109/ISMAR.2004.38
[10]
Andrew T. Duchowski, Krzysztof Krejtz, Izabela Krejtz, Cezary Biele, Anna Niedzielska, Peter Kiefer, Martin Raubal, and Ioannis Giannopoulos. 2018. The Index of Pupillary Activity: Measuring Cognitive Load Vis-à-vis Task Difficulty with Pupil Oscillation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 282, 13 pages. http://dx.doi.org/10.1145/3173574.3173856
[11]
Anna Maria Feit. 2018. Assignment Problems for Optimizing Text Input. Ph.D. Dissertation. Aalto University. http://urn.fi/URN:ISBN:978--952--60--8016--1
[12]
Andreas Fender, Philipp Herholz, Marc Alexa, and Jörg Müller. 2018. OptiSpace: Automated Placement of Interactive 3D Projection Mapping Content. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 269, 11 pages. http://dx.doi.org/10.1145/3173574.3173843
[13]
Andreas Fender, David Lindlbauer, Philipp Herholz, Marc Alexa, and Jörg Müller. 2017. HeatSpace: Automatic Placement of Displays by Empirical Analysis of User Behavior. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 611--621. http://dx.doi.org/10.1145/3126594.3126621
[14]
Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 93--100. http://dx.doi.org/10.1145/964442.964461
[15]
Ran Gal, Lior Shapira, Eyal Ofek, and Pushmeet Kohli. 2014. FLARE: Fast layout for augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 207--212. http://dx.doi.org/10.1109/ISMAR.2014.6948429
[16]
Nehla Ghouaiel, Jean-Marc Cieutat, and Jean-Pierre Jessel. 2014. Adaptive Augmented Reality: Plasticity of Augmentations. In Proceedings of the 2014 Virtual Reality International Conference (VRIC '14). ACM, New York, NY, USA, Article 10, 4 pages. http://dx.doi.org/10.1145/2617841.2620695
[17]
John F. Golding. 2006. Predicting individual differences in motion sickness susceptibility by questionnaire. Personality and Individual Differences 41, 2 (2006), 237 -- 248. http://dx.doi.org/10.1016/j.paid.2006.01.012
[18]
Raphael Grasset, Tobias Langlotz, Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. 2012. Image-driven View Management for Augmented Reality Browsers. In Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (ISMAR '12). IEEE Computer Society, Washington, DC, USA, 177--186. http://dx.doi.org/10.1109/ISMAR.2012.6402555
[19]
Jens Grubert, Daniel Hamacher, Rüdiger Mecke, Irina Böckelmann, Lutz Schega, Anke Huckauf, Mario Urbina, Michael Schenk, Fabian Doil, and Johannes Tümler. 2010. Extended investigations of user-related issues in mobile industrial AR. In 2010 IEEE International Symposium on Mixed and Augmented Reality (ISMAR '10). IEEE, 229--230. http://dx.doi.org/10.1109/ISMAR.2010.5643581
[20]
Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. 2017. Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality. IEEE Transactions on Visualization and Computer Graphics 23, 6 (June 2017), 1706--1724. http://dx.doi.org/10.1109/TVCG.2016.2543720
[21]
Eija Haapalainen, SeungJun Kim, Jodi F. Forlizzi, and Anind K. Dey. 2010. Psycho-physiological Measures for Assessing Cognitive Load. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (UbiComp '10). ACM, New York, NY, USA, 301--310. http://dx.doi.org/10.1145/1864349.1864395
[22]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139 -- 183. http://dx.doi.org/10.1016/S0166--4115(08)62386--9
[23]
Jennifer A. Healey and Rosalind W. Picard. 2005. Detecting Stress During Real-world Driving Tasks Using Physiological Sensors. Trans. Intell. Transport. Sys. 6, 2 (June 2005), 156--166. http://dx.doi.org/10.1109/TITS.2005.848368
[24]
Eckhard H Hess and James M Polt. 1964. Pupil size in relation to mental activity during simple problem-solving. Science 143, 3611 (1964), 1190--1192.
[25]
Nina Hollender, Cristian Hofmann, Michael Deneke, and Bernhard Schmitz. 2010. Review: Integrating Cognitive Load Theory and Concepts of Human-computer Interaction. Comput. Hum. Behav. 26, 6 (Nov. 2010), 1278--1288. http://dx.doi.org/10.1016/j.chb.2010.05.031
[26]
Scott Hudson, James Fogarty, Christopher Atkeson, Daniel Avrahami, Jodi Forlizzi, Sara Kiesler, Johnny Lee, and Jie Yang. 2003. Predicting Human Interruptibility with Sensors: A Wizard of Oz Feasibility Study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 257--264. http://dx.doi.org/10.1145/642611.642657
[27]
Simon Julier, Yohan Baillot, Dennis Brown, and Marco Lanzagorta. 2002. Information Filtering for Mobile Augmented Reality. IEEE Comput. Graph. Appl. 22, 5 (Sept. 2002), 12--15. http://dx.doi.org/10.1109/MCG.2002.1028721
[28]
Andreas Karrenbauer and Antti Oulasvirta. 2014. Improvements to Keyboard Optimization with Integer Programming. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 621--626. http://dx.doi.org/10.1145/2642918.2647382
[29]
Henry Lieberman and Ted Selker. 2000. Out of context: Computer systems that adapt to, and learn from, context. IBM systems journal 39, 3.4 (2000), 617--632. http://dx.doi.org/10.1147/sj.393.0617
[30]
Jacob Boesen Madsen, Markus Tatzqern, Claus B. Madsen, Dieter Schmalstieg, and Denis Kalkofen. 2016. Temporal Coherence Strategies for Augmented Reality Labeling. IEEE Transactions on Visualization and Computer Graphics 22, 4 (April 2016), 1415--1423. http://dx.doi.org/10.1109/TVCG.2016.2518318
[31]
Sandra P Marshall. 2002. The index of cognitive activity: Measuring cognitive workload. In Proceedings of the IEEE 7th conference on Human Factors and Power Plants. IEEE. http://dx.doi.org/10.1109/HFPP.2002.1042860
[32]
Silvano Martello and Paolo Toth. 1990. Knapsack Problems: Algorithms and Computer Implementations. John Wiley & Sons, Inc., New York, NY, USA.
[33]
Vivian Genaro Motti and Jean Vanderdonckt. 2013. A computational framework for context-aware adaptation of user interfaces. In IEEE 7th International Conference on Research Challenges in Information Science (RCIS). IEEE, 1--12. http://dx.doi.org/10.1109/RCIS.2013.6577709
[34]
Jens Müller, Roman R"adle, and Harald Reiterer. 2016. Virtual Objects As Spatial Cues in Collaborative Mixed Reality Environments: How They Shape Communication Behavior and User Task Load. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1245--1249. http://dx.doi.org/10.1145/2858036.2858043
[35]
Nargess Nourbakhsh, Fang Chen, Yang Wang, and Rafael A. Calvo. 2017. Detecting Users' Cognitive Load by Galvanic Skin Response with Affective Interference. ACM Trans. Interact. Intell. Syst. 7, 3, Article 12 (Sept. 2017), 20 pages. http://doi.acm.org/10.1145/2960413
[36]
Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D. Wilson. 2016. SnapToReality: Aligning Augmented Reality to the Real World. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1233--1244. http://dx.doi.org/10.1145/2858036.2858250
[37]
Antti Oulasvirta. 2017. User Interface Design with Combinatorial Optimization. Computer 50, 1 (Jan. 2017), 40--47. http://dx.doi.org/10.1109/MC.2017.6
[38]
Antti Oulasvirta, Samuli De Pascale, Janin Koch, Thomas Langerak, Jussi Jokinen, Kashyap Todi, Markku Laine, Manoj Kristhombuge, Yuxi Zhu, Aliaksei Miniukovich, Gregorio Palmas, and Tino Weinkauf. 2018. Aalto Interface Metrics (AIM): A Service and Codebase for Computational GUI Evaluation. In The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings (UIST '18 Adjunct). ACM, New York, NY, USA, 16--19. http://dx.doi.org/10.1145/3266037.3266087
[39]
Antti Oulasvirta and Andreas Karrenbauer. 2018. Combinatorial Optimization for Interface Design . In Computational Interaction. Oxford University Press.
[40]
Seonwook Park, Christoph Gebhardt, Roman R"adle, Anna Maria Feit, Hana Vrzakova, Niraj Ramesh Dayama, Hui-Shyong Yeo, Clemens N. Klokmose, Aaron Quigley, Antti Oulasvirta, and Otmar Hilliges. 2018. AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 184, 14 pages. http://dx.doi.org/10.1145/3173574.3173758
[41]
Bastian Pfleging, Drea K. Fekety, Albrecht Schmidt, and Andrew L. Kun. 2016. A Model Relating Pupil Diameter to Mental Workload and Lighting Conditions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5776--5788. http://dx.doi.org/10.1145/2858036.2858117
[42]
Edward Rosten, Gerhard Reitmayr, and Tom Drummond. 2005. Real-Time Video Annotations for Augmented Reality. In Proceedings of the First International Conference on Advances in Visual Computing (ISVC'05). Springer-Verlag, Berlin, Heidelberg, 294--302. http://dx.doi.org/10.1007/11595755_36
[43]
Sayan Sarcar, Jussi PP Jokinen, Antti Oulasvirta, Zhenxin Wang, Chaklam Silpasuwanchai, and Xiangshi Ren. 2018. Ability-Based Optimization of Touchscreen Interactions. IEEE Pervasive Computing 17, 1 (2018), 15--26. http://dx.doi.org/10.1109/MPRV.2018.011591058
[44]
John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257--285.
[45]
Arthur Tang, Charles Owen, Frank Biocca, and Weimin Mou. 2003. Comparative Effectiveness of Augmented Reality in Object Assembly. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 73--80. http://dx.doi.org/10.1145/642611.642626
[46]
Markus Tatzgern, Valeria Orso, Denis Kalkofen, Giulio Jacucci, Luciano Gamberini, and Dieter Schmalstieg. 2016. Adaptive information density for augmented reality displays. In 2016 IEEE Virtual Reality (IEEE VR '16). 83--92. http://dx.doi.org/10.1109/VR.2016.7504691
[47]
Beste F. Yuksel, Kurt B. Oleson, Lane Harrison, Evan M. Peck, Daniel Afergan, Remco Chang, and Robert JK Jacob. 2016. Learn Piano with BACh: An Adaptive Learning Interface That Adjusts Task Difficulty Based on Brain State. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5372--5384. http://dx.doi.org/10.1145/2858036.2858388

Cited By

View all
  • (2024)Survey of visualization methods for multiscene visual cue information in immersive environmentsJournal of Image and Graphics10.11834/jig.22114729:1(1-21)Online publication date: 2024
  • (2024)Practical approaches to group-level multi-objective Bayesian optimization in interaction technique designCollective Intelligence10.1177/263391372412413133:1Online publication date: 1-Jan-2024
  • (2024)Goldilocks Zoning: Evaluating a Gaze-Aware Approach to Task-Agnostic VR Notification PlacementProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682087(1-12)Online publication date: 7-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology
October 2019
1229 pages
ISBN:9781450368162
DOI:10.1145/3332165
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. context-awareness
  2. mixed reality
  3. ui optimization

Qualifiers

  • Research-article

Funding Sources

Conference

UIST '19

Acceptance Rates

Overall Acceptance Rate 842 of 3,967 submissions, 21%

Upcoming Conference

UIST '24

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)812
  • Downloads (Last 6 weeks)60
Reflects downloads up to 13 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Survey of visualization methods for multiscene visual cue information in immersive environmentsJournal of Image and Graphics10.11834/jig.22114729:1(1-21)Online publication date: 2024
  • (2024)Practical approaches to group-level multi-objective Bayesian optimization in interaction technique designCollective Intelligence10.1177/263391372412413133:1Online publication date: 1-Jan-2024
  • (2024)Goldilocks Zoning: Evaluating a Gaze-Aware Approach to Task-Agnostic VR Notification PlacementProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682087(1-12)Online publication date: 7-Oct-2024
  • (2024)MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIsProceedings of the ACM on Human-Computer Interaction10.1145/36611478:EICS(1-27)Online publication date: 17-Jun-2024
  • (2024)Shifting Focus with HCEye: Exploring the Dynamics of Visual Highlighting and Cognitive Load on User Attention and Saliency PredictionProceedings of the ACM on Human-Computer Interaction10.1145/36556108:ETRA(1-18)Online publication date: 28-May-2024
  • (2024)GraspUI: Seamlessly Integrating Object-Centric Gestures within the Seven Phases of GraspingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661551(1275-1289)Online publication date: 1-Jul-2024
  • (2024)Fidgets: Building Blocks for a Predictive UI ToolkitProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660731(1290-1305)Online publication date: 1-Jul-2024
  • (2024)FrameKit: A Tool for Authoring Adaptive UIs Using KeyframesProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645176(660-674)Online publication date: 18-Mar-2024
  • (2024)Adaptive 3D UI Placement in Mixed Reality Using Deep Reinforcement LearningExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651059(1-7)Online publication date: 11-May-2024
  • (2024)AdaptiveVoice: Cognitively Adaptive Voice Interface for Driving AssistanceProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642876(1-18)Online publication date: 11-May-2024
  • Show More Cited By

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media