research-article

ShadowClones: an Interface to Maintain a Multiple Sense of Body-space Coordination in Multiple Visual Perspectives

Authors:

Nanako Kumasaki,

Kazuhisa Shibata,

Shunichi KasaharaAuthors Info & Claims

AHs '23: Proceedings of the Augmented Humans International Conference 2023

Pages 171 - 178

https://doi.org/10.1145/3582700.3582706

Published: 14 March 2023 Publication History

Abstract

In this paper, we propose ShadowClones, an interface that supports interactions in which a single user can interact with multiple bodies in multiple spaces. Recent teleoperation technologies have allowed a user controlling multiple objects simultaneously, but at the same time, it also exhibited a significant challenge, which can be attributed to the high cognitive load caused by switching and recogning various spaces/perspectives repeatedly and instantly. To tackle this challenge, by taking advantage of pre-attentive visual cues for users’ simultaneous information processing, we designed and evaluated a new user interface, called Shadow Clones, that projects self-body information in unattended areas for increasing the awareness of body-space relationships and allowing users to seamlessly switch across different visual perspectives from avatars or remote robots. We then explored the proposed approach through a simple visual reaching task with a performance evaluation in terms of task completion time and success rate. The results showed superior performance when compared with a condition that presents no projections of users’ body movements in unattended areas. We conclude by discussing possible mechanisms of this enhancement as well as two potential scenarios using the shadow clones approach, including new entertainment content for virtual reality e-sports and multiple robot teleoperation such as in a construction site or a disaster site, without compromising operational performance.

References

[1]

Reem Alzahabi and Matthew S Cain. 2021. Ensemble perception during multiple-object tracking. Atten. Percept. Psychophys. 83, 3 (April 2021), 1263–1274.

[2]

A T Bahill, A Brockenbrough, and B T Troost. 1981. Variability and development of a normative data base for saccadic eye movements. Invest. Ophthalmol. Vis. Sci. 21, 1 Pt 1 (July 1981), 116–125.

[3]

Renaud Blanch and Michaël Ortega. 2009. Rake cursor: improving pointing performance with concurrent input channels. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 1415–1418.

Digital Library

[4]

Yuanzhi Cao, Tianyi Wang, Xun Qian, Pawan S Rao, Manav Wadhawan, Ke Huo, and Karthik Ramani. 2019. GhostAR: A Time-space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 521–534.

Digital Library

[5]

Li-Hung Chang, Kazuhisa Shibata, George J Andersen, Yuka Sasaki, and Takeo Watanabe. 2014. Age-related declines of stability in visual perceptual learning. Curr. Biol. 24, 24 (Dec. 2014), 2926–2929.

[6]

Linfeng Chen, Kazuki Takashima, Kazuyuki Fujita, and Yoshifumi Kitamura. 2021. PinpointFly: An Egocentric Position-control Drone Interface using Mobile AR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21, Article 150). Association for Computing Machinery, New York, NY, USA, 1–13.

Digital Library

[7]

H Collewijn, C J Erkelens, and R M Steinman. 1988. Binocular co-ordination of human horizontal saccadic eye movements. J. Physiol. 404 (Oct. 1988), 157–182.

[8]

Barbara Anne Dosher, Songmei Han, and Zhong-Lin Lu. 2010. Information-limited parallel processing in difficult heterogeneous covert visual search. J. Exp. Psychol. Hum. Percept. Perform. 36, 5 (Oct. 2010), 1128–1144.

[9]

John Duncan. 2010. The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour. Trends Cogn. Sci. 14, 4 (April 2010), 172–179.

[10]

Paul E Dux, Jason Ivanoff, Christopher L Asplund, and René Marois. 2006. Isolation of a central bottleneck of information processing with time-resolved FMRI. Neuron 52, 6 (Dec. 2006), 1109–1120.

[11]

Paul E Dux, Michael N Tombu, Stephenie Harrison, Baxter P Rogers, Frank Tong, and René Marois. 2009. Training improves multitasking performance by increasing the speed of information processing in human prefrontal cortex. Neuron 63, 1 (July 2009), 127–138.

[12]

Dylan F Glas, Takayuki Kanda, Hiroshi Ishiguro, and Norihiro Hagita. 2012. Teleoperation of Multiple Social Robots. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 42, 3 (May 2012), 530–544.

Digital Library

[13]

Arvid Guterstam, Dennis E O Larsson, Joanna Szczotka, and H Henrik Ehrsson. 2020. Duplication of the bodily self: a perceptual illusion of dual full-body ownership and dual self-location. R Soc Open Sci 7, 12 (Dec. 2020), 201911.

[14]

Suk Won Han and René Marois. 2013. The source of dual-task limitations: serial or parallel processing of multiple response selections?Atten. Percept. Psychophys. 75, 7 (Oct. 2013), 1395–1405.

[15]

James B Heald, Máté Lengyel, and Daniel M Wolpert. 2021. Contextual inference underlies the learning of sensorimotor repertoires. Nature 600, 7889 (Dec. 2021), 489–493.

[16]

Christopher G Healey, Kellogg S Booth, and James T Enns. 1996. High-speed visual estimation using preattentive processing. ACM Trans. Comput. Hum. Interact. 3, 2 (June 1996), 107–135.

Digital Library

[17]

Christopher G Healey and James T Enns. 2012. Attention and visual memory in visualization and computer graphics. IEEE Trans. Vis. Comput. Graph. 18, 7 (July 2012), 1170–1188.

Digital Library

[18]

Yoko Higuchi, Yoshiyuki Ueda, Kazuhisa Shibata, and Jun Saiki. 2019. Spatial variability induces generalization in contextual cueing. J. Exp. Psychol. Learn. Mem. Cogn. (Nov. 2019).

[19]

Matthias Hoppe, Andrea Baumann, Patrick Chofor Tamunjoh, Tonja-Katrin Machulla, Paweł W Woźniak, Albrecht Schmidt, and Robin Welsch. 2022. There Is No First- or Third-Person View in Virtual Reality: Understanding the Perspective Continuum. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22, Article 360). Association for Computing Machinery, New York, NY, USA, 1–13.

Digital Library

[20]

Hiroshi Imamizu and Mitsuo Kawato. 2009. Brain mechanisms for predictive control by switching internal models: implications for higher-order cognitive functions. Psychol. Res. 73, 4 (July 2009), 527–544.

[21]

Shunichi Kasahara, Ryuma Niiyama, Valentin Heun, and Hiroshi Ishii. 2013. exTouch: spatially-aware embodied manipulation of actuated objects mediated by augmented reality. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction (Barcelona, Spain) (TEI ’13). Association for Computing Machinery, New York, NY, USA, 223–228.

Digital Library

[22]

A M Clare Kelly and Hugh Garavan. 2005. Human functional neuroimaging of brain changes associated with practice. Cereb. Cortex 15, 8 (Aug. 2005), 1089–1102.

[23]

Masatomo Kobayashi and Takeo Igarashi. 2008. Ninja cursors: using multiple cursors to assist target acquisition on large screens. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 949–958.

Digital Library

[24]

Andrey Krekhov and Jens Krueger. 2021. Deadeye: A Novel Preattentive Visualization Technique Based on Dichoptic Presentation. (Jan. 2021). arxiv:2101.07048 [cs.HC]

[25]

Manu Kumar, Andreas Paepcke, and Terry Winograd. 2007. EyeExpos’e: Switching applications with your eyes.

[26]

Seungyeol Lee and Jeon-Il Moon. 2017. Management of a Single-User Multi-Robot Teleoperated System for Maintenance in Offshore Plants. In Multi-agent Systems. InTech.

[27]

Bingxin Li, Xiangqian Li, Gijsbert Stoet, and Martin Lages. 2019. Exploring individual differences in task switching. Acta Psychol. 193 (Feb. 2019), 80–95.

[28]

René Marois and Jason Ivanoff. 2005. Capacity limits of information processing in the brain. Trends Cogn. Sci. 9, 6 (June 2005), 296–305.

[29]

Antonella Maselli and Mel Slater. 2014. Sliding perspectives: dissociating ownership from self-location during full body illusions in virtual reality. Front. Hum. Neurosci. 8 (Sept. 2014), 693.

[30]

B McElree and M Carrasco. 1999. The temporal dynamics of visual search: evidence for parallel processing in feature and conjunction searches. J. Exp. Psychol. Hum. Percept. Perform. 25, 6 (Dec. 1999), 1517–1539.

[31]

Reiji Miura, Shunichi Kasahara, Michiteru Kitazaki, Adrien Verhulst, Masahiko Inami, and Maki Sugimoto. 2022. MultiSoma: Motor and Gaze Analysis on Distributed Embodiment With Synchronized Behavior and Perception. Frontiers in Computer Science 4 (2022).

[32]

Yukiya Nakanishi, Masaaki Fukuoka, Shunichi Kasahara, and Maki Sugimoto. 2022. Synchronous and Asynchronous Manipulation Switching of Multiple Robotic Embodiment Using EMG and Eye Gaze. In Augmented Humans 2022 (Kashiwa, Chiba, Japan) (AHs 2022). Association for Computing Machinery, New York, NY, USA, 94–103.

Digital Library

[33]

Ulric Neisser and Robert Becklen. 1975. Selective looking: Attending to visually specified events. Cogn. Psychol. 7, 4 (Oct. 1975), 480–494.

[34]

Mikhail Ostanin, Stanislav Mikhel, Alexey Evlampiev, Valeria Skvortsova, and Alexandr Klimchik. 2020. Human-robot interaction for robotic manipulator programming in Mixed Reality. In 2020 IEEE International Conference on Robotics and Automation (ICRA). 2805–2811.

[35]

H Pashler. 1994. Dual-task interference in simple tasks: data and theory. Psychol. Bull. 116, 2 (Sept. 1994), 220–244.

[36]

H Pashler. 1994. Graded capacity-sharing in dual-task interference?J. Exp. Psychol. Hum. Percept. Perform. 20, 2 (April 1994), 330–342.

[37]

Z W Pylyshyn and R W Storm. 1988. Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spat. Vis. 3, 3 (1988), 179–197.

[38]

Kari-Jouko Räihä and Oleg Špakov. 2009. Disambiguating ninja cursors with eye gaze. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 1411–1414.

Digital Library

[39]

Jonas Schjerlund, Kasper Hornbæk, and Joanna Bergström. 2021. Ninja Hands: Using Many Hands to Improve Target Selection in VR. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[40]

E H Schumacher, T L Seymour, J M Glass, D E Fencsik, E J Lauber, D E Kieras, and D E Meyer. 2001. Virtually perfect time sharing in dual-task performance: uncorking the central cognitive bottleneck. Psychol. Sci. 12, 2 (March 2001), 101–108.

[41]

Mariano Sigman and Stanislas Dehaene. 2008. Brain mechanisms of serial and parallel processing during dual-task performance. J. Neurosci. 28, 30 (July 2008), 7585–7598.

[42]

Kazuma Takada, Midori Kawaguchi, Yukiya Nakanishi, Akira Uehara, Mark Armstrong, Adrien Verhulst, Kouta Minamizawa, and Shunichi Kasahara. 2021. Parallel Ping-Pong: Demonstrating Parallel Interaction through Multiple Bodies by a Single User. In SIGGRAPH Asia 2021 Emerging Technologies(Tokyo, Japan) (SA ’21 Emerging Technologies, Article 12). Association for Computing Machinery, New York, NY, USA, 1–2.

[43]

Kazuma Takada, Midori Kawaguchi, Akira Uehara, Yukiya Nakanishi, Mark Armstrong, Adrien Verhulst, Kouta Minamizawa, and Shunichi Kasahara. 2022. Parallel Ping-Pong: Exploring Parallel Embodiment through Multiple Bodies by a Single User. In Augmented Humans 2022 (Kashiwa, Chiba, Japan) (AHs 2022). Association for Computing Machinery, New York, NY, USA, 121–130.

Digital Library

[44]

James T Townsend. 1990. Serial vs. Parallel Processing: Sometimes They Look like Tweedledum and Tweedledee but they can (and Should) be Distinguished. Psychol. Sci. 1, 1 (Jan. 1990), 46–54.

[45]

A M Treisman and G Gelade. 1980. A feature-integration theory of attention. Cogn. Psychol. 12, 1 (Jan. 1980), 97–136.

[46]

Adrien Verhulst, Yasuko Namikawa, and Shunlchl Kasahara. 2022. Parallel Adaptation: Switching between Two Virtual Bodies with Different Perspectives Enables Dual Motor Adaptation. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 169–177.

[47]

C Ware. 2004. Information Visualization: Perception for Design: Second Edition. unknown.

Digital Library

[48]

Kei Watanabe and Shintaro Funahashi. 2014. Neural mechanisms of dual-task interference and cognitive capacity limitation in the prefrontal cortex. Nat. Neurosci. 17, 4 (April 2014), 601–611.

[49]

Jeremy M Wolfe. 2014. Approaches to Visual Search. In The Oxford Handbook of Attention, Anna C. (Kia) Nobre and Sabine Kastner (Eds.). Oxford University Press.

Cited By

Cmentowski SKaraosmanoglu SKievelitz FSteinicke FKrüger J(2023)A Matter of Perspective: Designing Immersive Character Transitions for Virtual Reality GamesProceedings of the ACM on Human-Computer Interaction10.1145/36110237:CHI PLAY(73-103)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3611023

Index Terms

ShadowClones: an Interface to Maintain a Multiple Sense of Body-space Coordination in Multiple Visual Perspectives
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User interface design

Recommendations

Shadow Clones: an Unattended Visual Interface to Enhance the Task Performance in Multiple Space using Gaze-Switching
SUI '22: Proceedings of the 2022 ACM Symposium on Spatial User Interaction

We propose Shadow Clones; a user interface with unattended visual information to accomplish multiple tasks in multiple spaces using fast-gaze-switching. Shadow Clones interface provides not only a cursor on the view area the user is gazing at but also ...
Salient object detection via multiple saliency weights

Salient object detection aims to emulate the extraordinary capability of human visual system, which has the ability to find the most visually attractive objects in a complex visual scene. The human visual attention is often complicated and affected by ...
Overview Use in Multiple Visual Information Resolution Interfaces

In interfaces that provide multiple visual information resolutions (VIR), low-VIR overviews typically sacrifice visual details for display capacity, with the assumption that users can select regions of interest to examine at higher VIRs. Designers can ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

AHs '23: Proceedings of the Augmented Humans International Conference 2023

March 2023

395 pages

ISBN:9781450399845

DOI:10.1145/3582700

General Chairs:
Jamie A. Ward
Goldsmiths, University of London, United Kingdom
,
Mark McGill
University of Glasgow, United Kingdom
,
Karola Marky
Ruhr-University Bochum, Germany

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

JST Moonshot R\&D Program

Conference

AHs '23

AHs '23: Augmented Humans Conference

March 12 - 14, 2023

Glasgow, United Kingdom

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
274
Total Downloads

Downloads (Last 12 months)130
Downloads (Last 6 weeks)7

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cmentowski SKaraosmanoglu SKievelitz FSteinicke FKrüger J(2023)A Matter of Perspective: Designing Immersive Character Transitions for Virtual Reality GamesProceedings of the ACM on Human-Computer Interaction10.1145/36110237:CHI PLAY(73-103)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3611023

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents