Can a Robot’s Hand Bias Human Attention? Giulia Scorza Azzarà Joshua Zonca Francesco Rea DIBRIS, University of Genoa RBCS, Italian Institute of Technology Genoa, Italy giulia.scorza@iit.com CONTACT Unit, Italian Institute of Technology Genoa, Italy joshua.zonca@iit.com CONTACT Unit, Italian Institute of Technology Genoa, Italy francesco.rea@iit.com Joo-Hyun Song Alessandra Sciutti Cognitive, Linguistic, and Psychological Sciences, Brown University Providence, Rhode Island, USA joo-hyun_song@brown.edu CONTACT Unit, Italian Institute of Technology Genoa, Italy alessandra.sciutti@iit.com ABSTRACT 1 Previous studies have revealed that humans prioritize attention to the space near their hands (the so-called near-hand effect). This effect may also occur towards a human partner’s hand, but only after sharing a physical joint action. Hence, in human dyads, interaction leads to a shared body representation that may influence basic attentional mechanisms. Our project investigates whether a collaborative interaction with a robot might similarly influence attention. To this aim, we designed an experiment to assess whether the mere presence of a robot with an anthropomorphic hand could bias the human partner’s attention. We replicated a classical psychological paradigm to measure this attentional bias (i.e., the near-hand effect) by adding a robotic condition. Preliminary results found the near-hand effect when performing the task with the self-hand near the screen, leading to shorter reaction times on the same side of the hand. On the contrary, we found no effect on the robot’s hand in the absence of previous collaborative interaction with the robot, in line with studies involving human partners. In everyday activities, we often need to coordinate and synchronize with our partners, whose perceptual and motor abilities might differ from ours. Human beings effectively understand each other’s intentions while interacting in social contexts. Such capability is particularly remarkable if we consider that our visual perception of space is often inaccurate but can suffer from biases (e.g., central tendency [11], [22]), illusions (e.g., rescaling [20], [21]), and similar phenomena affecting human attention [4], [19]. One of the main goals of HRI is to design and employ robots that can interact and collaborate efficiently and naturally with human beings. To this aim, it is crucial to investigate the perceptual, motor, and attentional mechanisms that could support (or hinder) mutual understanding between the parties [16]. Previous studies proved that human attention is prioritized for the space near their hands [1], [15], resulting in shorter reaction times when detecting visual stimuli that appear close to their own hands (the so-called near-hand effect) [14]. Besides, after a collaborative task, this effect also occurs on the human partner’s hand [17]. These findings suggest that collaborative interaction with a human partner influences our shared body representation, biasing our attention toward the partner’s hand almost as if it was ours. Our project aims at investigating under which conditions attentional biases, such as the near-end effect, can also occur in a human-robot interaction context. To achieve this goal, we need to address two questions: CCS CONCEPTS · Human-centered computing → Empirical studies in HCI. KEYWORDS Human attention, Posner cueing task, Near-hand effect ACM Reference Format: Giulia Scorza Azzarà, Joshua Zonca, Francesco Rea, Joo-Hyun Song, and Alessandra Sciutti. 2023. Can a Robot’s Hand Bias Human Attention?. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’23 Companion), March 13ś16, 2023, Stockholm, Sweden. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3568294.3580074 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). HRI ’23 Companion, March 13ś16, 2023, Stockholm, Sweden © 2023 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9970-8/23/03. https://doi.org/10.1145/3568294.3580074 213 INTRODUCTION (1) Does the presence of a robot anthropomorphic hand bias human attention following a near-hand effect? (2) Can a collaborative joint physical interaction with the robot lead to the near-hand effect? To answer these research questions, we replicated a well-known psychology paradigm to study the near-hand effect in humans in individual and joint settings [12], using the iCub robot [7] as a controllable stimulus. From a technical point of view, iCub is an optimal choice since its hands have a similar structure and size as human hands and are characterized by many degrees of freedom that guarantee movements similar to human ones. Moreover, it can be programmed to act as a social agent and generate bio-inspired movements and actions [6], supporting compliance in physical interaction. HRI ’23 Companion, March 13ś16, 2023, Stockholm, Sweden Giulia Scorza Azzarà, Joshua Zonca, Francesco Rea, Joo-Hyun Song, & Alessandra Sciutti In this paper, we describe the methodology and results related to the first research question and introduce the plan to address the second research question in our future work. 2 (a) (b) METHODOLOGY We designed an experiment to analyze the near-hand effect of iCub’s hand. More precisely, we aimed firstly at quantifying each participant’s near-hand effect for their own hand and then evaluate whether such effect generalizes to a robotic anthropomorphic hand presented by the iCub robot close to them. The participants performed a well-known attention task called "Posner cueing task" [12] while sitting next to iCub. The Posner cueing task (Figure 1) is a classical paradigm used to study visual attention. There are two empty squares (3.4◦ ) on both sides (7.4◦ ) of a central fixation cross (3.4◦ ). After a random time interval between 1500-3000 ms, one square is cued by increasing the thickness of its borders for 200 ms, and then a target appears (black dot; 2.2◦ ). The participant has to press the space bar on a keyboard as soon as the target appears. If the target appears in the cued square, this is classified as a valid trial; otherwise, this is classified as an invalid trial. Sometimes the square remains cued for 2000 ms, but the square does not appear; these are the catch trials, and they are used to check if the participant is still focused on the task. We used 70% valid trials, 20% invalid trials, and 10% catch trials in random order, as it was done in [17]. (a) (b) Figure 2: Experimental conditions: human’s hand near the screen (a) and robot’s hand near the screen (b). 2.3 Procedure Participants performed the Posner cueing task under three possible conditions: no hand near the screen, self-hand (i.e., human’s hand) near the screen, and robot’s hand near the screen. Each experimental session included four blocks of 60 trials, two with no hand near the screen and two with a hand near the screen (i.e., human’s or robot’s hand), as displayed in Figure 2. The block order was randomized. Each participant did two sessions, one with the self-hand near the screen and the other with the robot’s hand near the screen. We asked participants to repeat the entire experiment with both hands to avoid possible effects depending on the use of the dominant hand (Figure 3). (c) Figure 1: Posner cueing task: valid trial (a), invalid trial (b), and catch trial (c). 3 RESULTS AND DISCUSSION The dependent measure of interest was the participants’ reaction times for target detection. The Reaction Times (RTs) of the responses were filtered between two thresholds: RTs > 200 ms, which is the average physiological threshold for RTs in human beings [8], and RTs < 1000 ms, following the research by Sun & Thomas [17]. 2.1 Participants Twenty-two right-handed people participated in the study (15 females, 7 males; mean age = 26.05 y.o.; std = 5.21 y.o.). All participants had a normal or corrected-to-normal vision and were naive to the purpose of the study. The Regional Ethical Committee approved the experimental protocol for protecting human participants in research, and all participants provided written informed consent before participating in the experiment. (a) (b) 2.2 Apparatus The experiment was programmed in MATLAB using the PsychToolbox extension. All visual stimuli were drawn in black against a light grey background on a monitor with a display resolution of 1024 × 768 pixels. The experimental setup was the same for all conditions. We used a chin rest to maintain the participant at a constant distance from the screen (i.e., 50 cm). The participants’ responses and reaction times were collected through the computer keyboard. When asked to place a hand near the computer screen, participants rested the forearm on support to minimize the discomfort associated with a prolonged extension of the hand and arm during the task. Figure 3: Experimental setup: P = participant, R = robot, and E = experimenter. Each participant did the task with their right hand (a) and left hand (b). 214 Can a Robot’s Hand Bias Human Attention? HRI ’23 Companion, March 13ś16, 2023, Stockholm, Sweden (a) (b) Figure 4: Effect on the human hand. (a) There is a significant difference in RTs between no-hand (green) and human hand (blue) conditions. (b) The effect on the human hand is consistent among all subjects. Almost all the dots lie under the bisector. (a) (b) Figure 5: Effect on the robot’s hand. (a) There is no significant difference in RTs between no-hand (green) and robot’s hand (red) conditions. (b) The effect on the robot’s hand is not consistent among the subjects. Data is distributed along the bisector. We excluded one participant from the analysis for excessive errors in catch trials (i.e., > 55%). The overall error rate of the participants in catch trials was 9.2%. Eventually, 5.5% of trials were discarded because they fell outside the 200ś1000 ms window. We analyzed participants’ reaction times using paired sample t-tests. The result of the first test concerns the Posner cueing task. We found a significant main effect of cue validity (t (21) = 9.03, p < 0.001), proving that participants detected faster the visual targets in valid than invalid trials; this is always verified, regardless of the experimental condition. Anyway, cue validity did not affect the near-hand effect. Afterward, we considered the hand validity variable, which prescribes whether the hand near the screen is on the same side as the appearing target. In particular, we refer to "hand valid" trials when the hand and the target are on the same side, whereas in "hand invalid" trials hand and target are on the opposite side. The bars in Figures 4 and 5 represent the difference between the average RT of "hand valid" and "hand invalid" trials. Results showed the expected near-hand effect when performing the attention task in the human hand condition, leading to significantly shorter reaction times on the same side of the hand near the screen (t (21) = 6.85, p < 0.001). Figure 4a displays the RT delta between the no-hand (green) and the human hand (blue) experimental conditions. The scatter plot of Figure 4b shows the results are very consistent between all the participants. Indeed, almost all the dots, each representing a participant, lie under the bisector, meaning most of the participants’ mean RTs are shorter in the human hand condition than in the no-hand condition. 215 HRI ’23 Companion, March 13ś16, 2023, Stockholm, Sweden Giulia Scorza Azzarà, Joshua Zonca, Francesco Rea, Joo-Hyun Song, & Alessandra Sciutti On the contrary, we found no significant effect on the robot’s hand condition (t (21) = 0.09, p = 0.928). Figure 5a displays the RT delta between the no-hand (green) and the robot’s hand (red) experimental conditions, showing they have similar values in average RT. In this case, data are more distributed across the reference line, as displayed in the scatter plot of Figure 5b, and the mean value lies on the bisector, meaning there is no significant difference between no-hand and robot’s hand conditions. Finally, the last test compares the human and robot conditions. We found a significant difference between the above two conditions (t (21) = -4.51, p < 0.001). The overall difference in reaction times is about 13 ms. This difference highlights the near-hand effect on the human hand and the latter’s absence on the robot’s hand. 4 research and innovation programme. G.A. No 804388, wHiSPER. We also acknowledge the support from the National Science Foundation (NSF) BCS 2043328. REFERENCES [1] Richard A Abrams, Christopher C Davoli, Feng Du, William H Knapp III, and Daniel Paull. 2008. Altered vision near the hands. Cognition 107, 3 (2008), 1035ś1047. [2] Muneeb Imtiaz Ahmad, Omar Mubin, and Joanne Orlando. 2017. A systematic review of adaptivity in human-robot interaction. Multimodal Technologies and Interaction 1, 3 (2017), 14. [3] James R Brockmole, Christopher C Davoli, Richard A Abrams, and Jessica K Witt. 2013. The world within reach: Effects of hand posture and tool use on visual cognition. Current Directions in Psychological Science 22, 1 (2013), 38ś44. [4] Joshua D Cosman and Shaun P Vecera. 2010. Attention affects visual perceptual processing near the hand. Psychological science 21, 9 (2010), 1254ś1258. [5] Christopher C Davoli and Philip Tseng. 2015. Taking a hands-on approach: Current perspectives on the effect of hand position on vision. Frontiers in psychology 6 (2015), 1231. [6] Sarah Degallier, Ludovic Righetti, Lorenzo Natale, Francesco Nori, Giorgio Metta, and Auke Ijspeert. 2008. A modular bio-inspired architecture for movement generation for the infant-like robot iCub. In 2008 2nd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics. IEEE, 795ś800. [7] Tobias Fischer, Jordi-Ysard Puigbò, Daniel Camilleri, Phuong DH Nguyen, Clément Moulin-Frier, Stéphane Lallée, Giorgio Metta, Tony J Prescott, Yiannis Demiris, and Paul FMJ Verschure. 2018. iCub-HRI: a software framework for complex human-robot interaction scenarios on the iCub humanoid robot. Frontiers in Robotics and AI 5 (2018), 22. [8] Robert J. Kosinski. 2008. A literature review on reaction time. Clemson University 10, 1 (2008), 337ś344. [9] Rebeka Kropivšek Leskovar, Jernej Čamernik, and Tadej Petrič. 2021. LeaderFollower Role Allocation for Physical Collaboration in Human Dyads. Applied Sciences 11, 19 (2021), 8928. [10] Carlo Mazzola, Francesco Rea, and Alessandra Sciutti. 2022. Shared perception is different from individual perception: a new look on context dependency. IEEE Transactions on Cognitive and Developmental Systems (2022). [11] Maria Olkkonen, Patrice F McCarthy, and Sarah R Allred. 2014. The central tendency bias in color perception: Effects of internal and external noise. Journal of vision 14, 11 (2014), 5ś5. [12] Michael I Posner. 1980. Orienting of attention. Quarterly journal of experimental psychology 32, 1 (1980), 3ś25. [13] Francesco Rea, Alessia Vignolo, Alessandra Sciutti, and Nicoletta Noceti. 2019. Human motion understanding for selecting action timing in collaborative humanrobot interaction. Frontiers in Robotics and AI 6 (2019), 58. [14] Catherine L Reed, Ryan Betz, John P Garza, and Ralph J Roberts. 2010. Grab it! Biased attention in functional hand and tool space. Attention, Perception, & Psychophysics 72, 1 (2010), 236ś245. [15] Catherine L Reed, Jefferson D Grubb, and Cleophus Steele. 2006. Hands up: attentional prioritization of space near the hand. Journal of Experimental Psychology: Human Perception and Performance 32, 1 (2006), 166. [16] Alessandra Sciutti, Martina Mara, Vincenzo Tagliasco, and Giulio Sandini. 2018. Humanizing human-robot interaction: On the importance of mutual understanding. IEEE Technology and Society Magazine 37, 1 (2018), 22ś29. [17] Hsin-Mei Sun and Laura E Thomas. 2013. Biased attention near another’s hand following joint action. Frontiers in Psychology 4 (2013), 443. [18] Ana Tanevska, Francesco Rea, Giulio Sandini, Lola Cañamero, and Alessandra Sciutti. 2020. A socially adaptable framework for human-robot interaction. Frontiers in Robotics and AI 7 (2020), 121. [19] Philip Tseng, Bruce Bridgeman, and Chi-Hung Juan. 2012. Take the matter into your own hands: a brief review of the effect of nearby-hands on visual processing. Vision research 72 (2012), 74ś77. [20] Björn Van der Hoort and H Henrik Ehrsson. 2014. Body ownership affects visual perception of object size by rescaling the visual representation of external space. Attention, Perception, & Psychophysics 76, 5 (2014), 1414ś1428. [21] Björn Van Der Hoort and H Henrik Ehrsson. 2016. Illusions of having small or large invisible bodies influence visual perception of object size. Scientific reports 6, 1 (2016), 1ś9. [22] Yang Xiang, Thomas Graeber, Benjamin Enke, and Samuel J Gershman. 2021. Confidence and central tendency in perceptual judgment. Attention, Perception, & Psychophysics 83, 7 (2021), 3024ś3034. CONCLUSION AND FUTURE WORK This project aims at assessing whether collaborative interaction with a humanoid robot can shape basic attentional and perceptual mechanisms in humans, as previously observed in human-human interaction scenarios. The preliminary results of the experiment proved the presence of the near-hand effect on the self-hand through the performance of an attention task, i.e., the Posner cueing task. Moreover, we found no effect caused by the mere presence of an anthropomorphic robot hand. This finding expands the results of research conducted with fake human-like hands or other persons’ hands [3], [5], demonstrating that an anthropomorphic robot hand, as a friend’s human hand, is not per se sufficient to shift human attention toward itself. With the next experiment, we will assess if a physical humanrobot interaction can bias human attention near the robot’s hand. The task will consist of a physical joint action between the human and the iCub, inspired by existing human-human research; then, participants will repeat the Posner task. We have built the collaborative task for the physical human-robot interaction and tested it in a pilot study. We hypothesize to find the near-hand effect after the physical HRI, as it happens between human dyads [17]. The use of the robot will allow us to precisely control and quantitatively assess the dynamics of the interaction to gain a better insight into which features of a joint action might influence the appearance of a "joint" near-hand effect as observed in humanhuman collaborative interaction. Furthermore, it will be possible to assess the role played by the social component of the interaction by manipulating the robot’s behavior to exhibit different levels of social intelligence. That has been shown to impact basic perceptual mechanisms such as human space perception [10]. A final consideration concerns adaptation which is a fundamental ability evident both at the behavioral and physiological levels and still quite an open issue in the field [2], [13], [18]. It would be interesting for this research project future if we enable the robot to adapt its behavior to that of the current interacting partner to allocate the roles of leader-follower naturally as it happens between human dyads [9]. ACKNOWLEDGMENTS This work has been supported by a Starting Grant from the European Research Council (ERC) under the European Union’s H2020 216