ShareYourReality: Investigating Haptic Feedback and Agency in Virtual Avatar Co-embodiment

Virtual co-embodiment refers to occurrences where multiple users can simultaneously interact with the virtual environment using a shared avatar. Given that two or more individuals share control over the avatar, there is an increase in social coordination (Fribourg et al., 2021). To this end, prior work has shown that participants who co-embody a virtual avatar reported high levels of perceived control, with lower levels of weighted percentage of actual control (Hapuarachchi et al., 2023; Kodama et al., 2022). This makes it a promising tool for VR-based rehabilitation (Juan et al., 2023) and training (Gonzalez-Franco et al., 2017; Harris et al., 2023) applications, since a learner with low control can feel a stronger sense of agency (SoA) while performing the activity with a teacher with high control. One domain in which researchers are trying to leverage the immersive capabilities of VR is in the support and treatment of dementia patients (Zhu et al., 2021) – these individuals are considered vulnerable and they typically find it challenging to operate basic VR controls (Lazar et al., 2014). In such cases, co-embodiment can enable assistive accessibility for these individuals, guided by support personnel. This would enable them to not only train, but also maintain a high level of agency during such immersive experiences. Furthermore, VR has shown potential for its ability to stimulate Mirror Neurons (MNs) of the internal sensorimotor system of stroke patients (Carvalho et al., 2013). In such settings, patients are immersed in training scenarios in virtual environments that involve executing motor actions, such as observing and visualizing mirror limb movements with the intent to imitate these actions. These have shown enhanced MN activation, leading to faster post-stroke recovery (Mekbib et al., 2021). To that end, co-embodiment can be further leveraged within these techniques in order to improve the effectiveness of such treatments through enhanced agency and nuanced control over movements executed by the patients actively guided by their trainers or caregivers.

In the context of avatar co-embodiment, the ‘Sense Of Embodiment’ (SoE) can be manifested through three main components: Sense of Self-Location, Sense of Body Ownership, and the Sense of Agency (Kilteni et al., 2012). Sense of Self-Location refers to the feeling of ‘being inside’ a virtual body, while sense of body ownership and agency refers to the feeling of ‘having’ and ‘controlling’ the virtual body, respectively. Studies have explored various factors and their influence on these components, and have shown that manipulations of the overall SoE are possible through changes in avatar representations, degree of control, and perspective of the users (Fribourg et al., 2020; Ogawa et al., 2019). Similarly, the influence of sharing the virtual body with another user and its effect on SoE was studied in experiments of virtual co-embodiment. Here, the sense of agency and body ownership play a pivotal role that determines the engagement level during the shared perceptual activity (Hagiwara et al., 2019; Fribourg et al., 2021; Hapuarachchi and Kitazaki, 2022; Hapuarachchi et al., 2023; Kodama et al., 2023).

To realize such co-embodiment, the motion of a shared avatar has previously been generated using two techniques: the weighted average co-embodiment method (Hagiwara et al., 2019; Fribourg et al., 2021; Kodama et al., 2022) and the body-part-segmented co-embodiment method (Hapuarachchi and Kitazaki, 2022; Hapuarachchi et al., 2023). The weighted-average method involves assigning a weight between 0 and 100 percent to each user and generating the movement of the shared avatar by interpolating the weighted average of the real-time position and orientation of the controllers of both the users (Figure 2) (Hagiwara et al., 2019; Fribourg et al., 2021; Kodama et al., 2022). The body-part-segmented co-embodiment method is a technique where the motion of independent limbs of a shared avatar is controlled by each user separately (Hapuarachchi and Kitazaki, 2022; Hapuarachchi et al., 2023). In this paper we focus on the first method, where shared interactions can be manipulated by both the users and their influence is determined by the percentage of control they possess. Since we focus on position-aware feedback mechanisms, we draw on the weighted-average method to enable this. Results from (Hagiwara et al., 2019; Fribourg et al., 2021; Kodama et al., 2022) all showed that sense of agency increased with the increase in the control weight for the participant during avatar co-embodiment. In all the studies, participants could coordinate their movements in joint action, leading to the sharing of motor intention and synchronization. In a follow-up study by Kodama et al. (2023), they evaluated participants’ task performance and motor skill learning ability. They concluded that learning using virtual co-embodiment was more efficient than the perspective-sharing method, in which a translucent teacher avatar was superimposed on the learner’s first-person perspective view. However, contrary to the previous studies, no significant differences were observed between the different control weight conditions. We draw on the design considerations from (Hagiwara et al., 2019; Fribourg et al., 2021; Kodama et al., 2022) to design our study protocol.

Since the early days of VR, haptic feedback has been a central component in many VR systems (Srinivasan and Basdogan, 1997) and has been used to enable a diversity of touch-based interactions in VR  (Wee et al., 2021), with the most common type of haptic feedback in VR applications being vibrotactile and force feedback (Yang et al., 2021). Studies that have explored haptics as a communication medium in the context of shared virtual spaces report enhanced user experiences (Jung et al., 2021; Zhang et al., 2023). Important to our present purposes, the addition of haptic feedback to social VR has been found to consistently enhance perceived social presence (Oh et al., 2018). In work on co-embodiment, the application of haptic feedback is essentially understudied. Hapuarachchi and Kitazaki (2022) explored the manipulation of the sense of agency by providing visual feedback of the partner’s target during co-embodiment, and Hapuarachchi et al. (2023) implemented passive haptics by attaching a back brace to both the users, allowing them to maintain consistent shoulder posture while controlling the shared avatar using the body-part-segmented co-embodiment method. These explorations highlight the value of identifying what type of feedback modalities can be integrated into the virtual co-embodiment paradigm to provide users with advanced perceptual capabilities. While visual feedback offers more information to the user, it leads to cluttered, chaotic experiences when scaled up. Thus, haptic feedback provides an alternative to overcome this limitation. The challenge in the context of co-embodiment is to design the feedback mechanism in a way that does not increase the cognitive load required to differentiate between the interaction with the environment and the presence of the other user. Given the foregoing, we implement haptic feedback as a communication medium to indicate the other users’ position during co-embodiment.

The perceptual crossing paradigm (Auvray et al., 2009) was conceived to study social interaction dynamics in real time through tactile sensorimotor interactions (Kojima et al., 2017). The classical paradigm features a minimalist 1D environment–a line–that loops around creating a continuous interaction space not visible to the users. Two users are each represented in the virtual space by an avatar (a dot) that they can control using a standard computer mouse. When their avatar encounters a virtual object in the 1D space they receive haptic feedback. There are three types of objects in the environment: a static object, the user’s avatars, and the user’s ’shadow,’ an object that moves with the users’ avatars at a set distance. All objects have exactly the same size and produce the same haptic feedback when encountered. When one user encounters the shadow of the other user, only the interacting user receives haptic feedback while the other user (to whom the shadow belongs), does not. The only condition when both users receive haptic feedback simultaneously is when both users’ avatars encounter each other. Users are tasked with clicking the mouse when they think they are interacting with the avatar of the other user. In the original studies (Auvray et al., 2009; Auvray and Rohde, 2012), users were successfully able to locate each other in the 1D virtual space. However, the probability of clicking when encountering another user’s avatar was not significantly higher than clicking when encountering another user’s shadow. Successful identification of the other could only be explained by the stability of mutual recognition; users would encounter one another, move back, encounter each other again, and repeat, creating an oscillating movement pattern of repeated encounters. In other words, users would only successfully recognize each other during perceptual crossing (i.e., perceptual activities of the same kind meet each other).

Extensions of the basic paradigm have shown that, in a team-based version of the paradigm where users were instructed to collaborate, participants successfully identified the other’s avatar and, for those encounters, reported the clearest awareness of the others’ presence (Froese et al., 2014a). A version of the paradigm that used a following task in a skewed 1D environment highlighted that users were successful in following each other’s movements through haptic perceptual crossing (Lenay, 2021). Though part of the strength of the perceptual crossing paradigm lies in the minimalist approach, 2D extensions have already been successful (Lenay et al., 2011). To the best of our knowledge, no 3D implementations of the paradigm have ever been attempted. We see an interesting opportunity in the implementation of the paradigm’s basic premise (i.e., haptic feedback upon contact in a virtual space to signify the presence of the other) as an interactive cue that could aid movement coordination as well as enhance perceived social presence in virtual co-embodiment scenarios.

Our study has two main Independent Variables (IVs), and follows a 3 (IV1: Control Distribution: 25-75% vs. 50-50% vs. 75-25%) x 2 (IV2: Haptic Feedback: None vs. Present) within-subject design, tested in a controlled, virtual environment. The control distribution consisted of three sets of weighing player one and player two’s control over the shared avatar: 25-75%, 50-50%, 75-25% (referred to as W25, W50, W75). There was either no haptic feedback (NH) or haptic feedback when participants’ hands overlapped (H) in the virtual space. This interaction was designed using virtual spheres (radius 8 cm), which is approximately the size of the virtual hand mesh that was attached to the controllers in the virtual environment. When the spheres of each user intersect with each other, the haptic feedback is triggered for both the users. The study was divided into three phases: Training, Task 1, and Task 2 (Figure 3). There were six distinct conditions (3 control distribution x 2 haptic feedback) for each task that was performed by a pair of participants. The experiment consisted of two tasks where each of these conditions were repeated twice, bringing the total to 24 (2 tasks x 2 repetitions) trials for the entire study. The subjective responses of the participants on the sense of agency were collected with a questionnaire after each trial, while the sense of co-presence and embodiment questionnaires were collected after each haptic feedback condition / block (after six trials). Task 1 was always performed before Task 2, and the two haptic feedback conditions were counterbalanced according to a Latin square design such that starting trial of each session consisted of all possible combinations of haptic and control conditions, with the remainder of trials subsequently randomized to mitigate order effects. The study was designed such that a sample consists of pairs of participants. For example, in a session, participant 1 would perform the task with 25% control four times (4x), twice with and twice without haptic feedback, while their counterpart had 75% control (performed also 4x). Similarly for 50% (4x) and 75% (4x). Therefore for ”25%”, ”75%”, and ”50%”, there were 4 samples for each task (twice with and twice without haptic feedback).

Participants performed the study using Oculus Quest 2 Head-Mounted Displays (HMDs) and Oculus Touch VR motion controllers connected to desktop computers. These computers ran the virtual environment we created using Unreal Engine 5.1²²2https://www.unrealengine.com/, and were connected with Ethernet to ensure minimum latency between the computers. One computer hosts a local server while the second computer joined this server as a client. Each computer recorded the rotation and position of their respective users. To create a co-embodied avatar, the level spawns a “shared hands” avatar in the virtual world. This virtual representation was chosen to model a gender neutral representation of hands (cf., (Schwind et al., 2017)). Since the controller was not represented in virtual space, we did not have the position of the hand to be wrapped around the controller, and instead showed a default open palm position pose. To determine the position of each of the shared hands, the avatar linearly interpolates between the position of User 1 and User 2, expressed by the following equation:

where $\alpha$ controls the interpolation such that the resulting position is 100% of Player 1 when $\alpha$ is 1 and 100% of Player 2’s position when $\alpha$ is 0. This value can be set to vary the control over the shared hands in each level to W25/W50/W75 to create the conditions outlined in the study design.

At the study location, a table was placed where participants would fill out all the forms: demographics, informed consent, pre- and post-study SSQ, and IPQ. Two computers were placed side by side on a separate table and were connected to HMDs (Figure 7). One participant was randomly assigned to the computer acting as User 1, and the other to the computer acting as User 2. A video camera was also in place to record both participants’ motions while they performed the tasks. A video showing this interaction is provided in Supplementary Material A. The position in which both participants would stand was marked on the floor. To reduce any possibility of injury and to simplify the interactions, participants conducted the trials while standing, and only used their right hand for the motion task. The spatial chime sound was channeled through the HMD speakers, set at a comfortable 60% volume.

Upon their arrival, participants were asked to read and sign the informed consent form and fill in a pre-study SSQ. Similar to Fribourg et al. (2021), participants were briefed that they would be sharing the avatar during all trials and instructed to avoid communicating with each other verbally. No instructions were provided regarding the haptic feedback, allowing participants to interpret the meaning of the vibrotactile cue when it occurs during the tasks. This was done to evaluate the effectiveness of the chosen vibration pattern in establishing synchronization patterns between participants autonomously, as was done in the original perceptual crossing experiments (Auvray et al., 2009; Auvray and Rohde, 2012). Participants provided subjective ratings on their sense of agency, co-presence, and body ownership after each set of trials using questionnaires presented inside the virtual environment. Each participant performed 25 trials (including repetitions), and answered 32 questions during the study (24 Sense of Agency + 4 Co-presence + 4 Body ownership). At the end of the study, participants filled in the post-study SSQ along with the IPQ. Finally, a semi-structured interview was conducted with both participants, which lasted around 15 minutes. During the interviews, we asked participants about their overall impression of the study, their perceptions of the shared motion and the haptic feedback, their impressions regarding the two tasks, and provocations regarding further use cases of virtual co-embodiment. The complete interview guide is provided in Supplementary Material B. Sessions lasted an average of 60 minutes, whereas the within-VR portion took approximately 25 minutes. Each participant was compensated with a €/$10 gift voucher for participating. Our study received approval from our institute’s ethics and data protection committee, where we also followed any guidelines pertaining to any prevailing cleanliness (cf., COVID-19) regulations.

We used an inductive thematic analysis (Cooper et al., 2012) approach. First, the lead author extensively reviewed the interview transcripts and recorded videos, generating initial codes and themes. Then, the other authors reviewed the codes and themes for consistency and offered additional themes as needed. Quotes are attributed to participants by indicating which pair (P1-P20) they belonged to, followed by the specific participant (PN-1 or PN-2) where appropriate.

First, we tested only a subset of questions in common co-presence and embodiment questionnaires – while such additional measures could shed further light on the experience of virtual co-embodiment, this was a deliberate design choice to ensure users do not experience fatigue and overload during the study. Second, while we followed closely Jeunet et al. (2018)’s question regarding the sense of agency, we found that some participants may have misinterpreted what was meant by ‘feeling of control’. They judged the question to be related to success in the task rather than actual control over their body movements. Indeed, agency within HCI can have multiple interpretations (see (Bennett et al., 2023) for a review), and we see this as a promising avenue for future work to explore other methods for evaluating the sense of agency in such shared virtual co-embodiment experiences. Third, it may be worthwhile to further extend the basic perceptual crossing paradigm in future work, by systematically investigating how varying the presence and type of haptics-related instructions and training beforehand would influence participants’ shared agency during co-embodiment tasks. Fourth, we restricted ourselves to studying hand ownership, we do not investigate realistic full-body avatar representations (cf., (Fribourg et al., 2021; Kodama et al., 2022)). Furthermore, previous studies have shown that the realism of the avatar (Fribourg et al., 2020) and users’ choice of avatar (Lim and Reeves, 2009) impacts their sense of embodiment. Given our focus was on better understanding the role of haptic feedback and shared control distribution across targeted and free-choice tasks, we kept our study variables to a minimum to avoid blowing up the parameter space. However, this provides an interesting area for further research – does the type of avatar body, or mixed hand representation shared amongst users similarly influences the sense of agency and co-presence? Fifth, it worth exploring how height differences between participants and their reach can impact experiences of shared avatar control. To this end, prior work has developed methods that can generate avatar body characteristics that can adapt to variable heights of participants that can be used (Ye et al., 2022) – this would help ensure that control is distributed precisely between the participants, even if this does not necessarily reflect real-world user characteristics. Finally, given our finding that the type of relationship with another person can influence following and leading behavior (cf., Sec 4.3.3), this opens up opportunities to further examine co-embodiment interactions in different dyad compositions.

Our study explored the impact of including haptic feedback and varying avatar control distribution on users’ sense of agency, co-presence, body ownership and motion synchrony across reaching tasks in a virtual co-embodiment scenario (RQ). To this end, one key objective was to assess how the condition in which participants would receive haptic feedback when their hands overlapped would affect these three factors in scenarios involving shared goals and free choice. Our findings indicate that the presence of haptic feedback yielded a significant effect on the sense of agency during our study, though in unexpected ways. Participants felt a significantly greater sense of agency during conditions without haptic feedback compared to conditions with haptic feedback. Given that haptic feedback is well-suited for conveying non-verbal cues (Moll and Sallnäs, 2009), we expected that haptics would facilitate, not hinder sensori-motor coordination and guidance (Melo et al., 2020). Furthermore, our qualitative findings (Sec 4.3.5) indicated that the vibrotactile patterns within our haptic feedback were perceived to be a hindrance and at times intrusive. Despite that we took care to ensure pleasant vibrotactile patterns through a pre-study, as participants reported, they were reminded of smartphone and smartwatch vibrations and notifications. To interpret this finding, we first note that given our focus on translating elements of perceptual crossing into the avatar co-embodiment paradigm, we restricted our study to the context of autonomous interaction processes during shared perceptual activities. This means that even without conscious awareness of the vibrotactile cues, we expected that such position-aware haptic feedback mechanisms would support shared perceptual experiences, in this case, shared motor activity during targeted and free-choice reaching tasks. However, given the salience of the haptic stimuli, we suspect that haptics may have lowered the sense of agency for participants as they may have felt overwhelmed by the other users’ guidance. This, along with the interplay of control, coordination, and physical attributes, would have then played a role in shaping the strategies the participants used in synchronizing with the other user during the haptic feedback conditions. Together, the foregoing raise cautions about how haptics can be integrated, suggesting that including vibrotactile-based haptic feedback as a positional guidance mechanism in 3D virtual space during such shared control interactions may not be an effective means for improving shared avatar co-embodiment experiences.

Our findings indicate that participants’ reported feelings of control (SoA) do not align with the actual levels of control. We found that participants’ sense of agency increased between 25% and 50% control conditions, while a decrease was observed between 50% and 75% conditions. This result echoes the findings of Kodama et al. (2023) (Kodama et al., 2023), who did not find a clear differentiation between the tested levels of control. Moreover, we found that participants felt a significantly greater sense of agency in Task 1 (targeted) compared with Task 2 (free-choice). Since the conditions were counterbalanced and trials randomized, participants may have had a difficult time to judge absolute control levels, as they had no relative comparison to indicate such experienced control levels. We use absolute judgements for measurement of subjective responses since it is the standard practice across prior work (Hagiwara et al., 2019; Fribourg et al., 2021) that investigates the sense of agency and also provided a means to test if haptics would lead to higher (perceived) sense of agency in a given trial, without referencing back to earlier trials (which may not have had haptics activated). As such, overestimation of control was apparent during Task 1. This lends credence to the findings by Fribourg et al. (2021), who also found that participants perceived a greater sense of agency when the goal was shared compared to situations where participants pursued different goals. Furthermore, participants felt a significantly greater sense of co-presence during Task 2 compared with Task 1, suggesting that strong motion synchronization effects may have diminished the awareness of the other. This was further echoed by participants, where some reported a lack of awareness that they were sharing an avatar with their partners during Task 1. Qualitative analyses of user responses in the perceptual crossing paradigm also highlight that movement synchronization might not always necessarily indicate recognition of each other (Kojima et al., 2017). In our specific implementation, in Task 1 (targeted), it might have been the case that the straight-forward task environment afforded high synchronization, but no real space for active exploration to take place (i.e., similar to the oscillating movements found upon successful recognition of each other in perceptual crossing studies) limiting opportunities to become aware of the other participant. As such, participants’ attention to their partners’ movement and need to explicitly communicate verbally with their partners during Task 2 also indicates that they were more inclined to consciously co-ordinate, compared to the more autonomous interaction that was observed during Task 1.

We also found that participants felt that the movements of the virtual hands were influencing their own movements (Body ownership 2; cf., Sec. 4.2.3) significantly greater during Task 2 compared with Task 1. This was further reflected upon by participants who stated they actively strategized to either exert more control over the virtual hand or to follow its movements during Task 2. Interestingly, similar strategizing about movements when encountering other users are found in the perceptual crossing paradigm, where in some cases users would spontaneously adopt leader and follower roles (Froese et al., 2014b). For example, users may choose to remain stationary and passively receive the other’s touch (Kojima et al., 2017). These parallels are interesting given the stark differences in available sensory information in our tasks compared to the perceptual crossing paradigm. It raises interesting questions about ways in which the basic paradigm can be extended and integrated into more multi-sensory shared virtual environments. The foregoing raise fundamental questions about the nature of shared control and social coordination as we integrate with machines and one another (Mueller et al., 2020): to what extent should we be consciously aware of bodily feedback mechanisms during shared activities? Given the importance of motion synchrony in varying the levels of conscious awareness of the virtual other, to what extent should shared body control systems, whether with humans or machines, leverage this without impeding on users’ sense of perceived and actual agency?