Designing Hand-held Controller-based Handshake Interaction in Social VR and Metaverse

The Social VR testbed was implemented using the Unity (v2021.3 LTS) game engine and its XR Interaction Toolkit plug-in. The VE was populated with 3D assets available free of charge and by custom-created ones modelled with Blender (v3.3 LTS). To let the users move in the VE, the joystick-based continuous locomotion method was used (as implemented in [Cannavò et al. 2020]). The Photon PUN 2 framework and its voice module were exploited to let multiple users operate together and talk in the VE.

The experience was deployed using the OpenXR framework targeting a Meta Quest 2 HMD (desktop-tethered via Air Link) and its bundled controllers. This HMD features a display resolution of 1832 × 1900 pixels per eye, spanning a horizontal 90° FOV with a 120Hz refresh rate. The user interacts with the VE using the controllers provided in bundle with the HMD.

The testbed scenario was designed with a low-poly/cartoonish aesthetic by taking inspiration from popular Social VR platforms like Mozilla Hubs, Roblox, RecRoom, etc. The VE layout was deliberately defined taking as a reference a little village square with a neighborhood of commercial activities (Fig. 2c).

The avatars follow the same essential, minimalist style, with sphere-shaped heads, no body, and stylized hands (Fig. 1-2a -2b) for the sake of compelling the user to focus on the gestural part of a handshaking experience.

In order to design the variants of the handshake interaction, the following phases of the gesture were identified and considered:

• Handshake Start (S): One of the two parties gets close to the other and initiates the handshake, while the other party is notified. The first party can decide to wait for the other party to grant the handshake (next phase) or cancel it before it happens. The other party can decide to decline the invitation or accept it (next phase);
  
• Handshake Grant (G): The other party accepts the invitation to handshake and completes it;
  
• Handshake Feedback (F): Both parties receive feedback on the successfully completed handshake.
  

For the implementation, a distance of 1.5 m among the parties was set as the minimum threshold for starting the handshake. If one of the parties gets outside of this range, the handshake is automatically considered as declined/cancelled. When designing the variants, the S and G phases were kept coherent, reaching the four designs described below (an example is given in Fig. 1).

• Handshake 1 (HS1): this variant was designed following the most common mechanics in Social VR platforms in which interaction with a user interface is used to S/G the handshake [Haans and IJsselsteijn 2006]. When the parties are within the handshake threshold distance, a floating menu is shown above the dominant hand of the user. The menu carries a button that can be pressed via ray-cast interaction to perform the S. The button turns into a cancel button for the party that initiated the handshake. When S, an animation of the first party's right hand extending and changing colour to light blue is played, and a menu is shown to the other party under the head of the avatar that initiated the handshake with a grant button to be pressed. A predefined animation in which both right hands get in touch and handshake three times with a sound effect is used for F.
  
• Handshake 2 (HS2): this variant was designed by considering minimal modifications to HS1 potentially allowing a higher comfort and efficiency. In particular, the GUI-based ray-cast interactions were replaced by others based on physical buttons on controllers. The menus (S/G) show up as before, but the user can S by pressing and holding one of the top buttons on the controller. If the button is released, the handshake is cancelled. The other party can G by pressing the same button on the controller. F is as before.
  
• Handshake 3 (HS3): this variant was designed with naturalness and fidelity of the handshake in mind, at the cost of potentially sacrificing learnability and control. The idea was to allow the user to directly grab the other party's hand to S. To perform the grabbing, the user is asked to reach the hand and subsequently press-holding the grip button on the controller. As before, if the button is released, the handshake is cancelled. It is worth noting that once the other party's hand is grabbed, the two users are provided with a discordant visualisation of their own virtual hand. In fact, from the perspective of the S user s/he believes to grab the actual hand controlled by the other user but actually the hand s/he sees attached to its own is a placeholder one. The other party still retains control on its own hand (not seeing neither the placeholder nor his/her own hand as controlled by the S user), and is notified via a GUI (and via the HS1 blue hand extending animation) that it is possible to G. To G, the party is mandated to grab the S user's extended hand. This implementation was devised to provide a more polite experience and reduce the risk of having the G user to perceive lower control and embodiment. F was kept as before (the hands will be already in the correct position for the animation), not to introduce further modifications to this variant.
  
• Handshake 4 (HS4): this latter variant was implemented as HS3, but with the aim to maximise naturalness and fidelity the F was modified by allowing the handshake to be freely performed without a predefined animation. Specifically, G is slightly modified so that the handshake is considered as granted not as soon as hands are mutually grabbed but when both users initiate a handshake gesture (moving the hand up and down while both parties are press-holding the grip button). F retained the sound effect notification whereas the animation was replaced with the possibility for the parties to handshake (i.e. controlling and moving) their own hands at will while seeing the hand of the other avatar attached to their own. This variant is shown in Fig. 1.
  

Videos depicting the four handshakes from both parties’ perspectives are available as supplemental material.

A handshaking activity was arranged in the testbed in order to have all the participants in the user study to experience all the possible actions of a given design. The participants were briefly instructed on the use of the variant using pre-made videos illustrating the functioning of the interaction. However, they were not allowed to practice with another avatar in this tutorial stage, since factors such as learnability and intuitiveness were going to be measured. After the tutorial, the participants were allowed to experience the handshake both with two NPCs and a UCA (which was operated by a confederate during the experimental evaluation). The two NPCs were programmed so that the first one (acting as the village's major) will never start a handshake but can grant it (or decline, 25% of the times), whereas the other one (acting as a retailer) will always start the handshake and let the user grant it (or decline). Likewise, the UCA is operated to let the participants experience the whole spectrum of actions. Furthermore, it was decided that once a handshake is successfully completed, the UCA moves ways and gets back after 30 seconds. However, to prime the illusion that, each time, the participants are interacting with an avatar controlled by a different human being, the colour of the avatar changes (in a predefined order, identical for all the participants) at every new appearance.

The spectrum of actions (six), obtained by combining the possible events, is as follows: the participant starts and cancels the handshake before the UCA/NPC grants it; the participant starts and the UCA/NPC declines; the participant starts and the UCA/NPC grants (successful handshake); the UCA/NPC starts and cancels before the participant grants; the UCA/NPC starts and the participant declines; the UCA/NPC starts and the participant grants (successful handshake). It was decided to have the participants experience at least two times each action with both the NPCs and the UCA, for a total of 36 actions. The average time to complete the handshaking activity was about 12 minutes.

The study was arranged by following a within-subject design. Prior to performing the handshaking activity, the participants were allowed to familiarise with locomotion in the VE using a sandbox scenario (the same used for the handshaking activity but without other avatars). Afterwards, they were asked to perform the handshaking activity with the four variants as described in Section 3.1. Latin square order of exposition was adopted to minimise possible biases and counterbalance learning effects. An a-priori power analysis was performed using the G*Power tool [Faul et al. 2009] to identify the required sample size. Setting α = 0.05 and aiming at detecting at least an effect size of medium entity (Cohen's f ≥ 0.25), it was found that a sample of 24 participants was adequate to reach a power of (1 − β) = 0.81 for the arranged study design [Cohen 1977]. Hence, 24 volunteers were recruited in the authors’ network of contacts, as well as among the staff and students at the authors’ university.

The statistical significance of the results was investigated by using the Friedman test with Conover correction and the Wilcoxon signed-ranks test as post-hoc by means of the RealStatistics tool (v7.3).

The four variants were deemed comparable in terms of embodiment, presence and spatial presence. No significant differences were found in the embodiment measure ($\bar{x}_{HS1-4}=4.95\pm 0.24$,p-value  = .113), including the related subscales: body ownership ($\bar{x}_{HS1-4}=5.29\pm 0.24$,p-value  = .112), body location ($\bar{x}_{HS1-4}=3.89\pm 0.43$,p-value  = .392), and agency ($\bar{x}_{HS1-4}=5.69\pm 0.23$, p-value  = .058). Also in terms of presence (IPQ), no significant differences were identified among the variants ($\bar{x}_{HS1-4}=4.33\pm 0.15$,p-value  = .392). Spatial presence (NMMS) was found as not significantly different overall ($\bar{x}_{HS1-4}=6.12\pm 0.09$,p-value  = .401) as well as in its subscales: mutual awareness ($\bar{x}_{HS1-4}=6.79\pm 0.09$,p-value  = .392), empathy ($\bar{x}_{HS1-4}=5.16\pm 0.21$,p-value  = .194), behavioural interdependence ($\bar{x}_{HS1-4}=5.33\pm 0.27$,p-value  = .598), mutual assistance ($\bar{x}_{HS1-4}=6.75\pm 0.13$,p-value  = .999). An exception is made by the dependent action subscale (p-value  = .037, $\bar{x}_{HS1}=1.50\pm 0.29$, $\bar{x}_{H2}=1.52\pm 0.30$, $\bar{x}_{HS3}=1.40\pm 0.27$, $\bar{x}_{HS4}=1.33\pm 0.26$), according to which a small but significant difference was found for HS1/HS4 (p-value  = 0.041, $\eta _{p}^{2}=0.28$) and HS2/HS4 (p-value  = 0.041, $\eta _{p}^{2}=0.28$), thus indicating that HS4 was perceived to allow slightly more freedom compared to HS1 and HS2 for which, instead, a more fundamental role for completing the handshake interaction was attributed to the other avatar. The results about usability were statistically significant for most of the factors investigated, highlighting an overall trend for scores, i.e. HS4 > HS3 > HS2 > HS1. This is shown by SUS results in Table 1, which are confirmed by the overall usability measured through the aggregate scale of the VR-USE (Fig. 3a).

The functionality scores follow the same trend, whereas in terms of error correction and robustness significant differences were found only for HS4 against the other three variants, suggesting that the lack of the handshake animation could have influenced the perception of robustness of the HS4 variant. The animation had no significant effects instead on the appropriateness dimension. Ease of use and system performance followed the general trend, as for intuitiveness and learnability for which, however, no significant differences were spotted between HS2 and HS3.

A more complete picture can be obtained by looking at the handshake experience (Fig. 3b). Although for the overall score, fidelity, efficiency and effect on bonding with the other avatar the general trend was confirmed, it is interesting to notice that no significant differences in the perception of who is leading the handshake were found. Linked to this latter result is the score about the assessment of the other avatar's personality through the handshaking: only HS1 was judged as significantly worse compared to HS3 and HS4, which seems to indicate that the usage of the GUI could act as a communication barrier. Regarding the comfort, as it may be expected, no significant differences were observed between HS3 and HS4, which were anyhow surprisingly rated as the best-performing variants in this regard.

Finally, by analysing the explicit ranks in Table 2 some additional considerations can be made.

For instance, the difference between H3 and H4 was not significant in those factors somehow related with the end of the handshake (#2, #3, #4, #5), which seems to indicate that the introduction of the different completion feedback did not play a role in the perception of the task completion. On the contrary, the lack of a predefined animation made the HS4 to be perceived as more controllable, less distracting, and associated with a higher level of embodiment (w.r.t. the hands) compared to HS3.

The above analysis seems to indicate that designers of Social VR platforms should consider limiting the use of GUI-based handshake interaction methods in favour of the other approaches proposed. Thus, interaction based on controller's buttons (HS2, HS3, HS4) shall be preferred when possible w.r.t. to HS1. Even though the allocation of functions mapped to a given button would be more complicated in an actual implementation of a Social VR platform than in the testbed used for the current study, difficulties could be mitigated by adopting context-aware practices, e.g., by relying on users’ respective positions, or on interaction history (has a handshake ever been performed before between these two users?). Furthermore, the introduction of a natural gesture appears to contribute in a non-negligible way to the handshake experience. Hence, designers should consider limiting the use of predefined animations in favour of more unconstrained interaction and possibly support gesture detection.