2.1 Affordances and Extensions to the Body
The relationships between one’s body and the environment present opportunities for action, or
affordances [
20,
21]. For example, a doorway affords passage if it is wider than the shoulders, and an object affords grasping if it is smaller than the hand. In other words, the perception of affordances is calibrated, or scaled, to relevant properties of the body [
46]. We are able to flexibly adjust affordance perception to account for changes in body dimensions or abilities in everyday life, such as carrying a wide object through a doorway [
43] or locomoting in a wheelchair [
26]. Studies on the perception of reaching capability indicate that when users wield a tool, the tool becomes incorporated into their body schemas. Thus, their reaching space becomes “re-mapped” to reflect their new ability [
3,
35] and objects that the tool may interact with can be perceived as closer [
47]. For the affordance of walking through a doorway, when users hold a rod that effectively widens their body, affordance perception recalibrates to accommodate the
person-plus-object (PPO) system (e.g., Wagman and Taylor [
45]) and their perception of the size of apertures that they are asked to consider walking through decreases [
43]. Users are also able to adjust their affordance perception to accommodate locomoting through an aperture in a wheelchair [
25,
26], and when wearing a backpack to squeeze sideways through an aperture [
12,
13,
14]. This research, all done in the real world, provides evidence for claims that observers can flexibly incorporate new objects into their body schemas and perception for action capabilities is then updated accordingly.
Extended or mixed realities (XR) have offered new tools to test the perception of affordances with changes to the body. XR encompasses AR and VR, including environments that contain both real and virtual components [
38,
42]. AR superimposes virtual objects onto the real world, whereas VR immerses the user in a visually virtual environment. Given its ability to portray a virtual body with different dimensions than an observer’s real physical body, VR has been used most often to manipulate body extensions or changes in size in order to assess the effect on affordance perception. For example, VR has been used to present larger hands [
34], longer arms [
6], and bigger feet [
28] in order to test the effects of displaying a change to virtual body size on affordance perception for grasping, reaching, and stepping. Most relevant to the current experiment is the work of Bhargava and colleagues who tested whether the presence of an avatar affected judgments of passability through a virtual aperture while wielding a virtual object [
5]. They found that rendering a self-avatar produced more realistic passability judgments compared to the no-avatar condition. These types of body manipulations would be very difficult, if not impossible, to achieve in the real world where users can see their true physical body size at all times. In AR, users see their bodies and can ascertain how it relates to an object they are carrying. In fully virtual environments, the body must be rendered. In the current study, we examine whether rendering just the object in VR will allow for an understanding of body size through feedback from action that may transfer to an AR environment.
2.2 Testing Feedback Effects on Perceived Affordances in XR
When an actor interacts with their environment, they receive information about the relationships between their body and features of the environment after performing an action. This information, or feedback, calibrates affordance perception. For example, in real environments participants who wore blocks on their feet, thereby increasing their height, could calibrate their affordance perception for sitting in a chair [
36] or walking under a barrier [
44] through feedback from postural sway about their new height. Postural sway is an example of exploratory feedback—a type of feedback that does not provide explicit information about the outcome of the afforded action. Participants wearing blocks in Mark et al. [
36] did not need to practice the action of sitting nor did they receive feedback about the accuracy of making sit-ability judgments. Yet their judgments did calibrate to their new height while wearing the blocks. However, exploratory feedback is not always sufficient for action calibration to occur [
12]. Often, feedback that explicitly reveals the outcome of the action is required for calibration. Thus, the bulk of research on affordance perception calibration has focused on
outcome feedback, which provides explicit information about the outcome of an action that can be judged statically or through action. With static outcome feedback, participants make a judgment about whether they can or cannot complete the action (without actually performing it), and then they receive visual or auditory feedback about the accuracy of that judgment. For action outcome feedback, the participant makes a judgment and then performs the afforded action, which provides feedback about the success or failure of their judgment as that action unfolds. Franchak et al. [
15] has shown that giving action-based outcome feedback may better calibrate passing through judgments than static feedback.
Affordance perception calibration for walking through a doorway without turning, but while holding an object, has been studied in mediated environments (VR and AR) as well as in the real world. At baseline, viewers tend to overestimate the smallest doorway width that affords passage when not holding an object [
32,
46]. Previous work in the real world and in VR shows that participants wielding an object are sensitive to the addition of an object to their bodies and can rescale judgments of passability accordingly [
5,
23,
45] (but see Petrucci et al. [
39]). Participants judging passability from a static viewpoint in the real world can accurately calibrate their judgments when either seeing a rod that increases their horizontal width or carrying it without seeing it [
45]. These findings suggest that only visual or only haptic information may be sufficient feedback to adjust passability estimates. Bhargava et al. [
5] examined judgments of passability through a doorway while wielding an object in VR. Practice walking through doorways with the object improved the accuracy of subsequent affordance judgments. Moreover, the calibration phase had a larger influence on judgments when an avatar was not present, suggesting that action outcome feedback may be especially important for calibration when visual representation of a virtual body is absent.
An important aspect of successfully judging passability and calibrating judgments is not colliding with the aperture during passage. Kondo et al. [
30] investigated collision-avoidance behavior in older adults who wielded a rod horizontally while walking through real and virtual apertures. In their study, participants completed a pretest in the real world where they practiced walking through apertures while holding a rod without receiving feedback. They then completed a training phase in VR, with the virtual aperture presented via a 3D stereo projection screen rather than an immersive head-mounted display (as in Bhargava et al.). Participants walked in place while the VR image moved to simulate them walking through the virtual aperture. Virtual collisions were indicated visually on the virtual walls that were hit. They then completed a posttest phase in the real world. The researchers found that the number of collisions did not change between pretest and posttest, but participants significantly reduced their body rotation angles for passing through the apertures. Mestre et al. [
37] compared vibrotractile feedback and avatar conditions for calibrating passing through judgments without an additional object. They found that in the absence of any feedback (avatar, haptic), subjects collided with the small virtual apertures in almost 50% of trials. This number of collisions was reduced when feedback or an avatar was added, with the smallest number of collisions occurring when both were present.
Affordance perception and recalibration are just beginning to be explored in AR. By recalibration, we mean the use of feedback to adapt an affordance judgment. Existing affordance perception research in AR has primarily been conducted in optical see-through AR [
16,
17,
18,
40,
41,
48]. Zhao and colleagues used mobile augmented reality (displayed via handheld devices) to assess providing a cue that represented the width of a user in the context of a set of apertures to be judged for passing through [
50]. They found that participants’ judgments of whether apertures were passable or not were improved after viewing the augmented visual cue that provided feedback about the width of one’s shoulders in the context of the virtual aperture. Gagnon and colleagues also recently investigated the role of static and action outcome feedback on passability judgments in the HoloLens 1, an optical see-through AR device [
16,
17]. Gagnon et al. [
16] used the method of adjustment to assess participants’ perception of the smallest aperture width that would be passable from a static position. In feedback trials, participants were presented with an aperture, made a yes/no judgment on whether they could successfully pass, and then the HoloLens provided auditory feedback (“correct” or “incorrect”) on the accuracy of that judgment. Baseline aperture width judgments were overestimated compared to shoulder width, but adjustments got closer to shoulder width following feedback. Thus, the results of this study provide evidence that static verbal feedback can successfully calibrate passing-through judgments in AR.
Gagnon et al. [
17] implemented the same stimuli and trial procedure as their prior work [
16] to investigate whether perceptual-motor collision-based feedback would calibrate passing-through affordance perception similarly to static verbal feedback in AR. Walking through an aperture provides additional perceptual cues, such as optic flow; perhaps the greater availability of cues could calibrate affordance judgments more efficiently than static feedback. However, providing collision-based outcome feedback in XR is challenging due to the virtual nature of the aperture walls. It is obvious if a collision is made while passing through an aperture in the real world because the actor can haptically feel contact with the wall, but this sensory cue is often absent in XR. It can also be difficult to visually perceive a collision in AR due to the opacity of the aperture walls and the limited field of view of the device. In the feedback trials of Gagnon et al. [
17], participants walked through the AR aperture and received auditory outcome feedback indicating whether a collision occurred during passage. Baseline perception of the just-passable aperture width was overestimated, but, contrary to Gagnon et al. [
16], perceived just-passable aperture increased following the collision-based feedback blocks. The authors conjectured that this increase in overestimation was due to the high number of collisions that participants experienced. Taken together, the results of studies by Gagnon and colleagues ([
16,
17]) demonstrate that affordance perception can be recalibrated in AR, but the direction of calibration may depend on the type of feedback that is provided. Additional research is needed to determine how different affordances are perceived in AR and to further examine the extent and type of feedback that is sufficient for recalibration.
2.3 Does Feedback Transfer to Other Environments?
XR can be useful for training skills that may be inconvenient to train in the real world [
29]. These include situations that are unknown or that may be unsafe, such as a future mission to Mars [
24], practice with difficult surgical procedures [
2], or scenarios involving complex manufacturing [
22]. For example, when training collision-avoidance behavior for walking through an aperture, it may be sub-optimal or dangerous for participants to collide with a physical doorway [
30]. If XR is used for training, it is important to evaluate how and when calibration transfers between virtual and real environments as well as whether it transfers across different types of mediated environments in order for the training to be most effective and generalizable.
The existing research on affordance perception calibration transfer has primarily focused on whether calibration transfers to a functionally similar action. In some studies, researchers have found that calibrating perception for one affordance transfers to a similar affordance in the real world. For example, participants who have calibrated their maximum leaping distance show more accurate judgments of maximum stepping distance [
7]. In other cases, calibration does not transfer. As previously described, Kondo et al. [
30] also used a real-world pretest, VR training, and real-world posttest design for training collision-avoidance behavior for walking through virtual apertures. They found that training in VR did not reduce collisions in the real-world posttest, but the training did result in participants making smaller body rotations during passing, suggesting that training in VR does have the potential to transfer some aspects of the perception of action capabilities to a real environment. However, the VR setup in this study was more similar to AR in the sense that a participant saw their real body and a real rod when completing the training task.
To the best of our knowledge, research on the transfer of affordance perception between VR and AR does not exist. However, some research shows that training in VR can transfer to the real world for police training [
4], maintenance procedures [
19], surgical skills [
27], and wayfinding tasks [
31]. Most relevant to the current study, Day et al. [
6] found that calibration to altered reaching capabilities in VR transferred to the real world. Participants completed a pretest phase in the real world, a calibration phase in VR in which they practiced reaching with an extended avatar arm or a normal avatar arm (action outcome feedback), and a posttest phase in the real world. Participants who practiced reaching with the extended arm in VR believed they could reach farther in the posttest phase compared to participants who experienced the unaltered avatar arm. The present research expands upon Day et al. [
6] by investigating whether calibration from action outcome feedback with a different affordance—passability—in VR transfers to AR.