Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments

Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments Cephise Louison, Fabien R Ferlay, Daniel R. Mestre To cite this version: Cephise Louison, Fabien R Ferlay, Daniel R. Mestre. Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments. 3DUI - IEEE Symposium on 3D User Interfaces, Mar 2017, Los Angeles, United States. pp.99-102, ฀10.1109/3DUI.2017.7893324฀. ฀hal01691483฀ HAL Id: hal-01691483 https://hal.science/hal-01691483 Submitted on 22 Dec 2022 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Spatialized vibrotactile feedback contributes to goal-directed movements in cluttered virtual environments Cephise Louison, Fabien Ferlay, Daniel Mestre To cite this version: Cephise Louison, Fabien Ferlay, Daniel Mestre. Spatialized vibrotactile feedback contributes to goaldirected movements in cluttered virtual environments. 2017 IEEE Symposium on 3D User Interfaces (3DUI), Mar 2017, Los Angeles, United States. pp.99-102, ฀10.1109/3DUI.2017.7893324฀. ฀hal03886902฀ HAL Id: hal-03886902 https://hal.archives-ouvertes.fr/hal-03886902 Submitted on 21 Dec 2022 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Distributed under a Creative Commons Attribution - NonCommercial| 4.0 International License Spatialized Vibrotactile Feedback Contributes to Goal-Directed Movements in Cluttered Virtual Environments Céphise LOUISON* Fabien FERLAY* CEA/IRFM, France Aix-Marseille University/ISM, France CEA/IRFM, France In this context, the lack of kinesthetic (force) feedback for example due to technical constraints, results in incomplete sensorial feedback as soon as the subject interacts with virtual objects. For example, collisions of the body with virtual objects do not typically result in proprioceptive and tactile feedback; nothing is actually there to stop the operator’s movement. To address these limitations, vibrotactile (cutaneous) stimulation appears as a good candidate to substitute for force feedback [1], and make operators aware of impending or actual collisions in highly constrained spaces. We thus hypothesized that vibrotactile feedback, positioned on the operator’s skin, will enhance spatial awareness of elements in the VE. In this paper, we present preliminary results from an experimental study, exploring the contribution of multi-localized vibrators to visuo-proprioceptive consistency, during goaldirected movements in a cluttered virtual environment. ABSTRACT In virtual reality (VR), spatial awareness is a dominant research topic. It plays an essential role in the assessment of human operators’ behavior within virtual environments (VE), notably for the evaluation of the feasibility of manual maintenance tasks in cluttered industrial settings. In such contexts, it is decisive to evaluate the spatial and temporal correspondence between the operator’s movement kinematics and that of his/her virtual avatar in the virtual environment. Often, in a cluttered VE, direct kinesthetic (force) feedback is limited or absent. We tested whether vibrotactile (cutaneous) feedback would increase visuoproprioceptive consistency, spatial awareness, and thus the validity of VR studies, by augmenting the perception of the operator’s contact(s) with virtual objects. We present preliminary experimental results, obtained using a head-mounted display (HMD) during a goal-directed task in a cluttered VE. Data suggest that spatialized vibrotactile feedback contributes to visuoproprioceptive consistency. 2 RELATED WORK AND BACKGROUND Vibration stimulation has long been used as an informational medium in different applications. It can be divided into three main application areas [2]: multimedia and entertainment, abstract information and physical information. This last area directly addresses spatial awareness. In simulated tasks, the use of vibrotactile devices has gained a lot of attention. For instance, Weber et al. [3] noted firstly that the cutaneous information channel is less overloaded than the visual and acoustic channels in complex work scenarios and secondly that vibrotactile stimulation is unobtrusive and informs the operator without disturbing other users. Finally, the stimulation is localized on the operator’s skin. It triggers an intuitive form of spatialized feedback. Moreover, vibrotactile interfaces are light and wearable. Therefore, vibrotactile feedback is a good candidate to substitute for kinesthetic stimulation sense and to make users aware of impending or actual collisions in highly constrained spaces. Recent studies, have successfully investigated the effect of vibrotactile stimulation while interacting with virtual objects. Results suggest that cutaneous stimulation is effective for improving the operator’s perception of impending or effective collisions between the operator’s body and environmental elements within a VE [1, 4–7]. A singular aspect of vibrotactile stimulation is the existence of several perceptual illusions arising from the spatial configuration of vibrators on the operator’s skin. One of these is called the “funneling illusion” [8, 9]. It corresponds to “phantom” localized sensations between two physical stimulations. Moreover, the position of the phantom sensation can be controlled by modulating the amplitude of the two stimulations/vibrations (the distance between the two actuators/stimuli has to be in a range between 40 and 80 mm) [10]. This modulation can be realized using a linear, a logarithmic model [8, 11] or the “energy model” Keywords: Visuo-proprioceptive consistency, Vibrotactile, Goaldirected movements, Cluttered environments. Index Terms: I.3.7 [Computer Graphics]:Three-Dimensional Graphics and Realism—Virtual reality; H.5.2 [User Interfaces]: Haptic I/O 1 Daniel R. MESTRE† Aix-Marseille University/ISM, CNRS, France INTRODUCTION In virtual reality (VR), spatial awareness is an important concept. It plays an essential role in the human operator’s behavioral adaptation to the virtual environment (VE). The operator’s behavior in future facilities has to be studied, to guarantee their viability, in terms of accessibility and maintenance. The operator must often work in a confined environment and has to pay attention to the position of his/her whole body relative to the position of various elements of the VE. In this context, visuoproprioceptive consistency is a key aspect for the validation of behavioral data obtained in VE. We use the term visuo-proprioceptive consistency to refer to the spatio-temporal correspondence between the operator’s and avatar’s kinematics. Spatio-temporal correspondence describes a bi-directional link of interaction. The avatar must adopt the same posture as the operator, being co-localized with the operator. In return, the user must integrate the posture limitations imposed on the avatar by the virtual environment (VE). * {cephise.louison|fabien.ferlay}@cea.fr † daniel.mestre@univ-amu.fr 1 between subjects. Different adjustments and configuration steps were made at the beginning of each session. At the end of this setup, the avatar was optimally co-localized with the subject’s own body, reproducing the subject’s movements in real-time. The subject could perceive the avatar’s body as his own body, from a first-person subjective view (Figure 1, bottom right). Subjects were instructed to reach, with their right hand, targets within the VE, then to maintain the reached position for one second, to validate it. They were asked to do it with the most fluid and natural movement. No specific information was given concerning the behavior to be adopted in case of contact with VE’s objects. During an initial training phase, they could get familiar with the task, with the virtual body and, depending on the condition, with the potential vibrotactile feedback. One trial consisted of the following sequence: the subject was standing with arms along the body, and on the experimenter input, a new target appeared. Following a beep, the subject had to reach for the target. When a trial was validated, another beep was played and the target disappeared. The subject had to take the initial position before another trial could start. Participants were allowed to have a short rest before starting the next trial, to avoid physical exhaustion. The experiment was interrupted by a short break between each session. Each session consisted of twenty trials (five repetitions of four targets). Within a given session, the targets’ succession order was randomized across subjects. The subjects were split into four groups. Subjects were randomly assigned to one group, each group having a different succession order of the four experimental conditions, to control for potential order or time effects (fatigue, learning, etc.). proposed by Israr and Poupyrev [12], which allows more precise control of the intensity and the localization of the phantom sensation. 3 MATERIAL AND METHODS The following experimentation was designed to test the hypothesis that vibrotactile feedback enhances spatial awareness and visuo-proprioceptive consistency. Furthermore, using multilocalized vibrators should improve the precision of the operator’s perception of impending or effective collisions with environmental elements in highly constrained spaces. 3.1 Experimental Conditions The independent variable was the type of vibrotactile feedback, with 4 conditions. The information was either only visual or augmented with vibrotactile stimulation. The vibrotactile feedback was used to represent any contact between the right arm and the environmental elements. There were three different conditions of vibrotactile feedback: nearest, funneling or single (See “Vibration Modes”). These three modes offer different levels of collisions rendering accuracy. No visual metaphor (e.g. contact arrows) was added to indicate collisions. 3.2 Procedure Twenty-eight subjects (8 female) voluntarily took part in this experiment. They were chosen from university students and staff, with ages between 20 and 34 years (mean = 23.6; SD = 3.4). Three volunteers were left-handed, but no differences were found, concerning handedness. All participants were naïve as to the purpose of the experiment. Participants were first informed about the general procedure, and were subsequently equipped with seventeen tracking markers, the vibrotactile device, and the head-mounted display (HMD) (Figure 1, left). 3.3 Virtual Environment The experimental environment represents a part of an industrial facility. The VE was composed of a platform to access to a complex machine made of a support beam, pipes, etc. Four small targets (blue spheres 7 cm in diameter) were positioned in this environment, in the middle of the pipe forest. These targets were positioned within arm’s reach (Figure 1, top right), distributed on the top, bottom, right and left sides. 3.4 Apparatus The experiment was conducted in a square area (3 x 3 meters) at CRVM (www.crvm.eu). Subjects were equipped with an Oculus Rift DK2 device. The tracking system (ArtTrack®), using optical trackers with eight cameras, was used to monitor the subject’s full-body posture and movements (Figure 1, left). The ARTHuman® software was used to reconstruct the subject’s skeleton from the tracked markers’ positions. This body reconstruction was used to calculate a real-time and co-localized virtual representation of the subject’s body in the Unity application (Figure 1, top right). The real-time VR system operated at 60 Hz. Figure 1: Left. A subject in the experimental setup. Top right. Global view of the experimental VE: avatar in the initial position and targets. Bottom right. First person view inside the HMD. A calibration step was required to build a biomechanical model of the subject, using the ART-Human® software (Advanced Realtime Tracking GmbH). A configuration file with the bone lengths was generated and used by the XDE engine (See “Apparatus”) to create a morphological avatar of the subject in the VE. The virtual avatar was made of simple primitives (cylinders) (Figure 1). It has been shown that abstract avatars don’t reduce the illusion of virtual body ownership [13] and they are easier to use for the physical engine computation. The experiment was divided into four sessions, one for each experimental condition, the succession order was counterbalanced Figure 2: Vibrotactile suit with the vibrotactile controller and position of the ten actuators along the arm . 2 individual trial. These positions (markers and avatar) had the same spatial frame of reference and were measured at the center of segments/bones. This integrated value gave us a direct indication of the subject’s behavior with respect to physical objects of the environment. We next computed the mean contact time with environmental objects during a trial. We measured contact time between the moment where a contact point was detected along the upper limb and the moment when no more contact points could be detected. This indicator was meant to analyze how subjects dealt with the occurrence of contacts, as a function of experimental conditions. The analyses were conducted using repeated measures analysis of variance (ANOVA). We then used post-hoc analyses with Bonferroni adjustment and Planned Comparison tests. Moreover, visual feedback (virtual environment and self-avatar) could be augmented by vibrotactile feedback. The vibrotactile device was based on an Arduino-like microcontroller communicating via Bluetooth with the PC on which the simulation was running. The controller addressed ten vibrators, positioned on the subject’s right upper limb (Figure 2); with variable amplitude using Pulse-width modulation (PWM). The controller activated vibrators (DC motor with an eccentric mass, from Parallax Inc. ref. 28822). Due to Bluetooth wireless connection and the vibrator activation, there was a delay between the instruction decision and when the vibration actually occurred. However it was never noticed by the subjects. The VR loop (motion capture to sensorial rendering) was controlled by a PC. The experimental software was developed with Unity3D. It allowed experimental control, data recording and all scenario actions. The XDE physical engine developed by CEA LIST (http://www-list.cea.fr/), integrated into Unity3D, allowed realistic and real-time physical simulation and human control from ART-Human data. 4 RESULTS We first computed the co-localization Root Mean Square Error (RMSE). The error of co-localization refers to the distance between the subject’s positions and the corresponding avatar positions. We computed RMS for the right (reaching) upper limb. The ANOVA with the experimental conditions as the withinsubjects factor revealed a significant effect of conditions (F (3, 81) = 7.5127; p =.00017). Subsequent post-hoc analyses with Bonferroni adjustment showed that RMS values in the three vibrotactile conditions were significantly lower than in the “visual only” condition (p <.005), and that these three conditions were not statistically different from one another (see Figure 3). This result suggests that visuo-proprioceptive consistency was improved by the vibration feedback. Spatialized feedback (funneling or nearest conditions) was not shown to have a significant effect compared to a single vibrator (located on the hand, whatever the contact position). 3.5 Vibration Modes The XDE physical engine allowed the computation of contact points between rigid bodies. The position of the contact, the normal and the force applied was computed for each contact point. The physical engine did not allow interpenetration between rigid bodies (for example, between the avatar and the VE): If a subject tried to walk through a wall in the VE, the avatar was stopped by the (virtual) wall. In that case, the distance between the avatar and the subject’s body was no longer null. It is a case of visuo-proprioceptive inconsistency. The contact information (position and force) was used to compute which vibrator should be activated to represent the contact on the subject’s arm and the amplitude of the vibration. In this experiment, three different conditions (modes) of vibration were used. Each mode had a specific algorithm to select the amplitude of vibrators during a contact. Single: In this mode, only one actuator (actuator ‘0’ on Figure 2, right) was used to represent the existence of a contact between the avatar and the VE, regardless of the position of the contact on the avatar’s arm. The vibration amplitude was mapped according to the contact force. Nearest: In this mode we selected the nearest vibrator to the contact position on the avatar’s arm. The amplitude was related to the force of contact. Several contacts could be represented at the same time by different actuators. Funneling: We decided to use this effect to create the sensation of a contact position that would be closest to the “actual” contact point. If possible, a pair of vibrators such that the contact point was between the positions of the two vibrators was chosen. We applied the energy model [12] to calculate the amplitude of both vibrators, according to the contact position and force. If no pair could be found, we applied the “Nearest” algorithm. Figure 3: Average values (with standard error) of RMS error, as a function of experimental conditions. The mean contact time during a single trial was analyzed. A significant effect of experimental conditions was found (F (3, 81) = 7.2249; p < .001). We observed a trend of decreasing contact time with increasingly localized vibration conditions (Figure 4). Post-hoc analysis (Bonferroni) shows that the “single” condition is not statistically different from the “visual” condition (p=.49). However, both “funneling” and “nearest” conditions gave significantly lower values of mean contact time as compared to the “visual” condition (p <.005). The “funneling” and “nearest” conditions do not differ statistically (p>.50). Finally, Planned Comparisons tests reveals that “Single” is statistically different from the group {“nearest”, “funneling”} (p <.05). This pattern of results suggests that the “single” non-localized vibrator informs the subjects about the occurrence of a contact. However, subjects still have to “search” for the contact location, as in the “visual” condition. As a consequence, they need a little more time to react. In that case, localized vibration (funneling or nearest) directly gives an indication about the contact location, thus reducing contact time. 3.6 Data Analysis During the experiment, several behavioral indicators and events were recorded and analyzed. To reduce noise in the data, we discarded initial failed attempts to reach the target and we kept only the final (and successful) reaching movement for each individual trial. The main behavioral metric was related to the concept of visuoproprioceptive consistency. To analyze that, we computed (for the right upper limb segments) the Root Mean Square (RMS) of the distance between the real subject’s 3D positions (obtained from the tracking system) and the avatar’s positions throughout each 3 “funneling illusion”. Nevertheless, preliminary results from the experimental study failed to show that this phenomenon had a significant effect. Future analysis and work will pursue that line of investigation. ACKNOWLEDGMENTS The authors wish to thank the CRVM team for decisive assistance in the experimental process. We also thank the CEA DEN/MAR/DTEC/SDTC/LSTD and CEA DRT/LIST/DIASI/LSI teams for their involvement in the project. Céphise Louison is supported by a doctoral grant form CEA. Figure 4: Average values (with standard error) of the mean contact time in a single trial, as a function of experimental conditions. 5 REFERENCES [1] DISCUSSION These preliminary results show evidence that individuals make fewer co-localization errors and react faster to the occurrence of a collision when presented with localized vibrotactile feedback. First of all, subjects showed a lower co-localization RMSE in the presence of vibrotactile stimulation. This positive effect can be explained by the fact that vibrotactile feedback increases spatial awareness and visuo-proprioceptive consistency, thus helping the subjects to maintain the spatio-temporal co-localization between their real arm and the virtual arm. This result also suggests that, globally, there is no major statistical difference between the different conditions of vibrotactile feedback. This suggests that, before localized information, vibrators deliver a “warning” signal. However, we also found evidence that, with the multi-localized vibrotactile feedback, subjects made shorter contact times, as compared to a single actuator or without vibrotactile feedback. It appears that the reaction to a contact is faster with localized information. It can be explained by the fact that the multi-localized vibrotactile feedback allows faster localization of a contact point without visual search and a faster reaction to it. This is supported by subjective data (post-hoc open interview). Subjects generally reported that the extra information given by the multi-localized vibrotactile device was relevant to find the position of contact points. Finally, no statistical difference was observed between the two multi-localized vibrotactile feedback conditions (funneling and nearest). The fact that there is no difference between these two algorithms may indicate that an increase in the resolution of the spatialized feedback display is not necessary. It could also be due to the limits of the vibrotactile device used in this experiment or to an insufficient configuration of the funneling algorithm. This question requires further investigation and additional data processing of the kinematics of reaching movements. 6 [2] [3] [4] [5] [6] [7] [8] [9] CONCLUSIONS AND FUTURE WORK [10] Our approach mixed a dynamic and co-localized visual representation (avatar) of the subject’s body and a multi-localized vibrotactile feedback as a substitute for force feedback. These preliminary results suggest that vibrotactile feedback enhances visuo-proprioceptive consistency. Moreover, a localized vibrotactile feedback allows faster localization of contact points by reducing the need of visual search, also favoring visuoproprioceptive consistency. It seems that using vibrotactile feedback to enhance the perception of collisions with virtual elements is a very promising approach to the improvement of spatial awareness of the operator in a cluttered environment. As such, it might be an important factor for the ecological validity of virtual experiments. To finish, a large amount of information can be displayed via tactile means, making use of perceptual phenomena such as the [11] [12] [13] 4 R. W. Lindeman, R. Page, Y. Yanagida, and J. L. Sibert, “Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system”, in Proceedings of the ACM symposium on Virtual reality software and technology, 2004, pp. 146–149. S. Choi and K. J. Kuchenbecker, “Vibrotactile Display: Perception, Technology, and Applications”, Proc. IEEE, vol. 101, no. 9, pp. 2093–2104, Sep. 2013. B. Weber, S. Schätzle, T. Hulin, C. Preusche, and B. Deml, “Evaluation of a vibrotactile feedback device for spatial guidance”, in World Haptics Conference (WHC), 2011 IEEE, 2011, pp. 349–354. A. Bloomfield and N. I. Badler, “Virtual training via vibrotactile arrays”, Presence Teleoperators Virtual Environ., vol. 17, no. 2, pp. 103–120, 2008. C. Louison, F. Ferlay, D. Keller, and D. Mestre, “Vibrotactile Feedback for Collision Awareness”, in Proceedings of the 2015 British HCI Conference, New York, NY, USA, 2015, pp. 277–278. D. R. Mestre, C. Louison, and F. Ferlay, “The Contribution of a Virtual Self and Vibrotactile Feedback to Walking Through Virtual Apertures”, in Human-Computer Interaction. Interaction Platforms and Techniques, M. Kurosu, Ed. Springer International Publishing, 2016, pp. 222–232. J. Ryu and G. J. Kim, “Using a vibro-tactile display for enhanced collision perception and presence”, in Proceedings of the ACM symposium on Virtual reality software and technology, 2004, pp. 89–96. D. S. Alles, “Information Transmission by Phantom Sensations”, IEEE Trans. Man-Mach. Syst., vol. 11, no. 1, pp. 85–91, Mar. 1970. G. v Békésy, “Funneling in the Nervous System and its Role in Loudness and Sensation Intensity on the Skin”, J. Acoust. Soc. Am., vol. 30, no. 5, pp. 399–412, May 1958. J. Cha, L. Rahal, and A. El Saddik, “A pilot study on simulating continuous sensation with two vibrating motors”, in Haptic Audio visual Environments and Games, 2008. HAVE 2008. IEEE International Workshop on, 2008, pp. 143–147. J. Seo and S. Choi, “Initial study for creating linearly moving vibrotactile sensation on mobile device”, in Haptics Symposium, 2010 IEEE, 2010, pp. 67–70. A. Israr and I. Poupyrev, “Tactile brush: drawing on skin with a tactile grid display”, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 2019–2028. J. L. Lugrin, J. Latt, and M. E. Latoschik, “Avatar anthropomorphism and illusion of body ownership in VR”, in 2015 IEEE Virtual Reality (VR), 2015, pp. 229–230.