A Multi-Objective Optimization Framework for Redirecting Pointing Gestures in Remote-Local Mixed/Augmented Reality

Collaborative augmented reality is an emerging field with the promise of simulating natural human-human interactions from remote locations. Users, represented by their photorealistic avatars, and relevant objects in their scenes can be teleported to each other's environments, with the capability of tracking their gaze, body pose, and other nonverbal behaviors using modern augmented reality devices. Pointing gestures play a key role for users to communicate about aspects related to their environment. Also, the body pose one uses during pointing, relays cues of the user's intentions and nonverbal behaviors during the interaction. Due to dissimilarities in multiple users’ environments, pointing gesture needs to be redirected since direct animation of the user's motion onto its avatar may introduce error. At the same time, the nonverbal behavior represented by the body pose also needs to be preserved for realistic interaction. While these objectives are not mutually exclusive, current approaches only solve the redirection of the gesture without preserving the body pose. In this paper, we present a systematic approach to solving the dual problem of redirection of gesture as well as preserving the body pose using a multi-objective optimization framework. The presented framework efficiently adjusts the weighting between the two objectives and gives the user the flexibility to set the minimum angular error tolerance for the pointing gesture redirection. We have tested our approach with the current state-of-the-art using both pointing gesture reference poses and redirecting against continuous gesture actions collected from an augmented reality human participant study. Results show that for a given user-defined error tolerance, our approach has a decrease of 33.5% in body pose error vs current-state-of-the-art for pointing gesture reference poses and 33.6% for pointing gestures recorded during the human participant study.