Remote collaboration in AR and VR using virtual replicas

In many complex tasks, a remote subject-matter expert may need to assist a local user, to guide their actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We demonstrate an approach that uses Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display (HWD). Our approach allows the remote expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. The remote expert can demonstrate actions in 3D by manipulating virtual replicas, supported by constraints and annotations, and point in 3D to portions of virtual replicas to annotate them.