SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing

Due to the development of 3D sensing and modeling techniques, the state-of-the-art mixed reality devices such as Microsoft Hololens have the ability of digitalizing the physical world. This unique feature bridges the gap between virtuality and reality and largely elevates the user experience. Unfortunately, the current solution only performs well if the virtual contents complement the real scene. It can easily cause visual artifacts when the reality needs to be modified due to the virtuality (e.g., remove real objects to offer more space for virtual objects), a common scenario in mixed reality applications such as room redecoration and environment design. We present a novel system, called emph{SceneCtrl}, that allows the user to interactively edit the real scene sensed by Hololens, such that the reality can be adapted to suit virtuality. Our proof-of-concept prototype employs scene reconstruction and understanding to enable efficient editing such as deleting, moving, and copying real objects in the scene. We also demonstrate emph{SceneCtrl} on a number of example scenarios in mixed reality, verifying the enhanced experience by resolving conflicts between virtuality and reality.