Computational Time Machine

Abstract

Time travel has always been one of the craziest dreams of humanity, which has yet to be achieved. The recent rapid advances in AR/VR, 3D vision, and image synthesis promise an exciting near future where we can freely change our immersive visual experience. This progress opens up unprecedented opportunities to bridge the gap of time and realize this wild dream in a computational way. In this thesis, I propose a computational time machine, simulating the experience of going back in time and re-experiencing the past. In particular, I focused on addressing two key elements of a time-travel experience --- 1) reproducing the appearance of the past as experienced by people at that time, and 2) a sense of copresence as if sharing the same space with historical people and scenes. Many historical people were only ever captured by old, faded, black and white photos. Their original appearance has been distorted due to the limitations of early cameras and the passage of time. I propose a novel technique to simulate traveling back in time with a modern camera to rephotograph famous subjects. Unlike conventional image restoration filters which apply independent operations like artifact-removal, colorization, and superresolution, the algorithm leverages the StyleGAN2 framework to project antique photos into the space of modern high-resolution photos, achieving all of these effects in a unified framework. For copresence, I first address visualizing history in 3D by introducing KeystoneDepth, the largest and most diverse collection of rectified historical stereo image pairs, consisting of tens of thousands of stereographs of people, events, objects, and scenes recorded between 1864 and 1966. In creating this collection, I propose a novel stereo rectification technique based on the unique geometry of antique stereo cameras. To synthesize 3D visualization, I also introduce a self-supervised deep view synthesis technique trained on historical imagery. Second, I tackle reconstructing video footage in 3D, and propose a novel algorithm for estimating geometrically consistent depth for all pixels in a monocular video. This method extracts scene-specific geometric constraints similar to classical structure-from-motion methods, and leverages data-driven geometric priors from a learning-based single-image depth estimation network by fine-tuning this network at test time. My method achieves high accuracy and geometric consistency, and is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. Third, I introduce a simple glasses-free 3D display, which can be built from a tablet computer and a plastic sheet folded into a cone, which I call the Pepper's Cone. This display enables viewing a three-dimensional object over 360 degrees without the use of a head-mounted display or special glasses. While using a curved reflector, our system can produce a perspective-correct image to the viewer by properly pre-distorting the displayed image. The end result is a natural and intuitive interface for inspecting 3D objects, as if they are suspended inside the reflector.