“Ruofei is a highly skilled and motivated researcher. I have first worked with him when we were both teaching assistant, then we did research in the same group. He is very autonomous and organized, and has shown that he can get things done quickly and efficiently. He is great at getting himself acquainted with scientific literature in a new area as well as keeping up with the most cutting-edge things. He is also superior in coding tasks, excels at several programming languages, and knows how to use a great deal of code libraries as well as how to quickly pick up new ones. All-in-all, Ruofei is a top-class computer science researcher.”
Ruofei Du
San Francisco, California, United States
2K followers
500+ connections
About
Ruofei Du serves as the Interactive Perception & Graphics Lead / Manager at Google AR…
Activity
-
My sabbatical has started—the first of my career. To mark the occassion and allow my mind to wander and think, I ran 5 contiguous days of 10ks. This…
My sabbatical has started—the first of my career. To mark the occassion and allow my mind to wander and think, I ran 5 contiguous days of 10ks. This…
Liked by Ruofei Du
-
Excited to showcase our research with Google DeepMind in bringing Gemini to Glasses #projectastra #GoogleIO
Excited to showcase our research with Google DeepMind in bringing Gemini to Glasses #projectastra #GoogleIO
Liked by Ruofei Du
-
Remember that demo I made that allows you to run Google's #Gemma2B model in browser beyond just a chat interface using #WebAI entirely client side in…
Remember that demo I made that allows you to run Google's #Gemma2B model in browser beyond just a chat interface using #WebAI entirely client side in…
Liked by Ruofei Du
Experience
Education
-
University of Maryland College Park
Activities and Societies: Augmentarium, Graphics and Visual Informatics Laboratory (GVIL), MBRC, Makeability Lab, Human-Computer Interaction Lab (HCIL), UMIACS
-
Activities and Societies: Vice President of ACM Class '09, SJTU. Volunteer Star Award at the World EXPO 2010. Teaching Assistant for CS484: Programming Practice for Computing Algorithm, Summer 2011. Research Assistant advised by Prof. Bao-liang Lu
Volunteer Experience
-
Associate Chair for CHI 2021-2024
ACM SIGCHI
Science and Technology
Engineering Interactive Systems & Technologies subcommittee.
-
Associate Chair for UIST 2022-2024
ACM UIST
Science and Technology
-
-
Publications
-
Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media
ACM
Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates…
Social media in virtual reality is in a high-growth market segment with influential products and services in virtual tourism, remote education, and business meetings. Nevertheless, previous systems have never achieved an online platform which renders a 6DoF mirrored world with geotagged social media in real time. In this paper, we introduce the technical detail behind Geollery.com which reconstructs a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. Our system integrates geotagged social media from both internal users and external sources such as Twitter, Yelp, and Flicker. We validate our real-time strategies of Geollery.com on various platforms including mobile phones, workstations, and head-mounted displays.
Other authorsSee publication -
Geollery: A Mixed Reality Social Media Platform
CHI 2019
We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We…
We present Geollery, an interactive mixed reality social media platform for creating, sharing, and exploring geotagged information. Geollery introduces a real-time pipeline to progressively render an interactive mirrored world with three-dimensional (3D) buildings, internal user-generated content, and external geotagged social media. This mirrored world allows users to see, chat, and collaborate with remote participants with the same spatial context in an immersive virtual environment. We describe the system architecture of Geollery, its key interactive capabilities, and our design decisions. Finally, we conduct a user study with 20 participants to qualitatively compare Geollery with another social media system, Social Street View. Based on the participants' responses, we discuss the benefits and drawbacks of each system and derive key insights for designing an interactive mirrored world with geotagged social media. User feedback from our study reveals several use cases for Geollery including travel planning, virtual meetings, and family gathering.
Other authorsSee publication -
ORC Layout: Adaptive GUI Layout with OR-Constraints
CHI 2019
We propose a novel approach for constraint-based graphical user interface (GUI) layout based on OR-constraints (ORC) in standard soft/hard linear constraint systems. ORC layout unifies grid layout and flow layout, supporting both their features as well as cases where grid and flow layouts individually fail. We describe ORC design patterns that enable designers to safely create flexible layouts that work across different screen sizes and orientations. We also present the ORC Editor, a GUI editor…
We propose a novel approach for constraint-based graphical user interface (GUI) layout based on OR-constraints (ORC) in standard soft/hard linear constraint systems. ORC layout unifies grid layout and flow layout, supporting both their features as well as cases where grid and flow layouts individually fail. We describe ORC design patterns that enable designers to safely create flexible layouts that work across different screen sizes and orientations. We also present the ORC Editor, a GUI editor that enables designers to apply ORC in a safe and effective manner, mixing grid, flow and new ORC layout features as appropriate. We demonstrate that our prototype can adapt layouts to screens with different aspect ratios with only a single layout specification, easing the burden of GUI maintenance. Finally, we show that ORC specifications can be modified interactively and solved efficiently at runtime.
Other authors -
Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video Textures
Journal of Computer Graphics Techniques
The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries…
The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution towards real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to their approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.
-
LUCSS: Language-based User-customized Colourization of Scene Sketches
We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text…
We introduce LUCSS, a language-based system for interactive colorization of scene sketches, based on their semantic understanding. LUCSS is built upon deep neural networks trained via a large-scale repository of scene sketches and cartoon-style color images with text descriptions. It consists of three sequential modules. First, given a scene sketch, the segmentation module automatically partitions an input sketch into individual object instances. Next, the captioning module generates the text description with spatial relationships based on the instance-level segmentation results. Finally, the interactive colorization module allows users to edit the caption and produce colored images based on the altered caption. Our experiments show the effectiveness of our approach and the desirability of its components to alternative choices.
-
SketchyScene: Richly-Annotated Scene Sketches
ECCV 2018
We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All…
We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.
Other authorsSee publication -
Kernel Foveated Rendering
Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D)
Foveated rendering coupled with eye-tracking has the potential to dramatically accelerate interactive 3D graphics with minimal loss of perceptual detail. In this paper, we parameterize foveated rendering by embedding polynomial kernel functions in the classic log-polar mapping. Our GPU-driven technique uses closed-form, parameterized foveation that mimics the distribution of photoreceptors in the human retina. We present a simple two-pass kernel foveated rendering (KFR) pipeline that maps well…
Foveated rendering coupled with eye-tracking has the potential to dramatically accelerate interactive 3D graphics with minimal loss of perceptual detail. In this paper, we parameterize foveated rendering by embedding polynomial kernel functions in the classic log-polar mapping. Our GPU-driven technique uses closed-form, parameterized foveation that mimics the distribution of photoreceptors in the human retina. We present a simple two-pass kernel foveated rendering (KFR) pipeline that maps well onto modern GPUs. In the first pass, we compute the kernel log-polar transformation and render to a reduced-resolution buffer. In the second pass, we carry out the inverse-log-polar transformation with anti-aliasing to map the reduced-resolution rendering to the fullresolution screen. We have carried out pilot and formal user studies to empirically identify the KFR parameters. We observe a 2.8X − 3.2X speedup in rendering on 4K U HD (2160p) displays with minimal perceptual loss of detail. The relevance of eye-tracking-guided kernel foveated rendering can only increase as the anticipated rise of display resolution makes it ever more difficult to resolve the mutually conflicting goals of interactive rendering and perceptual realism.
Other authorsSee publication -
Montage4D: Interactive Seamless Fusion of Multiview Video Textures
ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D)
The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries…
The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution towards real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to their approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.
Other authorsSee publication -
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
In Proceedings of the 21st Annual ACM SIGGRAPH Conference on Web3D
This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context…
This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context provided by immersive maps, such as Google Street View. This paper presents new algorithms for fusing and laying out the social media in an aesthetically pleasing manner with geospatial renderings, validates them with respect to visual saliency metrics, suggests spatio-temporal filters, and presents a system architecture that is able to stream geo-tagged social media and render it across a range of display platforms spanning tablets, desktops, head-mounted displays, and large-area room-sized curved tiled displays. The paper concludes by exploring several potential use cases including immersive social storytelling, learning about culture and crowd-sourced tourism.
Other authorsSee publication -
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment
In Proceedings of the 21st Annual ACM SIGGRAPH Conference on Web3D
Video Fields system fuses multiple videos, camera-world matrices from a calibration interface, static 3D models, as well as satellite imagery into a novel dynamic virtual environment. Video Fields integrates automatic segmentation of moving entities during the rendering pass and achieves view-dependent rendering in two ways: early pruning and deferred pruning. Video Fields takes advantage of the WebGL and WebVR technology to achieve cross-platform compatibility across smart phones, tablets…
Video Fields system fuses multiple videos, camera-world matrices from a calibration interface, static 3D models, as well as satellite imagery into a novel dynamic virtual environment. Video Fields integrates automatic segmentation of moving entities during the rendering pass and achieves view-dependent rendering in two ways: early pruning and deferred pruning. Video Fields takes advantage of the WebGL and WebVR technology to achieve cross-platform compatibility across smart phones, tablets, desktops, high-resolution tiled curved displays, as well as virtual reality head-mounted displays.
Other authorsSee publication -
Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras
ACM Transactions on Accessible Computing (TACCESS)
-
VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality
ACM Conference on Human Factors in Computing Systems (CHI), EA 2016
We present VRSurus, a smart device designed to recognize the puppeteer's gestures and render tactile feedback to enhance the interactivity of physical puppets in virtual reality (VR). VRSurus is wireless, self-contained, and small enough to be mounted upon any physical puppets. Using machine-learning techniques, VRSurus is able to recognize three gestures: swiping, shaking and thrusting. Actuators ({em e.g.,} solenoids, servos and vibration motors) assist with the puppetry visible to the…
We present VRSurus, a smart device designed to recognize the puppeteer's gestures and render tactile feedback to enhance the interactivity of physical puppets in virtual reality (VR). VRSurus is wireless, self-contained, and small enough to be mounted upon any physical puppets. Using machine-learning techniques, VRSurus is able to recognize three gestures: swiping, shaking and thrusting. Actuators ({em e.g.,} solenoids, servos and vibration motors) assist with the puppetry visible to the audience and provide tactile feedback on the puppeteer's forearm. As a proof of concept, we implemented a tangible serious VR game using VRSurus that aimed at inspiring children to protect the environment and demonstrated it at the ACM UIST 2015 Student Innovation Contest. Our 3D models, circuitry and the source code are publicly available at www.vrsurus.com at https://github.com/ruofeidu/ninjaterp.
Other authorsSee publication -
AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden
ACM Conference on Human Factors in Computing Systems (CHI), EA 2015
A Zen garden, also known as Japanese rock garden or Ryoanji garden, creates a peaceful way to visualize space and tranquility. In this paper, we introduce AtmoSPHERE, a new method for automatically imbuing a Zen garden with properties of its surrounding space and occupants. AtmoSPHERE uses a Microsoft Kinect to monitor and extract movement in a room and then visualizes representations of this movement physically via sand traces on a custom built XY servo sandbox table. We present our prototype…
A Zen garden, also known as Japanese rock garden or Ryoanji garden, creates a peaceful way to visualize space and tranquility. In this paper, we introduce AtmoSPHERE, a new method for automatically imbuing a Zen garden with properties of its surrounding space and occupants. AtmoSPHERE uses a Microsoft Kinect to monitor and extract movement in a room and then visualizes representations of this movement physically via sand traces on a custom built XY servo sandbox table. We present our prototype system, the design process and interaction modes, feedback from a preliminary deployment, and a discussion of future work.
Other authorsSee publication -
Online Vigilance Analysis Combining Video and Electrooculography Features
International Conference on Neural Information Processing (ICONIP) 2012
It is widely acknowledged that one can never emphasize vigilance too much, especially for drivers, policemen and soldiers. Unfortunately, almost every existing vigilance analysis system suffers from poor illumination, horizon of the cameras, together with various appearance and behaviors of the subjects. In this paper, we propose a novel system to analysis vigilance level combining both video and Electrooculography (EOG) features. For one thing, the video features extracted from an infrared…
It is widely acknowledged that one can never emphasize vigilance too much, especially for drivers, policemen and soldiers. Unfortunately, almost every existing vigilance analysis system suffers from poor illumination, horizon of the cameras, together with various appearance and behaviors of the subjects. In this paper, we propose a novel system to analysis vigilance level combining both video and Electrooculography (EOG) features. For one thing, the video features extracted from an infrared camera include percentage of closure (PERCLOS), eye blinks, slow eye movement (SEM), rapid eye movement (REM), which are also extracted from EOG signals. For another, other features like yawn frequency, body posture and face orientation are extracted from the video based on Active Shape Model (ASM). The results of our experiments indicate that our approach outperforms that based on either video or EOG merely. In addition, the prediction offered by our model is in close proximity to the actual error rate of the subject. We firmly believe that this method can be widely applied to prevent accidents like fatigued driving in the future.
Other authorsSee publication -
Experiencing a Mirrored World With Geotagged Social Media in Geollery
CHI 2019
We demonstrate the online deployment of Geollery, a mixed reality social media platform. We introduce an interactive pipeline to reconstruct a mirrored world at two levels of detail: the street level and the bird's-eye view. Instead of using offline 3D reconstruction approaches, our system streams and renders a mirrored world in real time, while depicting geotagged social media as billboards, balloons, framed photos, and virtual gifts. Geollery allows multiple users to see, chat, and…
We demonstrate the online deployment of Geollery, a mixed reality social media platform. We introduce an interactive pipeline to reconstruct a mirrored world at two levels of detail: the street level and the bird's-eye view. Instead of using offline 3D reconstruction approaches, our system streams and renders a mirrored world in real time, while depicting geotagged social media as billboards, balloons, framed photos, and virtual gifts. Geollery allows multiple users to see, chat, and collaboratively sketch with the spatial context in this mirrored world. We demonstrate a wide range of use cases including crowdsourced tourism, interactive audio guides with immersive spatial context, and meeting remote friends in mixed reality. We envision Geollery will be inspiring and useful as a standalone social media platform for those looking to explore new areas or looking to share their experiences. Please refer to https://geollery.com for the paper and live demos.
-
Interactive Fusion of 360° Images for a Mirrored World
VR 2019
Reconstruction of the physical world in real time has been a grand challenge in computer graphics and 3D vision. In this paper, we introduce an interactive pipeline to reconstruct a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local…
Reconstruction of the physical world in real time has been a grand challenge in computer graphics and 3D vision. In this paper, we introduce an interactive pipeline to reconstruct a mirrored world at two levels of detail. Given a pair of latitude and longitude coordinates, our pipeline streams and caches depth maps, street view panoramas, and building polygons from Google Maps and OpenStreetMap APIs. At a fine level of detail for close-up views, we render textured meshes using adjacent local street views and depth maps. When viewed from afar, we apply projection mappings to 3D geometries extruded from building polygons for a coarse level of detail. In contrast to teleportation, our system allows users to virtually walk through the mirrored world at the street level. We present an application of our approach by incorporating it into a mixed-reality social platform, Geollery, and validate our real-time strategies on various platforms including mobile phones, workstations, and head-mounted displays.
-
The Design and Preliminary Evaluation of a Finger- Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind
European Conference on Computer Vision (ECCV), Workshop on Assistive Computer Vision and Robotics. 2014
We introduce the preliminary design of a novel vision-augmented touch system called HandSight intended to support activities of daily living (ADLs) by sensing and feeding back non-tactile information about the physical world as it is touched. Though we are interested in supporting a range of ADL applications, here we focus specifically on reading printed text. We discuss our
vision for HandSight, describe its current implementation and results from an initial performance analysis of…We introduce the preliminary design of a novel vision-augmented touch system called HandSight intended to support activities of daily living (ADLs) by sensing and feeding back non-tactile information about the physical world as it is touched. Though we are interested in supporting a range of ADL applications, here we focus specifically on reading printed text. We discuss our
vision for HandSight, describe its current implementation and results from an initial performance analysis of finger-based text scanning. We then present a user study with four visually impaired participants (three blind) exploring how to continuously guide a user’s finger across text using three feedback conditions (haptic, audio, and both). Though preliminary, our results show that participants
valued the ability to access printed material, and that, in contrast to previous findings, audio finger guidance may result in the best reading performance.Other authorsSee publication -
Tracking-Tolerent Visual Cryptography
VR 2019
We introduce a novel secure display system, which uses visual cryptography with tolerance for tracking. Our system brings cryptographic privacy from text to virtual worlds. Much like traditional encryption that uses a public key and a private key, our system uses two images that are both necessary for visual decryption of the data. The public image could be widely shared on a printed page, on a traditional display (desktop, tablet, or smartphone), or in a multi-participant virtual world, while…
We introduce a novel secure display system, which uses visual cryptography with tolerance for tracking. Our system brings cryptographic privacy from text to virtual worlds. Much like traditional encryption that uses a public key and a private key, our system uses two images that are both necessary for visual decryption of the data. The public image could be widely shared on a printed page, on a traditional display (desktop, tablet, or smartphone), or in a multi-participant virtual world, while the other private image can be exclusively on a user's personal AR or VR display. Only the recipient with both images is able to visually decrypt the data by fusing them. In contrast to prior art, our system is able to provide tracking tolerance, making it more practically usable in modern VR and AR systems. We model the probability of misalignment caused by head or body jitter as a Gaussian distribution. Our algorithm diffuses the second image using the normalized probabilities, thus enabling the visual cryptography to be tolerant of alignment errors due to tracking.
Patents
Courses
-
Advanced Algebra II
MA067
-
Advanced Computer Graphics
CMSC 740
-
Advanced Topics in Information Processing
CMSC828B
-
Advanced Topics in Programming Languages; Tangible Interactive Computing
CMSC838F
-
Advanced Topics in Visual Computing
CMSC828B
-
Algorithm Analysis and Design
CS475
-
Analog Circuit
EI218
-
Artificial Intelligence
CS410
-
Artificial Intelligence Planning
CMSC722
-
Basic Japanese II
JA002
-
Compiler Principles
CS308
-
Computational Geometry
CMSC754
-
Computational Linguistic II
CMSC773
-
Computational Systems Biology
CMSC702
-
Computer Network
CS339
-
Computer Network Lab
CS379
-
Computer Organization Lab
CS203
-
Computer Organization and Architecture
CS202
-
Data Structure
CS361
-
Digital Logic
CS204
-
Graduate Seminar in Computer Science
CMSC798E
-
Graph Theory and Combinatoric
CS477
-
How to Conduct Research
CMSC798
-
Human-Computer Interface and Computer Graphcis
SE309
-
Information Visualization
CMSC734
-
Innovation and Entrepreneurship
AM923
-
Internet Economics and Management
EC008
-
Mathemtics for Computer Science II
MS104
-
Modern Calculus and Analysis II
MA058
-
Neural Network Theory and Applicatoin
CS483
-
Nuclear Energy Development and Perspective
SP060
-
Object-Oriented Analysis and Design
CS201
-
Operating System
CS307
-
Photography and Its Application in Engineering
ME906
-
Physics II
PH002
-
Physics Lab II
PH029
-
Pictorial Information Processing
CMSC733
-
Principles of Database System
CS304
-
Probability
MA206
-
Professional Practice
CS486
-
Project Management and its Application in IT
ME473
-
Project Workshop of Compiler Principles
CS216
-
Project Workshop of Operating System
CS376
-
Scientific Computing I
CMSC660
-
Set Theory and Mathematical Logic
CS026
-
The Theory of Computability
CS338
-
Undergraduate Project Thesis
BS054
Projects
-
Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context…
This paper presents an immersive geo-spatial social media system for virtual and augmented reality environments. With the rapid growth of photo-sharing social media sites such as Flickr, Pinterest, and Instagram, geo-tagged photographs are now ubiquitous. However, the current systems for their navigation are unsatisfyingly one- or two-dimensional. In this paper, we present our prototype system, Social Street View, which renders the geo-tagged social media in its natural geo-spatial context provided by immersive maps, such as Google Street View. This paper presents new algorithms for fusing and laying out the social media in an aesthetically pleasing manner with geospatial renderings, validates them with respect to visual saliency metrics, suggests spatio-temporal filters, and presents a system architecture that is able to stream geo-tagged social media and render it across a range of display platforms spanning tablets, desktops, head-mounted displays, and large-area room-sized curved tiled displays. The paper concludes by exploring several potential use cases including immersive social storytelling, learning about culture and crowd-sourced tourism.
Other creatorsSee project -
3D Reconstruction on Portable Devices
- Present
3D Reconstruction is a technology that empower people turn physical objects into high-quality 3D models with the normal RGB camera capturing on mobile device and support for 3D printing. In this project we intend to leverage the technology to portable devices. The challenge of the interaction design is how to offer the user the visual feedback and guidance to capture the entire surface of targeted object through a natural and intuitive way to enhance the user experience to lower the barriers…
3D Reconstruction is a technology that empower people turn physical objects into high-quality 3D models with the normal RGB camera capturing on mobile device and support for 3D printing. In this project we intend to leverage the technology to portable devices. The challenge of the interaction design is how to offer the user the visual feedback and guidance to capture the entire surface of targeted object through a natural and intuitive way to enhance the user experience to lower the barriers between common consumers and 3D reconstruction. Firstly, we map out the application workflow contain both of systematic backend and user-interface frontend; then we define the visual prompt as a translucent sphere skynet. Based on the augmented reality technology, the sphere could cover the object and blend in environment. According to the dynamic visual guidance and feedback on skynet, user can cover all the segments of the surfaces on the sphere, in the end the capture result could generate 3D models to support scenarios like AR furniture & space arrangement, e-commerce visualization, object recognition & search, etc.
Other creatorsSee project -
3DVAR: From 3D Reconstruction to Virtual & Augmented Reality
-
This video (http://www.duruofei.com/Research/3dvar) demonstrates an exploration from online 3D reconstruction to various applications including virtual reality, holography and augmented reality.
Firstly, we presents how portable 3D reconstruction with interactive segmentation enables ordinary users to create and consume 3D models. Secondly, we demonstrate that users could interact with reconstructed models in the virtual world using proxies and produce awesome artworks with favorite…This video (http://www.duruofei.com/Research/3dvar) demonstrates an exploration from online 3D reconstruction to various applications including virtual reality, holography and augmented reality.
Firstly, we presents how portable 3D reconstruction with interactive segmentation enables ordinary users to create and consume 3D models. Secondly, we demonstrate that users could interact with reconstructed models in the virtual world using proxies and produce awesome artworks with favorite reconstructed models. Moreover, we introduce "Holography" with reconstruction for both advertisement and comparison between virtual and real models. Eventually, we propose that reconstructed models could decorate your room, enrich magazines and innovate story-telling in augmented reality scenarios on the head-mounted or mobile hand-held devices.
In summary, our video presents a promising trend from 3D reconstruction to virtual and augmented reality.Other creatorsSee project
Honors & Awards
-
Best Paper Award
Web3D 2016
Best Paper Award for the paper - Social Street View: Blending Immersive Street Views with Geo-tagged Social Media
-
Best Student Poster Award
I3D 2018
For "A Pilot Study in Spherical Harmonics for Saliency Computation in 360 Videos"
-
Bronze Medalist
NOI 2008
Also ranked 35th in Chinese Team Selection Contest (CTSC 2008) as state representative. Consecutive First Prizes & Top 3 in National Olympiad in Informatics in Province (NOIP), in Heilongjiang state.
Test Scores
-
Graduate Record Examinations (GRE)
Score: 1520/1600
Languages
-
Chinese
Native or bilingual proficiency
-
English
Full professional proficiency
-
Japanese
Elementary proficiency
Recommendations received
1 person has recommended Ruofei
Join now to viewMore activity by Ruofei
-
Amazing blog post to share more about how Apple Intelligence models were built and optimized to run on-device! Many thanks to our great teams at…
Amazing blog post to share more about how Apple Intelligence models were built and optimized to run on-device! Many thanks to our great teams at…
Liked by Ruofei Du
-
Surfacing 3D content anchored to the physical world right inside Google Maps already installed on billions of phones. Make your AR content visible…
Surfacing 3D content anchored to the physical world right inside Google Maps already installed on billions of phones. Make your AR content visible…
Liked by Ruofei Du
-
Congratulations, Ranbir Singh, on being invited to give the commencement address to NCSU Engineering grads! I am sure they will benefit from your…
Congratulations, Ranbir Singh, on being invited to give the commencement address to NCSU Engineering grads! I am sure they will benefit from your…
Liked by Ruofei Du
-
1500+ JavaScript developers came to hear what's going on in #WebAI this year at #JSNation! Such a great crowd and thank you for the thoughtful…
1500+ JavaScript developers came to hear what's going on in #WebAI this year at #JSNation! Such a great crowd and thank you for the thoughtful…
Liked by Ruofei Du
-
I was joyously enthused to give the last "Design@Large" talk of the quarter at UCSD on May 29th. In it, I reflected on my history of studying how…
I was joyously enthused to give the last "Design@Large" talk of the quarter at UCSD on May 29th. In it, I reflected on my history of studying how…
Liked by Ruofei Du
-
It was a pleasure presenting the #AI Keynote and #WebML talks at #GoogleIO Connect #Shanghai to almost 3000 people or so, and many magnitudes more…
It was a pleasure presenting the #AI Keynote and #WebML talks at #GoogleIO Connect #Shanghai to almost 3000 people or so, and many magnitudes more…
Liked by Ruofei Du
-
Missed my #JSNation keynote on the state of #WebAI? The recording is now up! Or maybe you were already there but want to share with your wider team -…
Missed my #JSNation keynote on the state of #WebAI? The recording is now up! Or maybe you were already there but want to share with your wider team -…
Liked by Ruofei Du
-
Congratulations to my PhD student, Kenny Chen, for receiving an Honorable Mention Award at #SIGGRAPH2024! This is our 3rd #SIGGRAPH Tech Paper Award…
Congratulations to my PhD student, Kenny Chen, for receiving an Honorable Mention Award at #SIGGRAPH2024! This is our 3rd #SIGGRAPH Tech Paper Award…
Liked by Ruofei Du
-
I am excited to share what we have been working on over the past year. MLCEngine, is a universal LLM deployment engine that brings the power of…
I am excited to share what we have been working on over the past year. MLCEngine, is a universal LLM deployment engine that brings the power of…
Liked by Ruofei Du
-
My time as a student researcher at Google AR & VR has come to an end. Big thanks to all colleagues and friends for making this an amazing…
My time as a student researcher at Google AR & VR has come to an end. Big thanks to all colleagues and friends for making this an amazing…
Liked by Ruofei Du
-
🎉🎉🎉🎉🎉🎉 so excited to see this live!! Great work team!
🎉🎉🎉🎉🎉🎉 so excited to see this live!! Great work team!
Liked by Ruofei Du
-
David Li's (Ph.D. '24, computer science) next stop is Google! He'll be working as a software engineer on the augmented reality team with Ruofei…
David Li's (Ph.D. '24, computer science) next stop is Google! He'll be working as a software engineer on the augmented reality team with Ruofei…
Liked by Ruofei Du
Other similar profiles
Explore collaborative articles
We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
Explore MoreOthers named Ruofei Du in United States
-
Ruofei Du
Writer and editor at UCSD
-
Ruofei Du
Assistant Professor at UAMS
-
Ruofei Du
--
-
Ruofei Du
Graduated from University of Pittsburgh School of Health and Rehabilitation Sciences
7 others named Ruofei Du in United States are on LinkedIn
See others named Ruofei Du