I am Ruofei Du,
Interactive Perception & Graphics Lead,
at Google.

Welcome to ruofei's reality! Ruofei Du serves as Interactive Perception & Graphics Lead / Manager at Google and devotes to creating novel interactive technologies for XR. His research covers a wide range of topics in technical HCI, Graphics, and Perception, including augmented communication, visual programming, XR interactions, XR social platforms, digital human, foveated rendering, accessibility, and deep learning in graphics. Dr. Du serves as an Associate Chair in program committes of CHI and UIST; an Associate Editor of IEEE TCSVT. In his own words: I am passionate about inventing interactive technologies with graphics, perception, and HCI. See my research, artsy, projects, youtube, talks, github, and shadertoy demos for fun! bio

• 7/2/24: Artificial Object Intelligence is conditionally accepted at UIST 2024. Congrats Doga!
• 6/14/24: We published Human I/O on Google Research Blog, with 50K+ views.
• 5/29/24: We published ChatDirector on Google Research Blog, with 560K+ views.
  More news...

• CV
• Publications
• Projects
• Artsy

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual ProgrammingHonorable Mentions Award, 170K+ views

Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Bruce Liu, Ahmed Sabie, Sergio Orts-Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Olwal

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

Keywords: visual programming, node-graph editor, deep neural networks, data augmentation, deep learning, model comparison, visual analytics, interactive perception

pdf, lowres, doi | website, project, video, slides, code, demo, news | abstract | cited by, cite

Visual Captions: Augmenting Verbal Communication With On-the-fly VisualsOpen Source, Real-time, Live!

Xingyu Bruce Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang 'Anthony' Chen, and Ruofei Du

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, augmented reality, XR interaction

pdf, lowres, doi | project, video, code, data, news | abstract | cited by, cite

DepthLab: Real-time 3D Interaction With Depth Maps for Mobile Augmented Reality50K downloads

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; XR interaction; digital world, interactive graphics

pdf, doi | website, project, video, slides, code, demo, news, supp | abstract | cited by, cite

Geollery: A Mixed Reality Social Media PlatformLive Demo of a Metaverse of Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world; digital world; digital world; augmented communication

pdf, doi | website, project, video, slides, demo, supp | abstract | cited by, cite

SVG: 3D Stereoscopic Video Generation Via Denoising Frame Matrix

Peng Dai, Feitong Tan, Qiangeng Xu, David Futschik, Ruofei Du, Sean Fanello, Xiaojuan Qi, and Yinda Zhang

arxiv, 2024.

Keywords:

to appear | project | abstract | cited by, cite

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic FacesBest Student Paper Award

Safa Medin, Gengyan Li, Ruofei Du, Stephan Garbin, Philip Davidson, Gregory Wornell, Thabo Beeler, and Abhimitra Meka

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2024.

Keywords: Volumetric Rendering, Face Modeling, View Synthesis, PerformanceCapture, digital human

pdf, lowres, doi | website, project, video | abstract | cited by, cite

Human I/O: Towards a Unified Approach to Detecting Situational ImpairmentsBest Paper Honourable Mentions Award

Xingyu Bruce Liu, Jiahao Nick Li, David Kim, Xiang 'Anthony' Chen, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

Keywords: situational impairments, augmented reality, large language models, multimodal sensing, context awareness, XR interaction, interactive perception

pdf, doi | website, project, video, code, news | abstract | cited by, cite

ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition

Xun Qian, Feitong Tan, Yinda Zhang, Brian Moreno Collins, Alex Olwal, David Kim, Karthik Ramani, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

Keywords: augmented communication, video conferencing, 3D portrait avatar, co-presence, attention transition, depth estimation, video-mediated communication

pdf, doi | project, video, news | abstract | cited by, cite

UI Mobility Control in XR: Switching UI Positionings Between Static, Dynamic, and Self Entities

Siyou Pei, David Kim, Alex Olwal, Yang Zhang, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

Keywords: Extended Reality, User Interface, UI Mobility, UI Positioning, XR interaction, interactive graphics

pdf, doi | project, video | abstract | cited by, cite

Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du

Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

Keywords: Visual Programming; Large Language Models; Visual Prototyping;Node-graph Editor; Graph Compiler; Low-code Development; DeepNeural Networks; Deep Learning; Visual Analytics

pdf, doi | project | abstract | cited by, cite

Sandwiched Compression: Repurposing Standard Codecs With Neural Network Wrappers

Onur Guleryuz, Philip Chou, Berivan Isik, Hugues Hoppe, Danhang Tang, Ruofei Du, Jonathan Taylor, Philip Davidson, and Sean Fanello

arxiv, 2024.

Keywords: deep learning, image compression, nonlineartransform coding, high dynamic range, super-resolution, interactive perception

pdf, lowres, doi | project, code | abstract | cited by, cite

Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video TexturesMicrosoft TechFest 2018

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields; digital human

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social MediaBest Paper Award

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins; digital world

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

Fusing Multimedia Data Into Dynamic Virtual Environments

Ruofei Du

Ph.D. Dissertation, Computer Science Department., University of Maryland, College Park., 2018.

Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world

pdf, lowres, doi | project, slides | abstract | cited by, cite

InstructPipe: Building Visual Programming Pipelines With Human Instructions

Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du

https://arxiv.org/abs/2312.09672, 2023.

Keywords: Visual Programming; Large Language Models; Visual Prototyping; Nodegraph Editor; Graph Compiler; Low-code Development; Deep Neural Networks; Deep Learning; Visual Analytics; Interactive Perception

pdf, lowres, doi | project | abstract | cited by, cite

ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Erzhen Hu, Jens Emil Sloth Grønbæk, Wen Ying, Ruofei Du, and Seongkook Heo

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

Keywords: video-mediated communication, object-centered meetings, online meeting, collaborative work, augmented communication, XR interaction

pdf, doi | website, project, video | abstract | cited by, cite

Modeling and Improving Text Stability in Live Captions

Xingyu Bruce Liu, Jun Zhang, Leonardo Ferrer, Susan Xu, Vikas Bahirwani, Boris Smus, Alex Olwal, and Ruofei Du

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.

Keywords: live captions; real-time transcription; visual instability; flickering metric; speech-to-text; text stability; tokenized alignment; augmented communication

pdf, doi | project, video | abstract | cited by, cite

Next Steps for Human-Centered Generative AI: A Technical Perspective

Xiang 'Anthony' Chen, Jeffrey Burke, Ruofei Du, Matthew Hong, Jennifer Jacobs, Philippe Laban, Dingzeyu Li, Nanyun Peng, Karl Willis, Chien-Sheng Wu, and Bolei Zhou

https://arxiv.org/abs/2306.15774, 2023.

Keywords: Generative AI, Human-Centered Design

pdf, doi | project | abstract | cited by, cite

Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos

Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Mingsong Dou, Sergio Orts-Escolano, Rohit Pandey, Ping Tan, Thabo Beeler, Sean Fanello, and Yinda Zhang

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

Keywords: implicit 3D avatar, monocular RGB video, facial expressions, head poses, neural radiance field, photorealism, digital human

pdf, doi | website, project | abstract | cited by, cite

Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines

Ruofei Du, Na Li, Jing Jin, Michelle Carney, Xiuxiu Yuan, Kristen Wright, Mark Sherwood, Jason Mayes, Lin Chen, Jun Jiang, Jingtao Zhou, Zhongyi Zhou, Ping Yu, Adarsh Kowdle, Ram Iyengar, and Alex Olwal

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.

Keywords: visual programming, large language models, visual prototyping, multi-modal models, node-graph editor, deep neural networks, data augmentation, deep learning, visual analytics

pdf, doi | project, slides, code | abstract | cited by, cite

Experiencing Visual Captions: Augmented Communication With Real-time Visuals Using Large Language Models

Xingyu Bruce Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang 'Anthony' Chen, and Ruofei Du

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.

Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, dataset, textto-visual, AI agent, augmented reality

pdf, lowres, doi | project, slides, code | abstract | cited by, cite

Portrait Expression Editing With Mobile Photo Sequence

Yiqin Zhao, Rohit Pandey, Yinda Zhang, Ruofei Du, Feitong Tan, Chetan Ramaiah, Tian Guo, and Sean Fanello

SIGGRAPH Asia 2023 Technical Communications (SA), 2023.

Keywords: Neural rendering, Portrait expression editing, Mobile system

pdf, lowres, doi | project | abstract | cited by, cite

RetroSphere: Self-Contained Passive 3D Controller Tracking for Augmented RealityIMWUT Vol. 6 Distinguished Paper Award

Ananta Balaji, Clayton Kimber, David Li, Shengzhi Wu, Ruofei Du, and David Kim

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2022.

Keywords: Retroreflectors, Augmented reality, Virtual reality, Infrared marker tracking, Augmented reality glasses, XR interaction

pdf, lowres, doi | website, project, video, code | abstract | cited by, cite

Sandwiched Image Compression: Increasing the Resolution and Dynamic Range of Standard CodecsBest Paper Finalist

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2022 Picture Coding Symposium (PCS), 2022.

Keywords: deep learning, image compression, nonlineartransform coding, high dynamic range, super-resolution, interactive perception

pdf, doi | project, code | abstract | cited by, cite

PRIF: Primary Ray-based Implicit Function

Brandon Feng, Yinda Zhang, Danhang Tang, Ruofei Du, and Amitabh Varshney

European Conference on Computer Vision (ECCV), 2022.

Keywords: deep implicit functions, neural representation, signed distance function, interactive perception, interactive graphics

pdf, lowres, doi | website, project, video, supp | abstract | cited by, cite

“Slurp” Revisited: Using Software Reconstruction to Reflect on Spatial Interactivity and Locative Media

Shengzhi Wu, Daragh Byrne, Ruofei Du, and Molly Steenson

Proceedings of the Designing Interactive Systems Conference (DIS), 2022.

Keywords: system re-presencing, affordances, metaphor, software reconstruction, historical precedents, gestural interface, augmented reality, spatial interaction, XR interaction

pdf, lowres, doi | project, video | abstract | cited by, cite

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon E. Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

Keywords: accessibility, deaf, Deaf, hard of hearing, sound awareness

pdf, lowres, doi | website, project, video, code | abstract | cited by, cite

Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI

Ruofei Du, Alex Olwal, Mathieu Goc, Shengzhi Wu, Danhang Tang, Yinda Zhang, Jun Zhang, David Tan, Federico Tombari, and David Kim

Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

Keywords: augmented reality, everyday objects, tangible user interface, 3D user interface, 6 DoF, spatial interaction, markerless tracking, tangible interaction, hand gestures, XR interaction

pdf, lowres, doi | project, video | abstract | cited by, cite

OmniSyn: Intermediate View Synthesis Between Wide-baseline Panoramas

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du

2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2022.

Keywords: 360 image, virtual reality, view synthesis, panorama, neural rendering, depth map, mesh rendering, inpainting, digital world

pdf, doi | website, project, code | abstract | cited by, cite

GazeChat: Enhancing Virtual Conferences With Gaze-aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation, eye tracking, XR interaction

pdf, doi | website, project, video, code | abstract | cited by, cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

Keywords: deep implicit functions, neural representation, compression, levels of detail, MDIF, interactive perception

pdf, lowres, doi | project, code | abstract | cited by, cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

Keywords: correspondences, geodesic distance, embeddings, neural networks, digital human, interactive perception

pdf, lowres, doi | website, project, video, code, demo, supp | abstract | cited by, cite

A Log-Rectilinear Transformation for Foveated 360-degree Video StreamingTVCG Honorable Mentions

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table, eye tracking, digital world

pdf, doi | website, project, video, slides, code, data, supp | abstract | cited by, cite

Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP), 2021.

Keywords: deep learning, image compression, interactive perception

pdf, doi | project, code | abstract | cited by, cite

Saliency Computation for Virtual Cinematography in 360° Videos

Ruofei Du and Amitabh Varshney

IEEE Computer Graphics and Applications (CGA), 2021.

Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA, eye tracking, interactive graphics

pdf, doi | website, project, video | abstract | cited by, cite

CollaboVR: A Reconfigurable Framework for Multi-user to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication, XR interaction, augmented communication

pdf, doi | project, video, slides, code | abstract | cited by, cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization, eye tracking, interactive graphics

pdf, lowres, doi | website, project, video, code, data | abstract | cited by, cite

Experiencing Real-time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; interactive graphics, XR interaction

pdf, doi | website, project, video, slides | abstract | cited by, cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

Keywords: scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field, XR interaction

pdf, lowres, doi | project, video | abstract | cited by, cite

Eye-Dominance-guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking, interactive graphics

pdf, doi | project, video, code, supp | abstract | cited by, cite

Language-based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization, interactive graphics, interactive perception, augmented communication

pdf, lowres, doi | website, project, video, slides, code, data, supp | abstract | cited by, cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: GUI builder, layout manager, constraint-based layout, visual interface design, visual programming, interactive graphics

pdf, doi | project, video, code | abstract | cited by, cite

Kernel Foveated RenderingMost read in PACMCGIT

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays, eye tracking, interactive graphics

pdf, lowres, doi | website, project, video, slides, code, demo | abstract | cited by, cite

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world, interactive graphics, digital world

pdf, lowres, doi | project, video, slides | abstract | cited by, cite

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields, digital human

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang

European Conference on Computer Vision (ECCV), 2018.

Keywords: sketch dataset, scene sketch, sketch segmentation, interactive graphics

pdf, lowres, doi | website, project, code, data, demo | abstract | cited by, cite

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, and Jon E. Froehlich

ACM Transactions on Accessible Computing (TACCESS), 2016.

Keywords: accessibility, real-time OCR, visual impairments, wearables, XR interaction

pdf, doi | website, project, video, slides | abstract | cited by, cite

Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment

Ruofei Du, Sujal Bista, and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR; interactive graphics

pdf, lowres, doi | project, video, slides, data | abstract | cited by, cite

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.

Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world

pdf, doi | project, video | abstract | cited by, cite

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world

pdf, doi | project, video | abstract | cited by, cite

Tracking-Tolerent Visual Cryptography

Ruofei Du, Eric Lee, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: visual cryptography, augmented reality (AR), tracking, XR interaction

pdf, doi | project, video, data | abstract | cited by, cite

VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual RealityDemoed at UIST 2015

Ruofei Du and Liang He

Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.

Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display; XR interaction

pdf, lowres, doi | project, video, code, demo | abstract | cited by, cite

AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden

Ruofei Du, Kent Wills, Max Potasznik, and Jon E. Froehlich

Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.

Keywords: Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect; XR interaction

pdf, doi | website, project, video, code | abstract | cited by, cite

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon E. Froehlich

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.

Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind

pdf, doi | website, project, video, code | abstract | cited by, cite

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon E. Froehlich

Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.

Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch

pdf, doi | website, project, video | abstract | cited by, cite

Online Vigilance Analysis Combining Video and Electrooculography Features

Ruofei Du, Renjie Liu, Tianxiang Wu, and Bao-Liang Lu

Neural Information Processing - 19th International Conference (ICONIP), 2012.

Keywords: Vigilance Analysis, Fatigue Detection, Active Shape Model, Electrooculography, Support Vector Machine, eye tracking

pdf, doi | project, video | abstract | cited by, cite

A Pilot Study of Spherical Harmonics for Saliency Computation and Navigation in 360° Videos

Ruofei Du and Amitabh Varshney

ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA

pdf, doi | project | abstract | cited by, cite

Research on Fatigue Driving Detection System Based on Video Signals

Ruofei Du

Shanghai Jiao Tong University, 2013.

Keywords:

pdf, lowres | project | abstract | cited by, cite

Featured Publications

Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual ProgrammingHonorable Mentions Award, 170K+ views

Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Bruce Liu, Ahmed Sabie, Sergio Orts-Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Olwal

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

---

pdf, lowres, doi | website, project, video, slides, code, demo, news | cited by, cite

Visual Captions: Augmenting Verbal Communication With On-the-fly VisualsOpen Source, Real-time, Live!

Xingyu Bruce Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang 'Anthony' Chen, and Ruofei Du

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

---

pdf, lowres, doi | project, video, code, data, news | cited by, cite

DepthLab: Real-time 3D Interaction With Depth Maps for Mobile Augmented Reality50K downloads

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

---

pdf, doi | website, project, video, slides, code, demo, news, supp | cited by, cite

Geollery: A Mixed Reality Social Media PlatformLive Demo of a Metaverse of Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

---

pdf, doi | website, project, video, slides, demo, supp | cited by, cite

Recent Publications

Augmented Object Intelligence: Making the Analog World Interactable With XR-Objects

Mustafa Doga Dogan, Eric Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaco, Johnny Lee, Mar Gonzalez-Franco, and David Kim

Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024.

---

to appear | website, project | cited by, cite

SVG: 3D Stereoscopic Video Generation Via Denoising Frame Matrix

Peng Dai, Feitong Tan, Qiangeng Xu, David Futschik, Ruofei Du, Sean Fanello, Xiaojuan Qi, and Yinda Zhang

arxiv, 2024.

---

to appear | project | cited by, cite

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic FacesBest Student Paper Award

Safa Medin, Gengyan Li, Ruofei Du, Stephan Garbin, Philip Davidson, Gregory Wornell, Thabo Beeler, and Abhimitra Meka

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2024.

---

pdf, lowres, doi | website, project, video | cited by, cite

Human I/O: Towards a Unified Approach to Detecting Situational ImpairmentsBest Paper Honourable Mentions Award

Xingyu Bruce Liu, Jiahao Nick Li, David Kim, Xiang 'Anthony' Chen, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

---

pdf, doi | website, project, video, code, news | cited by, cite

ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition

Xun Qian, Feitong Tan, Yinda Zhang, Brian Moreno Collins, Alex Olwal, David Kim, Karthik Ramani, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

---

pdf, doi | project, video, news | cited by, cite

UI Mobility Control in XR: Switching UI Positionings Between Static, Dynamic, and Self Entities

Siyou Pei, David Kim, Alex Olwal, Yang Zhang, and Ruofei Du

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

---

pdf, doi | project, video | cited by, cite

Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming

Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du

Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.

---

pdf, doi | project | cited by, cite

Sandwiched Compression: Repurposing Standard Codecs With Neural Network Wrappers

Onur Guleryuz, Philip Chou, Berivan Isik, Hugues Hoppe, Danhang Tang, Ruofei Du, Jonathan Taylor, Philip Davidson, and Sean Fanello

arxiv, 2024.

---

pdf, lowres, doi | project, code | cited by, cite

InstructPipe: Building Visual Programming Pipelines With Human Instructions

Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du

https://arxiv.org/abs/2312.09672, 2023.

---

pdf, lowres, doi | project | cited by, cite

ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings

Erzhen Hu, Jens Emil Sloth Grønbæk, Wen Ying, Ruofei Du, and Seongkook Heo

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.

---

pdf, doi | website, project, video | cited by, cite

Modeling and Improving Text Stability in Live Captions

Xingyu Bruce Liu, Jun Zhang, Leonardo Ferrer, Susan Xu, Vikas Bahirwani, Boris Smus, Alex Olwal, and Ruofei Du

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.

---

pdf, doi | project, video | cited by, cite

Next Steps for Human-Centered Generative AI: A Technical Perspective

Xiang 'Anthony' Chen, Jeffrey Burke, Ruofei Du, Matthew Hong, Jennifer Jacobs, Philippe Laban, Dingzeyu Li, Nanyun Peng, Karl Willis, Chien-Sheng Wu, and Bolei Zhou

https://arxiv.org/abs/2306.15774, 2023.

---

pdf, doi | project | cited by, cite

Random Artsy Work

Professional Services

• Associate Chair on the program committee of UIST 2024.
• Associate Chair on the Blending Interaction: Engineering Interactive Systems and Tools committee of CHI 2024.
• Associate Editor of IEEE Transactions on Circuits and Systems for Video Technology.
• Associate Chair on the program committee of UIST 2023.
• Associate Chair on the Computational Interaction committee of CHI 2023.
• Associate Chair on the program committee of UIST 2022.
• Associate Chair on the Blending Interaction: Engineering Interactive Systems and Tools committee of CHI 2022.
• Associate Chair on the Engineering Interactive Systems and Technologies committee of CHI 2021.
• Chair for the "Write Right: Reading and Writing Tools" session at UIST 2023.
• Chair for the "Learning in VR/AR/XR/Immersive Environments" session at CHI 2023.
• Chair for the "XR Toolkits" session at UIST 2022.
• Chair for the "AI: Content Generation" session at CHI 2022.
• Chair for the "AR and VR (C)" session at CHI 2021.
• Chair for the "Making the Virtual Physical" session at CHI 2019.
• Reviewer for over 300 journal and conference papers on SIGGRAPH, UIST, CHI, ISMAR, VR, etc.

Calendar

Stay In Touch