I am Ruofei Du,
a Research Scientist,
at Google, San Francisco.

Welcome to ruofei's reality! Ruofei Du is a Senior Research Scientist at Google and works on creating novel interactive technologies for virtual and augmented reality. Du's research covers a wide range of topics in VR and AR, including AR interaction (DepthLab, Ad hoc UI), augmented communication (GazeChat), VR social platforms (Geollery), video-based rendering (Montage4D), foveated rendering (KFR, 360 video), accessibility (ProtoSound, HandSight), and deep learning in graphics (HumanGPS, SketchColorization). Du serves as an Associate Chair in program committes of CHI and UIST; an Associate Editor of Frontiers in Virtual Reality and IEEE TCSVT. In their own words: I am passionate about inventing interactive technologies with graphics, perception, and HCI. See my research, artsy, projects, youtube, talks, github, and shadertoy demos for fun! bio

• 5/11/22: Our work on live translation in AR was the closing feature at the Google I/O 2022 keynote!
• 5/10/22: We launched Portrait Depth API in TensorFlow.js, which turns a single photo into 3D!
• 5/9/22: DepthLab 2.1 is launched on Google Play Store with long range depth support! More news...

Prior to UMD, I received my B.S. degree from ACM Class, Computer Science at Shanghai Jiao Tong University (SJTU). I served as the vice president of the ACM Class and a undergraduate research assistant in the BCMI Lab. From Jul. 2012 to Feb. 2013, I worked as a full-time intern at Microsoft Research Asia working on 3DVAR: From 3D reconstruction To Virtual and Augmented Reality while serving as the Chief Technology Officer at the ARD Intern Community of Microsoft Asia-Pacific Research & Development Group.

• CV
• Publications
• Projects
• Artsy

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video, slides, code, demo, supp | abstract | cited by, cite

Geollery: A Mixed Reality Social Media Platform

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world

pdf, doi | website, project, video, slides, demo | abstract | cited by, cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

Fusing Multimedia Data Into Dynamic Virtual Environments

Ruofei Du

Ph.D. Dissertation, Computer Science Department., University of Maryland, College Park., 2018.

Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world

pdf, lowres, doi | project, slides | abstract | cited by, cite

Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI

Ruofei Du, Alex Olwal, Mathieu Goc, Shengzhi Wu, Danhang Tang, Yinda Zhang, Jun Zhang, David Tan, Federico Tombari, and David Kim

Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

Keywords: augmented reality, everyday objects, tangible user interface, 3D user interface, 6 DoF, spatial interaction, markerless tracking, tangible interaction, hand gestures

pdf, lowres, doi | project, video | abstract | cited by, cite

OmniSyn: Intermediate View Synthesis Between Wide-Baseline Panoramas

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du

2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2022.

Keywords: 360 image, virtual reality, view synthesis, panorama, neural rendering, depth map, mesh rendering, inpainting

pdf | project | abstract | cited by, cite

“Slurp” Revisited: Using Software Reconstruction to Reflect on Spatial Interactivity and Locative Media

Shengzhi Wu, Daragh Byrne, Ruofei Du, and Molly Steenson

Proceedings of the Designing Interactive Systems Conference (DIS), 2022.

Keywords: Historical Precedents; Software econstruction; Re-presencing; Augmented Reality; Gestural Interface; Metaphor; Affordances; Spatial Interaction

pdf, lowres | project, video | abstract | cited by, cite

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

Keywords: accessibility, deaf, Deaf, hard of hearing, sound awareness

pdf, lowres, doi | website, project, video, code | abstract | cited by, cite

GazeChat: Enhancing Virtual Conferences With Gaze-Aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation

pdf, doi | website, project, video, code | abstract | cited by, cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Haene, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

Keywords: deep implicit functions, neural representation, compression, levels of detail

pdf, lowres, doi | project, code | abstract | cited by, cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

Keywords: correspondences, geodesic distance, embeddings, neural networks

pdf, lowres, doi | website, project, video, code, demo, supp | abstract | cited by, cite

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table

pdf, doi | website, project, video, slides, code, data, supp | abstract | cited by, cite

Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP), 2021.

Keywords: deep learning, image compression

pdf, doi | project | abstract | cited by, cite

Saliency Computation for Virtual Cinematography in 360° Videos

Ruofei Du and Amitabh Varshney

IEEE Computer Graphics and Applications (CGA), 2021.

Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA

pdf, doi | website, project | abstract | cited by, cite

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication

pdf, doi | project, video, slides, code | abstract | cited by, cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization

pdf, lowres, doi | website, project, video, code, data | abstract | cited by, cite

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video | abstract | cited by, cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

Keywords: scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field

pdf, lowres, doi | project, video | abstract | cited by, cite

Eye-Dominance-Guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking

pdf, doi | project, video, code, supp | abstract | cited by, cite

Language-Based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization

pdf, lowres, doi | website, project, video, slides, code, data, supp | abstract | cited by, cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: GUI builder, layout manager, constraint-based layout, visual interface design, visual programming

pdf, doi | project, video, code | abstract | cited by, cite

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays

pdf, lowres, doi | website, project, video, slides, code, demo | abstract | cited by, cite

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world

pdf, lowres, doi | project, video, slides | abstract | cited by, cite

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cited by, cite

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang

European Conference on Computer Vision (ECCV), 2018.

Keywords: sketch dataset, scene sketch, sketch segmentation

pdf, lowres, doi | website, project, code, data, demo | abstract | cited by, cite

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, and Jon Froehlich

ACM Transactions on Accessible Computing (TACCESS), 2016.

Keywords: accessibility, real-time OCR, visual impairments, wearables

pdf, doi | website, project, video, slides | abstract | cited by, cite

Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment

Ruofei Du, Sujal Bista, and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR

pdf, lowres, doi | project, video, slides, data | abstract | cited by, cite

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.

Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world

pdf, doi | project, video | abstract | cited by, cite

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world

pdf, doi | project, video | abstract | cited by, cite

Tracking-Tolerent Visual Cryptography

Ruofei Du, Eric Lee, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: visual cryptography, augmented reality (AR), tracking

pdf, doi | project, video, data | abstract | cited by, cite

VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality

Ruofei Du and Liang He

Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.

Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display

pdf, lowres, doi | project, video, demo | abstract | cited by, cite

AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden

Ruofei Du, Kent Wills, Max Potasznik, and Jon Froehlich

Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.

Keywords: Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect

pdf, doi | website, project, video, code | abstract | cited by, cite

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon Froehlich

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.

Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind

pdf, doi | website, project, video, code | abstract | cited by, cite

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon Froehlich

Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.

Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch

pdf, doi | website, project, video, code | abstract | cited by, cite

Online Vigilance Analysis Combining Video and Electrooculography Features

Ruofei Du, Renjie Liu, Tianxiang Wu, and Bao-Liang Lu

Neural Information Processing - 19th International Conference (ICONIP), 2012.

Keywords: Vigilance Analysis Fatigue Detection Active Shape Model Electrooculography Support Vector Machine

pdf, doi | project, video | abstract | cited by, cite

Featured Publications

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented RealityApp Demo

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

---

pdf, doi | website, project, video, slides, code, demo, supp | cited by, cite

Geollery: A Mixed Reality Social Media PlatformOnline Demo of a Metaverse of Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

---

pdf, doi | website, project, video, slides, demo | cited by, cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

---

pdf, lowres, doi | website, project, video, slides | cited by, cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social MediaBest Paper Award

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

---

pdf, lowres, doi | website, project, video, slides | cited by, cite

Recent Publications

Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI

Ruofei Du, Alex Olwal, Mathieu Goc, Shengzhi Wu, Danhang Tang, Yinda Zhang, Jun Zhang, David Tan, Federico Tombari, and David Kim

Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

---

pdf, lowres, doi | project, video | cited by, cite

OmniSyn: Intermediate View Synthesis Between Wide-Baseline Panoramas

David Li, Yinda Zhang, Christian Häne, Danhang Tang, Amitabh Varshney, and Ruofei Du

2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2022.

---

pdf | project | cited by, cite

“Slurp” Revisited: Using Software Reconstruction to Reflect on Spatial Interactivity and Locative Media

Shengzhi Wu, Daragh Byrne, Ruofei Du, and Molly Steenson

Proceedings of the Designing Interactive Systems Conference (DIS), 2022.

---

pdf, lowres | project, video | cited by, cite

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

---

pdf, lowres, doi | website, project, video, code | cited by, cite

GazeChat: Enhancing Virtual Conferences With Gaze-Aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

---

pdf, doi | website, project, video, code | cited by, cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Haene, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

---

pdf, lowres, doi | project, code | cited by, cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

---

pdf, lowres, doi | website, project, video, code, demo, supp | cited by, cite

A Log-Rectilinear Transformation for Foveated 360-Degree Video StreamingTVCG Honorable Mentions

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

---

pdf, doi | website, project, video, slides, code, data, supp | cited by, cite

Random Artsy Work

Professional Services

• Associate Editor of Frontiers in Virtual Reality.
• Associate Editor of IEEE Transactions on Circuits and Systems for Video Technology.
• Associate Chair on the program committee of UIST 2022.
• Associate Chair on the Blending Interaction: Engineering Interactive Systems and Tools committee of CHI 2022.
• Associate Chair on the Engineering Interactive Systems and Technologies committee of CHI 2021.
• Associate Chair on the Late Breaking Work committee of CHI 2021.
• Associate Chair on the Late Breaking Work committee of CHI 2020.
• Chair for the "AI: Content Generation" session at CHI 2022.
• Chair for the "AR and VR (C)" session at CHI 2021.
• Chair for the "Augmented and Virtual Reality" session at Web3D 2019.
• Chair for the "Making the Virtual Physical" session at CHI 2019.
• Reviewer for over 200 journal and conference papers on SIGGRAPH, UIST, CHI, ISMAR, VR, etc.
• Student Volunteer for CHI 2014.

Calendar

Stay In Touch