I am Ruofei Du,
a Research Scientist,
at Google, San Francisco.

Welcome to ruofei's reality! Ruofei Du is a Senior Research Scientist at Google and works on creating novel interactive technologies for virtual and augmented reality. Du's research covers a wide range of topics in VR and AR, including depth-based interaction (DepthLab), mixed-reality social platforms (Geollery and Social Street View), 4D video-based rendering (Montage4D and VideoFields), foveated rendering (KFR, EFR, Foveated360, GazeChat), and deep learning in graphics (HumanGPS and SketchColorization). Du served as a committee member in CHI, SIGGRAPH Asia XR, ICMI and an Associate Editor of Frontiers in Virtual Reality. Du holds a Ph.D. in Computer Science from University of Maryland, College Park. In their own words: I am passionate about inventing interactive technologies with computer graphics, 3D vision, and HCI. Feel free to visit my research, artsy, projects, youtube, talks, github, and shadertoy demos for fun! bio

• 12/9/21: To serve as an Associate Chair on the program committee of UIST 2022.
• 12/2/21: Presented an invited talk on metaverse to University of Calgary.
• 11/15/21: Our paper on personalized sound recognition system is conditionally accepted in CHI 2022. More news...

Prior to UMD, I received my B.S. degree from ACM Class, Computer Science at Shanghai Jiao Tong University (SJTU). I served as the vice president of the ACM Class and a undergraduate research assistant in the BCMI Lab. From Jul. 2012 to Feb. 2013, I worked as a full-time intern at Microsoft Research Asia working on 3DVAR: From 3D reconstruction To Virtual and Augmented Reality while serving as the Chief Technology Officer at the ARD Intern Community of Microsoft Asia-Pacific Research & Development Group.

• CV
• Publications
• Projects
• Artsy

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

{{_Keywords}}depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video, slides, code, demo, supp | abstract | cite

Geollery: A Mixed Reality Social Media Platform

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

{{_Keywords}}metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world

pdf, doi | website, project, video, slides, demo | abstract | cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

{{_Keywords}}texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

{{_Keywords}}metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins

pdf, lowres, doi | website, project, video, slides | abstract | cite

Fusing Multimedia Data Into Dynamic Virtual Environments

Ruofei Du

Ph.D. Dissertation, Computer Science Department., University of Maryland, College Park., 2018.

{{_Keywords}}social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world

pdf, lowres, doi | project, slides | abstract | cite

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

{{_Keywords}}accessibility, deaf, Deaf, hard of hearing, sound awareness

to appear | website, project, code | abstract | cite

GazeChat: Enhancing Virtual Conferences With Gaze-Aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

{{_Keywords}}eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction

pdf, doi | website, project, video, code | abstract | cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Haene, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

{{_Keywords}}deep implicit functions, neural representation, compression, levels of detail

pdf, lowres | project, code | abstract | cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

{{_Keywords}}correspondences, geodesic distance, embeddings, neural networks

pdf, lowres | website, project, video, code, demo, supp | abstract | cite

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

{{_Keywords}}360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table

pdf, doi | website, project, video, slides, code, data, supp | abstract | cite

Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP), 2021.

{{_Keywords}}deep learning, image compression

pdf, doi | project | abstract | cite

Saliency Computation for Virtual Cinematography in 360° Videos

Ruofei Du and Amitabh Varshney

IEEE Computer Graphics and Applications (CGA), 2021.

{{_Keywords}}spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA

pdf, doi | website, project | abstract | cite

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

{{_Keywords}}chalktalk, virtual reality, collaborative work, layout, telepresence, communication

pdf, doi | project, video, slides, code | abstract | cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

{{_Keywords}}light field, foveated rendering, microscopic light field, eye tracking, visualization

pdf, lowres, doi | website, project, video, code, data | abstract | cite

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

{{_Keywords}}depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, project, video | abstract | cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

{{_Keywords}}scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field

pdf, lowres, doi | project, video | abstract | cite

Eye-Dominance-Guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

{{_Keywords}}virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking

pdf, doi | project, video, code, supp | abstract | cite

Language-Based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

{{_Keywords}}deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization

pdf, lowres, doi | website, project, video, slides, code, data, supp | abstract | cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

{{_Keywords}}GUI builder, layout manager, constraint-based layout, visual interface design, visual programming

pdf, doi | project, video, code | abstract | cite

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

{{_Keywords}}foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays

pdf, lowres, doi | website, project, video, slides, code, demo | abstract | cite

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

{{_Keywords}}virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world

pdf, lowres, doi | project, video, slides | abstract | cite

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

{{_Keywords}}texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, project, video, slides | abstract | cite

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang

European Conference on Computer Vision (ECCV), 2018.

{{_Keywords}}sketch dataset, scene sketch, sketch segmentation

pdf, lowres, doi | website, project, code, data, demo | abstract | cite

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, and Jon Froehlich

ACM Transactions on Accessible Computing (TACCESS), 2016.

{{_Keywords}}accessibility, real-time OCR, visual impairments, wearables

pdf, doi | website, project, video, slides | abstract | cite

Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment

Ruofei Du, Sujal Bista, and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

{{_Keywords}}virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR

pdf, lowres, doi | project, video, slides, data | abstract | cite

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.

{{_Keywords}}virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world

pdf, doi | project, video | abstract | cite

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

{{_Keywords}}virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world

pdf, doi | project, video | abstract | cite

Tracking-Tolerent Visual Cryptography

Ruofei Du, Eric Lee, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

{{_Keywords}}visual cryptography, augmented reality (AR), tracking

pdf, doi | project, video, data | abstract | cite

VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality

Ruofei Du and Liang He

Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.

{{_Keywords}}Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display

pdf, lowres, doi | project, video, demo | abstract | cite

AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden

Ruofei Du, Kent Wills, Max Potasznik, and Jon Froehlich

Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.

{{_Keywords}}Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect

pdf, doi | website, project, video, code | abstract | cite

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon Froehlich

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.

{{_Keywords}}Accessibility, Wearables, Real-time OCR, Text Reading for Blind

pdf, doi | website, project, video, code | abstract | cite

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon Froehlich

Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS), 2015.

{{_Keywords}}Blind; visually impaired; wearable computing; computer vision; vision-augmented touch

pdf, doi | website, project, video, code | abstract | cite

Online Vigilance Analysis Combining Video and Electrooculography Features

Ruofei Du, Renjie Liu, Tianxiang Wu, and Bao-Liang Lu

Neural Information Processing - 19th International Conference (ICONIP), 2012.

{{_Keywords}}Vigilance Analysis Fatigue Detection Active Shape Model Electrooculography Support Vector Machine

pdf, doi | project, video | abstract | cite

Featured Publications

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented RealityApp Demo

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

---

pdf, doi | website, project, video, slides, code, demo, supp | cited by, cite

Geollery: A Mixed Reality Social Media PlatformOnline Demo of a Metaverse of Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

---

pdf, doi | website, project, video, slides, demo | cited by, cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

---

pdf, lowres, doi | website, project, video, slides | cited by, cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social MediaBest Paper Award

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

---

pdf, lowres, doi | website, project, video, slides | cited by, cite

Recent Publications

ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users

Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon Froehlich

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.

---

to appear | website, project, code | cited by, cite

GazeChat: Enhancing Virtual Conferences With Gaze-Aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin

Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.

---

pdf, doi | website, project, video, code | cited by, cite

Multiresolution Deep Implicit Functions for 3D Shape Representation

Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Haene, Ruofei Du, Cem Keskin, and Danhang Tang

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

---

pdf, lowres | project, code | cited by, cite

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

---

pdf, lowres | website, project, video, code, demo, supp | cited by, cite

A Log-Rectilinear Transformation for Foveated 360-Degree Video StreamingTVCG Honorable Mentions

David Li, Ruofei Du, Adharsh Babu, Camelia Brumar, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.

---

pdf, doi | website, project, video, slides, code, data, supp | cited by, cite

Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec

Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello

2021 IEEE International Conference on Image Processing (ICIP), 2021.

---

pdf, doi | project | cited by, cite

Saliency Computation for Virtual Cinematography in 360° Videos

Ruofei Du and Amitabh Varshney

IEEE Computer Graphics and Applications (CGA), 2021.

---

pdf, doi | website, project | cited by, cite

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

---

pdf, doi | project, video, slides, code | cited by, cite

Random Artsy Work

Professional Services

• Associate Editor of Frontiers in Virtual Reality.
• Associate Chair on the program committee of UIST 2022.
• Associate Chair on the Blending Interaction: Engineering Interactive Systems and Tools committee of CHI 2022.
• Associate Chair on the Engineering Interactive Systems and Technologies committee of CHI 2021.
• Associate Chair on the Late Breaking Work committee of CHI 2021.
• Associate Chair on the Late Breaking Work committee of CHI 2020.
• Chair for the "AR and VR (C)" session at CHI 2021.
• Chair for the "Augmented and Virtual Reality" session at Web3D 2019.
• Chair for the "Making the Virtual Physical" session at CHI 2019.
• Reviewer for over 150 journal and conference papers on SIGGRAPH, UIST, CHI, ISMAR, VR, etc.
• Student Volunteer for CHI 2014.

Calendar

Stay In Touch