I am Ruofei Du,
a Research Scientist,
at Google, San Francisco.

Welcome to ruofei's reality! Ruofei Du is a Research Scientist at Google and works on creating novel interactive technologies for VR and AR. Ruofei holds a Ph.D. in Computer Science from University of Maryland, College Park, advised by Prof. Varshney. Their research covers a wide range of topics in VR and AR, including depth-based interaction (DepthLab), mixed-reality social platforms (Geollery and Social Street View), 4D video-based rendering (Montage4D and VideoFields), foveated rendering (KFR and EFR), and deep learning in graphics (SketchySceneColorization). I am passionate about inventing interactive technologies with computer graphics, 3D vision, and HCI. Feel free to visit my research, arts, blog, youtube, github, and shadertoy demos for fun!

• 10/22/20: Presented DepthLab talk and demo on ACM UIST 2020.
• 7/18/20: CollaboVR is conditionally accepted in IEEE ISMAR 2020.
• 6/25/20: Released Android app of Depth Lab and source code on GitHub. More news...

Prior to UMD, I received my B.S. degree from ACM Class, Computer Science at Shanghai Jiao Tong University (SJTU). I served as the vice president of the ACM Class and a undergraduate research assistant in the BCMI Lab. From Jul. 2012 to Feb. 2013, I worked as a full-time intern at Microsoft Research Asia working on 3DVAR: From 3D reconstruction To Virtual and Augmented Reality while serving as the Chief Technology Officer at the ARD Intern Community of Microsoft Asia-Pacific Research & Development Group.

• CV
• Publications
• Projects
• Artsy

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, lowres, doi | website, video, slides, code, demo, supp | abstract | cite

Geollery: A Mixed Reality Social Media Platform

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction

pdf, doi | website, video, slides, demo | abstract | cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, video, slides | abstract | cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social Media

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL

pdf, lowres, doi | website, video, slides | abstract | cite

Fusing Multimedia Data Into Dynamic Virtual Environments

Ruofei Du

Ph.D. Dissertation, Computer Science Department., University of Maryland, College Park., 2018.

Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography

pdf, lowres, doi | project page, slides | abstract | cite

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication

pdf | project page, video | abstract | cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization

pdf, lowres, doi | website, video, data | abstract | cite

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore

pdf, doi | website, video | abstract | cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

Keywords: scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field

pdf, lowres, doi | project page, video | abstract | cite

Eye-Dominance-Guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking

pdf, doi | project page, video, supp | abstract | cite

Language-Based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization

pdf, lowres, doi | website, video, slides, code, data, supp | abstract | cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

Keywords: GUI builder, layout manager, constraint-based layout, visual interface design, visual programming

pdf, lowres, doi | project page, video, code | abstract | cite

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays

pdf, lowres, doi | website, video, slides, demo | abstract | cite

Project Geollery.com: Reconstructing a Live Mirrored World With Geotagged Social Media

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.

Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL

pdf, lowres, doi | project page, video, slides | abstract | cite

Montage4D: Interactive Seamless Fusion of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.

Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields

pdf, lowres, doi | website, video, slides | abstract | cite

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang

European Conference on Computer Vision (ECCV), 2018.

Keywords: sketch dataset, scene sketch, sketch segmentation

pdf, lowres, doi | website, code, data, demo | abstract | cite

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, and Jon Froehlich

ACM Transactions on Accessible Computing (TACCESS), 2016.

Keywords: accessibility, real-time OCR, visual impairments, wearables

pdf, doi | website, video, slides | abstract | cite

Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment

Ruofei Du, Sujal Bista, and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

Keywords: virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR

pdf, lowres, doi | project page, video, slides, data | abstract | cite

Experiencing a Mirrored World With Geotagged Social Media in Geollery

Ruofei Du, David Li, and Amitabh Varshney

Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.

Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction

pdf, doi | project page, video | abstract | cite

Interactive Fusion of 360° Images for a Mirrored World

Ruofei Du, David Li, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world

pdf, doi | project page, video | abstract | cite

Tracking-Tolerent Visual Cryptography

Ruofei Du, Eric Lee, and Amitabh Varshney

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.

Keywords: visual cryptography, augmented reality (AR), tracking

pdf, doi | project page, video, data | abstract | cite

VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality

Ruofei Du and Liang He

Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.

Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display

pdf, lowres, doi | project page, video, demo | abstract | cite

AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden

Ruofei Du, Kent Wills, Max Potasznik, and Jon Froehlich

Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.

Keywords: Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect

pdf, doi | website, video | abstract | cite

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Leah Findlater, Rama Chellappa, and Jon Froehlich

Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.

Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind

pdf, doi | website, video | abstract | cite

Supporting Everyday Activities for Persons With Visual Impairments Through Computer Vision

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, and Jon Froehlich

Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS), 2015.

Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch

pdf, doi | website, video | abstract | cite

Online Vigilance Analysis Combining Video and Electrooculography Features

Ruofei Du, Renjie Liu, Tianxiang Wu, and Bao-Liang Lu

Neural Information Processing - 19th International Conference (ICONIP), 2012.

Keywords: Vigilance Analysis Fatigue Detection Active Shape Model Electrooculography Support Vector Machine

pdf, lowres, doi | project page, video | abstract | cite

Featured Publications

DepthLab: Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

---

pdf, lowres, doi | website, video, slides, code, demo, supp | cite

Geollery: A Mixed Reality Social Media PlatformJuried Demo at CHI 2019

Ruofei Du, David Li, and Amitabh Varshney

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

---

pdf, doi | website, video, slides, demo | cite

Montage4D: Real-Time Seamless Fusion and Stylization of Multiview Video Textures

Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney

Journal of Computer Graphics Techniques (JCGT), 2019.

---

pdf, lowres, doi | website, video, slides | cite

Social Street View: Blending Immersive Street Views With Geo-Tagged Social MediaBest Paper Award

Ruofei Du and Amitabh Varshney

Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.

---

pdf, lowres, doi | website, video, slides | cite

Recent Publications

CollaboVR: A Reconfigurable Framework for Multi-User to Communicate in Virtual Reality

Zhenyi He, Ruofei Du, and Ken Perlin

2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.

---

pdf | project page, video | cite

3D-Kernel Foveated Rendering for Light Fields

Xiaoxu Meng, Ruofei Du, Joseph JaJa, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.

---

pdf, lowres, doi | website, video, data | cite

Experiencing Real-Time 3D Interaction With Depth Maps for Mobile Augmented Reality in DepthLab

Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim

Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.

---

pdf, doi | website, video | cite

MeteoVis: Visualizing Meteorological Events in Virtual Reality

David Li, Eric Lee, Elijah Schwelling, Mason Quick, Patrick Meyers, Ruofei Du, and Amitabh Varshney

Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.

---

pdf, lowres, doi | project page, video | cite

Eye-Dominance-Guided Foveated Rendering

Xiaoxu Meng, Ruofei Du, and Amitabh Varshney

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.

---

pdf, doi | project page, video, supp | cite

Language-Based Colorization of Scene Sketches

Changqing Zou, Haoran Mo, Chengying Gao, Ruofei Du, and Hongbo Fu

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.

---

pdf, lowres, doi | website, video, slides, code, data, supp | cite

ORC Layout: Adaptive GUI Layout With OR-Constraints

Yue Jiang, Ruofei Du, Christof Lutteroth, and Wolfgang Stuerzlinger

Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.

---

pdf, lowres, doi | project page, video, code | cite

Kernel Foveated Rendering

Xiaoxu Meng, Ruofei Du, Matthias Zwicker, and Amitabh Varshney

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.

---

pdf, lowres, doi | website, video, slides, demo | cite

Random Artsy Work

Professional Services

• Associate Chair on the Engineering Interactive Systems and Technologies committee of CHI 2021.
• Associate Chair on the Late Breaking Work committee of CHI 2020.
• Chair for the "Augmented and Virtual Reality" session at Web3D 2019.
• Chair for the "Making the Virtual Physical" session at CHI 2019.
• Reviewer for over 100 journal and conference papers on SIGGRAPH, UIST, CHI, ISMAR, VR, etc.
• Student Volunteer for CHI 2014.

Calendar

Stay In Touch