XR Blocks: Accelerating Human-centered AI + XR InnovationOpensource SDK
arxiv, 2025.
pdf, doi | website, project, video, code, demo, blog | cited by, cite
Invent the Future
Welcome to ruofei's reality! Ruofei Du serves as Interactive Perception & Graphics Lead, XR Labs Manager at Google and devotes to leading AI + XR innovations. His research covers a wide range of topics in technical HCI, AI, Perception, and Graphics, including generative XR experiences, augmented communication, visual programming, XR interactions, social XR, avatars, foveated rendering, accessibility, and deep learning in graphics. Dr. Du serves as an Associate Chair in program committes of CHI and UIST; an Associate Editor of IEEE TCSVT. In his own words: I am passionate about inventing interactive technologies with AI, XR, and HCI. See my research, artsy, projects, youtube, talks, github, and shadertoy demos for fun! bio
- 4/13/26: We presented Vibe Coding XR at the CHI 2026 Google Booth.
- 3/25/26: Check out our latest blog that highlights XR Blocks Gem with educational use cases in XR!
- 3/10/26: Presented "Kicking off the FlyWheel of AI + XR Innovation with XR Blocks" at MIT HCI Seminar.
More news...
Sorry, but no results are found.
XR Blocks: Accelerating Human-centered AI + XR InnovationOpensource SDK
David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Evgenii Alekseev, Geonsun Lee, Alex Cooper, Min Xia, Scott Chung, Jeremy Nelson, Xiuxiu Yuan, Jolica Dias, Tim Bettridge, Benjamin Hersh, Michelle Huynh, Konrad Piascik, Ricardo Cabello, David Kim, and Ruofei Du
arxiv, 2025.
Keywords: Extended Reality, Software Development Kit, WebXR, WebGL, Programming Language, Depth-based Interaction, Mixed Reality, Augmented Reality, Virtual Reality, Toolkit, AI, Gemini, Android XR, TensorFlow Lite, LiteRT
Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR AgentFuture of AI + AR
Geonsun Lee, Min Xia, Nels Numan, Xun Qian, David Li, Yanhe Chen, Achin Kulshrestha, Ishan Chatterjee, Yinda Zhang, Dinesh Manocha, David Kim, and Ruofei Du
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
Keywords: Proactive Agents, Augmented Reality, Unobtrusive Interaction, Context-Awareness, Multimodal Interaction, Human-Agent Interaction, Large Multimodal Models, Adaptive Interfaces
InstructPipe: Building Visual Programming Pipelines in Visual Blocks with Human Instructions Using LLMs🎖️ Honorable Mentions Award
Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI), 2025.
Keywords: Visual Programming; Large Language Models; Visual Prototyping; Nodegraph Editor; Graph Compiler; Low-code Development; Deep Neural Networks; Deep Learning; Visual Analytics; Interactive Perception
Geollery: A Mixed Reality Social Media Platform🌎 Live Demo of a Metaverse of Mirrored World
Ruofei Du, David Li, and Amitabh Varshney
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.
Keywords: metaverse, virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, digital twins, mirrored world; digital world; digital world; augmented communication
Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and GeminiLive Demo
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Xiang 'Anthony' Chen, Faraz Faruqi, Steve Toh, and David Kim
arxiv, 2026.
Keywords: xr blocks, vibe coding
AgentHands: Generating Interactive Hands Gestures for Spatially Grounded Agent Conversations in XR
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI), 2026.
Keywords: co-speech gestures, conversational agents, extended reality
How Well Can 3D Accessibility Guidelines Support XR Development? an Interview Study with XR Practitioners in Industry
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI), 2026.
Keywords: accessibility, A11y, AR, VR, augmented and virtual reality, extended reality, XR, developerinterviews, guidelines
LegacyAvatars: Volumetric Face Avatars for Traditional Graphics Pipelines
Safa Medin, Gengyan Li, Ziqian Bai, Ruofei Du, Leonhard Helminger, Yinda Zhang, Stephan Garbin, Philip Davidson, Gregory Wornell, Thabo Beeler, and Abhimitra Meka
2026 International Conference on 3D Vision (3DV), 2026.
Keywords: Volumetric Rendering, Face Modeling, View Synthesis, PerformanceCapture, digital human, avatar
Enhance Foveated Rendering with Weighted Reservoir Sampling
Ville Cantory, Darya Biparva, Haoyu Tan, Tongyu Nie, John Schroeder, Ruofei Du, Victoria Interrante, and Piotr Didyk
Proceedings of the 2025 18th ACM SIGGRAPH Conference on Motion, Interaction, and Game (MIG), 2025.
Keywords: foveated rendering, weighted reservoir sampling
DialogLab: Authoring, Simulating, and Testing Dynamic Group Conversations in Hybrid Human-AI ConversationsLive Demo Available!
Erzhen Hu, Yanhe Chen, Mingyi Li, Vrushank Phadnis, Pingmei Xu, Xun Qian, Alex Olwal, David Kim, Seongkook Heo, and Ruofei Du
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
Keywords: Proactive Agents, Augmented Reality, Unobtrusive Interaction,Context-Awareness, Multimodal Interaction, Human-Agent Inter-action, Large Multimodal Models, Adaptive Interfaces
Thing2Reality: Enabling Spontaneous Creation of 3D Objects From 2D Images Using Generative AI in Distributed XR Meetings
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
Keywords: Extended Reality, Augmented Communication, Image-to-3D, Information Artifacts, Multi-modal, Remote Collaboration
ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"
Zhongyi Zhou, Kohei Uehara, Haoyu Zhang, Jingtao Zhou, Lin Gu, Ruofei Du, Zheng Xu, and Tatsuya Harada
arxiv, 2025.
Keywords: tool-use, LLM
Beyond the Phone: Exploring Context-Aware Interaction Between Mobile and Mixed Reality Devices
Fengyuan Zhu, Xun Qian, Daniel Kalmar, Mahdi Tayarani, Eric Gonzalez, Mar Gonzalez-Franco, David Kim, and Ruofei Du
2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2025.
Keywords: Cross-Device Interaction, Phone-XR Intergration
SVG: 3D Stereoscopic Video Generation Via Denoising Frame Matrix
Peng Dai, Feitong Tan, Qiangeng Xu, David Futschik, Ruofei Du, Sean Fanello, Xiaojuan Qi, and Yinda Zhang
The International Conference on Learning Representations (ICLR), 2025.
Keywords: Stereoscopic Video, Generative AI
Augmented Object Intelligence with XR-Objects🎖️ 2025 SXSW Innovation Awards Finalist
Mustafa Doga Dogan, Eric Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaco, Johnny Lee, Mar Gonzalez-Franco, and David Kim
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024.
Keywords: mixed reality; extended reality; augmented reality; augmented objects; spatial computing; user interfaces; context menus
Experiencing Thing2Reality: Transforming 2D Content Into Conditioned Multiviews and 3D Gaussian Objects for XR Communication
Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024.
Keywords: extended reality, augmented communication, image-to-3D, remote collaboration, spatial referencing, co-presence
Human I/O: Towards a Unified Approach to Detecting Situational Impairments🎖️ Best Paper Honourable Mentions Award
Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: situational impairments, augmented reality, large language models, multimodal sensing, context awareness, XR interaction, interactive perception
ChatDirector: Enhancing Video Conferencing with Space-Aware Scene Rendering and Speech-Driven Layout Transition500K+ Media Coverage
Xun Qian, Feitong Tan, Yinda Zhang, Brian Moreno Collins, Alex Olwal, David Kim, Karthik Ramani, and Ruofei Du
Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: augmented communication, video conferencing, 3D portrait avatar, co-presence, attention transition, depth estimation, video-mediated communication
UI Mobility Control in XR: Switching UI Positionings Between Static, Dynamic, and Self Entities
Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: Extended Reality, User Interface, UI Mobility, UI Positioning, XR interaction, interactive graphics
FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces🏆 Best Student Paper Award
Safa Medin, Gengyan Li, Ruofei Du, Stephan Garbin, Philip Davidson, Gregory Wornell, Thabo Beeler, and Abhimitra Meka
Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2024.
Keywords: Volumetric Rendering, Face Modeling, View Synthesis, PerformanceCapture, digital human
Experiencing InstructPipe: Building Multi-modal AI Pipelines Via Prompting LLMs and Visual Programming
Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du
Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: Visual Programming; Large Language Models; Visual Prototyping;Node-graph Editor; Graph Compiler; Low-code Development; DeepNeural Networks; Deep Learning; Visual Analytics
Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers
Onur Guleryuz, Philip Chou, Berivan Isik, Hugues Hoppe, Danhang Tang, Ruofei Du, Jonathan Taylor, Philip Davidson, and Sean Fanello
arxiv, 2024.
Keywords: deep learning, image compression, nonlineartransform coding, high dynamic range, super-resolution, interactive perception
Rapsai: Accelerating Machine Learning Prototyping of Multimedia Applications Through Visual Programming🎖️ Honorable Mentions Award, 170K+ views
Ruofei Du, Na Li, Jing Jin, Michelle Carney, Scott Miles, Maria Kleiner, Xiuxiu Yuan, Yinda Zhang, Anuva Kulkarni, Xingyu Bruce Liu, Ahmed Sabie, Sergio Orts-Escolano, Abhishek Kar, Ping Yu, Ram Iyengar, Adarsh Kowdle, and Alex Olwal
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: visual programming, node-graph editor, deep neural networks, data augmentation, deep learning, model comparison, visual analytics, interactive perception
Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals📹 Open Source, Real-time, Live!
Xingyu Bruce Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang 'Anthony' Chen, and Ruofei Du
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, augmented reality, XR interaction
ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: video-mediated communication, object-centered meetings, online meeting, collaborative work, augmented communication, XR interaction
Modeling and Improving Text Stability in Live Captions🚀 Landed in Live Transcribe App
Xingyu Bruce Liu, Jun Zhang, Leonardo Ferrer, Susan Xu, Vikas Bahirwani, Boris Smus, Alex Olwal, and Ruofei Du
Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.
Keywords: live captions; real-time transcription; visual instability; flickering metric; speech-to-text; text stability; tokenized alignment; augmented communication
Next Steps for Human-Centered Generative AI: A Technical Perspective
Xiang 'Anthony' Chen, Jeffrey Burke, Ruofei Du, Matthew Hong, Jennifer Jacobs, Philippe Laban, Dingzeyu Li, Nanyun Peng, Karl Willis, Chien-Sheng Wu, and Bolei Zhou
https://arxiv.org/abs/2306.15774, 2023.
Keywords: Generative AI, Human-Centered Design
Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos
Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Mingsong Dou, Sergio Orts-Escolano, Rohit Pandey, Ping Tan, Thabo Beeler, Sean Fanello, and Yinda Zhang
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Keywords: implicit 3D avatar, monocular RGB video, facial expressions, head poses, neural radiance field, photorealism, digital human
Experiencing Visual Blocks for ML: Visual Prototyping of AI Pipelines
Ruofei Du, Na Li, Jing Jin, Michelle Carney, Xiuxiu Yuan, Kristen Wright, Mark Sherwood, Jason Mayes, Lin Chen, Jun Jiang, Jingtao Zhou, Zhongyi Zhou, Ping Yu, Adarsh Kowdle, Ram Iyengar, and Alex Olwal
Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
Keywords: visual programming, large language models, visual prototyping, multi-modal models, node-graph editor, deep neural networks, data augmentation, deep learning, visual analytics
Experiencing Visual Captions: Augmented Communication with Real-time Visuals Using Large Language Models
Xingyu Bruce Liu, Vladimir Kirilyuk, Xiuxiu Yuan, Alex Olwal, Peggy Chi, Xiang 'Anthony' Chen, and Ruofei Du
Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, dataset, textto-visual, AI agent, augmented reality
Portrait Expression Editing with Mobile Photo Sequence
Yiqin Zhao, Rohit Pandey, Yinda Zhang, Ruofei Du, Feitong Tan, Chetan Ramaiah, Tian Guo, and Sean Fanello
SIGGRAPH Asia 2023 Technical Communications (SA), 2023.
Keywords: Neural rendering, Portrait expression editing, Mobile system
RetroSphere: Self-Contained Passive 3D Controller Tracking for Augmented Reality🏆IMWUT Vol. 6 Distinguished Paper Award
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2022.
Keywords: Retroreflectors, Augmented reality, Virtual reality, Infrared marker tracking, Augmented reality glasses, XR interaction
Sandwiched Image Compression: Increasing the Resolution and Dynamic Range of Standard Codecs🎖️ Best Paper Finalist
Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello
2022 Picture Coding Symposium (PCS), 2022.
Keywords: deep learning, image compression, nonlineartransform coding, high dynamic range, super-resolution, interactive perception
PRIF: Primary Ray-based Implicit Function
European Conference on Computer Vision (ECCV), 2022.
Keywords: deep implicit functions, neural representation, signed distance function, interactive perception, interactive graphics
“Slurp” Revisited: Using Software Reconstruction to Reflect on Spatial Interactivity and Locative Media
Proceedings of the Designing Interactive Systems Conference (DIS), 2022.
Keywords: system re-presencing, affordances, metaphor, software reconstruction, historical precedents, gestural interface, augmented reality, spatial interaction, XR interaction
ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard of Hearing Users
Dhruv Jain, Khoa Nguyen, Steven Goodman, Rachel Grossman-Kahn, Hung Ngo, Aditya Kusupati, Ruofei Du, Alex Olwal, Leah Findlater, and Jon E. Froehlich
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.
Keywords: accessibility, deaf, Deaf, hard of hearing, sound awareness
Opportunistic Interfaces for Augmented Reality: Transforming Everyday Objects Into Tangible 6DoF Interfaces Using Ad Hoc UI
Ruofei Du, Alex Olwal, Mathieu Goc, Shengzhi Wu, Danhang Tang, Yinda Zhang, Jun Zhang, David Tan, Federico Tombari, and David Kim
Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.
Keywords: augmented reality, everyday objects, tangible user interface, 3D user interface, 6 DoF, spatial interaction, markerless tracking, tangible interaction, hand gestures, XR interaction
OmniSyn: Intermediate View Synthesis Between Wide-baseline Panoramas
2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2022.
Keywords: 360 image, virtual reality, view synthesis, panorama, neural rendering, depth map, mesh rendering, inpainting, digital world
GazeChat: Enhancing Virtual Conferences with Gaze-aware 3D Photos
Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.
Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation, eye tracking, XR interaction
Multiresolution Deep Implicit Functions for 3D Shape Representation
Zhang Chen, Yinda Zhang, Kyle Genova, Thomas Funkhouse, Sean Fanello, Sofien Bouaziz, Christian Häne, Ruofei Du, Cem Keskin, and Danhang Tang
2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Keywords: deep implicit functions, neural representation, compression, levels of detail, MDIF, interactive perception
HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence
Feitong Tan, Danhang Tang, Mingsong Dou, Kaiwen Guo, Rohit Pandey, Cem Keskin, Ruofei Du, Deqing Sun, Sofien Bouaziz, Sean Fanello, Ping Tan, and Yinda Zhang
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Keywords: correspondences, geodesic distance, embeddings, neural networks, digital human, interactive perception
A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming🎖️ TVCG Honorable Mentions
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table, eye tracking, digital world
Sandwiched Image Compression: Wrapping Neural Networks Around a Standard Codec
Onur Guleryuz, Philip Chou, Hugues Hoppe, Danhang Tang, Ruofei Du, Philip Davidson, and Sean Fanello
2021 IEEE International Conference on Image Processing (ICIP), 2021.
Keywords: deep learning, image compression, interactive perception
Saliency Computation for Virtual Cinematography in 360° Videos
Ruofei Du and Amitabh Varshney
IEEE Computer Graphics and Applications (CGA), 2021.
Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA, eye tracking, interactive graphics
DepthLab: Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality🚀 100K downloads
Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; XR interaction; digital world, interactive graphics
CollaboVR: A Reconfigurable Framework for Multi-user to Communicate in Virtual Reality
Zhenyi He, Ruofei Du, and Ken Perlin
2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.
Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication, XR interaction, augmented communication
3D-Kernel Foveated Rendering for Light Fields
IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization, eye tracking, interactive graphics
Experiencing Real-time 3D Interaction with Depth Maps for Mobile Augmented Reality in DepthLab
Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, and David Kim
Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; interactive graphics, XR interaction
MeteoVis: Visualizing Meteorological Events in Virtual Reality
Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA), 2020.
Keywords: scientific visualization, virtual reality, meteorological data, immersion, interactive visualization, vector field, XR interaction
Eye-Dominance-guided Foveated Rendering
Xiaoxu Meng, Ruofei Du, and Amitabh Varshney
IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.
Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking, interactive graphics
Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video TexturesMicrosoft TechFest 2018
Journal of Computer Graphics Techniques (JCGT), 2019.
Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields; digital human
Language-based Colorization of Scene Sketches
ACM Transactions on Graphics (SIGGRAPH Asia), 2019.
Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization, interactive graphics, interactive perception, augmented communication
ORC Layout: Adaptive GUI Layout with OR-Constraints
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.
Keywords: GUI builder, layout manager, constraint-based layout, visual interface design, visual programming, interactive graphics
Kernel Foveated Rendering📖 Most read in PACMCGIT
Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.
Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays, eye tracking, interactive graphics
Project Geollery.com: Reconstructing a Live Mirrored World with Geotagged Social Media
Ruofei Du, David Li, and Amitabh Varshney
Proceedings of the 24th International Conference on Web3D Technology (Web3D), 2019.
Keywords: virtual reality, mixed reality, 360° image, GIS, 3D reconstruction, projection mapping, mirrored world, social media, WebGL, metaverse, mirrored world, interactive graphics, digital world
Experiencing a Mirrored World with Geotagged Social Media in Geollery
Ruofei Du, David Li, and Amitabh Varshney
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA), 2019.
Keywords: virtual reality, augmented reality, social media, GIS, street view, visualization, 3D user interface, 3D reconstruction, metaverse, mirrored world
Interactive Fusion of 360° Images for a Mirrored World
Ruofei Du, David Li, and Amitabh Varshney
2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.
Keywords: virtual reality, 360° image, 3D reconstruction, mixed reality, projection mapping, mirrored world, metaverse, mirrored world
Montage4D: Interactive Seamless Fusion of Multiview Video Textures
Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.
Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields, digital human
SketchyScene: Richly-Annotated Scene Sketches
Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang
European Conference on Computer Vision (ECCV), 2018.
Keywords: sketch dataset, scene sketch, sketch segmentation, interactive graphics
Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras
ACM Transactions on Accessible Computing (TACCESS), 2016.
Keywords: accessibility, real-time OCR, visual impairments, wearables, XR interaction
Video Fields: Fusing Multiple Surveillance Videos Into a Dynamic Virtual Environment
Ruofei Du, Sujal Bista, and Amitabh Varshney
Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.
Keywords: virtual reality; mixed-reality; video-based rendering; projection mapping; surveillance video; WebGL; WebVR; interactive graphics
Fusing Multimedia Data Into Dynamic Virtual EnvironmentsPh.D. Dissertation
Ruofei Du
Ph.D. Dissertation, University of Maryland, College Park., 2018.
Keywords: social street view, geollery, spherical harmonics, 360 video, multiview video, montage4d, haptics, cryptography, metaverse, mirrored world
Social Street View: Blending Immersive Street Views with Geo-Tagged Social Media 🏆 Best Paper Award
Ruofei Du and Amitabh Varshney
Proceedings of the 21st International Conference on Web3D Technology (Web3D), 2016.
Keywords: metaverse, spatial-temporal virtual reality; social media; street view; geographical information systems; mixed reality; WebGL; digital twins; digital world
VRSurus: Enhancing Interactivity and Tangibility of Puppets in Virtual Reality🎥 Demoed at UIST 2015
Ruofei Du and Liang He
Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.
Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display; XR interaction
AtmoSPHERE: Representing Space and Movement Using Sand Traces in an Interactive Zen Garden
Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2015.
Keywords: Visualization; Tangible Interactive Art; Machine Aesthetics; Calm Technology; XY Servo Table; Kinect; XR interaction
The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind
Computer Vision - ECCV 2014 Workshops (ECCVW), 2014.
Keywords: Accessibility, Wearables, Real-time OCR, Text Reading for Blind
Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision
Proceedings of the 17th International ACM SIGACCESS Conference on Computers Accessibility (ASSETS), 2015.
Keywords: Blind; visually impaired; wearable computing; computer vision; vision-augmented touch
Online Vigilance Analysis Combining Video and Electrooculography Features
Neural Information Processing - 19th International Conference (ICONIP), 2012.
Keywords: Vigilance Analysis, Fatigue Detection, Active Shape Model, Electrooculography, Support Vector Machine, eye tracking
A Pilot Study of Spherical Harmonics for Saliency Computation and Navigation in 360° Videos
Ruofei Du and Amitabh Varshney
ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D), 2018.
Keywords: spherical harmonics, virtual reality, visual saliency, 360°videos, omnidirectional videos, perception, Itti model, spectralresidual, GPGPU, CUDA
Featured Publications
Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR AgentFuture of AI + AR
Geonsun Lee, Min Xia, Nels Numan, Xun Qian, David Li, Yanhe Chen, Achin Kulshrestha, Ishan Chatterjee, Yinda Zhang, Dinesh Manocha, David Kim, and Ruofei Du
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
InstructPipe: Building Visual Programming Pipelines in Visual Blocks with Human Instructions Using LLMs🎖️ Honorable Mentions Award
Zhongyi Zhou, Jing Jin, Vrushank Phadnis, Xiuxiu Yuan, Jun Jiang, Xun Qian, Jingtao Zhou, Yiyi Huang, Zheng Xu, Yinda Zhang, Kristen Wright, Jason Mayes, Mark Sherwood, Johnny Lee, Alex Olwal, David Kim, Ram Iyengar, Na Li, and Ruofei Du
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI), 2025.
pdf, doi | project, video, code, demo, blog | cited by, cite
Geollery: A Mixed Reality Social Media Platform🌎 Live Demo of a Metaverse of Mirrored World
Ruofei Du, David Li, and Amitabh Varshney
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), 2019.
pdf, doi | website, project, video, slides, demo, supp | cited by, cite
Recent Publications
Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and GeminiLive Demo
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Xiang 'Anthony' Chen, Faraz Faruqi, Steve Toh, and David Kim
arxiv, 2026.
pdf, doi | project, video, code, demo, blog | cited by, cite
AgentHands: Generating Interactive Hands Gestures for Spatially Grounded Agent Conversations in XR
Ziyi Liu, David Li, Zhongyi Zhou, David Kim, Ruofei Du, and Xun Qian
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI), 2026.
How Well Can 3D Accessibility Guidelines Support XR Development? an Interview Study with XR Practitioners in Industry
Daniel Killough, Tiger Ji, Kexin Zhang, Yaxin Hu, Yu Huang, Ruofei Du, and Yuhang Zhao
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI), 2026.
LegacyAvatars: Volumetric Face Avatars for Traditional Graphics Pipelines
Safa Medin, Gengyan Li, Ziqian Bai, Ruofei Du, Leonhard Helminger, Yinda Zhang, Stephan Garbin, Philip Davidson, Gregory Wornell, Thabo Beeler, and Abhimitra Meka
2026 International Conference on 3D Vision (3DV), 2026.
Enhance Foveated Rendering with Weighted Reservoir Sampling
Ville Cantory, Darya Biparva, Haoyu Tan, Tongyu Nie, John Schroeder, Ruofei Du, Victoria Interrante, and Piotr Didyk
Proceedings of the 2025 18th ACM SIGGRAPH Conference on Motion, Interaction, and Game (MIG), 2025.
DialogLab: Authoring, Simulating, and Testing Dynamic Group Conversations in Hybrid Human-AI ConversationsLive Demo Available!
Erzhen Hu, Yanhe Chen, Mingyi Li, Vrushank Phadnis, Pingmei Xu, Xun Qian, Alex Olwal, David Kim, Seongkook Heo, and Ruofei Du
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
pdf, doi | project, video, code, demo, blog | cited by, cite
Thing2Reality: Enabling Spontaneous Creation of 3D Objects From 2D Images Using Generative AI in Distributed XR Meetings
Erzhen Hu, Mingyi Li, Jungtaek Hong, Xun Qian, Alex Olwal, David Kim, Seongkook Heo, and Ruofei Du
Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"
Zhongyi Zhou, Kohei Uehara, Haoyu Zhang, Jingtao Zhou, Lin Gu, Ruofei Du, Zheng Xu, and Tatsuya Harada
arxiv, 2025.
Beyond the Phone: Exploring Context-Aware Interaction Between Mobile and Mixed Reality Devices
Fengyuan Zhu, Xun Qian, Daniel Kalmar, Mahdi Tayarani, Eric Gonzalez, Mar Gonzalez-Franco, David Kim, and Ruofei Du
2025 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2025.
SVG: 3D Stereoscopic Video Generation Via Denoising Frame Matrix
Peng Dai, Feitong Tan, Qiangeng Xu, David Futschik, Ruofei Du, Sean Fanello, Xiaojuan Qi, and Yinda Zhang
The International Conference on Learning Representations (ICLR), 2025.
pdf, lowres, doi | website, project, code, data | cited by, cite
Arm Robot: AR-Enhanced Embodied Control and Visualization for Intuitive Robot Arm Manipulation
Siyou Pei, Alexander Chen, Ronak Kaoshik, Ruofei Du, and Yang Zhang
arxiv, 2024.
Augmented Object Intelligence with XR-Objects🎖️ 2025 SXSW Innovation Awards Finalist
Mustafa Doga Dogan, Eric Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaco, Johnny Lee, Mar Gonzalez-Franco, and David Kim
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST), 2024.
pdf, lowres, doi | website, project, video, code, blog | cited by, cite
Random Artsy Work
Professional Services
- Associate Chair on the program committee of UIST 2026.
- Associate Chair on the Computational Interaction committee of CHI 2026.
- Associate Editor of IEEE Transactions on Circuits and Systems for Video Technology.
- Keynote Speaker & Judge on the “Immerse the Bay” 2025 committee of Stanford XR.
- Associate Chair on the Engineering Interactive Systems and Technologies committee of CHI 2021.
- Chair for the "XR Toolkits" session at UIST 2022.
- Chair for the "Making the Virtual Physical" session at CHI 2019.
- Reviewer for over 400 journal and conference papers on SIGGRAPH, UIST, CHI, TCSVT, ISMAR, VR, etc.
Calendar
I am open to invited talks and external collaboration. If you would like to book a conversation with me, please refer to my calendar and drop me an email.Additional links: Tools, References, Legacy Blog.
Stay In Touch
{{Email}}
click to reveal{{Phone}}
click to revealLast Updated
by Ruofei Du
(杜若飞)