We aim to enhance the user experience in Extended Reality (XR) by pioneering innovative tools and interactive techniques. In DepthLab, we present a comprehensive software library that integrates a range of UI/UX features focused on depth, such as geometry-aware rendering, depth interactive behaviors, and visual effects. In our Ad hoc UI project, we unveil opportunistic interfaces that empower users to conjure virtual interfaces on commonplace objects using voice commands or tapping gestures. With SlurpAR, we take inspiration from the historical work of Slurp, adapting it to a fresh AR modality.
VIDEO
Publications Ruofei Du ,
Eric Turner ,
Maksym Dzitsiuk ,
Luca Prasso ,
Ivo Duarte ,
Jason Dourgarian ,
Joao Afonso ,
Jose Pascoal ,
Josh Gladstone ,
Nuno Cruces ,
Shahram Izadi ,
Adarsh Kowdle ,
Konstantine Tsotsos , and
David Kim Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; XR interaction; digital world, interactive graphics
Xiang 'Anthony' Chen ,
Jeffrey Burke ,
Ruofei Du ,
Matthew Hong ,
Jennifer Jacobs ,
Philippe Laban ,
Dingzeyu Li ,
Nanyun Peng ,
Karl Willis ,
Chien-Sheng Wu , and
Bolei Zhou https://arxiv.org/abs/2306.15774, 2023.
Keywords: Generative AI, Human-Centered Design
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT), 2022.
Keywords: Retroreflectors, Augmented reality, Virtual reality, Infrared marker tracking, Augmented reality glasses, XR interaction
Proceedings of the Designing Interactive Systems Conference (DIS), 2022.
Keywords: system re-presencing, affordances, metaphor, software reconstruction, historical precedents, gestural interface, augmented reality, spatial interaction, XR interaction
Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI), 2022.
Keywords: augmented reality, everyday objects, tangible user interface, 3D user interface, 6 DoF, spatial interaction, markerless tracking, tangible interaction, hand gestures, XR interaction
Ruofei Du ,
Eric Turner ,
Maksym Dzitsiuk ,
Luca Prasso ,
Ivo Duarte ,
Jason Dourgarian ,
Joao Afonso ,
Jose Pascoal ,
Josh Gladstone ,
Nuno Cruces ,
Shahram Izadi ,
Adarsh Kowdle ,
Konstantine Tsotsos , and
David Kim Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2020.
Keywords: depth map; interactive 3D graphics; real time; interaction; augmented reality; mobile AR; rendering; GPU; ARCore; interactive graphics, XR interaction
2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019.
Keywords: visual cryptography, augmented reality (AR), tracking, XR interaction
Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA), 2016.
Keywords: Virtual Reality; Tangible User Interface; Haptics; Gesture Recognition; Head-Mounted Display; XR interaction
Cited By Real-time Occlusion Handling for Augmented Reality Assistance Assembly Systems With Monocular Images Journal of Manufacturing Systems. Wang Li , Junfeng Wang , Maoding Liu , and Shiwen Zhao . source | cite | search InDepth: Real-time Depth Inpainting for Mobile Augmented Reality Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Yunfan Zhang , Tim Scargill , Ashutosh Vaishnav , Gopika Premsankar , Mario Francesco , and Maria Gorlatova . source | cite | search Neural Cameras: Learning Camera Characteristics for Coherent Mixed Reality Rendering 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR. David Mandl , Peter Mohr , Tobias Langlotz , Christoph Ebner , Shohei Mori , Stefanie Zollmann , Peter M Roth , and Denis Kalkofen . source | cite | search TransforMR: Pose-Aware Object Substitution for Composing Alternate Mixed Realities 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Mohamed Kari , Tobias Grosse-Puppendahl , Luis Falconeri Coelho , Andreas Fender , David Bethge , Reinhard Schutte , and Christian Holz . source | cite | search Holodeck: Immersive 3D Displays Using Swarms of Flying Light Specks https://arxiv.org/pdf/2111.03657. Shahram Ghandeharizadeh . source | cite | search Testing of a Deep Learning Model Providing Monocular Depth Estimation on Mobile Devices Via Web Service 2021 5th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT). Alper Tunga Akin and Cetin Comert . source | cite | search Real-time Tracking of Surrounding Objects in Augmented and Mixed Reality Applications Proceedings Volume 11897, Optoelectronic Imaging and Multimedia Technology VIII. Vitalii Nezhelskii , Iuliia Golovchanskaia , Andrey Zhdanov , and Dmitry Zhdanov . source | cite | search Sparse Depth Completion With Semantic Mesh Deformation Optimization https://arxiv.org/pdf/2112.05498.pdf. Bing Zhou , Matias Aiskovich , and Sinem Guven . source | cite | search Analysis and Development of Augmented Reality Applications for the Dissemination of Cultural Heritage Department of Cartographic Engineering Geodesy and Photogrammetry, Universitat Politècnica de València. Silvia Blanco Pons . source | cite | search Hybrid Mobile Vision for Emerging Applications Proceedings of the 23rd Annual International Workshop on Mobile Computing Systems and Applications. Nan Wu , Felix Xiaozhu Lin , Feng Qian , and Bo Han . source | cite | search A Smartphone-Based Mobility Assistant Using Depth Imaging for Visually Impaired and Blind Applied Sciences. Aaron Raymond See , Bien Grenier Sasing , and Welsey Daniel Advincula . source | cite | search ScalAR: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality CHI Conference on Human Factors in Computing Systems. Xun Qian , Fengming He , Xiyun Hu , Tianyi Wang , Ananya Ipsita , and Karthik Ramani . source | cite | search Future Landscape Visualization Using a City Digital Twin: Integration of Augmented Reality and Drones With Implementation of 3D Model-based Occlusion Handling Journal of Computational Design and Engineering. Naoki Kikuchi , Tomohiro Fukuda , and Nobuyoshi Yabuki . source | cite | search When Virtuality Surpasses Reality: Possible Futures of Ubiquitous XR CHI Conference on Human Factors in Computing Systems Extended Abstracts. Ross Johnstone , Neil McDonnell , and Julie Williamson . source | cite | search A Generic Framework for Depth Reconstruction Enhancement Journal of Imaging. Hendrik Sommerhoff and Andreas Kolb . source | cite | search Globally Consistent Video Depth and Pose Estimation With Efficient Test-Time Training arXiv.2208.02709. Yao-Chih Lee , Kuan-Wei Tseng , Guan-Sheng Chen , and Chu-Song Chen . source | cite | search Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence ACM Computing Surveys. Jacky Cao , Kit-Yung Lam , Lik-Hang Lee , Xiaoli Liu , Pan Hui , and Xiang Su . source | cite | search New Approaches in Training for in Situ Visualization of Ionizing Radiation Measurements University of Ontario Institute of Technology. Joseph Chaput . source | cite | search Developing a Play-anywhere Handheld AR Storytelling App Using Remote Data Collection Frontiers in Computer Science. Gideon Raeburn , Martin Welton , and Laurissa Tokarchuk . source | cite | search LITAR: Visually Coherent Lighting for Mobile Augmented Reality Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Yiqin Zhao , Chongyang Ma , Haibin Huang , and Tian Guo . source | cite | search Collision-Aware AR Telemanipulation Using Depth Mesh 2022 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM). Chanapol Piyavichayanon , Masanobu Koga , Eiji Hayashi , and Sakmongkon Chumkamon . source | cite | search A Vision System Using Depth Inpainting for Virtual Content Reconstruction in Augmented Reality Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies VI. Viacheslav Voronin , Nikolay Gapon , Iliy Khamidullin , Olga Tokareva , Alexander Zelensky , and Evgeny A. Semenishchev . source | cite | search WeLDCFNet: Convolutional Neural Network Based on Wedgelet Filters and Learnt Deep Correlation Features for Depth Maps Features Extraction 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP). Mariem Sehli , Dorsaf Sebai , and Faouzi Ghorbel . source | cite | search XSpace: An Augmented Reality Toolkit for Enabling Spatially-Aware Distributed Collaboration Proceedings of the ACM on Human-Computer Interaction. Jaylin Herskovitz , Yi Fei Cheng , Anhong Guo , Alanson P. Sample , and Michael Nebeling . source | cite | search The Usability of a Novel Mobile Augmented Reality Application for Excavation Process Considering Safety and Productivity in Construction Construction Innovation. Fathima Nishara Abdeen , Randima Nirmal Gunatilaka , Samad M.E. Sepasgozar , and David John Edwards . source | cite | search Fighting Pandemics With Augmented Reality and Smart Sensing-based Social Distancing IEEE Computer Graphics and Applications. Jacky Cao , Hassan Mehmood , Xiaoli Liu , Sasu Tarkoma , Ekaterina Gilman , and Xiang Su . source | cite | search Affordance-Based and User-Defined Gestures for Spatial Tangible Interaction Proceedings of the 2023 ACM Designing Interactive Systems Conference. Weilun Gong , Stephanie Santosa , Tovi Grossman , Michael Glueck , Daniel Clarke , and Frances Lai . source | cite | search Dynamic Iterative Refinement for Efficient 3D Hand Pose Estimation https://arxiv.org/abs/2111.06500.pdf. John Yang , Yash Bhalgat , Simyung Chang , Fatih Porikli , and Nojun Kwak . source | cite | search PneuFetch: Supporting Blind and Visually Impaired People to Fetch Nearby Objects Via Light Haptic Cues Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. Liang He , Ruolin Wang , and Xuhai Xu . source | cite | search Sensory VR TEI '18: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction. Daniel Harley , Alexander Verni , Mackenzie Willis , Ashley Ng , Lucas Bozzo , and Ali Mazalek . source | cite | search A Character in Your Hand: Puppetry to Inform Game Controls Digital Games Research Association DiGRA. Michael Nitsche and Pierce McBride . source | cite | search Manipulating Puppets in VR 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). Michael Nitsche and Pierce McBride . source | cite | search Cinemacraft: Exploring Fidelity Cues in Collaborative Virtual World Interactions 0. Siddharth Narayanan , Nicholas Polys , and Ivica Ico Bukvic . source | cite | search Cinemacraft: Exploring Fidelity Cues in Collaborative Virtual World Interactions Virginia Polytechnic Institute and State University. Siddharth Narayanan . source | cite | search ARToken: A Tangible Device for Dynamically Binding Real-world Objects With Virtual Representation Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. Hsuan-Yu Hsueh , Chien-Hua Chen , Irene Chen , Chih-Yuan Yao , and Hung-Kuo Chu . source | cite | search Lines of Play: Using Vector Tools to Place Game Objects in Augmented Reality Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play. Jasmine Roberts , John Mernacaj , Suzanne Chambers , Dror Ayalon , and Errol King . source | cite | search VRFromX: From Scanned Reality to Interactive Virtual Experience With Human-in-the-Loop Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Ananya Ipsita , Hao Li , Runlin Duan , Yuanzhi Cao , Subramanian Chidambaram , Min Liu , and Karthik Ramani . source | cite | search Binary Descriptors for Dense Stereo Matching International Journal of Intelligent Computing and Information Sciences. Hanaa Ibrahim , Heba Khaled , Noha AbdElSabour Seada , and Hossam Faheem . source | cite | search Snap2cad: 3D Indoor Environment Reconstruction for AR/VR Applications Using a Smartphone Device Computers & Graphics. Alessandro Manni , Damiano Oriti , Andrea Sanna , Francesco De Pace , and Federico Manuri . source | cite | search Portalware: Exploring Free-Hand AR Drawing With a Dual-Display Smartphone-Wearable Paradigm Designing Interactive Systems Conference 2021. Jing Qian , Tongyu Zhou , Meredith Young-Ng , Jiaju Ma , Angel Cheung , Xiangyu Li , Ian Gonsher , and Jeff Huang . source | cite | search Measuring Forest Carbon With Mobile Phones Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services. Amelia Holcomb , Bill Tong , Megan Penny , and Srinivasan Keshav . source | cite | search Enabling Real-time Sign Language Translation on Mobile Platforms With On-board Depth Cameras Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. HyeonJung Park , Youngki Lee , and JeongGil Ko . source | cite | search Mobile Augmented Reality: User Interfaces, Frameworks, and Intelligence https://arxiv.org/abs/2106.08710. Jacky Cao , Kit-Yung Lam , Lik-Hang Lee , Xiaoli Liu , Pan Hui , and Xiang Su . source | cite | search DistanciAR: Authoring Site-Specific Augmented Reality Experiences for Remote Environments Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Zeyu Wang , Cuong Nguyen , Paul Asente , and Julie Dorsey . source | cite | search Semantic Mapping: A Semantics-based Approach to Virtual Content Placement for Immersive Environments 2021 17th International Conference on Intelligent Environments (IE). Jingyang Liu . source | cite | search Validation of Robot Model With Mobile Augmented Reality 2021 6th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS). Chanapol Piyavichayanon and Masanobu Koga . source | cite | search Outdoor Objekt-Detektion Und -Tracking Von Verkehrsteilnehmern Für Augmented Reality Fakultät Technik und Informatik. Ruben Horn . source | cite | search Landscape Visualization by Integrating Augmented Reality and Drones With Occlusion Handling to Link Real and Virtual Worlds - Towards City Digital Twin Realization . Kikuch Kikuchi , Naok Naoki , Fukud Fukuda , Tomohir Tomohiro , Yabuk Yabuki , and Nobuyosh Nobuyoshi . source | cite | search AdaBins: Depth Estimation Using Adaptive Bins Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Shariq Farooq Bhat , Ibraheem Alhashim , and Peter Wonka . source | cite | search SemanticAdapt: Optimization-based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology. Yifei Cheng , Yukang Yan , Xin Yi , Yuanchun Shi , and David Lindlbauer . source | cite | search Real-time Tracking of Surrounding Objects in Augmented and Mixed Reality Applications Proceedings Volume 11897, Optoelectronic Imaging and Multimedia Technology VIII. Vitalii Nezhelskii , Iuliia Golovchanskaia , Andrey Zhdanov , and Dmitry Zhdanov . source | cite | search Interaction Techniques for 3D-positioning Objects in Mobile Augmented Reality ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction. Carl-Philipp Hellmuth , Miroslav Bachinski , and Jörg Müller . source | cite | search Monitoring Social Distancing With Single Image Depth Estimation arXiv.2204.01693. Alessio Mingozzi , Andrea Conti , Filippo Aleotti , Matteo Poggi , and Stefano Mattoccia . source | cite | search Visual Cryptography for Biometric Privacy, Authentication and General Access Structure: A Review Journal Of Critical Reviews. B. Rebecca Jeyavadhanam and B. Angel Rubavathy . source | cite | search Teachable Reality: Prototyping Tangible Augmented Reality With Everyday Objects by Leveraging Interactive Machine Teaching Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Kyzyl Monteiro , Ritik Vatsal , Neil Chulpongsatorn , Aman Parnami , and Ryo Suzuki . source | cite | search Deep Depth Completion: A Survey arXiv.2205.05335. Junjie Hu , Chenyu Bao , Mete Ozay , Chenyou Fan , Qing Gao , Honghai Liu , and Tin Lun Lam . source | cite | search Scene Responsiveness for Visuotactile Illusions in Mixed Reality Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. Mohamed Kari , Reinhard Schütte , and Raj Sodhi . source | cite | search AdapTUI: Adaptation of Geometric-Feature-Based Tangible User Interfaces in Augmented Reality Proceedings of the ACM on Human-Computer Interaction. Fengming He , Xiyun Hu , Xun Qian , Zhengzhe Zhu , and Karthik Ramani . source | cite | search Current Trend of Artificial Intelligence-Augmented Reality in Science Learning: Systematic Literature Review Jurnal Penelitian Pendidikan IPA. Dona Budi Kharisma , Sudirman Sudirman , Firman Edi , and Ruth Rize Paas Megahati S . source | cite | search DualStream: Spatially Sharing Selves and Surroundings Using Mobile Devices and Augmented Reality arXiv.2309.00842. Rishi Vanukuru , Suibi Che-Chuan Weng , Krithik Ranjan , Torin Hopkins , Amy Banic , Mark D. Gross , and Ellen Yi-Luen Do . source | cite | search SiTAR: Situated Trajectory Analysis for In-the-Wild Pose Error Estimation arXiv.2308.16813. Tim Scargill , Ying Chen , Tianyi Hu , and Maria Gorlatova . source | cite | search A Systematic Threat Analysis and Defense Strategies for the Metaverse and Extended Reality Systems Computers & Security. Sara Qamar , Zahid Anwar , and Mehreen Afzal . source | cite | search Pull Gestures With Coordinated Graphics on Dual-Screen Devices Proceedings of the 2022 International Conference on Multimodal Interaction. Vivian Shen and Chris Harrison . source | cite | search Environment-Aware Rendering and Interaction in Web-Based Augmented Reality Journal of Imaging. José Ferrão , Paulo Dias , Beatriz Sousa Santos , and Miguel Oliveira . source | cite | search Real-time Bi-directional Real-Virtual Interaction Framework Using Automatic Simulation Model Generation 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). Junyoung Yun , Hyeongil Nam , Taesoo Kwon , Hyungmin Kim , and Jong Park . source | cite | search Harnessing the Metaverse for Livestock Welfare: Unleashing Sensor Data and Navigating Ethical Frontiers 2. Suresh Neethirajan . source | cite | search Virtual Occlusions Through Implicit Depth arXiv.2305.07014. Jamie Watson , Mohamed Sayed , Zawar Qureshi , Gabriel Brostow , Sara Vicente , Oisin Mac Aodha , and Michael Firman . source | cite | search Magic NeRF Lens: Interactive Fusion of Neural Radiance Fields for Virtual Facility Inspection arXiv.2307.09860. Ke Li , Susanne Schmidt , Tim Rolff , Reinhard Bacher , Wim Leemans , and Frank Steinicke . source | cite | search Toward Scalable and Controllable AR Experimentation arXiv.2307.08587. Ashkan Ganj , Yiqin Zhao , Federico Galbiati , and Tian Guo . source | cite | search Enhancing Depth Completion With Multi-View Monitored Distillation arXiv.2303.15840. Jia-Wei Guo , Cong Li , Sen-Hua Zhu , Chang-Zheng Zhang , Ming Ouyang , Ning Ding , and Hung-Chyun Chou . source | cite | search Augmented Geometry in University Education INTED2023 Proceedings. Michele Russo , Alessandra Marina Giugliano , Giulia Flenghi , Laura Carnevali , Maria Martone , Andrea Marrella , and Francesco Sapio . source | cite | search Shakes on a Plane: Unsupervised Depth Estimation From Unstabilized Photography arXiv.2212.12324. Ilya Chugunov , Yuxuan Zhang , and Felix Heide . source | cite | search LitAR: Visually Coherent Lighting for Mobile Augmented Reality arXiv.2301.06184. Yiqin Zhao , Chongyang Ma , Haibin Huang , and Tian Guo . source | cite | search Depth Map Estimation of Single-View Image Using Smartphone Camera for a 3-Dimension Image Generation in Augmented Reality 2023 Sixth International Symposium on Computer, Consumer and Control (IS3C). Jun'Nosuke Takarabe , Irawati Nurmala Sari , and Weiwei Du . source | cite | search DySPN: Learning Dynamic Affinity for Image-guided Depth Completion IEEE Transactions on Circuits and Systems for Video Technology. Yuankai Lin , Hua Yang , Tao Cheng , Wending Zhou , and Zhouping Yin . source | cite | search V2Depth: Monocular Depth Estimation Via Feature-Level Virtual-View Simulation and Refinement Proceedings of the 31st ACM International Conference on Multimedia. Zizhang Wu , Zhuozheng Li , Zhi-Gang Fan , Yunzhe Wu , Jian Pu , and Xianzhi Li . source | cite | search Demonstrating BrightMarkers: Fluorescent Tracking Markers Embedded in 3D Printed Objects Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. Mustafa Doga Dogan , Raul Garcia-Martin , Patrick William Haertel , Jamison John O{\textquotesingle}Keefe , Raul Sanchez-Reillo , and Stefanie Mueller . source | cite | search 
