Our research on "Augmented Communication" aims to enhance remote communication in virtual and augmented reality through the integration of cutting-edge technologies such as machine learning, eye tracking, visual augmentation, and gesture recognition. Through this research, we have developed innovative solutions such as Visual Captions, which proactively suggests relevant visuals to aid in open-vocabulary conversations; ThingShare, a video-conferencing system that facilitates the sharing of physical objects; GazeChat, a remote communication system that utilizes gaze-awareness to represent users in 3D profile photos; and CollaboVR, a framework that enables multi-user communication in virtual reality through the design of interactive and reconfigurable layouts. Our goal is to further the state-of-the-art in real-time systems for augmented communication in VR and AR, ultimately making remote communication more universally accessible and effective.
VIDEO
Publications Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, augmented reality, XR interaction
Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2024.
Keywords: extended reality, augmented communication, image-to-3D, remote collaboration, spatial referencing, co-presence
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI), 2023.
Keywords: video-mediated communication, object-centered meetings, online meeting, collaborative work, augmented communication, XR interaction
Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA), 2023.
Keywords: live captions; real-time transcription; visual instability; flickering metric; speech-to-text; text stability; tokenized alignment; augmented communication
Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
Keywords: augmented communication, large language models, video-mediated communication, online meeting, collaborative work, dataset, textto-visual, AI agent, augmented reality
Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.
Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation, eye tracking, XR interaction
2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020.
Keywords: chalktalk, virtual reality, collaborative work, layout, telepresence, communication, XR interaction, augmented communication
Cited By I Cannot See Students Focusing on My Presentation; Are They Following Me? Continuous Monitoring of Student Engagement Through "Stungage" arXiv.2204.08193. Snigdha Das , Sandip Chakraborty , and Bivas Mitra . source | cite | search ARENA: The Augmented Reality Edge Networking Architecture 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Nuno Pereira , Anthony Rowe , Michael W Farb , Ivan Liang , Edward Lu , and Eric Riebling . source | cite | search What Was Hybrid? a Systematic Review of Hybrid Collaboration and Meetings Research https://arxiv.org/pdf/2111.06172.pdf. Thomas Neumayr , Banu Saatci , Sean Rintel , Clemens Klokmose , and Mirjam Augstein . source | cite | search A Dynamically Weighted Multi-Objective Optimization Approach to Positional Interactions in Remote-Local Augmented/Mixed Reality 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). Akshith Ullal , Cadence Watkins , and Nilanjan Sarkar . source | cite | search Usability Testing of Virtual Reality Applications\textemdashThe Pilot Study Sensors. Dorota Kamińska , Grzegorz Zwoliński , and Anna Laska-Le{\'{s}}niewicz . source | cite | search ARcall: Real-Time AR Communication Using Smartphones and Smartglasses https://arxiv.org/abs/2203.04358. Hemant Bhaskar Surale , Yu Jiang , ThamBrian A. , and SmithRajan Vaish . source | cite | search Ubiq: A System to Build Flexible Social Virtual Reality Experiences Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. Sebastian Friston , Ben J Congdon , David Swapp , Lisa Izzouzi , Klara Brandstätter , Daniel Archer , Otto Olkkonen , Felix Johannes Thiel , and Anthony Steed . source | cite | search How Will VR Enter University Classrooms? Multi-stakeholders Investigation of VR in Higher Education CHI Conference on Human Factors in Computing Systems. Qiao Jin , Yu Liu , Svetlana Yarosh , Bo Han , and Feng Qian . source | cite | search Mixed Reality Collaboration for Complementary Working Styles Special Interest Group on Computer Graphics and Interactive Techniques Conference Immersive Pavilion. Keru Wang , Zhu Wang , Karl Rosenberg , Zhenyi He , Dong Woo Yoo , Un Joo Christopher , and Ken Perlin . source | cite | search Local Free-View Neural 3D Head Synthesis for Virtual Group Meetings 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). Sebastian Rings and Frank Steinicke . source | cite | search Augmented Chironomia for Presenting Data to Remote Audiences arXiv.2208.04451. Brian D. Hall , Lyn Bartram , and Matthew Brehmer . source | cite | search A State of the Art and Scoping Review of Embodied Information Behavior in Shared, Co-present Extended Reality Experiences Electronic Imaging. Kathryn Hays , Arturo Barrera , Lydia Ogbadu-Oladapo , Olumuyiwa Oyedare , Julia Payne , Mohotarema Rashid , Jennifer Stanley , Lisa Stocker , Christopher Lueg , Michael Twidale , and Ruth West . source | cite | search Smart Factory Using Virtual Reality and Online Multi-User: Towards a Metaverse for Experimental Frameworks Applied Sciences. Luis Omar Alpala , Dar{\'{\i}}o J. Quiroga-Parra , Juan Carlos Torres , and Diego H. Peluffo-Ord{\'{o}}{\~{n}}ez . source | cite | search Visual Transitions Around Tabletops in Mixed Reality: Study on a Visual Acquisition Task Between Vertical Virtual Displays and Horizontal Tabletops Proceedings of the ACM on Human-Computer Interaction. Gary Perelman , Emmanuel Dubois , Alice Probst , and Marcos Serrano . source | cite | search Investigating Document Layout and Placement Strategies for Collaborative Sensemaking in Augmented Reality CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Weizhou Luo , Anke Lehmann , Yushan Yang , and Raimund Dachselt . source | cite | search Colocation for SLAM-Tracked VR Headsets With Hand Tracking Computers. Dennis Reimer , Iana Podkosova , Daniel Scherzer , and Hannes Kaufmann . source | cite | search A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems ACM Computing Surveys. Alexander Schäfer , Gerd Reis , and Didier Stricker . source | cite | search OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Erzhen Hu , Jens Emil Sloth Gr{\o}nb{\ae}k , Austin Houck , and Seongkook Heo . source | cite | search ChameleonControl: Teleoperating Real Human Surrogates Through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms arXiv.2302.11053. Mehrad Faridan , Bheesha Kumari , and Ryo Suzuki . source | cite | search ViGather: Inclusive Virtual Conferencing With a Joint Experience Across Traditional Screen Devices and Mixed Reality Headsets Proceedings of the ACM on Human-Computer Interaction. Huajian Qiu , Paul Streli , Tiffany Luong , Christoph Gebhardt , and Christian Holz . source | cite | search Investigating Psychological Ownership in a Shared AR Space: Effects of Human and Object Reality and Object Controllability arXiv.2308.13953. Dongyun Han , Donghoon Kim , Kangsoo Kim , and Isaac Cho . source | cite | search A Reference Framework for Evaluating Virtual Conferences/submitted by Alexander Gindlhumer JOHANNES KEPLER UNIVERSITY LINZ. Alexander Gindlhumer . source | cite | search iBall: Augmenting Basketball Videos With Gaze-moderated Embedded Visualizations Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Zhutian Chen , Qisen Yang , Jiarui Shan , Tica Lin , Johanna Beyer , Haijun Xia , and Hanspeter Pfister . source | cite | search A Reference Framework for Evaluating Virtual Conferences Universitätsbibliothek Linz. Lisa-Marie Huber and Alexander Gindlhumer . source | cite | search An Extended AI-Experience: Industry 5.0 in Creative Product Innovation Sensors. Amy Grech , Jörn Mehnen , and Andrew Wodehouse . source | cite | search DisETrac: Distributed Eye-Tracking for Online Collaboration Proceedings of the 2023 Conference on Human Information Interaction and Retrieval. Bhanuka Mahanama , Mohan Sunkara , Vikas Ashok , and Sampath Jayarathna . source | cite | search SceneFusion: Room-Scale Environmental Fusion for Efficient Traveling Between Separate Virtual Environments IEEE Transactions on Visualization and Computer Graphics. Miao Wang , Yi-Jun Li , Jinchuan Shi , and Frank Steinicke . source | cite | search How Users Cognitively Appraise And~emotionally Experience The~metaverse: Focusing on Social Virtual Reality Information Technology & People. Ayoung Suh . source | cite | search Meet Me in VR! Can VR Space Help Remote Teams Connect: A Seven-week Study With Horizon Workrooms International Journal of Human-Computer Studies. Katarzyna Abramczuk , Zbigniew Bohdanowicz , Bartosz Muczy{\'{n}}ski , Kinga H. Skorupska , and Daniel Cnotkowski . source | cite | search An End-to-End Review of Gaze Estimation and Its Interactive Applications on Handheld Mobile Devices ACM Computing Surveys. Yaxiong Lei , Shijing He , Mohamed Khamis , and Juan Ye . source | cite | search CallMap: A Multi-dialogue Participative Chatting Environment Based on~Participation Structure Lecture Notes in Computer Science. Masanari Ichikawa and Yugo Takeuchi . source | cite | search Pseudo-mutual Gazing Enhances Interbrain Synchrony During Remote Joint Attention Tasking Brain and Behavior. Chun-Hsiang Chuang and Hao-Che Hsu . source | cite | search Key Technologies for Networked Virtual Environments Multimedia Tools and Applications. Juan Gonz{\'{a}}lez Salinas , Fernando Boronat Segu{\'{\i}} , Almanzor Sapena Piera , and Francisco Javier Pastor Castillo . source | cite | search Facilitated Model-based Reasoning in Immersive Virtual Reality: Meaning-making and Embodied Interactions With Dynamic Processes International Journal of Computer-Supported Collaborative Learning. Michelle Lui , Kit-Ying Angela Chong , Martha Mullally , and Rhonda McEwen . source | cite | search CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. Haijun Xia , Tony Wang , Aditya Gunturu , Peiling Jiang , William Duan , and Xiaoshuo Yao . source | cite | search Feeling Present! From Physical to Virtual Cinematography Lighting Education With Metashadow Proceedings of the 31st ACM International Conference on Multimedia. Zheng Wei , Xian Xu , Lik-Hang Lee , Wai Tong , Huamin Qu , and Pan Hui . source | cite | search Gazing Heads: Investigating Gaze Perception in Video-Mediated Communication ACM Transactions on Computer-Human Interaction. Martin Schuessler , Luca Hormann , Raimund Dachselt , Andrew Blake , and Carsten Rother . source | cite | search Efficient VR-AR Communication Method Using Virtual Replicas in XR Remote Collaboration International Journal of Human-Computer Studies. Eunhee Chang , Yongjae Lee , Mark Billinghurst , and Byounghyun Yoo . source | cite | search From Process-based to Technology-driven: A Study on Functionalities As Key Elements of Collaborative Planning Methods for Construction Projects Production Planning amp; Control. Moslem Sheikhkhoshkar , Hind Bril El-Haouzi , Alexis Aubry , Farook Hamzeh , and Farzad Rahimian . source | cite | search Every “Body” Gets a Say: An Augmented Optimization Metric to Preserve Body Pose During Avatar Adaptation in Mixed/Augmented Reality IEEE Transactions on Visualization and Computer Graphics. Alexandra Watkins , Akshith Ullal , and Nilanjan Sarkar . source | cite | search Effects of Avatar Transparency on Social Presence in Task-centric Mixed Reality Remote Collaboration IEEE Transactions on Visualization and Computer Graphics. Boram Yoon , Jae-eun Shin , Hyung-il Kim , Seo Young Oh , Dooyoung Kim , and Woontack Woo . source | cite | search 
