GazeChat: Enhancing Virtual Conferences with Gaze-aware 3D Photos

Communication software such as Clubhouse and Zoom has evolved to be an integral part of many people's daily lives. However, due to network bandwidth constraints and concerns about privacy, cameras in video conferencing are often turned off by participants. This leads to a situation in which people can only see each others' profile images, which is essentially an audio-only experience. Even when switched on, video feeds do not provide accurate cues as to who is talking to whom. This paper introduces GazeChat, a remote communication system that visually represents users as gaze-aware 3D profile photos. This satisfies users' privacy needs while keeping online conversations engaging and efficient. GazeChat uses a single webcam to track whom any participant is looking at, then uses neural rendering to animate all participants' profile images so that participants appear to be looking at each other. We have conducted a remote user study (N=16) to evaluate GazeChat in three conditions: audio conferencing with profile photos, GazeChat, and video conferencing. Based on the results of our user study, we conclude that GazeChat maintains the feeling of presence while preserving more privacy and requiring lower bandwidth than video conferencing, provides a greater level of engagement than to audio conferencing, and helps people to better understand the structure of their conversation.

Publications

teaser image of GazeChat: Enhancing Virtual Conferences with Gaze-aware 3D Photos

GazeChat: Enhancing Virtual Conferences with Gaze-aware 3D Photos

Zhenyi He, Keru Wang, Brandon Feng, Ruofei Du, and Ken Perlin
Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.
Keywords: eye contact, gaze awareness, video conferencing, video-mediated communication, gaze interaction, augmented communication, augmented conversation, eye tracking, XR interaction
pdf, doi | website, project, video, code | abstract | cited by, cite

Videos

Talks

Interactive Perception & Graphics for a Universally Accessible XR Teaser Image.

Interactive Perception & Graphics for a Universally Accessible XR

Ruofei Du

CVPR 2025 , Nashville, TN, USA.

pdf |

Interactive Perception & Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Perception & Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.

pdf | gSlides

Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at ECL Seminar Series by Dr. Alaeddin Nassani , Remote Talk.

pdf | gSlides

Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at Empathic Computing Lab , Remote Talk.

pdf |

Interactive Graphics for a Universally Accessible Metaverse Teaser Image.

Interactive Graphics for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at UMD by Prof. Amitabh Varshney , College Park, Maryland.

pdf | video | gSlides

Computational Interaction for a Universally Accessible Metaverse Teaser Image.

Computational Interaction for a Universally Accessible Metaverse

Ruofei Du

Invited Talk at GAMES Seminar , Remote Talk.

pdf | gSlides

Fusing Physical and Virtual Worlds into An Interactive Metaverse Teaser Image.

Fusing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at UCLA by Prof. Yang Zhang , Remote Talk.

pdf | gSlides

Polymerizing Physical and Virtual Worlds into An Interactive Metaverse Teaser Image.

Polymerizing Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk by Prof. Arthur Theil at Birmingham City University , Remote Talk.

pdf | gSlides

Blending Physical and Virtual Worlds into An Interactive Metaverse Teaser Image.

Blending Physical and Virtual Worlds into An Interactive Metaverse

Ruofei Du

Invited Talk at Wayne State University , Remote Talk.

pdf | gSlides

Fusing Physical and Virtual Worlds into Interactive Mixed Reality Teaser Image.

Fusing Physical and Virtual Worlds into Interactive Mixed Reality

Ruofei Du

Invited Talk at George Mason University , Remote Talk.

pdf | gSlides

Cited By

  • I Cannot See Students Focusing on My Presentation; Are They Following Me? Continuous Monitoring of Student Engagement Through "Stungage". arXiv.2204.08193.Snigdha Das, Sandip Chakraborty, and Bivas Mitra. source | cite | search
  • Local Free-View Neural 3D Head Synthesis for Virtual Group Meetings. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).Sebastian Rings and Frank Steinicke. source | cite | search
  • Augmented Chironomia for Presenting Data to Remote Audiences. arXiv.2208.04451.Brian D. Hall, Lyn Bartram, and Matthew Brehmer. source | cite | search
  • A State of the Art and Scoping Review of Embodied Information Behavior in Shared, Co-present Extended Reality Experiences. Electronic Imaging.Kathryn Hays, Arturo Barrera, Lydia Ogbadu-Oladapo, Olumuyiwa Oyedare, Julia Payne, Mohotarema Rashid, Jennifer Stanley, Lisa Stocker, Christopher Lueg, Michael Twidale, and Ruth West. source | cite | search
  • OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.Erzhen Hu, Jens Emil Sloth Gr{\o}nb{\ae}k, Austin Houck, and Seongkook Heo. source | cite | search
  • ViGather: Inclusive Virtual Conferencing with a Joint Experience Across Traditional Screen Devices and Mixed Reality Headsets. Proceedings of the ACM on Human-Computer Interaction.Huajian Qiu, Paul Streli, Tiffany Luong, Christoph Gebhardt, and Christian Holz. source | cite | search
  • A Reference Framework for Evaluating Virtual Conferences/submitted by Alexander Gindlhumer. JOHANNES KEPLER UNIVERSITY LINZ.Alexander Gindlhumer. source | cite | search
  • iBall: Augmenting Basketball Videos with Gaze-moderated Embedded Visualizations. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.Zhutian Chen, Qisen Yang, Jiarui Shan, Tica Lin, Johanna Beyer, Haijun Xia, and Hanspeter Pfister. source | cite | search
  • A Reference Framework for Evaluating Virtual Conferences. Universitätsbibliothek Linz.Lisa-Marie Huber and Alexander Gindlhumer. source | cite | search
  • DisETrac: Distributed Eye-Tracking for Online Collaboration. Proceedings of the 2023 Conference on Human Information Interaction and Retrieval.Bhanuka Mahanama, Mohan Sunkara, Vikas Ashok, and Sampath Jayarathna. source | cite | search
  • An End-to-End Review of Gaze Estimation and Its Interactive Applications on Handheld Mobile Devices. ACM Computing Surveys.Yaxiong Lei, Shijing He, Mohamed Khamis, and Juan Ye. source | cite | search
  • CallMap: A Multi-dialogue Participative Chatting Environment Based on~Participation Structure. Lecture Notes in Computer Science.Masanari Ichikawa and Yugo Takeuchi. source | cite | search
  • Pseudo-mutual Gazing Enhances Interbrain Synchrony During Remote Joint Attention Tasking. Brain and Behavior.Chun-Hsiang Chuang and Hao-Che Hsu. source | cite | search
  • Gazing Heads: Investigating Gaze Perception in Video-Mediated Communication. ACM Transactions on Computer-Human Interaction.Martin Schuessler, Luca Hormann, Raimund Dachselt, Andrew Blake, and Carsten Rother. source | cite | search

    • Stay In Touch