ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition
ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition
Remote video conferencing systems (RVCS) are widely adopted in personal and professional communication. However, they often lack the co-presence experience of in-person meetings. This is largely due to the absence of intuitive visual cues and clear spatial relationships among remote participants, which can lead to loss of attention and speech interruptions. In this paper, we present ChatDirector, a novel RVCS that overcomes these limitations by incorporating space-aware visual presence and speech-aware attention transition assistance. ChatDirector employs a real-time pipeline that converts participants' RGB video streams into 3D portrait avatars and renders them in a 3D virtual scene. We also design a decision tree algorithm that directs the avatar layouts and behaviors based on participants' speech states. We evaluated ChatDirector through a user study (N=16). The satisfactory algorithm performance and complimentary subject user feedback imply that ChatDirector significantly enhances communication efficacy and user engagement.
Publications
ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition500K+ Media Coverage
Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: augmented communication, video conferencing, 3D portrait avatar, co-presence, attention transition, depth estimation, video-mediated communication
Videos
Talks
Augmented Communication for a Universally Accessible Metaverse
Ruofei Du
Invited Talk at UMD by Dr. Varshney , College Park, Maryland.
pdf |