ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition

Remote video conferencing systems (RVCS) are widely adopted in personal and professional communication. However, they often lack the co-presence experience of in-person meetings. This is largely due to the absence of intuitive visual cues and clear spatial relationships among remote participants, which can lead to loss of attention and speech interruptions. In this paper, we present ChatDirector, a novel RVCS that overcomes these limitations by incorporating space-aware visual presence and speech-aware attention transition assistance. ChatDirector employs a real-time pipeline that converts participants' RGB video streams into 3D portrait avatars and renders them in a 3D virtual scene. We also design a decision tree algorithm that directs the avatar layouts and behaviors based on participants' speech states. We evaluated ChatDirector through a user study (N=16). The satisfactory algorithm performance and complimentary subject user feedback imply that ChatDirector significantly enhances communication efficacy and user engagement.

Publications

teaser image of ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition

ChatDirector: Enhancing Video Conferencing With Space-Aware Scene Rendering and Speech-Driven Layout Transition

Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI), 2024.
Keywords: augmented communication, video conferencing, 3D portrait avatar, co-presence, attention transition, depth estimation, video-mediated communication


Videos

Talks

Cited By

Stay In Touch