The "Avatars and Digital Human for Co-presence Systems" project aims to develop advanced techniques for creating high-quality digital avatars and virtual humans. Montage4D focuses on real-time seamless texture montage for dynamic multiview reconstruction, which can be used for immersive telepresence in various applications such as business, training, and live entertainment. HumanGPS aims to build dense correspondences between human images under arbitrary camera viewpoints and body poses using a deep learning framework. The proposed embeddings can produce accurate correspondences between images with remarkable generalization capabilities on both intra and inter subjects. HeadAvatar proposes a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. The learnt avatar can be driven by a parametric face model to achieve user-controlled facial expressions and head poses, with more accurate expression-dependent details and good generalization to out-of-training expressions compared to other state-of-the-art approaches.
VIDEO
Publications Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2024.
Keywords: Volumetric Rendering, Face Modeling, View Synthesis, PerformanceCapture, digital human
Journal of Computer Graphics Techniques (JCGT), 2019.
Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields; digital human
Ziqian Bai ,
Feitong Tan ,
Zeng Huang ,
Kripasindhu Sarkar ,
Danhang Tang ,
Di Qiu ,
Abhimitra Meka ,
Ruofei Du ,
Mingsong Dou ,
Sergio Orts-Escolano ,
Rohit Pandey ,
Ping Tan ,
Thabo Beeler ,
Sean Fanello , and
Yinda Zhang 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Keywords: implicit 3D avatar, monocular RGB video, facial expressions, head poses, neural radiance field, photorealism, digital human
SIGGRAPH Asia 2023 Technical Communications (SA), 2023.
Keywords: Neural rendering, Portrait expression editing, Mobile system
Feitong Tan ,
Danhang Tang ,
Mingsong Dou ,
Kaiwen Guo ,
Rohit Pandey ,
Cem Keskin ,
Ruofei Du ,
Deqing Sun ,
Sofien Bouaziz ,
Sean Fanello ,
Ping Tan , and
Yinda Zhang 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Keywords: correspondences, geodesic distance, embeddings, neural networks, digital human, interactive perception
Cited By IBRNet: Learning Multi-View Image-Based Rendering 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Qianqian Wang , Zhicheng Wang , Kyle Genova , Pratul Srinivasan , Howard Zhou , Jonathan Barron , Ricardo Martin-Brualla , Noah Snavely , and Thomas Funkhouser . website , source | cite | search Neural Body: Implicit Neural Representations With Structured Latent Codes for Novel View Synthesis of Dynamic Humans CVPR 2021. Sida Peng , Yuanqing Zhang , Yinghao Xu , Qianqian Wang , Qing Shuai , Hujun Bao , and Xiaowei Zhou . source | cite | search Dance in the Wild: Monocular Human Animation With Neural Dynamic Appearance Synthesis https://arxiv.org/pdf/2111.05916.pdf. Tuanfeng Y. Wang , Duygu Ceylan , Krishna Kumar Singh , and Niloy J. Mitra . source | cite | search Human View Synthesis Using a Single Sparse RGB-D Input arXiv.2112.13889. Phong Nguyen , Nikolaos Sarafianos , Christoph Lassner , Janne Heikkila , and Tony Tung . source | cite | search BodyMap: Learning Full-Body Dense Correspondence Map arXiv.2205.09111. Anastasia Ianina , Nikolaos Sarafianos , Yuanlu Xu , Ignacio Rocco , and Tony Tung . source | cite | search A Self-occlusion Aware Lighting Model for Real-time Dynamic Reconstruction IEEE Transactions on Visualization and Computer Graphics. Chengwei Zheng , Wenbin Lin , and Feng Xu . source | cite | search Scalable Neural Indoor Scene Rendering ACM Transactions on Graphics. Xiuchao Wu , Jiamin Xu , Zihan Zhu , Hujun Bao , Qixing Huang , James Tompkin , and Weiwei Xu . source | cite | search Progressive Multi-scale Light Field Networks arXiv.2208.06710. David Li and Amitabh Varshney . source | cite | search Free-Viewpoint RGB-D Human Performance Capture and~Rendering Lecture Notes in Computer Science. Phong Nguyen-Ha , Nikolaos Sarafianos , Christoph Lassner , Janne Heikkilä , and Tony Tung . source | cite | search Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments arXiv.2211.14310. Leif Van Holland , Patrick Stotko , Stefan Krumpen , Reinhard Klein , and Michael Weinmann . source | cite | search EditableNeRF: Editing Topologically Varying Neural Radiance Fields by Key Points arXiv.2212.04247. Chengwei Zheng , Wenbin Lin , and Feng Xu . source | cite | search VoLux-GAN: A Generative Model for 3D Face Synthesis With HDRI Relighting Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings. Feitong Tan , Sean Fanello , Abhimitra Meka , Sergio Orts-Escolano , Danhang Tang , Rohit Pandey , Jonathan Taylor , Ping Tan , and Yinda Zhang . source | cite | search Video Content Representation to Support the Hyper-reality Experience in Virtual Reality 2021 IEEE Virtual Reality and 3D User Interfaces (VR). Hyerim Park and Woontack Woo . source | cite | search Multi-View Neural Human Rendering 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Minye Wu , Yuehao Wang , Qiang Hu , and Jingyi Yu . source | cite | search Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera Computer Graphics Forum. Hyomin Kim , Jungeon Kim , Hyeonseo Nam , Jaesik Park , and Seungyong Lee . source | cite | search RealityCheck: Blending Virtual Environments With Situated Physical Reality Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Jeremy Hartmann , Christian Holz , Eyal Ofek , and Andrew Wilson . source | cite | search High-Precision 5DoF Tracking and Visualization of Catheter Placement in EVD of the Brain Using AR ACM Transactions on Computing for Healthcare. Xuetong Sun , Sarah B. Murthi , Gary Schwartzbauer , and Amitabh Varshney . source | cite | search Volumetric Capture of Humans With a Single RGBD Camera Via Semi-Parametric Learning 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Rohit Pandey , Cem Keskin , Shahram Izadi , Sean Fanello , Anastasia Tkach , Shuoran Yang , Pavel Pidlypenskyi , Jonathan Taylor , Ricardo Martin-Brualla , Andrea Tagliasacchi , George Papandreou , and Philip Davidson . source | cite | search SIGNET: Efficient Neural Representation for Light Fields 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Brandon Feng and Amitabh Varshney . source | cite | search Pri3D: Can 3D Priors Help 2D Representation Learning? https://arxiv.org/abs/2104.11225.pdf. Ji Hou , Saining Xie , Benjamin Graham , Angela Dai , and M. Nießner . source | cite | search Versatile Multi-Modal Pre-Training for Human-Centric Perception arXiv.2203.13815. Fangzhou Hong , Liang Pan , Zhongang Cai , and Ziwei Liu . source | cite | search The Relightables: Volumetric Performance Capture of Humans With Realistic Relighting ACM Transactions on Graphics. Kaiwen Guo , Peter Lincoln , Philip Davidson , Jay Busch , Xueming Yu , Matt Whalen , Geoff Harvey , Sergio Orts-Escolano , Rohit Pandey , Jason Dourgarian , Danhang Tang , Anastasia Tkach , Adarsh Kowdle , Emily Cooper , Mingsong Dou , Sean Fanello , Graham Fyffe , Christoph Rhemann , Jonathan Taylor , Paul Debevec , and Shahram Izadi . source | cite | search Instant Panoramic Texture Mapping With Semantic Object Matching for Large-Scale Urban Scene Reproduction IEEE Transactions on Visualization and Computer Graphics. Jinwoo Park , Ik-Beom Jeon , Sung-Eui Yoon , and Woontack Woo . source | cite | search LookinGood: Enhancing Performance Capture With Real-time Neural Re-Rendering ACM Transactions on Graphics. Ricardo Martin-Brualla , Rohit Pandey , Shuoran Yang , Pavel Pidlypenskyi , Jonathan Taylor , Julien Valentin , Sameh Khamis , Philip Davidson , Anastasia Tkach , Peter Lincoln , Adarsh Kowdle , Christoph Rhemann , Dan B Goldman , Cem Keskin , Steve Seitz , Shahram Izadi , and Sean Fanello . source | cite | search Image-guided Neural Object Rendering 8th International Conference on Learning Representations. Justus Thies , Michael Zollhöfer , Christian Theobalt , Marc Stamminger , and Matthias Nießner . source | cite | search MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. Yongjie Guan , Xueyu Hou , Nan Wu , Bo Han , and Tao Han . source | cite | search Normal-guided Garment UV Prediction for Human Re-Texturing arXiv.2303.06504. Yasamin Jafarian , Tuanfeng Y. Wang , Duygu Ceylan , Jimei Yang , Nathan Carr , Yi Zhou , and Hyun Soo Park . source | cite | search LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling arXiv.2208.08622. Boyan Jiang , Xinlin Ren , Mingsong Dou , Xiangyang Xue , Yanwei Fu , and Yinda Zhang . source | cite | search Dynamic Surface Capture for Human Performance by Fusion of Silhouette and Multi-view Stereo The 18th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry. Zheng Zhang , You Li , Xiangrong Zeng , Sheng Tan , and Changhua Jiang . source | cite | search ConVol-E: Continuous Volumetric Embeddings for Human-Centric Dense Correspondence Estimation 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Amogh Tiwari , Pranav Manu , Nakul Rathore , Astitva Srivastava , and Avinash Sharma . source | cite | search Advances in 3D Generation: A Survey arXiv.2401.17807. Xiaoyu Li , Qi Zhang , Di Kang , Weihao Cheng , Yiming Gao , Jingbo Zhang , Zhihao Liang , Jing Liao , Yan-Pei Cao , and Ying Shan . source | cite | search 
