Avatars and Digital Human for Co-presence Systems

The "Avatars and Digital Human for Co-presence Systems" project aims to develop advanced techniques for creating high-quality digital avatars and virtual humans. Montage4D focuses on real-time seamless texture montage for dynamic multiview reconstruction, which can be used for immersive telepresence in various applications such as business, training, and live entertainment. HumanGPS aims to build dense correspondences between human images under arbitrary camera viewpoints and body poses using a deep learning framework. The proposed embeddings can produce accurate correspondences between images with remarkable generalization capabilities on both intra and inter subjects. HeadAvatar proposes a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. The learnt avatar can be driven by a parametric face model to achieve user-controlled facial expressions and head poses, with more accurate expression-dependent details and good generalization to out-of-training expressions compared to other state-of-the-art approaches.

Publications

teaser image of FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces

FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic FacesBest Student Paper Award

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2024.
Keywords: Volumetric Rendering, Face Modeling, View Synthesis, PerformanceCapture, digital human

teaser image of Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video Textures

Montage4D: Real-time Seamless Fusion and Stylization of Multiview Video TexturesMicrosoft TechFest 2018

Journal of Computer Graphics Techniques (JCGT), 2019.
Keywords: texture montage, 3d reconstruction, texture stitching, view-dependent rendering, discrete geodesics, projective texture mapping, differential geometry, temporal texture fields; digital human

teaser image of Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos

Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Keywords: implicit 3D avatar, monocular RGB video, facial expressions, head poses, neural radiance field, photorealism, digital human


teaser image of Portrait Expression Editing With Mobile Photo Sequence

Portrait Expression Editing With Mobile Photo Sequence

SIGGRAPH Asia 2023 Technical Communications (SA), 2023.
Keywords: Neural rendering, Portrait expression editing, Mobile system
teaser image of HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Keywords: correspondences, geodesic distance, embeddings, neural networks, digital human, interactive perception

Videos

HumanGPS: Geodesic PreServing Feature for Dense Human Correspondence


Talks

Cited By

  • IBRNet: Learning Multi-View Image-Based Rendering. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. website, source | cite | search
  • Neural Body: Implicit Neural Representations With Structured Latent Codes for Novel View Synthesis of Dynamic Humans. CVPR 2021.Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. source | cite | search
  • Dance in the Wild: Monocular Human Animation With Neural Dynamic Appearance Synthesis. https://arxiv.org/pdf/2111.05916.pdf.Tuanfeng Y. Wang, Duygu Ceylan, Krishna Kumar Singh, and Niloy J. Mitra. source | cite | search
  • Human View Synthesis Using a Single Sparse RGB-D Input. arXiv.2112.13889.Phong Nguyen, Nikolaos Sarafianos, Christoph Lassner, Janne Heikkila, and Tony Tung. source | cite | search
  • BodyMap: Learning Full-Body Dense Correspondence Map. arXiv.2205.09111.Anastasia Ianina, Nikolaos Sarafianos, Yuanlu Xu, Ignacio Rocco, and Tony Tung. source | cite | search
  • A Self-occlusion Aware Lighting Model for Real-time Dynamic Reconstruction. IEEE Transactions on Visualization and Computer Graphics.Chengwei Zheng, Wenbin Lin, and Feng Xu. source | cite | search
  • Scalable Neural Indoor Scene Rendering. ACM Transactions on Graphics.Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang, James Tompkin, and Weiwei Xu. source | cite | search
  • Progressive Multi-scale Light Field Networks. arXiv.2208.06710.David Li and Amitabh Varshney. source | cite | search
  • Free-Viewpoint RGB-D Human Performance Capture and~Rendering. Lecture Notes in Computer Science.Phong Nguyen-Ha, Nikolaos Sarafianos, Christoph Lassner, Janne Heikkilä, and Tony Tung. source | cite | search
  • Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments. arXiv.2211.14310.Leif Van Holland, Patrick Stotko, Stefan Krumpen, Reinhard Klein, and Michael Weinmann. source | cite | search
  • EditableNeRF: Editing Topologically Varying Neural Radiance Fields by Key Points. arXiv.2212.04247.Chengwei Zheng, Wenbin Lin, and Feng Xu. source | cite | search
  • VoLux-GAN: A Generative Model for 3D Face Synthesis With HDRI Relighting. Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings.Feitong Tan, Sean Fanello, Abhimitra Meka, Sergio Orts-Escolano, Danhang Tang, Rohit Pandey, Jonathan Taylor, Ping Tan, and Yinda Zhang. source | cite | search
  • Video Content Representation to Support the Hyper-reality Experience in Virtual Reality. 2021 IEEE Virtual Reality and 3D User Interfaces (VR).Hyerim Park and Woontack Woo. source | cite | search
  • Multi-View Neural Human Rendering. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu. source | cite | search
  • Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera. Computer Graphics Forum.Hyomin Kim, Jungeon Kim, Hyeonseo Nam, Jaesik Park, and Seungyong Lee. source | cite | search
  • RealityCheck: Blending Virtual Environments With Situated Physical Reality. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.Jeremy Hartmann, Christian Holz, Eyal Ofek, and Andrew Wilson. source | cite | search
  • High-Precision 5DoF Tracking and Visualization of Catheter Placement in EVD of the Brain Using AR. ACM Transactions on Computing for Healthcare.Xuetong Sun, Sarah B. Murthi, Gary Schwartzbauer, and Amitabh Varshney. source | cite | search
  • Volumetric Capture of Humans With a Single RGBD Camera Via Semi-Parametric Learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Rohit Pandey, Cem Keskin, Shahram Izadi, Sean Fanello, Anastasia Tkach, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Ricardo Martin-Brualla, Andrea Tagliasacchi, George Papandreou, and Philip Davidson. source | cite | search
  • SIGNET: Efficient Neural Representation for Light Fields. 2021 IEEE/CVF International Conference on Computer Vision (ICCV).Brandon Feng and Amitabh Varshney. source | cite | search
  • Pri3D: Can 3D Priors Help 2D Representation Learning?. https://arxiv.org/abs/2104.11225.pdf.Ji Hou, Saining Xie, Benjamin Graham, Angela Dai, and M. Nießner. source | cite | search
  • Versatile Multi-Modal Pre-Training for Human-Centric Perception. arXiv.2203.13815.Fangzhou Hong, Liang Pan, Zhongang Cai, and Ziwei Liu. source | cite | search
  • The Relightables: Volumetric Performance Capture of Humans With Realistic Relighting. ACM Transactions on Graphics.Kaiwen Guo, Peter Lincoln, Philip Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts-Escolano, Rohit Pandey, Jason Dourgarian, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, Sean Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul Debevec, and Shahram Izadi. source | cite | search
  • Instant Panoramic Texture Mapping With Semantic Object Matching for Large-Scale Urban Scene Reproduction. IEEE Transactions on Visualization and Computer Graphics.Jinwoo Park, Ik-Beom Jeon, Sung-Eui Yoon, and Woontack Woo. source | cite | search
  • LookinGood: Enhancing Performance Capture With Real-time Neural Re-Rendering. ACM Transactions on Graphics.Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, and Sean Fanello. source | cite | search
  • Image-guided Neural Object Rendering. 8th International Conference on Learning Representations.Justus Thies, Michael Zollhöfer, Christian Theobalt, Marc Stamminger, and Matthias Nießner. source | cite | search
  • MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time. Proceedings of the 29th Annual International Conference on Mobile Computing and Networking.Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. source | cite | search
  • Normal-guided Garment UV Prediction for Human Re-Texturing. arXiv.2303.06504.Yasamin Jafarian, Tuanfeng Y. Wang, Duygu Ceylan, Jimei Yang, Nathan Carr, Yi Zhou, and Hyun Soo Park. source | cite | search
  • LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling. arXiv.2208.08622.Boyan Jiang, Xinlin Ren, Mingsong Dou, Xiangyang Xue, Yanwei Fu, and Yinda Zhang. source | cite | search
  • Dynamic Surface Capture for Human Performance by Fusion of Silhouette and Multi-view Stereo. The 18th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry.Zheng Zhang, You Li, Xiangrong Zeng, Sheng Tan, and Changhua Jiang. source | cite | search
  • ConVol-E: Continuous Volumetric Embeddings for Human-Centric Dense Correspondence Estimation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).Amogh Tiwari, Pranav Manu, Nakul Rathore, Astitva Srivastava, and Avinash Sharma. source | cite | search
  • Advances in 3D Generation: A Survey. arXiv.2401.17807.Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. source | cite | search
  • Stay In Touch