Multimodal Large‑Model Driven Virtual Digital Humans: Background, Methods, and Applications

Overview – With the continuous advancement of artificial intelligence, multimodal human‑computer dialogue technologies have matured and are widely applied across various fields. The presentation focuses on three parts: background of virtual digital humans, development methods, and application cases.

1. Background of Virtual Digital Humans – Digital humans are increasingly influential in IP influence and fan economies, appearing as virtual actors, hosts, streamers, customer service agents, guides, and experts. Their market is expanding rapidly, offering higher intelligence, more human‑like experiences, and serving as foundational infrastructure for the metaverse. However, high modeling costs (hundreds of thousands to millions) and long production cycles hinder continuous output.

Key Challenges – (1) Expensive modeling; (2) Limited driving capabilities—static 2D idols versus 3D avatars with unrealistic appearance and stiff motions; (3) Constrained application scenarios, often limited to simple customer‑service tasks.

2. Development Methods – The mainstream approach combines NeRF and GAN for generation and rendering, enabling low‑cost, rapid creation of digital humans. AIGC reduces modeling time to about three weeks and cost to 10‑100 k RMB, supporting multiple styles (realistic, cartoon, etc.) through a flexible base library. Production can be batch‑generated for IP creation.

Digital human driving is divided into human‑in‑the‑loop (using motion capture, facial capture, etc.) and non‑human‑in‑the‑loop (talking‑head) methods. Human‑in‑the‑loop extracts dynamic and identity features via a dual‑stream network and reconstructs new faces. Non‑human‑in‑the‑loop uses 3D reconstruction, pose estimation, and Transformer‑based lip‑sync to drive avatars from text or speech, supporting full‑pose facial replacement and video‑driven motion transfer.

3. Multimodal Dialogue – Modern digital humans are moving from single‑modal to multimodal interaction, leveraging massive dialogue data for self‑supervised learning. Cross‑modal representation learning integrates image, text, audio, and video, enabling more human‑like understanding and generation. The proprietary "Zidong Tai‑Chu" multimodal model (~100 billion parameters) unifies these modalities for image‑text‑audio generation, scene creation, and question‑answer retrieval.

4. Application Cases – Examples include: (a) Real‑time multimodal dialogue where keywords trigger image generation; (b) Creative image synthesis (e.g., a teddy bear swimming); (c) Automotive cockpit integration—users upload a photo and the system generates a personalized avatar that can control vehicle functions; (d) Tourism guide "Hang Xiaoyi" combining multimodal dialogue with a historical knowledge graph; (e) The world’s first multimodal sign‑language avatar integrating visual, textual, audio, and facial cues.

5. Q&A Highlights – The model can generate real‑time sign language by mapping pre‑modeled word gestures; the sign‑language teaching‑exam device converts gestures to text/audio and provides visual feedback; the "Xiao Chu" IP design reflects a youthful Chinese‑style AI persona, symbolizing domestic AI innovation.

Overall, the presentation demonstrates how large multimodal models empower cost‑effective, high‑quality digital humans with advanced dialogue and driving capabilities, opening new possibilities for immersive human‑computer interaction.