Voice‑Driven Facial Animation for Digital Humans: Techniques and OPPO XiaoBu Assistant Practice

The presentation provides an overview of digital‑human driving technologies, focusing on three mainstream methods: motion‑capture (requiring wearable sensors and high hardware cost), audio‑driven (using audio features, ASR, TTS, emotion analysis to control facial and body motion), and key‑point driving (leveraging ordinary RGB cameras for low‑cost real‑time animation).

It emphasizes that for intelligent assistants, audio‑driven approaches are more suitable for personalized, many‑to‑many interactions.

Voice‑driven facial algorithms are described, including traditional phoneme‑based analysis (extracting formants, low computational complexity but requiring post‑processing), generative methods such as Wav2Lip (GAN‑based audio‑to‑lip synthesis), BlendShape prediction networks (e.g., NetEase 2019 work), and Nvidia's Audio2Face (predicting 3D facial key‑point displacements with emotion encoding).

OPPO XiaoBu Assistant implementation is detailed. The system supports two digital‑human avatars—XiaoBu (cartoon) and BuMeiMei (human‑like)—with both cloud and end‑side capabilities. End‑side Audio2Lip achieves ~92% lip‑sync accuracy for XiaoBu and 90% for BuMeiMei at 60 fps, while cloud‑side reaches ~97% and 95% respectively. The end‑side pipeline includes (1) on‑device phoneme feature extraction, (2) a customized timbre library, and (3) adaptive BlendShape driving.

The cloud‑side pipeline uses wav2vec 2.0 for audio feature extraction, aligns phoneme timelines, and fuses features with an adaptive network to output BlendShape coefficients, achieving similar high accuracy.

Sing2Lip is introduced for singing scenarios, incorporating genre recognition, melody/rhythm/pitch analysis, and a music‑vowel database to generate BlendShape weights, supporting Chinese, English, and Cantonese.

The article compares BlendShape and Mesh driving methods. BlendShape offers industry‑standard generalization and low bandwidth (transmitting 60‑100 weight values), while Mesh provides higher realism but requires transmitting thousands of vertex displacements and has lower generalization. A hybrid model converting Mesh to BlendShape coefficients is suggested.

Current challenges in the field are outlined: high cost of 4D scanning for modeling, gaps between algorithmic UV/lighting and artist‑rendered results, limited real‑time high‑quality rendering on edge devices, insufficient evaluation metrics for naturalness and the uncanny valley, and lack of coordinated facial‑body‑expression pipelines.

The Q&A section addresses lip‑sync evaluation (MOS‑like human rating), data availability (limited Chinese 4D datasets), editability of Mesh models (possible via Maya/3ds Max), and motion capture for actions.