EchoMimicV2: An End-to-End Audio‑Driven Semi‑Body Human Animation Framework

EchoMimicV2 is an open‑source digital‑human project released by Ant Group’s Alipay algorithm team. The system can synthesize high‑quality semi‑body animation videos from a single reference image, an audio clip, and a hand‑gesture sequence, ensuring tight synchronization between the generated avatar and the audio content.

Resources Paper: EchoMimicV2: Towards Striking, Simplified, and Semi‑Body Human Animation Project page: https://antgroup.github.io/ai/echomimic_v2/ Code repository: https://github.com/antgroup/echomimic_v2

Motivation and Challenges Existing high‑quality digital‑human methods focus on head‑only animation and ignore the torso and hands, while multi‑modal conditioning (audio, pose, image) makes models heavier, unstable, and slow at inference.

Key Technical Contributions

1. Audio‑Pose Dynamic Harmonization (APDH) : a training strategy that gradually reduces condition complexity and coordinates audio with pose inputs, reducing redundant pose information.

2. Head Partial Attention (HPA) : a lightweight head‑region attention module that seamlessly incorporates head‑only image augmentation to improve facial expression quality without extra plugins.

3. Multi‑Stage PhD Loss : a three‑phase denoising loss (pose‑dominant, detail‑dominant, quality‑dominant) that stabilizes training and enhances visual fidelity.

Network Architecture The backbone follows the ReferenceNet design with a Reference UNet and a Denoising UNet. Three core components drive the audio‑conditioned generation:

Pose Encoder – encodes hand‑keypoint maps into latent pose features.

Audio Encoder – a pretrained audio feature extractor that provides cross‑attention signals.

Denoising UNet – receives noisy latent frames together with audio conditioning to produce the final video frames.

Training Strategy APDH consists of Pose Sampling (PS) and Audio Diffusion (AD). Pose Sampling provides diverse pose inputs while Audio Diffusion gradually introduces audio conditioning, enabling the model to handle multimodal inputs robustly.

Future Directions The authors outline several research avenues: (1) generating hand‑keypoint sequences directly from audio to remove manual pose input; (2) extending the method to arbitrary full‑body reference images; (3) improving generalization across diverse portrait styles.

Related Work The paper surveys recent progress in diffusion‑based video generation, pose‑driven, text‑driven, and audio‑driven human animation methods such as MagicPose, AnimateAnyone, V‑Express, MegActor‑Σ, and others, highlighting the shift toward multimodal conditioning and the need for stable, efficient pipelines.

References A comprehensive bibliography of recent diffusion, video synthesis, and human motion generation papers is provided (e.g., Dhariwal & Nichol 2021, Ho et al. 2020, Rombach et al. 2022, Chen et al. 2023, Hu 2024, etc.).