AIGC Video Generation Techniques for E‑commerce: Lip‑Sync, Head/Body Driving, and Business Applications

This article presents a comprehensive overview of recent advances in AI‑generated content (AIGC) for video creation, focusing on applications within the Taobao e‑commerce platform. It describes how low‑cost, high‑throughput AIGC pipelines can produce diverse video assets such as feed‑style introductions, search‑driven explanations, and product detail clips.

Lip‑Sync Driving (Wav2Lip) : The classic Wav2Lip model uses a GAN architecture with three modules—Speech Encoder, Identity Encoder, and Face Decoder—to generate frame‑wise lip‑synchronized video. Its loss combines a visual fidelity discriminator, pixel‑wise L1 loss, and a sync loss that aligns audio and visual streams.

Improved Lip‑Sync (MuseTalk) : MuseTalk replaces iterative diffusion with an image‑inpainting backbone, incorporates Whisper‑derived audio features, cross‑attention fusion, VGG‑based perceptual loss, and a SyncNet sync loss, achieving higher visual quality and near‑real‑time performance.

Head‑Driven Generation : Open‑source methods such as Hallo (Fudan) and EchoMimic (Ant Group) extend lip‑sync to full head motion. Hallo introduces hierarchical audio‑visual cross‑attention, while EchoMimic adds a Landmark Encoder and random landmark sampling to improve fidelity and flexibility.

Body‑Driven / Co‑Speech Generation : The article distinguishes generative (speech‑to‑pose‑to‑video) and retrieval‑based pipelines. AnimateAnyone exemplifies the generative route with a VAE‑based denoising UNet, PoseGuider, ReferenceNet, and CLIP encoder. Tango illustrates retrieval‑based generation using a motion graph, cross‑modal similarity search (AuMoCLIP), and diffusion‑based interpolation, enhanced by a Background Guider.

Business Workflow : A four‑stage pipeline—material generation & selection, character driving, quality filtering, and compositing—is described. Techniques for face diversity (FuseAnyPart) and video‑level garment swapping (GPD‑VVTO) are introduced to enrich the persona library and support virtual try‑on scenarios.

Results & Outlook : Deployments of these technologies in Taobao’s marketing videos have significantly increased click‑through rates. Ongoing efforts aim to scale production, expand model capabilities, and explore new creative applications.