How Hallo‑Live Achieves Real‑Time Streaming Text‑Driven Audio‑Video Avatar Generation

Problem

Text‑driven audio‑video digital humans must simultaneously understand the input text (character, scene, tone, acoustic environment) and generate synchronized speech and talking‑face video. Joint audio‑video synthesis is high‑dimensional and computationally heavy. Two core bottlenecks prevent real‑time streaming:

Strict block‑causal attention blocks the video stream from seeing short‑term future audio, which is needed for anticipatory lip movements.

Few‑step distillation speeds up generation but introduces “mean‑flattening” degradation: blurred video texture, mechanical speech, and increased audio‑visual drift.

Proposed Method: Hallo‑Live

Hallo‑Live combines asynchronous dual‑stream diffusion with human‑centric preference‑guided distillation (HP‑DMD) . The system is trained in two stages.

Stage 1: Dual‑Stream ODE Init

Audio and video blocks are injected with different noise levels into a dual‑stream DiT. Training uses a block‑causal mask that aligns visibility constraints with streaming inference, ensuring that each block only attends to past and current tokens.

Stage 2: Self‑Rollout + Dual‑Stream DMD

A student model generates full audio‑video sequences autoregressively using KV caches. Reward‑weighted distillation aligns the student distribution with a preference‑adjusted teacher distribution.

Causal Fusion Block

The core unit processes the two streams as follows:

Separate block‑causal self‑attention for video and audio.

Injection of text conditioning.

Block‑causal cross‑attention to exchange information.

Video‑to‑audio attention employs a Future‑Expanding Block‑Causal Mask , allowing the current video block to attend to a short look‑ahead window of future audio keys/values while keeping the video query strictly causal.

Key Technology 1: Future‑Expanding Attention

Strict block‑causal attention limits the video stream to current and past audio, hindering natural anticipatory lip motions. Future‑Expanding Attention makes the video‑to‑audio cross‑attention asymmetric: the video query remains causal, but the audio key‑value range is extended forward by a small look‑ahead window. This creates a temporary “pre‑read” region that improves lip‑sync without affecting the final audio output because the future audio block is later overwritten.

Key Technology 2: Preference‑Guided Distillation (HP‑DMD)

Instead of directly mimicking the teacher distribution, the student is trained to match a reward‑weighted teacher distribution. Three reward modules score generated samples:

VideoAlign : measures visual alignment with text and scene.

SyncNet : evaluates lip‑audio synchronization.

AudioBox : assesses speech naturalness and acoustic quality.

Scores are exponentiated and used to re‑weight the distillation loss, effectively shaping a new target distribution rather than applying policy‑gradient RL.

Experimental Results

Speed : 20.38 FPS with 0.94 s end‑to‑end latency on two NVIDIA H200 GPUs, a 16.0× throughput increase and 99.3 % latency reduction versus the teacher model Ovi.

Quality : VideoAlign Overall = 2.32, Sync‑C = 4.72, human fidelity scores = 0.90 / 0.98 / 0.92. These metrics are comparable to Ovi and LTX‑2, showing no obvious quality loss despite the speedup.

Limitations

Synchronization and speech quality still lag behind the strongest offline models, and the current implementation requires two NVIDIA H200 GPUs, leaving room for lower‑cost hardware optimization.

References

Paper: https://arxiv.org/abs/2604.23632

Code: https://github.com/fudan-generative-vision/Hallo-Live