Bernini: An Open‑Source AI Model that Masterfully Handles Diverse Video Editing Tasks

Today the Bernini team announced and open‑sourced a heavyweight research result called Bernini, a unified framework that cleverly fuses a multimodal large model with a diffusion model for video generation and editing, immediately achieving top scores on major video‑editing leaderboards.

Bernini can easily handle a wide range of video‑editing tasks, including adding people, adding effects, style transfer, viewpoint change, background replacement, reference‑object replacement, and multi‑reference generation.

The core architecture follows a clear division of labor. A multimodal large language model acts as a semantic planner that predicts target semantic embeddings from masked objectives. A diffusion model based on the DiT architecture serves as a pixel renderer that performs latent‑space flow matching guided by both the semantic embeddings and source video VAE features. Communication between the two is mediated by a visual Transformer (ViT) embedding space, allowing the planner to output compact high‑dimensional plans rather than full‑resolution frames.

To avoid visual token overlap when multiple video segments are concatenated, the system introduces a segment‑aware mixed‑attention mechanism and a segment‑aware 3D rotary position encoding (SA‑3D RoPE). SA‑3D RoPE assigns each segment a unique anti‑collision index and decouples segment identity from raw spatio‑temporal coordinates, enabling the attention module to instantly distinguish original video frames from reference images.

Beyond the architectural split, Bernini incorporates a two‑engine chain‑of‑thought reasoning mechanism. For simple edit commands, a textual self‑reasoning engine rewrites the prompt into a detailed script containing shot composition, actions, and temporal logic. For more complex causal edits, a visual‑textual self‑reasoning engine first generates an intermediate frame that reflects the desired visual change and then smoothly propagates this change across the entire timeline, ensuring logical consistency from understanding to generation.

Training follows a three‑stage progressive strategy. Stage 1 hones the planner using a mask‑generation paradigm to fill missing visual information from multimodal context. Stage 2 focuses on the renderer, improving high‑fidelity generation and detail preservation. Stage 3 performs lightweight joint fine‑tuning to bind the semantic planner and pixel renderer, preserving each component’s strengths while drastically reducing joint‑training overhead.

Data collection is equally massive. The team built a large‑scale, task‑rich corpus containing 20 million high‑quality video pairs and nearly 30 million image‑pair samples extracted from web‑scale corpora, plus a million‑scale motion‑aware dataset with skeleton annotations to teach the model human‑level body dynamics.

System‑level optimizations include a surgical redesign of the parallel configuration that replaces divergent operations with pre‑allocated buffers and index scattering, saving 17 GB of intermediate memory. Combined with compute‑offloading and Ulysses sequence parallelism, the model’s sequence throughput increases 4.4×, comfortably handling 440 K token sequences. A two‑stage classifier‑free guided distillation further reduces inference to four forward passes, matching the quality of an 80‑step generation.

To rigorously evaluate the framework, the researchers handcrafted the Bernini‑Bench benchmark: 300 challenging test cases covering 22 fine‑grained categories such as background replacement, viewpoint shift, focus transfer, and causally‑driven edits that require world‑knowledge reasoning. Human pairwise preference tests show Bernini excels at maintaining video consistency and faithfully following instructions, with minimal unrelated distortions.

Quantitative results on public leaderboards confirm the qualitative findings. On the OpenVE benchmark Bernini achieves a composite score of 4.04, leading in global style, background, and local modifications. On EditVerse, which emphasizes text‑video alignment, Bernini ranks first. In text‑to‑video generation tests it retains an overall score of 84.64, matching the best native generation models, and scores 62.94 in multi‑reference generation, surpassing all closed‑source competitors.

Overall, Bernini demonstrates that a clean semantic‑planner + pixel‑renderer split, reinforced by segment‑aware position encoding and chain‑of‑thought reasoning, can deliver state‑of‑the‑art video editing performance while keeping training and inference costs manageable.