Testing Gemini Omni: Turn Sketches into Cinematic Videos with One Prompt

Overview

Gemini Omni is Google DeepMind’s new “world model” that learns directly from raw video, audio, image, and text data and produces multimodal outputs (video, image, audio, text) without treating any modality as an optional conditioning layer.

Key differentiators (a16z)

Large‑language‑model‑level conversational editing is integrated into the video model, enabling iterative modifications and role extensions across scenarios.

A “digital‑twin” capability lets users create a personal avatar (cloned appearance and voice) that can be embedded into generated scenes.

Training objective and evaluation pipeline

The training goal from day one is “multimodal in, multimodal out.” Image, audio, video, and text are treated as primary training data rather than auxiliary conditioning. During evaluation the team runs five parallel pipelines: video generation, video editing, image generation, text alignment, and audio sync. Optimizing one pipeline can cause regressions in another, so trade‑offs require deep intuition.

Emergent capabilities

Despite never being trained on paired “same video, different style” data, Omni can perform style transfer (e.g., converting a video to a crayon‑drawn style). It can also continue a narrative: given a prompt such as “a woman walks down a hallway and a monster emerges from a door,” the model preserves hallway geometry, lighting, and the woman’s appearance while adding the monster and smoothly transitioning the camera.

Cross‑modal benefits

Training jointly on multiple modalities creates a mutually reinforcing relationship:

Learning to generate music improves video coherence.

Learning to draw enhances physical reasoning about light and perspective.

Learning video editing sharpens causal understanding because editing requires knowing how changes propagate.

Safety and transparency measures (the “cages”)

Avatar Flow : Users must capture multi‑angle facial images and record a spoken numeric passphrase to create an “Avatar.” The Avatar must be used for any personal‑face generation, preventing arbitrary image uploads.

Forced watermark : Every Omni‑generated video embeds Google’s invisible SynthID watermark plus C2PA cross‑platform metadata. The watermark survives editing, compression, and redistribution, enabling provenance checks (e.g., querying the Gemini app to determine if a video was AI‑generated).

Why Omni is not Veo 4

Veo’s training target was classic text‑to‑video: generate a video from a text prompt, with later extensions adding a conditioning layer for image references. Omni’s target is fundamentally different—learning the world from raw multimodal data rather than layering additional inputs onto an existing model. Product lead Nicole Brichtova described the change as a “step change” and emphasized that the foundation had to be rethought from the ground up.

Google’s naming history (Gemini 1.5, 2.0, 2.5; Veo 1‑3) follows a conservative, engineering‑culture pattern. The new name “Omni” breaks that pattern, signaling a new product line and a shift in model design.

Quantitative details

During evaluation the team runs five pipelines simultaneously (video generation, video editing, image generation, text alignment, audio sync). Optimizing any one pipeline can cause regressions in the others, requiring careful trade‑off decisions.

References

https://x.com/MTSlive/status/2056895733207597244

https://x.com/joshwoodward/status/2056827449556845051

https://x.com/jerrod_lew/status/2056865054130319828

https://www.youtube.com/watch?v=5T0yRNmNRi4