Understanding OpenAI's Sora Video Generation Model: Architecture, Workflow, and Core Technologies

OpenAI released Sora, a video generation model capable of producing up to one‑minute high‑quality videos from text prompts, supporting resolutions such as 1920×1080 and 1080×1920, surpassing earlier models like Stable Video Diffusion.

The accompanying technical report outlines Sora's overall architecture without deep theoretical exposition; this article expands on the model by summarizing its key steps.

Training pipeline: large collections of videos at various resolutions and durations are down‑sampled into a latent space, where they receive textual annotations (enhanced by DALL·E 3 re‑annotation) for supervised learning.

Video generation process: a random noise video is encoded by a video compression network into low‑dimensional latent representations; these latents are split into spacetime patches, flattened into a sequence of tokens, and fed to a Diffusion Transformer (DiT) that leverages attention to denoise tokens conditioned on the text prompt.

After denoising, a visual decoder reconstructs the latent sequence back into pixel‑level video frames, supporting arbitrary resolutions without prior scaling or cropping.

The article also revisits the image‑generation background: Stable Diffusion uses a Latent Diffusion Model (LDM) with an auto‑encoder (encoder ε, decoder δ) to compress images, a UNet‑based diffusion model, and a Transformer‑enhanced attention mechanism for prompt control.

Transformer architecture, introduced in 2017, relies on self‑attention to handle long‑range dependencies and parallel computation, and its multi‑head attention layer is crucial for both text and visual tasks.

Sora extends this by training a video‑specific auto‑encoder that compresses both spatial and temporal dimensions, generating spacetime patches that are tokenized and processed by DiT, which incorporates position encodings (e.g., (x, y, t)) to preserve temporal order.

During inference, the model iteratively samples denoised latents across diffusion steps, then decodes them into the final video output.

References include OpenAI's research page, a Zhihu article, and several WeChat posts for further reading.