The Magic of GPT‑4o: Technical Overview and Speculated Architecture

Recently the GPT‑4o image generation model was released, bringing breakthrough performance and new usage patterns. The author collected current technical information, hoping to spark discussion among experts.

Contents

1. GPT‑4o’s magical capabilities 2. Speculated technical route of GPT‑4o 3. Conclusion

1. GPT‑4o’s magical capabilities

GPT‑4o can generate extremely long texts, produce images from textual prompts, edit images via dialogue, and follow style references (e.g., creating posters, manga, or emojis). The article shows several examples with screenshots, demonstrating the model’s ability to generate high‑quality images, perform style transfer, and edit images interactively.

More examples can be found on the official OpenAI page: https://openai.com/index/introducing-4o-image-generation/.

2. Speculated technical route

The author summarizes community speculation that GPT‑4o still relies on an autoregressive (auto‑regression) backbone, unlike most modern text‑to‑image models that use diffusion. The autoregressive approach generates images token‑by‑token, similar to how text is generated.

Key points include:

Autoregressive generation predicts the next token based on all previous tokens, which works well for language because tokens are naturally discrete.

For images, a tokenization pipeline is required. Early attempts (e.g., Image‑GPT) treated each pixel as a token, but this is inefficient. Modern pipelines use auto‑encoders (VAE, VQ‑VAE, VQ‑GAN) to compress images into a discrete codebook.

VQ‑VAE introduces vector quantization to map continuous latent vectors to a finite set of tokens, enabling autoregressive modeling.

VQ‑GAN adds a discriminator to improve visual fidelity.

FlowMo combines a multimodal transformer (MMDiT) with a rectified flow decoder, offering a more efficient diffusion‑style decoder while keeping the autoregressive encoder.

The article also presents the training stages used by large multimodal models:

Pre‑training on massive text, image, and video data.

Quality‑focused fine‑tuning with high‑resolution, high‑quality samples.

DPO (Direct Preference Optimization) reinforcement learning using human feedback to align generation quality and text‑image consistency.

For illustration, the author shows the token format used by Emu3 (a multimodal autoregressive model):

[BOS]{text}[SOV]{metadata}[SOT]{visual tokens}[EOV][EOS]

This format unifies text and visual tokens, allowing the model to ingest mixed‑modality inputs in a single sequence.

3. Conclusion

Autoregressive image generation is not new, but it was overtaken by diffusion due to speed and quality. However, diffusion struggles to provide a truly unified model for all tasks, whereas an autoregressive multimodal model can handle generation, editing, and understanding in a single framework. The author believes GPT‑4o follows a similar path: massive data, large scale, extensive SFT and RLHF, resulting in a powerful, general‑purpose model.