DeepSeek-V3, DeepSeek-R1, and Janus‑Pro: Architecture, Training Techniques, and Performance Insights

The piece begins with a brief introduction highlighting the significance of open‑source large language models and positioning DeepSeek‑V3 as a competitive alternative in the AI landscape.

DeepSeek‑V3 Architecture

DeepSeek‑V3 employs a Mixture‑of‑Experts (MoE) design where multiple expert networks are gated by a routing network, enabling activation of only a fraction of the total 671 B parameters (approximately 37 B per token). It introduces finer‑grained experts with shared experts and a top‑k routing mechanism.

The model also features Multi‑head Latent Attention (MLA), which compresses the key and value vectors into a low‑rank latent space, dramatically reducing KV cache size. An auxiliary‑loss‑free load‑balancing strategy adds bias terms to routing scores, dynamically adjusting them during training to keep expert loads balanced without degrading performance.

Node‑limited routing limits token communication to a few nodes, achieving near‑perfect compute‑communication overlap. Multi‑Token Prediction (MTP) predicts several consecutive tokens in a single forward pass, increasing training signal density and data efficiency.

For efficient training, DeepSeek‑V3 uses DualPipe cross‑node communication to hide most communication overhead and a custom all‑to‑all kernel that fully utilizes IB and NVLink bandwidth. FP8 mixed‑precision training is adopted, with GEMM operations performed in FP8 while critical layers (embeddings, output heads, MoE gates, normalization, attention) retain BF16/FP32 precision, achieving up to a 2× speedup with less than 0.25 % relative loss compared to BF16 baselines.

Training data includes 14.8 T tokens and a two‑stage context window expansion up to 128 K tokens. The model is fine‑tuned with 1.5 M SFT samples and further refined using reinforcement learning (RL) with rule‑based and model‑based reward models, employing Group Relative Policy Optimization (GRPO) and auxiliary‑loss‑free strategies.

https://github.com/deepseek-ai/DeepSeek-V3

DeepSeek‑R1

DeepSeek‑R1 focuses on pure RL training without an initial supervised fine‑tuning step, targeting OpenAI‑o1‑level reasoning performance. It uses a small amount of cold‑start data, multi‑stage training, and a rejection‑sampling pipeline to generate high‑quality SFT data. Reward modeling combines rule‑based accuracy rewards, format rewards, and model‑based rewards derived from a separate reward model trained on DeepSeek‑R1 outputs.

Key training steps include collecting cold‑start data, applying RL with GRPO, and generating additional SFT data via rejection sampling, resulting in roughly 800 k curated samples for further fine‑tuning.

https://github.com/deepseek-ai/DeepSeek-R1

Janus‑Pro (Multimodal Unified Modeling)

Janus‑Pro extends the unified multimodal modeling approach by separating visual encoders for understanding and generation tasks, mitigating conflicts between the two. Training incorporates a two‑stage strategy: an extended first stage on ImageNet for visual grounding, followed by a second stage using text‑to‑image data to improve generation quality.

Data augmentation includes 90 M multimodal pre‑training samples (image captions, tables, documents) and 72 M synthetic aesthetic images, balancing real and synthetic data 1:1. Model sizes of 1 B and 7 B demonstrate scalability, achieving strong performance on multimodal benchmarks and text‑to‑image instruction following.

https://github.com/deepseek-ai/Janus/blob/main/janus_pro_tech_report.pdf

Cost Statistics & Comparative Results

The article presents cost‑effective training metrics showing DeepSeek‑V3 achieving first‑tier performance with lower training expense compared to contemporaries, and includes visual charts (omitted here) illustrating these efficiencies.

Limitations

Current constraints include a 384 × 384 input resolution for multimodal understanding, affecting fine‑grained tasks like OCR, and lower resolution in text‑to‑image generation leading to less detailed outputs; increasing image resolution is suggested as a remedy.

Case Showcase

A visual case study demonstrates the model’s capabilities (image omitted).

Overall, the article serves as a comprehensive technical reference for the design, training, and evaluation of state‑of‑the‑art large language and multimodal models.