DeepSeek‑R1: Training Innovations and Architecture for High‑Performance Reasoning LLMs

Overview

DeepSeek‑R1’s release is a milestone for AI development, especially for the machine‑learning research community, because it provides an open‑source distilled version and fully transparent training methods for building inference‑oriented models similar to OpenAI’s O1.

Basic Large‑Model Training Pipeline

The standard training process consists of three stages:

Pre‑training: next‑token prediction on massive internet data to acquire fundamental abilities.

Supervised fine‑tuning (SFT): instruction understanding and execution training to build basic dialogue capability.

Preference alignment: optimizing model behavior according to human preferences to produce a usable version.

DeepSeek‑R1 Innovative Training Methods

2.1 Long‑Chain Reasoning Data

60 000 high‑quality reasoning samples were generated with detailed thought processes. Because manual annotation is prohibitively expensive, a special data‑generation pipeline was used.

2.2 Transitional Inference Model

An intermediate model focused on reasoning was first trained; although it performs modestly on other tasks, it requires only a small amount of labeled data to excel at reasoning problems. This model then generates large‑scale training data for the final version.

2.3 Large‑Scale Reinforcement Learning

Reinforcement learning (RL) training is divided into two key phases:

R1‑Zero: a reasoning‑oriented RL model that creates SFT data points.

Using the transitional model to generate high‑quality training data (cold‑start data) through methods such as few‑shot prompting, self‑generated reflective answers, and human‑curated outputs.

Cold‑start data consists of about 5 000 samples, which are amplified to 600 000 high‑quality examples via the intermediate model.

2.3.1 R1‑Zero: Reasoning‑Focused RL

R1‑Zero is built from a pre‑trained model and, through RL, achieves performance competitive with OpenAI’s O1 without requiring massive labeled datasets.

2.3.2 Using the Transitional Model to Generate Training Data

The transitional model is fine‑tuned on a few thousand reasoning examples (the “cold‑start” set) and then used to produce large‑scale, high‑quality data for the final model.

2.3.3 Comprehensive RL Optimization

The final R1 model incorporates additional RL components: verification for non‑reasoning tasks, helpfulness evaluation similar to Llama, safety reward models, and user‑experience optimizations, enabling strong reasoning while remaining versatile for everyday dialogue.

Architecture Details

DeepSeek‑R1 follows a transformer decoder design with 61 blocks; the first three are dense, and the remaining are expert‑mixing layers, balancing performance and computational efficiency.

Model dimensions and hyper‑parameters are illustrated in the accompanying diagrams (omitted here for brevity).

Conclusion

DeepSeek‑R1 demonstrates a reproducible, high‑performance approach to building reasoning‑capable LLMs, offering the community valuable insights and a complete technical recipe.

Example Code Snippet Used in RL Evaluation

编写Python代码，接受一个数字列表，按排序顺序返回它们，但也在开头添加42。