DeepSeek-R1: Enhancing Reasoning Capabilities in LLMs via Reinforcement Learning

The article discusses the DeepSeek-R1 series of large language models, focusing on how reasoning capabilities can be enhanced through large-scale reinforcement learning (RL) without relying on extensive supervised fine‑tuning (SFT). It introduces DeepSeek‑R1‑Zero, a model trained purely via RL using Group Relative Policy Optimization (GRPO), which eliminates the need for a critic model and reduces training cost.

The work shows that even a small amount of SFT for cold‑start can further improve performance. DeepSeek‑R1 builds on this by adding a cold‑start phase with high‑quality chain‑of‑thought data before large‑scale RL, yielding better readability and reasoning performance.

Key technical components include GRPO for efficient RL, a rule‑based reward system combining accuracy and format rewards, language‑consistency rewards to mitigate multilingual mixing, rejection sampling for generating SFT data, and a two‑stage RL alignment process that optimizes helpfulness and harmlessness.

The article also covers model distillation, showing that transferring DeepSeek‑R1’s knowledge to smaller models (e.g., Qwen, Llama) via SFT yields significant gains, whereas pure RL on small models requires massive compute and still falls short of distilled performance.

Finally, it discusses limitations such as language mixing, unsuccessful attempts with process reward models and Monte Carlo tree search, and outlines future directions like exploring ensemble learning and further RL alignment.