DeepSeek‑V3 Training Efficiency, Knowledge Distillation, and the Risks of Synthetic Data

DeepSeek‑V3 achieved top‑tier performance with a training budget of only US$5.57 million, using 2048 H800 GPUs for 2 788 kGPU‑hours, which is roughly one‑sixth the compute used for GPT‑4 MoE, highlighting its cost‑effectiveness.

The model’s efficiency stems from low‑precision computation, a small parameter count, and especially the use of data‑distillation techniques that generate high‑quality training data, thereby reducing the amount of raw data needed.

Experts such as Peter Bentley (UCL) warn that relying on synthetic data—i.e., training new models on outputs of existing AI—could cause models to collapse, emphasizing the need for human‑generated high‑quality content.

Knowledge distillation is defined as transferring the knowledge of a large “teacher” model to a smaller “student” model, preserving performance while lowering computational and storage requirements; the process includes training the teacher, preparing distilled data, training the student with teacher outputs, and optimizing the student architecture.

However, distillation has a “ceiling effect”: the student model cannot surpass the teacher’s capabilities, limiting its ability to generalize to new domains.

Additional concerns include the possibility that DeepSeek‑V3 was trained on ChatGPT outputs or used GPT‑based distillation, which could introduce low‑quality AI‑generated data into the training pipeline.

Overall, while DeepSeek‑V3 demonstrates impressive efficiency through distillation and mixed‑precision training, the technique is not a panacea, and reliance on synthetic data poses significant risks to model robustness.