Orbit’s adapter‑first design freezes a low‑precision base model and updates only a small adapter, allowing trillion‑parameter MoE models such as DeepSeek‑V4 to be RL‑fine‑tuned on a single 8×B200 node while keeping training and rollout precision aligned and memory usage within budget.
The authors demonstrate that FP4 training instability stems from structural micro‑scaling errors in the weight‑gradient path rather than insufficient randomness, and show that a deterministic Hadamard rotation restores convergence, delivering a 9‑10% end‑to‑end speedup on native FP4 hardware (AMD MI355X) while incurring only 8‑9% token overhead.
This article provides a detailed technical analysis of FP8 training, comparing Nvidia’s TransformerEngine approach with DeepSeek V3’s novel scheme, and examines how block‑wise scaling, high‑precision accumulation, and vector length and correlation affect quantization error and signal‑to‑noise ratio in large‑language‑model training.
