How Orbit Enables Single-Node RL Fine-Tuning of Trillion-Parameter Models like DeepSeek‑V4

Large‑scale reinforcement‑learning (RL) post‑training for MoE models reaches the terabyte‑parameter level, turning RL into both an algorithmic and a systems challenge. Training must store massive weights, gradients, and optimizer states, while rollout must generate samples at high throughput, and reference policies further stress memory and scheduling.

Orbit addresses these issues by fixing the base model in a low‑precision representation used for both training and rollout, and updating only a lightweight adapter. This “adapter‑first” approach compresses RL fine‑tuning of 1‑trillion‑parameter models such as Kimi‑K2.6 and DeepSeek V4 onto a single 8×B200 node.

Memory and Precision Alignment

With an 8×B200 node offering roughly 1536 GB of HBM, full‑parameter fine‑tuning would exceed the budget, but Orbit’s frozen low‑precision base plus adapter keeps memory within limits. By using the same INT4/F​P4 base and BF16 adapter for both training and rollout, the system eliminates the discrepancy between training‑time and deployment‑time precision that often destabilizes RL.

Adapter‑First System Design

Only the MB‑scale adapter is transferred between training and inference engines, reducing synchronization volume and avoiding frequent inference engine rebuilds.

Active‑expert‑chunked dequantization groups router‑selected experts into fixed‑size batches, performs grouped GEMM on temporarily dequantized weights, and releases high‑precision weights after computation, preventing OOM in low‑precision MoE training.

Adapter‑native async with double‑buffered rollout maintains versioned adapters, streams new adapters to an inactive slot, and atomically switches when ready, cutting rollout bubble. In Qwen3‑4B + OFT on 8×B200 (TP=2) this yields a 1.42× step‑time improvement and 44 % higher rollout throughput without loss of eval accuracy.

DeepSeek V4 optimizations include Full‑CUDA graph decoding, DeepGEMM, DeepEP V2, and custom GEMM backward kernels that bypass base‑weight gradients, leveraging the frozen base.

Experimental Results

Kimi‑K2.6 : Running on a single 8×B200 node with INT4 base + BF16 adapter, reward, eval accuracy, and pass@k all rose steadily over ~200 RL steps, while the train‑rollout log‑probability difference remained stable.

DeepSeek V4 Flash : Using FP4 base + BF16 adapter, the model showed similar upward trends in reward, eval accuracy, and pass@k over 100+ steps, with a stable log‑probability gap.

DeepSeek V4 Pro (1.6 T) : Although the RL data did not improve the already strong base, the experiment demonstrated that Orbit’s pipeline scales to 1.6‑trillion‑parameter MoE models, maintaining stable log‑probability differences and controllable GPU memory on a single node.

These results prove that Orbit can compress what traditionally required multi‑node RL fine‑tuning into a single node, and that the same hardware budget can support even larger models or allow smaller models to run on a single card with higher batch sizes, longer responses, and more frequent updates.

Conclusion

Orbit’s value lies not only in making giant models trainable but also in simplifying RL post‑training for smaller models by freezing a low‑precision base, aligning training and deployment precision, and replacing full‑model synchronization with lightweight adapter synchronization.