How Bengio’s TBA Decouples Sampling and Learning to Speed Up LLM RL by 50×

LLM post‑training with reinforcement learning is often limited not by parameter updates but by the time spent generating rollouts; on‑policy methods such as PPO, RLOO and GRPO must wait for each token‑wise generation, leaving compute resources under‑utilized.

The Bengio team proposes Trajectory Balance with Asynchrony (TBA), an asynchronous framework that splits the system into two independent pipelines: a Searcher that continuously generates trajectories using a slightly stale model, and a Trainer that samples batches from a global replay buffer to update the policy. Searchers store generated responses and rewards locally, periodically synchronising their buffers and model weights with the Trainer every k optimisation steps.

To make off‑policy data useful, TBA adopts the Trajectory Balance (TB) objective, originally from GFlowNets, which guarantees correctness as long as the sampling distribution has full support. The authors use a VarGrad‑TB variant that estimates the TB loss from multiple responses to the same prompt, avoiding the need for a separate flow‑model. In the on‑policy limit the loss reduces to a REINFORCE‑style estimator with a mean baseline and KL‑regularised rewards, while in the asynchronous off‑policy setting it shows markedly higher robustness.

The replay buffer is sampled with a mixed strategy called Most‑On‑Policy Probability . With probability p the Trainer selects the most recently added experiences (high policy freshness), and with probability 1‑p it draws from the entire history using a softmax over reward scores combined with uniform sampling. This balances data freshness against exploration diversity.

Extensive experiments on three downstream tasks demonstrate TBA’s advantages. On the GSM8K math‑reasoning benchmark, TBA reduces wall‑clock training time by nearly 50× compared with VinePPO and improves Pass@1 accuracy by 1.2‑1.8 %. In preference fine‑tuning (TL;DR summarisation) it achieves a better Pareto front between KL/perplexity and win‑rate. Across model scales from 410 M to 2.8 B parameters, TBA consistently attains higher win‑rates faster than an optimised asynchronous DPO baseline (3.8‑5.3× speed‑up). For automated red‑team evaluation, increasing the number of Searchers raises both attack success rate and prompt diversity, with TBA outperforming a non‑distributed GFlowNet baseline by up to 7× wall‑clock speed.

Scaling to a 7 B Qwen 2.5 model, the authors test a simplified variant TBA′ (based on PRIME‑RL) against Dr. GRPO. In a highly off‑policy 10‑step setting, TBA′ shows smoother learning curves and greater stability than Dr. GRPO, confirming the TB objective’s resilience under extreme off‑policy conditions.

The paper also notes a trade‑off: trajectory‑level objectives increase gradient variance, which the authors mitigate by using more responses per query (e.g., K=20 vs K=40 in GSM8K ablations). Consequently, TBA imposes stricter requirements on batch construction and sampling policies.

In summary, TBA reorganises the relationship between sampling and learning, turning massive parallel exploration into effective training signal and delivering substantial speed‑ups for LLM reinforcement learning while maintaining or improving task performance.