Large-Model RL Advances: Credit Allocation, Complex Reasoning, Agent Learning

Reinforcement learning (RL) is a paradigm where agents continuously improve their policies through a perception‑decision‑action‑feedback loop, contrasting with static supervised learning that relies on fixed data distributions. By emphasizing trial‑and‑error interaction, RL aims to move AI from passive answer generation to autonomous action, overcoming sparse rewards and static supervision.

This week HyperAI selected six recent papers on large‑model RL from top universities and tech companies, each proposing novel solutions to credit assignment, reasoning, or agent learning challenges.

ECHO: Terminal Agents Learn World Models for Free

The paper observes that terminal agents generate massive feedback but conventional RL only uses sparse rewards, wasting observations. ECHO adds a cross‑entropy prediction loss on terminal feedback while keeping the action loss unchanged, incurring no extra forward‑pass cost. This enables the policy to simultaneously predict responses to commands, effectively learning a world model for free.

Experiments show a doubling of first‑response accuracy on terminal‑control benchmarks, markedly better prediction of unseen terminal dynamics, reduced reliance on expert demonstrations, and the ability to self‑evolve without external validation.

DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

Standard RL suffers from overly coarse credit assignment, where high‑frequency patterns dominate updates and true high‑reward tokens are obscured. DelTA computes a dedicated coefficient to re‑weight a self‑normalized objective, amplifying gradient directions unique to positive or negative reward tokens while suppressing weak, shared directions.

On mathematical reasoning and code‑generation benchmarks, DelTA surpasses the strongest same‑scale baselines and demonstrates strong generalization across different model architectures.

GoLongRL: Capability‑Oriented Long Context Reinforcement Learning with Multitask Alignment

Long‑context RL is limited by homogeneous retrieval data and reward‑scale mismatches that distort advantage estimates in multitask settings. GoLongRL introduces a capability‑oriented framework covering nine core abilities and a custom reward dataset. It employs a TMN‑Reweight mechanism that normalizes tasks and applies difficulty‑adaptive weighting to focus on high‑value hard samples.

Evaluations on multiple long‑text benchmarks show comprehensive superiority over existing leading models and prevent degradation of reasoning and memory capabilities.

Anti‑Self‑Distillation for Reasoning RL via Pointwise Mutual Information

Conventional self‑distillation in math reasoning encourages models to shortcut by over‑relying on known answers, suppressing multi‑step search. The proposed Anti‑SD reverses this by maximizing Jensen‑Shannon divergence to generate gradient signals that reward exploratory tokens, complemented by an entropy‑based gate for training stability.

Across several large‑model configurations, Anti‑SD reaches target performance with only 20‑50% of the baseline training steps and improves final accuracy on math reasoning benchmarks by up to 11.5 percentage points.

RubricEM: Meta‑RL with Rubric‑guided Policy Decomposition beyond Verifiable Rewards

Long‑horizon research tasks often lack objective rewards, resulting in coarse feedback. RubricEM introduces a rubric‑guided interface that splits trajectories into planning, retrieval, review, and answer phases, enabling fine‑grained credit allocation. An asynchronous meta‑policy trains on reflective memory built from past interactions.

The 8B‑parameter model outperforms many open‑source baselines, approaches top closed‑source systems, learns efficiently with few training steps, and exhibits strong cross‑task generalization.

Poly‑EPO: Training Exploratory Reasoning Models

Post‑training of large‑model RL often collapses diversity, hindering exploration of new reasoning paths. Poly‑EPO adopts ensemble RL, defining a joint objective that multiplies the average reward of a set of responses by a diversity score, embedding the diversity signal directly into the advantage function.

On mathematical reasoning evaluations, Poly‑EPO prevents policy homogenization, raises pass@k coverage by up to 20%, and shows stronger scaling under various voting mechanisms.

These six papers collectively illustrate how large‑model RL is advancing credit allocation, complex reasoning, and autonomous agent capabilities. For more frontier AI research, visit HyperAI’s “Latest Papers” section.