FineFT: Efficient Risk-Aware Reinforcement Learning for Futures Trading

Background

Crypto futures have high leverage and liquidity, but most reinforcement‑learning (RL) methods target spot markets. High leverage amplifies reward volatility, making training unstable, and existing methods lack self‑awareness of capability limits, risking large capital loss in unseen market states.

Problem definition

Two challenges for RL in high‑leverage futures: (1) convergence difficulty caused by large reward fluctuations; (2) risk‑control deficiency because agents cannot recognize when they operate outside their learned capability boundary.

FineFT framework

FineFT is a three‑stage ensemble RL framework designed to address these challenges.

3.1 Selective‑update efficient ensemble

Initialize N deep Q‑networks (DQNs) with identical MLP architecture but different random seeds. Construct a constrained optimal‑action value to generate demonstration transitions. Pre‑train each learner using Huber TD error and an optimal‑value supervisor, accelerating acquisition of basic trading principles. For each transition compute an ensemble TD‑error (ETD) matrix; assign weights proportional to ETD accuracy and update only the most accurate learner and its neighbors. This selective update avoids random updates of poorly performing agents and creates a positive feedback loop that specializes agents for specific market dynamics.

3.2 Ensemble filtering and boundary identification

Segment market dynamics by slope, producing distinct market sequences. Back‑test each ensemble element on each segment and retain the agent with the highest average profitability, forming a diverse high‑performing policy pool. For each market dynamic train a variational auto‑encoder (VAE) with negative log‑likelihood and KL‑divergence losses, recording a reference score. During testing, VAE reconstruction scores detect out‑of‑distribution (OOD) states, signalling that a learner has exceeded its capability boundary.

3.3 Risk‑aware heuristic routing

Define a conservative strategy: if a single trade’s maximum drawdown exceeds 5 %, close the position; otherwise maintain it. This reduces position‑change frequency and ensures timely liquidation in unfamiliar market states. Routing uses a sliding window of recent market states, computes VAE scores, builds an empirical CDF for each state, applies EMA smoothing, and selects the dynamic with the highest score. If the score falls below a risk threshold, the conservative strategy is chosen; otherwise the corresponding learner is deployed.

Experimental setup

Data: four major cryptocurrency futures covering >2 years (bull, bear, volatile periods). Split chronologically: training (first year), validation (last six months of second year), test (final six months). Evaluation metrics: total return (TR), annualized Sharpe ratio (ASR), Calmar ratio (CR), annualized Sortino ratio (ASoR), maximum drawdown (MDD), volatility (AVOL). Baselines (12): PPO, DRA, DQN, CRP, MACD, IV, EDQN, SUNRISE, RAQR, WINOW, EHFT, MHFT. Hardware: 4 × NVIDIA RTX 4090 GPUs, AMD Ryzen Threadripper PRO 5995WX CPU; training time ≈ 6 h. Transaction cost = 0.02 %, leverage = 5.

Results

FineFT significantly outperforms all baselines on the six financial metrics, maintaining profitability in familiar market states and switching to the conservative policy in unfamiliar states, thereby avoiding large losses. Statistical tests show p < 0.05 for return and Sharpe ratio improvements and p < 0.005 % for maximum‑drawdown reduction.

Ablation studies

Selective‑update effectiveness: Comparing training strategies shows FineFT’s pre‑training (FP) requires the fewest convergence steps and achieves the highest final reward, confirming that selective updates focus agents on appropriate market dynamics and improve convergence.

Risk‑aware routing effectiveness: Removing the risk threshold (FineFT_wo_risk) leads to mis‑identification of dynamics, erroneous trades, and substantial losses, demonstrating the importance of the risk‑aware component.

Case study

Visualization of the selective‑update mechanism shows different agents specialize in distinct market dynamics (e.g., some update during sudden price spikes, others during sharp declines), validating the mechanism’s effectiveness. VAE‑based boundary detection successfully identifies OOD market states during testing, prompting FineFT to liquidate positions and prevent further losses.

Paper: https://arxiv.org/pdf/2512.23773

Code: https://github.com/qinmoelei/FineFT_code_space