Reinforcement Learning Theory Overview and Its Application to News Recommendation

With AlphaGo’s successive victories, reinforcement learning has attracted increasing interest from both academia and industry; traditional recommendation algorithms cannot capture the relationships among news items within a request or across multiple requests, whereas reinforcement learning offers a way to learn recommendation policies.

1. Reinforcement Learning Theory Simple Review

1.1 Basic Concepts Reinforcement learning, a major branch of artificial intelligence, models sequential decision‑making by interacting with an environment to maximize cumulative reward, involving five core elements: Agent, Action, Environment, State, and Reward.

The Agent observes State and Reward from the Environment, selects an Action via policy π, and the Environment returns a new State and Reward, forming a loop that aims to maximize the Value Function V or Action‑Value Function Q .

1.2 Differences from Supervised Learning Supervised learning provides labeled data and immediate feedback, while reinforcement learning lacks explicit labels, receives delayed rewards, deals with temporally correlated State‑Action sequences, and its actions influence future states.

1.3 Overview of RL Methods For discrete, finite state‑action spaces, tabular methods (dynamic programming, policy/value iteration) or model‑free approaches (Monte‑Carlo, Q‑learning, SARSA) are applicable. When states are continuous or the search space is large, neural networks approximate the state, leading to value‑based methods (DQN, DDQN) and policy‑gradient methods (REINFORCE, Actor‑Critic). For continuous or high‑dimensional actions, deterministic policy gradients such as DPG and DDPG are used.

2. News Recommendation Modeling

2.1 Definition of Entities In a recommendation session, the Agent is the recommendation engine, the Environment is the user, Reward is +1 for a clicked news item (0 otherwise), State consists of user profile, candidate news tags, and the user's last four screen clicks, and Action is the set of ten news items presented.

2.2 Difference from CTR Prediction CTR models predict click‑through rates without learning a recommendation policy; they rely on handcrafted ranking rules. Reinforcement learning simultaneously predicts scores and learns the policy, enabling optimization of long‑term user engagement across multiple recommendation steps, though it requires more complex training.

3. Specific Methods

3.1 Deep Deterministic Policy Gradient (DDPG) DDPG combines an Actor network that outputs deterministic continuous actions with a Critic network that evaluates the action‑value. The loss functions involve gradients of Q with respect to actions and network parameters, allowing learning in high‑dimensional continuous action spaces.

The framework consists of an Actor that proposes a 10‑dimensional action (probabilities for each news item) and a Critic that predicts the value V; the NFM module encodes user‑news matching, while an RNN models the sequential selection of news.

3.2 Deep Q Network (DQN) / Double DQN (DDQN) DQN approximates the Q‑function with a neural network, using techniques such as Replay Buffer and Target Network for stability. Double DQN separates action selection (online network) from Q‑value estimation (target network) to reduce overestimation bias.

The DQN framework for news recommendation treats the action as a 10‑dimensional vector, selecting the highest‑valued news at each step; alternative designs concatenate actions from multiple requests to fit the standard DQN formulation.

3.3 Summary Both DDPG and DQN face challenges: online vs. offline training, efficient sample sampling, and training instability due to hyper‑parameter sensitivity. DDPG requires probability feedback unavailable in offline logs, making offline pre‑training essential, while DQN can be trained directly on logged data but must adapt the action representation. Consequently, the simpler DQN is often preferred for production news recommendation systems.