Understanding How ChatGPT Works: RLHF, PPO, and Consistency Challenges

ChatGPT is OpenAI's latest large language model, built on the GPT‑3 architecture and offering improved accuracy, narrative detail, and contextual coherence compared with its predecessor. Like other large language models, it generates text in various styles and for diverse purposes.

OpenAI fine‑tunes ChatGPT using a combination of supervised learning and reinforcement learning from human feedback (RLHF). The RLHF pipeline involves three main steps: supervised fine‑tuning (SFT), training a reward model (RM) from human‑ranked outputs, and applying proximal policy optimization (PPO) to further improve the SFT model.

Step 1 – Supervised Fine‑Tuning : A small, high‑quality dataset (≈12‑15 k examples) is collected by asking annotators to write desired responses to a set of prompts. The pre‑trained GPT‑3.5 model (e.g., text‑davinci‑003 ) is then fine‑tuned on this data, producing the SFT model. Because the dataset is limited, the SFT model may still generate inconsistent or undesired outputs.

Step 2 – Training the Reward Model : For each prompt, the SFT model generates multiple responses (4‑9). Human annotators rank these responses from best to worst, creating a comparison dataset roughly ten times larger than the SFT dataset. This data trains the reward model, which learns to predict human preferences.

Step 3 – PPO Fine‑Tuning : The reward model guides a PPO algorithm that updates the SFT model. PPO, an on‑policy reinforcement‑learning method, uses a trust‑region objective and a value function to estimate expected returns, applying a KL‑penalty to keep the policy close to the original SFT model and avoid over‑optimization.

The article also discusses the distinction between model ability (optimizing its training objective) and consistency (aligning outputs with human expectations). Large language models trained solely on next‑token prediction or masked‑language‑modeling often exhibit high ability but low consistency, leading to issues such as providing invalid help, fabricating facts, lacking interpretability, and exhibiting harmful bias.

Evaluation of ChatGPT relies on human‑rated prompts and outputs, measuring helpfulness, truthfulness, and harmlessness. Additional zero‑shot benchmarks (question answering, reading comprehension, summarization) reveal an "alignment tax" where RLHF improves consistency at the cost of some task performance.

The method has several limitations: dependence on annotator preferences, lack of controlled studies comparing RLHF to pure supervised fine‑tuning, potential variance in comparison data without factual grounding, and the assumption of homogeneous human preferences. Moreover, reward models may be sensitive to prompt phrasing, and PPO can sometimes over‑optimize to exploit the reward model.

References to key papers are provided, including the original RLHF paper ("Training language models to follow instructions with human feedback"), PPO algorithm paper, and related works on human‑preference reinforcement learning and alternative alignment approaches.

Training language models to follow instructions with human feedback (arXiv:2203.02155)

Learning to summarize from Human Feedback (arXiv:2009.01325)

PPO: Proximal Policy Optimization Algorithms (arXiv:1707.06347)

Deep Reinforcement Learning from Human Preferences (arXiv:1706.03741)

DeepMind Sparrow and GopherCite as alternative alignment methods

Overall, the article provides a detailed overview of how ChatGPT is trained, the role of RLHF and PPO in improving consistency, and the open challenges that remain in aligning large language models with human intent.