How ChatGPT Works: Training, RLHF, and Consistency Issues

ChatGPT is OpenAI’s newest large language model, an evolution of GPT‑3 that offers better accuracy, narrative detail, and contextual coherence. It is trained using a combination of supervised learning and reinforcement learning from human feedback (RLHF), which distinguishes it from earlier models.

The model’s training pipeline consists of three main steps: (1) supervised fine‑tuning (SFT) on a curated set of prompts and high‑quality responses, (2) training a reward model (RM) by having annotators rank multiple SFT outputs for the same prompt, and (3) applying proximal policy optimization (PPO) to further fine‑tune the SFT model using the RM as a learned objective.

RLHF addresses the “consistency” problem—where a model may follow its training objective (e.g., minimizing token‑level loss) but produce outputs that diverge from human expectations. By incorporating human preferences, the model learns to generate responses that are more helpful, truthful, and harmless.

Evaluation of ChatGPT relies on human‑rated benchmarks covering helpfulness (adherence to user instructions), truthfulness (avoidance of fabricated facts), and harmlessness (absence of toxic content). Additional zero‑shot tests on traditional NLP tasks reveal an “alignment tax”: alignment techniques can slightly reduce performance on some benchmarks.

The article also discusses limitations of the RLHF approach, such as annotator bias, lack of contrastive studies, and the difficulty of capturing diverse human values. Potential over‑optimization and KL‑penalty tricks are mentioned as mitigations.

References to key papers (e.g., the RLHF arXiv paper, PPO algorithm, DeepMind’s Sparrow) are provided for readers who wish to explore the technical details further.