ChatGPT: Technical Principles, Architecture, and the Role of Human‑Feedback Reinforcement Learning

The article introduces ChatGPT, OpenAI's latest large language model, highlighting its significant improvements over GPT‑3 in accuracy, detail, and contextual coherence, and noting its ability to generate text in diverse styles for interactive applications.

It explains that ChatGPT is trained using a combination of supervised learning and Reinforcement Learning from Human Feedback (RLHF), where human annotators provide preferences that are used to create a reward model guiding the model toward more aligned outputs.

The training pipeline consists of three steps: (1) Supervised Fine‑Tuning (SFT) on a small, high‑quality dataset of prompt‑response pairs; (2) Training a Reward Model (RM) by having annotators rank multiple SFT outputs for the same prompt; and (3) Applying Proximal Policy Optimization (PPO) to further fine‑tune the SFT model using the RM as a learned objective.

The article discusses the consistency problem in large language models, distinguishing between capability (optimizing a loss function) and consistency (aligning model behavior with human intent). It lists typical inconsistency manifestations such as providing invalid help, hallucinating facts, lacking explainability, and exhibiting harmful bias.

While RLHF aims to improve consistency, the article points out several limitations: data bias from annotator preferences, lack of proper control studies, heterogeneous human values, sensitivity of the reward model to prompt variations, and the risk of over‑optimizing the model to the reward signal.

ChatGPT's performance is evaluated on three human‑rated criteria—helpfulness, truthfulness, and harmlessness—as well as zero‑shot performance on standard NLP tasks, revealing an “alignment tax” where consistency improvements can reduce performance on some tasks.

The article concludes with references to the original RLHF paper, the PPO algorithm paper, and related works on instruction‑following models and alternative alignment methods, providing readers with further technical resources.