Understanding ChatGPT: Architecture, Training Strategies, and Alignment Challenges

Since its release, ChatGPT has attracted massive attention, prompting the question of how it actually works despite the lack of publicly disclosed implementation details.

ChatGPT is OpenAI's latest large language model, a significant upgrade over GPT‑3, capable of generating text in many styles and purposes with improved accuracy, detail, and contextual coherence, and it is specifically designed for interactive use.

OpenAI fine‑tunes ChatGPT using a combination of supervised learning and Reinforcement Learning from Human Feedback (RLHF). The RLHF component incorporates human feedback to minimize unhelpful, distorted, or biased outputs.

The article analyses GPT‑3's limitations, explains the principles of RLHF, shows how ChatGPT leverages RLHF to address those limitations, and finally examines the remaining challenges of this approach.

Capabilities vs. Consistency – Capability refers to a model's ability to optimize its objective function (e.g., predicting the next token), while consistency concerns whether the model’s behavior aligns with human expectations and the intended task. An example of inconsistency is a model that achieves low training loss but performs poorly on a test set because the loss does not reflect the true goal.

Typical inconsistency problems in large language models include:

Providing invalid help that does not follow user instructions.

Fabricating facts or hallucinating information.

Lacking interpretability, making it hard to understand decisions.

Producing harmful bias inherited from training data.

These issues stem from the core training objectives—next‑token prediction and masked‑language modeling—which focus on statistical patterns rather than semantic meaning, causing models to optimize for token likelihood without guaranteeing correct or useful outputs.

"The cat sat on the" → model may predict "mat", "chair", or "floor" based on context.

"The [MASK] sat on the" → model may fill the mask with "cat" or "dog".

Researchers are exploring ways to mitigate inconsistency, and ChatGPT addresses it by applying RLHF, the first model to use this technique in a real‑world setting.

RLHF Pipeline

The pipeline consists of three steps:

Supervised Fine‑Tuning (SFT) : Collect a small, high‑quality dataset (≈12‑15 k examples) of prompts and desired outputs, and fine‑tune a pre‑trained GPT‑3.5/text‑davinci‑003 model, often using a code‑model variant.

Reward Model (RM) Training : Human annotators rank multiple SFT outputs for each prompt, creating a dataset roughly ten times larger than the SFT data, which is used to train a reward model that predicts human preference.

Proximal Policy Optimization (PPO) Fine‑Tuning : Initialize the policy with the SFT model and the value function with the RM. PPO optimizes the policy using the RM as a reward while applying a KL‑penalty to prevent over‑optimizing the RM.

Performance Evaluation

Evaluation relies on human ratings of helpfulness, truthfulness, and harmlessness, using prompts that were not seen during training. The model also undergoes zero‑shot testing on traditional NLP tasks, revealing an “alignment tax” where RLHF improves alignment but can reduce performance on some tasks.

Limitations

Human annotator bias influences the fine‑tuning data.

Lack of controlled studies makes it unclear how much RLHF alone contributes to improvements.

Human preferences are heterogeneous, yet RLHF treats them as homogeneous.

Reward model stability under prompt variations is not well studied.

Potential for over‑optimization where the policy learns to game the reward model.