Why GPT Can Exhibit Intelligence Through Next‑Token Prediction: A Comprehensive Exploration of Compression, Knowledge Circuits, and Model Scaling

The piece begins with the "Octopus Test" analogy, contrasting two camps—those who view GPT‑4 as merely a sophisticated statistical pattern matcher and those who claim it has learned deeper, world‑level regularities, and it outlines the prominent proponents on each side.

It then introduces the central thesis that next‑token prediction (NTP) is essentially a data‑compression task: during training the model learns probability distributions over tokens, which can be turned into arithmetic codes. The length of the resulting code for each token equals the cross‑entropy loss, linking compression rate directly to model intelligence.

Using compression efficiency as a proxy for intelligence, the article explains how higher‑capacity models (e.g., larger LLaMA variants) achieve shorter codes, implying stronger internal representations. It connects this to the Minimum Description Length principle, illustrating with the prime‑number example.

The next section surveys empirical studies on how GPT extracts knowledge, describing the three‑stage process (attention aggregation, feed‑forward enrichment, final token extraction) and distinguishing monosemantic neurons (single‑concept encoders) from polysemantic neurons that participate in superposition coding.

It further discusses discovered knowledge circuits—task‑specific pathways involving attention heads and MLP layers—and how these circuits become more elaborate in larger models. The "circuit competition" hypothesis is presented: multiple overlapping sub‑circuits compete during inference, with larger models better able to resolve complex tasks.

Implications for in‑context learning (ICL) and chain‑of‑thought prompting are explored, proposing that examples activate task‑specific circuits while enhanced induction heads act like a K‑nearest‑neighbors mechanism, sometimes cooperating and sometimes competing with the underlying knowledge circuit.

Finally, the article reflects philosophically on GPT as a "parameter mirror" of the physical world, likening human evolution to continual pre‑training and suggesting that large language models not only reproduce observed reality but can also generate coherent alternative worlds, illustrated by a thought experiment about a brain kept alive in a nutrient‑filled vat.