Emergence in Large Language Models: Phenomena, Explanations, and Implications

The article, originally presented at the 2023 China AI Society "ChatGPT and Large Model" symposium, introduces the concept of emergence in complex systems and applies it to large language models (LLMs), describing how macro‑level capabilities arise that cannot be explained by individual components alone.

Examples of emergence in everyday life—such as snowflake formation, traffic jams, and animal migration—illustrate how many simple interacting units can produce unexpected, organized patterns when their number reaches a critical threshold.

LLMs exhibit similar emergence: as model parameters grow beyond certain limits (often >100 B), tasks that previously performed poorly suddenly achieve high accuracy. Three categories of task behavior are identified:

Scaling law tasks (knowledge‑intensive) improve steadily with size.

Emergent tasks (multi‑step, complex) show a sharp performance jump once a critical scale is crossed, exemplified by few‑shot (in‑context) learning and chain‑of‑thought (CoT) prompting.

U‑shaped tasks, where performance first declines then rises with scale, can be transformed into scaling‑law behavior using CoT.

In‑context learning allows LLMs to solve tasks without parameter updates by providing a few examples; performance dramatically improves after a model reaches a size specific to each task (e.g., 13 B for 3‑digit addition, 540 B for the Word‑in‑Context benchmark). Similarly, CoT prompting—explicit step‑by‑step reasoning—enables emergent capabilities across mathematical and symbolic reasoning tasks.

The article also examines whether reducing model size harms emergence. Small models such as DeepMind's Chinchilla (70 B) and Meta's LLaMA, trained on substantially more data, still demonstrate emergent behavior on benchmarks like MMLU, suggesting that data efficiency can compensate for fewer parameters.

Finally, the phenomenon of "Grokking"—a sudden generalization phase observed during training on low‑data tasks—is introduced as a potential analogy to emergence. While Grokking reflects training dynamics and emergence reflects scale‑driven performance, both share abrupt transitions that merit further study.

Overall, the article argues that model scaling inevitably leads to emergent capabilities, that careful data‑model trade‑offs can preserve these effects in smaller models, and that understanding Grokking may provide insights into the underlying mechanisms of emergence.