Paradigm Shifts in Large Language Model Research and Future Directions

Author: Zhang Junlin, New Technology R&D Lead (Weibo). Source: Zhihu.

Overview: The rise of ChatGPT surprised many by showing how effective large language models (LLMs) can be, while also highlighting a widening gap between Chinese and world‑leading LLM research. The author reflects on this gap and argues that the difference stems mainly from divergent understandings of LLM development philosophy rather than just compute resources.

Historical timeline: After the introduction of BERT, Chinese research kept pace for a couple of years, but the launch of GPT‑3 (mid‑2020) marked a watershed. GPT‑3 demonstrated a new paradigm—LLMs should be built as generative, self‑supervised models that learn to perform many tasks without task‑specific fine‑tuning. Since then, OpenAI has stayed roughly six months ahead of Google and two years ahead of domestic teams.

Impact on NLP research: The first paradigm shift (deep learning → two‑stage pre‑training) collapsed many “intermediate” NLP tasks (e.g., POS tagging, parsing) because models like BERT/GPT internalize these features. The second shift (pre‑training → AGI‑oriented models) is driven by scaling, prompting, and instruction‑following techniques. As models grow, tasks that previously required fine‑tuning become solvable via zero‑shot or few‑shot prompting.

LLM knowledge acquisition: LLMs absorb two major knowledge types: linguistic knowledge (syntax, semantics) stored mainly in lower and middle Transformer layers, and world knowledge (facts, common sense) stored in middle and higher layers. Recent work treats the Feed‑Forward Network (FFN) as a key‑value memory, where individual neurons act as <Beijing, is‑capital‑of, China> stores.

Knowledge editing: Three families of methods exist to modify outdated or incorrect facts: (1) trace the original training data and remove it, (2) fine‑tune on corrected examples (risking catastrophic forgetting), and (3) directly edit FFN parameters to replace <UK, current‑prime‑minister, Boris> with <UK, current‑prime‑minister, Sunak> .

Scaling effects: Scaling laws show that increasing model size, data, or compute monotonically reduces pre‑training loss. Optimal compute allocation suggests scaling both data and parameters together (e.g., Chinchilla’s 70B‑parameter model trained on 300B tokens outperforms larger dense models). Larger models exhibit three performance patterns: (a) smooth improvement on knowledge‑intensive tasks, (b) emergent abilities that appear after a size threshold, and (c) U‑shaped curves where mid‑size models under‑perform before larger models recover.

Prompting techniques: Zero‑shot CoT adds "Let's think step by step" to elicit reasoning; few‑shot CoT provides explicit reasoning examples; self‑consistency generates multiple reasoning paths and votes; least‑to‑most decomposes complex problems into sub‑questions. These methods unlock latent reasoning abilities without changing model parameters.

Code‑augmented pre‑training: Models trained on both natural language and source code (e.g., Codex) show substantially higher reasoning performance across benchmarks, suggesting that code data injects algorithmic reasoning patterns into the model.

Future research directions: (1) Explore the ultimate scale limits of LLMs, (2) Enhance complex reasoning capabilities, (3) Incorporate non‑NLP modalities (vision, audio, robotics) toward AGI, (4) Design more natural human‑LLM interfaces, (5) Build high‑difficulty, user‑driven evaluation suites, (6) Improve data quality and diversity, (7) Develop sparse Transformer architectures to reduce training and inference costs, and (8) Study systematic knowledge‑editing techniques.

Practical guidance for reproducing ChatGPT‑like systems: Choose an auto‑regressive architecture (GPT style), pre‑train with large amounts of code and text, leverage high‑quality diverse data, consider retrieval‑augmented models to keep parameter counts modest, adopt sparse routing for efficiency, and focus on instruction‑tuned fine‑tuning using real user prompts.

Conclusion: The LLM field is undergoing rapid paradigm changes; understanding these shifts and the associated technical trends is essential for both academic researchers and industry practitioners aiming to build the next generation of intelligent systems.