Emerging Paths Toward General AI: Trends in Large‑Scale Pretrained Models

In 2017 a ten‑page paper introduced the attention‑based Transformer architecture, famously summarized as “Attention is All You Need,” which has since become a cornerstone of modern AI, driving unprecedented advances across many fields.

The new architecture enabled two major shifts: it allowed neural networks to scale to much larger parameter counts, opening new research spaces, and it leveraged the rise of cloud computing to distribute massive models across thousands of servers.

Since 2019, the “data + compute + deep learning” surge has accelerated: OpenAI released the 175‑billion‑parameter GPT‑3 in 2020, Google unveiled the 1.6‑trillion‑parameter Switch Transformer in 2021, and Alibaba’s DAMO Academy quickly followed with the AliceMind language series and the multimodal M6 model, both exceeding 10 trillion parameters while exploring commercial deployment and green‑AI considerations.

Recent research trends highlighted by DAMO Academy’s “Top Ten Technology Trends” suggest that while the race for ever larger models may plateau, collaborative evolution between large and small models, multimodal integration, and more refined, interpretable AI will dominate future work.

Notable examples include DeepMind’s Flamingo and Google’s Pathways, which shift focus from sheer parameter count to multimodal fusion, and Meta’s work on “world models” aiming to enhance reasoning and explainability.

To foster open discussion on these developments, Alibaba will host a “Large‑Scale Pretrained Models” forum at the upcoming World AI Conference, featuring keynotes and round‑table sessions that bring together academia and industry to explore algorithmic, data, and compute breakthroughs and promote open, shared research.