From DATA for AI to AI for DATA: Evolution of Ant Group’s Intelligent Data System

The presentation at the 2024 Bund Conference, titled “From DATA for AI to AI for DATA,” delivered by Luo Ji of Ant Group, outlines the rapid breakthroughs in generative AI and the parallel acceleration of data technologies, marking a new historical phase where data becomes a central source of value.

It recounts the evolution of data technologies: the 1990s database foundations, the 2003‑2006 big‑data era sparked by MapReduce, Bigtable, and GFS, the mobile‑internet data surge, and the 2017 transformer paper that launched large‑model AI, illustrating a shift from cost‑efficiency to value creation.

In the emerging “intelligent‑data fusion” era, data production expands beyond web crawling to include IoT devices, wearables, and smart appliances, emphasizing high‑quality, annotated, and multimodal data as essential for effective AI models.

Data assets are transitioning from structured to unstructured formats; IDC forecasts that by 2027 unstructured data will comprise 86.8% of total data, introducing challenges in cleaning, quality assessment, multimodal fusion, and specialized auditing.

Data services are extending from user‑oriented to machine‑ and agent‑oriented, requiring new semantic expressions, efficient encoding/decoding, and low‑latency, high‑throughput pipelines to support multi‑agent collaboration.

The talk identifies three key challenges for future data applications: (1) hybrid scalar‑vector retrieval for new search and interaction scenarios; (2) uncertainty in AI‑driven application outcomes, necessitating native experimental pipelines; and (3) building open data ecosystems for value discovery, secure sharing, and performance measurement.

Ant Group’s recent two‑year efforts have built a multimodal storage and compute engine, vector database capabilities, mixed‑storage architecture, and an experimental data‑quality framework to underpin this intelligent‑data ecosystem.

In conclusion, the data system is moving from a cost‑center to a value‑center, aiming to deliver inclusive, data‑driven AI services that enhance everyday life.