Evolving Enterprise Data Architecture for the Large‑Model Era: Practices and Case Studies

In the era of large language models, enterprises face three major data challenges: predominance of unstructured, scattered data; siloed platforms lacking a unified access method; and poor discoverability due to mismatched resource vocabularies. To support large‑model consumption, the author maps these issues to a three‑node workflow: (1) Data pipeline to a vector database, providing high‑quality, unified data assets; (2) Data orchestration, which normalizes heterogeneous structures (KV, table, graph, hierarchical) via model‑driven algorithms; (3) Data interaction, requiring high‑throughput I/O and efficient compute.

The proposed solution is a data‑virtualization engine that builds a logical abstraction layer over heterogeneous sources. Core techniques include:

Virtual tables : logical tables that expose unified views (wide tables, streaming‑batch hybrid tables) while preserving data lineage.

Intelligent materialization : AI‑driven, real‑time generation of materialized views for both offline warehouses and OLAP workloads, balancing performance and cost.

High‑performance compute : a custom KV storage offering ten‑fold I/O throughput over traditional engines and a vectorized execution engine delivering up to twice the performance of mainstream engines on large‑table joins and aggregations.

Broad compatibility : plug‑in support for external engines such as Hive, Spark, ClickHouse, and StarRocks, reducing deployment complexity.

Large‑model optimization : multi‑index and vector‑function support, optional integration with external vector databases, and storage‑level enhancements for vector‑search workloads.

Case Study 1 – Bank Data‑Governance : A leading financial institution partnered on a proof‑of‑concept to apply virtualization for data‑warehouse health monitoring. The workflow involved deep data‑warehouse analysis, ingestion of SQL and metrics, automated generation of dependency graphs, and health‑score visualizations. Results included 100 % data accuracy, minute‑level model‑change alerts, and a 95 % recommendation‑accuracy rate for optimization rules.

Case Study 2 – Bank Data‑Analysis Platform Acceleration : Another bank struggled with data redundancy, SQL divergence, and duplicated processing across OLAP and streaming pipelines. Deploying the virtualization engine enabled dynamic BSP and streaming handling, unified SQL interfaces, automatic task orchestration, and gradual engine‑capability fusion. The outcome was a consolidated system that accelerated queries, merged batch‑stream workloads, and simplified data management.

Overall, the analysis concludes that integrating data‑virtualization techniques is essential for transforming chaotic, multi‑system data landscapes into a unified, high‑performance foundation that can fully serve large‑model applications.