Practical Deployment of Delta Lake in BI and AI Products

The talk, organized by DataFun and presented by Li Dili, R&D Director at Guanyuan Data, introduces the company’s analytics platform and its goal of making data-driven decisions accessible to business users.

Guanyuan Data’s product suite supports end‑to‑end data workflows—ingestion, development, analysis, and application—offering low‑code smart ETL and a Delta Lake‑based data explanation module that enables multidimensional analysis.

In the BI+AI scenario, a major bank’s platform serves over 40,000 active users with sub‑5‑second query latency, powered by a Spark cluster of 18,000 cores and Delta Lake storage.

The architecture relies on a data‑lake layer built on HDFS, object storage, or cloud storage, with Delta Lake providing ACID transactions, incremental updates, schema evolution, versioning, partitioning, and support for multiple compute engines such as Spark, Pandas, and delta‑rs.

Key operational components include Spark for batch processing, ClickHouse for query acceleration, DolphinScheduler for orchestration, and Delta Lake for storage management; additional tools like delta‑rs and Standalone Reader enable lightweight data access.

Delta Lake’s ACID implementation uses a delta_log directory with JSON commit files and checkpoint files to speed up reads; write isolation is serializable with optimistic concurrency control.

Performance optimizations focus on small‑file compaction, vacuuming old versions, column‑pruned reads, and keeping the platform up‑to‑date with new features such as Z‑Order and enhanced DML.

The presentation concludes with plans to adopt more cloud‑native designs, integrate additional engines (Databricks, ClickHouse), expose SQL interfaces, and develop data‑catalog‑based asset management, while continuing contributions to the open‑source Delta Lake ecosystem.