Building a Millisecond‑Response Lakehouse Platform with Apache Iceberg: Architecture, Query Acceleration, and Intelligent Optimization

Introduction This article presents Bilibili’s technical practice of building a millisecond‑response lake‑warehouse integrated platform based on Apache Iceberg, outlining the motivation, architecture, query acceleration techniques, intelligent optimizations, and current production status.

Background The platform addresses Hive’s performance, complexity, data‑isolation, and latency issues by providing a unified, engine‑agnostic storage layer on HDFS, supporting FIink, Spark, Java API, Alluxio caching, and Trino for interactive queries, while still allowing downstream writes to ClickHouse or ES when needed.

Query Acceleration Acceleration is achieved through multi‑dimensional sorting (using Hibert Curve), file‑level statistics for predicate push‑down, and a suite of indexes (BloomFilter, Bitmap, BloomRF, Token‑based indexes). Additionally, pre‑aggregated Cube files and star‑tree indexes are generated to speed up OLAP aggregations, with adaptive fallback when cubes are missing.

Intelligent Optimization The Magnus service continuously monitors Iceberg write events, automatically applying sorting, index creation, and cube generation. It visualizes table metadata, recommends schema and index adjustments based on query logs, and can auto‑apply validated optimizations to reduce user effort.

Current Status The solution is in production for BI reporting, metric services, A/B testing, audience selection, and log analysis, managing roughly 5 PB of Iceberg tables with 75 TB daily growth and serving ~200 k Trino queries per day with 5‑second P95 latency.