Building a Sub‑Second Response Lakehouse Platform with Apache Iceberg at Bilibili

Background: Bilibili built a lake‑warehouse integrated platform on Apache Iceberg to address Hive’s performance, complexity, data silo, and latency issues, aiming for interoperable access, fast interactive queries, and low user barriers.

Architecture: Data is stored as Iceberg tables on HDFS, ingested via Flink, Spark, or Java APIs; Magnus continuously optimizes tables (sorting, indexing, cube building). Alluxio provides caching, and Trino serves interactive queries, with fallback to ClickHouse or Elasticsearch for millisecond‑level latency.

Iceberg Table Structure: Iceberg manages file‑level metadata with snapshots and manifests, offering an open storage format that facilitates extensions.

Query Acceleration: Multi‑dimensional sorting (Hibert Curve preferred over Z‑ORDER) reduces scanned files; file‑level indexes (BloomFilter, Bitmap, BloomRF, Token/N‑gram variants) further prune data. Pre‑computed aggregates (Cubes) are generated per file and stored in manifests, supporting global merge at query time. Star‑Tree indexes enhance Cube performance for varied dimension combinations.

Intelligent Optimization (Magnus): The service automatically monitors Iceberg writes, triggers Spark jobs for sorting, indexing, and Cube creation, visualizes table metadata, and provides recommendation based on query logs, table statistics, and user‑defined preferences.

Current Status: The platform serves BI reports, metric services, A/B testing, audience selection, and log analysis. Iceberg tables total ~5 PB with a daily growth of 75 TB; Trino handles ~200 k queries per day, with 95th‑percentile latency around 5 seconds, targeting sub‑second to 10‑second response times.