Tencent Real-time Lakehouse Intelligent Optimization Practice

Tencent's real-time lakehouse architecture consists of three parts: data lake compute, data lake management, and data lake storage. Spark serves as the batch ETL engine, Flink handles near‑real‑time streaming, while StarRocks and Presto provide ad‑hoc OLAP queries. The management layer centers on Iceberg, exposing simple APIs and an Auto Optimize Service that boosts query performance and reduces storage costs. Underlying storage uses HDFS and Tencent Cloud Object Storage (COS), with Alluxio providing a unified cache layer.

The Intelligent Optimization Service is composed of six modules: Compaction Service (merging small files), Expiration Service (removing expired snapshots), Cleaning Service (lifecycle and orphan file cleanup), Clustering Service (data redistribution), Index Service (secondary index recommendation), and Auto Engine Service (automatic engine acceleration). Each module has recent enhancements, such as RowGroup‑level and Page‑level copy optimizations for compaction, Bloom‑filter‑assisted delete‑file application, incremental rewrite based on modify time, and a two‑stage index recommendation workflow that extracts SQL, performs coarse filtering, builds incremental indexes, and evaluates effectiveness via dual‑run experiments.

Clustering Service leverages Z‑order and range‑ID techniques to reorder data for better data skipping, achieving more than fourfold performance gains in practice. Auto Engine Service collects OLAP engine events to heat relevant partitions, routing them to StarRocks for optimized query execution.

Scenario capabilities include multi‑stream stitching, where two message queues update different columns and are merged by id with timestamp ordering, and primary‑key tables that enable row‑level updates via bucketed writes and column‑family storage. In‑place migration tools allow seamless transition from legacy Hive/Thive tables to Iceberg metadata without moving data files, supporting strict, append, and overwrite modes. The new name‑mapping mechanism enhances partition pruning for built‑in functions.

PyIceberg provides a JVM‑free Python API for Iceberg tables, enabling seamless integration with Pandas, TensorFlow, and PyTorch for data analysis and AI model training, while supporting column pruning and predicate push‑down via DuckDB.

Future directions focus on further optimizing the real‑time lakehouse: expanding Auto Optimize Service with hot‑cold separation, materialized view acceleration, intelligent sensing, and refined compaction; enhancing primary‑key tables with deletion vectors; and exploring AI‑driven lakehouse formats and distributed DataFrames that unify metadata and compute.