Impala Architecture, Concurrency, CBO Join Optimization, and Storage Layer in Tencent Financial Big Data Scenarios

Speaker and Platform Guest: Deng Wei, Senior Engineer at Tencent; Organizer: DataFunTalk.

Impala Overview Impala is deployed in Tencent's financial big‑data environment to provide interactive, low‑latency analytics. It stores data in Kudu (real‑time click‑stream) and HDFS (large‑scale batch). Main application scenarios include interactive analysis, tag‑factory construction, user‑profile analysis, and AB‑testing support.

Key Characteristics • No reliance on YARN; runs as resident processes. • Uses RPC‑based data shuffle with batch‑wise streaming between stages. • LLVM‑driven dynamic code generation for high performance.

Concurrency Principles

Thread hierarchy: instance threads, scan (decompression) threads, and I/O threads. In default mode, computation and scanning are one‑to‑many; in multi‑thread mode, scanning merges into instance threads.

Issues encountered: Severe jitter under high concurrency caused by RPC exchange queues filling up, leading to deferred RPC processing. "Cast to string" problem that degrades concurrency.

Solutions: Adjust datastream_service_num_deserialization_threads from the default to 80 (CPU hyper‑threads = 96) to improve stability and query efficiency. Recognize that the smallest concurrency unit is a file; too few rows per file reduces parallelism, while splitting files too much harms I/O efficiency.

CBO‑Based Join Optimization

Basic workflow: collect table statistics (row count, column selection, join type), decide broadcast vs. partition vs. ordered joins, and determine which side is the large table.

Three typical problems and remedies: Outer‑join data skew – use a "pivot table" strategy to control join order, improving efficiency. Inconsistent GROUP BY ordering across sub‑queries – enforce uniform GROUP BY order to reduce unnecessary exchanges. Average‑distribution assumption leading to inaccurate size estimates and unwanted broadcast joins – adjust statistics or rewrite queries.

Broad application: virtual‑cube multi‑table joins replace heavyweight user‑segment packages, dramatically speeding up cross‑theme analyses such as banner‑view to purchase conversion tracking.

Storage Layer

Principles: pre‑compute common dimension combinations, build inverted indexes and columnar statistics; columnar storage reduces I/O by reading only needed columns.

Data filtering pipeline: row‑group statistics → page index → dictionary encoding (e.g., PLAIN DICTIONARY) to prune data early.

Optimization techniques: global sorting, hash‑partition‑then‑sort, and Z‑order sorting to enhance filter effectiveness.

Practical impact: profile analysis time for 18 million users dropped from 20 minutes to 1 minute after migrating from Spark‑based pipelines to Impala‑driven joins.

Summary and Reflections

OLAP engine performance roadmap: vectorization and dynamic code generation to minimize type‑checking branches.

Impala optimization approaches: Problem‑driven (basic and advanced tuning based on profiling and source‑code adjustments). Metric‑driven (stress‑test I/O, CPU, network, then tune parameters and monitor key runtime metrics).

Thank you for listening. For those interested in big‑data architecture and optimization, feel free to contact [email protected].

PS: We are recruiting engineers with experience in Flink, Spark, HBase, Impala, Presto, etc., to join our financial big‑data platform team.