Optimizing Query Performance in Apache Iceberg with Z‑Order Data Organization

As enterprise data volumes and formats grow, analysts demand faster query speeds while maintaining accuracy. Apache Iceberg’s DataSkipping, based on file metrics, can quickly filter required data files, but its efficiency drops sharply when filter fields increase, sometimes leading to full‑table scans.

The presentation outlines five parts: I/O efficiency of query analysis, data‑organization design, technical implementation, performance evaluation, and future roadmap.

1. Query Analysis I/O Efficiency

A typical SQL query (SELECT count(*) FROM employee WHERE first_name LIKE 'Tho%' AND last_name LIKE 'Frank%' AND birthplace='newyork') uses birthplace as a partition column. Iceberg processes the query through three filtering layers: partition pruning, file‑level min‑max filtering, and RowGroup filtering, dramatically reducing scanned files.

2. Potential Problems

Min‑max filtering relies on sorted data; unsorted columns render it ineffective.

When many columns are sorted, only the first few retain good ordering, causing later columns to produce “fault” data that forces full scans.

3. Data‑Organization Optimization Design

To address these issues, the solution leverages space‑filling curves to reduce dimensionality. The concept is illustrated with a Geohash example, converting 2‑D coordinates into a 1‑D address while preserving locality.

Fault data can appear when the 1‑D range covers points outside the original 2‑D query region, increasing scan cost.

4. Z‑Order Algorithm

Tencent Iceberg implements a Z‑Order based data‑organization optimization, exposed via the native OPTIMIZE syntax.

OPTIMIZE TABLE employee ZORDER BY first_name, last_name

The workflow consists of four steps:

Candidate file selection (full table or specific partitions, with optional WHERE on partition columns).

Generate Z‑order address for each row by interleaving bits of selected columns (e.g., first_name "Thomas" → 68, last_name "More" → 102, then interleaved to form the Z‑address).

Repartition data by Z‑order address (equivalent to Dataset.repartitionByRange(ZOrderAddress) ).

Write back using Copy‑on‑Write, creating a new snapshot.

5. Performance Evaluation

Two test categories were conducted:

Key Parameter Tests – varying aggregation columns, output file size, and CUBE size. More aggregation columns degrade performance but still outperform the unoptimized baseline. Smaller output files slightly improve query speed; 1 GB is a balanced default. Larger CUBE sizes improve filtering; 150 GB is recommended.

SSB Benchmark – using modified Star Schema Benchmark queries Q3.1‑Q3.4. With Z‑Order, query latency drops below 1 s versus >12 s for 10 k small files without optimization. File‑level scanning is reduced dramatically, confirming the effectiveness of data‑organization.

6. Future Plans

Enhance continuous data ingestion capabilities.

Improve query performance, including compute‑storage separation and index support.

Boost operability and system extensibility (new compute engines, cloud catalog integration).

The session concludes with a thank‑you to the audience.