Iceberg Data Lake Query Optimization Practices and Governance

Introduction : The session introduces Apache Iceberg, a high‑performance table format that works with engines like Spark, Flink, and Presto, providing ACID semantics, snapshot‑based isolation, and flexible metadata management.

Iceberg Architecture : Iceberg consists of three layers – the data layer (Parquet/ORC files), the metadata layer (metadata files, manifest lists, and manifest files), and the catalog layer that gives access to tables.

Query Process : A typical SELECT query loads the current snapshot from the catalog, follows the manifest list to locate relevant data files, and reads only the necessary files, enabling full‑table scans when needed.

Hidden Partitioning : Iceberg can create hidden partitions to avoid full scans on high‑cardinality columns, pushing down filters to the storage layer and achieving significant data‑skipping.

Time Travel : All historical snapshots are retained, allowing queries on past data states via snapshot IDs, which also capture DDL and schema changes.

Key Features : Snapshot‑based read/write separation, stream‑batch unified I/O, schema and partition evolution, engine‑agnostic table format, and ACID guarantees.

Data Ingestion : Two main approaches are presented – real‑time pipelines (Flink/Spark → Iceberg) that replace traditional MQ for incremental consumption, and batch migration of existing Hive tables to Iceberg.

Query Optimization Challenges : Small file proliferation from minute‑level writes and lack of data‑skipping for large, unpartitioned datasets degrade performance.

Optimization Techniques : Compaction Service: automatically merges small files. Expiration Service: cleans up obsolete metadata and data files. Cluster Service: re‑partitions data to improve multi‑condition queries. Cleaning Service: manages lifecycle similar to Hive.

Data Organization Optimizations : Bin‑Packing: merges small files based on size thresholds. Sort: orders data by a single column to improve min/max statistics for data‑skipping. Z‑Ordering: multi‑dimensional space‑filling curve that groups rows by multiple columns, dramatically reducing scanned data.

Z‑Ordering Implementation : The process samples column ranges, computes Z‑values by interleaving binary representations, and rewrites files partitioned by these Z‑values. SQL extensions and APIs expose this functionality to users.

Experimental Results : Using the SSB benchmark (scale‑100, ~60 GB, tens of thousands of files), three scenarios were compared – raw scan, bin‑packing, and Z‑ordering. Z‑ordering reduced scanned data by over 80 % compared to the other methods.

Future Roadmap : Implement generic Bloom filter support for Parquet. Integrate Alluxio for transparent acceleration. Continue enhancing SQL extensions and Z‑ordering capabilities. Improve metric collection for read/write operations. Deploy online query platforms and refine file‑merge rules.

End of the presentation.