Building Data Lake Solutions with Iceberg and Object Storage: Architecture, Write/Read Processes, and Storage Optimization

1. Data Lake and Iceberg Overview

Data lake requires massive storage (object storage, cloud, HDFS), support for various data types, unified metadata, and multiple compute engines such as Flink, Spark, Hive, Presto.

2. Structured Data Use Cases

Describes typical scenarios where structured data is ingested, requiring schema flexibility, ACID guarantees, and lightweight schema evolution.

3. Iceberg Table Architecture

Explains Iceberg’s snapshot metadata, manifest list, manifest files, and data files (Parquet, ORC, Avro) and how they enable ACID and efficient queries.

4. Write Process

Shows Flink‑based write flow: Data Workers parse records, write to Iceberg, partitioned files are generated, checkpoint triggers commit worker to merge manifests and create a new snapshot.

5. Read Process

Describes Flink table scan: selects appropriate snapshot, filters manifest list/files, reads data files, and returns records without costly list operations.

6. Catalog Functionality

Iceberg Catalog abstracts storage and metadata; implementations can customize file I/O, namespace, and table operations.

7. Object Storage vs HDFS

Compares scalability, small‑file handling, multi‑site deployment, and storage overhead, highlighting advantages of object storage for data lakes.

8. Append Upload & Atomic Commit Challenges

Discusses S3 multipart upload (MPU) and Dell EMC ECS append upload, and solutions for atomic commits using distributed locks or If‑Match semantics.

9. Demonstration Setup

Uses Pravega as a Kafka‑like stream, Flink reads from Pravega, writes to Iceberg with ECS Catalog, achieving a fully object‑storage‑based data lake.

10. Storage Optimization Ideas

Proposes reducing Parquet redundancy by generating on‑the‑fly data files from source files and pre‑computing frequently used columns to save space while maintaining query performance.