Lakehouse: Concepts, Architecture, Implementation, and Cloud Practices

The article, derived from a DataFunCon 2021 talk by Alibaba technical expert Chen Xinwei, introduces Lakehouse as the emerging data architecture that combines the strengths of data warehouses and data lakes, gaining popularity since the 2019 Databricks paper.

1. Evolution of Data Architecture – Traditional data warehouses (e.g., Teradata, Oracle) offered high‑performance, ACID‑compliant analytics but suffered from storage‑compute coupling and limited support for unstructured data. Data lakes built on HDFS or object storage (S3/OSS) provided low‑cost, schema‑less storage but introduced reliability, consistency, and performance challenges.

2. Lakehouse Emergence – Lakehouse retains low‑cost, open storage while adding management layers (Delta Lake, Iceberg, Hudi) that deliver ACID transactions, versioning, and optimized query performance, reducing data movement and improving reliability.

3. Architecture and Implementation – A typical Lakehouse stack consists of three layers: the access layer (metadata lookup, object‑storage access, open data formats, declarative DataFrame APIs), the optimization layer (caching, indexing, data layout, governance), the transaction layer (ACID isolation, multi‑version, time‑travel), and the storage layer (cloud object storage with Parquet/ORC formats).

4. Lake Formats – The article details Delta Lake (transaction log, MVCC, file‑level updates, Z‑ordering, Optimize, Savepoint, Rollback, automatic metadata sync) and Apache Iceberg (snapshot‑based metadata, incremental commits, snapshot diff for incremental consumption) and highlights Alibaba Cloud EMR contributions for both formats.

5. Selection Guidance – A comparative chart shows how Lakehouse integrates the advantages of warehouses and lakes, helping practitioners choose based on workload, consistency, and performance requirements.

6. Cloud Lakehouse Practice – Alibaba Cloud’s Lakehouse solution adds unified metadata (DLF), CDC ingestion, and multi‑engine support (Spark, Flink, Trino, Presto, Hive, MaxCompute). It offers zero‑code data pipelines, automatic metadata synchronization, and governance features such as cost analysis, hot‑cold data profiling, and automated optimization via a K8s‑based workload scheduler.

7. Case Studies – Two customer scenarios are presented: a fully managed data lake migration from on‑premise CDH to OSS + DLF + DDI, and a transition from a monolithic Hive architecture to a compute‑storage‑separated real‑time lake, both achieving lower operational cost and higher scalability.

8. Future Outlook – Lakehouse is expected to evolve toward database‑level capabilities (multi‑table transactions, richer Optimize functions), tighter integration of lake formats with storage and compute APIs, and increasingly intelligent management platforms that automate governance and optimization.

Overall, the article demonstrates that Lakehouse is poised to become the next‑generation big‑data architecture.