Snowflake and Delta Lake: Architecture, Features, and Comparative Analysis

Overview – Since the 1980s data analysis has progressed from traditional Enterprise Data Warehouses (EDW) to Data Lakes and now to cloud‑native data warehouses and lakehouse solutions. Snowflake and Delta Lake represent the latest generation, aiming to combine the strengths of both worlds while leveraging cloud technologies such as Hadoop and Spark.

Snowflake Introduction – Snowflake is a fully cloud‑native, SaaS data‑warehouse platform that runs on AWS, Azure, and Google Cloud. It offers a pure SaaS experience, ANSI‑SQL compatibility, ACID transactions, and native support for semi‑structured data. Its architecture consists of three layers: a cloud‑service layer (metadata, security, query planning), virtual warehouses (elastic compute clusters), and storage (Amazon S3 or equivalent object stores). Storage and compute are loosely coupled, enabling independent scaling, caching, and multi‑cluster shared‑data operation.

Snowflake Core Technologies – The compute engine follows a shared‑nothing design with a proprietary vectorized columnar executor, supporting push‑down predicates, dynamic and static pruning, and file‑stealing to balance load. Concurrency is managed via snapshot isolation (SI) built on MVCC, providing ACID guarantees, time‑travel, and efficient cloning. Security is end‑to‑end encrypted with role‑based access control.

Delta Lake Introduction – Delta Lake, developed by Databricks, is an open‑source storage layer that adds ACID transactions, schema evolution, and time‑travel to Apache Parquet files stored in cloud object stores. It maintains a transaction log in the _delta_log directory, uses optimistic concurrency control, and checkpoints metadata in Parquet format. Features include upserts/merges, streaming ingest, automatic data layout optimization (Z‑ordering), and audit logging.

Delta Lake Core Concepts – Data files are immutable Parquet files identified by GUIDs. The transaction log records add/remove actions with min/max statistics for pruning. Reads reconstruct the latest snapshot by applying the most recent checkpoint and subsequent JSON log files. Writes involve locating the latest log ID, reading the current snapshot, writing new data files, appending a new log entry atomically, and optionally creating a new checkpoint. Isolation is provided at snapshot‑isolation level, with optional linearizable reads.

Comparison Summary – Snowflake offers a turnkey SaaS experience with strong ease‑of‑use, while Delta Lake provides an open, Parquet‑based lakehouse that integrates tightly with Spark and other analytics engines. Snowflake’s proprietary format can yield performance benefits but limits external engine compatibility; Delta Lake’s open format enables broader ecosystem use. Snowflake relies on micro‑batch ingestion (Snowpipe) for near‑real‑time data, whereas Delta Lake supports true streaming with second‑level latency. Both platforms are cloud‑agnostic, but Delta Lake also runs on on‑premise Hadoop clusters.

References

[1] The Snowflake Elastic Data Warehouse [2] Delta Lake: High‑Performance ACID Table Storage over Cloud Object Stores [3] Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics [4] https://mp.weixin.qq.com/s/PgpTUs_B2Kg3T5klHEpFVw [5] https://www.sohu.com/a/431026284_315839 [6] https://www.datagrom.com/data-science-machine-learning-ai-blog/snowflake-vs-databricks