Apache Hudi 1.0: Design Reconsiderations and Key New Features

Apache Hudi is an efficient transactional data‑lake platform that offers table‑format definitions, comprehensive transaction support, indexing, and change‑data‑capture, integrating with engines like Spark, Flink, and query layers such as Presto and Trino.

The 1.0 release revisits the original design, introducing five development directions: deeper query‑engine integration, a generalized relational data model, hybrid server‑less architectures, support for unstructured data, and enhanced self‑management of tables.

Key new features include:

LSM‑tree based timeline storage, replacing linear logs with a hierarchical structure to improve write and read efficiency.

Function indexes that enable data skipping by indexing transformed fields (e.g., hour‑level timestamps) without fine‑grained physical partitioning.

File‑group readers and writers that support merge‑on‑read tables, partial updates, and position‑based merges, reducing storage overhead and improving query performance.

Partial update support that records only changed fields, cutting write size dramatically and accelerating merges.

Non‑blocking concurrency control using MVCC and global monotonic timestamps to allow concurrent writes and GDPR‑style deletions without frequent conflicts.

Performance tests show significant gains: LSM‑tree timeline loading for 1 M commits in 367 ms, partial updates reducing write size by 70× and improving update latency by 1.4×, and overall merge speedups of 12‑20% on large tables.

Additional resources such as the 1.0 technical spec, documentation, blogs, Slack community, and GitHub repository are provided for deeper exploration.