Alluxio: Introduction, Architecture, and Practical Experience for Big Data Construction

Alluxio is an open‑source data orchestration component that sits between storage and compute layers, providing a unified API to access heterogeneous storage systems, improving data locality, scalability, and enabling cloud‑native deployments.

Modern big‑data environments involve diverse compute engines (e.g., Presto, Spark) and storage systems (e.g., HDFS, Iceberg), leading to tight coupling, difficult upgrades, migration challenges, and continuous iteration pressure.

Alluxio addresses these issues by offering a standard API, decoupling compute from storage, multi‑level caching, cloud‑native support, short‑circuit NIO for same‑node access, and elastic worker registration with heartbeat mechanisms.

The architecture consists of an Alluxio Master (HA via ZooKeeper or embedded Raft) that manages configuration and worker coordination, and Alluxio Workers that handle cross‑storage reads/writes, maintain multi‑tier caches, perform access short‑circuit, and register with the Master.

Key features include a unified namespace with flexible mounting (root and nested mounts), fine‑grained access control (read‑only mounts), transparent access to underlying storage, configurable caching policies (read/write strategies, eviction, TTL, metadata synchronization), and support for POSIX APIs.

Practical experiences show Alluxio simplifying lake‑house integration by synchronizing metadata between old and new clusters, accelerating hot data access through cache‑rich clusters, enabling mixed deployments (e.g., Trino + Alluxio on SSD), supporting Spark RSS acceleration, and facilitating hot‑cold data separation via two architectural approaches.

Future directions focus on lighter, more flexible deployment, richer cache request strategies, cross‑cluster I/O throttling, client‑side UFS reads, tag‑based cache quotas, unified JVM/RAM cache management, and fault‑tolerant local cache loading to make Alluxio even more stable and performant.