Evolution of MatrixOne: From NewSQL MPP to a Modern HTAP Architecture

Introduction – MatrixOne is an open‑source distributed database that has evolved from a NewSQL/MPP architecture to a modern HTAP design.

Early Architecture and Challenges – The initial design combined storage and compute in each node, using a proxy for load balancing and three storage engines (AOE, TPE, TAE). This caused scalability issues (tightly coupled storage and compute), performance bottlenecks (Raft leader hotspot, uneven engine performance), and high cost due to triple‑replication.

Upgrade Path – Starting in March 2022, the team re‑architected MatrixOne by separating the system into three layers: a compute layer (SQL front‑end, parser, MPP executor), a transaction layer (log service, DN nodes), and a storage layer (TAE columnar engine). The distributed framework MatrixCube was retained but refactored, and a new transaction layer with Logtail was introduced.

Key Design Decisions – Adopted a single HTAP engine (TAE) to replace multiple engines, reducing code duplication and cost. Moved hot data to node‑local caches and cold data to object storage (AWS S3), achieving a 2/3 cost reduction. Implemented a flexible resource isolation strategy using tags and separate CN (compute) groups for TP and AP workloads. Introduced Logtail to buffer writes, improve durability, and offload bulk writes directly to S3.

Current Architecture Overview – The system now consists of: File Service: a unified interface to read/write data from S3. Transaction Layer: DN nodes handling metadata, conflict detection, and Logtail. Compute Layer: Stateless CN nodes with caches and a pipeline optimizer (DAG‑based executor). Storage Layer: TAE columnar store providing ACID guarantees and HTAP capabilities.

Benefits and Learnings – Decoupling layers enables independent scaling, reduces hotspots, lowers storage cost, simplifies code maintenance, and improves overall performance. The team also gained deeper insights into SQL execution, transaction handling, and distributed storage integration.

Q&A Highlights – Answers covered future storage engine support (MinIO, OSS), enterprise‑level customization, reasons for separating Logtail, and how hot/cold data and read/write separation are achieved.

Conclusion – MatrixOne’s redesign delivers a flexible, cost‑effective HTAP solution with clear separation of compute, transaction, and storage, supporting both TP and AP workloads while maintaining ACID properties.