Evolution of MatrixOne: From NewSQL/MPP Roots to a Modern HTAP Architecture

The article presents MatrixOne, an open‑source distributed NewSQL database, and walks through its architectural evolution from an early NewSQL/MPP design to a modern HTAP‑focused system.

Early Architecture and Problems

The initial architecture combined storage and compute in each server, using a proxy for load balancing and three storage engines (AOE, TPE, TAE). While this design delivered good performance, it suffered from scalability, performance, and cost issues:

Scalability: storage and compute had to be expanded together, and adding a node required full three‑replica data synchronization, leading to long warm‑up times.

Performance: the Raft leader became a hotspot; heterogeneous engine performance caused bottlenecks; slower storage (HDD) limited overall throughput.

Cost: linear cost growth with node size, and multi‑replica storage dramatically increased expenses, especially in public‑cloud deployments.

Path to Architecture Upgrade

Starting in March 2022, the team iterated from version 0.1 to 0.5, eventually recognizing that the monolithic design could no longer meet requirements. Three major pain points were identified: the tightly coupled distributed framework, the proliferation of storage engines, and the lack of compute‑storage separation.

New Three‑Layer Architecture

The redesign decouples the system into compute, transaction, and storage layers:

SQL Front‑End : MySQL‑compatible protocol for client connectivity.

Compute Layer : Includes a SQL parser, a new DAG‑based MPP executor, and a custom pipeline that supports sub‑queries, window functions, CTEs, and memory‑overflow handling.

Transaction Layer : Implements a dedicated DN (transaction node) service, Logtail for write‑ahead logging, and a distributed transaction protocol that offloads heavy writes to S3 object storage.

Storage Layer : Consolidates the three previous engines into a single column‑store engine (TAE) that provides full ACID guarantees, OLAP capabilities, and optional column‑family optimizations for TP workloads.

MatrixCube : An open‑source distributed storage framework providing Raft‑based replication, load balancing, and a scheduler named Prophet.

File Service : Unified interface to read/write data from S3 (or MinIO/OSS in private deployments), allowing stateless compute nodes.

Challenges and Solutions During Upgrade

Key difficulties included finding a low‑redundancy, low‑cost storage (solved by adopting S3 with cold‑hot separation), avoiding transaction‑node bottlenecks (addressed by introducing Logtail and batching writes), and isolating different workload types. The solution uses tagged compute nodes (TP vs. AP) and dynamic resource allocation based on workload tags.

Additional gains include:

Re‑architecting the SQL execution path for deeper understanding of plans and standards.

Moving from multi‑engine to a single HTAP engine, reducing code duplication and maintenance effort.

Replacing factor‑based algorithms with a DAG executor.

Cutting storage cost to roughly one‑third by leveraging object storage and Logtail.

Achieving fine‑grained resource isolation through compute‑storage separation and workload tagging.

Summary

The redesign separates compute, transaction, and storage, enables HTAP capabilities with a single columnar engine, adopts a DAG‑based executor, and dramatically lowers storage cost while improving scalability and performance. The project is currently in beta (v0.8) with a public cloud offering and a beta‑program for early adopters.

Q&A Highlights

• MatrixOne supports MinIO for private deployments and S3/OSS for public clouds. • Enterprise customers receive dedicated operation tools and can request custom solutions. • Logtail acts as an intermediate cache to accelerate writes and enable shared access across compute nodes. • Hot‑cold data separation is achieved via S3’s tiering and per‑node caches.