Overview of Alipay System Architecture and the Open‑Source Distributed Messaging Middleware Metamorphosis (MetaQ)

Alipay System Architecture Overview

Typical Processing Defaults

Capital Handling Platform

Financial Accounting

Payment Settlement

Accounting Center

Transaction Processing

Flexible Transactions

Alipay’s Open‑Source Distributed Messaging Middleware – Metamorphosis (MetaQ)

Metamorphosis (MetaQ) is a high‑performance, highly available, and scalable distributed messaging middleware written in Java, similar to LinkedIn’s Kafka. It supports sequential writes, large throughput, local and XA transactions, and is suitable for high‑throughput, ordered, broadcast, and log‑data transmission scenarios; it is widely used in both Taobao and Alipay and has been open‑sourced.

The author rewrote MetaQ because Kafka is implemented in Scala, a language they were not familiar with, and because Kafka lacks several required features such as transaction support, multiple offset storage mechanisms, and robust high‑availability solutions.

Text‑based protocol design that is transparent and allows monitoring via a memcached‑style stats protocol.

Pure Java implementation covering communication, storage, client, and server.

Provides transaction support, including local and XA distributed transactions.

HA replication with both asynchronous and synchronous modes to ensure message reliability.

Supports asynchronous message sending.

Local recovery for failed message consumption.

Multiple offset storage options (database, disk, Zookeeper) with customizable implementations and group commit for higher reliability and throughput.

Message broadcast mode.

Accompanying projects such as a Python client, a Twitter‑Storm spout, and tail4j.

MetaQ processes roughly 12 billion messages per day for Alipay and hundreds of millions per day for Taobao.

Suitable Applications for MetaQ

High‑throughput log transport.

Message broadcasting, e.g., cache‑invalidation notifications.

Ordered data synchronization such as MySQL binlog replication.

Message routing in distributed environments (clustered brokers, producers, and consumers) where order and reliability are critical.

General purpose MQ use cases.

For more Java‑related technical sharing, follow the author’s Java community (scan the QR code for additional resources).

PS: If you found this sharing useful, feel free to like and repost.