Database Migration and Architecture Upgrade at Qunar: From MMM to PXC
This article analyzes common database migration strategies, compares two approaches—master‑slave cut‑over and proxy‑based migration—describes Qunar's internal MMM and Percona XtraDB Cluster (PXC) architectures, and details a step‑by‑step, zero‑downtime migration process from MMM to PXC with practical lessons and best practices.
Database migration and splitting are inevitable parts of daily operations; this article examines typical migration solutions, their scenarios, advantages, and disadvantages, and introduces the Quanr database architecture and its evolution.
Migration Schemes
Designing a migration plan depends on the existing data architecture and business requirements, aiming for minimal downtime and data consistency.
Scheme 1: Master‑Slave Cut‑over
Establish master‑slave replication, synchronize data, stop writes during low‑traffic periods, switch applications to the new cluster, and verify migration results. Advantages: simple architecture, DBA‑friendly. Disadvantages: requires service interruption, potential traffic cut‑over issues.
Scheme 2: Proxy‑Based Migration
Use a proxy layer to control traffic flow. Steps: (1) set up master‑slave replication; (2) add a proxy in front of the original cluster and have applications connect to the proxy; (3) after traffic is routed through the proxy, cut the replication and switch traffic to the target database. Advantages: no service downtime, decouples DBA and development work, allows rolling releases. Disadvantages: more complex architecture, added operational overhead.
Summary of Schemes
Scheme 1 suits scenarios where brief downtime is acceptable; Scheme 2 is better for zero‑downtime migrations with rolling releases.
Qunar Cluster Architecture
3.1 MMM Cluster
MMM (Master‑Master replication manager) is an older MySQL HA solution using monitor and agent components, providing a primary‑secondary setup with virtual IP failover. It offers simple failover but only asynchronous replication, limiting data consistency.
3.2 PXC Cluster
Percona XtraDB Cluster (PXC) provides multi‑master synchronous replication with strong consistency and decentralized architecture. It supports read‑write on any node but has constraints such as InnoDB‑only storage and transaction size limits.
Qunar's PXC deployment typically uses three nodes, with a sentinel cluster monitoring health, a config server storing namespace and node information, and Zookeeper for versioning and client notifications. Clients access the database via namespace, abstracting the underlying topology.
Qunar Cluster Migration
4.1 Migrating MMM to PXC
Due to legacy MMM clusters, Qunar decided to upgrade them to PXC without downtime. The process includes registering MMM VIP as a PXC namespace, notifying services to use the namespace, monitoring traffic via Zookeeper, and sequentially upgrading MMM nodes to PXC while maintaining high availability through automated scripts.
Key steps: (1) replace MMM master with PXC node; (2) promote the second MMM master as donor for the next PXC node; (3) retire remaining MMM slaves and integrate them into the PXC cluster; (4) update Zookeeper version and client connections.
4.2 Migration and Splitting Case
Applying the same in‑place upgrade approach, a heavily loaded MMM cluster can be split by promoting one master to a PXC donor, registering its IP, and gradually migrating a specific business service to the new PXC namespace while keeping the original cluster operational.
Conclusion
Different database architectures and business scenarios require tailored migration methods. Proxy‑based, zero‑downtime migrations decouple DBA and development work, improve efficiency, and can be applied to various cases without service interruption.
Qunar Tech Salon
Qunar Tech Salon is a learning and exchange platform for Qunar engineers and industry peers. We share cutting-edge technology trends and topics, providing a free platform for mid-to-senior technical professionals to exchange and learn.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.