XDB: A MySQL High‑Availability Solution with Horus, Isis, and Atum

MySQL Community Edition provides excellent performance and compatibility but lacks a built‑in high‑availability (HA) solution, which is only offered in the commercial edition; XDB is introduced as an open‑source HA framework to fill this gap.

The article first reviews MySQL master‑slave replication, emphasizing that the master handles writes while slaves handle reads, and notes that a single‑point master failure would block all write operations, necessitating an HA strategy that can promote a slave to master.

XDB consists of seven components: Horus (monitoring instance health and orchestrating master promotion), Isis (tracking topology changes and updating DBProxy routing), Atum (CLI tool for manual failover and topology inspection), Zookeeper (distributed coordination), MySQL , and DBProxy (SQL routing, read/write splitting, IP‑based access control, load balancing, and SQL injection protection).

The core component Horus operates similarly to MHA: it compares master‑slave binlogs, fills missing data, and elects a new master. Horus instances run on each MySQL host, register their metadata (port, role, hostname) in Zookeeper, and expose the replication topology as JSON.

Horus uses a state machine with seven states (S0‑START_STATUS, S1‑TSC_WITH_MASTER_STATUS, S2‑TSC_WITHOUT_MASTER_STATUS, S3‑REGISTER_VAR_NODE_STATUS, S4‑STEADY_STATUS, S5‑EXCHANGE_STATUS, S6‑LEAD_EXCHANGE_STATUS) to manage transitions during normal operation, failover, and node addition.

Four typical scenarios are described: (1) Horus starts after the replication cluster is already deployed; (2) master crash triggers automatic promotion of a slave; (3) a new slave joins during an ongoing failover; (4) a new slave joins when the cluster is stable.

Monitoring features include disk health checks (performed in a separate thread to avoid blocking) and replication delay monitoring, where Horus writes a heartbeat timestamp every three seconds and computes delay with the formula delay_time_correct = current_timestamp - last_update_timestamp_in_heartbeat .

For log recovery, Horus parses MySQL binlog and relay‑log files after a master crash to compare offsets and synchronize data, a process similar to MHA’s log comparison workflow.

Horus stores its configuration and runtime data in Zookeeper nodes such as /root/product/cluster/menu/default_config , /root/product/cluster/tsc/default_node_config , and /root/product/cluster/var/tsc/tsc_0 , which hold static topology, dynamic state, and lock information.

Automatic log management rotates and deletes logs based on size, retention time, and disk usage thresholds; deletion is performed via hard‑linking and incremental inode cleanup to minimize I/O impact on live services.

In Baidu Waimai, Horus has been tested for years and was deployed on the orderlist cluster during a large‑scale traffic replay; due to lack of dedicated Horus operators, automatic failover was disabled and DBA‑driven manual switchovers were used.

Deployment requires manual installation of Horus on each MySQL node, Isis on DBProxy nodes, and integration with an existing Zookeeper ensemble; no automated deployment tooling is currently provided.

Operational experience shows that Horus logs reveal node identities (e.g., node_29156251931628 ) and replication status, and the Atum tool can visualize the cluster topology.

In conclusion, studying Horus source code reveals its state‑machine design, process model, key structures, and smooth master‑slave transition mechanisms, offering valuable insights for building other HA systems despite some imperfections.