PostgreSQL High Availability (PGHA) at Qunar: Architecture, Customization, Testing, and Metrics

Qunar's DBA team needed a robust PostgreSQL high‑availability (HA) solution to meet strict availability targets (multiple 9s) and reduce manual switchover/failover errors.

Background : Hardware upgrades, manual switchover/failover steps, and unpredictable failures required an automated HA layer.

Solution selection : After evaluating popular open‑source PGHA tools, Patroni was chosen for its active community, Python‑based automation, and strong feature set.

Patroni architecture :

Key advantages include synchronous replication support, minimal data‑loss switchover, high automation to avoid split‑brain, and pg_rewind for recovery.

Operational impact : Patroni takes full control of cluster configuration files, making the cluster more invasive and requiring HA‑driven operational procedures.

Customizations for Qunar :

DCS (Distributed Configuration Store) was set to ZooKeeper (zk) because Qunar's stack is Java + zk.

zk is deployed as a centralized cluster for easier maintenance.

To avoid unreliable watchdog‑based power‑off, a custom OS console power‑reset ("kill") mechanism was implemented for rapid master shutdown.

Virtual IP (VIP) strategy: a floating VIP is used instead of PgBouncer/Pgpool/Haproxy, simplifying the architecture.

Both master and slave VIPs were customized to support automatic failover and switchover without service interruption.

Testing : After each customization iteration, comprehensive regression tests are run to ensure all test cases meet expectations.

Metrics :

Target availability: 99.95% (≈262.8 minutes downtime per year) and higher (up to 5 9s) with the HA layer.

Master failover timeline includes TTL, safety margin, loop_wait, OPS reset, promote time, and VIP bind (≈10 seconds total).

Switchover for master or slave typically completes within 3 seconds; slave failover within 10 seconds.

Formulas for safe parameter settings:

ttl - safety_margin > loop_wait
ttl >= loop_wait * 2 + retry_timeout
# defaults: ttl=30s, safety_margin=5s, loop_wait=5s, retry_timeout=10s

Future directions :

Namespace‑based connection strings without fixed host/IP, enabling cross‑region deployments.

Language‑specific connection pool implementations (Python, Java, Go) that map logical namespaces to physical hosts.

Leveraging native JDBC failover for Qunar's Java‑centric stack.

Conclusion : The multi‑phase PGHA rollout at Qunar is stable, achieves >5 9s availability, and continues to evolve through community feedback and internal optimizations.