Airbnb’s Database Scaling Using MySQL Replication and Vertical Sharding

Airbnb, a San Francisco‑based marketplace for lodging, experienced an 800% growth in 2011, leading to massive data volume increase and pressure on storage scalability and stability.

To address this, Airbnb launched a project that vertically partitions high‑traffic tables into separate databases using application‑level functions, requiring engineering effort for application changes, data migration, and reliability testing while aiming for minimal downtime.

Background

The team preferred vertical sharding over horizontal sharding, assigning dedicated RDS instances to individual services. Analysis showed that the “message inbox” feature consumed about one‑third of writes on the primary database, making it a prime candidate for isolation.

Constraints included limited engineering resources and the need to keep downtime short; therefore MySQL’s built‑in replication (similar to Amazon RDS “Read Replica Promotion”) was chosen to simplify consistency handling.

Plan

The plan leveraged MySQL replication on Amazon RDS to create a new master (message‑master) and a replica (message‑replica). The upgrade required a brief write pause, after which the replica could be promoted to the new primary.

Replication filters were used to exclude unrelated tables, reducing upgrade time. The architecture diagram (image) shows the components involved.

Phase 1 – Pre‑planning

Identify tables to move, ensure cross‑team communication, and redirect traffic to the new database once ready, while keeping naming conventions consistent.

Phase 2 – Execution

Coordinate with customer‑service to limit inbox downtime to under ten minutes.

Deploy new database credentials for the inbox queries.

Switch write traffic to the new master, allowing reads to continue from the replica.

Cut connections from the old master, promoting the replica without a full deployment.

Validate replication catch‑up, then upgrade the new master (≈30 s downtime for reads, ~4 min for writes).

Enable Multi‑AZ on the upgraded master before the next automated backup window.

After stability is confirmed, drop obsolete tables from the old databases.

If any step fails, Zookeeper entries are rolled back, restoring the original inbox functionality, though lost writes would need manual recovery.

Results

Post‑migration metrics (image) show a significant drop in primary‑database writes, a 20 % reduction in database size, and a 33 % decrease in write queries, cutting inbox downtime to 7.5 minutes and improving overall stability.

Surprise: RDS Snapshots Increase Latency

During Multi‑AZ deployments, snapshots can add several minutes of latency under high load, potentially causing query back‑log and instability. The team learned that snapshot‑induced latency scales non‑linearly with load, making it a critical factor for future scaling plans.