Airbnb’s Big Data Platform Architecture: Design, Evolution, and Lessons Learned

Airbnb was founded in August 2008, offering world‑class customer service and a growing user community. As its business grew more complex, the volume of data in its big‑data platform exploded.

This article, authored by Airbnb engineer James Mayfield, analyzes Airbnb’s big‑data platform architecture, providing detailed insights and implementation details.

Part 1: Philosophy Behind the Big‑Data Architecture

Airbnb promotes a data‑driven culture, letting data speak for itself. Metrics are collected, hypotheses are validated through experiments, machine‑learning models are built, and business opportunities are mined, enabling rapid and flexible growth.

After many iterations, the big‑data stack has become stable, reliable, and scalable. This article shares Airbnb’s architectural experience with the community, and future posts will cover the distributed components in depth. James Mayfield notes, “We use many excellent open‑source projects every day; after benefiting from them we give back to the community.”

Based on the experience of building Airbnb’s platform, the following viewpoints are offered:

Engage with the open‑source community: adopt proven projects and contribute useful ones back.

Prefer standard components and methods over reinventing the wheel; consider maintenance and hidden costs of custom solutions.

Ensure platform scalability: data growth is now explosive, not merely linear.

Listen to colleague feedback to guide architectural road‑maps.

Reserve excess resources: over‑provisioning fosters a culture of exploration and prepares for future data scale.

Part 2: Big‑Data Architecture Overview

Below is an overview diagram of the platform.

Data sources come mainly from two channels: event logs sent to Kafka via instrumentation, and MySQL dumps stored in AWS RDS that are transferred to the Hive “gold” cluster using Sqoop (Airbnb actually runs two Hive clusters, “gold” and “silver”, whose separation is explained later).

Behavioral data and dimensional snapshots are ingested into the gold Hive cluster, where they are cleaned, enriched with business logic, aggregated, and validated.

The separation of gold and silver clusters isolates storage from compute, enabling disaster‑recovery. The gold cluster runs critical jobs with invisible resource impact, while the silver cluster serves as a product environment.

Raw data resides in the gold cluster and is replicated to the silver cluster; data generated on silver is not copied back. Silver acts as a superset and must receive low‑latency updates from gold. Airbnb built a custom synchronization tool because no open‑source solution met their needs.

Optimizations have been made in HDFS storage and Hive table management, emphasizing immutability (partitioned tables). Airbnb avoids proliferating disparate data systems to reduce ETL burden and inconsistency. Presto is used to query Hive tables, replacing legacy warehouses such as Oracle, Teradata, Vertica, and Redshift, and future plans include direct Presto‑Tableau connections.

Notable open‑source tools include Airpal, a Presto‑based web UI for ad‑hoc SQL queries used by over one‑third of Airbnb staff, and Airflow, a scheduler that can run Hive, Presto, Spark, MySQL jobs and provide monitoring. Spark is favored for machine‑learning and stream processing. S3 is used as an independent storage tier; moving portions of data from HDFS to S3 reduces costs and updates Hive tables to point to S3 files for easier access and metadata management.

Part 3: Hadoop Cluster Evolution

Airbnb migrated its clusters to the gold and silver architecture. Two years ago the company moved from Amazon EMR to EC2‑based HDFS, initially handling 300 TB. Today it operates two independent HDFS clusters storing 11 PB, with several PB on S3.

Key challenges and solutions:

A) Running Hadoop on Mesos Early experiments showed that Mesos could host Hadoop, Hive, Presto, Chronos, and Marathon on c3.8xlarge instances with 3 TB on EBS. Problems included invisible job logs, lack of health visibility, Mesos only supporting MR1, task‑tracker performance issues, high system load, and incompatibility with Kerberos. Solution: Adopt proven external solutions rather than building custom ones.

B) Remote read/write latency All HDFS data resided on EBS volumes accessed over the network, contrary to Hadoop’s design favoring local disk I/O. Replicated blocks across different racks caused remote reads/writes and occasional data loss during moves. Solution: Use instances with local storage and run on single nodes.

C) Mixed workloads on homogeneous machines Storage‑intensive workloads (Hive/Hadoop/HDFS) and compute‑intensive workloads (Presto/Spark) were sharing the same c3.8xlarge instances, making 3 TB of EBS storage expensive. Solution: After moving to Mesos, assign different instance types to different clusters (e.g., Spark on c3.8xlarge, later on d2.8xlarge with 4 TB local storage), saving over $100 M in three years.

D) HDFS federation challenges Early “Pinky” and “Brain” clusters shared storage but ran independent map‑reduce jobs, requiring queries on both clusters and suffering instability. Solution: Migrate data to separate HDFS nodes for machine‑level isolation, simplifying disaster recovery.

E) Heavy system monitoring burden Custom monitoring and alerting had to be built for Hadoop, Hive, and HDFS, which are complex and error‑prone. Solution: Partner with Cloudera for expert support and use its Manager tool to reduce monitoring overhead.

Final Statement

After evaluating legacy issues and inefficiencies, Airbnb performed a seamless migration of petabyte‑scale data and thousands of jobs—a long‑term effort. The author plans additional articles and will open‑source the tools developed for the community.

Evolution of the platform has yielded substantial cost savings and performance gains, as shown by the following statistics:

Disk read/write speeds increased from 70‑150 MB/s to over 400 MB/s.

Hive job CPU time doubled.

Read throughput tripled.

Write throughput doubled.

Overall costs reduced by 70%.