Inside Twitter’s Manhattan: How a Massive Distributed Database Powers Real‑Time Ads

Various Types of Databases

Twitter’s overall storage architecture consists of four systems: NoSQL for user information and small numbers, a large file system for images and videos, Hadoop for backend data processing, and MySQL for relational queries.

CAP Theorem of Different Databases

CAP theorem states that a distributed system can satisfy at most two of consistency, availability, and partition tolerance. Different databases make trade‑offs accordingly.

Different databases make trade‑offs:

Relational databases prioritize strong consistency, sacrificing availability.

NoSQL offers high concurrency with eventual consistency.

NewSQL such as Google Spanner attempts to satisfy all three properties by reducing partition likelihood, though it cannot truly break CAP.

Specialized databases include time‑series stores (OpenTSDB), document stores (MongoDB), and graph databases (Neo4j).

Manhattan’s Evolution

Twitter originally used Cassandra, which suffered from scaling limits and gossip‑protocol inconsistencies at large node counts. To overcome these issues, Twitter built its own NoSQL system, Manhattan, focusing on reliability, availability, operability, low latency, and efficiency.

Manhattan now runs on over 30,000 nodes, each handling tens of thousands of QPS with a 99th‑percentile latency of 10 ms.

Manhattan Storage Engine Design

Manhattan uses a modular design with interchangeable storage nodes and engines, tailored to different workloads.

Common Data Engines

SeaDB – read‑optimized for batch updates and read‑heavy workloads.

SSTable – write‑optimized, similar to Google’s Bigtable.

RMDBS – traditional relational engine supporting indexes and real‑time updates.

Special‑Purpose Engines

Strong consistency.

Time‑series services.

Secondary indexes.

Manhattan Architecture

Coordinator module packages requests and distributes them across backend nodes, reassembling responses.

Bloom filters quickly eliminate non‑existent keys, improving query efficiency.

Read path for SSTable‑based storage is expensive; optimizations include compaction, two‑level indexing, and Bloom filters.

Writes go directly to memory via a fast commit log.

Reconciliation copies written data across replicas and periodically resolves inconsistencies to achieve final consistency.

Large‑Scale Operations Practices

Twitter handles tens of millions of requests per second across dozens of clusters, making operations challenging.

Introducing DevOps improves efficiency through organizational structure, processes, and tools.

Organization

Integrating development and operations ensures shared responsibility; operations must sign off designs, and developers run their own services for a month after launch.

Process

Newcomers receive training, shadow experienced engineers, then handle incidents independently.

Tools

Self‑service UI reduces communication overhead and provides debugging interfaces.

Deployment pipelines support canary testing, rollback, and staged rollouts to handle data format changes safely.

Topology transition mechanisms enable incremental scaling without full‑stop migrations.

Challenges and Opportunities in Operations

Manual fault diagnosis is painful given billions of metrics; automated anomaly detection and log aggregation help pinpoint issues.

Intelligent operations use machine learning to correlate front‑end and back‑end failures, generate flexible alerts, and build a knowledge base that continuously learns from incidents.