Introducing Apache Pulsar: Technical Benefits and Solutions for Didi Big Data Messaging System

Apache Pulsar is an Apache Software Foundation top‑level project and a next‑generation cloud‑native distributed messaging platform that integrates messaging, storage, and lightweight function‑as‑a‑service. It originated at Yahoo, where it handled over 1 billion messages per day for 1.4 million topics.

Didi Big Data began evaluating Pulsar in January 2021, created an internal Pulsar 2.7 branch, and launched the first Pulsar data‑channel synchronization cluster in August 2021. The cluster has been serving Log→ES, BamaiLog→ES, BamaiLog→ClickHouse, and Log→HDFS pipelines and has run stably for more than two years.

The legacy DKafka cluster suffered from severe operational pain points: uneven disk I/O, storage capacity imbalance, and CPU utilization spikes, especially during peak periods. Scaling quickly was difficult despite automation such as disk‑IO‑based partition adjustment, batch migration, and storage load‑balancing.

Pulsar offers elegant solutions to these problems.

1. Resolving SATA HDD I/O bottlenecks – Pulsar’s storage‑separation architecture stores messages in Bookie entry‑logs. Messages from different partitions are sorted in memory and flushed sequentially to a single entry‑log, eliminating random writes and allowing support for millions of partitions without performance degradation.

2. Overcoming cluster capacity limits – By adopting a heterogeneous storage model (large‑capacity SATA HDD for bulk data plus NVMe for journal and RocksDB index), Pulsar can lower costs while extending storage periods and increasing replica counts.

3. Eliminating cache/IO isolation issues – Pulsar implements a multi‑level cache (read‑only write cache, read‑write write cache, and read cache). Cache misses trigger storage reads, which are then prefetched and cached, dramatically reducing read‑write interference and page‑cache pollution.

4. Mitigating single‑disk/host hot‑spots – Pulsar’s storage layer distributes segment data across Bookies using an ensemble mechanism. This automatic dispersion keeps per‑disk utilization variance within ~10 % and removes the need for manual rebalancing.

5. Reducing metadata and rebalance pressure – Pulsar’s bundle mechanism groups millions of partitions into a limited number of bundles (hash‑ring based). Unloading a hot‑bundle automatically reassigns its partitions to a less‑loaded broker, simplifying load‑balancing compared with DKafka’s heavyweight leader‑rebalance.

6. Simplifying cluster scaling – For compute‑layer scaling, Pulsar unloads bundles from overloaded brokers, which are then automatically reassigned. For storage‑layer scaling, adding Bookies requires no manual steps; as new segments are written, Pulsar gradually shifts writes to low‑load Bookies, achieving painless expansion and shrinkage.

The document also presents a fault‑simulation Q&A, demonstrating how Pulsar’s stateless compute nodes, bundle migration, and multi‑level caching address DKafka’s failure scenarios such as leader overload, storage hot‑spots, and inability to expand during peak traffic.

In summary, Pulsar’s compute‑storage separation, cache/IO isolation, peer‑to‑peer node design, bundle and ensemble mechanisms substantially improve operational experience for large‑scale messaging systems. However, the richer architecture introduces more monitoring metrics and complexity, posing new challenges for developers and operators.