Practical Experience of Deploying and Optimizing Apache Pulsar on Kubernetes at 360

1. Architecture Design

1.1 Cluster Deployment

Our Pulsar service platform runs robustly on a Kubernetes (K8s) cluster, aiming to maximize resource utilization, improve message service flexibility and operational efficiency, and enable rapid user onboarding. To fit the internal service ecosystem, we made several localized improvements when deploying the Pulsar cluster:

Log Management Integration: Implemented automated log collection that seamlessly feeds into the company's big‑data analysis platform, providing data for log analysis, business insight, and intelligent alerts.

Unified Monitoring and Alerting: Automatically integrated Pulsar into the corporate monitoring and alerting system, visualizing cluster status in real time to ensure service stability and responsiveness.

Storage Adaptation and Optimization: Chose open‑local as the persistent volume (PVC) plugin to align with company storage policies, enhancing flexibility and performance while ensuring compliance.

Permission Management Integration: Unified cluster permission management, simplifying access control and improving security and user experience.

Load‑Balancing Integration: Tight integration with the corporate load‑balancer platform provides stable, low‑latency message services across the entire company.

In the future we plan to explore the Pulsar Operator to further automate cluster management and fine‑tune control for more complex workloads.

1.2 Disk Selection

Benchmark results show that under a 1 GB/s throughput, NVMe disks keep end‑to‑end latency below 8 ms, while ordinary SSDs stay around 40 ms. Considering cost‑effectiveness and most business scenarios, we adopted local SSDs as the primary storage, offering NVMe disks for latency‑critical services.

To balance cost and performance we are testing a heterogeneous storage architecture that combines HDDs with NVMe disks. Ledger data, which is asynchronously persisted, will be stored on high‑capacity HDDs to reduce cost without sacrificing write efficiency, while Journal logs and RocksDB indexes will reside on fast NVMe drives for low‑latency access.

1.3 Cluster Modes

To meet varying stability and flexibility requirements, we provide single‑node, cross‑AZ multi‑active, and cross‑region multi‑active Pulsar deployment options. In multi‑active mode, clusters are connected in a full‑mesh topology, ensuring real‑time synchronization of messages and consumption progress. If one cluster fails, traffic seamlessly switches to a standby cluster, delivering near‑zero service interruption.

Testing revealed a risk of duplicate consumption when producing and consuming across two clusters simultaneously. We recommend a "single‑cluster produce‑consume" strategy: at any time, use only one cluster (or its mirror) for both production and consumption to maintain data consistency.

2. Pulsar Service Discovery

Pulsar brokers run in a K8s cluster, while client applications may run on bare metal, VMs, or other K8s clusters. To expose brokers externally, we capture each broker pod’s IP via an environment variable and set advertisedAddress in the broker configuration. The pod IP is registered in Zookeeper at startup, enabling automatic address updates.

Because pod IPs are unstable, we place an internal LoadBalancer in front of all brokers, exposing a stable address. All brokers are accessed through a single domain name, reducing impact of data‑center migrations and improving flexibility and stability.

In practice, users resolve the domain name to obtain the target topic’s broker addresses, then connect directly to the broker pod IPs for publishing and consuming.

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: status.podIP

3. Performance Optimization

3.1 Challenges

Using the open‑source benchmark tool OpenMessagingBenchmark (OMB), we identified several issues:

High journal creation latency (P999 latency reaches seconds).

High production latency (P999 latency reaches seconds).

High consumption latency (P999 latency reaches seconds).

Uneven traffic between Bookie and Broker nodes.

3.2 Solutions

3.2.1 Benchmark Tool Optimization

We customized OMB to achieve reliable results:

Synchronize Pulsar client version with the cluster version.

Add TTL and retention policies to automatically clean up benchmark data.

Expose persistence parameters (EnsembleSize, WriteQuorum, AckQuorum) for consistency with production.

Adjust batchingMaxPublishDelayMs to reduce publish latency.

Fine‑tune parameters such as ioThreads , connectionsPerBroker , and batchingMaxBytes for optimal performance.

Enable topic‑specific testing to simulate catch‑up reads.

Fix OMB’s post‑test process hang and memory leak issues.

3.2.2 Pulsar Optimizations

Key actions include:

Build comprehensive monitoring dashboards in Grafana and fill missing metrics.

Deeply understand broker and Bookie data flow to locate bottlenecks.

Separate Zookeeper, ledger, and journal directories onto different SSD/NVMe disks; enable journal sync writes.

Limit the number of journal directories per disk to preserve sequential writes.

3.2.2.1 Journal Write Optimization

Adjustments:

Verify namespace persistence parameters (EnsembleSize, WriteQuorum, AckQuorum).

Monitor journal creation, addition, and sync latencies; increase journalPageCacheFlushIntervalMSec if needed.

Increase numJournalCallbackThreads under high load.

Check disk I/O utilization and hardware health.

3.2.2.2 Ledger Write Optimization

Actions:

Monitor ledger disk I/O utilization.

Adjust dbStorage_writeCacheMaxSizeMb to control flush frequency.

Tune RocksDB parameters ( writeBufferSizeMB , sstSizeInMB , numLevels , maxSizeInLevel1MB ) to reduce write amplification.

3.2.2.3 Read Optimization

Increase write cache size to improve cache hits, configure dbStorage_readAheadCacheBatchSize for pre‑loading, and adjust RocksDB block size to enlarge index cache.

3.2.3 Optimization Results

After iterative tuning, we achieved:

Peak throughput up to 2 GB/s even with large messages (5 MB) and >10 k partitions.

P99 publish latency ≤ 5 ms under 1 GB/s sustained load.

Average end‑to‑end latency ≤ 3.2 ms, P99 ≤ 8 ms.

Stable 1 GB/s catch‑up read throughput during 1.8 GB/s tailing‑read scenarios.

4. Monitoring and Alerting

Pulsar’s built‑in monitoring covers JVM, broker, Bookie, Zookeeper, topic, and core K8s components. When Grafana consumes metrics from a K8s Prometheus exporter, chart adjustments may be required due to format differences.

We built a multi‑layer alerting system covering hardware resources, K8s cluster and node health, pod status, message E2E latency (using a mock client), and Pulsar‑specific metrics such as Bookie read/write status.

Log data is also shipped to the big‑data analysis platform, enabling deeper issue detection and reducing operational workload.

5. Future Plans

Continuing to evolve the Pulsar platform, we plan to:

Integrate with S3 for long‑term, low‑cost storage.

Refine heterogeneous storage strategies combining SATA HDDs and NVMe SSDs.

Improve load‑balancing algorithms for more efficient resource distribution.

Develop automatic elastic scaling to handle traffic spikes.

Mature the Pulsar Operator for simplified K8s management.

Deepen integration with internal big‑data, log management, and function‑compute platforms.

These efforts aim to broaden Pulsar’s applicability across diverse technical ecosystems.