Cloud-Native Architecture and Tiered Storage for Xiaohongshu Kafka: Cost Reduction, Elastic Migration, and Performance Optimization

Facing rapid growth of Kafka clusters, Xiaohongshu's big‑data storage team adopted a cloud‑native architecture that introduces hot‑cold tiered storage, containerization, and a self‑developed load‑balancing service called "Balance Control". This practice achieved significant storage cost reduction, minute‑level elastic migration, high‑performance data access, and automated resource scheduling.

The Kafka cluster’s scale has reached TB‑level peak throughput, and storage costs remain high due to multi‑replica and cloud‑disk/SSD usage. Moreover, the tight coupling between compute and storage nodes hampers scaling efficiency.

Key problems identified:

Cost : expensive storage and wasted compute resources.

Efficiency : long‑running cluster expansion/reduction and weak scheduling capability.

Stability : poor emergency response and degraded performance during catch‑up reads.

To address the “compute‑storage integration” limitation, a cloud‑native elastic architecture was introduced, decoupling storage from compute. Three solution paths were compared:

Self‑developed compute‑storage separation (on ObjectStore) : Uses object storage with erasure coding for cost‑effective, elastic storage, but requires an additional acceleration layer due to higher latency.

Hot‑cold tiered storage (on ObjectStore) : Keeps a small amount of hot data locally and offloads the majority of cold data to cheap object storage. This reduces storage cost without changing core write or HA mechanisms, though some coupling remains.

Apache Pulsar : A cloud‑native messaging engine with native compute‑storage separation, but its underlying BookKeeper still relies on disk storage, limiting cost benefits, and migration effort is high.

The tiered storage approach was selected for its low cost, moderate elasticity, and short implementation cycle. After six months of rollout, 80% of clusters were upgraded, delivering up to 60% storage cost savings and extending topic retention from 1 day to 7+ days.

The overall cloud‑native architecture consists of four layers:

Access layer : Custom SDK providing authentication, rate‑limiting, and service discovery.

Compute layer : Kafka brokers with tiered storage, achieving “weak state” for local data.

Scheduling layer : Load‑balancing scheduler distributes traffic north‑bound and integrates with containers for automatic elastic scaling.

Infrastructure layer : Container and object‑storage foundation offering PaaS‑style management and unlimited low‑cost storage.

Tiered storage stores hot data on high‑performance cloud disks and cold data on inexpensive, highly reliable object storage, dramatically lowering storage costs while maintaining high availability.

Performance optimizations include:

Batch reads : 8 MB‑aligned segment reads to reduce I/O calls.

Cache pre‑loading : Concurrent pre‑read based on locality and sequential read patterns, achieving 99% cache hit rate and a 30% improvement in cold‑data read performance.

Containerization further improves elasticity. By deploying Kafka on Kubernetes with StatefulSets, persistent volumes, ConfigMaps, and Supervisord‑based process monitoring, the system gains:

Automatic restart of failed Kafka processes.

Hot‑fix capability inside containers.

Deletion protection via webhook when unhealthy pods exceed thresholds.

Mixed‑workload scheduling ("混部") leverages idle CPU in off‑peak periods, raising overall CPU utilization from ~10% to >40% after cloud migration.

Cluster Autoscaler (CA) automates node scaling, reducing manual intervention and enabling rapid response to traffic spikes.

For load balancing, a self‑developed service "Balance Control" uses multi‑objective optimization to consider partition replicas, network bandwidth, disk, and compute resources. It performs:

Data collection via Kafka Metrics Reporter.

Metric aggregation into a ClusterModel snapshot.

Scheduling decisions that relocate or swap partitions to rebalance traffic.

Balance Control also supports auto‑scaling (AutoScala): monitoring cluster load and automatically expanding or shrinking the cluster within minutes, achieving "elastic scaling, pay‑as‑you‑go".

Overall results show a 60% reduction in storage cost, a ten‑fold improvement in scaling efficiency, and a fully commercialized elastic scaling model. Future work includes further research on compute‑storage separation, multi‑active disaster recovery, and continued enhancements to the cloud‑native messaging engine.