Design and Implementation of 58 Cloud Search Platform Using Kubernetes and Containerization

Background: 58.com’s original search system, initially based on Solr and later on a custom eSearch engine, could not keep up with growing traffic and required lengthy manual setup for each new vertical search service.

To address these limitations, the 58 Cloud Search platform (WCS) was created, leveraging Kubernetes and Docker to provide a fully self‑service solution where developers only configure index structures and upload source data, after which a searchable service is provisioned within minutes.

Architecture: WCS runs all components as containers managed by Kubernetes, which acts as the control plane. A HAProxy + Keepalived layer with a virtual IP provides load‑balancing and high availability for the Kubernetes master nodes. Multi‑cluster deployment isolates API layers per business ID, enabling seamless rolling upgrades and migration between clusters.

Automatic Indexing and Fault Recovery: Each search pod consists of three containers – an init container that fetches and configures index data, a main search container that loads the index, and a data‑update container that consumes messages to build incremental indexes. Kubernetes liveness and readiness probes detect failed pods and automatically replace them, synchronizing missing index data from healthy peers.

Resource Optimization: To prevent offline indexing jobs from impacting online query performance, pods are labeled as “online” or “offline” and scheduled onto separate nodes using node affinity. A custom scheduler plugin scores nodes based on real CPU and memory usage, improving load distribution and reducing resource hotspots.

Results: The platform now supports more than 200 internal search services, handling around 70 billion queries per day with peak QPS of 150 k, while continuously improving availability and resource utilization through automated orchestration and custom scheduling.