Beike Zhaofang's 秒X Real‑Time Analytics Platform: Architecture, Implementation, and Use Cases

Background: With the development of big data technology, real‑time, accurate and stable data processing has become a business requirement, leading Beike Zhaofang to build the “秒X” real‑time analytics platform for instant metric collection, monitoring and alerting.

Architecture: The platform uses Spark Streaming for low‑latency, high‑throughput stream processing, Druid.io for real‑time multidimensional OLAP analysis, and Elasticsearch for detailed data search, forming a pipeline that ingests data from Kafka, cleanses, stores, and serves it.

Fast configuration: Each Kafka topic is treated as a task; users configure parsing, field extraction, filtering, transformation, indexing and dimension/metric definitions through a UI, enabling rapid setup of new data sources.

Data processing flow: (1) Data collection via Kafka; (2) Real‑time cleaning and transformation with Spark Streaming; (3) Storage in Druid, HBase, Elasticsearch, Redis; (4) Applications such as dashboards, alerts, detailed queries and API services.

Monitoring and alerting: The platform aggregates metrics and performs fine‑grained checks, triggering SMS or email alerts when thresholds are breached, ensuring timely response to anomalies.

Visualization and retrieval: Users can build OLAP models, create charts, dashboards, and perform interactive Elasticsearch queries for detailed log inspection.

Use cases: Security department uses 秒X for domain‑level peak traffic, IP access monitoring and volume statistics; Kylin engine uses it to monitor query latency and IP query counts, with automatic alerts for queries exceeding 10 seconds.

Conclusion: 秒X has been stable in production, supporting multiple business units, and the team will continue to enhance real‑time capabilities, simplify configuration and improve data accuracy.

About the team: The Beike Zhaofang big‑data architecture team builds and maintains the company’s data storage, compute and streaming platforms, providing high‑performance OLAP engines and toolchains.