Design and Architecture of the Boussole Real-Time Multi-Dimensional Data Analysis Engine

Background: As Shopee’s business expands, the need for real‑time, multi‑dimensional data analysis grew. Traditional offline pipelines could not meet the requirements for instant operational insights, activity monitoring, and anomaly detection. The team therefore built Boussole, a real‑time analytics platform that supports massive multi‑dimensional queries with millisecond latency.

Overview: Boussole follows a typical real‑time analytics data flow – data is ingested, pre‑processed, and then aggregated in real time according to user‑defined metrics, dimensions, and aggregation methods. The results are stored persistently and visualized through configurable dashboards.

Storage Model: The underlying storage model converts each dimension into a key‑value (KV) pair, enabling a uniform representation regardless of the number of dimensions. HBase is used as the primary backend, with an abstraction layer that can also plug into Redis, Memcache, RocksDB, TiKV, etc. To reduce query I/O, the system performs pre‑aggregation and stores a mapping of which dimension values exist for each metric.

Pre‑Aggregation: Pre‑aggregation materializes results for selected dimension combinations so that queries can be answered without on‑the‑fly computation. This improves query speed at the cost of storage expansion, which can grow up to 2^N for N dimensions. Users must choose dimension sets wisely to balance storage and performance.

Metric and Tag Storage: Metrics and tags are hashed with FNV64a, timestamps are encoded as uint32, and values are stored as float64. Tags are stored as key‑only KV entries, with timestamps aligned to the hour to save space. The storage savings from hour‑alignment are roughly (3600-10)/3600 ≈ 99.72% .

Query Process: A user submits a PromQL (via MetricsQL) query. The engine parses and optimizes the query, determines which dimensions need to be fetched, and performs dimension filtering, aggregation, and metric calculations. Complex filters (e.g., regex, inequality) may require a full dimension scan, while exact matches can be resolved directly from pre‑aggregated data.

Distributed PromQL Executor: To handle high‑dimensional queries, a distributed executor inspired by CockroachDB’s design distributes sub‑plans across multiple nodes, reducing hotspot resource usage. The executor also integrates HBase Coprocessor with Delta‑of‑Delta compression, achieving up to 13.1% compression on high‑frequency time‑series data.

Sampling and Data Reduction: The system caps the number of points per series to 3840 (matching typical 4K display resolution) and uses Delta‑of‑Delta compression to keep payloads small. This limits bandwidth and rendering overhead while preserving trend visibility.

Future Work: Ongoing efforts include further optimization of the distributed executor, adaptive pre‑aggregation based on query patterns, exploration of new time‑series storage models, and continued performance tuning to keep query latency low and cost minimal.