Full-Chain Traffic Data Detection in DiDi's Omega Platform

Omega is an internal DiDi platform that provides end‑to‑end mobile user‑behavior collection, processing, storage, presentation, and application. It starts with SDK‑based data collection, performs real‑time or offline ETL to generate business‑required metrics, and supports more than 1,500 applications across most business lines.

The data pipeline consists of six core modules:

Collection SDK – gathers, assembles, and sends event data with caching to reduce loss and duplication, and receives server‑side policies for targeted collection.

Data Reception – a high‑throughput lightweight web service that receives SDK data and configuration pushes.

Real‑time ETL – handles heavy processing such as format conversion, geo‑enrichment, and whitelist filtering.

Offline Data Warehouse – Kafka to Hive cleaning, building both generic ODS and session/device‑oriented warehouses.

Real‑time Split – routes data to real‑time warehouses and algorithmic strategy services.

Behavior Analysis – provides downstream analytics like event segmentation, funnel, and path analysis.

Key challenges of this long chain include the diversity of data sources requiring multiple SDKs, the massive data volume demanding high accuracy and timeliness, and the continuous evolution of SDK hosts and components that may introduce issues.

The detection service measures three aspects of data quality: completeness, precision, and accuracy. Completeness is evaluated by comparing expected request counts (derived from SDK counters) with actual received counts, yielding loss‑rate and duplicate‑rate metrics. Precision focuses on the consistency of measurements, while accuracy ensures reproducibility across multiple runs.

Two core metrics are used:

Loss Rate – the ratio of actual received SDK requests to the expected number of requests, reflecting SDK‑side collection quality.

Storage Rate – the ratio of event‑level PVs that successfully flow through the four internal nodes to the total SDK request PVs, reflecting the quality of the internal channel.

Hourly and daily detection services have been implemented, achieving a native SDK loss rate of 0.5% and a web SDK loss rate of 2%, well below the industry average of ~5% for web events.

The platform also provides integrity tags in the offline ODS layer using Flink checkpoint IDs, ensures strict monotonicity, and prevents downstream ETL from reading failed checkpoints.

Additional features include:

Global detection of packet loss and duplication across the entire chain.

Monitoring of unregistered events and attributes, enabling easy registration and governance.

Attribute fill‑rate monitoring (e.g., uid, city ID, channel) and Android/iOS ratio checks.

Dashboard visualizations and push notifications for data‑quality alerts.

Beyond the technical content, the article concludes with a recruitment notice for senior data engineering positions in the DiDi Data Platform team.