Design and Implementation of Youzan ABTest System for Data‑Driven Growth

Youzan, a merchant service platform, faces diminishing traffic growth from mobile internet and increasing complexity of SaaS business scenarios. To improve product iteration efficiency and support data‑driven growth, the company built an internal A/B testing system.

Background : A/B testing is a core tool for data‑driven growth, allowing comparison of alternative strategies by randomly splitting traffic and measuring conversion metrics.

ABTest Overview : The article defines A/B testing as a comparative analysis method that isolates traffic, runs random experiments, and monitors results. Typical application scenarios include gray‑release, feature optimization, content optimization, and operational optimization.

Core Concepts (inspired by Google’s “Overlapping Experiment Infrastructure”): • Application : a logical grouping of traffic and systems (e.g., a product detail page). • Scene : a business unit where multiple experiments can run; traffic can be reused across scenes within the same application. • Experiment : a concrete strategy described by a configuration; experiments in the same scene are mutually exclusive. • Traffic Source : defines the granularity and proportion of traffic allocated to an experiment.

ABTest System Design :

1. Interaction Flow : The system consists of three parts – the ABTest platform, ABTest SDKs (Java and Node), and a data pipeline. Users create and manage metadata on the platform, SDKs fetch experiment configurations and report logs, and the data pipeline aggregates real‑time and offline metrics.

2. ABTest Platform provides metadata management, SDK integration support, real‑time and offline reporting, and anomaly monitoring/alerting.

3. ABTest SDK (Java/Node) implements: • Dynamic experiment configuration distribution using consistent hashing (MurmurHash) with 1/16384 granularity. • Automatic logging of request, custom, and performance data. • Generation of tracing identifiers (abTraceId, bcm) for front‑end tracking.

4. Data Flow uses NSQ, Kafka, Flink, Hive/Spark, HBase, and Druid to collect SDK logs, process real‑time streams, and store both real‑time and offline reports.

Usability Guarantees : The team improves platform ease‑of‑use (metadata UI, role/permission, approval workflow, whitelist), optimizes SDK performance (multi‑level caching, batch reporting), and provides extensive testing support (sample code, environment selection, user‑ID based validation, log query).

Metric Production includes:

Data‑warehouse snapshots of ABTest metadata and logs.

Real‑time reports (5‑minute granularity) generated by Flink.

Effect reports with a generic conversion model (request → exposure → click → purchase) and custom attribution support.

Support for both count‑based and user‑based metrics, with HyperLogLog and precise de‑duplication for user counts.

Distinction between direct (product‑level) and indirect (store‑level) conversion effects.

Anti‑fraud filtering using absolute thresholds and 3σ rules.

Statistical significance testing using Z‑test and p‑value comparison (e.g., 95% confidence).

System Evaluation : The primary north‑star metric is the maximal GMV uplift across experiments. The evaluation also considers coverage, impact, and risk avoidance.

Future Outlook : Planned work includes front‑end invisible tracking, integration with growth analysis platform, richer custom conversion goals, automated experiment evaluation reports, and broader adoption across the company.

The article concludes that A/B testing is essential for data‑driven growth, and the ABTest system, together with growth analysis and event‑tracking platforms, forms the “three swords” of Youzan’s data growth team.