Fundamentals and Implementation of A/B Testing at Qunar

A/B testing is introduced as a single‑variable controlled experiment where one factor (e.g., ad background color) is varied while others remain constant, and the effect is measured using metrics such as click‑through rate.

A concrete demo compares an old advertising strategy with a new one by randomly splitting traffic 55/45, collecting click data for 30 days, and observing that the new version yields a higher click‑through rate (6.23% vs 5.26%).

The article emphasizes that conclusions must rely on statistical testing rather than intuition, recommending Z‑test to assess significance.

Experiment space is divided horizontally and vertically; exclusive zones allow isolated traffic, while layered zones enable parallel experiments without interference.

Qunar's A/B testing platform consists of three main parts: a management backend for creating experiments and generating codes, an SDK integrated into application code to perform traffic splitting and logging, and a reporting system that aggregates logs, computes common metrics, and provides customizable dashboards.

The workflow includes adding experiment versions, setting routing rules, defining experiment duration, and confirming configuration before activation.

Statistical validation covers Z‑test calculation (Z = mean / standard error) with a threshold of |Z| > 1.65 for significance, and methods for estimating the minimum required sample size based on desired confidence (α) and power (β), including formulas for unequal traffic splits.

Confidence intervals for the difference (Δ) between versions are derived from the normal distribution of Δ; a 95% interval that crosses zero indicates no clear advantage, while a wholly positive interval suggests the new version outperforms the old.