A Comprehensive Guide to A/B Testing: Principles, Design, Metrics, and Decision Making

1 What is A/B testing?

A/B testing validates product capabilities or strategies by creating two (A/B) or multiple (A/B/n) versions, randomly exposing comparable visitor groups to each version, collecting user experience and business data, and analyzing the results with statistical methods to select the best version for deployment.

Pre‑validation through A/B testing provides evidence‑based decisions and allows limited traffic testing to avoid large‑scale negative impacts.

2 Why introduce A/B testing

Pricing systems serve diverse business scenarios, requiring flexible quoting capabilities and strategies that must adapt to market changes, operational shifts, or new algorithms.

Because price adjustments can be sensitive and outcomes uncertain, data‑driven experimentation helps assess potential gains or risks before full rollout.

A/B testing acts as a powerful tool for decision verification and improves the overall product development workflow.

3 How to conduct A/B testing

A/B testing is an iterative learning loop driven by business needs and data‑driven decisions, forming a closed cycle of hypothesis, metric definition, experiment, analysis, release, and new hypothesis.

3.1 Formulate hypothesis

The first step is to clearly state the background and goal of the experiment, defining a null hypothesis and an alternative hypothesis that can be evaluated from both user and data perspectives.

3.2 Define evaluation metrics

Metrics must reflect the experimenter's intent and be measurable; they are categorized as primary (core goal), secondary (supporting perspectives), and guardrail (long‑term safety) indicators.

3.3.1 Select experimental unit

Traffic splitting divides overall traffic into mutually exclusive groups; the key is to find homogeneous sub‑populations that represent the larger audience, ensuring accurate evaluation of strategies.

3.3.2 Compute sample size

While larger samples increase confidence, practical constraints (limited traffic, high error cost) often require balancing sample size against experiment speed.

Sample‑size formulas differ for means and proportions:

Mean:

Proportion:

3.3.3 Traffic segmentation

After determining experiment traffic, allocate it uniformly to control and variant groups, often validated by an AA “dry run” to ensure no pre‑existing differences.

3.3.4 Experiment duration calculation

Experiment length balances sufficient traffic for statistical power against the need for rapid iteration; it should span at least a week to mitigate weekly patterns and allow novelty effects to settle.

4 Implement traffic splitting and data reporting

During execution, each request is tagged with experiment identifiers; logs are processed to compute metric values and generate analysis reports, with alerts for anomalies.

5 Result analysis (hypothesis verification) and decision

After the experiment, statistical analysis validates the original hypothesis, typically using hypothesis testing methods.

5.1 T‑test

T‑tests (and Z‑tests) assess the significance of differences between two population means; the choice depends on sample size and whether population variance is known.

The T‑statistic formula (degrees of freedom f, sample standard deviations S1, S2) is shown below:

Decision thresholds are set (commonly α = 0.05); if the T‑statistic falls in the rejection region, the null hypothesis is rejected.

5.2 Decision using P‑value

The P‑value represents the probability of observing the data (or more extreme) if the null hypothesis is true; a small P‑value leads to rejecting the null hypothesis at the chosen significance level.

5.3 Scientific evaluation based on hypothesis verification

Before the experiment, AA testing ensures unbiased groups; after the experiment, T‑statistics and P‑values guide the decision.

5.4 Decision

Based on analysis, decide whether to iterate, terminate, or scale the experiment. Scaling follows three stages:

Stage 1 – Small traffic (≤5%): monitor for negative impact over 3‑5 days.

Stage 2 – Scaling: gradually increase traffic, observing weekday/weekend effects for at least a week.

Stage 3 – Long‑term hold: retain ≤5% of the original strategy as a “reverse bucket” for ongoing observation.

6 Summary

A/B testing is a highly effective method for measuring online optimization, provided that experiment goals are measurable, traffic splitting is sound, and results are correctly interpreted.

About the author: Wang Menglong, Software Engineer, R&D Technology Department, Zhuanzhuan.