A/B Testing Framework for Online Experiments: Design, Implementation, Analysis, and Decision Making

The article introduces A/B testing within an online traffic splitting framework, outlining its purpose to compare two strategies under consistent conditions.

It details a four-step experimental process: design, implementation, effect evaluation, and decision making.

In the design phase, it covers defining objectives (e.g., comparing ROI, long-term effects) and structuring experiments via metric design (invariants vs. variables) and traffic acquisition/allocation, including random unit selection, sample size estimation using formulas involving Type I/II errors, and considerations for multiple metrics.

The implementation section discusses random unit granularity (page, session, user levels), orthogonal vs. mutually exclusive experiment designs, and robustness checks using AA tests and overlapping stratified bucketing to mitigate bucket bias.

Effect evaluation covers descriptive statistics (central tendency, dispersion, distribution shape, frequency and trend analysis, correlation) and statistical inference including hypothesis testing (bootstrap + t-test), p-value interpretation, statistical power, and confidence intervals.

Depth analysis includes dimensional subdivision (e.g., user segmentation, position-based analysis) using methods like stratified chi‑square (CMH test) for rate metrics and ANOVA/LSD for mean metrics, and attribution analysis via process metrics, dropout data, and post‑conversion retention, employing models such as funnel, user path, session analysis, and regression/ML techniques.

Finally, the decision‑making step outlines how to interpret results: assessing core metric significance, examining growth across dimensions, and deciding on full‑scale rollout, partial rollout, or rollback based on multidimensional evidence.