Handling Outliers in Internet A/B Experiments: Concepts, Methods, and Practical Recommendations

01 Background In many online A/B experiments, practitioners encounter unstable traffic allocation, large fluctuations in historical metrics, and inconsistent results after removing a few users, raising the question of how to handle outliers effectively.

02 Concept Analysis An outlier is generally a data point that deviates markedly from the rest of the sample, but its precise definition varies across domains. The classic 3‑sigma rule assumes normality and is ill‑suited for the heavy‑tailed, power‑law distributions common in internet metrics.

03 Causes of Outliers Typical sources include measurement errors, sampling randomness, fraudulent behavior (e.g., fake orders), and heterogeneous user groups such as B‑side users on a retail platform.

04 Role and Limitations of Outlier Removal in A/B Tests Removing outliers can improve traffic balance and reduce metric variance, but it may also discard valuable information, introduce bias, and require larger sample sizes or variance‑reduction techniques like ANCOVA or CUPED.

05 Traditional Statistical Methods: Trim & Winsorize Winsorizing caps extreme values at a chosen percentile, while Trimming discards them. Experiments show that, for the same percentile, trimming often yields larger bias in mean estimates, whereas winsorizing provides greater variance reduction for comparable bias.

06 Application of Risk‑Control Models A case study demonstrates that detecting and blocking fraudulent users before data collection, combined with server‑side reporting, can substantially mitigate abnormal spikes in duration metrics.

07 Outlier Detection Techniques Simple, low‑cost methods suitable for experiment platforms include using kurtosis thresholds to flag top‑percentile extreme values. More sophisticated approaches (e.g., Z‑score, higher‑order moment analysis) are also discussed.

Appendix The relationship between experiment variance, sample size, and statistical power is presented, highlighting how reducing variance improves the t‑statistic and the likelihood of detecting true effects.