Online Experiment Design and Analysis: Practices, Case Studies, and Guidelines from Tencent Data Platform

The speaker, a senior data R&D engineer at Tencent, introduces the concept of experiments as controlled interventions to validate hypotheses, emphasizing that experiment design—including target, objective, and intervention—is the core of reliable analysis.

AB testing is described as a specific case of completely randomized designs (CRD), requiring fully randomized assignment of homogeneous subjects to treatment groups; more complex scenarios may involve synthetic control, time‑slice rotation, or multi‑armed bandit (MAB) methods.

In the context of Tencent's information‑flow product, over 500 parallel experiments run daily with thousands of candidate metrics, illustrating the scale and diversity of online testing.

Two detailed case studies are examined: (1) a flash‑screen ad timing experiment that showed modest metric changes overall but significant improvements for the subset of users with 1‑5 minute intervals, highlighting the importance of analyzing the effective exposure population and using ITT versus CACE analyses; (2) a Spring red‑envelope activity that increased short‑term engagement but raised concerns about spill‑over effects and the need to evaluate post‑experiment performance.

Additional recommendation‑system experiments (hotspot algorithm upgrades) demonstrate how to trace strategy impact through exposure, CTR, and overall platform metrics, and why contradictory signals require a detailed causal chain analysis.

The article outlines three key analysis recommendations: align experiment units with analysis units, keep analysis focused on the original hypothesis and design, and leverage online experiments as the most direct causal inference tool.

Challenges specific to B‑side experiments are discussed, such as mismatched experiment and analysis units and complex data pipelines, with suggestions to transform B‑side tests into C‑side equivalents.

Finally, a standardized experiment analysis workflow is presented—hypothesis formulation, design, deployment, monitoring, and result interpretation—along with best‑practice guidelines (e.g., ensuring sufficient sample size, monitoring metric trends, handling spill‑over, and documenting conclusions for future iterations).