Practical Insights into Online Experiment Design and Analysis at Tencent Lookpoint

Speaker Qian Cheng, senior data R&D engineer at Tencent, introduces the fundamentals of online experiments, defining experiments as controlled interventions to validate hypotheses and describing AB testing as a special case of completely randomized designs.

He discusses various experiment types beyond simple AB tests, such as synthetic control, time‑slice rotation, and multi‑armed bandits, and explains how experiment design must consider target population, objectives, and intervention methods.

Case studies from the Tencent Lookpoint platform illustrate experiment implementation in information‑flow scenarios, including flash‑screen ad timing, holiday red‑packet activity, and hotspot recommendation upgrades, highlighting the importance of analyzing both intention‑to‑treat and complier‑average causal effect metrics.

He emphasizes best practices: align analysis unit with experiment unit, verify sample homogeneity, use causal inference tools (instrumental variables, matching, uplift modeling), and construct effect maps to detect heterogeneous impacts.

The talk concludes with a standardized experiment workflow—hypothesis formulation, design, deployment, monitoring, and result interpretation—and a Q&A addressing challenges in B‑side experiments, sample size, spillover effects, and risk assessment.