Regional Heterogeneity in Game AB Experiments: Detection, Decomposition, and Prediction

Introduction – The article discusses regional differences in game AB experiments, highlighting the need to detect and understand heterogeneity across regions.

Current Situation – An example of three regions shows varying effects, where an overall positive impact can mask a negative impact in a specific region.

Common Handling – Simple region‑level drill‑down provides unbiased estimates but lacks generalization to other regions.

Heterogeneity Detection – Meta‑analysis is used to test whether observed regional differences are statistically significant rather than mere noise.

Decomposition – After confirming heterogeneity, the framework separates its sources: differing conditional treatment effects (CATE) versus differing user distributions.

Modeling CATE – Describes ATE and CATE τ(x), and mentions machine‑learning learners such as X‑learner and T‑learner to estimate treatment effects based on user features.

Sources of Heterogeneity – Identifies two origins: (1) true treatment‑effect variation across regions, and (2) distributional differences of user characteristics.

Prediction – Once the source is known, the model can predict effects in unseen regions, showing high correlation between predicted and actual effects.

Summary – Presents a workflow: detect heterogeneity, decompose its source, and use insights for strategy iteration, emphasizing that distributional factors often dominate over regional labels.

Q&A – Addresses practical questions about blank experiments, ensuring balanced AB groups, and extending the methodology beyond regions to other dimensions such as age.