Post‑Darwin Method for Game Business Effect Evaluation Using Stratified Sampling

Authors: vivo Internet Data Analysis Team – Luo Yandong, Zhang Lingchao

This article introduces common problems and methods for evaluating the effectiveness of internet business data, and proposes an optimized "Post‑Darwin" analysis method based on stratified sampling. The method is suitable for scenarios where A/B testing is unavailable or user groups are uneven.

1. Introduction

Game business, as a major revenue source, frequently launches operational activities to boost player spending. Analysts need scientific models to accurately assess the value and risk of these activities.

A typical problem illustrated is a Simpson’s paradox in profit‑rate metrics caused by uneven monthly user payment distribution.

Over three years, the team refined the "Post‑Darwin" methodology to address these evaluation challenges.

2. Common Issues in Game Business Effect Evaluation

Impact of natural factors (holidays) on payment growth.

Participation thresholds that create inconsistent user groups.

Discrepancies between overall and segment‑level effects.

Inability to run fair A/B tests due to anti‑cheating policies.

Short‑term spikes vs. long‑term value of promotional activities.

3. Development of Evaluation Methods

3.1 What is Effect Evaluation?

It measures how activities (discounts, rebates, promotions) improve core metrics such as revenue and profit.

3.2 Evolution Stages

Time‑Series Comparison – compare post‑activity metrics with the previous period.

Natural Filtering – separate natural seasonal changes from activity impact.

A/B Testing – strict controlled experiments (limited by user distribution and risk).

Post‑Darwin – combines advantages of methods 2 and 3 by stratifying users into uniform layers and comparing activity vs. non‑activity groups within each layer.

3.3 Advantages and Applicability

4. Post‑Darwin Methodology

4.1 Define Research Objects – target groups (participants vs. non‑participants) and metrics (e.g., ARPU, coupon redemption).

4.2 Time‑Series Comparison for Each Group

Formulas used:

C1 = A*(A4‑A3*(B4/B3));

C2 = A*(A2‑A1*(B2/B1));

4.3 Stratified Comparison – split users into uniform layers based on pre‑activity payment levels, then compute incremental impact within each layer.

4.4 Result Aggregation – subtract natural growth (non‑participants) from activity growth (participants) for each layer, then sum across layers to obtain overall impact.

Special notes:

If a layer’s confidence is insufficient (<5% difference), it may be excluded or weighted according to its contribution.

When participant vs. non‑participant size ratio is extreme (<1:10), sampling can be applied to the non‑participant group.

5. Practical Cases

5.1 Dragon‑Boat Festival Recharge Activity

Background: tiered recharge gifts (100, 1000, 4000 RMB). Key Findings: Users exposed to the activity showed significantly higher payment uplift, especially in low‑payment tiers; non‑receiving users had lower or no uplift.

5.2 Game Version Reservation Effect

Background: analyzing the impact of pre‑release version reservation on user activity and spending.

Findings: Users who perceived the upcoming version paid ~6% more after release; reservation behavior further amplified post‑release spending.

6. Conclusion and Outlook

The "Post‑Darwin" method, built on stratified user comparison, effectively resolves uneven user distribution and lack of A/B testing in game business effect evaluation. Future work includes integrating the methodology into big‑data platforms for real‑time, automated analysis.