Challenges of Traditional Experiment Systems and the Vision for Next‑Generation Evaluation Platforms

In many internet businesses, experiment systems are used to test strategies such as UI changes or ad placement, providing objective decision support; however, traditional systems are based on 19th‑century scientific assumptions that do not fully address today’s complex online scenarios.

The core of an experiment is to intervene on a variable, collect data, analyze it, and verify statistical significance, but three critical stages—intervention, measurement, and analysis—can each fail, rendering the whole process ineffective.

Problems that traditional experiment methods cannot solve:

1. Inability to intervene : Randomized controlled trials require clear A/B groups, which is impossible for many real‑world cases such as evaluating a live concert’s impact on user preference or protecting minors with a policy that cannot be selectively applied.

2. Inability to measure : Dynamic user populations and spill‑over effects (e.g., emoji‑pack usage contaminating control groups or inventory depletion in e‑commerce) make static measurement inaccurate.

3. Inability to analyze : High‑dimensional parameter spaces in recommendation algorithms explode the combinatorial search space, making manual analysis infeasible.

These limitations motivate a vision for the next‑generation experiment platform.

Future directions:

1. An observation system : When intervention is impossible, the platform should support observational studies—methods borrowed from economics and sociology—to evaluate policy effects.

2. A dynamic system : The platform must handle dynamic interactions (e.g., social graph‑based grouping to avoid sample contamination) and provide real‑time estimation.

3. A decision system : By employing surrogate models and Bayesian optimization, the system can automatically explore high‑dimensional parameter spaces and make data‑driven decisions without exhaustive human analysis.

The speaker, Rex, an A/B testing R&D engineer at Volcano Engine, concludes the session and provides a link to the replay video.