Understanding AB Testing: Design, Execution, and Analysis

The purpose of AB testing is to obtain representative conclusions through scientific experimental design, representative sampling, traffic segmentation, and small‑traffic testing, and to ensure that these conclusions can be generalized to the entire traffic. It is now widely used in recommendation algorithms, product interaction design, advertising placement, product feature iteration, operational strategies, and is the most commonly used and accurate method for evaluating resource usage ROI.

In 2000, Google engineers applied this method to internet product testing, after which AB testing became increasingly important and gradually turned into a key tool for scientific product operation iteration and data‑driven growth.

From foreign companies such as Apple, Airbnb, Amazon, Facebook, Google, LinkedIn, Microsoft, Uber to domestic companies like Baidu, Alibaba, Tencent, Didi, ByteDance, Meituan, massive AB tests run on various endpoints (websites, PC applications, mobile apps, emails, etc.).

In typical online AB experiments, users are randomly and evenly divided into different groups; users within the same group experience the same strategy during the experiment, while different groups may use the same or different strategies.

Simultaneously, the logging system tags users according to the experiment system to record their behavior, and the data computation system calculates various experimental metrics from the tagged logs. Experimenters use these metrics to understand and analyze how different strategies affect users and whether they meet the pre‑experiment hypotheses.

As shown in Figure 1‑1, the diagram summarizes the classic AB testing workflow.

Figure 1‑1 AB Testing Process

Applying the process shown in Figure 1‑1 to product iteration means releasing product versions with different features or strategies simultaneously to two or more user groups. These groups are randomly sampled from the overall user base, usually representing a small fraction, and the groups have similar attributes and composition.

For example, as shown in Figure 1‑2, we use an AB test to determine which banner color yields a higher click‑through rate.

Group A sees a light‑colored banner, Group B sees a dark‑colored banner, and we analyze which color attracts more user attention and increases clicks. If the dark banner performs better, it is rolled out to all users.

Figure 1‑2

In practice, AB testing evaluation is rarely limited to click‑through rate; other metrics must also be considered comprehensively.

How to design the AB testing process? How to design the experiment plan? How to conduct experiment analysis?

These topics are detailed in the AB Testing section of the "Data Intelligence Knowledge Map". Follow the public account below to download the full version of the map.

Other artificial intelligence sections include: intelligent risk control, user profiling, recommendation systems, pre‑training, privacy computing knowledge system, causal inference.

The full‑size "Data Intelligence Knowledge Map" can be obtained by following the "Big Talk Data Intelligence" public account and replying with “knowledge map”.

References:

1. "Dry Goods | All the ABTest Knowledge You Want to Know Is Here" – https://www.6aiq.com/article/1606260721974

2. "Finally Someone Explained AB Testing Clearly" – https://www.51cto.com/article/715248.html