Understanding ABTest: Concepts, Design, Multi‑Layer Experiments, and Practical Implementation

With the rise of data‑driven decision making, ABTest has become a core tool for many internet companies, providing scientific traffic splitting, real‑time experiment monitoring, and reliable results to support business decisions.

What is ABTest? ABTest (also called A/B experiment or small‑traffic random experiment) involves creating a control strategy A and a new strategy B, randomly assigning users to two groups, and analyzing metric changes to determine if the new strategy meets expectations.

The relationship among applications, scenarios, experiments, and traffic is illustrated in Figure 1‑1. Key terms include:

Application: a logical division of traffic and system, e.g., a product detail page or a shopping cart.

Scenario: a business situation where different strategies need comparison; a scenario can contain multiple experiments.

Experiment: a concrete strategy described by an experiment configuration; experiments within the same scenario are mutually exclusive.

Orthogonal traffic: each experiment layer receives an independent random split of traffic, ensuring experiments do not affect each other.

Exclusive traffic: traffic split within the same layer does not overlap, guaranteeing isolation when traffic is sufficient.

Splitter: the component that routes users to different versions based on defined rules.

ABTest Design – After understanding the concept, the design flow is shown in Figure 2‑1. Users from front‑end APP, H5 or PC are first filtered for target groups, then the type of experiment (orthogonal or exclusive) and traffic allocation (e.g., 5 % or 10 %) are decided. A hash of pin/uuid/deviceId is computed, mod‑processed, and assigned to buckets that are further split into groups A‑A, A‑B, and B.

Single‑layer vs Multi‑layer Experiments

Single‑layer experiments (Figure 3‑1) suffer from limited scalability, traffic starvation, and bias. Multi‑layer experiments (Figure 1‑1) allocate 100 % of traffic to each layer, eliminating starvation and bias, and allow unlimited concurrent tests.

Mixed designs combining single‑ and multi‑layer experiments are illustrated in Figure 3‑2, suitable for complex business scenarios where different layers (e.g., UI, search results, ad results) operate independently.

Evaluating ABTest Results

The primary goal of ABTest is to select the optimal strategy and avoid inferior ones. The “Extreme GMV Lift” metric measures the improvement of the best experiment over the worst, normalized by total request volume. Additional evaluation metrics are shown in Figure 4‑1.

Project Practice

A concrete case study walks through the full lifecycle: product requirement (improve customer‑service efficiency via an invoice prompt), experiment creation on the ABTest platform, SOA service integration (code snippet below), result analysis, and conclusion.

/**
 * Get ABTest split result
 * @param clientInfo
 * @param paramMap
 * @return
 */
public static Map
getABTest(ClientInfo clientInfo, Map
paramMap){
    // Initialize product line, register experiment (base version = control A)
    String productLine = "productLine123";
    // Experiment ID
    String expId = Util.toString(paramMap.get("expId"));
    // Base flag – fallback to control A if ABTest platform fails
    String baseFlag = Util.toString(paramMap.get("baseFlag"));
    // Split type: 1‑single experiment, 2‑batch experiment
    int abType = Util.toInt(paramMap.get("abType"));
    // Split mode: 1‑pin, 2‑userid, 3‑deviceId
    List
mode = Util.swapList(paramMap.get("mode"));

    ABPower abPower = new ABPower.ABPowerBuilder(productLine).register(expId, baseFlag).build();
    // Determine split identifiers
    String pin = "";
    String uuid = "";
    String deviceId = "";
    if (mode != null && mode.contains(1)) { pin = clientInfo.getPin(); }
    if (mode != null && mode.contains(2)) { uuid = clientInfo.getUuid(); }
    if (mode != null && mode.contains(3)) { deviceId = clientInfo.getDeviceId(); }

    ABUser abUser = new ABUser(pin, uuid, deviceId); // default split by pin
    // Optional pre‑conditions
    if (MapUtils.isNotEmpty(preCondition)) {
        preCondition.forEach((k,v) -> abUser.setPreCondition(k, v));
    }
    if (StringUtils.isNotEmpty(preCondition) && "version".equals(preCondition)) {
        abUser.setPreCondition(preCondition, clientInfo.getClientVersion());
    }
    if (abType == 1) {
        // Single experiment routing
        ABSingleResult abSingleResult = abPower.router(abUser, expId);
        result = abSingleResult.getABData();
    }
    if (abType == 2) {
        // Batch experiment routing
        ABBatchResult abResult = (ABBatchResult) abPower.batchRouter(abUser);
        result = abResult.getABData();
    }
    if (result == null) {
        return abTestResult;
    }
    // ... further processing ...
    return result;
}
/**
 * Report ABTest split result and upload tracking data to the platform
 */
Map
abTestResult = getABTest(sopParam.getClientInfo(), abTestParamMap);
if (MapUtils.isNotEmpty(abTestResult)) {
    Map
invoiceABTestExpInfo = Maps.newHashMap();
    invoiceABTestExpInfo.put("touchstone_expids", Util.toString(abTestResult.get("touchstone_expids")));
    pointData.put("invoiceABTestExpInfo", invoiceABTestExpInfo);
}

Analysis of the experiment showed that exposing the invoice prompt increased customer‑service efficiency, although the test was still in a small‑traffic pilot phase and further validation is needed at full scale.

Conclusion

ABTest is a core tool for data‑driven growth; building a robust ABTest system enables product and research teams to iterate faster, help merchants grow, and lays the foundation for further data‑driven initiatives.