Design and Implementation of the Replay Comparison Engine for Automated Service Regression Testing

Youzan, a SaaS company, faces rapid business growth and frequent functional iterations across its product lines (e‑commerce, retail, beauty, etc.). Ensuring functional correctness and service stability during these changes is a major challenge for developers and testers.

Existing quality‑assurance mechanisms rely on conventional automated testing tools that require large amounts of test data and extensive script writing, leading to high cost and low efficiency. Moreover, as system complexity grows, the number of regression scenarios increases, making it difficult to cover all cases with manual test cases.

To address these problems, Youzan built the Replay Comparison Engine , an automated service regression verification platform that replays real online requests in a pre‑release environment and compares the responses to automatically detect interface correctness issues, dramatically reducing testing cost and improving efficiency.

What the engine does : it captures live traffic, replays the same requests against a staging environment, and compares the online and pre‑release responses. Most business changes manifest as differences in API return values, so detecting response discrepancies validates service correctness.

Advantages include:

Comprehensive coverage by using real online traffic.

Real‑time automated collection and verification, greatly boosting efficiency.

Quick detection of added or removed response fields.

Discovery of hidden bugs that only appear after long‑running operations.

Phase 1 Design and Implementation :

The engine consists of three main components:

Client SDK – intercepts online requests/responses, assembles them, and sends the data to a message queue (MQ).

Comparison Engine Server – core service that replays requests (Dubbo generic calls and REST) and compares results, storing mismatches in a database and KV store.

Comparison Engine Console – UI for configuring replay parameters and viewing inconsistency reports.

Key design points of the Client SDK:

Support request sampling to avoid overwhelming the backend.

Buffer low‑traffic requests for later replay.

Asynchronous collection to minimize impact on service latency.

Server‑side design highlights:

Mitigate false positives by handling data‑change side effects (e.g., inventory updates during order placement) through multiple replays.

Failed replays are sent to a retry topic to prevent blocking normal processing.

Thread‑pool scheduling based on request latency (RT) to efficiently handle I/O‑bound replay tasks.

Customizable comparison rules, such as ignoring specific fields or list order.

Phase 2 (in development): Support for Write‑Interface Replay

Challenges include determining write‑interface correctness, preventing dirty data, capturing all data changes, and reconstructing the application context.

Write‑request recording uses ReplayClientAspect to record request context and a Framework component to capture interactions with third‑party services.

Write‑request replay loads the recorded context, mocks third‑party interactions, and ensures no real data is modified. A special marker distinguishes replayed write requests, and the framework mocks all external calls.

Integration is simple: add the @Replay annotation to the target interface. Example:

@Replay

public ListResult
listBySkuCodes(ListBySkuCodesParam param) { ... }

Configuration is performed via the console, where users set request sampling QPS, replay parameters, and comparison options (e.g., ignored fields, list‑order tolerance).

The engine has been adopted by most Youzan services (product, inventory, marketing, membership), significantly improving scenario coverage and regression efficiency.

Conclusion and Outlook : Future work focuses on test‑case library management, automatic request template identification, cross‑environment regression, and further enhancing core‑scenario testing efficiency.