Boost Backend Batch Testing Efficiency with a Comparative Test Platform

Background

The core financial system of Vipshop processes large‑scale batch data for orders, payments, and inventory. Supplier billing involves a series of ordered backend batch steps such as data extraction, preprocessing, payment calculation, discount allocation, and recomputation. The flexible, heavily‑configured design supports many business models but makes the payment‑calculation logic tightly coupled and hard to test.

Whenever a new business model is added, the billing logic must be modified, and ensuring that changes do not break existing calculations becomes a major testing challenge. Traditional regression testing can require more than seven person‑days, while new features need only two person‑days, making comprehensive coverage difficult.

To address these issues, a comparative testing tool platform was developed. After adoption, testing efficiency and quality improved significantly, regression testing time dropped to under one day, and billing logic changes resulted in almost no errors.

Platform Principle

The platform can import real online business data for automated regression testing and also create custom scenario data for functional testing. It is independent of application code and deployment environment, working only with test business data. Currently it supports MySQL databases and VMS message drivers.

If business rules remain unchanged, the output data (Result1 and Result2) from before‑ and after‑optimization runs should be identical. Expected results can be stored in a baseline database, and each regression run compares current results against this baseline.

Application Scenarios

The platform suits backend job tasks with the following characteristics:

Message jobs that consume external messages and write to tables.

Data‑processing jobs that transform data from multiple tables into others.

Hybrid jobs that consume messages, combine with internal data, and write results.

Examples include:

Refactor Regression: Copy real production data into a test environment for regression, achieving one‑click end‑to‑end testing without affecting live business.

Functional Testing: Prepare scenario data and expected results; the platform initializes data, starts the job, stops it, and automatically compares outcomes, greatly improving efficiency.

Workflow

Test cases consist of input data and expected output stored in database tables. When the input and expected results are unchanged, the job should produce identical outputs, such as supplier billing data for a given period.

The platform collects deployment, runtime, and dependency information, stores scenario inputs and expected outputs, links them to the job, and upon execution initializes data, runs the job, collects results, and compares them with the baseline.

Overall Architecture

The architecture includes:

Base Data Module: Manages environment info, startup scripts, and database instance details.

Template Management: Provides data and message templates; supports VMS message triggering and scenario input/result templates with backup and cleanup options.

Scenario Management: Stores scenario inputs, expected outputs, and job launch information.

Test Plan Management: Groups multiple scenarios into test sets for one‑click execution and flexible regression.

Scheduling Execution Management: Handles task queuing, data initialization, job execution, result collection, and comparison, with configurable wait times.

Report Management: Tracks execution and maintenance status of scenarios and test plans.

Practical Effects

All core financial modules (VSP, AR, VCP) now use the tool for functional and regression testing. Billing scenario coverage is comprehensive, test quality and efficiency have markedly improved, and VCP can be released within two workdays after development hand‑off.

Tool Advantages

One‑Click Online Version Regression: Import one to two months of production data, run the job, and generate expected results; a single click validates post‑refactor correctness.

No Learning Curve for Testers: Testers create scenario data without coding, focusing on critical test design.

Low Maintenance Cost: Only modify templates when related table fields change; result templates remain unchanged.

Test‑First Approach: Testers prepare scenario data alongside development, concentrating on logical outcomes.

Increased Test Efficiency: Writing a case simultaneously creates a regression case; developers trigger automated regression with one click.

Developer Participation in Functional Testing: Developers can self‑test via the platform, reducing reliance on separate test environments.

Version Enhancements

Future plans include supporting additional databases (Hive, Oracle), adding API automation testing, providing change alerts for table schema modifications, one‑click template updates, and comparing results across different versions with the same input data.