Iterative Load Testing Process Improvement and Full-Process Evolution

Background

System service load testing is time‑consuming and labor‑intensive, often performed after 23:00 in production environments, requiring overnight work and causing high human‑cost and fatigue for the team.

To alleviate fatigue and reduce labor cost, the existing load‑testing workflow was analyzed. The current modes—routine (regular) load testing and large‑scale full‑link testing—are driven by business needs but suffer from timeliness issues and high manpower.

To make routine testing more efficient and business‑aligned, some routine tests are shifted to daily iteration projects, creating an iterative load‑testing approach that provides finer‑grained, smaller‑scope testing to address timeliness and labor challenges.

Load‑Testing Process Improvement Analysis

1. Historically, load testing focused on routine testing with full‑link testing as a supplement, scheduled only during major promotions (e.g., 618, Double‑11).

2. Tests mainly target performance metrics of promotional traffic; issues in individual service stages often surface only during full‑link testing, causing delayed detection.

3. Although results are obtained, the manpower cost and timeliness are unsatisfactory.

4. Inter‑service traffic capacity and request limits are discovered only through routine or full‑link tests; lack of daily data leads to timeouts, rate‑limiting, or circuit‑breaker events that reduce test effectiveness.

Iterative Load‑Testing Process

Based on the analysis, an iterative load‑testing workflow is proposed:

The workflow adjusts the existing process to achieve cost reduction and efficiency gains by testing after each daily iteration release, ensuring business alignment and meeting testing needs across five improvement dimensions:

Stability of Load‑Testing Service

Testing after iteration releases reveals hidden stability issues early, allowing flexible optimization and rapid issue resolution.

Stability of Load‑Testing Environment

Daily iteration testing exposes production‑environment machine problems earlier, enabling proactive mitigation before large‑scale tests.

Testing Timeliness

Fast‑track testing assigns minimal‑unit tasks immediately after release, with scheduled tests and next‑day report collection for quick results.

Testing Personnel

1. Lower the learning curve for QA engineers by simplifying the testing process and platform.

2. Open testing to all business QA members, not just a few specialists.

Business Alignment

For each iteration, participants assess whether testing is needed based on business characteristics:

1. If testing is unnecessary, the impact on promotional activities is minimized.

2. If testing is required, the impact scope is evaluated:

Backend refactor project, affecting main workflow
New/changed business requirements involving core interfaces
Core business scenario adjustments
Dependent service interface changes
…

Project‑level testing defines a small scope, quickly clarifying metrics and scenarios.

3. Increase QA involvement, turning performance testing into a routine skill.

4. Freshly completed projects have up‑to‑date knowledge, allowing low‑cost script and data creation; scheduled tests yield low‑cost analysis.

5. Accumulated project‑level results inform future large‑scale testing plans and help pre‑empt service bottlenecks.

Full‑Process Load‑Testing Model Evolution

Iterative testing becomes the smallest unit, aggregated into routine and full‑link testing to ensure coverage across different granularity levels. References: Routine Load Testing , Full‑Link Load Testing .

Service traffic flow diagram:

Load‑Testing Model Comparison

Iterative Load‑Testing Feedback Effects

Business Value: Since February, several iteration tests have uncovered six hidden performance issues, allowing services to avoid hidden risks.

Result Feedback:

Conclusion

1. Business QA staff should acquire basic performance‑testing skills to identify hidden risks in project requirements.

2. Testing by project scope may not cover all service functions, but frequent iterative testing gradually expands coverage.

3. Dilute the pressure of large‑scale and routine testing preparation by integrating testing into the demand lifecycle, achieving lightweight, distributed test‑asset accumulation.

About the author: Zhuang Jindi, head of Shenzhen Business Support Group / Performance Virtual Group, responsible for case platform construction, performance testing process establishment, and full‑link testing implementation.