Baidu Intelligent Testing: Analysis and Practice in Testing Activities

This article introduces intelligent practices in the testing analysis phase, which is a critical component of the five-step testing process (test input, execution, analysis, localization, and evaluation). Testing analysis determines the problem recall capability of testing activities, distinguishing between VE (correctness verification) and VA (performance-based verification). The article presents five key intelligent testing approaches:

1. Contract Testing-based Automatic Assertion Point Generation: Contract testing verifies whether code changes cause contract failures. The approach involves capturing different consumer requirements to form contract interface content, recording traffic (request/response), preprocessing data, extracting schemas, and importing to YAPI for mock data generation. Through reverse engineering, code changes can automatically detect contract modifications and update mock data accordingly.

2. Time-slicing C++ Memory Leak Detection: Uses DTW (Dynamic Time Warping) curve similarity algorithm to compare memory curves between versions, identifying leaks when curves are dissimilar. Further optimization with Cart decision tree improved prediction accuracy from 75% to 98%. Historical data prediction reduces test time by 1/3.

3. Dynamic Threshold-based Performance Diff Detection: Addresses performance test accuracy issues caused by environmental variations. Box plot analysis uses normal distribution to set reasonable thresholds based on historical data. LOF (Local Outlier Factor) algorithm identifies anomalies by comparing point density with neighboring points. These strategies significantly reduced yellow-light rates and retry rates.

4. Intelligent Assertion Completion for Functional Testing: Applies machine translation models (Transformer, TestNMT, Reformer) to understand tested methods and infer verifiable content for generating appropriate assertions. The approach addresses the fact that less than 5% of functions have manually verified assertions. Achieved 41% assertion accuracy.

5. Visual Recall with and without Reference: UI Diff compares screenshots to identify visual differences. Supports various strategies including full-page diff, custom region diff, multi-resolution testing, and different product versions. Achieved 90%+ accuracy, covering 12% of frontend projects and recalling 100+ issues quarterly. No-reference recall detects anomalies like white/black screens, component overlap, and text truncation using CNN, achieving 98%+ accuracy.