Building a Quality Assurance System for Alibaba Video Search

Alibaba Youku video search serves as a front‑line content distribution channel, handling massive video catalogs and providing search services across the entire Alibaba Entertainment product line.

Given the high demands for stability and precision, a comprehensive quality assurance (QA) system is essential.

1. Business Characteristics

Video search architecture features:

Supports complex and diverse upstream business scenarios with intricate logic.

The end‑to‑end business chain from query to result is long, involving many modules and external dependencies.

Algorithms depend heavily on data; changes in underlying data affect algorithm outputs.

2. Testing Challenges

Long, complex business chain makes coverage measurement difficult.

Offline and real‑time data changes impact business; data quality monitoring must be tightly coupled with business.

Algorithm modules are complex and often non‑interpretable, making effectiveness evaluation hard.

Bad cases are sparse in massive data; discovering common failure patterns is challenging.

3. Quality Assurance Solutions

The QA framework consists of three layers: engineering quality, algorithm quality, and user experience.

3.1 Engineering Quality – Regression

Regression testing is performed before each release to catch bugs early. Each module maintains its own regression suite, automated via an internal smoke‑testing platform. Over 5,000 regression cases are executed across environments, with minute‑level online inspections.

3.2 Engineering Quality – Monitoring

Functional Monitoring: Layered monitoring based on module segmentation, storing daily smoke‑test results and real bug data. Over 50 bugs have been discovered through continuous inspection.

Effect Monitoring: Real‑time effect monitoring detects performance regressions, supports trend analysis, and closes the loop: monitoring → issue detection → resolution.

3.3 Algorithm Quality – Data Monitoring

Offline Monitoring: Custom monitoring rules are defined for each table. Steps include selecting tables, creating partitioned monitoring tables, configuring rules (e.g., absolute value > threshold, 7‑day trend > 10‑20%), and subscribing to alerts via SMS, DingTalk, or email.

Real‑time Monitoring: Real‑time stream processing monitors data services, full‑link flow, and indexing. A closed‑loop pipeline (monitoring → detection → resolution) is implemented.

3.4 Algorithm Quality – UDF Unit Testing

UDFs (User‑Defined Functions) are widely used in feature computation for both offline and online pipelines. Testing follows a function‑test model (input → output). Three categories of UDF tests are defined:

Fixed‑rule UDFs: simple input‑output equality checks.

General‑rule UDFs: validation via regex or generic rules.

Model‑based UDFs: evaluate recall/accuracy against a benchmark dataset.

3.5 Algorithm Quality – Feature Column Testing

Feature columns are assembled as DAGs of UDF operators. The compiled DAG is stored as column metadata and executed during runtime. Tests generate feature input sets, run the DAG, and verify outputs against expected distributions and business rules.

3.6 Algorithm Quality – Full‑Graph Index Testing

Full‑graph index tests evaluate row‑level pipelines that combine multiple feature columns (e.g., OCG pipeline). These tests validate the end‑to‑end behavior of the indexing engine.

3.7 Effect Baselines

Effect baselines are built for critical modules such as the intent analysis component. Dynamic test sets are maintained in a data platform; expected results are curated by evaluation engineers. Monitoring includes rule checks (e.g., TOP‑N recall constraints) and noise reduction (retries, filtering unrelated cards).

Search chain effect baseline.

Intent module QP effect baseline.

3.8 Impact Assessment

Automatic impact assessment calculates the proportion of total SQV (search query volume) affected by anomalies, using layered sampling and noise removal techniques.

3.9 Dashboard

Effect metrics are integrated into a minute‑level monitoring dashboard, providing real‑time visibility into algorithm health.

4. User Experience – Badcase Mining

Bad cases are extracted from high‑jump traffic and hot topics. The process includes:

High‑jump badcase detection via competitor comparison.

Timeliness analysis of hot content across platforms.

Twice‑daily remediation cycles, leading to a noticeable reduction in badcase ratio.

5. Public Opinion Closed‑Loop

Using the Youku public‑opinion platform, user feedback (e.g., "cannot find" or "search poorly") is aggregated and addressed, solving five major badcase categories.