Evaluation Framework and Methodology for OPPO XiaoBu AI Assistant

The presentation begins by defining the concept and purpose of evaluation, emphasizing that evaluation combines assessment and measurement to quantify product performance and guide optimization.

It outlines five essential evaluation elements: evaluation method, data sampling, dimension and scoring rules, annotation scoring, and the evaluation report.

Evaluation methods are illustrated for both search and voice‑assistant domains, including overall satisfaction (Per‑page), side‑by‑side (SBS), single‑item scoring (PI), and recall/precision metrics.

Data sampling techniques are described, such as random sampling, deduplication sampling, stratified sampling, and vertical sampling, each with its advantages and limitations for covering long‑tail queries.

The article details the selection of evaluation dimensions—legality, spam/low‑quality, intent understanding, relevance, timeliness, ranking, diversity, authority, convenience, and richness—and explains how to define rules and grading tiers for each.

Annotation scoring involves four query‑understanding methods (direct understanding, everyday experience, deep reasoning, and search‑based) and two result‑assessment steps (relevance to user need and dimension compliance).

Reporting requirements stress a one‑page summary containing key metrics, statistical findings, major issues, and background information, tailored to the audience (management, product, or algorithm teams).

The XiaoBu assistant evaluation framework is then introduced, focusing on four core bottlenecks: wake‑up, listening, understanding, and speaking. Specific assessments include wake‑up rate, ASR word‑error and sentence‑error rates, intent recall/precision, user satisfaction, and TTS MOS scores, with considerations for multilingual and multi‑device scenarios.

Finally, the Q&A section addresses practical concerns such as handling long‑tail sampling, boundary cases, the role of human annotators, and the most critical metrics for different evaluation goals.