Practical Guide to Evaluating Recommendation Systems: Metrics, Scenarios, and Best Practices

Recommendation systems are one of the most common and important technologies on the Internet today, powering content delivery in apps, websites, and mini‑programs.

Building a high‑quality recommender is valuable but challenging; evaluating its performance is essential because without scientific metrics there is no clear direction for improvement.

1. Choose Evaluation Methods According to the Recommendation Scenario

The scenario determines the relationship between content type, presentation format, and user needs. For example, long‑form video recommendations (movies) require fast, accurate results to help users pick a film, while short‑form video feeds (TikTok‑style) prioritize diversity and novelty to keep users engaged.

2. Factors Influencing Evaluation

Slot type: Fixed‑slot Top‑N recommendations (similar to search results) often use CTR, NDCG, MRR, MAP, etc., while feed‑style infinite slots rely more on exposure‑click ratios such as PV‑CTR, UV‑CTR.

Business model: E‑commerce platforms focus on transaction‑related metrics (order rate, GMV proportion), whereas ad‑driven apps emphasize user dwell time, click volume, and ad revenue.

Offline vs. online evaluation: Offline methods use static datasets and metrics like MSE, RMSE, R‑squared, while online A/B testing provides real‑time feedback but can be affected by external factors and may not reflect module‑level performance.

Ecosystem balance: When content comes from many UGC/PGC sources, metrics such as source coverage, diversity, novelty, and even the Gini coefficient are needed to avoid a “rich‑get‑richer” effect.

Human nature: Purely catering to user impulses can lead to low‑quality content; guiding users toward higher‑quality material requires metrics beyond clicks, such as serendipity, novelty, and long‑term satisfaction.

3. Practical Metric Setting Methods

Method 1: Define different metrics for distinct user segments (e.g., new users vs. premium users) to capture varied goals like rapid conversion or long‑term retention.

Method 2: Assign metrics based on recommendation placement (homepage banner, feed, detail‑page suggestions) using appropriate measures such as precision‑recall for related items or CTR for top‑banner slots.

Method 3: Combine multiple metrics with weighted sums to balance commercial, user‑experience, and technical considerations; the exact weighting depends on product positioning and lifecycle stage.

4. Summary of Common Metric Types

Conversion‑type metrics: exposure‑click rate, PV‑CTR, UV‑CTR, add‑to‑cart rate, share rate, purchase rate, AUC, etc.

Content‑quality metrics: diversity, novelty, timeliness, confidence/trust.

User‑satisfaction metrics: retention, dwell time, completion rate, average reading time, engagement, serendipity.

In a given scenario, avoid using too many metrics; select a few core indicators that reflect the primary goals, but never rely on a single metric as it can lead to suboptimal optimization.

Author Bio

Chen Yunwen, Founder & CEO of Daguang Data, Ph.D. in Computer Science from Fudan University, expert in AI with numerous publications in IEEE Transactions, SIGKDD, and other top venues; experienced in machine learning, NLP, search and recommendation systems.