Why Accuracy Misleads and How to Pick Better ML Evaluation Metrics

Scenario Description

During model evaluation, classification, ranking, and regression problems require different metrics. However, many metrics only reflect a part of a model's ability; improper combination can hide issues or lead to wrong conclusions. Using Hulu business cases, we illustrate several evaluation scenarios.

Problem Description

Limitations of Accuracy

Trade‑off between Precision and Recall

The “surprise” of Root Mean Square Error (RMSE)

Key Points

Accuracy, Precision, Recall, RMSE

Answer and Analysis

1. Limitations of Accuracy

Hulu’s luxury‑goods advertisers want to target luxury users. Hulu built a classification model using data from a DMP, achieving >95% overall accuracy, yet most ads were shown to non‑luxury users. Accuracy is defined as the proportion of correctly classified samples.

When class distribution is highly imbalanced, a model can achieve high accuracy by predicting the majority class. In Hulu’s case, luxury users are a small fraction, so overall accuracy does not reflect performance on that segment. Using average accuracy (mean per‑class accuracy) is more effective.

The question is open‑ended; besides metric choice, issues like over/under‑fitting, train‑test split, and distribution shift may also affect results, but metric selection is the most evident factor.

2. Precision‑Recall Trade‑off

Hulu’s fuzzy video search returns top‑5 results with high precision, yet users often cannot find less popular videos. Precision is the proportion of true positives among predicted positives; recall is the proportion of true positives among all actual positives.

In ranking models, Precision@N and Recall@N are used. High precision at top‑5 does not guarantee sufficient recall for the whole set, leading to missing relevant items. Evaluating both precision and recall across different N, and plotting a Precision‑Recall curve, provides a fuller picture.

The P‑R curve’s x‑axis is recall, y‑axis is precision; each point corresponds to a threshold. Comparing curves of different models shows trade‑offs.

Additional metrics such as F1‑score and ROC curve can also reflect performance.

3. The “Surprise” of RMSE

Hulu wants to predict viewership trends of TV shows. A regression model yields a high RMSE even though 95% of predictions have <1% error. RMSE is sensitive to outliers; a few extreme errors can inflate the metric.

Solutions: filter out noise points during preprocessing, improve the model to handle outliers, or use more robust metrics such as MAPE (Mean Absolute Percentage Error), which normalizes errors and reduces outlier impact.

Summary and Extension

Through three hypothetical Hulu scenarios, we demonstrated the importance of selecting appropriate evaluation metrics. No single metric suffices; a complementary set of metrics enables comprehensive model assessment and helps identify and resolve issues in real‑world applications.

Next Question Preview

Feature Engineering – Numerical Features