Understanding Confusion Matrix, ROC Curve, and Evaluation Metrics for Binary Classification Models

Model evaluation is a crucial step after developing a binary classification model, and this article introduces the main evaluation methods.

Confusion Matrix

The confusion matrix displays the counts of true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN), as illustrated in the figure below.

From the confusion matrix, several metrics can be derived:

Accuracy measures the proportion of correctly classified samples, but it can be misleading in imbalanced datasets.

Precision is the proportion of predicted positive samples that are truly positive, while recall is the proportion of actual positive samples that are correctly predicted. Their trade‑off is illustrated with a watermelon example.

In credit scoring, high precision is preferred to avoid losses, whereas in credit marketing, high recall is desired to reach more customers.

To balance precision and recall, the F‑score (especially F1) is used, defined as the harmonic mean of precision and recall.

F1 ranges from 0 to 1, with higher values indicating better model performance.

ROC Curve

The ROC curve evaluates a classifier’s performance across all possible probability thresholds, plotting the false positive rate (FPR) against the true positive rate (TPR). The area under the curve (AUC) quantifies overall discriminative ability.

By analyzing the ROC curve, one can assess how threshold adjustments affect TP, FP, TN, and FN, and select models that perform well regardless of the chosen threshold.

Choosing appropriate evaluation metrics depends on the specific business context and data distribution.