Mastering Core ML Evaluation Metrics: From Bias‑Variance to ROC Curves

1. Bias‑Variance Trade‑off

This is a fundamental concept in machine learning theory; most algorithms, including deep learning, strive to balance bias and variance, as illustrated by the accompanying diagram.

2. Gini Impurity and Entropy

Gini impurity (a measure of heterogeneity) and entropy (a measure of randomness) are both used to assess node impurity in decision trees.

Gini impurity is usually easier to compute because entropy involves logarithmic calculations.

3. Precision‑Recall Curve

The precision‑recall curve shows the trade‑off between precision and recall at different thresholds; a larger area under the curve indicates both high precision and high recall.

Precision’s denominator varies with false positives, while recall’s denominator (the total number of true instances) remains constant.

This explains why precision may fluctuate toward the end of the curve while recall stays stable.

4. ROC Curve

The ROC curve visualizes a classifier’s performance across all classification thresholds.

True Positive Rate

False Positive Rate and the area under the curve (AUC), which serves as a performance metric; higher AUC indicates a better model.

5. Elbow Curve (K‑Means)

Used to select the optimal number of clusters in K‑means; the within‑cluster sum of squares (WCSS) decreases as the number of clusters increases, forming an “elbow” shape.

6. Scree Plot (PCA)

Helps visualize the percentage of variance explained by each principal component after performing PCA on high‑dimensional data, guiding the selection of a suitable number of components.

7. Linear and Logistic Regression Curves

For linearly separable data, both linear regression and logistic regression can serve as decision boundaries.

In logistic regression, because there are typically only two classes, a simple linear decision boundary may not suffice; the probability transition region is better modeled with a sigmoid transformation, producing a smooth S‑shaped curve.

8. Support Vector Machine (Geometric Understanding)

9. Standard Normal Distribution Rule (Z‑distribution)

The empirical rule states that about 68% of data lie within one standard deviation, 95% within two, and 99.7% within three standard deviations of the mean.

10. Student’s t‑Distribution

The t‑distribution resembles the normal curve but has heavier tails; it is used for small sample sizes, approaching the normal distribution as the sample size exceeds about 30.

Source: https://mp.weixin.qq.com/s/bTNI7_jOaoyR1oRu5zJg0w